# Track Specification: Test Architecture Integrity & Simulation Audit

## Overview
Comprehensive audit of testing infrastructure and simulation framework to identify false positive risks, coverage gaps, and simulation fidelity issues. This analysis was triggered by a request to review how tests and simulations are setup, whether tests can report passing grades when they fail, and if simulations are rigorous enough or are just rough emulators.

## Current State Audit (as of 20260304)

### Already Implemented (DO NOT re-implement)
- **Testing Infrastructure** (	ests/conftest.py):
  - live_gui fixture for session-scoped GUI lifecycle management
  - Process cleanup with kill_process_tree()
  - VerificationLogger for diagnostic logging
  - Artifact isolation to 	ests/artifacts/ and 	ests/logs/

- **Simulation Framework** (simulation/):
  - sim_base.py: Base simulation class with setup/teardown
  - workflow_sim.py: Workflow orchestration
  - sim_context.py, sim_ai_settings.py, sim_tools.py, sim_execution.py
  - user_agent.py: Simulated human agent
- **Testing Infrastructure** (tests/conftest.py):
  - live_gui fixture for session-scoped GUI lifecycle management
  - Process cleanup with kill_process_tree()
  - VerificationLogger for diagnostic logging
  - Artifact isolation to tests/artifacts/ and tests/logs/
  - Ban on arbitrary core mocking
- **Mock Provider** (tests/mock_gemini_cli.py):
  - Keyword-based response routing
  - JSON-L protocol matching real CLI output

#### Critical False Positive Risks Identified
1. **Mock Provider Always Returns Success**: Never validates input, never produces errors, never tests failure paths
2. **Auto-Approval Pattern**: All HITL gates auto-clicked, never verifying dialogs appear or rejection flows
3. **Substring-Based Assertions**: Only check existence of content, not validity or structure
4. **State Existence Only**: Tests check fields exist but not their correctness or invariants
5. **No Negative Path Testing**: No coverage for rejection, timeout, malformed input, concurrent access
6. **No Visual Verification**: Tests verify logical state via Hook API but never check what's actually rendered
7. **No State Machine Validation**: No verification that status transitions are legal or complete

#### Simulation Rigor Gaps Identified
1. **No Real-Time Latency Simulation**: Fixed delays don't model variable LLM/network latency
2. **No Human-Like Behavior**: Instant actions, no typing speed, hesitation, mistakes, or task switching
3. **Arbitrary Polling Intervals**: 1-second polls may miss transient states
4. **Mock CLI Redirection**: Bypasses subprocess spawning, environment passing, and process cleanup paths
5. **No Stress Testing**: No load testing, no edge case bombardment

#### Test Coverage Gaps
- No tests for approval dialog rejection flows
- No tests for malformed LLM response handling
- No tests for network timeout/failure scenarios
- No tests for concurrent duplicate requests
- No tests for out-of-order event sequences
- No thread-safety tests for shared resources
- No visual rendering verification (modal visibility, text overflow, color schemes)

#### Structural Testing Contract Gaps
- Missing rule requiring negative path testing
- Missing rule requiring state validation beyond existence
- Missing rule requiring visual verification
- No enforcement for thread-safety testing

## Goals

1. Document all identified testing pitfalls with severity ratings (HIGH/MEDIUM/LOW)
2. Create actionable recommendations for each identified issue
3. Map existing test coverage gaps to specific missing test files
4. Provide architecture recommendations for simulation framework enhancements

## Functional Requirements

- [ ] Document all false positive risks in a structured format
- [ ] Document all simulation fidelity gaps in a structured format
- [ ] Create severity matrix for each issue
- [ ] Generate list of missing test cases by category
- [ ] Provide concrete examples of how current tests would pass despite bugs
- [ ] Provide concrete examples of how simulations would miss UX issues

## Non-Functional Requirements

- Report must include author attribution (GLM-4.7) and derivation methodology
- Analysis must cite specific file paths and line numbers where applicable
- Recommendations must be prioritized by impact and implementation effort

## Architecture Reference

Refer to:
- docs/guide_simulations.md - Current simulation contract and patterns
- docs/guide_mma.md - MMA orchestration architecture
- docs/guide_architecture.md - Thread domains, event system, HITL mechanism
- conductor/tracks/*/spec.md - Existing track specifications for consistency

## Out of Scope

- Implementing the actual test fixes (that's for subsequent tracks)
- Refactoring the simulation framework (documenting only)
- Modifying the mock provider (analyzing only)
- Writing new tests (planning phase for future tracks)
