manual_slop/conductor/tracks/test_integrity_audit_20260307/plan.md

Plan: Test Integrity Audit & Intent Documentation

Phase 1: Pattern Detection & Analysis

Focus: Identify test files with simplification patterns

Tasks

• Task 1.1: Analyze tests/test_gui_updates.py for simplification

  • File: tests/test_gui_updates.py
  • Check: Mock patching changes, removed assertions, skip additions
  • Reference: git diff shows changes to mock structure (lines 28-48)
  • Intent: Verify _refresh_api_metrics and _process_pending_gui_tasks work correctly

• Task 1.2: Analyze tests/test_gui_phase3.py for simplification

  • File: tests/test_gui_phase3.py
  • Check: Collapsed structure, removed test coverage
  • Reference: 22 lines changed, structure simplified
  • Intent: Verify track proposal editing, conductor setup scanning, track creation

• Task 1.3: Analyze tests/test_conductor_engine_v2.py for simplification

  • File: tests/test_conductor_engine_v2.py
  • Check: Engine execution changes, assertion removal
  • Reference: 4 lines changed

• Task 1.4: Analyze tests/test_gui2_performance.py for inappropriate skips

  • File: tests/test_gui2_performance.py
  • Check: New skip conditions, weakened assertions
  • Reference: Added skip for zero FPS (line 65-66)
  • Intent: Verify GUI maintains 30+ FPS baseline

• Task 1.5: Run git blame analysis on modified test files

  • Command: git blame tests/ --since="2026-02-07" to identify AI-modified tests
  • Identify commits from AI agents (look for specific commit messages)

• Task 1.6: Analyze simulation tests for simplification (test_sim_*.py)

  • Files: test_sim_base.py, test_sim_context.py, test_sim_tools.py, test_sim_execution.py, test_sim_ai_settings.py
  • These tests simulate user actions - critical for regression detection
  • Check: Puppeteer patterns, mock overuse, assertion removal

• Task 1.7: Analyze live workflow tests

  • Files: test_live_workflow.py, test_live_gui_integration_v2.py
  • These tests verify end-to-end user flows
  • Check: End-to-end verification integrity

• Task 1.8: Analyze major feature tests (core application)

  • Files: test_dag_engine.py, test_conductor_engine_v2.py, test_mma_orchestration_gui.py
  • Core orchestration - any simplification is critical
  • Check: Engine behavior verification

• Task 1.9: Analyze GUI feature tests

  • Files: test_gui2_layout.py, test_gui2_events.py, test_gui2_mcp.py, test_gui_symbol_navigation.py
  • UI functionality - verify visual feedback is tested
  • Check: UI state verification

Phase 2: Test Intent Documentation

Focus: Add docstrings and anti-simplification comments to all audited tests

Tasks

• Task 2.1: Add docstrings to test_gui_updates.py tests

  • File: tests/test_gui_updates.py
  • Tests: test_telemetry_data_updates_correctly, test_performance_history_updates, test_gui_updates_on_event
  • Add: Docstring explaining what behavior each test verifies
  • Add: "ANTI-SIMPLIFICATION" comments on critical assertions

• Task 2.2: Add docstrings to test_gui_phase3.py tests

  • File: tests/test_gui_phase3.py
  • Tests: test_track_proposal_editing, test_conductor_setup_scan, test_create_track
  • Add: Docstring explaining track management verification purpose

• Task 2.3: Add docstrings to test_conductor_engine_v2.py tests

  • File: tests/test_conductor_engine_v2.py
  • Check all test functions for missing docstrings
  • Add: Verification intent for each test

• Task 2.4: Add docstrings to test_gui2_performance.py tests

  • File: tests/test_gui2_performance.py
  • Tests: test_performance_baseline_check
  • Clarify: Why 30 FPS threshold matters (not arbitrary)

• Task 2.5: Add docstrings to simulation tests (test_sim_*.py)

  • Files: test_sim_base.py, test_sim_context.py, test_sim_tools.py, test_sim_execution.py
  • These tests verify user action simulation - add purpose documentation
  • Document: What user flows are being simulated

• Task 2.6: Add docstrings to live workflow tests

  • Files: test_live_workflow.py, test_live_gui_integration_v2.py
  • Document: What end-to-end scenarios are being verified

• Task 2.7: Add docstrings to major feature tests

  • Files: test_dag_engine.py, test_conductor_engine_v2.py
  • Document: What core orchestration behaviors are verified

Phase 3: Test Restoration

Focus: Restore improperly removed assertions and fix inappropriate skips

Tasks

• Task 3.1: Restore assertions in test_gui_updates.py

  • File: tests/test_gui_updates.py
  • Issue: Check if test_gui_updates_on_event still verifies actual behavior
  • Verify: _on_api_event triggers proper state changes

• Task 3.2: Evaluate skip necessity in test_gui2_performance.py

  • File: tests/test_gui2_performance.py:65-66
  • Issue: Added skip for zero FPS
  • Decision: Document why skip exists or restore assertion

• Task 3.3: Verify test_conductor_engine tests still verify engine behavior

  • File: tests/test_conductor_engine_v2.py
  • Check: No assertions replaced with mocks

• Task 3.4: Restore assertions in simulation tests if needed

  • Files: test_sim_*.py
  • Check: User action simulations still verify actual behavior

• Task 3.5: Restore assertions in live workflow tests if needed

  • Files: test_live_workflow.py, test_live_gui_integration_v2.py
  • Check: End-to-end flows still verify complete behavior

Phase 4: Anti-Simplification Pattern Application

Focus: Add permanent markers to prevent future simplification

Tasks

• Task 4.1: Add ANTI-SIMPLIFICATION header to test_gui_updates.py

  • File: tests/test_gui_updates.py
  • Add: Module-level comment explaining these tests verify core GUI state management

• Task 4.2: Add ANTI-SIMPLIFICATION header to test_gui_phase3.py

  • File: tests/test_gui_phase3.py
  • Add: Module-level comment explaining these tests verify conductor integration

• Task 4.3: Add ANTI-SIMPLIFICATION header to test_conductor_engine_v2.py

  • File: tests/test_conductor_engine_v2.py
  • Add: Module-level comment explaining these tests verify engine execution

• Task 4.4: Add ANTI-SIMPLIFICATION header to simulation tests

  • Files: test_sim_base.py, test_sim_context.py, test_sim_tools.py, test_sim_execution.py
  • Add: Module-level comments explaining these tests verify user action simulations
  • These are CRITICAL - they detect regressions in user-facing functionality

• Task 4.5: Add ANTI-SIMPLIFICATION header to live workflow tests

  • Files: test_live_workflow.py, test_live_gui_integration_v2.py
  • Add: Module-level comments explaining these tests verify end-to-end flows

• Task 4.6: Run full test suite to verify no regressions

  • Command: uv run pytest tests/test_gui_updates.py tests/test_gui_phase3.py tests/test_conductor_engine_v2.py -v
  • Verify: All tests pass with restored assertions

Phase 5: Checkpoint & Documentation

Focus: Document findings and create checkpoint

• Task 5.1: Document all simplification patterns found

  • Create: findings.md in track directory
  • List: Specific patterns detected and actions taken

• Task 5.2: Create checkpoint commit

  • Commit message: conductor(checkpoint): Test integrity audit complete

Checkpoint: [TO BE ADDED]