fk these ai
This commit is contained in:
152
conductor/tracks/test_integrity_audit_20260307/plan.md
Normal file
152
conductor/tracks/test_integrity_audit_20260307/plan.md
Normal file
@@ -0,0 +1,152 @@
|
||||
# Plan: Test Integrity Audit & Intent Documentation
|
||||
|
||||
## Phase 1: Pattern Detection & Analysis
|
||||
Focus: Identify test files with simplification patterns
|
||||
|
||||
### Tasks
|
||||
|
||||
- [ ] Task 1.1: Analyze tests/test_gui_updates.py for simplification
|
||||
- File: tests/test_gui_updates.py
|
||||
- Check: Mock patching changes, removed assertions, skip additions
|
||||
- Reference: git diff shows changes to mock structure (lines 28-48)
|
||||
- Intent: Verify _refresh_api_metrics and _process_pending_gui_tasks work correctly
|
||||
|
||||
- [ ] Task 1.2: Analyze tests/test_gui_phase3.py for simplification
|
||||
- File: tests/test_gui_phase3.py
|
||||
- Check: Collapsed structure, removed test coverage
|
||||
- Reference: 22 lines changed, structure simplified
|
||||
- Intent: Verify track proposal editing, conductor setup scanning, track creation
|
||||
|
||||
- [ ] Task 1.3: Analyze tests/test_conductor_engine_v2.py for simplification
|
||||
- File: tests/test_conductor_engine_v2.py
|
||||
- Check: Engine execution changes, assertion removal
|
||||
- Reference: 4 lines changed
|
||||
|
||||
- [ ] Task 1.4: Analyze tests/test_gui2_performance.py for inappropriate skips
|
||||
- File: tests/test_gui2_performance.py
|
||||
- Check: New skip conditions, weakened assertions
|
||||
- Reference: Added skip for zero FPS (line 65-66)
|
||||
- Intent: Verify GUI maintains 30+ FPS baseline
|
||||
|
||||
- [ ] Task 1.5: Run git blame analysis on modified test files
|
||||
- Command: git blame tests/ --since="2026-02-07" to identify AI-modified tests
|
||||
- Identify commits from AI agents (look for specific commit messages)
|
||||
|
||||
- [ ] Task 1.6: Analyze simulation tests for simplification (test_sim_*.py)
|
||||
- Files: test_sim_base.py, test_sim_context.py, test_sim_tools.py, test_sim_execution.py, test_sim_ai_settings.py
|
||||
- These tests simulate user actions - critical for regression detection
|
||||
- Check: Puppeteer patterns, mock overuse, assertion removal
|
||||
|
||||
- [ ] Task 1.7: Analyze live workflow tests
|
||||
- Files: test_live_workflow.py, test_live_gui_integration_v2.py
|
||||
- These tests verify end-to-end user flows
|
||||
- Check: End-to-end verification integrity
|
||||
|
||||
- [ ] Task 1.8: Analyze major feature tests (core application)
|
||||
- Files: test_dag_engine.py, test_conductor_engine_v2.py, test_mma_orchestration_gui.py
|
||||
- Core orchestration - any simplification is critical
|
||||
- Check: Engine behavior verification
|
||||
|
||||
- [ ] Task 1.9: Analyze GUI feature tests
|
||||
- Files: test_gui2_layout.py, test_gui2_events.py, test_gui2_mcp.py, test_gui_symbol_navigation.py
|
||||
- UI functionality - verify visual feedback is tested
|
||||
- Check: UI state verification
|
||||
|
||||
## Phase 2: Test Intent Documentation
|
||||
Focus: Add docstrings and anti-simplification comments to all audited tests
|
||||
|
||||
### Tasks
|
||||
|
||||
- [ ] Task 2.1: Add docstrings to test_gui_updates.py tests
|
||||
- File: tests/test_gui_updates.py
|
||||
- Tests: test_telemetry_data_updates_correctly, test_performance_history_updates, test_gui_updates_on_event
|
||||
- Add: Docstring explaining what behavior each test verifies
|
||||
- Add: "ANTI-SIMPLIFICATION" comments on critical assertions
|
||||
|
||||
- [ ] Task 2.2: Add docstrings to test_gui_phase3.py tests
|
||||
- File: tests/test_gui_phase3.py
|
||||
- Tests: test_track_proposal_editing, test_conductor_setup_scan, test_create_track
|
||||
- Add: Docstring explaining track management verification purpose
|
||||
|
||||
- [ ] Task 2.3: Add docstrings to test_conductor_engine_v2.py tests
|
||||
- File: tests/test_conductor_engine_v2.py
|
||||
- Check all test functions for missing docstrings
|
||||
- Add: Verification intent for each test
|
||||
|
||||
- [ ] Task 2.4: Add docstrings to test_gui2_performance.py tests
|
||||
- File: tests/test_gui2_performance.py
|
||||
- Tests: test_performance_baseline_check
|
||||
- Clarify: Why 30 FPS threshold matters (not arbitrary)
|
||||
|
||||
- [ ] Task 2.5: Add docstrings to simulation tests (test_sim_*.py)
|
||||
- Files: test_sim_base.py, test_sim_context.py, test_sim_tools.py, test_sim_execution.py
|
||||
- These tests verify user action simulation - add purpose documentation
|
||||
- Document: What user flows are being simulated
|
||||
|
||||
- [ ] Task 2.6: Add docstrings to live workflow tests
|
||||
- Files: test_live_workflow.py, test_live_gui_integration_v2.py
|
||||
- Document: What end-to-end scenarios are being verified
|
||||
|
||||
- [ ] Task 2.7: Add docstrings to major feature tests
|
||||
- Files: test_dag_engine.py, test_conductor_engine_v2.py
|
||||
- Document: What core orchestration behaviors are verified
|
||||
|
||||
## Phase 3: Test Restoration
|
||||
Focus: Restore improperly removed assertions and fix inappropriate skips
|
||||
|
||||
### Tasks
|
||||
|
||||
- [ ] Task 3.1: Restore assertions in test_gui_updates.py
|
||||
- File: tests/test_gui_updates.py
|
||||
- Issue: Check if test_gui_updates_on_event still verifies actual behavior
|
||||
- Verify: _on_api_event triggers proper state changes
|
||||
|
||||
- [ ] Task 3.2: Evaluate skip necessity in test_gui2_performance.py
|
||||
- File: tests/test_gui2_performance.py:65-66
|
||||
- Issue: Added skip for zero FPS
|
||||
- Decision: Document why skip exists or restore assertion
|
||||
|
||||
- [ ] Task 3.3: Verify test_conductor_engine tests still verify engine behavior
|
||||
- File: tests/test_conductor_engine_v2.py
|
||||
- Check: No assertions replaced with mocks
|
||||
|
||||
- [ ] Task 3.4: Restore assertions in simulation tests if needed
|
||||
- Files: test_sim_*.py
|
||||
- Check: User action simulations still verify actual behavior
|
||||
|
||||
- [ ] Task 3.5: Restore assertions in live workflow tests if needed
|
||||
- Files: test_live_workflow.py, test_live_gui_integration_v2.py
|
||||
- Check: End-to-end flows still verify complete behavior
|
||||
|
||||
## Phase 4: Anti-Simplification Pattern Application
|
||||
Focus: Add permanent markers to prevent future simplification
|
||||
|
||||
### Tasks
|
||||
|
||||
- [ ] Task 4.1: Add ANTI-SIMPLIFICATION header to test_gui_updates.py
|
||||
- File: tests/test_gui_updates.py
|
||||
- Add: Module-level comment explaining these tests verify core GUI state management
|
||||
|
||||
- [ ] Task 4.2: Add ANTI-SIMPLIFICATION header to test_gui_phase3.py
|
||||
- File: tests/test_gui_phase3.py
|
||||
- Add: Module-level comment explaining these tests verify conductor integration
|
||||
|
||||
- [ ] Task 4.3: Add ANTI-SIMPLIFICATION header to test_conductor_engine_v2.py
|
||||
- File: tests/test_conductor_engine_v2.py
|
||||
- Add: Module-level comment explaining these tests verify engine execution
|
||||
|
||||
- [ ] Task 4.4: Run full test suite to verify no regressions
|
||||
- Command: uv run pytest tests/test_gui_updates.py tests/test_gui_phase3.py tests/test_conductor_engine_v2.py -v
|
||||
- Verify: All tests pass with restored assertions
|
||||
|
||||
## Phase 5: Checkpoint & Documentation
|
||||
Focus: Document findings and create checkpoint
|
||||
|
||||
- [ ] Task 5.1: Document all simplification patterns found
|
||||
- Create: findings.md in track directory
|
||||
- List: Specific patterns detected and actions taken
|
||||
|
||||
- [ ] Task 5.2: Create checkpoint commit
|
||||
- Commit message: conductor(checkpoint): Test integrity audit complete
|
||||
|
||||
## Checkpoint: [TO BE ADDED]
|
||||
Reference in New Issue
Block a user