conductor(track): Document real bug - self.engine gets overwritten

This commit is contained in:
2026-05-07 07:39:14 -04:00
parent afef5b5791
commit 9099b02002
2 changed files with 29 additions and 36 deletions
@@ -4,16 +4,16 @@
- [x] Task: Compare test_mma_concurrent_tracks_sim.py with working tests - Same fixture, same pattern
- [x] Task: Check subprocess/port cleanup between live_gui fixture tests - Not cleanup issue, single test also times out
- [x] Task: Verify get_mma_workers() API returns expected format - Returns {"workers": Dict[str, str]}
- [x] Task: Run isolated concurrent test with verbose debugging - Workers ARE running (confirmed via user screenshot)
- [x] Task: Run isolated concurrent test with verbose debugging - Confirmed: only ONE worker appears when starting two tracks
## Phase 2: Identify Root Cause
- [x] Task: Workers appear in active_streams but test times out - Workers run but completion detection fails
- [x] Task: Issue is NOT fixture cleanup - Even isolated runs timeout
- [x] Task: Issue is likely in btn_mma_accept_tracks background thread - Accept tracks completes (proposed_tracks works), but something after that hangs
## Phase 2: Identify Root Cause - CONFIRMED BUG
- [x] Task: Root cause found: `self.engine` is single attribute that gets overwritten
- [x] Task: When second track starts, it overwrites `self.engine`, orphaning first track's engine
## Phase 3: Implement Fix
- [ ] Task: Simplify test to avoid complex polling - Just verify workers appear, don't wait for completion
- [ ] Task: Or fix the underlying issue causing the hang
## Phase 3: Implement Fix - NEEDS REAL FIX, NOT TEST SIMPLIFICATION
- [ ] Task: Change `self.engine: Optional[ConductorEngine]` to `self.engines: Dict[str, ConductorEngine]`
- [ ] Task: Update all `self.engine` references to track-specific lookup
- [ ] Task: Verify two concurrent tracks produce two concurrent workers
## Phase 4: Final Verification
- [ ] Task: Run all concurrent MMA tests
@@ -1,37 +1,30 @@
# Track: Fix Concurrent MMA Live GUI Tests
## Problem Statement
The live GUI integration tests for concurrent MMA track execution timeout:
- `test_mma_concurrent_tracks_sim.py::test_mma_concurrent_tracks_execution`
- `test_mma_concurrent_tracks_stress_sim.py::test_mma_concurrent_tracks_stress`
- `test_visual_sim_mma_v2.py::test_mma_complete_lifecycle`
When starting two MMA tracks concurrently via `btn_mma_start_track`, only ONE worker appears instead of two. The test `test_mma_concurrent_tracks_stress_sim.py` correctly identifies this bug:
- Two tracks are started
- But only ONE Tier 3 worker (`mock-ticket-1`) appears
- The second worker never starts
While simpler single-track live GUI tests pass, these concurrent execution tests timeout (300s).
## Root Cause Analysis
In `app_controller.py` `_cb_start_track()`:
```python
self.engine = multi_agent_conductor.ConductorEngine(...)
threading.Thread(target=engine.run, daemon=True).start()
```
## Observations
1. Workers ARE running - both ticket-A-1 and ticket-B-1 appear in `active_streams` (confirmed via user screenshot)
2. The issue is in the test infrastructure's ability to handle concurrent MMA workloads
3. Other live GUI tests work (visual_orchestration, mma_step_mode_sim, system_prompt_sim)
`self.engine` is a SINGLE attribute. When a second track is started, it **overwrites** `self.engine`, orphaning the first track's engine. The first track's engine thread continues running but loses its reference in `self.engine`.
## Investigation Tasks
1. [ ] Compare `test_mma_concurrent_tracks_sim.py` with `test_visual_orchestration.py` to identify differences in test structure
2. [ ] Check if `live_gui` fixture cleanup between concurrent tests causes issues
3. [ ] Investigate if subprocess management from previous tests pollutes subsequent test runs
4. [ ] Analyze timing issues - the concurrent test polls at 1s intervals but workers may complete faster
5. [ ] Verify `get_mma_workers()` API endpoint returns data in expected format
## Required Fix
1. Replace `self.engine: Optional[ConductorEngine]` with `self.engines: Dict[str, ConductorEngine]` - one engine per track
2. Update all references to `self.engine` to use track-specific engine lookup
3. Update `kill_worker`, `pause_mma`, `resume_mma`, `approve_ticket`, `mutate_dag` to target specific track engines
4. Ensure cleanup of engines when tracks complete
## Hypotheses
1. **Subprocess leakage**: Previous test's GUI process isn't fully killed before next test starts
2. **API response format**: `mma_streams` dict may serialize differently than expected by test
3. **Timing race**: Workers complete before completion check loop starts, making completion undetectable
4. **Thread pool exhaustion**: Multiple concurrent tracks exhaust some shared resource
## Impact
This affects ALL concurrent MMA track execution - not just tests. Multiple tracks started simultaneously will interfere with each other.
## Success Criteria
- `test_mma_concurrent_tracks_execution` completes in <60s
- `test_mma_concurrent_tracks_stress` completes in <60s
- No other tests regress
## Approach
1. First, isolate whether the issue is in the test setup (fixture) or test logic
2. Add debugging output to identify exact failure point
3. Fix the root cause - likely requires improving test infrastructure robustness
- Two concurrent tracks produce two concurrent workers
- No regression in single-track execution
- All concurrent tests pass