conductor(track): initialize fix_mma_concurrent_tracks_sim_20260627
Followup track to post_module_taxonomy_de_cruft_20260627 (shippedd74b9822). The 1 remaining test failure in tier-3-live_gui is test_mma_concurrent_tracks_execution. Three of the four stacked root causes were already fixed in commit635ca552(partial fix in the prior session): 1. flat.setdefault(...)[...] = ... on frozen ProjectContext (3 sites) 2. t_data['id'] on Ticket objects (1 site) 3. mock_concurrent_mma.py --resume handling The fourth root cause (2nd track's _start_track_logic never fires) remains unresolved. This track instruments _start_track_logic_result with stderr diagnostics, runs the test in isolation, identifies the failure mode, and fixes it. Per user directive: 'those issues must get resolved we are not sweeping them under the rug'. Per workflow.md §Tier 1 Track Initialization Rules: scope is 1 production file + 1 test mock + 1 report update; 4-6 atomic commits total; no day estimates.
This commit is contained in:
@@ -0,0 +1,52 @@
|
||||
{
|
||||
"track_id": "fix_mma_concurrent_tracks_sim_20260627",
|
||||
"name": "Fix MMA Concurrent Tracks Sim Test (tier-3-live_gui regression)",
|
||||
"status": "active",
|
||||
"type": "fix",
|
||||
"date_created": "2026-06-27",
|
||||
"created_by": "tier2-tech-lead",
|
||||
"blocks": [],
|
||||
"blocked_by": {
|
||||
"post_module_taxonomy_de_cruft_20260627": "shipped (the parent track; this is the followup fix for the 1 remaining tier-3 failure)"
|
||||
},
|
||||
"scope": {
|
||||
"new_files": [
|
||||
"docs/reports/TRACK_COMPLETION_fix_mma_concurrent_tracks_sim_20260627.md"
|
||||
],
|
||||
"modified_files": [
|
||||
"src/app_controller.py",
|
||||
"tests/mock_concurrent_mma.py",
|
||||
"docs/reports/OUTSTANDING_MMA_TEST_FAILURES_20260627.md"
|
||||
],
|
||||
"deleted_files": []
|
||||
},
|
||||
"verification_criteria": [
|
||||
"VC1: tests/test_mma_concurrent_tracks_sim.py::test_mma_concurrent_tracks_execution passes in isolation",
|
||||
"VC2: Tier 3 (tier-3-live_gui) of the batched test suite shows 0 failures",
|
||||
"VC3: No diagnostic stderr lines remain in src/app_controller.py (instrumentation removed)",
|
||||
"VC4: docs/reports/OUTSTANDING_MMA_TEST_FAILURES_20260627.md updated to RESOLVED status",
|
||||
"VC5: docs/reports/TRACK_COMPLETION_fix_mma_concurrent_tracks_sim_20260627.md written",
|
||||
"VC6: No git restore/checkout/reset/stash used during the track (per AGENTS.md HARD BAN)",
|
||||
"VC7: All atomic commits have git notes (per workflow.md Per-Task Commit Protocol)"
|
||||
],
|
||||
"estimated_effort": {
|
||||
"method": "scope (per workflow.md §Tier 1 Track Initialization Rules). NO day estimates.",
|
||||
"scope": "1 task: instrument + diagnose + fix + verify (1 production file + 1 test mock file + 1 report). 3-5 atomic commits."
|
||||
},
|
||||
"risk_register": [
|
||||
"R1 (low): Instrumentation incomplete; failure mode remains hidden - mitigated by adding diagnostics at 3 strategic points (before/after generate_tickets, in except block)",
|
||||
"R2 (medium): Production fix regresses other tests - mitigated by running the targeted tier-3 batched test suite after the fix",
|
||||
"R3 (medium): Mock fix requires deeper understanding of gemini_cli_adapter session reuse - mitigated by reading src/ai_client.py to understand session_id lifecycle",
|
||||
"R4 (low): 30-second test poll may be too short for test infrastructure - mitigated by not changing the poll time; the fix should make the test pass within the existing budget",
|
||||
"R5 (low): Instrumentation leaks into production - mitigated by removing the instrumentation in the same commit that fixes the bug (or follow-up commit)",
|
||||
"R6 (medium): User does not give permission to run the full 11-tier batch - mitigated by running only the targeted tier-3 batch (--tier tier-3-live_gui); ask user for full batch separately"
|
||||
],
|
||||
"out_of_scope": [
|
||||
"Refactoring src/multi_agent_conductor.py (the MMA engine itself)",
|
||||
"Refactoring _cb_accept_tracks or _start_track_logic beyond the minimum fix",
|
||||
"Refactoring tests/mock_concurrent_mma.py beyond the minimum fix",
|
||||
"Adding new MMA concurrent execution tests",
|
||||
"Fixing any other tier failures (RAG flake is pre-existing and out of scope)",
|
||||
"Updating conductor/tracks/post_module_taxonomy_de_cruft_20260627/spec.md (the parent track is SHIPPED)"
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,163 @@
|
||||
# Plan: fix_mma_concurrent_tracks_sim_20260627
|
||||
|
||||
3 phases, 4 tasks, 3-5 atomic commits. Per-task TDD red-first. The "test" is the existing failing test in `tests/test_mma_concurrent_tracks_sim.py`; the "fix" is the production code in `src/app_controller.py` and the mock in `tests/mock_concurrent_mma.py`.
|
||||
|
||||
## Phase 0: Instrument + diagnose (Tier 2, 1 commit)
|
||||
|
||||
**Focus:** Per workflow.md "The Deduction Loop (kill it)", you are allowed to run a failing test at most 2 times in a single investigation. After 2 failures, STOP running the test. Read the code, predict the failure mode, and instrument ALL the relevant state in one pass. So Phase 0 is the instrumentation pass.
|
||||
|
||||
- [ ] **Task 0.1** [Tier 2]: Add stderr diagnostics to `src/app_controller.py:_start_track_logic_result`
|
||||
- WHERE: `src/app_controller.py:4750-4840` (the `_start_track_logic_result` function)
|
||||
- WHAT: Add 3 stderr write/flush calls:
|
||||
1. BEFORE `conductor_tech_lead.generate_tickets(goal, skeletons)` — log title, goal
|
||||
2. AFTER `generate_tickets` returns — log length of `raw_tickets`
|
||||
3. INSIDE the `except` block at line 4831 — log full traceback via `import traceback; traceback.print_exc()`
|
||||
- HOW: `manual-slop_edit_file` surgical edit (3-10 lines per edit)
|
||||
- SAFETY: `uv run -m pytest tests/test_mma_concurrent_tracks_sim.py -v` still parses (py_check_syntax exits 0)
|
||||
- INSTRUMENTATION LIFETIME: This commit is INTERIM. The instrumentation must be removed in Phase 2 once the root cause is identified. (Per AGENTS.md "No Diagnostic Noise in Production".)
|
||||
- [ ] **COMMIT 0.1:** `chore(diag): add stderr instrumentation to _start_track_logic_result` (Tier 2)
|
||||
- [ ] **GIT NOTE:** "Temporary instrumentation to diagnose test_mma_concurrent_tracks_execution failure. Will be removed in the next commit after root cause is identified."
|
||||
|
||||
- [ ] **Task 0.2** [Tier 2]: Run the test in isolation with the instrumentation
|
||||
- HOW: `uv run -m pytest tests/test_mma_concurrent_tracks_sim.py::test_mma_concurrent_tracks_execution -v -s > tests/artifacts/tier2_state/fix_mma_concurrent_tracks_sim_20260627/test_run_0.log 2>&1`
|
||||
- Per workflow.md: redirect to log file (NEVER filter output, NEVER use `head`/`tail`)
|
||||
- Read the log file: `manual-slop_read_file tests/artifacts/tier2_state/fix_mma_concurrent_tracks_sim_20260627/test_run_0.log`
|
||||
- Identify the failure mode for the 2nd track
|
||||
- **DO NOT** run the test more than 2 times in total (workflow.md "Deduction Loop")
|
||||
|
||||
## Phase 1: Fix the root cause (Tier 3, 1-2 commits)
|
||||
|
||||
**Focus:** Based on Phase 0 diagnosis, fix the actual root cause.
|
||||
|
||||
- [ ] **Task 1.1** [Tier 3]: Fix the root cause in `src/app_controller.py` OR `tests/mock_concurrent_mma.py`
|
||||
- **If Phase 0 diagnosis is "mock routing broken for 2nd call"** (cause A in spec):
|
||||
- WHERE: `tests/mock_concurrent_mma.py` (the routing logic at lines 64-90)
|
||||
- WHAT: The `gemini_cli_adapter` reuses the session_id returned by the previous call. So track-b's call comes in with `--resume mock-sprint-A` (the session_id returned by the previous track's sprint call). The mock must handle this case.
|
||||
- HOW: Add a routing case for `if session_id == "mock-sprint-A" and call_n == N: _emit_sprint_ticket("B")` — but ALSO handle the case where the gemini_cli_adapter passes the latest session_id for both the track-b sprint call and the track-b worker call.
|
||||
- The cleanest fix: don't rely on session_id alone. After epic + sprint-A, the next call is ALWAYS track-b sprint (since we only have 2 tracks). Add a per-call counter that maps to (call_n // 2) % 2 for the track index.
|
||||
- **If Phase 0 diagnosis is "production bug" (cause B/C/D in spec):**
|
||||
- WHERE: `src/app_controller.py:_start_track_logic_result` (line 4750-4840)
|
||||
- WHAT: Fix the specific bug (disk I/O, flat dict missing field, silent exception)
|
||||
- HOW: Surgical `manual-slop_edit_file` fix
|
||||
- SAFETY: `uv run -m pytest tests/test_mma_concurrent_tracks_sim.py -v` shows PASS
|
||||
- [ ] **COMMIT 1.1:** `fix(mma_concurrent): fix 2nd track _start_track_logic not firing` (Tier 3)
|
||||
- Commit message body: explain which root cause was identified and what was changed.
|
||||
- [ ] **GIT NOTE:** "Fixes test_mma_concurrent_tracks_execution by <specific fix>."
|
||||
|
||||
- [ ] **Task 1.2** [Tier 2]: Run the test in isolation to verify the fix
|
||||
- HOW: `uv run -m pytest tests/test_mma_concurrent_tracks_sim.py::test_mma_concurrent_tracks_execution -v > tests/artifacts/tier2_state/fix_mma_concurrent_tracks_sim_20260627/test_run_1.log 2>&1`
|
||||
- Read the log file and verify PASS
|
||||
- If still failing, **STOP and report to the user** (per workflow.md "Surrender" anti-pattern is OK only after the 5-step checklist)
|
||||
|
||||
- [ ] **Task 1.3** [Tier 2]: Run the targeted tier-3 batched test suite to verify no regressions
|
||||
- HOW: `uv run python scripts/run_tests_batched.py --tier tier-3-live_gui > tests/artifacts/tier2_state/fix_mma_concurrent_tracks_sim_20260627/test_run_tier3.log 2>&1`
|
||||
- Verify: 0 failures in tier-3
|
||||
- Per workflow.md "Isolated-Pass Verification Fallacy" — the only verification that matters is the batched run, not the isolated run
|
||||
|
||||
## Phase 2: Remove instrumentation + write report (Tier 2, 1-2 commits)
|
||||
|
||||
**Focus:** Clean up the temporary instrumentation and write the end-of-track report.
|
||||
|
||||
- [ ] **Task 2.1** [Tier 2]: Remove the stderr instrumentation from `src/app_controller.py:_start_track_logic_result`
|
||||
- WHERE: `src/app_controller.py:4750-4840` (where the 3 stderr lines were added in Phase 0)
|
||||
- WHAT: Remove the 3 stderr write/flush calls
|
||||
- HOW: `manual-slop_edit_file` surgical edit (3 sites)
|
||||
- SAFETY: `git grep "_start_track_logic_result.*stderr" src/app_controller.py` returns 0 hits
|
||||
- [ ] **COMMIT 2.1:** `chore(cleanup): remove diagnostic instrumentation from _start_track_logic_result` (Tier 2)
|
||||
- [ ] **GIT NOTE:** "Removes the temporary stderr instrumentation added in 0.1. The bug fix is in 1.1; this is cleanup."
|
||||
|
||||
- [ ] **Task 2.2** [Tier 2]: Update `docs/reports/OUTSTANDING_MMA_TEST_FAILURES_20260627.md` to RESOLVED
|
||||
- WHERE: `docs/reports/OUTSTANDING_MMA_TEST_FAILURES_20260627.md` (the "4. UNRESOLVED" section)
|
||||
- WHAT: Replace "⚠️ UNRESOLVED" with "✅ RESOLVED" and add a link to the fixing commit
|
||||
- HOW: `manual-slop_edit_file` surgical edit
|
||||
- [ ] **COMMIT 2.2:** `docs(report): mark OUTSTANDING_MMA_TEST_FAILURES_20260627.md as RESOLVED` (Tier 2)
|
||||
- [ ] **GIT NOTE:** "Per FR8 of the track spec. The MMA concurrent tracks test is now passing in the batched test suite."
|
||||
|
||||
- [ ] **Task 2.3** [Tier 2]: Write `docs/reports/TRACK_COMPLETION_fix_mma_concurrent_tracks_sim_20260627.md`
|
||||
- WHERE: `docs/reports/TRACK_COMPLETION_fix_mma_concurrent_tracks_sim_20260627.md` (new file)
|
||||
- WHAT: Follow the precedent of `TRACK_COMPLETION_post_module_taxonomy_de_cruft_20260627.md`:
|
||||
- Executive summary
|
||||
- 3 root causes already fixed in 635ca552
|
||||
- The 1 root cause fixed in this track
|
||||
- Files changed
|
||||
- Verification results
|
||||
- Suggested next steps
|
||||
- HOW: `Write` tool to create the file
|
||||
- [ ] **COMMIT 2.3:** `docs(reports): TRACK_COMPLETION_fix_mma_concurrent_tracks_sim_20260627` (Tier 2)
|
||||
- [ ] **GIT NOTE:** "End-of-track report. Track is complete; tier-3 of post_module_taxonomy_de_cruft_20260627 is now PASS."
|
||||
|
||||
- [ ] **Task 2.4** [Tier 2]: Update `conductor/tracks/fix_mma_concurrent_tracks_sim_20260627/state.toml` to status = "completed"
|
||||
- WHERE: `conductor/tracks/fix_mma_concurrent_tracks_sim_20260627/state.toml`
|
||||
- WHAT: Set `[meta].status = "completed"`, `[meta].current_phase = "complete"`, fill in task commit SHAs
|
||||
- HOW: `Write` tool
|
||||
- [ ] **COMMIT 2.4:** `conductor(state): fix_mma_concurrent_tracks_sim_20260627 SHIPPED` (Tier 2)
|
||||
- [ ] **GIT NOTE:** "Track SHIPPED. All 7 VCs pass. Tier-3 of the parent track is now PASS."
|
||||
|
||||
## Commit Log (Expected, 4-6 atomic commits)
|
||||
|
||||
1. (Phase 0) `chore(diag): add stderr instrumentation to _start_track_logic_result` (Tier 2)
|
||||
2. (Phase 1) `fix(mma_concurrent): fix 2nd track _start_track_logic not firing` (Tier 3)
|
||||
3. (Phase 2) `chore(cleanup): remove diagnostic instrumentation from _start_track_logic_result` (Tier 2)
|
||||
4. (Phase 2) `docs(report): mark OUTSTANDING_MMA_TEST_FAILURES_20260627.md as RESOLVED` (Tier 2)
|
||||
5. (Phase 2) `docs(reports): TRACK_COMPLETION_fix_mma_concurrent_tracks_sim_20260627` (Tier 2)
|
||||
6. (Phase 2) `conductor(state): fix_mma_concurrent_tracks_sim_20260627 SHIPPED` (Tier 2)
|
||||
|
||||
Plus per-task plan-update commits per workflow.md.
|
||||
|
||||
## Verification Commands
|
||||
|
||||
```bash
|
||||
# Phase 0: Run the test in isolation with instrumentation
|
||||
uv run -m pytest tests/test_mma_concurrent_tracks_sim.py::test_mma_concurrent_tracks_execution -v -s > tests/artifacts/tier2_state/fix_mma_concurrent_tracks_sim_20260627/test_run_0.log 2>&1
|
||||
|
||||
# Phase 1: Run the test in isolation after the fix
|
||||
uv run -m pytest tests/test_mma_concurrent_tracks_sim.py::test_mma_concurrent_tracks_execution -v > tests/artifacts/tier2_state/fix_mma_concurrent_tracks_sim_20260627/test_run_1.log 2>&1
|
||||
|
||||
# Phase 1: Run the targeted tier-3 batched suite
|
||||
uv run python scripts/run_tests_batched.py --tier tier-3-live_gui > tests/artifacts/tier2_state/fix_mma_concurrent_tracks_sim_20260627/test_run_tier3.log 2>&1
|
||||
|
||||
# Phase 2 (optional, ASK USER FIRST per user directive): Run the full 11-tier batch
|
||||
uv run python scripts/run_tests_batched.py > tests/artifacts/tier2_state/fix_mma_concurrent_tracks_sim_20260627/test_run_full.log 2>&1
|
||||
|
||||
# Verify VC3: No diagnostic lines in production
|
||||
git grep "_start_track_logic_result.*stderr" src/app_controller.py
|
||||
# Expect: 0 hits
|
||||
|
||||
# Verify VC4: Report is updated
|
||||
grep "RESOLVED" docs/reports/OUTSTANDING_MMA_TEST_FAILURES_20260627.md
|
||||
# Expect: 1+ hits
|
||||
|
||||
# Verify VC5: TRACK_COMPLETION exists
|
||||
ls docs/reports/TRACK_COMPLETION_fix_mma_concurrent_tracks_sim_20260627.md
|
||||
# Expect: file exists
|
||||
```
|
||||
|
||||
## Notes for Tier 3 worker (Phase 1)
|
||||
|
||||
- The "test" is `tests/test_mma_concurrent_tracks_sim.py::test_mma_concurrent_tracks_execution`. It is the spec.
|
||||
- The fix is in `src/app_controller.py:_start_track_logic_result` OR `tests/mock_concurrent_mma.py`. Choose based on Phase 0 diagnosis.
|
||||
- Use `manual-slop_edit_file` for surgical edits (3-10 lines per edit).
|
||||
- 1-space indentation. CRLF line endings. No comments.
|
||||
- Per `conductor/code_styleguides/python.md` §17: no `dict[str, Any]`, no `Any`, no `Optional[T]`, no `hasattr()` for entity dispatch.
|
||||
- If the fix requires changing the mock's response shape, do NOT change the test — the test exercises the production pipeline.
|
||||
|
||||
## Notes for Tier 2 reviewer (Phases 0 and 2)
|
||||
|
||||
- Phase 0 is the instrumentation pass. The diagnostics are INTERIM and must be removed in Phase 2.
|
||||
- Phase 1 is the fix. Read the test log from Phase 0 BEFORE choosing the fix; don't guess.
|
||||
- Phase 2 is cleanup + report.
|
||||
- Per `AGENTS.md` HARD BAN: no `git restore`, no `git checkout`, no `git reset`, no `git stash`.
|
||||
- Per `AGENTS.md` "No Diagnostic Noise in Production": the instrumentation in Phase 0 must be removed in Phase 2.
|
||||
- Per `conductor/workflow.md` "Pre-commit verification gate": after every commit, run `git diff --cached --stat` + `git show HEAD --stat` + `uv run python scripts/audit_tier2_leaks.py --strict`.
|
||||
|
||||
## See also
|
||||
|
||||
- `conductor/tracks/fix_mma_concurrent_tracks_sim_20260627/spec.md` — the canonical reference
|
||||
- `docs/reports/OUTSTANDING_MMA_TEST_FAILURES_20260627.md` — the 4 stacked root causes
|
||||
- `conductor/tracks/post_module_taxonomy_de_cruft_20260627/spec.md` — the parent track spec
|
||||
- `conductor/tracks/post_module_taxonomy_de_cruft_20260627/state.toml` — the parent track state
|
||||
- `conductor/code_styleguides/error_handling.md` — the Result[T] + nil-sentinel convention
|
||||
- `conductor/code_styleguides/data_oriented_design.md` §8.5 — the Python Type Promotion Mandate
|
||||
- `conductor/code_styleguides/python.md` §17 — the LLM Default Anti-Patterns
|
||||
- `conductor/workflow.md` §"Process Anti-Patterns" — the 8 anti-patterns to avoid
|
||||
- `AGENTS.md` — the project operating rules + HARD BANs
|
||||
@@ -0,0 +1,207 @@
|
||||
# Track Specification: fix_mma_concurrent_tracks_sim_20260627
|
||||
|
||||
## Overview
|
||||
|
||||
Single-test fix track. The `tier-3-live_gui::test_mma_concurrent_tracks_sim::test_mma_concurrent_tracks_execution` test was failing on the `tier2/post_module_taxonomy_de_cruft_20260627` branch. Per the user directive ("those issues must get resolved we are not sweeping them under the rug"), this track fixes the test to pass in the batched test suite, ships it, and the parent branch is then ready for review.
|
||||
|
||||
The test exercises the full concurrent-MMA flow: plan an epic (returns 2 proposed tracks), accept both, start both concurrently, verify both ticket-A and ticket-B workers appear, verify both tracks complete. The failure was at "accept-tracks" — after `btn_mma_accept_tracks`, only 1 of the 2 proposed tracks was created in the project.
|
||||
|
||||
This track is the **TDD fix for one specific test**. It is NOT a sweep or a refactor; it is a focused investigation + fix + verification.
|
||||
|
||||
## Current State Audit (branch `tier2/post_module_taxonomy_de_cruft_20260627`, measured 2026-06-27)
|
||||
|
||||
| Component | State | Source |
|
||||
|---|---|---|
|
||||
| `tests/test_mma_concurrent_tracks_sim.py` | 144 lines; fails at line 66 ("Tracks not created in project") | `manual-slop_read_file` |
|
||||
| `tests/mock_concurrent_mma.py` | 144 lines; uses file-based call counter; parses `--resume` arg | commit 635ca552 |
|
||||
| `src/app_controller.py:_cb_accept_tracks._bg_task` | Loops `for i, track_data in enumerate(self.proposed_tracks): self._start_track_logic(...)`; only track-a's mock call observed | `manual-slop_get_file_slice` lines 4665-4680 |
|
||||
| `src/app_controller.py:_start_track_logic_result` | Calls `conductor_tech_lead.generate_tickets(goal, skeletons)` → mock returns sprint ticket → `project_manager.save_track_state(track_id, state, ...)` → `self.tracks.append(...)` | `manual-slop_get_file_slice` lines 4750-4840 |
|
||||
| 3 production sites fixed in 635ca552 | `flat.setdefault(...)["paths"] = ...` → `flat.to_dict() then setdefault`; `t_data["id"]` → `t_data.id` | `OUTSTANDING_MMA_TEST_FAILURES_20260627.md` |
|
||||
| 1 test mock fix in 635ca552 | `--resume` arg parsing + call counter | commit 635ca552 |
|
||||
|
||||
## The 4 Stacked Regressions (Root Cause Analysis)
|
||||
|
||||
### 1. `flat_config()` return type change (PRODUCTION BUG — FIXED in 635ca552)
|
||||
|
||||
`flat_config()` in `src/project.py` was changed by `cruft_elimination_20260627` (commit 0d2a9b5e) from `dict[str, Any]` to a **frozen `@dataclass ProjectContext`**. The change was semantic, not just cosmetic. But 3 sites in `src/app_controller.py` mutated the returned object:
|
||||
|
||||
- `_do_generate` (line 4027): `flat["files"] = ...; flat["files"]["paths"] = ...`
|
||||
- `_cb_plan_epic` (line 4604): `flat.setdefault("files", {})["paths"] = ...`
|
||||
- `_start_track_logic_result` (line 4793): `flat.setdefault("files", {})["paths"] = ...`
|
||||
|
||||
Each raised `TypeError: 'ProjectContext' object does not support item assignment`.
|
||||
|
||||
**Fix in 635ca552:** Call `flat.to_dict()` to get a mutable dict.
|
||||
|
||||
### 2. `topological_sort()` return type change (PRODUCTION BUG — FIXED in 635ca552)
|
||||
|
||||
`conductor_tech_lead.topological_sort()` in `src/mma_conductor.py` was changed (also in commit 0d2a9b5e) from `list[str]` to `list[Ticket]`. The `_start_track_logic_result` consumer used dict-style access (`t_data["id"]`, `t_data.get("description")`).
|
||||
|
||||
**Fix in 635ca552:** Use Ticket attribute access (`t_data.id`, `t_data.description`, etc.).
|
||||
|
||||
### 3. `gemini_cli_adapter` `--resume` session reuse (MOCK BUG — FIXED in 635ca552)
|
||||
|
||||
The gemini_cli_adapter now reuses the session_id from the epic call (`mock-epic`) for all subsequent Tier 2/3 calls via `--resume mock-epic`. The original mock `tests/mock_concurrent_mma.py` was written when each LLM call was stateless; it routed on prompt substrings ("PATH: Epic Initialization", "generate the implementation tickets", "You are assigned to Ticket"). In resume mode the prompt is empty (the session is the context), so the routing fell to the default case.
|
||||
|
||||
**Fix in 635ca552:** Parse `--resume` from `sys.argv` and use a persistent file-based call counter to route to per-track responses.
|
||||
|
||||
### 4. ⚠️ UNRESOLVED — 2nd track's `_start_track_logic` never fires
|
||||
|
||||
After fixes 1-3, the test still fails: only 1 sprint-ticket mock call is observed (for track-a); the 2nd call for track-b never happens. The 30-second test poll times out.
|
||||
|
||||
**Hypothesized root cause:** `_start_track_logic` for track-a either hangs OR fails silently. The for loop in `_cb_accept_tracks._bg_task` continues to track-b which also calls `_start_track_logic` and also fails/hangs. The test poll times out before either track completes.
|
||||
|
||||
**Possible causes to investigate:**
|
||||
- `conductor_tech_lead.generate_tickets(goal, skeletons)` returns `[]` (no tickets) for track-a when the adapter can't reuse the session properly → no track created, no error
|
||||
- `project_manager.save_track_state(track_id, state, ...)` blocks on disk I/O
|
||||
- The IO pool is saturated (the bg_task is `submit_io(_bg_task)` and each `_start_track_logic` is synchronous on its own thread)
|
||||
- `aggregate.run(flat)` hangs (the new `flat.to_dict()` conversion may be missing a field that `aggregate.run` requires)
|
||||
- The exception in `except (OSError, IOError, ValueError, TypeError, KeyError, AttributeError, RuntimeError) as e:` at line 4831 catches an exception and returns `Result(data=None, errors=[err])` — but the caller `_start_track_logic` (line 4744) prints `ERROR in _start_track_logic: {err.message}` and continues to the next track in the loop, which also fails. The test poll times out because no track is appended to `self.tracks`.
|
||||
|
||||
## Goals
|
||||
|
||||
| ID | Goal | Acceptance |
|
||||
|---|---|---|
|
||||
| G1 | Diagnose why only 1 of 2 tracks is created in `_cb_accept_tracks._bg_task` | stderr diagnostics + log file show the actual failure mode for each track |
|
||||
| G2 | Fix the production OR test-mock bug that causes the 2nd track to fail | Test passes in isolation AND in the full batched suite |
|
||||
| G3 | Update `docs/reports/OUTSTANDING_MMA_TEST_FAILURES_20260627.md` to reflect the fix | Report shows RESOLVED status |
|
||||
| G4 | Tier 3 of `tier2/post_module_taxonomy_de_cruft_20260627` goes from FAIL to PASS | `uv run python scripts/run_tests_batched.py --tier tier-3-live_gui` shows 0 failures |
|
||||
| G5 | All 11 batched test tiers pass | `uv run python scripts/run_tests_batched.py` shows 11/11 PASS (or pre-existing RAG flake) |
|
||||
|
||||
## Non-Goals
|
||||
|
||||
- Refactoring the MMA concurrent execution engine (`src/multi_agent_conductor.py`)
|
||||
- Refactoring `_cb_accept_tracks` or `_start_track_logic` beyond the minimum fix
|
||||
- Refactoring `tests/mock_concurrent_mma.py` beyond the minimum fix
|
||||
- Adding new tests for MMA concurrent execution
|
||||
- Fixing any other tier failures (RAG flake is pre-existing and out of scope)
|
||||
- Updating `conductor/tracks/post_module_taxonomy_de_cruft_20260627/spec.md` (the parent track is SHIPPED; this is a follow-up)
|
||||
|
||||
## Functional Requirements
|
||||
|
||||
### FR1: Instrument `_start_track_logic_result` with stderr diagnostics (Tier 3)
|
||||
|
||||
Add 3 `sys.stderr.write` + `sys.stderr.flush` calls:
|
||||
1. BEFORE `conductor_tech_lead.generate_tickets(goal, skeletons)` — log title, goal
|
||||
2. AFTER `generate_tickets` returns — log length of `raw_tickets`
|
||||
3. INSIDE the `except` block at line 4831 — log full traceback via `import traceback; traceback.print_exc()`
|
||||
|
||||
**WHY:** Per workflow.md "The Deduction Loop (kill it)", you are allowed to run a failing test at most 2 times in a single investigation. After 2 failures, STOP running the test. Read the code, predict the failure mode, and instrument ALL the relevant state in one pass.
|
||||
|
||||
### FR2: Run the test in isolation (Tier 2)
|
||||
|
||||
`uv run -m pytest tests/test_mma_concurrent_tracks_sim.py::test_mma_concurrent_tracks_execution -v -s` and capture:
|
||||
- stderr output from `_start_track_logic_result` instrumentation
|
||||
- the mock call counter file at `artifacts/.mock_concurrent_mma_call_count`
|
||||
- the sloppy.py stderr (via the test's log capture)
|
||||
|
||||
**Per workflow.md "Pre-commit verification gate"**, redirect to log file: `... > tests/artifacts/tier2_state/fix_mma_concurrent_tracks_sim_20260627/test_run.log 2>&1`
|
||||
|
||||
### FR3: Diagnose the failure mode (Tier 2)
|
||||
|
||||
Based on FR2 output, identify ONE of:
|
||||
- A. `generate_tickets` returns `[]` (mock routing broken for 2nd call)
|
||||
- B. `project_manager.save_track_state` raises (disk I/O issue)
|
||||
- C. `aggregate.run(flat)` raises (flat dict missing field)
|
||||
- D. The `except` block catches a `RuntimeError` (or other) and the test poll times out
|
||||
|
||||
### FR4: Fix the root cause (Tier 3)
|
||||
|
||||
**Per the user directive: "we should adjust the tests instead"** — but the test exercises the production code path. The test is the spec; the production must be correct. Fix in this priority order:
|
||||
|
||||
1. **If cause A** (mock routing): fix `tests/mock_concurrent_mma.py` to handle the `--resume mock-sprint-A` session reuse (the adapter reuses the session_id returned by the previous call, so track-b's call is `--resume mock-sprint-A` not `--resume mock-epic`).
|
||||
2. **If cause B/C/D** (production bug): fix `src/app_controller.py:_start_track_logic_result` to handle the error gracefully, log the error to the test log, and continue to the next track (instead of silently aborting the loop).
|
||||
|
||||
### FR5: Verify the test passes in isolation (Tier 2)
|
||||
|
||||
`uv run -m pytest tests/test_mma_concurrent_tracks_sim.py::test_mma_concurrent_tracks_execution -v`
|
||||
|
||||
Must show PASS.
|
||||
|
||||
### FR6: Verify the test passes in the full batched suite (Tier 2)
|
||||
|
||||
**Per workflow.md "Isolated-Pass Verification Fallacy"** — the only verification that matters for `live_gui` tests is the batch run. The test must pass with the other tier-3 tests in the suite.
|
||||
|
||||
`uv run python scripts/run_tests_batched.py --tier tier-3-live_gui`
|
||||
|
||||
Must show 0 failures in tier-3.
|
||||
|
||||
### FR7: Verify all 11 tiers pass (Tier 2)
|
||||
|
||||
`uv run python scripts/run_tests_batched.py`
|
||||
|
||||
**Per user directive ("stop running the batch yourself, ask me")** — ASK the user before running the full 11-tier batch. Show them the targeted tier-3 result first.
|
||||
|
||||
Expected: 11/11 PASS (or 10/11 if the RAG flake is the only remaining failure).
|
||||
|
||||
### FR8: Update `OUTSTANDING_MMA_TEST_FAILURES_20260627.md` (Tier 2)
|
||||
|
||||
Mark the section "4. UNRESOLVED — Second track's `_start_track_logic` never fires" as RESOLVED with a link to the fixing commit.
|
||||
|
||||
### FR9: Write `TRACK_COMPLETION_fix_mma_concurrent_tracks_sim_20260627.md` (Tier 2)
|
||||
|
||||
Follow the precedent of `TRACK_COMPLETION_post_module_taxonomy_de_cruft_20260627.md`:
|
||||
- Executive summary
|
||||
- 3 root causes fixed (the 3 already in 635ca552)
|
||||
- The 1 root cause fixed in this track
|
||||
- Files changed
|
||||
- Verification results
|
||||
- Suggested next steps
|
||||
|
||||
## Non-Functional Requirements
|
||||
|
||||
- NFR1: 1-space indentation
|
||||
- NFR2: CRLF line endings on Windows
|
||||
- NFR3: No comments in source code
|
||||
- NFR4: Per-task atomic commits with git notes
|
||||
- NFR5: No new pip dependencies
|
||||
- NFR6: Result[T] returns for fallible fns
|
||||
- NFR7: No `git restore` / `git checkout` / `git reset` / `git stash` (per AGENTS.md HARD BAN)
|
||||
- NFR8: Stderr diagnostics must be removed before the final commit (no diagnostic noise in production per workflow.md)
|
||||
|
||||
## Architecture Reference
|
||||
|
||||
- `src/app_controller.py:_cb_accept_tracks._bg_task` (line 4635-4682) — the for loop that should create 2 tracks
|
||||
- `src/app_controller.py:_start_track_logic_result` (line 4750-4840) — the per-track pipeline
|
||||
- `src/multi_agent_conductor.py:ConductorEngine.run` — the engine that spawns workers
|
||||
- `src/ai_client.py:gemini_cli_adapter` (or similar) — the adapter that uses `--resume` for session reuse
|
||||
- `src/mma_conductor.py:topological_sort` — returns `list[Ticket]` (was `list[str]` pre-cruft)
|
||||
- `src/project.py:flat_config` — returns `frozen @dataclass ProjectContext` (was `dict[str, Any]` pre-cruft)
|
||||
- `conductor/code_styleguides/error_handling.md` — the Result[T] + nil-sentinel convention
|
||||
- `conductor/code_styleguides/data_oriented_design.md` §8.5 — the Python Type Promotion Mandate
|
||||
- `conductor/code_styleguides/python.md` §17 — the LLM Default Anti-Patterns
|
||||
|
||||
## Risks
|
||||
|
||||
| # | Risk | Likelihood | Mitigation |
|
||||
|---|---|---|---|
|
||||
| R1 | The instrumentation is incomplete and the failure mode remains hidden | low | Add diagnostics at 3 strategic points: before/after generate_tickets, in the except block |
|
||||
| R2 | The fix requires changes to the production code that may regress other tests | medium | Run the full batched test suite after the fix (with user permission) |
|
||||
| R3 | The mock fix requires a deeper understanding of the gemini_cli_adapter's session reuse | medium | Read `src/ai_client.py:gemini_cli_adapter` (or similar) to understand the session_id lifecycle |
|
||||
| R4 | The test has a 30-second poll that may be too short for the test infrastructure (IO pool + bg_task + subprocess spawn) | low | Document the timing in the test, but don't change the test's poll time (the fix should make the test pass within the existing poll budget) |
|
||||
| R5 | The instrumentation leaks into production (per AGENTS.md "No Diagnostic Noise in Production") | low | Remove the instrumentation in the same commit that fixes the bug (or in a follow-up commit) |
|
||||
| R6 | The user does not give permission to run the full 11-tier batched test suite | medium | Run only the targeted tier-3 batched test (`--tier tier-3-live_gui`); ask user for the full batch separately |
|
||||
|
||||
## Verification Criteria (Definition of Done)
|
||||
|
||||
| # | Criterion | Verification |
|
||||
|---|---|---|
|
||||
| VC1 | The test `test_mma_concurrent_tracks_execution` passes in isolation | `uv run -m pytest tests/test_mma_concurrent_tracks_sim.py -v` shows PASS |
|
||||
| VC2 | Tier 3 of the batched test suite passes (0 failures) | `uv run python scripts/run_tests_batched.py --tier tier-3-live_gui` shows 0 failures |
|
||||
| VC3 | The instrumentation is removed from `src/app_controller.py` | `git grep "_start_track_logic_result.*stderr" src/app_controller.py` returns 0 hits |
|
||||
| VC4 | `OUTSTANDING_MMA_TEST_FAILURES_20260627.md` is updated to RESOLVED | grep "RESOLVED" OUTSTANDING_MMA_TEST_FAILURES_20260627.md returns hits |
|
||||
| VC5 | `TRACK_COMPLETION_fix_mma_concurrent_tracks_sim_20260627.md` is written | `ls docs/reports/TRACK_COMPLETION_fix_mma_concurrent_tracks_sim_20260627.md` exists |
|
||||
| VC6 | All diagnostic stderr lines are removed from `src/app_controller.py` | No `[DEBUG] _start_track_logic:` lines remain in production |
|
||||
| VC7 | No `git restore` / `git checkout` / `git reset` / `git stash` used | Audit the git reflog for the branch |
|
||||
|
||||
## See also
|
||||
|
||||
- `docs/reports/OUTSTANDING_MMA_TEST_FAILURES_20260627.md` — the 4 stacked root causes (this track fixes the 4th)
|
||||
- `docs/reports/END_OF_SESSION_post_module_taxonomy_de_cruft_20260627_iteration3.md` — the prior iteration report
|
||||
- `conductor/tracks/post_module_taxonomy_de_cruft_20260627/spec.md` — the parent track spec
|
||||
- `conductor/tracks/post_module_taxonomy_de_cruft_20260627/state.toml` — the parent track state
|
||||
- `conductor/code_styleguides/error_handling.md` — the Result[T] + nil-sentinel convention
|
||||
- `conductor/code_styleguides/data_oriented_design.md` §8.5 — the Python Type Promotion Mandate
|
||||
- `conductor/code_styleguides/python.md` §17 — the LLM Default Anti-Patterns
|
||||
- `conductor/workflow.md` §"Process Anti-Patterns" — the 8 anti-patterns to avoid
|
||||
- `AGENTS.md` — the project operating rules + HARD BANs
|
||||
@@ -0,0 +1,49 @@
|
||||
# Track state for fix_mma_concurrent_tracks_sim_20260627
|
||||
# Updated by Tier 2 Tech Lead as tasks complete
|
||||
|
||||
[meta]
|
||||
track_id = "fix_mma_concurrent_tracks_sim_20260627"
|
||||
name = "Fix MMA Concurrent Tracks Sim Test (tier-3-live_gui regression)"
|
||||
status = "active"
|
||||
current_phase = 0
|
||||
last_updated = "2026-06-27"
|
||||
|
||||
[blocked_by]
|
||||
post_module_taxonomy_de_cruft_20260627 = "shipped (the parent track; this is the followup fix for the 1 remaining tier-3 failure)"
|
||||
|
||||
[blocks]
|
||||
|
||||
[phases]
|
||||
phase_0 = { status = "in_progress", checkpointsha = "", name = "Instrument + diagnose (1 commit: add stderr diagnostics to _start_track_logic_result)" }
|
||||
phase_1 = { status = "pending", checkpointsha = "", name = "Fix the root cause (1-2 commits: fix in app_controller.py OR mock_concurrent_mma.py)" }
|
||||
phase_2 = { status = "pending", checkpointsha = "", name = "Remove instrumentation + write report (4 commits: cleanup + report updates + TRACK_COMPLETION + state.toml)" }
|
||||
|
||||
[tasks]
|
||||
t0_1 = { status = "in_progress", commit_sha = "", description = "Add stderr diagnostics to _start_track_logic_result" }
|
||||
t0_2 = { status = "pending", commit_sha = "", description = "Run the test in isolation with instrumentation; capture log; identify failure mode" }
|
||||
t1_1 = { status = "pending", commit_sha = "", description = "Fix the root cause (mock routing OR production bug) based on Phase 0 diagnosis" }
|
||||
t1_2 = { status = "pending", commit_sha = "", description = "Run the test in isolation to verify the fix" }
|
||||
t1_3 = { status = "pending", commit_sha = "", description = "Run the targeted tier-3 batched test suite to verify no regressions" }
|
||||
t2_1 = { status = "pending", commit_sha = "", description = "Remove the stderr instrumentation from _start_track_logic_result" }
|
||||
t2_2 = { status = "pending", commit_sha = "", description = "Update OUTSTANDING_MMA_TEST_FAILURES_20260627.md to RESOLVED" }
|
||||
t2_3 = { status = "pending", commit_sha = "", description = "Write TRACK_COMPLETION_fix_mma_concurrent_tracks_sim_20260627.md" }
|
||||
t2_4 = { status = "pending", commit_sha = "", description = "Update state.toml to status = completed; final SHIPPED commit" }
|
||||
|
||||
[verification]
|
||||
phase_0_complete = false
|
||||
phase_1_complete = false
|
||||
phase_2_complete = false
|
||||
|
||||
[track_specific]
|
||||
test_failing = "tests/test_mma_concurrent_tracks_sim.py::test_mma_concurrent_tracks_execution"
|
||||
parent_track = "post_module_taxonomy_de_cruft_20260627"
|
||||
parent_track_shipped_commit = "d74b9822"
|
||||
prior_partial_fix_commit = "635ca552"
|
||||
prior_fixes_in_635ca552 = [
|
||||
"flat.setdefault(...)[...] = ... on frozen ProjectContext (3 sites)",
|
||||
"t_data['id'] on Ticket objects (1 site)",
|
||||
"mock_concurrent_mma.py --resume handling"
|
||||
]
|
||||
diagnosis_approach = "instrument _start_track_logic_result with 3 stderr calls; run test in isolation; identify failure mode for 2nd track"
|
||||
fix_approach = "based on Phase 0 diagnosis: fix in src/app_controller.py (production) OR tests/mock_concurrent_mma.py (test mock)"
|
||||
verification_target = "tier-3-live_gui batched test suite: 0 failures"
|
||||
Reference in New Issue
Block a user