# Track Specification: Test Batching — Post-Refactor Polish

**Status:** Active (spec authored 2026-06-08)
**Initialized:** 2026-06-08
**Owner:** Tier 2 Tech Lead
**Priority:** Medium (developer ergonomics + observability; not a regression blocker)
**Blocked by:** `test_batching_refactor_20260606` (must be SHIPPED before this track begins; the new orchestrator from the refactor is the target of the polish)
**Blocks:** None

---

## 1. Problem Statement

`test_batching_refactor_20260606` will replace the current `scripts/run_tests_batched.py` with a tier-based orchestrator that:
- Uses `subprocess.run(cmd, capture_output=True, text=True)` to invoke each batch's pytest
- On failure, prints the last 2000 chars of stdout (the new spec/plan, Phase 3 Task 3.1, line 1304: `print(proc.stdout[-2000:] if proc.returncode != 0 else ...)`)
- Has no mechanism to surface the **actual failed file paths** to the user

This is a regression in failure visibility vs. the current script (which lists every file in a failed batch — bad, but at least explicit). The new script will print a tail of pytest output that the user must manually scan for `FAILED ` lines.

Three concrete improvements are deferred from the refactor to this track:

1. **Per-file FAILED-line extraction** in the new orchestrator. When a tier batch fails, the script's summary should list the specific test files pytest reported as failed (parsed via str ops only, no regex per `AGENTS.md` standing ban). Same contract the current legacy script's `_extract_failed_files` (when fixed) will provide.
2. **`live_gui` subprocess window foregrounding.** When the `live_gui` fixture spawns `sloppy.py`, the OS window must be raised to the foreground so the user watching the test can see the activity. Tier 3 (consolidated `live_gui`, 14+ `*_sim.py` files in one pytest invocation) amplifies this: without foregrounding, the user sees a hidden window for 30-60s while the tier runs.
3. **`focus_test_panel(name)` test helper.** Live_gui tests should signal which panel they're exercising. The helper uses the existing `ApiHookClient.set_value` to toggle `show_windows[name] = True` and is called from individual `*_sim.py` test setup. The refactor's Tier 3 consolidation makes this signal-critical: the user needs to see WHICH panel is being driven, not just that something is happening.

A fourth improvement is housekeeping: ~45 scratch files in `tests/artifacts/` from prior sessions (regex experimentation, layout baking debugging, sub-track task notes). These are gitignored but clutter the directory. Safe deletion is non-trivial (some files may be referenced by other tests or fixtures) so it's deferred to this track where it can be done carefully with verification.

---

## 2. Current State Audit (as of `2db14361 TEST LAYOUT`)

### Already Implemented (DO NOT re-implement)

| What | Where | Status |
|---|---|---|
| `App._diag_layout_state()` method | `src/gui_2.py:507-544` | Committed `818537b3`. Logs `[GUI] show_windows entries: N`, `[GUI] layout file: <path> (<bytes>)`, `[GUI] WARNING: layout has N stale window name(s)...` |
| `manualslop_layout_default.ini` (user's preferred 2-column layout) | `tests/artifacts/manualslop_layout_default.ini` (2,699 bytes) | Whitelisted in `.gitignore` line 17. Confirmed loaded by `_diag_layout_state` log. |
| `tests/conftest.py:418-421` copies the layout artifact into the test workspace | `tests/conftest.py:418-421` | Replaces the prior "do NOT copy" block from `7a4f71e7` |
| `_default_windows` updated for 12-window visible-by-default set | `src/app_controller.py:1832-1855` | MMA Dashboard=False, Log Management=True, Diagnostics=True |
| `_STALE_WINDOW_NAMES` set | `src/gui_2.py:530-533` | 10 names (Theme removed; was incorrectly flagged as stale) |
| Skip markers from `e09e6823` resolved | `8d58d7fc` (warmup races), `a36aad50` (gui_events_v2), `91b34ae8` (live_gui_filedialog), `ff523f7e` (project_switch_persona) | 3 of 5 fixed in subsequent commits; 2 in `8d58d7fc` |
| `RUN_MMA_INTEGRATION` env-var gate on `test_mma_step_mode_sim.py` | `tests/test_mma_step_mode_sim.py:24-27` | Appropriate opt-in integration gate, not a broken test |
| `scripts/cleanup_orphaned_processes.py` | Committed `5e1867bb` | Manages stale subprocesses; preserves MCP servers |
| `_extract_failed_files` (in legacy `run_tests_batched.py`, if Phase 0 ships) | `scripts/run_tests_batched.py:30-50` (post-Phase-0) | Str-ops-only FAILED-line parser; 11 unit tests in `tests/test_run_tests_batched.py` |

### Gaps to Fill (This Track's Scope)

| Gap | Severity | Where the fix lands |
|---|---|---|
| New orchestrator's `subprocess.run(capture_output=True)` only prints stdout tail on failure — no per-file failure list | **High** | New `scripts/run_tests_batched.py` (post-refactor) — the `_run_batch` helper around line 1296-1308 of the refactor's plan |
| `live_gui` fixture doesn't bring sloppy.py's window to front | **Medium** | `tests/conftest.py:live_gui` fixture |
| `live_gui` tests have no per-test focus signal | **Medium** | `tests/conftest.py` (new helper) + per-test callsites in 14+ `*_sim.py` files |
| `tests/artifacts/` has ~45 scratch files from prior sessions | **Low** | `tests/artifacts/*.py`, `tests/artifacts/*.txt`, `tests/artifacts/*.toml` (verify references first) |
| The `_extract_failed_files` from Phase 0 of the refactor (if shipped) lives in the LEGACY script that gets renamed to `.legacy` in Phase 3, then deleted in Phase 4 | **Critical** | The function needs to be lifted to a shared location (e.g., `scripts/test_failure_parser.py`) so both legacy and new orchestrator use the same code |

---

## 3. Goals

1. **Per-file FAILED-line extraction in the new orchestrator.** When any tier batch fails, the summary lists the specific test files pytest reported as failed (via str ops only, no regex). On timeout, fall back to listing the whole batch with `(timeout)` annotation.
2. **Lift `_extract_failed_files` to a shared library.** The function lives in `scripts/test_failure_parser.py` (or similar); both the legacy script and the new orchestrator import it. No code duplication.
3. **`live_gui` subprocess window foregrounding.** When the fixture spawns `sloppy.py`, find the child window by PID and call `ShowWindow` + `SetForegroundWindow`. No-op on non-Windows or when pywin32 is unavailable. Wrapped in `try/except`; never raises.
4. **`focus_test_panel(name)` helper.** New module-level function in `tests/conftest.py` that uses the existing `ApiHookClient.set_value` to toggle `show_windows[name] = True`. Returns True/False (False if hook server unreachable).
5. **Wire `focus_test_panel` into at least 3 starter `*_sim.py` tests** so the pattern is established for the refactor's consolidated Tier 3.
6. **Clean up `tests/artifacts/` scratch files** (with verification of non-reference first).

---

## 4. Functional Requirements

### 4.1 Shared `_extract_failed_files` library

**FR-1.** Create `scripts/test_failure_parser.py` containing the `_extract_failed_files(output: str) -> list[str]` function. Str-ops-only (no `re` import per `AGENTS.md`).

**FR-2.** The function SHALL:
- Accept the full captured stdout+stderr from a pytest invocation
- Parse lines beginning with the literal 7-character prefix `FAILED ` (note trailing space)
- Extract the test ID, ending at the first ` - ` (space-dash-space) separator
- If the test ID contains `::`, take the file path portion (before the first `::`)
- Normalize backslashes to forward slashes (Windows path safety)
- Strip a leading `tests/` prefix to return the bare filename
- Deduplicate (preserve first-occurrence order)

**FR-3.** Update the legacy `scripts/run_tests_batched.py` to import `_extract_failed_files` from the new shared module (if it was implemented locally in the refactor's Phase 0; otherwise add it there for the first time).

**FR-4.** Update the new orchestrator (post-refactor) to call `_extract_failed_files` on the captured stdout/stderr in `_run_batch` when `returncode != 0`. Use the returned list to populate the SUMMARY table's per-file failure list.

**FR-5.** Add 11+ unit tests in `tests/test_test_failure_parser.py` covering the contract from FR-2 (same set as the original 11 tests for the legacy script, ported to the new module).

### 4.2 New Orchestrator Per-File Failure List

**FR-6.** In the new `scripts/run_tests_batched.py:_run_batch` (post-refactor), on non-zero exit:
- Call `_extract_failed_files(proc.stdout + proc.stderr)` (combined)
- If the returned list is non-empty, add those files to the per-tier failure list
- If the returned list is empty (rare; collection errors, plugin crashes), add the whole batch's files with a `(no FAILED lines; treating as batch failure)` annotation

**FR-7.** On `subprocess.TimeoutExpired` (the batch exceeded `--timeout`): fall back to `failed_files.extend(batch)` with `(timeout)` annotation (per-file accuracy impossible on timeout — same as legacy).

**FR-8.** The SUMMARY table (new orchestrator's `_print_summary`) SHALL include a per-file failure listing when any tier failed:
```
[TIER 3] live_gui              FAIL   14/14  47.2s
   - tests/test_foo.py
   - tests/test_bar.py
```

**FR-9.** The orchestrator's worst-case exit code SHALL be 1 if any tier has a per-file failure list, 0 if all tiers passed or were skipped.

### 4.3 Live_Gui Window Foregrounding (`tests/conftest.py`)

**FR-10.** Add module-level function `_foreground_subprocess_window(pid: int, attempts: int = 3, delay_s: float = 0.5) -> None` to `tests/conftest.py`.

**FR-11.** The function SHALL:
- No-op immediately on `os.name != "nt"`
- Try-except `import win32gui, win32con`; no-op on `ImportError`
- Loop `attempts` times: `win32gui.EnumWindows` to find a top-level visible window whose owning PID matches `pid`; on match, call `win32gui.ShowWindow(hwnd, win32con.SW_SHOWNORMAL)` then `win32gui.SetForegroundWindow(hwnd)`
- Sleep `delay_s` between attempts (the subprocess may take 1-2s to create its window)
- Wrap the whole body in `try/except Exception`; log a `[Fixture] WARNING: ...` line and return on any error; NEVER raise into the test fixture

**FR-12.** Wire the helper into the `live_gui` fixture: insert one line `_foreground_subprocess_window(proc.pid)` immediately after the `subprocess.Popen(...)` call returns.

**FR-13.** Add 3 unit tests in `tests/test_live_gui_foregrounding.py` asserting: helper exists and is callable; helper is no-op on invalid PIDs; helper is no-op when `win32gui`/`win32con` import fails (monkeypatched).

### 4.4 `focus_test_panel` Helper

**FR-14.** Add module-level function `focus_test_panel(panel_name: str, host: str = "127.0.0.1", port: int = 8999) -> bool` to `tests/conftest.py`.

**FR-15.** The function SHALL:
- Try-except `from src.api_hook_client import ApiHookClient`; return False on `ImportError`
- Instantiate `ApiHookClient(host=host, port=port)`
- Call `client.wait_for_server(timeout=0.5)`; return False if the server is not reachable
- Call `client.set_value(f'show_windows["{panel_name}"]', True)`
- Wrap the whole body in `try/except Exception`; log a `[focus_test_panel] ...` line and return False on any error
- Return True on success

**FR-16.** The function is OPTIONAL for tests: tests that don't call it get existing behavior. Tests that call it signal intent. The function's return value is informational (caller may choose to skip on False).

**FR-17.** Wire `focus_test_panel` into at least 3 starter `*_sim.py` files (one-line addition in test setup, immediately after `client.wait_for_server(...)`):
- `tests/test_command_palette_sim.py`: `focus_test_panel("Command Palette")`
- `tests/test_workflow_sim.py`: `focus_test_panel("Discussion Hub")`
- `tests/test_undo_redo_sim.py`: `focus_test_panel("Discussion Hub")`

### 4.5 `tests/artifacts/` Scratch Cleanup

**FR-18.** Verify each candidate scratch file is NOT referenced by any test or fixture (use `rg "<filename_without_ext>" tests/ scripts/ src/ docs/` and confirm zero matches).

**FR-19.** For files with zero references, delete them. The candidate list (from prior session's report + my own audit of `tests/artifacts/`):
- `test_parser.py`, `test_patterns.py`, `test_regex.py` (regex experimentation)
- `verify_layout.py`, `check_cwd.py`, `check_cwd_uv.py`, `exists.py`, `fix_stale_names.py`, `fix_conftest_layout.py` (layout + cwd debugging)
- `fake_test_output.txt` (sample data for parser testing)
- `agents_skip_msg.txt`, `commit_layout_diag_msg.txt`, `configpath_msg.txt`, `context_presets_msg.txt`, `hooks_dictkey_msg.txt`, `reset_layout_msg.txt`, `st2a_prompt.txt`, `st2a_task.toml`, `st2g_msg.txt` (3 copies), `stale_test_msg.txt`, `synthesis_crash_msg.txt`, `warmup_fix_msg.txt`, `workflow_skip_msg.txt` (agent scratch messages)
- `task1.toml`–`task4.toml`, `task1.txt`–`task_3_1.txt` (task notes)
- `temp_config.toml`, `temp_data.txt`, `temp_live*.toml`, `temp_notes.txt`, `temp_project.toml`, `temp_settings.toml`, `temp_simproject.toml` (temp scratch)
- `test_001.md` (25KB scratch markdown)

**FR-20.** The following SHALL be PRESERVED:
- `tests/artifacts/manualslop_layout_default.ini` (whitelisted in `.gitignore`)
- `tests/artifacts/manual_slop.toml`, `repro_project.toml`, `test_snapshot_project.toml` (referenced by fixtures)
- `tests/artifacts/live_gui_workspace/`, `repro_workspace/`, `temp_workspace/`, `gui_ux_sim/`, `test_isolated_project/`, `test_link_workspace/`, `conductor/`, `.slop_cache/` (runtime state)
- `tests/artifacts/.gitignore` (in-place gitignore for the subdirectory)

---

## 5. Non-Functional Requirements

**NFR-1.** 1-space indentation throughout all Python changes (per `conductor/product-guidelines.md`).
**NFR-2.** CRLF line endings on Windows for all changed `.py` files.
**NFR-3.** No inline comments in production code (per `AGENTS.md`).
**NFR-4.** No `re` (regex) module imports in the failure parser. Verify with `grep -n "import re\|from re" scripts/test_failure_parser.py` returning empty after the change.
**NFR-5.** No new external dependencies. No `pyproject.toml` change.
**NFR-6.** Type hints required for all new functions and the modified `run_batch` signature in the new orchestrator.
**NFR-7.** The window-foregrounding helper SHALL NOT call `SetForegroundWindow` more than 3 times per session (Windows throttles repeated foreground-stealing attempts).
**NFR-8.** All commits are atomic per-task (per `conductor/workflow.md` "Definition of Done").

---

## 6. Architecture Reference

- **`docs/guide_architecture.md` "Thread domains"** — the live_gui fixture runs in the pytest process (foreground); sloppy.py runs in a subprocess. The fixture → subprocess communication is over the Hook API (`127.0.0.1:8999`). Window-foregrounding uses a separate channel (Windows OS API; `win32gui`).
- **`docs/guide_testing.md` "live_gui fixture"** — the session-scoped fixture's lifecycle.
- **`docs/guide_api_hooks.md` "ApiHookClient.set_value"** — the existing mechanism for toggling `show_windows[name]`. The new `focus_test_panel` helper uses this.
- **`docs/guide_simulations.md` "Puppeteer pattern"** — existing pattern for live_gui tests; the new `focus_test_panel` is a small variant of the same shape.
- **`conductor/tracks/test_batching_refactor_20260606/spec.md` §3.3 "Six Tiers"** — Tier 3 (live_gui) is the upstream system this track polishes. The new orchestrator's `_run_batch` is the integration point for the per-file failure list.
- **`conductor/tracks/startup_speedup_20260606/state.toml` §`conftest_warmup_wait`** — the fixture's existing warmup-blocking wait runs at conftest load time, before the live_gui fixture executes. The new window-foregrounding code runs AFTER the subprocess spawns (not at load time) and is therefore orthogonal.
- **`AGENTS.md` "Critical Anti-Patterns"** — re-affirms the standing ban on `re` (regex) module imports in the codebase. The user has threatened a 10-page report if they see regex.

---

## 7. Coordination with `test_batching_refactor_20260606`

| Refactor phase | What this track does after it ships |
|---|---|
| **Phase 1** (Library + dry-run) | Nothing; legacy script unchanged. |
| **Phase 2** (Shadow run) | Nothing; shadow run still uses legacy + new in parallel. |
| **Phase 3** (Switch default, rename legacy to `.legacy`) | The legacy's `_extract_failed_files` (if implemented in refactor's Phase 0) is moved to `scripts/test_failure_parser.py` so the new orchestrator can use it without forking. The new orchestrator's `_run_batch` is updated to call the shared parser. |
| **Phase 4** (Cleanup, delete legacy) | The legacy is deleted; `scripts/test_failure_parser.py` is the sole home of the FAILED-line parser. |

### 7.1 Open question for the refactor (recorded, not fixed here)

The refactor's `scripts/test_categorizer.py::auto_classify()` rule #2 uses **regex** in the spec (`AGENTS.md` ban conflict):
> `\(live_gui\)\s*[:,)]` regex match in source

The user has confirmed they will instruct the implementing agent to convert this to AST-based detection (`ast.parse` → walk `FunctionDef` for `live_gui` in args). This is **the refactor's responsibility**, not this post-refactor track's.

---

## 8. Out of Scope

- **The test batching refactor itself** — owned by `test_batching_refactor_20260606`.
- **Auto-classification regex → AST conversion** — the user will instruct the agent directly; not part of this track.
- **Tracked `manualslop_layout.ini` at repo root** — requires explicit user permission per the user's HARD BAN on `git restore`/`git checkout --`. The conftest no longer copies it to the test workspace (regression fixed in `7a4f71e7`).
- **User's TOML files** (`config.toml`, `project.toml`, `project_history.toml`) — explicitly excluded per the user's standing constraint.
- **New audit scripts** — none introduced. The existing audit set is sufficient.
- **The skip markers from `e09e6823`** — 3 fixed in subsequent commits, 2 in `8d58d7fc`. No skip markers remain that this track needs to address.
- **The `__getattr__` cheat audit work** — separate track referenced in `conductor/reports/AUDIT_ARCHITECTURAL_CHEATS_20260607.md`.
- **Performance baseline** — the refactor's `--durations` feature records runtimes. Generating that file is a Phase 1 task of the refactor, not this track.

---

## 9. Verification Criteria

This track is "done" when **all** of the following are true:

- [ ] `scripts/test_failure_parser.py` exists and exports `_extract_failed_files` (no `re` import; verify with `grep -n "import re\|from re" scripts/test_failure_parser.py` returning empty).
- [ ] 11+ unit tests in `tests/test_test_failure_parser.py` all pass.
- [ ] The legacy `scripts/run_tests_batched.py` (if not yet deleted by the refactor) imports `_extract_failed_files` from the new module.
- [ ] The new `scripts/run_tests_batched.py` (post-refactor) `_run_batch` calls `_extract_failed_files` on captured output and includes the per-file failure list in the SUMMARY table.
- [ ] `tests/conftest.py:_foreground_subprocess_window` exists; 3 unit tests pass; the live_gui fixture calls it after `subprocess.Popen(...)`.
- [ ] `tests/conftest.py:focus_test_panel` exists; 3+ `*_sim.py` tests call it in setup.
- [ ] The scratch files from FR-19 are deleted; the directory only contains the preserved files/directories from FR-20.
- [ ] The existing test suite still passes for batches 1-4 (no regressions).
- [ ] Batch 5's timeout (test_z_negative_flows) is reported as exactly 1 failed file, not all 42.
- [ ] All commits are atomic per-task with descriptive messages.
- [ ] No commits include the user's TOML files.
- [ ] No commits include `manualslop_layout.ini` at the repo root.
