# Root Cause Report: test_full_live_workflow batch failure (v2)

**Supersedes:** `test_full_live_workflow_root_cause_20260608.md` (older 6-cause analysis, dated 2026-06-08)
**Date:** 2026-06-08
**Status:** Investigation complete via diagnostic logging, no fix attempted
**Failure reproducibility:** 100% in `tier-3-live_gui` batch (5+ tests, ~200s total), 0% in isolation (`pytest tests/test_live_workflow.py` → 11.69s PASS)
**Related commits (reverted/no-op):** `4a338486` (io_pool 4→8), `c9a991bb` (timeout 30→120s), `87d7c5bf` (test_io_pool assertion)

---

## TL;DR — The Real Root Cause

**`test_full_live_workflow` does not fail because of a slow `_do_project_switch`.** The switch runs in ~8-10ms when it executes. The test fails because **the GUI subprocess crashes mid-batch** due to an ImGui scope mismatch in some render function, which leaves the controller's `io_pool` in a shutdown state. Subsequent clicks to the still-alive hook server fail with `RuntimeError: cannot schedule new futures after shutdown`.

The `_do_project_switch` was a SYMPTOM, not the cause. The previous report's 6-cause analysis (cwd-relative paths, race conditions, click fire-and-forget, etc.) addressed symptoms of the same underlying issue but did not address the IM_ASSERT.

---

## Evidence Trail

### Diagnostic instrumentation (temporarily added, then reverted)

Added `[switch-diag] +N.NNNs <step>` prints to stderr at every step inside `_do_project_switch` (production code). Output goes to `logs/sloppy_py_test.log` because the `live_gui` fixture captures the subprocess's stderr/stdout to that file. Pattern matches the existing `[startup]` and `[HOOKS]` instrumentation style.

### Key log findings (`logs/sloppy_py_test.log`)

#### Finding 1: All 4 sims' switches complete in ~8-10ms each

```
[switch-diag] +0.000s enter path=temp_livecontextsim.toml
[switch-diag] +0.000s flush_to_project_start
[switch-diag] +0.000s flush_to_project_done
[switch-diag] +0.000s load_project_start
[switch-diag] +0.002s load_project_done
[switch-diag] +0.002s preset_manager_start
[switch-diag] +0.002s preset_manager_done
[switch-diag] +0.002s persona_manager_start
[switch-diag] +0.002s persona_manager_done
[switch-diag] +0.002s refresh_start
[switch-diag] +0.010s refresh_done
[switch-diag] +0.010s mcp_configure_start
[switch-diag] +0.010s mcp_configure_done
[switch-diag] +0.010s success
[switch-diag] +0.010s finally_enter
[switch-diag] +0.010s finally_done
```

Same pattern for all 4 sims. **The switch itself is fast. There is no hang inside `_do_project_switch`.**

#### Finding 2: An ImGui `IM_ASSERT` fires at 71.5s into GUI lifetime

```
[02214] [imgui-error] In window 'MainDockSpace': Missing End()
[startup] main_call: 71518.5ms
Traceback (most recent call last):
  File "C:\projects\manual_slop\sloppy.py", line 75, in <module>
    main()
  File "C:\projects\manual_slop\src\gui_2.py", line 1478, in main
    app.run()
  File "C:\projects\manual_slop\src\gui_2.py", line 618, in run
    immapp.run(self.runner_params, ...)
  File "...\imgui_bundle\_patch_runners_add_save_screenshot_param.py", line 38, in patched_run
    run_backup(*args, **kwargs)
RuntimeError: IM_ASSERT( (0) && "Missing End()" )   ---   imgui.cpp:11662
```

The `IM_ASSERT` is an ImGui scope-tracking assertion: a `begin()` call was not matched with a corresponding `end()`. The window reported is 'MainDockSpace' — a special window managed by `hello_imgui` for the dock space layout. Some child widget within the dock space has an unbalanced begin/end.

#### Finding 3: The test's `btn_project_new_automated` click hits a shutdown pool

```
[HOOKS] POST /api/session data length: 1       ← test_live_workflow's post_session
[HOOKS] GET /api/warmup_wait?timeout=60.0
[HOOKS] GET /api/project_switch_status
[HOOKS] POST /api/gui data length: 3           ← test's btn_reset
[HOOKS] POST /api/gui data length: 3           ← test's btn_project_new_automated
Error executing GUI task (click): cannot schedule new futures after shutdown
Traceback (most recent call last):
  File "...\src\app_controller.py", line 1637, in _process_pending_gui_tasks
    self._gui_task_handlers[action](self, task)
  File "...\src\app_controller.py", line 580, in _handle_click
    controller._cb_new_project_automated(user_data)
  File "...\src\app_controller.py", line 2723, in _cb_new_project_automated
    self._switch_project(user_data)
  File "...\src\app_controller.py", line 2809, in _switch_project
    self.submit_io(self._do_project_switch, path)
  File "...\src\app_controller.py", line 2282, in submit_io
    future = self._io_pool.submit(fn, *args, **kwargs)
  File "...\concurrent\futures\thread.py", line 167, in submit
    raise RuntimeError('cannot schedule new futures after shutdown')
RuntimeError: cannot schedule new futures after shutdown
```

The IM_ASSERT happened ~71.5s into the GUI. The test_live_workflow runs after the 4 sims (~80s into GUI). Between the IM_ASSERT and the test's click, the `io_pool` was shut down.

#### Finding 4: The hook server (FastAPI/stdlib http.server) stays alive

The subprocess continues to respond to hooks (GET /api/events, GET /api/session, POST /api/gui) AFTER the IM_ASSERT and AFTER the io_pool is shut down. The test's `wait_for_project_switch` polls `/api/project_switch_status` 1200+ times in 120s — the server is responsive. Only `submit_io` fails.

This proves: the IO thread pool is the casualty, not the entire process. Something selectively shut down `_io_pool` while the rest of the controller (and the hook server) kept running.

---

## What Shuts Down The io_pool?

This is the unresolved question. The only places `_io_pool.shutdown(wait=False)` is called in `src/`:

1. `src/app_controller.py:762` — in the `_on_sigint` SIGINT handler. This requires SIGINT to be delivered to the subprocess. On Windows, `taskkill /F` does NOT deliver signals.
2. `src/app_controller.py:2325` — in `controller.shutdown()`. This is called from `src/gui_2.py:869` (in `App.shutdown`), which is only called at `src/gui_2.py:620` AFTER `immapp.run()` returns successfully.

The IM_ASSERT raises `RuntimeError` from inside `immapp.run()`. The exception propagates up. **There is no `try/finally` around `immapp.run`**, so `App.shutdown()` is NOT called via this path.

**Hypothesis A:** Python's interpreter finalization calls `__del__` on the controller (or its `_io_pool`) during exception propagation. `ThreadPoolExecutor.__del__` defaults to `shutdown(wait=False)` per the io_pool.py module docstring. This is the most likely path on a `RuntimeError` propagating through `main()`.

**Hypothesis B:** `immapp.run` internally catches the IM_ASSERT and returns normally. Then `app.shutdown()` is called, which calls `controller.shutdown()`, which calls `_io_pool.shutdown(wait=False)`. Then `app.run()` returns, `main()` returns, and the sloppy.py process exits. But the log shows more activity AFTER the IM_ASSERT (clicks being processed), so this hypothesis is inconsistent with the evidence.

**Hypothesis C:** The hook server thread (FastAPI on port 8999) runs in a separate thread from the main ImGui loop. The IM_ASSERT crashes the main thread but the hook server thread keeps running. The `_io_pool` is the GUI's pool, not the hook server's. When the main thread crashes, Python's atexit / finalization shuts down `_io_pool`. The hook server thread continues independently.

**Hypothesis C is most consistent with the evidence.** The `_io_pool` is created in `AppController.__init__` (line ~810 in current code). The hook server is a separate `ThreadingHTTPServer` (line 11 of `src/api_hooks.py`). They are independent. The IM_ASSERT kills the ImGui main loop, the io_pool gets shut down during finalization, the hook server thread continues serving requests.

The exact mechanism of `_io_pool.shutdown` being called in this scenario is not directly observable from the log. It could be:
- `ThreadPoolExecutor.__del__` during GC (Hypothesis C path)
- An atexit handler installed by the warmup system or another module
- A signal delivery I haven't identified (e.g., the Python interpreter catching the exception and sending SIGTERM internally)

**This matters less than the actual fix.** The IM_ASSERT is the trigger; the io_pool shutdown is a downstream consequence. Fixing the IM_ASSERT (the real bug) prevents all of this.

---

## Why Did The IM_ASSERT Only Fire In Batch?

The IM_ASSERT is deterministic: it fires every time the offending code path is rendered. In isolation, `test_full_live_workflow` runs alone — it does not exercise the same render functions as the 4 sims. In batch, the sims run first:
- `test_context_sim_live` (ContextSimulation)
- `test_ai_settings_sim_live` (AISettingsSimulation)
- `test_tools_sim_live` (ToolsSimulation)
- `test_execution_sim_live` (ExecutionSimulation)

Each sim opens specific panels (Context, AI Settings, Tools, Execution) and triggers render paths that may be unique to those simulations. After 4 sims, the cumulative state of `ImGui`'s internal scope stack is corrupted — a `begin()` was called in a panel that's only opened in sim mode, and its matching `end()` was either:
- In a code path that was skipped due to a conditional render
- In a `defer` block that early-returned
- After a `return` inside a panel function
- Skipped due to a conditional in the render loop

The IM_ASSERT then fires at frame 71.5s, which is some specific frame AFTER the sims have set up state. The exact render function is unknown without running `scripts/check_imgui_scopes.py` against the full codebase.

---

## Why Did My Previous Fixes Fail?

### Fix 1: io_pool 4→8 (commit `4a338486`)

**Wrong diagnosis:** I assumed the io_pool was saturated with sims' AI discussion turn workers, causing the new switch to queue forever.

**Actual cause:** The io_pool isn't the bottleneck. The switch runs in ~8-10ms. The pool wasn't saturated at the time of the test's switch.

**Why the commit doesn't hurt:** Bigger pool is a marginal improvement to startup concurrency. It's not a regression, just a fix for an issue that isn't the root cause.

### Fix 2: Test timeout 30s→120s (commit `c9a991bb`)

**Wrong diagnosis:** I assumed the switch was slow and just needed more time.

**Actual cause:** The switch is fast. The test fails because the click can't even reach the switch handler — `submit_io` throws at line 2282 because the pool is shut down.

**Why the commit doesn't hurt:** A longer timeout gives a clearer error message (the actual `RuntimeError: cannot schedule new futures after shutdown` is surfaced) but doesn't change the outcome. If anything, it makes the test more annoying to wait for.

### Fix 3 (uncommitted, reverted): Dedicated executor for switches

**Wrong diagnosis:** I assumed the project switch should not share a pool with background work.

**Actual cause:** The pool is fine. The pool gets killed by the GUI crash.

**Why the commit was reverted:** It added complexity for a non-issue. Per `conductor/workflow.md`: "Don't ship a known regression to save time."

---

## The Three Fixes I Have NOT Yet Attempted

Per the systematic-debugging skill, the architectural question needs user input. The 3 viable directions are documented in `docs/reports/test_full_live_workflow_propagation_digest_20260608.md` (to be written — see TODOs).

### Direction A: Fix the actual ImGui scope bug

**What:** Run `scripts/check_imgui_scopes.py` to find the `begin()`/`end()` mismatch. Fix the offending render function.

**Pros:** Real fix. Solves the root cause.

**Cons:** May require deep investigation across 90+ render functions. May be in a render path that's only triggered by a specific sim panel combination. Could take significant time.

**Risk:** Medium. A wrong fix could break other tests or hide the real issue.

### Direction B: Wrap `immapp.run` in `try/except RuntimeError`

**What:** In `src/gui_2.py:618`, wrap `immapp.run(...)` in a `try/except RuntimeError` (or broader). On exception, log it and let the app continue in a degraded state (e.g., skip the rest of the frame, return to event loop).

**Pros:** Band-aid that prevents the GUI crash from propagating to the process. Tests continue to work. Easier to implement than Direction A.

**Cons:** Hides the actual ImGui scope bug. Future tests may exhibit other weirdness from the scope mismatch. The user has said they don't want a "wrap" that just silently continues.

**Risk:** Low for tests, but masks a real bug. Per user: "I don't want the entire test to just linger or silently continue."

### Direction C: Make `_io_pool.shutdown` recoverable

**What:** In `submit_io`, check if the pool is shut down. If so, recreate it (lazily).

**Pros:** Decouples the test from the io_pool's lifecycle. Makes the controller more robust to GUI crashes.

**Cons:** Doesn't address the IM_ASSERT root cause. The GUI is still crashing — we're just hiding the consequences.

**Risk:** Low. Standard pattern for resilient thread pools.

### Direction D: Make the batch runner handle the failure cleanly

**What:** When a test file fails, the `run_tests_batched.py` runner currently continues to the next batch. The fix is to ensure: (1) the failing test file is marked as failed, (2) the next batch can start with a clean state (kill and restart the sloppy.py subprocess per batch, not session-wide).

**Pros:** Doesn't require fixing the underlying bug. Tests can fail without poisoning subsequent batches.

**Cons:** Doesn't fix `test_full_live_workflow`. The test still fails in its own batch.

**Risk:** Low. Standard pattern for test isolation.

### User's explicit guidance

Per the user:
- "I don't want the entire test to just linger or silently continue"
- "I also don't want a batch to be too fragile where I can't restart the app and continue with the next test file if it fails"
- "Just has to note that the new file didn't get to deal with a dirty state"
- "The wrap might be worth it if that properly lets us handle the assert"

The user wants:
1. A report (this document)
2. A todo list for the actual fixes
3. A digest of probable solutions
4. Batch-level resilience (kill+restart sloppy.py per file)
5. The wrap-around-`immapp.run` is acceptable IF it properly handles the assert (not just swallows it)

---

## Files Referenced

- `src/app_controller.py:2282` — `submit_io` line that throws the RuntimeError
- `src/app_controller.py:762` — `_on_sigint` shutdown
- `src/app_controller.py:2325` — `controller.shutdown` pool shutdown
- `src/gui_2.py:618` — `immapp.run(...)` call site (the IM_ASSERT trigger)
- `src/gui_2.py:620` — `self.shutdown()` (only called on normal exit)
- `src/api_hooks.py:117-136` — `/api/project_switch_status` endpoint
- `src/api_hooks.py:11` — `from http.server import ThreadingHTTPServer` (independent of io_pool)
- `tests/test_live_workflow.py:90-94` — `wait_for_project_switch` call
- `tests/test_live_workflow.py:84-89` — defensive `os.path.exists` check
- `tests/conftest.py:263-280` — `kill_process_tree` (uses `taskkill /F` on Windows; no signal)
- `tests/conftest.py:111-126` — pytest smart watchdog (300s timeout in PARENT process)
- `tests/conftest.py:516-547` — `live_gui` fixture finally block (session-scoped, only fires at end)
- `scripts/check_imgui_scopes.py` — EXISTING audit script that can detect this class of bug
- `logs/sloppy_py_test.log` — captured subprocess stderr from the failing test run

## Diagnostic Logging (temporarily added, then reverted)

```python
# src/app_controller.py:2731-2763 (REVERTED)
import sys as _diag_sys
_diag_t0 = time.time()
def _diag(step: str) -> None:
    print(f"[switch-diag] +{time.time()-_diag_t0:.3f}s {step} path={Path(path).name}",
          file=_diag_sys.stderr, flush=True)
_diag("enter")
# ... at every step ...
```

All diagnostic logging has been removed. The production code is back to the pre-diagnostic state. The pattern can be re-applied if needed (e.g., to find which `begin()` in which render function is unbalanced).
