# Digest: Probable Solutions for ImGui Assert Propagation Failure (2026-06-08)

**Companion to:** `docs/reports/test_full_live_workflow_imgui_assert_20260608.md`
**Companion to:** `conductor/todos/TODO_test_full_live_workflow_v2.md`
**Status:** Pre-implementation analysis. User has not yet chosen a direction.
**Audience:** Future implementer, the user (decision reference)

---

## 1. The Problem (restated)

When `tests/test_extended_sims.py` (4 sims) runs before `tests/test_live_workflow.py` in the tier-3 batch, an ImGui `IM_ASSERT((0) && "Missing End()")` fires at ~71.5s into GUI lifetime in window 'MainDockSpace'. The `RuntimeError` propagates from `immapp.run` through `app.run()` and `main()`. The hook server thread (separate `ThreadingHTTPServer`) survives. The `_io_pool` ends up in a shutdown state (mechanism unclear — likely `ThreadPoolExecutor.__del__` during GC). Subsequent test clicks fail with `RuntimeError: cannot schedule new futures after shutdown`.

The test poll loop then waits 120s before timing out.

---

## 2. Constraints (from user)

Per the user's session feedback (2026-06-08):
- "I don't want the entire test to just linger or silently continue" — silent failure is not acceptable
- "I also don't want a batch to be too fragile where I can't restart the app and continue with the next test file if it fails" — batch isolation is required
- "Just has to note that the new file didn't get to deal with a dirty state" — the failure should be observable, not hidden
- "The wrap might be worth it if that properly lets us handle the assert" — a proper wrap (with logging + observable state) is acceptable; a silent swallow is not

The user wants:
1. A real fix where possible
2. A wrap that surfaces the failure (not swallows it)
3. Batch resilience (a failed batch should not poison the next)
4. Tests should be able to detect a degraded GUI and fail fast with a clear message

---

## 3. Solution Matrix

The 6 tasks in `conductor/todos/TODO_test_full_live_workflow_v2.md` are presented here as a solutions matrix. Each is evaluated on:
- **Real-fix value** (does it address the root cause?)
- **Test impact** (does it make `test_full_live_workflow` pass in batch?)
- **Effort** (hours)
- **Risk** (chance of regression)
- **User alignment** (does it match the user's stated constraints?)

| # | Solution | Real fix? | Test impact | Effort | Risk | User-aligned? |
|---|----------|-----------|------------|--------|------|---------------|
| 1 | Run `check_imgui_scopes.py` to find the scope mismatch | **Yes** (the actual bug) | Yes (if successful) | 1-2h | Med | Yes |
| 2 | Fix the identified ImGui scope mismatch | **Yes** (the actual bug) | Yes (if successful) | 1-4h | Med | Yes |
| 3 | Wrap `immapp.run` in `try/except RuntimeError` | No (band-aid) | Yes (prevents crash) | 1-2h | Low | Yes (per user: "might be worth it if it properly handles") |
| 4 | Kill+restart sloppy.py per test file | No (isolation) | Yes (clean state) | 2-4h | Low | Yes (per user: "I can't restart the app and continue with the next test file if it fails") |
| 5 | Make `submit_io` recover from a shut-down pool | No (resilience) | Yes (survives crash) | 0.5h | Low | Yes (defense in depth) |
| 6 | Add `/api/gui_health` endpoint | No (observability) | Yes (fast-fail) | 1-2h | Low | Yes (per user: "Just has to note that the new file didn't get to deal with a dirty state") |

---

## 4. Solution Details (Probable Approaches)

### 4.1 Solution 1+2: Audit + Fix the ImGui Scope Mismatch

**Approach A: Manual scope audit (lowest risk)**

1. Run `python scripts/check_imgui_scopes.py` against `src/gui_2.py`
2. Triage findings — many will be false positives (e.g., `begin()` inside a conditional that has an `end()` in the other branch via context manager)
3. Identify the SPECIFIC `render_*` function with the unbalanced scope
4. Inspect that function's render path and find the missing `end()` or the extra `begin()`

**Probable location of the bug:** A render function that's only called in one of the sims' panel render paths. Candidates:
- Render functions for the AI Settings panel (test_ai_settings_sim_live)
- Render functions for the Tools panel (test_tools_sim_live)
- Render functions for the Execution/Modals panel (test_execution_sim_live)
- Render functions for the Context & Chat panel (test_context_sim_live)

**Probable cause of the bug:** A recent render refactor that added a `begin()` to show a tooltip or popup, but the matching `end()` is inside an `if` branch that can early-return. Or a `begin()` that's only conditionally reached, with the `end()` always called but the stack now has an extra entry.

**Approach B: Defer-not-catch pattern (per `conductor/workflow.md` known pitfall)**

The known pitfall section describes:
> `imgui-bundle` (and similar native extension libraries) expose C-level functions that can crash the Python process with a Windows access violation (`0xc0000005`) or a SIGSEGV on Linux. **These crashes are not catchable from Python** — `try/except Exception` does not intercept native access violations, only Python exceptions.

The IM_ASSERT we observed IS a Python RuntimeError (catchable). But the fix pattern is similar: use `imscope` context managers (per `src/imgui_scopes.py`) to ensure scopes are balanced even on early returns.

**Implementation pattern:**
```python
# Before (buggy)
def render_my_panel(app):
    imgui.begin("My Panel")
    if not app.some_state:
        return  # <-- Missing end()!
    imgui.text("hello")
    imgui.end()

# After (fixed)
def render_my_panel(app):
    with imscope(imgui.begin, "My Panel"):  # auto end() on exit
        if not app.some_state:
            return
        imgui.text("hello")
        # end() called automatically
```

**Effort:** 1-4 hours (depends on what the audit finds).

**Risk:** Medium. The fix may need to be applied to multiple render functions, each requiring careful testing.

**Confidence in success:** Medium. The IM_ASSERT is deterministic and ImGui's scope tracking is reliable. Once the offending function is found, the fix is mechanical.

### 4.2 Solution 3: Wrap `immapp.run` in `try/except RuntimeError`

**Approach: Catch and recover**

```python
# src/gui_2.py:617-621
try:
    immapp.run(self.runner_params, add_ons_params=immapp.AddOnsParams(with_markdown_options=md_options))
except RuntimeError as e:
    # IM_ASSERT (Missing End()) or similar. Log the error, mark the GUI
    # as degraded, and let the hook server continue. Per user feedback
    # (2026-06-08): the wrap is acceptable IF it surfaces the failure
    # (does not silently swallow).
    self.controller._gui_degraded_reason = f"immapp.run raised: {e}"
    self.controller._last_imgui_assert = traceback.format_exc()
    print(f"[GUI-DEGRADED] {self.controller._gui_degraded_reason}", file=sys.stderr, flush=True)
    # Do NOT call self.shutdown() — keep the hook server alive for tests.
    # The io_pool may be in a weird state; lazy-recreate on next submit_io (Task 5).
# On normal exit
if not self.controller._gui_degraded_reason:
    self.shutdown()
    session_logger.close_session()
```

**Key design choices:**
1. **The wrap does NOT call `self.shutdown()`** when the assert fires. This keeps the hook server alive so subsequent tests can still query state.
2. **The error is logged at ERROR level** with the full assert message and stack trace. This is observable, not silent.
3. **The controller sets `_gui_degraded_reason` and `_last_imgui_assert`** so the new `/api/gui_health` endpoint (Task 6) can expose the state to tests.
4. **Tests can detect the degraded state** via `client.get_gui_health()` and fail fast with a clear message.

**Effort:** 1-2 hours (wrap + logging + state).

**Risk:** Low. The wrap is a band-aid, but a transparent one. The error is still surfaced.

**Confidence in success:** High. The wrap itself is trivial. The integration with `/api/gui_health` and the test-side fast-fail is straightforward.

### 4.3 Solution 4: Kill+Restart sloppy.py per Test File

**Approach: Per-file fixture scope**

Currently, the `live_gui` fixture in `tests/conftest.py` is session-scoped. All live_gui tests share the same `sloppy.py` subprocess for the entire test session. If the subprocess crashes mid-session, all subsequent tests are poisoned.

**Change:** Make the fixture per-file-scoped (or function-scoped with smart re-spawn). When a test in file A finishes, the subprocess is killed. When file B starts, a fresh subprocess is spawned.

**Implementation pattern:**

```python
# tests/conftest.py (modified)
@pytest.fixture(scope="module")  # was: "session"
def live_gui(request):
    # If the prior file's subprocess died, this is a fresh start.
    # If the user wants true per-test isolation, change to "function".
    ...
    try:
        yield process, gui_script
    finally:
        kill_process_tree(process.pid)
        log_file.close()
        shutil.rmtree(temp_workspace)
```

**Considerations:**
- **Performance:** Each test file's fixture spawn takes ~1-2s. With 49 live_gui files, that's 49-98s added to the tier-3 batch. Probably acceptable.
- **State persistence:** Currently, tests can rely on state from a prior test (e.g., a project loaded by sim 1 is still loaded when sim 2 runs). Making the fixture per-file-scoped breaks this. **Most live_gui tests should NOT depend on prior test state** — they should set up their own state. This is the principle the v1 report identified.
- **Watchdog interaction:** The conftest's smart watchdog (300s) and unconditional watchdog (900s) are based on `_pytest_finished_event`. The per-file fixture change is independent.

**Alternative approach: Smart re-spawn**

Keep the fixture session-scoped, but add a "re-spawn" check at the start of each test. If the subprocess is dead, spawn a new one. The fixture becomes a "lazy" fixture that may spawn multiple subprocesses over the session.

**Probable implementation:**
```python
@pytest.fixture(scope="session")
def live_gui(request):
    process, gui_script = _spawn_sloppy()
    yield _LazyLiveGui(process, gui_script)
    kill_process_tree(process.pid)

class _LazyLiveGui:
    def __init__(self, process, gui_script):
        self._process = process
        self._gui_script = gui_script
        self._lock = threading.Lock()
    
    def get_process(self):
        with self._lock:
            if self._process.poll() is not None:
                # Respawn
                self._process = _spawn_sloppy()
            return self._process
```

This is more complex but preserves the per-session state model for tests that need it.

**Effort:** 2-4 hours (per-file approach is simpler; lazy re-spawn is more invasive).

**Risk:** Low. The per-file approach is straightforward. The lazy re-spawn is more complex but doesn't change the API surface for tests.

**Confidence in success:** High. The pattern is well-established in test infrastructure.

### 4.4 Solution 5: Make `submit_io` Recover from a Shut-Down Pool

**Approach: Lazy recreation**

```python
# src/app_controller.py:2275-2284
def submit_io(self, fn, *args, **kwargs):
    if not hasattr(self, "_io_pool") or self._io_pool is None or self._is_io_pool_shutdown():
        # Recreate the pool (it was shut down, e.g., by a GUI crash).
        self._io_pool = make_io_pool()
        self._io_pool_inflight = 0
    if not hasattr(self, "_io_pool_inflight_lock"):
        self._io_pool_inflight_lock = threading.Lock()
    with self._io_pool_inflight_lock:
        self._io_pool_inflight = getattr(self, "_io_pool_inflight", 0) + 1
    future = self._io_pool.submit(fn, *args, **kwargs)
    future.add_done_callback(lambda _f: self._io_pool_inflight_done())
    return future

def _is_io_pool_shutdown(self) -> bool:
    """True if the io_pool has been shut down (e.g., via __del__ or controller.shutdown)."""
    pool = getattr(self, "_io_pool", None)
    if pool is None:
        return True
    # ThreadPoolExecutor doesn't expose a clean "is_shutdown" method.
    # Use a try/except probe.
    try:
        pool.submit(lambda: None).result(timeout=0.001)
        return False
    except RuntimeError:
        return True
    except Exception:
        return False
```

**Key design choices:**
1. **The lazy-recreate happens transparently.** Callers don't need to know about pool lifecycle.
2. **The inflight counter is reset** when the pool is recreated (the old workers are dead).
3. **The probe (`_is_io_pool_shutdown`) is best-effort.** `ThreadPoolExecutor` doesn't expose a clean shutdown check; a `submit().result()` probe is the standard pattern.

**Effort:** 30 minutes.

**Risk:** Low. The pool was already designed to be replaceable (per the existing test infrastructure). The probe is a non-invasive check.

**Confidence in success:** High. Standard pattern for resilient thread pools.

### 4.5 Solution 6: Add `/api/gui_health` Endpoint

**Approach: Read-only health endpoint**

```python
# src/api_hooks.py (new elif branch in do_GET)
elif self.path == "/api/gui_health":
    self.send_response(200)
    self.send_header("Content-Type", "application/json")
    self.end_headers()
    controller = _get_app_attr(app, "controller", None)
    if controller is None:
        payload = {"healthy": True, "degraded_reason": None, "last_assert": None, "io_pool_alive": True}
    else:
        payload = {
            "healthy": getattr(controller, "_gui_degraded_reason", None) is None,
            "degraded_reason": getattr(controller, "_gui_degraded_reason", None),
            "last_assert": getattr(controller, "_last_imgui_assert", None),
            "io_pool_alive": not controller._is_io_pool_shutdown() if hasattr(controller, "_is_io_pool_shutdown") else True,
        }
    self.wfile.write(json.dumps(payload).encode("utf-8"))
```

**Test integration:**

```python
# tests/test_live_workflow.py (new pre-check at start)
def test_full_live_workflow(live_gui) -> None:
    client = ApiHookClient()
    assert client.wait_for_server(timeout=10)
    # New: check GUI health before proceeding
    health = client.get_gui_health()
    if not health.get("healthy"):
        pytest.fail(
            f"GUI is degraded before test starts: "
            f"degraded_reason={health.get('degraded_reason')}, "
            f"last_assert={health.get('last_assert')}"
        )
    ...
```

**Effort:** 1-2 hours.

**Risk:** Low. Read-only endpoint + trivial client method + simple pre-check.

**Confidence in success:** High.

---

## 5. Recommended Combination

The user's constraints suggest a **layered defense**:

### Layer 1: Real fix (Tasks 1+2)
Find and fix the ImGui scope mismatch. This is the right thing to do.

### Layer 2: Safety net (Task 3)
Wrap `immapp.run` so a future scope mismatch doesn't kill the process. Log the error, mark GUI as degraded.

### Layer 3: Observability (Task 6)
Expose the degraded state via `/api/gui_health`. Tests can fast-fail with a clear message.

### Layer 4: Resilience (Task 5)
Make `submit_io` recover from a shut-down pool. Defense in depth.

### Layer 5: Batch isolation (Task 4)
Per-file (or per-test) fixture scope for `live_gui`. A failed batch doesn't poison the next.

**Recommended PR order:**
- **PR 1 (highest priority):** Tasks 1+2 (real fix)
- **PR 2 (in parallel with PR 1):** Tasks 3+6 (wrap + observability) — these don't depend on finding the bug
- **PR 3:** Task 4 (batch isolation) — independent of the bug, valuable on its own
- **PR 4 (defense in depth):** Task 5 (lazy pool recreation) — only needed if PR 2 doesn't fully solve the problem

### What's NOT recommended

- **Task 5 alone, without Tasks 1+2+3:** The lazy-recreate hides the bug. The io_pool will keep getting shut down by the GUI crash, then recreated, then shut down again, etc. The test will pass but the GUI is still broken.
- **Task 4 alone, without the real fix:** Batch isolation is good hygiene, but the test still fails in its own batch. The bug is not fixed.
- **A silent swallow in Task 3:** The user explicitly rejected this. A silent failure is worse than a visible failure.

---

## 6. Open Questions for the User

Before implementation, these need clarification:

1. **Task 3 wrap behavior:** When the IM_ASSERT fires, should the wrap:
   a. Just log and return (degraded mode, no further renders)
   b. Try to continue the render loop (skip the current frame, retry next frame)
   c. Restart the ImGui context entirely (clear and reinitialize)
   
   My recommendation: (a) is the safest. (b) risks infinite loops if the scope is genuinely broken. (c) is invasive.

2. **Task 4 scope:** Per-file or per-test? Per-test is more isolated but adds 49+ fixture spawns. Per-file is a middle ground.
   
   My recommendation: Per-file for now. Per-test can be added later if needed.

3. **Task 5 proactive recreation:** Should the pool be recreated eagerly (at controller init) or lazily (on first submit after shutdown)?
   
   My recommendation: Lazy. The pool is normally alive; recreating eagerly is wasteful.

4. **New state attributes:** `_gui_degraded_reason` and `_last_imgui_assert` — should they be persisted to disk (e.g., for crash analysis) or just in-memory?
   
   My recommendation: In-memory. Disk persistence is overkill for transient runtime state.

5. **Compatibility with existing tests:** Some existing tests may assume the io_pool is always available. Will the lazy-recreate break them?
   
   My recommendation: No. The lazy-recreate preserves the pool's API surface. The only difference is that the pool may be a fresh instance after a crash. Existing tests don't care about the instance identity.

---

## 7. Effort & Risk Summary

| Solution | Effort | Risk | User-aligned | Recommendation |
|----------|--------|------|--------------|----------------|
| 1+2 (audit + fix) | 2-6h | Med | Yes | **PRIORITY 1: implement first** |
| 3 (wrap) | 1-2h | Low | Yes | **PRIORITY 2: implement in parallel** |
| 4 (batch isolation) | 2-4h | Low | Yes | **PRIORITY 3: independent, implement anytime** |
| 5 (lazy recreate) | 0.5h | Low | Yes | **PRIORITY 4: only if 3 doesn't fully solve** |
| 6 (health endpoint) | 1-2h | Low | Yes | **PRIORITY 2: implement with 3** |

**Total effort (all):** 6-14 hours
**Recommended PR sequence:** 1+2 → 3+6 → 4 → 5

---

## 8. References

- `docs/reports/test_full_live_workflow_imgui_assert_20260608.md` — full debugging report
- `conductor/todos/TODO_test_full_live_workflow_v2.md` — task list
- `scripts/check_imgui_scopes.py` — existing audit script (use for Task 1)
- `src/imgui_scopes.py` — ImGuiScope context manager (use for fix in Task 2)
- `src/api_hooks.py:11` — `ThreadingHTTPServer` (independent of io_pool)
- `src/gui_2.py:618` — `immapp.run(...)` call site
- `src/app_controller.py:2282` — `submit_io` line that throws the RuntimeError
- `conductor/workflow.md` "Known Pitfalls (2026-06-05)" — Defer-Not-Catch Pattern
- `conductor/workflow.md` "Skip-Marker Policy" — don't add skip markers as workarounds
- `AGENTS.md` "Critical Anti-Patterns" — no comments, no batch fragility
