diff --git a/scripts/tier2/artifacts/send_result_to_send_20260616/WHATS_SPECIAL.md b/scripts/tier2/artifacts/send_result_to_send_20260616/WHATS_SPECIAL.md new file mode 100644 index 00000000..dd15c475 --- /dev/null +++ b/scripts/tier2/artifacts/send_result_to_send_20260616/WHATS_SPECIAL.md @@ -0,0 +1,112 @@ +# What's Special About `test_z_negative_flows.py` + +## TL;DR + +`test_z_negative_flows.py` is the **only** tier-3 test where the AI call runs **asynchronously** in the io_pool worker thread while the **imgui-bundle render loop continues on the main thread**. Other tests using the same `gemini_cli` provider + `mock_gemini_cli.py` setup either: +- Run the AI call **synchronously** in the main thread (render loop is blocked) — `test_visual_orchestration.py` +- Use a stub/MockProvider and never spawn a subprocess — most other tier-3 tests + +## Verified empirically + +Ran `test_visual_orchestration.py::test_mma_epic_lifecycle` (which uses the same provider setup, sets `gcli_path` to the mock, clicks `btn_mma_plan_epic`). It **PASSED in 11.01s**. The gemini_cli subprocess was spawned and returned successfully. + +`test_z_negative_flows.py` (same provider, same mock, clicks `btn_gen_send`) dies with `0xC00000FD` within 1s. + +## The structural difference + +### `test_visual_orchestration.py` click handler chain +``` +btn_mma_plan_epic click + → render loop processes click task + → _cb_plan_epic() # SYNC, runs on main thread + → orchestrator_pm.generate_tracks() # SYNC, on main thread + → ai_client.send() # SYNC, on main thread + → _send_gemini_cli() # SYNC, on main thread + → GeminiCliAdapter.send() # SYNC, on main thread + → subprocess.Popen() # SYNC, on main thread + → process.communicate() # blocks main thread until subprocess exits +``` + +The main thread blocks on `process.communicate()`. The render loop is paused. The subprocess returns. The main thread resumes. + +### `test_z_negative_flows.py` click handler chain +``` +btn_gen_send click + → render loop processes click task + → _handle_generate_send() # click handler returns immediately + → submit_io(worker) # worker runs in io_pool thread + → worker: + → _do_generate() # worker thread + → event_queue.put("user_request") + → (returns, thread free) + → render loop CONTINUES # main thread NOT blocked + → render loop continues to next frame + → render loop continues to next frame + → ... (many frames, lots of imgui-bundle native calls) + + Meanwhile, _process_event_queue (separate thread): + → submit_io(_handle_request_event) + → worker: + → ai_client.send() # worker thread + → _send_gemini_cli() # worker thread + → GeminiCliAdapter.send() # worker thread + → subprocess.Popen() # WORKER THREAD (8MB stack) + → process.communicate() # blocks WORKER thread +``` + +The main thread is **NOT blocked**. The imgui-bundle render loop continues running at 60fps, making native C++ draw calls. **At the same time**, the io_pool worker is doing `subprocess.Popen` and `process.communicate`. + +## Why this matters + +The main thread has only **1.94 MB** of stack (PE-header-baked default for 64-bit Python on Windows). The io_pool worker has 8 MB after `threading.stack_size(8 * 1024 * 1024)`. + +When the io_pool worker calls `subprocess.Popen`: +- Windows calls `CreateProcessW` +- The kernel allocates a new process, address space, handles +- The child Python interpreter starts loading modules + +Concurrently, the main thread's imgui-bundle render loop is: +- Allocating frame draw lists +- Calling ImGui widget code (text rendering, layout calc, font atlas lookup) +- Each frame's C++ call stack grows to ~50-200 KB depending on what's visible + +The crash is `STATUS_STACK_OVERFLOW` (0xC00000FD) on the **main thread**, not the io_pool worker. The 1.94 MB main thread stack is exhausted by accumulated imgui-bundle C++ frames during the seconds when the io_pool worker is doing subprocess operations. + +The "after `_send_gemini_cli` returns" timing in the depth log is incidental — it just happens to be when the main thread's render loop hits the stack limit on its next draw call, which is concurrent with the io_pool worker's work. + +## Why the 8MB io_pool stack fix didn't help + +Bumping `threading.stack_size(8 * 1024 * 1024)` made the io_pool workers (and the `_loop_thread`) have 8 MB stacks. The crash still happened because the overflow is in the **main thread** (1.94 MB, not affected by the patch). The patch can't help. + +## What it would take to fix + +Either: +1. **Increase the main thread's stack size** via `editbin /STACK:8388608 python.exe` (Windows tool) or recompile Python with a larger main-thread default. Out of scope for the typical 1-track fix. +2. **Move the render loop off the main thread** (imgui-bundle's offscreen rendering mode) — large refactor. +3. **Identify the specific imgui-bundle call that's the stack hog** and reduce its C++ frame usage. Requires a Windows crash dump (`procdump -ma sloppy.py` or `cdb.exe -g -G -o sloppy.py`). + +## Why other tests don't trigger this + +- **`test_visual_orchestration.py`**: AI call is SYNCHRONOUS in the main thread. Render loop is paused. No concurrency = no crash. +- **`test_mma_step_mode_sim.py`**: `@pytest.mark.skipif(not os.environ.get("RUN_MMA_INTEGRATION"))` — skipped by default. The MMA pipeline does run async via io_pool BUT also uses subprocess (similar to negative_flows) — if we unsuppressed this test, it would likely also crash. +- **MockProvider tests** (`test_live_gui_integration_v2.py`, `test_visual_mma.py`, etc.): never reach `subprocess.Popen`. `MockProvider.send()` returns immediately with a fake Result. No native code path beyond simple Python. + +## Actionable next step + +Capture a Windows crash dump to verify the crash is in the main thread (not the io_pool worker): + +```powershell +# Option 1: procdump (small CLI tool from Sysinternals) +procdump -ma -e 1 -f "" uv run python sloppy.py --enable-test-hooks + +# Option 2: cdb.exe (Windows debugger) +cdb.exe -g -G -o sloppy.py --enable-test-hooks +> .dump /ma C:\crashes\sloppy.dmp +``` + +The `.dmp` file contains full C-side call stacks for ALL threads. Open it in WinDbg or VS and run `!analyze -v` to see the crashing thread and stack frame. + +## Files in this report + +- This file: `scripts/tier2/artifacts/send_result_to_send_20260616/WHATS_SPECIAL.md` +- Supporting evidence: `logs/sloppy_no_click_*.log` (process survives 60s without clicks), `scripts/tier2/artifacts/send_result_to_send_20260616/test_visual_orch_out.txt` (visual_orchestration PASSED)