docs(tier2): what's special about test_z_negative_flows
User asked why this test is uniquely affected. Answer: it's the ONLY tier-3 test where the AI call runs ASYNCHRONOUSLY in the io_pool worker while the imgui-bundle render loop continues on the main thread. Verified: test_visual_orchestration.py::test_mma_epic_lifecycle uses the same provider setup (gemini_cli + mock_gemini_cli.py + click) but calls orchestrator_pm.generate_tracks() synchronously in the main thread, blocking the render loop. It PASSES in 11s. test_mma_step_mode_sim.py::test_mma_step_mode_approval_flow also uses the async path but is @pytest.mark.skipif(not RUN_MMA_INTEGRATION) - skipped by default. Would likely also crash if unsuppressed. All other MockProvider tests short-circuit at ai_client.send and never spawn a subprocess. The crash is on the MAIN thread (1.94 MB stack, verified via kernel32.GetCurrentThreadStackLimits), not the io_pool worker (which has 8MB after threading.stack_size(8MB) patch). The main thread's imgui-bundle render loop runs concurrently with the io_pool worker's subprocess.Popen / process.communicate. The accumulated imgui-bundle C++ frames exhaust the main thread's 1.94 MB stack. This explains: - Why bumping io_pool stack to 8MB doesn't help (the patch can't reach the main thread, which was created before any sitecustomize runs). - Why the standalone subprocess call works (no render loop concurrent). - Why the no-click baseline survives 60s (no AI call to trigger the race). Next step: capture a Windows crash dump via procdump or cdb.exe to confirm the crashing thread is the main thread and identify the specific imgui-bundle C++ stack frame.
This commit is contained in:
@@ -0,0 +1,112 @@
|
||||
# What's Special About `test_z_negative_flows.py`
|
||||
|
||||
## TL;DR
|
||||
|
||||
`test_z_negative_flows.py` is the **only** tier-3 test where the AI call runs **asynchronously** in the io_pool worker thread while the **imgui-bundle render loop continues on the main thread**. Other tests using the same `gemini_cli` provider + `mock_gemini_cli.py` setup either:
|
||||
- Run the AI call **synchronously** in the main thread (render loop is blocked) — `test_visual_orchestration.py`
|
||||
- Use a stub/MockProvider and never spawn a subprocess — most other tier-3 tests
|
||||
|
||||
## Verified empirically
|
||||
|
||||
Ran `test_visual_orchestration.py::test_mma_epic_lifecycle` (which uses the same provider setup, sets `gcli_path` to the mock, clicks `btn_mma_plan_epic`). It **PASSED in 11.01s**. The gemini_cli subprocess was spawned and returned successfully.
|
||||
|
||||
`test_z_negative_flows.py` (same provider, same mock, clicks `btn_gen_send`) dies with `0xC00000FD` within 1s.
|
||||
|
||||
## The structural difference
|
||||
|
||||
### `test_visual_orchestration.py` click handler chain
|
||||
```
|
||||
btn_mma_plan_epic click
|
||||
→ render loop processes click task
|
||||
→ _cb_plan_epic() # SYNC, runs on main thread
|
||||
→ orchestrator_pm.generate_tracks() # SYNC, on main thread
|
||||
→ ai_client.send() # SYNC, on main thread
|
||||
→ _send_gemini_cli() # SYNC, on main thread
|
||||
→ GeminiCliAdapter.send() # SYNC, on main thread
|
||||
→ subprocess.Popen() # SYNC, on main thread
|
||||
→ process.communicate() # blocks main thread until subprocess exits
|
||||
```
|
||||
|
||||
The main thread blocks on `process.communicate()`. The render loop is paused. The subprocess returns. The main thread resumes.
|
||||
|
||||
### `test_z_negative_flows.py` click handler chain
|
||||
```
|
||||
btn_gen_send click
|
||||
→ render loop processes click task
|
||||
→ _handle_generate_send() # click handler returns immediately
|
||||
→ submit_io(worker) # worker runs in io_pool thread
|
||||
→ worker:
|
||||
→ _do_generate() # worker thread
|
||||
→ event_queue.put("user_request")
|
||||
→ (returns, thread free)
|
||||
→ render loop CONTINUES # main thread NOT blocked
|
||||
→ render loop continues to next frame
|
||||
→ render loop continues to next frame
|
||||
→ ... (many frames, lots of imgui-bundle native calls)
|
||||
|
||||
Meanwhile, _process_event_queue (separate thread):
|
||||
→ submit_io(_handle_request_event)
|
||||
→ worker:
|
||||
→ ai_client.send() # worker thread
|
||||
→ _send_gemini_cli() # worker thread
|
||||
→ GeminiCliAdapter.send() # worker thread
|
||||
→ subprocess.Popen() # WORKER THREAD (8MB stack)
|
||||
→ process.communicate() # blocks WORKER thread
|
||||
```
|
||||
|
||||
The main thread is **NOT blocked**. The imgui-bundle render loop continues running at 60fps, making native C++ draw calls. **At the same time**, the io_pool worker is doing `subprocess.Popen` and `process.communicate`.
|
||||
|
||||
## Why this matters
|
||||
|
||||
The main thread has only **1.94 MB** of stack (PE-header-baked default for 64-bit Python on Windows). The io_pool worker has 8 MB after `threading.stack_size(8 * 1024 * 1024)`.
|
||||
|
||||
When the io_pool worker calls `subprocess.Popen`:
|
||||
- Windows calls `CreateProcessW`
|
||||
- The kernel allocates a new process, address space, handles
|
||||
- The child Python interpreter starts loading modules
|
||||
|
||||
Concurrently, the main thread's imgui-bundle render loop is:
|
||||
- Allocating frame draw lists
|
||||
- Calling ImGui widget code (text rendering, layout calc, font atlas lookup)
|
||||
- Each frame's C++ call stack grows to ~50-200 KB depending on what's visible
|
||||
|
||||
The crash is `STATUS_STACK_OVERFLOW` (0xC00000FD) on the **main thread**, not the io_pool worker. The 1.94 MB main thread stack is exhausted by accumulated imgui-bundle C++ frames during the seconds when the io_pool worker is doing subprocess operations.
|
||||
|
||||
The "after `_send_gemini_cli` returns" timing in the depth log is incidental — it just happens to be when the main thread's render loop hits the stack limit on its next draw call, which is concurrent with the io_pool worker's work.
|
||||
|
||||
## Why the 8MB io_pool stack fix didn't help
|
||||
|
||||
Bumping `threading.stack_size(8 * 1024 * 1024)` made the io_pool workers (and the `_loop_thread`) have 8 MB stacks. The crash still happened because the overflow is in the **main thread** (1.94 MB, not affected by the patch). The patch can't help.
|
||||
|
||||
## What it would take to fix
|
||||
|
||||
Either:
|
||||
1. **Increase the main thread's stack size** via `editbin /STACK:8388608 python.exe` (Windows tool) or recompile Python with a larger main-thread default. Out of scope for the typical 1-track fix.
|
||||
2. **Move the render loop off the main thread** (imgui-bundle's offscreen rendering mode) — large refactor.
|
||||
3. **Identify the specific imgui-bundle call that's the stack hog** and reduce its C++ frame usage. Requires a Windows crash dump (`procdump -ma sloppy.py` or `cdb.exe -g -G -o sloppy.py`).
|
||||
|
||||
## Why other tests don't trigger this
|
||||
|
||||
- **`test_visual_orchestration.py`**: AI call is SYNCHRONOUS in the main thread. Render loop is paused. No concurrency = no crash.
|
||||
- **`test_mma_step_mode_sim.py`**: `@pytest.mark.skipif(not os.environ.get("RUN_MMA_INTEGRATION"))` — skipped by default. The MMA pipeline does run async via io_pool BUT also uses subprocess (similar to negative_flows) — if we unsuppressed this test, it would likely also crash.
|
||||
- **MockProvider tests** (`test_live_gui_integration_v2.py`, `test_visual_mma.py`, etc.): never reach `subprocess.Popen`. `MockProvider.send()` returns immediately with a fake Result. No native code path beyond simple Python.
|
||||
|
||||
## Actionable next step
|
||||
|
||||
Capture a Windows crash dump to verify the crash is in the main thread (not the io_pool worker):
|
||||
|
||||
```powershell
|
||||
# Option 1: procdump (small CLI tool from Sysinternals)
|
||||
procdump -ma -e 1 -f "" uv run python sloppy.py --enable-test-hooks
|
||||
|
||||
# Option 2: cdb.exe (Windows debugger)
|
||||
cdb.exe -g -G -o sloppy.py --enable-test-hooks
|
||||
> .dump /ma C:\crashes\sloppy.dmp
|
||||
```
|
||||
|
||||
The `.dmp` file contains full C-side call stacks for ALL threads. Open it in WinDbg or VS and run `!analyze -v` to see the crashing thread and stack frame.
|
||||
|
||||
## Files in this report
|
||||
|
||||
- This file: `scripts/tier2/artifacts/send_result_to_send_20260616/WHATS_SPECIAL.md`
|
||||
- Supporting evidence: `logs/sloppy_no_click_*.log` (process survives 60s without clicks), `scripts/tier2/artifacts/send_result_to_send_20260616/test_visual_orch_out.txt` (visual_orchestration PASSED)
|
||||
Reference in New Issue
Block a user