End-of-track report for the 4 sandbox bugs hit by the first Tier 2
run (send_result_to_send_20260616) and the audit infrastructure
added to prevent regression. 5 fixes (4 bugs + 1 audit) shipped as
6 atomic commits on master.
See the report for:
- Per-fix description, root cause, and file:line refs
- Live clone state after the fixes
- 38 default-on + 3 opt-in test inventory
- 4 conventions established
- Next steps for the user (re-run, merge review branch, etc.)
- Known follow-ups NOT in this track
Per user feedback this round:
1. T-shirt size removed from conductor/workflow.md (policy),
conductor/tracks.md (registry), and the prior
NEGATIVE_FLOWS_INVESTIGATION_20260617.md report.
2. Layout regenerated from _default_windows (17KB -> 3KB, 10 stale
windows -> 3). Layout fix did NOT fix the crash.
Three new diagnostic experiments (results appended to the report):
- diag_no_click.py: process survives 60s without clicks (render loop
is stable in isolation; crash is click-triggered).
- diag_thread.py: standalone ThreadPoolExecutor + adapter call works
fine in all 3 MOCK_MODE modes (subprocess spawn is not the issue).
- diag_realbig2_run.py: bumping threading.stack_size(8MB) does NOT
prevent the crash (io_pool worker is not where the stack is exhausted).
Refined hypothesis: the crash is in the MAIN THREAD's imgui-bundle
render loop (1.94 MB stack), running concurrently with the io_pool
worker's adapter call. The subprocess spawn + CreateProcessW causes
the kernel to allocate resources at the moment the main thread is
deep in imgui-bundle C++ frames, exhausting the main thread's small
guard page.
What's needed for definitive diagnosis: a Windows crash dump (procdump
-ma or cdb.exe) to see the actual C-side stack frame, OR a
SetUnhandledExceptionFilter in sitecustomize.py that logs the
crashing thread's TEB and call stack to stderr before the process dies.
Per user feedback 2026-06-17:
- T-shirt size is not an acceptable sizing metric. Remove it from
conductor/workflow.md (the policy file), conductor/tracks.md (the
registry), and docs/reports/NEGATIVE_FLOWS_INVESTIGATION_20260617.md.
- Regenerate manualslop_layout.ini to remove 83 stale window references
that pointed to deleted/renamed windows (Projects, Files, Screenshots,
Provider, System Prompts, Discussion History, Comms History, etc.).
Layout now matches the windows registered in src/app_controller.py
_default_windows (lines 1862-1886). Stale window count: 10 -> 3.
T-shirt size removal details:
- conductor/workflow.md: Removed the S/M/L/XL table, the replacement
pattern row, and the 'reasonable effort' guard's reference. Scope
(N files, M sites, N tasks) is the only effort dimension.
- conductor/tracks.md: Removed the T-shirt column from the table header
and removed T-shirt size mentions from the Fable track entry.
- docs/reports/NEGATIVE_FLOWS_INVESTIGATION_20260617.md: Removed the
T-shirt size mention in the follow-up track suggestion.
Layout fix:
- manualslop_layout.ini went from 17,360 bytes (102 windows, 83 stale)
to 3,361 bytes (23 windows, all matching _default_windows). The
stale window warning dropped from 10 windows to 3 (Message, Tool
Calls, Response - these are in _default_windows but reference
separate panels in the layout).
Verification: layout fix did NOT fix the underlying stack overflow crash.
After layout fix, the test still dies with rc=3221225725 (0xC00000FD).
The user noted 'Something more fundamental is wrong.' Investigation
continues; this commit only addresses the explicit ask (remove T-shirt,
fix layout).
Per user feedback:
1. Removed T-shirt size metric from the report. The T-shirt size
convention is defined in conductor/tracks.md (lines 47, 738, 748,
790) and conductor/workflow.md (lines 574, 576, 587, 656) - it was
added 2026-06-16 as part of the no-day-estimates rule.
2. Re-investigated the actual call stack depth. The Python call chain
at crash time is only 13 frames deep. This is NOT a Python
recursion bug.
3. Measured the main thread stack via kernel32.GetCurrentThreadStackLimits.
It is 1.94 MB on this Python 3.11.6 installation. The sitecustomize
sets threading.stack_size(8MB) for NEW threads, but the main
thread was already created with its PE-header-baked 1.94MB.
4. Bumped io_pool workers to 8MB via threading.stack_size(8MB) in
sitecustomize.py. Process STILL dies with 0xC00000FD. So the
stack overflow is NOT in the io_pool worker. It is in the main
thread, running the imgui-bundle render loop.
5. The main thread is 1.94MB. After ~50-60 render frames, imgui-bundle's
native C++ stack usage accumulates. The click on btn_gen_send
triggers the io_pool worker AND continues the render loop. The
next render frame's C++ stack usage overflows the main thread's
1.94MB guard page, killing the process.
The fix is NOT about the io_pool thread stack. It is about either:
(a) reducing imgui-bundle's per-frame C++ stack usage (e.g., fix the
stale manualslop_layout.ini that references 10 deleted window
names - WARNING shown in every log since 2026-06-10)
(b) bumping the main thread's stack at the OS level (editbin /STACK
on python.exe)
(c) running the render loop in a subprocess
Capture a WER crash dump to identify the exact C-side stack frame
that overflows. Add SetUnhandledExceptionFilter via sitecustomize.py
to log the crashing thread's TEB to stderr before the process dies.
User asked to continue investigation of the 3 failing tests in
tests/test_z_negative_flows.py. Ran the test in batched tier-3 mode,
isolated the failure to a native Windows STATUS_STACK_OVERFLOW
(0xC00000FD) in the io_pool worker thread when calling
GeminiCliAdapter.send -> subprocess.Popen -> communicate.
Verified the failure:
- Reproduces 100% on a fresh subprocess (no xdist, no other tests).
- Is NOT caused by the send_result -> send rename (purely mechanical).
- Happens on MOCK_MODE=malformed_json, error_result, AND success
(rules out the exception/traceback construction as cause).
- Adapter body completes normally; process dies immediately after.
- Is the io_pool worker thread's 1MB C stack being exhausted by the
deep call chain (run_with_tool_loop -> asyncio cross-thread
dispatch -> _send -> adapter.send -> subprocess.Popen -> communicate
+ Windows ReadFile/WaitForSingleObject).
Conclusion: pre-existing bug. The test file (originally test_negative_flows.py
from 2026-03-06, renamed to test_z_negative_flows.py on 2026-03-07) is the
ONLY test in the suite that exercises a real subprocess AI call end-to-end
through the io_pool worker. Other tier-3 tests use MockProvider and
short-circuit at the ai_client.send level.
Documented: root cause, reproduction evidence, 4 proposed solutions
(thread stack bump, multiprocessing migration, blocking main thread,
xfail), and a follow-up track suggestion for the long-term fix.
This is an investigation report only; no code changes. The theme fix in
9fcf0517 is unaffected. The rename track in 8c6d9aa0 is unaffected.
The 9fcf0517 fix(theme) commit had also overwritten the track completion
report at 219b653a with a combined analysis. Per user feedback, the
completion report and the post-completion bug analysis belong in two
separate files.
This commit:
- Restores the original completion report (219b653a) unchanged.
- Adds a new report (THEME_BUG_ANALYSIS_*) documenting the
post-completion bug, the actual root cause, the fix, and the
process feedback from the user.
The theme fix itself is unchanged in 9fcf0517.
src/theme_nerv_fx.py:97 was calling draw_list.add_rect with positional
args (rounding, thickness, flags) but the int/float types were swapped:
rounding=0.0 (correct)
thickness=0 (int, signature expects float)
flags=10.0 (float, signature expects int)
The TypeError fires every render frame once ai_status starts with
'error'. App.run's except RuntimeError eventually catches and calls
self.shutdown() -> controller.shutdown() -> _io_pool.shutdown(wait=False).
Subsequent tests in the same live_gui session can't submit_io.
Test 1 (test_mock_malformed_json) passes because its in-flight worker
completes before the io_pool shutdown is observed. Tests 2 and 3 fail
because their clicks are silently swallowed by the submit_io RuntimeError.
Switch to keyword args with correct types. Update test_theme_nerv_fx
assertion to match.
Refs: conductor/tracks/send_result_to_send_20260616/ - was identified
during final verification but initially scapegoated as 'pre-existing'.
Per user feedback, the bug is fixed now.
Verified: test_theme_nerv_fx 5/5 pass. test_z_negative_flows.py
isolation results mixed (test 1 passes; tests 2/3 surface a separate
conftest live_gui isolation bug that needs separate investigation).
Adds a manual-first pipeline for finding UX regressions in long screen recordings: ffmpeg re-encode to proxy, LAB-palette frame-change detection (kasa-style), pixel-diff backup, manual triage into a triage overlay on the existing ASCII UI Layout Map DSL (docs/guide_ascii_layout_map.md). The overlay adds only a thin meta-layer (entry headers, @delta, @ux_finding) on top of the existing visual grammar; the existing DSL remains the source of truth for the visual layer. Includes 8 edge-case worked examples ranked by LLM difficulty and a findings-report template for the user-in-the-loop iteration. Future track candidates: build the keyframe-extraction tool (scripts/dogfood_extract.py) after ≥3 manual dogfoods validate the DSL shape.
User feedback from the first sandbox run (send_result_to_send_20260616,
2026-06-17) identified 6 conventions Tier 2 must follow. Update the agent
prompt template, slash command template, user guide, and workflow doc:
1. Test runner: ALWAYS use 'uv run python scripts/run_tests_batched.py'
(NOT 'uv run pytest'). The batched runner provides tier filtering,
parallelization (xdist), and a summary table that direct pytest lacks.
2. Default branch: this repo uses 'master', not 'main'. The Tier 2 slash
command now does 'git fetch origin master' (was 'origin main').
3. Line endings: preserve existing. This repo has a mix of CRLF and LF;
a repo-wide LF standardization is a future track.
4. Throw-away scripts: write to 'scripts/tier2/artifacts/<track>/', NOT
the base 'scripts/tier2/' directory. The base is reserved for
production code; throw-away scripts are kept for archival but
isolated per-track.
5. End-of-track report: write 'docs/reports/TRACK_COMPLETION_<track>.md'
and update 'state.toml' to 'status=completed'. The user reads this
to decide merge. Previously this was implicit; now it's explicit.
6. Run-time expectation: tracks are 1-4 hours. If context runs out, Tier
2 notes progress to disk and continues. The --resume flag picks up
from the last completed task.
Also updated the user guide with a 'Conventions' section and a
troubleshooting entry for the resume flow. The verify-the-sandbox
checklist now uses 'origin master' instead of 'origin main'.
This one was important to keep is it was the first attempt at an autonomous run.
Essentially worked except for a turn exhaustion on ai side (need to tweak some config maybe).
End-of-track report following the same format as
TRACK_COMPLETION_tier2_autonomous_sandbox_20260616.md. Documents:
- 24-commit inventory (10 atomic renames + 14 plan/script commits)
- All 6 phases completed, all 9 verification flags = true
- Pre-existing failures (7 tests, all credentials.toml, confirmed
against origin/master baseline where they also fail)
- 2 surgical doc fixes in error_handling.md (deprecation section +
line 204 contradiction)
- Sandbox enforcement contracts held (4 of 4 hard bans + 4 of 4
secondary contracts)
- User handoff instructions (fetch + diff + merge + per-commit review)
The track is the first end-to-end test of the tier2_autonomous_sandbox;
this report is the final deliverable for that test.
Doc consistency: guide_ai_client.md, guide_app_controller.md, and
the error_handling styleguide now reference the new symbol name.
Also fixes two consistency issues in error_handling.md introduced by
the mechanical rename:
1. The 'Deprecation: send -> send_result' section (lines 623-642) was
rewritten as a 'Historical deprecation (added 2026-06-15, reverted
2026-06-16)' note that points to the relevant track specs.
2. Line 204 (the 'Current State Audit' summary for src/ai_client.py)
had a self-contradictory claim ('send() is the new public API;
send() is @deprecated') after the rename. Updated to describe
the canonical public API.
Historical archives (conductor/tracks/*/spec.md, conductor/tracks/*/plan.md,
docs/reports/*) are NOT modified - they document the 2026-06-15
public_api_migration decision and stay as historical record.
Comprehensive 12-section completion report following the format of
TRACK_COMPLETION_ai_loop_regressions_20260615.md. Documents:
- 4 atomic commits, 1288+4+0 fully green baseline
- 2 defensive guards in src/rag_engine.py (lines 150 and 331)
- 3 new unit tests in tests/test_rag_sync_none_error.py
- 4 plan deviations (spec wrong about root cause, test_rag_visual_sim
was already passing, traceback diagnostic was a dead end, temp dir
cleanup retry loop for Windows)
- 5 followup recommendations for Tier 1 review
Documents the two bugs fixed in the rag_test_failures_20260615 track:
1. get_all_indexed_paths: m.get('path') failing on None metadata
2. _validate_collection_dim_result: 'if not embeddings' raising
ValueError on non-empty numpy arrays
Also documents the 'no such table: tenants' chromadb corruption
symptom (wipe .slop_cache/chroma_* to recover).
Plus: 'rag_status' shows 'error: ' prefix is the failure indicator;
the actual error message is the part after the prefix.
The headless batch hang the user reported was caused by an xdist worker
crash on test_headless_verification_full_run, not a test logic failure.
The same root cause as the 4 Phase 2 follow-ups (mock returns raw string
but production does 'if not result.ok:'), but with a different failure
mode (worker crash that hangs the batched test runner).
Documented in section 3 of the report as deviation #2.5 with:
- Where it went wrong (missed in the 4 follow-ups)
- The specific symptom in the user's session
- The fix (out-of-band commit e35b6a34)
- Lesson for the next spec (verification must include xdist mode)
531-line completion report for Tier 1 review covering:
- Goal & scope (per spec)
- 7 phases of delivery (per commit)
- 6 plan deviations to flag (CRITICAL: 7 production-affected test files
+ 4 follow-up mock fixes were missed in the original spec; the user's
stated mass-rename send_result->send plan; the track was done on
master not a feature branch)
- Files changed (per category)
- Verification (per the spec's 15 verification criteria)
- Definition of Done
- Recommended next track (send_result -> send rename)
- Tier 1 review checklist
Per plan Task 7.1: removed all deprecation language about ai_client.send()
from docs/guide_ai_client.md:
- Removed the 'Public API > ai_client.send(...) deprecated' section
- Updated 'Migration Notes for Existing Callers' to reflect the
public_api_migration_and_ui_polish_20260615 completion
- Updated 'Public API Result Migration' line in the see-also section
to mark the follow-up track as COMPLETED (not 'planned')
Verification: rg -i 'deprecat.*send|send.*deprecat' docs/guide_ai_client.md
returns 0 hits (the only remaining 'deprecat' mention is the resolved
Public API Result Migration bullet which now describes the resolution
path, not a deprecation).
In-depth handoff for Tier 1 review covering:
- Executive summary with TL;DR
- Goal & scope (planned vs delivered)
- Per-phase delivery summary
- Test coverage analysis (7 new + 2 adapted + 2 smoke)
- Deferred items documentation (3 cross-references)
- Pre-existing failures (14, verified not caused by this track)
- Plan deviations (6 items, with rationale)
- Post-ship risk register
- Commit inventory with diff stat
- 7 recommendations for the Tier 1 reviewer
- Handoff checklist
Working tree was clean before adding the report (no other changes to commit).
Adds 3 entries to the See Also section:
1. Gemini / Gemini CLI thinking-format compatibility (deferred from
ai_loop_regressions_20260614) - investigate empirically
2. <think> (half-width) marker support in thinking_parser (deferred)
3. Public API Result Migration (planned, separate track public_api_migration_20260606)
Each entry links to the corresponding spec section for traceability.
ed