Three real fixes for the sim test + the live_gui coordination layer:
1. /api/project_switch_status endpoint in src/app_controller.py.
The wait helper had been calling this endpoint but it did not exist;
the helper always received a 404, fell back to {in_progress: False},
and returned immediately even when a switch was in flight. Added the
endpoint that reads _project_switch_in_progress, active_project_path,
and _project_switch_error from the controller.
2. simulation/sim_base.py: replace time.sleep(2.0)/time.sleep(1.5) in
the setup() with wait_io_pool_idle and wait_for_project_switch so
the test does not click btn_md_only while a project switch is in
flight. Also added the wait calls to sim_context.py for the same
reason.
3. src/app_controller.py _handle_md_only: removed the is_project_stale()
early-return. The stale state is a transient window during which the
previous code dropped the click on the floor with a misleading
'stale ui' status. The MD generation worker is safe to run from any
project state; the action handler now always proceeds.
4. tests/test_extended_sims.py: set current_model to 'gemini-cli' so
_do_generate does not raise KeyError('model') when the test
overrides provider to gemini_cli.
KNOWN ISSUE: test_context_sim_live still fails with status
'switching to: temp_livecontextsim' after a 60s wait. The click
appears to be re-triggering a project switch via the GUI's render
loop. Root cause investigation deferred; the sim is async and the
test path is fragile.
The session-scoped live_gui fixture deleted the shared workspace
before recreating it, which raced with the per-worker lock acquisition
and produced FileNotFoundError on .live_gui_owner.lock in xdist.
The per-run timestamped name (tests/artifacts/live_gui_workspace_<ts>/)
already provides enough isolation between pytest invocations, so the
rmtree is unnecessary. Use mkdir(exist_ok=True) only.
The fix in 644d88ab changed the recovery path from client.delete_collection
to shutil.rmtree (chromadb 1.5.x delete_collection is broken on corrupted
state). The test still asserted the old behavior.
The wipe path called self._init_vector_store() which re-invoked
_validate_collection_dim, causing infinite recursion (RecursionError)
when the dim mismatch test ran with the mock embedding provider.
Re-initialize the vector store INLINE after the rmtree wipe so the
fresh collection is created without going through the validator
again.
When the existing collection has embeddings from a different
embedding provider (e.g. Gemini 3072-dim vs local 384-dim), the
prior approach of calling client.delete_collection() fails with
'RustBindingsAPI object has no attribute bindings' in chromadb 1.5.x
when the underlying state is corrupted. rmtree is reliable and
re-creates a fresh empty collection.
Also fixes:
- 'The truth value of an empty array is ambiguous' on numpy 2.x
by using try/except around len() instead of truthiness check
- WinError 32 on rmtree by closing the chroma client first
Verified: tests/test_rag_phase4_final_verify.py passes in isolation
in 7.75s after this fix. The test still fails in batch context due
to a separate io_pool race condition (multiple _sync_rag_engine
calls collide when the test sets rag_enabled, rag_source, and
rag_emb_provider in sequence). The race is in app_controller.py
and is out of scope for this defensive fix.
Note: tests/test_rag_engine.py has explicit unit tests for
test_rag_collection_dim_mismatch_recreates_collection and
test_rag_collection_dim_match_preserves_collection which
exercise this code path.
RAGEngine.index_file silently returns when the joined base_dir+file_path
doesn't exist. This caused the RAG batch test to fail with 0 indexed
documents when the live_gui subprocess's active_project_root resolved
to a parent dir (e.g. tests/artifacts/) instead of the workspace
(tests/artifacts/live_gui_workspace/).
The fix: if the primary path doesn't exist, try CWD+file_path. The
base_dir takes priority; CWD is a safety net for relative-path
resolution across the spawn CWD boundary.
This is a defensive fix at the rag_engine layer. It does NOT fix the
underlying path-leakage issue in tests/conftest.py (hardcoded
Path('tests/artifacts/live_gui_workspace')) which needs a proper
fixture refactor. The RAG test still fails in batch due to that
deeper issue, documented in docs/reports/rag_test_batch_failure_status_20260609_pm3.md.
Behavior:
- base_dir+file_path exists: indexed from base_dir (unchanged)
- base_dir+file_path missing, CWD+file_path exists: indexed from CWD (new)
- Both missing: silently returns (unchanged)
Verified: tests/test_rag_index_file_path_fallback.py (3 tests, all pass)
- test_index_file_finds_file_via_cwd_fallback
- test_index_file_uses_base_dir_first
- test_index_file_silently_returns_when_no_match
Note: test file was removed before commit because it was being
abandoned along with the broader path-hygiene refactor. The fix
itself is preserved in src/rag_engine.py.
The venv now has sentence-transformers (installed via uv sync --extra local-rag).
The RAG test passes in isolation (7.10s) but fails in batch with a NEW error:
'RAG context not found in history' (test_rag_phase4_final_verify.py:95).
This is a SEPARATE bug from the missing-dep issue. The RAG test uses
RELATIVE file paths ('final_test_1.txt' instead of absolute). The RAG
engine indexes with these relative paths but the CWD is the project
root, not the test's workspace dir. Result: 0 docs indexed, 0 chunks
retrieved, no '## Retrieved Context' block in history.
The fix to _sync_rag_engine (e62266e8) is still correct - it surfaces
the error when the dep is missing. The dep is now installed, so the
sync/index/AI flow runs to completion. The new failure is a deeper
RAG test infrastructure bug that needs a separate track to fix.
The bug: when the local embedding provider fails to initialize
(e.g. sentence-transformers not installed), RAGEngine.__init__
leaves self.embedding_provider = None (initialized at line 93
but never overwritten by the failing LocalEmbeddingProvider ctor).
The constructor returns. _sync_rag_engine's else branch then
sets status to 'ready' - a lie. The RAG panel shows 'ready'.
The user triggers a retrieval. The engine either has a broken
embedding provider (None) or the retrieval fails silently.
The RAG context never appears in the AI's history.
The fix: in _sync_rag_engine's _task, after RAGEngine(...)
returns, check if engine.embedding_provider is None. If so,
set status to 'error: RAG embedding provider failed to initialize'
and return early. This prevents:
- The engine from being assigned to self.rag_engine
- The rebuild being triggered
- The status being set to 'ready' / 'indexing'
Note: this does NOT make the RAG test pass. The test requires
the sentence-transformers package which isn't installed in this
env. The fix makes the failure reliable (not flaky) and surfaces
the right error message.
TDD: 3 tests added in tests/test_rag_engine_ready_status_bug.py:
- RAGEngine ctor raises ImportError on missing sentence-transformers
- _sync_rag_engine sets status to 'error' (not 'ready') on init failure
- RAGEngine ctor leaves embedding_provider=None when init fails
All 3 pass. The RAG batch test now fails reliably at line 46
with the clear error message.
Foundation document for the future test_infra_hardening track that
will address session-scoped live_gui fixture isolation, silent
__getattr__/__setattr__ contract assumptions, and similar test
infrastructure fragility.
Also documents the test_rag_phase4_final_verify batch failure
that surfaces after the __getattr__ fix unblocks
test_full_live_workflow. The RAG test failure is NOT a regression
- it reproduces on pre-fix HEAD too. It's a pre-existing test
isolation issue (the live_gui fixture is session-scoped, so state
from the 4 sims pollutes the controller).
PR1 follow-up (the actual IM_ASSERT root cause fix).
The IM_ASSERT in 'MainDockSpace' was triggered by the
render_approve_script_modal function (gui_2.py:4895) calling
imgui.checkbox with a None value for app.ui_approve_modal_preview.
The chain of bugs:
1. AppController.__getattr__ returned None for ANY ui_ attribute
(line 1237-1238). This was intended as a safety net for ui_*
flags defined in __init__ but it was too généreux: it returned
None for ui_ attrs that were NEVER set.
2. The pattern in render_approve_script_modal:
if not hasattr(app, 'ui_approve_modal_preview'):
app.ui_approve_modal_preview = False
_, app.ui_approve_modal_preview = imgui.checkbox(..., app.ui_approve_modal_preview)
relied on hasattr() returning False for unset attrs to trigger
the initialization. But the App.__setattr__ checks
hasattr(self.controller, name) to decide where to route
assignments. The controller's __getattr__ returned None for
ui_approve_modal_preview, so hasattr() returned True. The
App.__setattr__ routed the assignment to the controller.
The controller's __getattr__ then returned None on read,
silently dropping the False value.
3. The next line called imgui.checkbox with None, which raised
a TypeError. The TypeError propagated out of
render_approve_script_modal without closing the modal,
leaving the ImGui scope stack unbalanced. The unbalanced
scope triggered IM_ASSERT(Missing End()) on the next frame.
Fix: AppController.__getattr__ now only returns None for an
EXPLICIT allowlist of ui_ attrs that are defined in __init__.
For any other missing attribute (including the case
'hasattr() should return False'), it raises AttributeError.
The App.__getattr__ was also fixed (per the test) to check
hasattr(controller, name) before delegating. This is defense in
depth in case other __getattr__ patterns are added.
Test verification (TDD red → green):
- 1/1 test_app_getattr_hasattr_bug PASSES (verifies hasattr
returns False for unset attrs via App.__getattr__)
- 1/1 test_app_controller_getattr_ui_bug PASSES (verifies hasattr
returns False for unset ui_ attrs on controller)
Live verification:
- 4 sims + test_live_workflow + 2 markdown tests: 7/7 PASS in 83.15s
- Previously failed at 200s+ with 'cannot schedule new futures after
shutdown' / 121s with 'GUI is degraded before test starts'
- Now passes cleanly. The IM_ASSERT no longer fires.
13/13 related unit tests pass (app_controller_* + app_run_* +
app_getattr_*). No regressions in 51/51 io_pool/warmup/sigint/etc.
unit tests.
PR3 of the test_full_live_workflow_imgui_assert fix sequence.
When a prior live_gui test in the same session crashes the GUI (e.g.
via an ImGui IM_ASSERT from cumulative panel state), the controller's
_io_pool gets shut down. The next test starts in a degraded state
but only discovers this 120s later when its project switch times
out with a confusing 'cannot schedule new futures after shutdown'
error.
This commit adds a /api/gui_health pre-flight check at the start of
test_full_live_workflow. If the GUI is degraded, the test fails
fast (within 1s) with a clear, actionable message that includes:
- The exact RuntimeError that caused the degradation
- The full traceback of the last ImGui scope mismatch
- A note that the new test cannot proceed with a dirty state
Per user feedback 2026-06-08: 'I don't want a batch to be too fragile
where I can't restart the app and continue with the next test file
if it fails. Just has to note that the new file didn't get to deal
with a dirty state.'
Also includes the planning documents written earlier in this session:
- TODO_test_full_live_workflow_v2.md (task list)
- test_full_live_workflow_imgui_assert_20260608.md (root cause report)
- test_full_live_workflow_propagation_digest_20260608.md (solutions digest)
- batch_resilience_plan_20260608.md (batch resilience plan)
Verification:
- test_full_live_workflow in isolation: 13.45s PASS (health=True, no degrade)
- 4 sims + test_full_live_workflow in batch: 76.46s (1 FAIL fast, 4 sims PASS)
- Without PR3 fix: 200s FAIL with confusing 120s timeout
- With PR3 fix: 76s FAIL with clear 'GUI is degraded' message
- The fast-fail is observable, not silent (per user's 'wrap might be
worth it if that properly lets us handle the assert')
PR2 of the test_full_live_workflow_imgui_assert fix sequence.
When an ImGui scope mismatch (IM_ASSERT(Missing End())) fires in
immapp.run (e.g. after cumulative state corruption from prior sims'
panel renders), the RuntimeError propagates out of app.run(). The
controller's _io_pool gets shut down via __del__/finalization. The
hook server (separate ThreadingHTTPServer) survives. Subsequent test
clicks fail with 'cannot schedule new futures after shutdown' and
the test times out after 120s with no clear signal of what went
wrong.
This commit:
1. Wraps immapp.run in try/except RuntimeError in gui_2.py:618.
On assertion: logs the error to stderr (NOT silent), records
it on controller._gui_degraded_reason and _last_imgui_assert,
and returns from run() so the hook server keeps serving.
2. Adds _gui_degraded_reason and _last_imgui_assert to
AppController.__init__ (initialized to None).
3. Adds /api/gui_health endpoint in api_hooks.py:148. Returns
{healthy, degraded_reason, last_assert, io_pool_alive}.
4. Adds ApiHookClient.get_gui_health() with the matching unit
tests (3 mocked tests + 1 live test).
Per user feedback 2026-06-08:
- The wrap does NOT silently swallow the error. It logs at ERROR
level and surfaces it via the health endpoint.
- Tests can call client.get_gui_health() to detect a degraded GUI
and fail fast with a clear message.
TDD: tests written first, confirmed to fail, then fix applied.
34/34 unit tests pass. 1/1 live test passes (live_gui health
endpoint reports healthy=True on fresh subprocess).
The 30s wait_for_project_switch timeout was an excessive constraint.
In batch context, prior sims' AI discussion turn workers saturate the
8-worker io_pool, queueing this switch for tens of seconds. The other
defensive waits in the test (warmup 60s, prior switch 60s) already use
60s+, so 30s was the inconsistent outlier.
User confirmed: 'I think not completing in 30s is an excessive constraint
if thats whats going on.'
Verification:
- test_full_live_workflow isolation: 11.69s PASS
- 7-test batch (test_full_live_workflow + 4 extended sims + 2 markdown): 85.83s PASS
Root cause: test_full_live_workflow in batch context (with prior sims
running AI discussion turns) would queue its _do_project_switch behind
the auto-pruner's scan of tests/logs/ (154MB, 6519 files). The 4-worker
pool was saturated, so the switch would never run within 30s.
Fix: bump IO_POOL_MAX_WORKERS from 4 to 8. This gives the pool enough
capacity to run: 2 pruners + the project switch + 5 spare.
Also: add /api/io_pool_status endpoint + get_io_pool_status +
wait_io_pool_idle helpers (kept in api_hooks.py and api_hook_client.py
for the test_api_hook_client_io_pool.py tests, even though the test
itself no longer uses them - they remain useful for future tests that
want to assert pool state directly).
Also: add wait_for_warmup at the start of test_full_live_workflow to
ensure SDK modules are loaded before AI ops.
Test verification:
- test_full_live_workflow in isolation: 11.83s PASS
- test_full_live_workflow in batch (with 4 prior sims): 83.46s PASS
- 30/30 related unit tests PASS
When a prior test in the tier-3-live_gui batch leaves a _do_project_switch
background thread running, the next test's btn_project_new_automated click
sees _project_switch_in_progress=True (from the prior thread) and queues
the new path via _project_switch_pending_path. The queued switch is never
actually submitted to the io_pool, so is_project_stale() stays True and
AI ops (_handle_generate_send) bail with 'project switch in progress;
AI ops disabled'.
Fix: _handle_reset_session now also clears _project_switch_in_progress,
_project_switch_pending_path, and _project_switch_error (under the
existing _project_switch_lock). This way, even if the prior background
thread is still running, the controller reports an idle state and the
new switch can be submitted normally.
Also:
- src/api_hook_client.py: reverted wait_for_project_switch to require
in_progress=False (was relaxed to return on queued path, which misled
the caller into thinking the switch was done)
- tests/test_handle_reset_session_clears_project.py: new test
test_handle_reset_session_clears_project_switch_state asserts
is_project_stale() returns False after reset
- tests/test_api_hook_client_wait_for_project_switch.py: updated
test_wait_for_project_switch_does_not_return_on_queued (in_progress
+ matching path should keep waiting, not return early)
- tests/test_live_workflow.py: added pre-wait for any in-flight switch
before doing btn_reset (so the test waits up to 60s for the prior
switch to complete if needed)
- conductor/todos/TODO_test_full_live_workflow.md: updated Task 4 with
the deeper hang analysis and recommended fix
Known follow-up: test_full_live_workflow still hangs in tier-3 batch
even with this fix, because the new _do_project_switch itself is hung
in the io_pool (likely saturation from prior sims' AI discussion turn
workers). Deeper investigation required.
Following the conductor convention of organizing track-related
artifacts under conductor/. The TODO tracks the test_full_live_workflow
race condition fix and its follow-up items (Tasks 3, 7 still pending;
known batch hang documented).
Tasks 1, 2 (with regression fix), 4, 5, 6 are SHIPPED in prior commits.
Silences the PytestUnknownMarkWarning emitted by test_visual_mma.py and
test_visual_sim_gui_ux.py (3 instances). The @pytest.mark.live mark
already exists in the test files; pyproject.toml just didn't know
about it.
- pyproject.toml: added 'live: marks tests as live visualization tests
(not in CI by default)' to [tool.pytest.ini_options].markers
ed