From bed332fbbb47ddc5ac182604632e2517457a0cd5 Mon Sep 17 00:00:00 2001 From: Ed_ Date: Sat, 27 Jun 2026 23:38:44 -0400 Subject: [PATCH] docs(report): ADDENDUM 2 - definitive root cause (stale sim project files) After Tier 2's fixes (ab16f2f2 + f3d823b7), 28/29 RAG tests pass but test_rag_phase4_final_verify still fails. Traced the remaining failure: the subprocess's active_project_path points to tests/artifacts/temp_livecontextsim.toml (created by simulation/sim_base.py:84, never cleaned up), so active_project_root = tests/artifacts. The RAG engine uses tests/artifacts as base_dir, so index_file looks for final_test_1.txt in tests/artifacts/ (not found) and silently no-ops. Collection stays empty -> 0 chunks -> no RAG context block. Verified via /api/project endpoint (project.name='temp_livecontextsim', not 'TestProject') and in-process RAGEngine test (engine works perfectly with correct base_dir). The ui_files_base_dir temp-path issue (Tier 2's fix) is a separate, real polluter but NOT the current failure's cause. Fix: clean up stale temp_*.toml files in tests/artifacts/, add teardown to simulation/sim_base.py, and make index_file log when it no-ops on missing files (the silent return is why this took 3 sessions to find). --- ...GATION_rag_phase4_final_verify_20260627.md | 59 +++++++++++++++++++ 1 file changed, 59 insertions(+) diff --git a/docs/reports/INVESTIGATION_rag_phase4_final_verify_20260627.md b/docs/reports/INVESTIGATION_rag_phase4_final_verify_20260627.md index db573e08..edd23a91 100644 --- a/docs/reports/INVESTIGATION_rag_phase4_final_verify_20260627.md +++ b/docs/reports/INVESTIGATION_rag_phase4_final_verify_20260627.md @@ -118,6 +118,65 @@ Per `conductor/workflow.md` "Isolated-Pass Verification Fallacy": the fix must p --- +## ADDENDUM 2: Definitive root cause after Tier 2's fixes (2026-06-27 22:50) + +Tier 2 applied 2 commits (`ab16f2f2` stop polluters + `f3d823b7` dim check NameError). 28/29 RAG tests now pass. But `test_rag_phase4_final_verify` STILL fails. Tier 2's `SESSION_REPORT_TIER1_FOLLOWUP.md` reports a "diag files never created" paradox and guesses the rebuild isn't called. + +**I reproduced the failure post-fixes and traced the actual root cause. It is NOT the `ui_files_base_dir` temp path (Tier 2's reset fix works — force-setting `ui_files_base_dir='.'` confirmed the reset takes effect). The real issue is `active_project_path` points to a STALE simulation project file.** + +### The evidence (diagnostic via Hook API + `/api/project` endpoint) + +1. Force-set `ui_files_base_dir='.'` via `_set_attr` custom_callback → confirmed reset works, `files_base_dir='.'` +2. RAG sync reaches `ready`; `btn_rebuild_rag_index` click produces `indexing...` → `ready` transition (rebuild DID run) +3. BUT `_slop_cache/` is NEVER created in the workspace (chroma collection never initialized there) +4. `/api/project` endpoint returns `project.name = 'temp_livecontextsim'` — NOT `TestProject` (the workspace's manual_slop.toml has `name = 'TestProject'`) +5. In-process `RAGEngine` test (`diag3.py`, deleted) proved the engine works perfectly: collection created, `index_file` indexes, `search` returns chunks. The RAG engine code is correct. + +### The root cause: `active_project_path` → stale sim project → wrong `base_dir` + +`simulation/sim_base.py:84` creates project files at `tests/artifacts/temp_.toml`: +```python +self.project_path = os.path.abspath(f"tests/artifacts/temp_{project_name.lower()}.toml") +``` +For `test_extended_sims.py::test_context_sim_live`, this is `tests/artifacts/temp_livecontextsim.toml`. The simulation calls `setup_new_project` which switches the subprocess's `active_project_path` to this file. **These temp project files are NEVER cleaned up** (confirmed: `temp_livecontextsim.toml` exists on disk from 6/25). + +When `test_rag_phase4_final_verify` runs (even in isolation): +- The subprocess's `active_project_path` = `tests/artifacts/temp_livecontextsim.toml` (somehow loaded — either from a stale config or the project-loading fallback) +- `active_project_root` = `Path("tests/artifacts/temp_livecontextsim.toml").parent` = `tests/artifacts` +- `_do_rag_sync` builds `RAGEngine(config, base_dir="tests/artifacts")` +- `index_file('final_test_1.txt')` does `os.path.join("tests/artifacts", "final_test_1.txt")` = `tests/artifacts/final_test_1.txt` → file NOT FOUND (the test created it in the workspace, not `tests/artifacts/`) +- `index_file` silently returns → collection stays empty → search returns 0 chunks → no `## Retrieved Context` block + +### Why `active_project_path` is stale even in isolation + +The subprocess loads `config.toml` (passed via `--config=/config.toml`). The conftest-generated config has `projects.active = "/manual_slop.toml"`. But the diagnostic shows `project.name = 'temp_livecontextsim'`. This means `_load_active_project` is NOT loading the workspace manual_slop.toml — it's falling back to a stale project file. The most likely path: `_load_active_project` fails to load the workspace manual_slop.toml (maybe the file doesn't exist at the time init runs, or `load_project` fails), falls through to `project_paths` iteration, and finds a stale `temp_*.toml` file. OR a prior session's `btn_project_save` wrote `temp_livecontextsim.toml` into the config's `projects.paths` and the config file persisted. + +**The `temp_*.toml` files in `tests/artifacts/` are the pollution source.** They are created by `simulation/sim_base.py` and never cleaned up. Any subprocess that loads one as the active project gets `active_project_root = tests/artifacts`, which breaks the RAG engine's file resolution. + +### Why the "diag files never created" paradox (Tier 2's report) + +Tier 2 added diag writes to `_rebuild_rag_index`, `index_file`, `_handle_generate_send`, `_handle_request_event` — and none of the log files appeared. The paradox is explained: **the subprocess IS running the new code** (verified: cleared all `__pycache__`, re-ran, same behavior). The diag writes likely failed silently because: +- The `_sandbox_audit_hook` in the TEST process blocks writes outside `./tests/`, but the SUBPROCESS doesn't have the hook. So that's not it. +- The diag writes were to `tests/artifacts/tier2_state/...` but the directory didn't exist (Tier 2 may have forgotten `mkdir -p`). The `try/except: pass` around the diag write swallowed the `FileNotFoundError`. +- OR the diag was added to `index_file` but `index_file` returned early (file not found at the stale `base_dir`) BEFORE reaching the diag line. + +The most likely: the diag was placed AFTER the `os.path.exists(full_path)` check in `index_file`, which returns early. The diag never ran because `index_file` no-op'd on the missing file. + +### The fix (for Tier 2) + +**Primary fix — clean up stale `temp_*.toml` files and prevent future pollution:** +1. `simulation/sim_base.py:setup` — add cleanup of `self.project_path` in a `finally` or teardown. The sim creates `tests/artifacts/temp_.toml` but never removes it. Add `os.remove(self.project_path)` + the `_history.toml` sibling in cleanup. +2. `tests/conftest.py` `_reset_clean_baseline` — after `reset_session()`, also re-load the workspace project: click `btn_project_switch` with the workspace's `manual_slop.toml` path, OR call `reset_session()` then force `active_project_path` back to the workspace. The current `reset_session()` does NOT reset `active_project_path` (by design, per the comment about `_flush_to_project`), but for `@clean_baseline` tests, the project SHOULD be reset to the workspace. +3. `src/app_controller.py` `_handle_reset_session` — consider resetting `active_project_path` to the config's `projects.active` value (re-reading the config). This is risky (the comment warns about infinite re-switch loops), but the current behavior leaves stale project paths that break RAG. + +**Defensive fix — make `index_file` log when it no-ops on missing files:** +4. `src/rag_engine.py` `index_file` — the `if not os.path.exists(full_path): return` is a silent no-op. Add a `sys.stderr.write` or `logging.debug` when this fires, so the "empty collection" failure mode is visible in the subprocess log instead of silent. This would have made the diagnosis trivial. + +**Stale-file cleanup (immediate):** +5. `rm tests/artifacts/temp_*.toml` and `rm tests/artifacts/temp_*_history.toml` — remove the 7 stale sim project files from `tests/artifacts/`. These are the pollution source. + +--- + ## ADDENDUM: Insane path-resolution audit (added per user directive 2026-06-27) The user asked to verify there are no more "insane path resolutions" pointing to temp/AppData. Findings: