docs(report): ADDENDUM 2 - definitive root cause (stale sim project files)

After Tier 2's fixes (ab16f2f2 + f3d823b7), 28/29 RAG tests pass but
test_rag_phase4_final_verify still fails. Traced the remaining failure:
the subprocess's active_project_path points to
tests/artifacts/temp_livecontextsim.toml (created by
simulation/sim_base.py:84, never cleaned up), so active_project_root =
tests/artifacts. The RAG engine uses tests/artifacts as base_dir, so
index_file looks for final_test_1.txt in tests/artifacts/ (not found)
and silently no-ops. Collection stays empty -> 0 chunks -> no RAG
context block.

Verified via /api/project endpoint (project.name='temp_livecontextsim',
not 'TestProject') and in-process RAGEngine test (engine works perfectly
with correct base_dir). The ui_files_base_dir temp-path issue (Tier 2's
fix) is a separate, real polluter but NOT the current failure's cause.

Fix: clean up stale temp_*.toml files in tests/artifacts/, add teardown
to simulation/sim_base.py, and make index_file log when it no-ops on
missing files (the silent return is why this took 3 sessions to find).

docs/reports/INVESTIGATION_rag_phase4_final_verify_20260627.md

@@ -118,6 +118,65 @@ Per `conductor/workflow.md` "Isolated-Pass Verification Fallacy": the fix must p
 
 ---
 
+## ADDENDUM 2: Definitive root cause after Tier 2's fixes (2026-06-27 22:50)
+
+Tier 2 applied 2 commits (`ab16f2f2` stop polluters + `f3d823b7` dim check NameError). 28/29 RAG tests now pass. But `test_rag_phase4_final_verify` STILL fails. Tier 2's `SESSION_REPORT_TIER1_FOLLOWUP.md` reports a "diag files never created" paradox and guesses the rebuild isn't called.
+
+**I reproduced the failure post-fixes and traced the actual root cause. It is NOT the `ui_files_base_dir` temp path (Tier 2's reset fix works — force-setting `ui_files_base_dir='.'` confirmed the reset takes effect). The real issue is `active_project_path` points to a STALE simulation project file.**
+
+### The evidence (diagnostic via Hook API + `/api/project` endpoint)
+
+1. Force-set `ui_files_base_dir='.'` via `_set_attr` custom_callback → confirmed reset works, `files_base_dir='.'`
+2. RAG sync reaches `ready`; `btn_rebuild_rag_index` click produces `indexing...` → `ready` transition (rebuild DID run)
+3. BUT `_slop_cache/` is NEVER created in the workspace (chroma collection never initialized there)
+4. `/api/project` endpoint returns `project.name = 'temp_livecontextsim'` — NOT `TestProject` (the workspace's manual_slop.toml has `name = 'TestProject'`)
+5. In-process `RAGEngine` test (`diag3.py`, deleted) proved the engine works perfectly: collection created, `index_file` indexes, `search` returns chunks. The RAG engine code is correct.
+
+### The root cause: `active_project_path` → stale sim project → wrong `base_dir`
+
+`simulation/sim_base.py:84` creates project files at `tests/artifacts/temp_<name>.toml`:
+```python
+self.project_path = os.path.abspath(f"tests/artifacts/temp_{project_name.lower()}.toml")
+```
+For `test_extended_sims.py::test_context_sim_live`, this is `tests/artifacts/temp_livecontextsim.toml`. The simulation calls `setup_new_project` which switches the subprocess's `active_project_path` to this file. **These temp project files are NEVER cleaned up** (confirmed: `temp_livecontextsim.toml` exists on disk from 6/25).
+
+When `test_rag_phase4_final_verify` runs (even in isolation):
+- The subprocess's `active_project_path` = `tests/artifacts/temp_livecontextsim.toml` (somehow loaded — either from a stale config or the project-loading fallback)
+- `active_project_root` = `Path("tests/artifacts/temp_livecontextsim.toml").parent` = `tests/artifacts`
+- `_do_rag_sync` builds `RAGEngine(config, base_dir="tests/artifacts")`
+- `index_file('final_test_1.txt')` does `os.path.join("tests/artifacts", "final_test_1.txt")` = `tests/artifacts/final_test_1.txt` → file NOT FOUND (the test created it in the workspace, not `tests/artifacts/`)
+- `index_file` silently returns → collection stays empty → search returns 0 chunks → no `## Retrieved Context` block
+
+### Why `active_project_path` is stale even in isolation
+
+The subprocess loads `config.toml` (passed via `--config=<workspace>/config.toml`). The conftest-generated config has `projects.active = "<workspace>/manual_slop.toml"`. But the diagnostic shows `project.name = 'temp_livecontextsim'`. This means `_load_active_project` is NOT loading the workspace manual_slop.toml — it's falling back to a stale project file. The most likely path: `_load_active_project` fails to load the workspace manual_slop.toml (maybe the file doesn't exist at the time init runs, or `load_project` fails), falls through to `project_paths` iteration, and finds a stale `temp_*.toml` file. OR a prior session's `btn_project_save` wrote `temp_livecontextsim.toml` into the config's `projects.paths` and the config file persisted.
+
+**The `temp_*.toml` files in `tests/artifacts/` are the pollution source.** They are created by `simulation/sim_base.py` and never cleaned up. Any subprocess that loads one as the active project gets `active_project_root = tests/artifacts`, which breaks the RAG engine's file resolution.
+
+### Why the "diag files never created" paradox (Tier 2's report)
+
+Tier 2 added diag writes to `_rebuild_rag_index`, `index_file`, `_handle_generate_send`, `_handle_request_event` — and none of the log files appeared. The paradox is explained: **the subprocess IS running the new code** (verified: cleared all `__pycache__`, re-ran, same behavior). The diag writes likely failed silently because:
+- The `_sandbox_audit_hook` in the TEST process blocks writes outside `./tests/`, but the SUBPROCESS doesn't have the hook. So that's not it.
+- The diag writes were to `tests/artifacts/tier2_state/...` but the directory didn't exist (Tier 2 may have forgotten `mkdir -p`). The `try/except: pass` around the diag write swallowed the `FileNotFoundError`.
+- OR the diag was added to `index_file` but `index_file` returned early (file not found at the stale `base_dir`) BEFORE reaching the diag line.
+
+The most likely: the diag was placed AFTER the `os.path.exists(full_path)` check in `index_file`, which returns early. The diag never ran because `index_file` no-op'd on the missing file.
+
+### The fix (for Tier 2)
+
+**Primary fix — clean up stale `temp_*.toml` files and prevent future pollution:**
+1. `simulation/sim_base.py:setup` — add cleanup of `self.project_path` in a `finally` or teardown. The sim creates `tests/artifacts/temp_<name>.toml` but never removes it. Add `os.remove(self.project_path)` + the `_history.toml` sibling in cleanup.
+2. `tests/conftest.py` `_reset_clean_baseline` — after `reset_session()`, also re-load the workspace project: click `btn_project_switch` with the workspace's `manual_slop.toml` path, OR call `reset_session()` then force `active_project_path` back to the workspace. The current `reset_session()` does NOT reset `active_project_path` (by design, per the comment about `_flush_to_project`), but for `@clean_baseline` tests, the project SHOULD be reset to the workspace.
+3. `src/app_controller.py` `_handle_reset_session` — consider resetting `active_project_path` to the config's `projects.active` value (re-reading the config). This is risky (the comment warns about infinite re-switch loops), but the current behavior leaves stale project paths that break RAG.
+
+**Defensive fix — make `index_file` log when it no-ops on missing files:**
+4. `src/rag_engine.py` `index_file` — the `if not os.path.exists(full_path): return` is a silent no-op. Add a `sys.stderr.write` or `logging.debug` when this fires, so the "empty collection" failure mode is visible in the subprocess log instead of silent. This would have made the diagnosis trivial.
+
+**Stale-file cleanup (immediate):**
+5. `rm tests/artifacts/temp_*.toml` and `rm tests/artifacts/temp_*_history.toml` — remove the 7 stale sim project files from `tests/artifacts/`. These are the pollution source.
+
+---
+
 ## ADDENDUM: Insane path-resolution audit (added per user directive 2026-06-27)
 
 The user asked to verify there are no more "insane path resolutions" pointing to temp/AppData. Findings: