# RAG Phase 4 Sync Fix — Specification (2026-06-10)

## Overview

This track fixes a pre-existing RAG test failure that halted the `tier-3-live_gui` batch during the `mma_tier_usage_reset_fix_20260610` verification run on 2026-06-10.

**The original bug (FIXED):** `tests/test_rag_phase4_final_verify.py::test_phase4_final_verify` failed with "RAG sync failed. Status: idle" because `_handle_reset_session` set `self.rag_config = None` and the `rag_*` setters check `if self.rag_config:` before doing anything — so the 4 setters fired by the test were all no-ops.

**Fix:** reset `rag_config` to a fresh `RAGConfig()` default (not None) in `_handle_reset_session`, so the setters can mutate it and trigger the sync.

**Status (post-fix):** RAG sync now reaches `'ready'`; the test fails on a SEPARATE downstream assertion (retrieval order — see "Residual issue" below).

## Reproduction (already verified)

```powershell
cd C:\projects\manual_slop; uv run pytest tests/test_rag_phase4_final_verify.py::test_phase4_final_verify -v --timeout=120
```

**Result:** 1 failed in 57.39s — `AssertionError: RAG sync failed. Status: idle`

## Suspected root cause

Looking at `src/app_controller.py:1463-1500`:

```python
def _sync_rag_engine(self) -> None:
    with self._rag_sync_lock:
        self._rag_sync_token += 1
        self._rag_sync_dirty = True
        token = self._rag_sync_token
    self.submit_io(lambda: self._do_rag_sync(token))

def _do_rag_sync(self, token: int) -> None:
    while True:
        with self._rag_sync_lock:
            if token != self._rag_sync_token:
                return                              # ← BUG: returns silently
            self._rag_sync_dirty = False
        self._set_rag_status("initializing...")    # ← only sets after the check
        ...
```

The coalescing logic is the prime suspect: if 5 setters are called in quick succession (`rag_collection_name`, `files`, `rag_enabled`, `rag_source`, `rag_emb_provider`), each increments the token and submits a worker. The 5 workers all run concurrently. The first worker checks `if token != self._rag_sync_token` — the token from the first call is now stale (token 1 vs current 5), so it returns without setting status. The second worker (token 2) also returns. The third worker (token 3) also returns. Only the LAST worker (token 5) actually proceeds and sets status.

But the io_pool has limited concurrency (4 workers in startup_speedup_20260606, plus more in `_io_pool` for general use). With 5 setters fired in quick succession from the API, 5 workers are submitted. They all race. The LAST one to acquire `_rag_sync_lock` wins.

This SHOULD work — only the worker with the latest token should set the status. But there's a subtle race: if worker for token 5 acquires the lock first, sees its own token, and proceeds. But what if all 5 workers start before any of them acquires the lock? Then the order of acquisition is non-deterministic.

Looking more carefully: the first worker (token 1) runs, acquires lock, sees token=1 but current=5, returns. Now `self._rag_sync_dirty` is whatever it was BEFORE the first worker (let's say False, because no one has set it True yet — wait, but token 1's setter set `self._rag_sync_dirty = True` BEFORE submitting).

Actually, let me re-read:
```python
def _sync_rag_engine(self) -> None:
    with self._rag_sync_lock:
        self._rag_sync_token += 1
        self._rag_sync_dirty = True
        token = self._rag_sync_token
    self.submit_io(lambda: self._do_rag_sync(token))
```

So each setter:
1. Acquires lock
2. Increments token
3. Sets dirty=True
4. Releases lock
5. Captures `token` (the new value)
6. Submits worker with the captured `token`

So worker 1 captures token=1, worker 5 captures token=5. All 5 workers are submitted.

In `_do_rag_sync`:
```python
while True:
    with self._rag_sync_lock:
        if token != self._rag_sync_token:
            return                              # stale, return
        self._rag_sync_dirty = False
    self._set_rag_status("initializing...")
    # ... do work ...
    with self._rag_sync_lock:
        if not self._rag_sync_dirty:
            return                              # no more setters, done
        token = self._rag_sync_token
        self._rag_sync_dirty = False
```

So worker 1 acquires lock, sees token (1) != self._rag_sync_token (5), returns immediately. Worker 2 same. Worker 3 same. Worker 4 same. Worker 5 acquires lock, sees token (5) == self._rag_sync_token (5), proceeds. Sets status to "initializing...". Does work. Then checks dirty; if no more setters, returns. Sets status to "ready".

This SHOULD work. So why doesn't it?

Possibility 1: The io_pool doesn't process the 5th worker. Maybe the io_pool is full with other work (the test sets a lot of other things, all going through submit_io).

Possibility 2: The worker for token 5 crashes before setting status. The except branch sets status to "error: ...", not "ready". But the test shows "idle", not "error: ...".

Possibility 3: The status is reset by something else. Looking at `_handle_reset_session`:
```python
self.rag_status = 'idle'
```
But the test doesn't call reset.

Possibility 4: The test is checking the wrong state. The Hook API's `get_value` might be returning a cached value.

Let me look at how `get_value` works in the API hooks.

## Diagnostic plan

1. Add a print or log line in `_do_rag_sync` to see if it's being called and with what token
2. Add a print after `_set_rag_status` to see what status is being set
3. Run the test and observe
4. Once we know the actual failure mode, fix it

## Goals

1. The RAG phase 4 test passes in isolation
2. The RAG phase 4 test passes in the full tier-3-live_gui batch (or at least doesn't halt it)
3. No regression in the 4 sim tests in tests/test_extended_sims.py
4. No regression in other RAG tests in tests/test_rag_*.py

## Non-Goals

- Refactoring `_do_rag_sync` (just fix the bug)
- Changing the RAG test design
- Adding new RAG features
- Updating documentation
- Filing follow-up tracks

## Functional Requirements

### FR1. RAG sync reaches 'ready' after configuration

**Where:** `src/app_controller.py` (or `src/rag_engine.py` if the issue is in RAGEngine init)

**What:** After the test sets `rag_enabled=True`, `rag_source='chroma'`, `rag_emb_provider='local'`, the `_do_rag_sync` worker must complete and set `rag_status='ready'` (or 'error: ...' with a clear message if it can't).

**Why:** The RAG test polls for 'ready' and fails if it doesn't see it within 50s.

**Acceptance:**
- `test_rag_phase4_final_verify.py::test_phase4_final_verify` passes
- 4 sim tests in `test_extended_sims.py` still pass

## Non-Functional Requirements

- NFR1: 1-2 line fix, surgical
- NFR2: No new dependencies
- NFR3: 1 atomic commit

## Architecture Reference

- `src/app_controller.py:1463-1500`: `_sync_rag_engine` + `_do_rag_sync` (the coalescing logic)
- `src/app_controller.py:1848-1852`: rag_config initialization in project load
- `src/rag_engine.py:22-53`: lazy imports (`_get_sentence_transformers`, etc.)
- `src/rag_engine.py:88-108`: RAGEngine `__init__` + `_init_embedding_provider`
- `tests/test_rag_phase4_final_verify.py`: the failing test

## Out of Scope

- Refactoring `_do_rag_sync` to a state machine
- Adding observability/metrics to the RAG sync
- Speeding up RAG startup
- Adding new RAG embedding providers
