chore(conductor): Add new track 'RAG Support'

conductor/tracks/rag_support_20260308/spec.md

@@ -0,0 +1,38 @@
+# Specification: RAG Support
+
+## Overview
+This track introduces Retrieval-Augmented Generation (RAG) capabilities to Manual Slop. It allows agents to search and retrieve relevant information from large local codebases, project documentation, or external knowledge bases, overcoming context window limitations and reducing hallucination.
+
+## Functional Requirements
+- **Multi-Source Retrieval:**
+    - **Local Vector Store:** Integrate a local vector database (e.g., Chroma or Qdrant) for indexing and searching local project files.
+    - **Native Retrieval:** Support vendor-specific retrieval features (e.g., Gemini's file search/caching mechanisms).
+    - **External RAG APIs:** Provide a generic interface to connect to external RAG services or specialized MCP servers.
+- **Configurable Indexing:**
+    - Support both **Manual Indexing** (triggering index builds for specific folders/files) and **Auto-Indexing** (indexing files added to the Context Hub).
+    - Users can configure indexing preferences (e.g., which extensions to include, chunking strategy) in `config.toml` or `manual_slop.toml`.
+- **Embedding Support:**
+    - Support for **Gemini** embeddings and **Local** embedding models (e.g., via HuggingFace/Sentence-Transformers).
+- **Retrieval UI & Visualization:**
+    - **Retrieved Blocks:** Display retrieved context fragments directly in the **Discussion History** before the agent's response.
+    - **Source Links:** Provide clickable links/buttons to jump to the original source file and line for each retrieved chunk.
+    - **Retrieval Settings:** A dedicated panel in **AI Settings** to adjust retrieval parameters (top-k, similarity threshold, active RAG source).
+- **Agent Integration:**
+    - Update `ai_client.py` to perform a retrieval step before sending the final prompt to the model.
+    - Ensure retrieved context is properly formatted and injected into the agent's context window.
+
+## Non-Functional Requirements
+- **Performance:** Indexing should be performed in a background thread to avoid GUI freezing. Retrieval must be fast enough to not noticeably delay agent response times.
+- **Scalability:** The RAG system should handle codebases with thousands of files efficiently.
+- **Privacy:** Ensure that local indexing stays local and sensitive data is not inadvertently sent to external embedding providers without user consent.
+
+## Acceptance Criteria
+- [ ] Users can index their local project using Gemini or local embeddings.
+- [ ] Agents can successfully retrieve and use information from indexed files that were not part of the active context window.
+- [ ] Retrieved context is displayed in the GUI with links back to the source code.
+- [ ] Users can switch between local, native, and external RAG sources in the settings.
+- [ ] Auto-indexing works when files are added or modified in the Context Hub.
+
+## Out of Scope
+- Building a complex web crawler for RAG (focusing on local files and specific documentation URLs).
+- Support for advanced semantic search beyond standard vector similarity (e.g., knowledge graphs) in this initial track.