conductor(archive): move 39 completed tracks (2026-05 to 2026-06) to archive/

2026-06-03 00:09:52 -04:00
parent 0ffeccc7d3
commit 594f14f943
160 changed files with 94 additions and 21 deletions
@@ -0,0 +1,23 @@
+# AI Loop: Optimization & Consolidation Targets
+
+Based on the technical trace and sequence mapping of the AI interaction loop, the following areas are identified as primary targets for "Heavy Curation".
+
+### 1. Unified Provider Loop (`ai_client.py`)
+- **Observation:** `_send_anthropic`, `_send_gemini`, and `_send_gemini_cli` all implement their own `for r_idx in range(MAX_TOOL_ROUNDS + 2)` loops.
+- **Problem:** Significant boilerplate duplication for tool execution, error handling, and file re-reading.
+- **Curation Goal:** Refactor the multi-turn recursion into a single `_base_send_loop` method that takes a provider-specific `generate_turn` callback.
+
+### 2. Threading Model Management (`app_controller.py`)
+- **Observation:** `_process_event_queue` spawns a new `threading.Thread` for every `user_request`.
+- **Problem:** Potential for thread explosion if multiple asynchronous requests are triggered rapidly (though rare in typical usage).
+- **Curation Goal:** Consolidate into a single dedicated "AI Worker" thread with a task queue, or use a small `ThreadPoolExecutor` to manage background lifetimes.
+
+### 3. Redundant Context Markers
+- **Observation:** `_FILE_REFRESH_MARKER` and `_get_context_marker()` are used in multiple places to inject diffs.
+- **Problem:** String duplication and fragmented logic for deciding when to "refresh" the AI's file context.
+- **Curation Goal:** Centralize the context-refresh injection logic within the `aggregate` module or a dedicated `ContextRefresher` class.
+
+### 4. Blocking Call Audit
+- **Observation:** `asyncio.run_coroutine_threadsafe(...).result()` is used to call async tool logic from the sync worker thread.
+- **Problem:** This bridge is technically correct but adds complexity.
+- **Curation Goal:** If possible, move more of the AI loop logic into a proper `async` context to avoid the `.result()` blocking pattern.
@@ -0,0 +1,86 @@
+# AI Interaction Pipeline: Intensive Technical Trace
+
+This document provides a low-level technical trace of the AI interaction loop, following a pipeline-oriented architectural model. It identifies thread context switches, data transformation overhead, and synchronization bottlenecks.
+
+## 1. Sequence Diagram: Asynchronous Interaction Pipeline
+
+```mermaid
+sequenceDiagram
+    autonumber
+    participant UI as gui_2.py (Main/Render Thread)
+    participant EV as app_controller.py (Event Dispatcher)
+    participant WK as ai_client.py (Worker Thread Pool)
+    participant AI as ai_client.py (Provider Pipeline)
+    participant MCP as mcp_client.py (FileSystem Pipeline)
+    participant SR as shell_runner.py (Subprocess Pipeline)
+
+    Note over UI, WK: [Phase A: Request Initiation]
+    UI->>EV: SyncEventQueue.put("user_request", dict)
+    Note right of UI: Data: Raw Prompt + Context Pointers
+    EV->>EV: polling loop (event_queue.get())
+    EV->>WK: threading.Thread(target=_handle_request_event).start()
+    Note right of EV: Context Switch: Event Thread -> AI Worker Thread
+
+    Note over WK, AI: [Phase B: Context Synthesis & Generation]
+    WK->>AI: ai_client.send(md_content, history)
+    AI->>AI: _build_chunked_context_blocks()
+    Note right of AI: Perf: O(N) string concatenation + regex scans
+    AI->>Vendor: Provider API Request (HTTPS/JSON)
+    Note right of AI: Bottleneck: Network Latency (1-30s)
+    Vendor-->>AI: ToolCall(s) or StopReason
+
+    Note over AI, SR: [Phase C: Multi-Turn Tool Execution Loop]
+    loop MAX_TOOL_ROUNDS (r_idx <= 10)
+        alt Tool Use Detected
+            AI->>WK: _execute_tool_calls_concurrently()
+            
+            alt Read-Only (MCP)
+                WK->>MCP: read_file / list_dir / search
+                MCP-->>WK: stdout_string
+            else Mutating (Shell)
+                WK->>EV: _pending_gui_tasks.append(approval_modal)
+                Note over UI: UI Polling Detects Task
+                UI->>UI: Render ImGui Popup (Wait for HITL)
+                Note over UI: User Approval Interaction
+                UI-->>WK: threading.Condition.notify()
+                Note right of WK: Resume AI Worker Thread
+                WK->>SR: run_powershell(script)
+                SR->>OS: Subprocess Spawn (powershell.exe)
+                OS-->>SR: stdout/stderr (JSON-L Stream)
+                SR-->>WK: COMBINED_OUTPUT_STRING
+            end
+            
+            WK-->>AI: Aggregate Tool Results
+            AI->>AI: _reread_file_items() (Context Refresh)
+            Note right of AI: Perf: IO Bound (File MTime Scans)
+            AI->>Vendor: Follow-up Prompt (with Tool Result)
+        else Terminal Text
+            AI-->>WK: Final AI Response Text
+        end
+    end
+
+    Note over WK, UI: [Phase D: Result Synchronization]
+    WK->>EV: SyncEventQueue.put("response", result)
+    EV->>EV: _pending_gui_tasks.append(response_obj)
+    loop Every Frame (~16.6ms)
+        UI->>EV: _process_pending_gui_tasks()
+        Note right of UI: Data Copy: Controller State -> UI History Buffer
+        UI->>UI: Update Rendering State (Markdown/Syntax Highlight)
+    end
+```
+
+## 2. Technical Performance Audit
+
+### 2.1 Threading & Synchronization
+- **Context Switches:** The pipeline traverses four distinct execution contexts: Main Thread -> Event Thread -> Daemon Worker -> Subprocess.
+- **Lock Contention:** `_pending_gui_tasks_lock` is acquired twice per AI response turn (once by background thread to append, once by UI thread to process).
+- **Blocking Sites:** `ai_client.send` blocks the dedicated `WK` thread. `_confirm_and_run` blocks the `WK` thread using a `Condition` variable waiting on UI input.
+
+### 2.2 Data Transformation Costs
+- **Context Bloat:** `md_content` is a monolithic string. During synthesis, this string is often copied or chunked (`_chunk_text`), increasing memory pressure on the Python heap.
+- **Serialization Overhead:** Every tool call involves: Python dict -> JSON String -> Subprocess Stdin -> (Tools) -> Subprocess Stdout -> JSON String -> Python dict.
+
+### 2.3 Curation Targets (Intensive)
+1. **Reduce Memory Copies:** The monolithic Markdown context should be handled as a stream or a shared buffer to avoid redundant copies between `aggregate` and `ai_client`.
+2. **Deterministic Status Polling:** Replace string-based status polling (`ai_status`) with an enum-based state machine to reduce regex comparisons in the simulator and UI.
+3. **Subprocess Pooling:** `shell_runner` spawns a new process for every script. For high-frequency tool use, a persistent PowerShell session could reduce overhead.
@@ -0,0 +1,5 @@
+# Track ai_interaction_call_graph_20260507 Context
+
+- [Specification](./spec.md)
+- [Implementation Plan](./plan.md)
+- [Metadata](./metadata.json)
@@ -0,0 +1,8 @@
+{
+  "track_id": "ai_interaction_call_graph_20260507",
+  "type": "chore",
+  "status": "new",
+  "created_at": "2026-05-07T16:00:00Z",
+  "updated_at": "2026-05-07T16:00:00Z",
+  "description": "Exhaustive function-to-function call graph tracing the AI loop from request to terminal execution."
+}
@@ -0,0 +1,22 @@
+# Implementation Plan: AI Interaction Call Graph (ai_interaction_call_graph_20260507)
+
+## Phase 1: Trace Mapping
+- [x] Task: Use `py_find_usages` to trace `ai_client.send` callers and callees.
+- [x] Task: Map the asynchronous hand-off from `AppController` to the AI worker threads.
+- [x] Task: Trace the recursion depth of the tool-call loop (`MAX_TOOL_ROUNDS`).
+
+## Phase 2: Documentation & Synthesis
+- [x] Task: Create a high-fidelity Mermaid sequence diagram of the entire loop.
+- [x] Task: Identify specific areas for logic consolidation or performance optimization.
+
+## Phase 3: Automated Path Derivation Tooling
+- [x] Task: Develop `derive_code_path` MCP tool using tree-sitter.
+- [~] Task: Implement cross-file call-chain tracing and data hand-off detection.
+- [ ] Task: Verify tool output against the manual AI Loop trace.
+
+## Phase 4: Comprehensive Pipeline Mapping
+- [x] Task: Map the **Context Aggregation Pipeline** using the new tool.
+- [x] Task: Map the **GUI Event & State Synchronization** pipeline.
+- [x] Task: Map the **Simulation Lifecycle** and turn-loop.
+- [x] Task: Consolidate all intensive traces into a final Phase 5 Architectural Audit.
+- [x] Task: Conductor - User Manual Verification 'Final Audit' (Protocol in workflow.md)
@@ -0,0 +1,22 @@
+# Specification: AI Interaction Call Graph (ai_interaction_call_graph_20260507)
+
+## Overview
+A low-level technical trace of the AI interaction loop. The goal is to map every single function call and data hand-off from the moment a user message is sent to the final terminal execution of a PowerShell script or tool result.
+
+## Scope
+- **Entry Point:** `src/gui_2.py:App._render_discussion_panel` (Send button action).
+- **Subsystems:** `ai_client.py`, `mcp_client.py`, `shell_runner.py`, `app_controller.py`.
+
+## Functional Requirements
+1. **Call Graph Generation:**
+   - Document the sequence of synchronous and asynchronous calls.
+   - Identify thread boundaries (GUI thread vs. Background worker thread).
+2. **Data Transformation Trace:**
+   - Track the transformation of a message: raw text -> GenerateRequest -> AI History -> Provider Prompt -> AI Response -> Tool Call -> PS Script.
+3. **Error & Retry Paths:**
+   - Map how exceptions are caught, classified, and bubbled back to the UI.
+
+## Acceptance Criteria
+- [ ] Detailed call graph in Mermaid format.
+- [ ] List of all internal private methods involved in the loop.
+- [ ] Identification of any blocking calls in the async pipeline.