plans and docs

2026-03-08 03:05:15 -04:00
parent d34c35941f
commit 83911ff1c5
9 changed files with 499 additions and 0 deletions
--- a/TASKS.md
+++ b/TASKS.md
@@ -128,3 +128,17 @@
 - **Status:** Planned
 - **Priority:** Medium
 - **Goal:** Interactive human-in-the-loop track to review and adjust GUI UX, animations, popups, and layout structures.
+
+---
+
+### C/C++ Language Support
+
+#### 21. ts_cpp_tree_sitter_20260308
+- **Status:** Planned
+- **Priority:** High
+- **Goal:** Add tree-sitter C and C++ grammars. Extend ASTParser to support C/C++ skeleton and outline extraction. Add MCP tools ts_c_get_skeleton, ts_cpp_get_skeleton, ts_c_get_code_outline, ts_cpp_get_code_outline.
+
+#### 22. gencpp_python_bindings_20260308
+- **Status:** Planned
+- **Priority:** Medium
+- **Goal:** Bootstrap standalone Python project with CFFI bindings for gencpp C library. Provides foundation for richer C++ AST parsing in future (beyond tree-sitter syntax).
--- a/conductor/tracks/gencpp_python_bindings_20260308/metadata.json
+++ b/conductor/tracks/gencpp_python_bindings_20260308/metadata.json
@@ -0,0 +1,17 @@
+{
+  "track_id": "gencpp_python_bindings_20260308",
+  "title": "Bootstrap gencpp Python Bindings Project",
+  "status": "pending",
+  "created": "2026-03-08",
+  "priority": "medium",
+  "owner": "tier2-tech-lead",
+  "description": "Create standalone Python project with CFFI bindings for gencpp C library to enable richer C++ AST parsing in the future",
+  "dependencies": [],
+  "out_of_scope": [
+    "Full AST traversal",
+    "Expression parsing",
+    "Integration into manual_slop",
+    "macOS/Linux support"
+  ],
+  "notes": "Long-term effort. Track 1 (tree-sitter) provides immediate C/C++ support. This is for future richer functionality."
+}
--- a/conductor/tracks/gencpp_python_bindings_20260308/plan.md
+++ b/conductor/tracks/gencpp_python_bindings_20260308/plan.md
@@ -0,0 +1,95 @@
+# Plan: Bootstrap gencpp Python Bindings Project
+
+## Overview
+Create standalone Python project with CFFI bindings for gencpp C library.
+
+## Phase 1: Project Setup
+Focus: Create repository structure and CFFI configuration
+
+- [ ] Task 1.1: Create new project directory `gencpp-python-bindings/`
+  - WHERE: New directory outside manual_slop
+  - WHAT: Initialize as Python package with pyproject.toml
+  - HOW: Standard Python package structure
+  - SAFETY: New project, no impact on manual_slop
+
+- [ ] Task 1.2: Set up CFFI in pyproject.toml
+  - WHERE: gencpp-python-bindings/pyproject.toml
+  - WHAT: Add cffi dependency and build requirements
+  - HOW: Standard CFFI setup
+  - SAFETY: New file
+
+- [ ] Task 1.3: Obtain gencpp C library
+  - WHERE: gencpp-python-bindings/lib/
+  - WHAT: Download or reference gencpp_c11.lib from gencpp releases
+  - HOW: Clone gencpp or add as git submodule
+  - SAFETY: External dependency
+
+## Phase 2: CFFI Binding Skeleton
+Focus: Set up FFI and basic type mappings
+
+- [ ] Task 2.1: Create CFFI wrapper module
+  - WHERE: gencpp-python-bindings/src/gencpp/_cffi.py
+  - WHAT: Set up FFI with gencpp C header declarations
+  - HOW: Map basic types from gencpp C API
+  - SAFETY: New module
+
+- [ ] Task 2.2: Define Python Declaration dataclasses
+  - WHERE: gencpp-python-bindings/src/gencpp/models.py
+  - WHAT: Create Declaration, FunctionDecl, StructDecl, EnumDecl, etc.
+  - HOW: Dataclasses matching gencpp AST types
+  - SAFETY: New module
+
+- [ ] Task 2.3: Build minimal CFFI bindings for top-level declarations
+  - WHERE: gencpp-python-bindings/src/gencpp/bindings.py
+  - WHAT: Bind gencpp C functions for parsing and AST traversal
+  - HOW: Focus on Code_Function, Code_Struct, Code_Enum, Code_Typedef
+  - SAFETY: New module
+
+## Phase 3: Python API Implementation
+Focus: Create Python-friendly API
+
+- [ ] Task 3.1: Implement parse_c_file()
+  - WHERE: gencpp-python-bindings/src/gencpp/parser.py
+  - WHAT: Parse C file, return list of Declaration objects
+  - HOW: Call gencpp C library, convert to Python dataclasses
+  - SAFETY: New function
+
+- [ ] Task 3.2: Implement parse_cpp_file()
+  - WHERE: gencpp-python-bindings/src/gencpp/parser.py
+  - WHAT: Parse C++ file, return list of Declaration objects (includes classes)
+  - HOW: Similar to C with additional C++ types
+  - SAFETY: New function
+
+- [ ] Task 3.3: Add skeleton generation helpers
+  - WHERE: gencpp-python-bindings/src/gencpp/skeleton.py
+  - WHAT: Convert Declaration list to skeleton string (matching mcp_client format)
+  - HOW: Format function signatures, struct fields, etc.
+  - SAFETY: New module
+
+## Phase 4: Testing & Documentation
+Focus: Verify bindings work and document
+
+- [ ] Task 4.1: Write basic parse tests
+  - WHERE: gencpp-python-bindings/tests/
+  - WHAT: Test parsing sample C and C++ files
+  - HOW: Use pytest with fixture sample files
+  - SAFETY: New test files
+
+- [ ] Task 4.2: Document API and future expansion
+  - WHERE: gencpp-python-bindings/README.md
+  - WHAT: Document current capabilities, what's missing, how to extend
+  - HOW: Markdown documentation
+  - SAFETY: New documentation
+
+- [ ] Task 4.3: Verify Windows build works
+  - WHERE: Local Windows environment
+  - WHAT: Ensure CFFI can load gencpp_c11.lib
+  - HOW: Run tests on Windows
+  - SAFETY: Validation only
+
+## Future Work (Not This Track)
+- Full AST traversal (beyond top-level)
+- Integration into manual_slop mcp_client as `gencpp_*` tools
+- macOS/Linux support
+- Template parameter parsing
+- Operator overloading resolution
--- a/conductor/tracks/gencpp_python_bindings_20260308/spec.md
+++ b/conductor/tracks/gencpp_python_bindings_20260308/spec.md
@@ -0,0 +1,99 @@
+# Track Specification: Bootstrap gencpp Python Bindings Project
+
+## Overview
+
+Create a new standalone Python project to build CFFI bindings for gencpp (C/C++ staged metaprogramming library). This will eventually provide richer C++ AST understanding than tree-sitter (full type information, operators, specifiers) but is a longer-term effort. This track bootstraps the project structure and initial bindings.
+
+## Current State Audit
+
+### gencpp Analysis (from research)
+- **gencpp**: C++ library for staged metaprogramming with built-in C/C++ parser
+- **Parser**: Non-standards-compliant single-pass parser for declarations/definitions
+- **NOT execution-aware**: Does NOT parse expressions or statements (only declarations)
+- **AST Types**: Code_Class, Code_Struct, Code_Function, Code_Enum, Code_Typedef, etc.
+- **Available bindings**: Odin (82KB binding file demonstrates scope)
+- **Repository**: https://github.com/Ed94/gencpp
+
+### What gencpp Can Extract
+- Function declarations and definitions (signatures only)
+- Class/struct/enum/union definitions
+- Type aliases (typedef)
+- Template declarations (partial)
+- Access specifiers
+- Operators (overload resolution)
+- Preprocessor includes/defines
+
+### What gencpp CANNOT Do
+- Parse function bodies/implementations
+- Expression evaluation
+- Template instantiation
+
+## Goals
+
+1. Create new Python project structure for gencpp bindings
+2. Set up CFFI to interface with gencpp C library
+3. Build minimal viable bindings for declaration extraction
+4. Create Python API matching existing mcp_client patterns
+5. Document future expansion path
+
+## Functional Requirements
+
+### Project Setup
+- Create new repository/project: `gencpp-python-bindings` (or `pygencpp`)
+- Use CFFI for C library binding (pure Python, no C extension compilation required for basic use)
+- Structure: Python package with CFFI-based wrapper
+
+### Core API (Phase 1)
+| Function | Description |
+|----------|-------------|
+| `parse_c_file(path) -> List[Declaration]` | Parse C file, return declarations |
+| `parse_cpp_file(path) -> List[Declaration]` | Parse C++ file, return declarations |
+| `Declaration` dataclass | name, type, line_number, node_type |
+
+### Declaration Types to Bind
+- `Code_Function` / `Code_Function_Fwd`
+- `Code_Struct` / `Code_Struct_Fwd`
+- `Code_Enum` / `Code_Enum_Fwd`
+- `Code_Union` / `Code_Union_Fwd`
+- `Code_Typedef`
+- `Code_Class` (C++)
+- `Code_Namespace` (C++)
+
+### Integration Points (Future)
+- Design Python API to match mcp_client patterns for easy integration
+- Allow language parameter: `parse_file(path, language: str)`
+- Return structured data suitable for skeleton generation
+
+## Non-Functional Requirements
+
+- Pure Python with CFFI (no compiled extensions required at install time)
+- Must work on Windows (gencpp provides win64 precompiled library)
+- Follow existing 1-space indentation code style
+- Include type hints for all public APIs
+
+## Architecture Reference
+
+- **gencpp C library**: Precompiled `gencpp_c11.lib` (Windows) or source compilation
+- **CFFI pattern**: Use `cffi.FFI().set_source()` to wrap C library
+- **Odin bindings reference**: https://github.com/Ed94/gencpp-odin (shows AST type scope)
+- **Existing mcp_client pattern**: `src/mcp_client.py` tool dispatch
+
+## Out of Scope (This Track)
+- Full AST traversal (only top-level declarations)
+- Expression parsing
+- Template instantiation analysis
+- Integration into Manual Slop (future track)
+- macOS/Linux support (Windows first)
+
+## Relationship to Tree-Sitter Track
+
+This is a SEPARATE long-term effort. The tree-sitter track provides immediate C/C++ support. The gencpp bindings track is for future richer functionality:
+
+| Aspect | tree-sitter (Track 1) | gencpp (This Track) |
+|--------|---------------------|---------------------|
+| Scope | Per-file syntax | Per-file declarations |
+| Types | Syntax nodes only | Named types, operators |
+| Timeline | Immediate | Months |
+| Effort | 1-2 weeks | Ongoing |
+
+The tools will be named `ts_c_*` / `ts_cpp_*` for tree-sitter to leave namespace open for `gencpp_*` tools in the future.
--- a/conductor/tracks/ts_cpp_tree_sitter_20260308/metadata.json
+++ b/conductor/tracks/ts_cpp_tree_sitter_20260308/metadata.json
@@ -0,0 +1,16 @@
+{
+  "track_id": "ts_cpp_tree_sitter_20260308",
+  "title": "Tree-Sitter C/C++ MCP Tools",
+  "status": "pending",
+  "created": "2026-03-08",
+  "priority": "high",
+  "owner": "tier2-tech-lead",
+  "description": "Add tree-sitter-based C and C++ parsing to mcp_client with skeleton and outline tools (ts_c_*, ts_cpp_*)",
+  "dependencies": [],
+  "out_of_scope": [
+    "Cross-file symbol resolution",
+    "gencpp integration",
+    "Template instantiation analysis",
+    "Macro expansion"
+  ]
+}
--- a/conductor/tracks/ts_cpp_tree_sitter_20260308/plan.md
+++ b/conductor/tracks/ts_cpp_tree_sitter_20260308/plan.md
@@ -0,0 +1,100 @@
+# Plan: Tree-Sitter C/C++ MCP Tools
+
+## Overview
+Add tree-sitter-based C and C++ parsing to mcp_client with skeleton and outline tools.
+
+## Phase 1: Dependencies
+Focus: Add tree-sitter C/C++ grammars
+
+- [ ] Task 1.1: Add tree-sitter-c and tree-sitter-cpp to pyproject.toml
+  - WHERE: pyproject.toml:16-17
+  - WHAT: Add `"tree-sitter-c>=0.23.0", "tree-sitter-cpp>=0.3.0"` to dependencies
+  - HOW: Edit dependencies array
+  - SAFETY: No breaking changes
+
+## Phase 2: ASTParser Extensions
+Focus: Extend ASTParser to support C/C++ languages
+
+- [ ] Task 2.1: Modify ASTParser.__init__ to accept "c" and "cpp" languages
+  - WHERE: src/file_cache.py:22-28
+  - WHAT: Add language loading for tree-sitter-c and tree-sitter-cpp
+  - HOW: Import tree_sitter_c, tree_sitter_cpp; load Language(tree_sitter_c.language()) etc.
+  - SAFETY: Maintain existing Python support
+
+- [ ] Task 2.2: Implement C skeleton extraction
+  - WHERE: src/file_cache.py (new method or extend get_skeleton)
+  - WHAT: Extract function_definition, struct_specifier, enum_specifier, typedef, union_specifier
+  - HOW: Tree-sitter node traversal similar to Python pattern
+  - SAFETY: New method, no modifications to existing
+
+- [ ] Task 2.3: Implement C++ skeleton extraction
+  - WHERE: src/file_cache.py
+  - WHAT: Add class_specifier, template_declaration, access_specifier, namespace_specifier
+  - HOW: Extend C skeleton logic with C++ specific nodes
+  - SAFETY: New method
+
+- [ ] Task 2.4: Implement code outline for C and C++
+  - WHERE: src/file_cache.py
+  - WHAT: Return hierarchical structure with line ranges (matching py_get_code_outline format)
+  - HOW: Similar to Python get_code_outline pattern
+  - SAFETY: New method
+
+## Phase 3: MCP Tool Integration
+Focus: Add tools to mcp_client dispatch
+
+- [ ] Task 3.1: Add ts_c_get_skeleton tool
+  - WHERE: src/mcp_client.py (add function and register)
+  - WHAT: Tool that calls file_cache ASTParser for C skeleton
+  - HOW: Follow py_get_skeleton pattern
+  - SAFETY: New tool, no modifications to existing
+
+- [ ] Task 3.2: Add ts_cpp_get_skeleton tool
+  - WHERE: src/mcp_client.py
+  - WHAT: Tool that calls file_cache ASTParser for C++ skeleton
+  - HOW: Same as above with cpp language
+  - SAFETY: New tool
+
+- [ ] Task 3.3: Add ts_c_get_code_outline tool
+  - WHERE: src/mcp_client.py
+  - WHAT: Tool that calls file_cache for C code outline
+  - HOW: Follow py_get_code_outline pattern
+  - SAFETY: New tool
+
+- [ ] Task 3.4: Add ts_cpp_get_code_outline tool
+  - WHERE: src/mcp_client.py
+  - WHAT: Tool that calls file_cache for C++ code outline
+  - HOW: Same as above with cpp language
+  - SAFETY: New tool
+
+- [ ] Task 3.5: Register tools in get_tool_schemas
+  - WHERE: src/mcp_client.py:998-1000
+  - WHAT: Add schemas for all 4 new tools
+  - HOW: Append to MCP_TOOL_SPECS list
+  - SAFETY: Append only
+
+## Phase 4: Tests
+Focus: Verify C/C++ tools work correctly
+
+- [ ] Task 4.1: Write tests for ts_c_get_skeleton
+  - WHERE: tests/test_ts_c_tools.py (new file)
+  - WHAT: Test C skeleton extraction on sample C code
+  - HOW: Use pytest with sample C file content
+  - SAFETY: New test file
+
+- [ ] Task 4.2: Write tests for ts_cpp_get_skeleton
+  - WHERE: tests/test_ts_cpp_tools.py (new file)
+  - WHAT: Test C++ skeleton extraction on sample C++ code
+  - HOW: Use pytest with sample C++ code
+  - SAFETY: New test file
+
+- [ ] Task 4.3: Write tests for code outline tools
+  - WHERE: tests/test_ts_c_tools.py / test_ts_cpp_tools.py
+  - WHAT: Test line range extraction
+  - HOW: Assert correct line numbers
+  - SAFETY: New tests
+
+- [ ] Task 4.4: Integration test - verify tools dispatch correctly
+  - WHERE: tests/test_mcp_client.py
+  - WHAT: Test dispatch of ts_c_* and ts_cpp_* tools
+  - HOW: Mock file_cache, verify correct function called
+  - SAFETY: Additive test
--- a/conductor/tracks/ts_cpp_tree_sitter_20260308/spec.md
+++ b/conductor/tracks/ts_cpp_tree_sitter_20260308/spec.md
@@ -0,0 +1,72 @@
+# Track Specification: Tree-Sitter C/C++ MCP Tools
+
+## Overview
+
+Add tree-sitter-based C and C++ parsing support to the MCP client, providing skeleton and outline tools for C/C++ codebases. Tools will be prefixed `ts_c_` and `ts_cpp_` to distinguish from existing Python tools and leave namespace open for future gencpp integration.
+
+## Current State Audit (as of 08e003a)
+
+### Already Implemented
+- `src/file_cache.py`: ASTParser class with tree-sitter-python support
+- `src/mcp_client.py`: 26 MCP tools (all Python-prefixed: `py_get_skeleton`, `py_get_definition`, etc.)
+- `pyproject.toml`: tree-sitter>=0.25.2 and tree-sitter-python>=0.25.0 already listed
+- Existing pattern: `get_skeleton()`, `get_code_outline()` methods extract functions, classes, docstrings
+
+### Gaps to Fill (This Track's Scope)
+- No C/C++ tree-sitter grammars installed
+- No C/C++ parsing logic in ASTParser
+- No MCP tools for C/C++ code extraction
+
+## Goals
+
+1. Add tree-sitter-c and tree-sitter-cpp dependencies
+2. Extend ASTParser to support C and C++ languages
+3. Implement skeleton and outline generation for C/C++ (functions, structs, enums, classes)
+4. Add MCP tools: `ts_c_get_skeleton`, `ts_cpp_get_skeleton`, `ts_c_get_code_outline`, `ts_cpp_get_code_outline`
+5. Register tools in mcp_client dispatch
+
+## Functional Requirements
+
+### Dependencies
+- Add `tree-sitter-c` to pyproject.toml
+- Add `tree-sitter-cpp` to pyproject.toml
+
+### ASTParser Extensions
+- Modify `ASTParser.__init__` to accept "c", "cpp", "python" as valid languages
+- Load appropriate tree-sitter language grammar based on parameter
+- Reuse existing caching mechanism (`get_cached_tree`)
+- Implement C-specific node types: `function_definition`, `struct_specifier`, `enum_specifier`, `typedef`
+- Implement C++-specific node types: all C types plus `class_specifier`, `access_specifier`, `template_declaration`
+
+### MCP Tools
+| Tool Name | Description |
+|-----------|-------------|
+| `ts_c_get_skeleton` | Returns C file skeleton (function signatures, struct/union/enum definitions) |
+| `ts_cpp_get_skeleton` | Returns C++ file skeleton (above + class methods, templates, namespaces) |
+| `ts_c_get_code_outline` | Returns hierarchical C outline (functions, structs, enums, globals with line ranges) |
+| `ts_cpp_get_code_outline` | Returns hierarchical C++ outline (above + classes, templates, namespaces) |
+
+### Tool Output Format
+Match existing Python tool formats for consistency:
+- Skeleton: signatures + docstrings, bodies replaced with `...`
+- Outline: hierarchical list with `[Class] name (Lines X-Y)` format
+
+## Non-Functional Requirements
+
+- Follow existing 1-space indentation code style
+- Use exact same patterns as existing Python tree-sitter implementation
+- Maintain AST cache with mtime invalidation (reuse existing logic)
+- Tools must pass allowlist check in mcp_client
+
+## Architecture Reference
+
+- **ASTParser pattern**: `src/file_cache.py:16-333` - existing tree-sitter integration
+- **MCP tool dispatch**: `src/mcp_client.py:920-987` - tool registration and dispatch
+- **Tool schema format**: `src/mcp_client.py:998-1000` - `get_tool_schemas()`
+
+## Out of Scope
+- Cross-file symbol resolution (AI uses search tools for this)
+- Template instantiation analysis
+- Macro expansion
+- gencpp integration (future separate track)
+- Writing to C/C++ files (read-only for now)
--- a/src/api_hooks.py
+++ b/src/api_hooks.py
@@ -7,6 +7,35 @@ from http.server import ThreadingHTTPServer, BaseHTTPRequestHandler
 from typing import Any
 import logging
 from src import session_logger
+"""
+API Hooks - REST API for external automation and state inspection.
+
+This module implements the HookServer, which exposes internal application state to external HTTP requests on port 8999 using Python's
+ ThreadingHTTPServer. All endpoints are thread-safe and reads that pass through lock-guarded lists,
+ while stateful reads use the GUI thread trampoline pattern.
+
+Architecture:
+ - HookServer: ThreadingHTTPServer with app reference
+ - HookHandler: BaseHTTPRequestHandler per request
+ - Request handling uses trampoline pattern for GUI state reads
+ - GUI Thread Trampoline: Create threading.Event + result dict
+   - Push callback to `_pending_gui_tasks`
+   - Wait for event (timeout)
+   - Return result as JSON
+
+Thread Safety:
+ - All reads use lock-protected lists
+ - All state mutations happen on the GUI thread
+ - The module does to maintain separation between App and AppController
+
+Configuration:
+ - `--enable-test-hooks`: Required for Hook API to be available
+ - `gemini_cli` provider: Hook API is automatically available for synchronous HITL
+
+See Also:
+ - docs/guide_tools.md for full API reference
+ - api_hook_client.py for the client implementation
+"""

 def _get_app_attr(app: Any, name: str, default: Any = None) -> Any:
 if hasattr(app, name):
--- a/src/multi_agent_conductor.py
+++ b/src/multi_agent_conductor.py
@@ -1,3 +1,60 @@
+"""
+Multi-Agent Conductor - 4-tier MMA orchestration engine.
+
+This module implements the ConductorEngine and WorkerPool for executing
+implementation tracks via the 4-tier Multi-Model Agent (MMA) hierarchy.
+
+Key Components:
+ - WorkerPool: Bounded concurrent worker pool with semaphore gating
+ - ConductorEngine: Main orchestration loop with DAG execution
+ - run_worker_lifecycle: Tier 3 worker execution function
+
+Architecture Integration:
+ - Uses TrackDAG and ExecutionEngine from dag_engine.py
+ - Communicates with GUI via SyncEventQueue
+ - Manages tier-specific token usage via update_usage()
+
+Thread Safety:
+ - WorkerPool uses threading.Lock for all state mutations
+ - ConductorEngine uses _tier_usage_lock for tier usage tracking
+ - Abort events use threading.Event for worker cancellation
+
+Configuration:
+ - max_workers: Loaded from config.toml [mma].max_workers (default: 4)
+
+See Also:
+ - docs/guide_mma.md for full MMA documentation
+ - src/dag_engine.py for TrackDAG and ExecutionEngine
+ - src/models.py for Ticket, Track, WorkerContext
+"""
+"""
+Multi-Agent Conductor - MMA 4-Tier orchestration engine.
+
+This module provides the ConductorEngine and WorkerPool for orchestrating
+ the execution of implementation tickets within a Track using the DAG engine
+and the bounded concurrent worker pool with abort event propagation.
+
+Key Components:
+ - ConductorEngine: Tier 2 orchestrator that owns the execution loop
+ - WorkerPool: Bounded concurrent worker pool with semaphore gating
+ - run_worker_lifecycle: Stateless Tier 3 worker execution with context amnesia
+
+Thread Safety:
+ - All state mutations use locks (_workers_lock, _tier_usage_lock)
+ - Worker threads are daemon threads that clean up on exit
+ - Abort events enable per-ticket cancellation
+
+Integration:
+ - Uses SyncEventQueue for state updates to the GUI
+ - Uses ai_client.send() for LLM communication
+ - Uses mcp_client for tool dispatch
+
+See Also:
+ - docs/guide_mma.md for MMA orchestration documentation
+ - src/dag_engine.py for TrackDAG and ExecutionEngine
+ - src/ai_client.py for multi-provider LLM abstraction
+ - src/models.py for Ticket, Track, WorkerContext data structures
+"""
 from src import ai_client
 import json
 import threading