docs update (wip)

2026-03-08 01:46:34 -05:00
parent d9a06fd2fe
commit d34c35941f
14 changed files with 1213 additions and 105 deletions
--- a/docs/Readme.md
+++ b/docs/Readme.md
@@ -1,6 +1,12 @@
 # Documentation Index

-[Top](../Readme.md)
+[Top](../README.md)
+
+---
+
+## Overview
+
+This documentation suite provides comprehensive technical reference for the Manual Slop application — a GUI orchestrator for local LLM-driven coding sessions. The guides follow a strict old-school technical documentation style, emphasizing architectural depth, state management details, algorithmic breakdowns, and structural formats.

 ---

@@ -8,68 +14,341 @@

 | Guide | Contents |
 |---|---|
-| [Architecture](guide_architecture.md) | Thread domains, cross-thread data structures, event system, application lifetime, task pipeline (producer-consumer), Execution Clutch (HITL), AI client multi-provider architecture, Anthropic/Gemini caching strategies, context refresh, comms logging, state machines |
-| [Meta-Boundary](guide_meta_boundary.md) | Explicit distinction between the Application's domain (Strict HITL) and the Meta-Tooling domain (autonomous agents), preventing feature bleed and safety bypasses via shared bridges like `mcp_client.py`. |
-| [Tools & IPC](guide_tools.md) | MCP Bridge 3-layer security model, all 26 native tool signatures, Hook API GET/POST endpoints with request/response formats, ApiHookClient method reference, `/api/ask` synchronous HITL protocol, session logging, shell runner |
-| [MMA Orchestration](guide_mma.md) | Ticket/Track/WorkerContext data structures, DAG engine (cycle detection, topological sort), ConductorEngine execution loop, Tier 2 ticket generation, Tier 3 worker lifecycle with context amnesia, Tier 4 QA integration, token firewalling, track state persistence |
-| [Simulations](guide_simulations.md) | `live_gui` pytest fixture lifecycle, `VerificationLogger`, process cleanup, Puppeteer pattern (8-stage MMA simulation), approval automation, mock provider (`mock_gemini_cli.py`) with JSON-L protocol, visual verification patterns, ASTParser (tree-sitter) vs summarizer (stdlib `ast`) |
+| [Architecture](guide_architecture.md) | Thread domains (GUI Main, Asyncio Worker, HookServer, Ad-hoc), cross-thread data structures (AsyncEventQueue, Guarded Lists, Condition-Variable Dialogs), event system (EventEmitter, SyncEventQueue, UserRequestEvent), application lifetime (boot sequence, shutdown sequence), task pipeline (producer-consumer synchronization), Execution Clutch (HITL mechanism with ConfirmDialog, MMAApprovalDialog, MMASpawnApprovalDialog), AI client multi-provider architecture (Gemini SDK, Anthropic, DeepSeek, Gemini CLI, MiniMax), Anthropic/Gemini caching strategies (4-breakpoint system, server-side TTL), context refresh mechanism (mtime-based file re-reading, diff injection), comms logging (JSON-L format), state machines (ai_status, HITL dialog state) |
+| [Meta-Boundary](guide_meta_boundary.md) | Explicit distinction between the Application's domain (Strict HITL — `gui_2.py`, `ai_client.py`, `multi_agent_conductor.py`, `dag_engine.py`) and the Meta-Tooling domain (`scripts/mma_exec.py`, `scripts/claude_mma_exec.py`, `scripts/tool_call.py`, `scripts/mcp_server.py`, `.gemini/`, `.claude/`), preventing feature bleed and safety bypasses via shared bridges like `mcp_client.py`. Documents the Inter-Domain Bridges (`cli_tool_bridge.py`, `claude_tool_bridge.py`) and the `GEMINI_CLI_HOOK_CONTEXT` environment variable. |
+| [Tools & IPC](guide_tools.md) | MCP Bridge 3-layer security model (Allowlist Construction, Path Validation, Resolution Gate), all 26 native tool signatures with parameters and behavior (File I/O, AST-Based, Analysis, Network, Runtime), Hook API GET/POST endpoints with request/response formats, ApiHookClient method reference (Connection Methods, State Query Methods, GUI Manipulation Methods, Polling Methods, HITL Method), `/api/ask` synchronous HITL protocol (blocking request-response over HTTP), session logging (comms.log, toolcalls.log, apihooks.log, clicalls.log, scripts/generated/*.ps1), shell runner (mcp_env.toml configuration, run_powershell function with timeout handling and QA callback integration) |
+| [MMA Orchestration](guide_mma.md) | Ticket/Track/WorkerContext data structures (from `models.py`), DAG engine (TrackDAG class with cycle detection, topological sort, cascade_blocks; ExecutionEngine class with tick-based state machine), ConductorEngine execution loop (run method, _push_state for state broadcast, parse_json_tickets for ingestion), Tier 2 ticket generation (generate_tickets, topological_sort), Tier 3 worker lifecycle (run_worker_lifecycle with Context Amnesia, AST skeleton injection, HITL clutch integration via confirm_spawn and confirm_execution), Tier 4 QA integration (run_tier4_analysis, run_tier4_patch_callback), token firewalling (tier_usage tracking, model escalation), track state persistence (TrackState, save_track_state, load_track_state, get_all_tracks) |
+| [Simulations](guide_simulations.md) | Structural Testing Contract (Ban on Arbitrary Core Mocking, `live_gui` Standard, Artifact Isolation), `live_gui` pytest fixture lifecycle (spawning, readiness polling, failure path, teardown, session isolation via reset_ai_client), VerificationLogger for structured diagnostic logging, process cleanup (kill_process_tree for Windows/Unix), Puppeteer pattern (8-stage MMA simulation with mock provider setup, epic planning, track acceptance, ticket loading, status transitions, worker output verification), mock provider strategy (`tests/mock_gemini_cli.py` with JSON-L protocol, input mechanisms, response routing, output protocol), visual verification patterns (DAG integrity, stream telemetry, modal state, performance monitoring), supporting analysis modules (ASTParser with tree-sitter, summarize.py heuristic summaries, outline_tool.py hierarchical outlines) |

 ---

 ## GUI Panels

-### Projects Panel
+### Context Hub

-Configuration and context management. Specifies the Git Directory (for commit tracking) and tracked file paths. Project switching swaps the active file list, discussion history, and settings via `<project>.toml` profiles.
+The primary panel for project and file management.

- **Word-Wrap Toggle**: Dynamically swaps text rendering in large read-only panels (Responses, Comms Log) between unwrapped (code formatting) and wrapped (prose).
+- **Project Selector**: Switch between `<project>.toml` configurations. Changing projects swaps the active file list, discussion history, and settings.
+- **Git Directory**: Path to the repository for commit tracking and git operations.
+- **Main Context File**: Optional primary context document for the project.
+- **Output Dir**: Directory where generated markdown files are written.
+- **Word-Wrap Toggle**: Dynamically swaps text rendering in large read-only panels between unwrapped (code formatting) and wrapped (prose).
+- **Summary Only**: When enabled, sends file structure summaries instead of full content to reduce token usage.
+- **Auto-Scroll Comms/Tool History**: Automatically scrolls to the bottom when new entries arrive.

-### Discussion History
+### Files & Media Panel
+
+Controls what context is compiled and sent to the AI.
+
+- **Base Dir**: Root directory for path resolution and MCP tool constraints.
+- **Paths**: Explicit files or wildcard globs (`src/**/*.py`).
+- **File Flags**:
+  - **Auto-Aggregate**: Include in context compilation.
+  - **Force Full**: Bypass summary-only mode for this file.
+- **Cache Indicator**: Green dot (●) indicates file is in provider's context cache.
+
+### Discussion Hub

 Manages conversational branches to prevent context poisoning across tasks.

 - **Discussions Sub-Menu**: Create separate timelines for different tasks (e.g., "Refactoring Auth" vs. "Adding API Endpoints").
 - **Git Commit Tracking**: "Update Commit" reads HEAD from the project's git directory and stamps the discussion.
- **Entry Management**: Each turn has a Role (User, AI, System). Toggle between Read/Edit modes, collapse entries, or open in the Global Text Viewer via `[+ Max]`.
+- **Entry Management**: Each turn has a Role (User, AI, System, Context, Tool, Vendor API). Toggle between Read/Edit modes, collapse entries, or open in the Global Text Viewer via `[+ Max]`.
 - **Auto-Add**: When toggled, Message panel sends and Response panel returns are automatically appended to the current discussion.
+- **Truncate History**: Reduces history to N most recent User/AI pairs.

-### Files & Screenshots
+### AI Settings Panel

-Controls what is fed into the context compiler.
+- **Provider**: Switch between API backends (Gemini, Anthropic, DeepSeek, Gemini CLI, MiniMax).
+- **Model**: Select from available models for the current provider.
+- **Fetch Models**: Queries the active provider for the latest model list.
+- **Temperature / Max Tokens**: Generation parameters.
+- **History Truncation Limit**: Character limit for truncating old tool outputs.

- **Base Dir**: Defines the root for path resolution and MCP tool constraints.
- **Paths**: Explicit files or wildcard globs (`src/**/*.rs`).
- Full file contents are inlined by default. The AI can call `get_file_summary` for compact structural views.
+### Token Budget Panel

-### Provider
+- **Current Usage**: Real-time token counts (input, output, cache read, cache creation).
+- **Budget Percentage**: Visual indicator of context window utilization.
+- **Provider-Specific Limits**: Anthropic (180K prompt), Gemini (900K input).

-Switches between API backends (Gemini, Anthropic, DeepSeek, Gemini CLI). "Fetch Models" queries the active provider for the latest model list.
+### Cache Panel

-### Message & Response
+- **Gemini Cache Stats**: Count, total size, and list of cached files.
+- **Clear Cache**: Forces cache invalidation on next send.

- **Message**: User input field.
+### Tool Analytics Panel
+
+- **Per-Tool Statistics**: Call count, total time, failure count for each tool.
+- **Session Insights**: Burn rate estimation, average latency.
+
+### Message & Response Panels
+
+- **Message**: User input field with auto-expanding height.
 - **Gen + Send**: Compiles markdown context and dispatches to the AI via `AsyncEventQueue`.
 - **MD Only**: Dry-runs the compiler for context inspection without API cost.
 - **Response**: Read-only output; flashes green on new response.

-### Global Text Viewer & Script Outputs
+### Operations Hub

- **Last Script Output**: Pops up (flashing blue) whenever the AI executes a script. Shows both the executed script and stdout/stderr. `[+ Maximize]` reads from stored instance variables, not DPG widget tags, so it works regardless of word-wrap state.
- **Text Viewer**: Large resizable popup invoked by `[+]` / `[+ Maximize]` buttons. For deep-reading long logs, discussion entries, or script bodies.
- **Confirm Dialog**: The `[+ Maximize]` button in the script approval modal passes script text as `user_data` at button-creation time — safe to click even after the dialog is dismissed.
-
-### Tool Calls & Comms History
-
-Real-time display of MCP tool invocations and raw API traffic. Each comms entry: timestamp, direction (OUT/IN), kind, provider, model, payload.
+- **Focus Agent Filter**: Show comms/tool history for specific tier (All, Tier 2, Tier 3, Tier 4).
+- **Comms History**: Real-time display of raw API traffic (timestamp, direction, kind, provider, model, payload preview).
+- **Tool Calls**: Sequential log of tool invocations with script/args and result preview.

 ### MMA Dashboard

-Displays the 4-tier orchestration state: active track, ticket DAG with status indicators, per-tier token usage, output streams. Approval buttons for spawn/step/tool gates.
+The 4-tier orchestration control center.

-### System Prompts
+- **Track Browser**: List of all tracks with status, progress, and actions (Load, Delete).
+- **Active Track Summary**: Color-coded progress bar, ticket status breakdown (Completed, In Progress, Blocked, Todo), ETA estimation.
+- **Visual Task DAG**: Node-based visualization using `imgui-node-editor` with color-coded states (Ready, Running, Blocked, Done).
+- **Ticket Queue Management**: Bulk operations (Execute, Skip, Block), drag-and-drop reordering, priority assignment.
+- **Tier Streams**: Real-time output from Tier 1/2/3/4 agents.

-Two text inputs for instruction overrides:
-1. **Global**: Applied across every project.
-2. **Project**: Specific to the active workspace.
+### Tier Stream Panels

-Concatenated onto the base tool-usage guidelines.
+Dedicated windows for each MMA tier:
+
+- **Tier 1: Strategy**: Orchestrator output for epic planning and track initialization.
+- **Tier 2: Tech Lead**: Architectural decisions and ticket generation.
+- **Tier 3: Workers**: Individual worker output streams (one per active ticket).
+- **Tier 4: QA**: Error analysis and diagnostic summaries.
+
+### Log Management
+
+- **Session Registry**: Table of all session logs with metadata (start time, message count, size, whitelist status).
+- **Star/Unstar**: Mark sessions for preservation during pruning.
+- **Force Prune**: Manually trigger aggressive log cleanup.
+
+### Diagnostics Panel
+
+- **Performance Telemetry**: FPS, Frame Time, CPU %, Input Lag with moving averages.
+- **Detailed Component Timings**: Per-panel rendering times with threshold alerts.
+- **Performance Graphs**: Historical plots for selected metrics.
+
+---
+
+## Configuration Files
+
+### config.toml (Global)
+
+```toml
+[ai]
+provider = "gemini"
+model = "gemini-2.5-flash-lite"
+temperature = 0.0
+max_tokens = 8192
+history_trunc_limit = 8000
+system_prompt = ""
+
+[projects]
+active = "path/to/project.toml"
+paths = ["path/to/project.toml"]
+
+[gui]
+separate_message_panel = false
+separate_response_panel = false
+separate_tool_calls_panel = false
+show_windows = { "Context Hub": true, ... }
+
+[paths]
+logs_dir = "logs/sessions"
+scripts_dir = "scripts/generated"
+conductor_dir = "conductor"
+
+[mma]
+max_workers = 4
+```
+
+### <project>.toml (Per-Project)
+
+```toml
+[project]
+name = "my_project"
+git_dir = "./my_repo"
+system_prompt = ""
+main_context = ""
+
+[files]
+base_dir = "."
+paths = ["src/**/*.py"]
+tier_assignments = { "src/core.py" = 1 }
+
+[screenshots]
+base_dir = "."
+paths = []
+
+[output]
+output_dir = "./md_gen"
+
+[gemini_cli]
+binary_path = "gemini"
+
+[deepseek]
+reasoning_effort = "medium"
+
+[agent.tools]
+run_powershell = true
+read_file = true
+list_directory = true
+search_files = true
+get_file_summary = true
+web_search = true
+fetch_url = true
+py_get_skeleton = true
+py_get_code_outline = true
+get_file_slice = true
+set_file_slice = false
+edit_file = false
+py_get_definition = true
+py_update_definition = false
+py_get_signature = true
+py_set_signature = false
+py_get_class_summary = true
+py_get_var_declaration = true
+py_set_var_declaration = false
+get_git_diff = true
+py_find_usages = true
+py_get_imports = true
+py_check_syntax = true
+py_get_hierarchy = true
+py_get_docstring = true
+get_tree = true
+get_ui_performance = true
+
+[mma]
+epic = ""
+active_track_id = ""
+tracks = []
+```
+
+### credentials.toml
+
+```toml
+[gemini]
+api_key = "YOUR_KEY"
+
+[anthropic]
+api_key = "YOUR_KEY"
+
+[deepseek]
+api_key = "YOUR_KEY"
+
+[minimax]
+api_key = "YOUR_KEY"
+```
+
+### mcp_env.toml (Optional)
+
+```toml
+[path]
+prepend = ["C:/custom/bin"]
+
+[env]
+MY_VAR = "some_value"
+EXPANDED = "${HOME}/subdir"
+```
+
+---
+
+## Environment Variables
+
+| Variable | Purpose |
+|---|---|
+| `SLOP_CONFIG` | Override path to `config.toml` |
+| `SLOP_CREDENTIALS` | Override path to `credentials.toml` |
+| `SLOP_MCP_ENV` | Override path to `mcp_env.toml` |
+| `SLOP_TEST_HOOKS` | Set to `"1"` to enable test hooks |
+| `SLOP_LOGS_DIR` | Override logs directory |
+| `SLOP_SCRIPTS_DIR` | Override generated scripts directory |
+| `SLOP_CONDUCTOR_DIR` | Override conductor directory |
+| `GEMINI_CLI_HOOK_CONTEXT` | Set by bridge scripts to bypass HITL for sub-agents |
+| `CLAUDE_CLI_HOOK_CONTEXT` | Set by bridge scripts to bypass HITL for sub-agents |
+
+---
+
+## Exit Codes
+
+| Code | Meaning |
+|---|---|
+| 0 | Normal exit |
+| 1 | General error |
+| 2 | Configuration error |
+| 3 | API error |
+| 4 | Test failure |
+
+---
+
+## File Layout
+
+```
+manual_slop/
+├── conductor/              # Conductor system
+│   ├── tracks/             # Track directories
+│   │   └── <track_id>/     # Per-track files
+│   │       ├── spec.md
+│   │       ├── plan.md
+│   │       ├── metadata.json
+│   │       └── state.toml
+│   ├── archive/            # Completed tracks
+│   ├── product.md          # Product definition
+│   ├── product-guidelines.md
+│   ├── tech-stack.md
+│   └── workflow.md
+├── docs/                   # Deep-dive documentation
+│   ├── guide_architecture.md
+│   ├── guide_meta_boundary.md
+│   ├── guide_mma.md
+│   ├── guide_simulations.md
+│   └── guide_tools.md
+├── logs/                   # Runtime logs
+│   ├── sessions/           # Session logs
+│   │   └── <session_id>/   # Per-session files
+│   │       ├── comms.log
+│   │       ├── toolcalls.log
+│   │       ├── apihooks.log
+│   │       └── clicalls.log
+│   ├── agents/             # Sub-agent logs
+│   ├── errors/             # Error logs
+│   └── test/               # Test logs
+├── scripts/                # Utility scripts
+│   ├── generated/          # AI-generated scripts
+│   └── *.py                # Build/execution scripts
+├── src/                    # Core implementation
+│   ├── gui_2.py            # Primary ImGui interface
+│   ├── app_controller.py   # Headless controller
+│   ├── ai_client.py        # Multi-provider LLM abstraction
+│   ├── mcp_client.py       # 26 MCP tools
+│   ├── api_hooks.py        # HookServer REST API
+│   ├── api_hook_client.py  # Hook API client
+│   ├── multi_agent_conductor.py  # ConductorEngine
+│   ├── conductor_tech_lead.py    # Tier 2 ticket generation
+│   ├── dag_engine.py       # TrackDAG + ExecutionEngine
+│   ├── models.py           # Ticket, Track, WorkerContext
+│   ├── events.py           # EventEmitter, SyncEventQueue
+│   ├── project_manager.py  # TOML persistence
+│   ├── session_logger.py   # JSON-L logging
+│   ├── shell_runner.py     # PowerShell execution
+│   ├── file_cache.py       # ASTParser (tree-sitter)
+│   ├── summarize.py        # Heuristic summaries
+│   ├── outline_tool.py     # Code outlining
+│   ├── performance_monitor.py  # FPS/CPU tracking
+│   ├── log_registry.py     # Session metadata
+│   ├── log_pruner.py       # Log cleanup
+│   ├── paths.py            # Path resolution
+│   ├── cost_tracker.py     # Token cost estimation
+│   ├── gemini_cli_adapter.py  # CLI subprocess adapter
+│   ├── mma_prompts.py      # Tier system prompts
+│   └── theme*.py           # UI theming
+├── simulation/             # Test simulations
+│   ├── sim_base.py         # BaseSimulation class
+│   ├── workflow_sim.py     # WorkflowSimulator
+│   ├── user_agent.py       # UserSimAgent
+│   └── sim_*.py            # Specific simulations
+├── tests/                  # Test suite
+│   ├── conftest.py         # Fixtures (live_gui)
+│   ├── artifacts/          # Test outputs
+│   └── test_*.py           # Test files
+├── sloppy.py               # Main entry point
+├── config.toml             # Global configuration
+└── credentials.toml        # API keys
+```