4 Commits

Author SHA1 Message Date
ed 98d2f17fc6 project toml 2026-02-22 12:15:16 -05:00
ed b75b4a7c32 still botched 2026-02-22 12:10:19 -05:00
ed ebf9ffd653 Merge branch 'master' into wip
# Conflicts:
#	config.toml
#	manual_slop.toml
2026-02-22 12:03:23 -05:00
ed 13ad7aea17 fixing 2026-02-22 12:03:07 -05:00
39 changed files with 240 additions and 1241 deletions
-1
View File
@@ -6,4 +6,3 @@ md_gen
scripts/generated scripts/generated
logs logs
dpg_layout.ini dpg_layout.ini
.env
-47
View File
@@ -1,47 +0,0 @@
# Project Overview
**Manual Slop** is a local GUI application designed as an experimental, "manual" AI coding assistant. It allows users to curate and send context (files, screenshots, and discussion history) to AI APIs (Gemini and Anthropic). The AI can then execute PowerShell scripts within the project directory to modify files, requiring explicit user confirmation before execution.
**Main Technologies:**
* **Language:** Python 3.11+
* **Package Management:** `uv`
* **GUI Framework:** Dear PyGui (`dearpygui`), ImGui Bundle (`imgui-bundle`)
* **AI SDKs:** `google-genai` (Gemini), `anthropic`
* **Configuration:** TOML (`tomli-w`)
**Architecture:**
* **`gui.py`:** The main entry point and Dear PyGui application logic. Handles all panels, layouts, user input, and confirmation dialogs.
* **`ai_client.py`:** A unified wrapper for both Gemini and Anthropic APIs. Manages sessions, tool/function-call loops, token estimation, and context history management.
* **`aggregate.py`:** Responsible for building the `file_items` context. It reads project configurations, collects files and screenshots, and builds the context into markdown format to send to the AI.
* **`mcp_client.py`:** Implements MCP-like tools (e.g., `read_file`, `list_directory`, `search_files`, `web_search`) as native functions that the AI can call. Enforces a strict allowlist for file access.
* **`shell_runner.py`:** A sandboxed subprocess wrapper that executes PowerShell scripts (`powershell -NoProfile -NonInteractive -Command`) provided by the AI.
* **`project_manager.py`:** Manages per-project TOML configurations (`manual_slop.toml`), serializes discussion entries, and integrates with git (e.g., fetching current commit).
* **`session_logger.py`:** Handles timestamped logging of communication history (JSON-L) and tool calls (saving generated `.ps1` files).
# Building and Running
* **Setup:** The application uses `uv` for dependency management. Ensure `uv` is installed.
* **Credentials:** You must create a `credentials.toml` file in the root directory to store your API keys:
```toml
[gemini]
api_key = "****"
[anthropic]
api_key = "****"
```
* **Run the Application:**
```powershell
uv run .\gui.py
```
# Development Conventions
* **Configuration Management:** The application uses two tiers of configuration:
* `config.toml`: Global settings (UI theme, active provider, list of project paths).
* `manual_slop.toml`: Per-project settings (files to track, discussion history, specific system prompts).
* **Tool Execution:** The AI acts primarily by generating PowerShell scripts. These scripts MUST be confirmed by the user via a GUI modal before execution. The AI also has access to read-only MCP-style file exploration tools and web search capabilities.
* **Context Refresh:** After every tool call that modifies the file system, the application automatically refreshes the file contents in the context using the files' `mtime` to optimize reads.
* **UI State Persistence:** Window layouts and docking arrangements are automatically saved to and loaded from `dpg_layout.ini`.
* **Code Style:**
* Use type hints where appropriate.
* Internal methods and variables are generally prefixed with an underscore (e.g., `_flush_to_project`, `_do_generate`).
* **Logging:** All API communications are logged to `logs/comms_<ts>.log`. All executed scripts are saved to `scripts/generated/`.
+21 -38
View File
@@ -12,16 +12,16 @@ Is a local GUI tool for manually curating and sending context to AI APIs. It agg
- `uv` - package/env management - `uv` - package/env management
**Files:** **Files:**
- `gui.py` - main GUI, `App` class, all panels, all callbacks, confirmation dialog, layout persistence, rich comms rendering; `[+ Maximize]` buttons in `ConfirmDialog` and `win_script_output` now pass text directly as `user_data` / read from `self._last_script` / `self._last_output` instance vars instead of `dpg.get_value(tag)` — fixes glitch when word-wrap is ON or dialog is dismissed before viewer opens - `gui.py` - main GUI, `App` class, all panels, all callbacks, confirmation dialog, layout persistence, rich comms rendering
- `ai_client.py` - unified provider wrapper, model listing, session management, send, tool/function-call loop, comms log, provider error classification, token estimation, and aggressive history truncation - `ai_client.py` - unified provider wrapper, model listing, session management, send, tool/function-call loop, comms log, provider error classification
- `aggregate.py` - reads config, collects files/screenshots/discussion, builds `file_items` with `mtime` for cache optimization, writes numbered `.md` files to `output_dir` using `build_markdown_from_items` to avoid double I/O; `run()` returns `(markdown_str, path, file_items)` tuple; `summary_only=False` by default (full file contents sent, not heuristic summaries) - `aggregate.py` - reads config, collects files/screenshots/discussion, writes numbered `.md` files to `output_dir`
- `shell_runner.py` - subprocess wrapper that runs PowerShell scripts sandboxed to `base_dir`, returns stdout/stderr/exit code as a string - `shell_runner.py` - subprocess wrapper that runs PowerShell scripts sandboxed to `base_dir`, returns stdout/stderr/exit code as a string
- `session_logger.py` - opens timestamped log files at session start; writes comms entries as JSON-L and tool calls as markdown; saves each AI-generated script as a `.ps1` file - `session_logger.py` - opens timestamped log files at session start; writes comms entries as JSON-L and tool calls as markdown; saves each AI-generated script as a `.ps1` file
- `project_manager.py` - per-project .toml load/save, entry serialisation (entry_to_str/str_to_entry with @timestamp support), default_project/default_discussion factories, migrate_from_legacy_config, flat_config for aggregate.run(), git helpers (get_git_commit, get_git_log) - `project_manager.py` - per-project .toml load/save, entry serialisation (entry_to_str/str_to_entry with @timestamp support), default_project/default_discussion factories, migrate_from_legacy_config, flat_config for aggregate.run(), git helpers (get_git_commit, get_git_log)
- `theme.py` - palette definitions, font loading, scale, load_from_config/save_to_config - `theme.py` - palette definitions, font loading, scale, load_from_config/save_to_config
- `gemini.py` - legacy standalone Gemini wrapper (not used by the main GUI; superseded by `ai_client.py`) - `gemini.py` - legacy standalone Gemini wrapper (not used by the main GUI; superseded by `ai_client.py`)
- `file_cache.py` - stub; Anthropic Files API path removed; kept so stale imports don't break - `file_cache.py` - stub; Anthropic Files API path removed; kept so stale imports don't break
- `mcp_client.py` - MCP-style tools (read_file, list_directory, search_files, get_file_summary, web_search, fetch_url); allowlist enforced against project file_items + base_dirs for file tools; web tools are unrestricted; dispatched by ai_client tool-use loop for both Anthropic and Gemini - `mcp_client.py` - MCP-style read-only file tools (read_file, list_directory, search_files, get_file_summary); allowlist enforced against project file_items + base_dirs; dispatched by ai_client tool-use loop for both Anthropic and Gemini
- `summarize.py` - local heuristic summariser (no AI); .py via AST, .toml via regex, .md headings, generic preview; used by mcp_client.get_file_summary and aggregate.build_summary_section - `summarize.py` - local heuristic summariser (no AI); .py via AST, .toml via regex, .md headings, generic preview; used by mcp_client.get_file_summary and aggregate.build_summary_section
- `config.toml` - global-only settings: [ai] provider+model+system_prompt, [theme] palette+font+scale, [projects] paths array + active path - `config.toml` - global-only settings: [ai] provider+model+system_prompt, [theme] palette+font+scale, [projects] paths array + active path
- `manual_slop.toml` - per-project file: [project] name+git_dir+system_prompt+main_context, [output] namespace+output_dir, [files] base_dir+paths, [screenshots] base_dir+paths, [discussion] roles+active+[discussion.discussions.<name>] git_commit+last_updated+history - `manual_slop.toml` - per-project file: [project] name+git_dir+system_prompt+main_context, [output] namespace+output_dir, [files] base_dir+paths, [screenshots] base_dir+paths, [discussion] roles+active+[discussion.discussions.<name>] git_commit+last_updated+history
@@ -87,7 +87,7 @@ Is a local GUI tool for manually curating and sending context to AI APIs. It agg
- All tool calls (script + result/rejection) are appended to `_tool_log` and displayed in the Tool Calls panel - All tool calls (script + result/rejection) are appended to `_tool_log` and displayed in the Tool Calls panel
**Dynamic file context refresh (ai_client.py):** **Dynamic file context refresh (ai_client.py):**
- After the last tool call in each round, project files from `file_items` are checked via `_reread_file_items()`. It uses `mtime` to only re-read modified files, returning only the `changed` files to build a minimal `[FILES UPDATED]` block. - After the last tool call in each round, all project files from `file_items` are re-read from disk via `_reread_file_items()`. The `file_items` variable is reassigned so subsequent rounds see fresh content.
- For Anthropic: the refreshed file contents are injected as a `text` block appended to the `tool_results` user message, prefixed with `[FILES UPDATED]` and an instruction not to re-read them. - For Anthropic: the refreshed file contents are injected as a `text` block appended to the `tool_results` user message, prefixed with `[FILES UPDATED]` and an instruction not to re-read them.
- For Gemini: refreshed file contents are appended to the last function response's `output` string as a `[SYSTEM: FILES UPDATED]` block. On the next tool round, stale `[FILES UPDATED]` blocks are stripped from history and old tool outputs are truncated to `_history_trunc_limit` characters to control token growth. - For Gemini: refreshed file contents are appended to the last function response's `output` string as a `[SYSTEM: FILES UPDATED]` block. On the next tool round, stale `[FILES UPDATED]` blocks are stripped from history and old tool outputs are truncated to `_history_trunc_limit` characters to control token growth.
- `_build_file_context_text(file_items)` formats the refreshed files as markdown code blocks (same format as the original context) - `_build_file_context_text(file_items)` formats the refreshed files as markdown code blocks (same format as the original context)
@@ -141,12 +141,10 @@ Entry layout: index + timestamp + direction + kind + provider/model header row,
- `log_tool_call(script, result, script_path)` writes the script to `scripts/generated/<ts>_<seq:04d>.ps1` and appends a markdown record to the toolcalls log without the script body (just the file path + result); uses a `threading.Lock` for the sequence counter - `log_tool_call(script, result, script_path)` writes the script to `scripts/generated/<ts>_<seq:04d>.ps1` and appends a markdown record to the toolcalls log without the script body (just the file path + result); uses a `threading.Lock` for the sequence counter
- `close_session()` flushes and closes both file handles; called just before `dpg.destroy_context()` - `close_session()` flushes and closes both file handles; called just before `dpg.destroy_context()`
**Anthropic prompt caching & history management:** **Anthropic prompt caching:**
- System prompt + context are combined into one string, chunked into <=120k char blocks, and sent as the `system=` parameter array. Only the LAST chunk gets `cache_control: ephemeral`, so the entire system prefix is cached as one unit. - System prompt + context are combined into one string, chunked into <=120k char blocks, and sent as the `system=` parameter array. Only the LAST chunk gets `cache_control: ephemeral`, so the entire system prefix is cached as one unit.
- Last tool in `_ANTHROPIC_TOOLS` (`run_powershell`) has `cache_control: ephemeral`; this means the tools prefix is cached together with the system prefix after the first request. - Last tool in `_ANTHROPIC_TOOLS` (`run_powershell`) has `cache_control: ephemeral`; this means the tools prefix is cached together with the system prefix after the first request.
- The user message is sent as a plain `[{"type": "text", "text": user_message}]` block with NO cache_control. The context lives in `system=`, not in the first user message. - The user message is sent as a plain `[{"type": "text", "text": user_message}]` block with NO cache_control. The context lives in `system=`, not in the first user message.
- `_add_history_cache_breakpoint` places `cache_control:ephemeral` on the last content block of the second-to-last user message, using the 4th cache breakpoint to cache the conversation history prefix.
- `_trim_anthropic_history` uses token estimation (`_CHARS_PER_TOKEN = 3.5`) to keep the prompt under `_ANTHROPIC_MAX_PROMPT_TOKENS = 180_000`. It strips stale file refreshes from old turns, and drops oldest turn pairs if still over budget.
- The tools list is built once per session via `_get_anthropic_tools()` and reused across all API calls within the tool loop, avoiding redundant Python-side reconstruction. - The tools list is built once per session via `_get_anthropic_tools()` and reused across all API calls within the tool loop, avoiding redundant Python-side reconstruction.
- `_strip_cache_controls()` removes stale `cache_control` markers from all history entries before each API call, ensuring only the stable system/tools prefix consumes cache breakpoint slots. - `_strip_cache_controls()` removes stale `cache_control` markers from all history entries before each API call, ensuring only the stable system/tools prefix consumes cache breakpoint slots.
- Cache stats (creation tokens, read tokens) are surfaced in the comms log usage dict and displayed in the Comms History panel - Cache stats (creation tokens, read tokens) are surfaced in the comms log usage dict and displayed in the Comms History panel
@@ -182,15 +180,13 @@ Entry layout: index + timestamp + direction + kind + provider/model header row,
**MCP file tools (mcp_client.py + ai_client.py):** **MCP file tools (mcp_client.py + ai_client.py):**
- Four read-only tools exposed to the AI as native function/tool declarations: `read_file`, `list_directory`, `search_files`, `get_file_summary` - Four read-only tools exposed to the AI as native function/tool declarations: `read_file`, `list_directory`, `search_files`, `get_file_summary`
- Access control: `mcp_client.configure(file_items, extra_base_dirs)` is called before each send; builds an allowlist of resolved absolute paths from the project's `file_items` plus the `base_dir`; any path that is not explicitly in the list or not under one of the allowed directories returns `ACCESS DENIED` - Access control: `mcp_client.configure(file_items, extra_base_dirs)` is called before each send; builds an allowlist of resolved absolute paths from the project's `file_items` plus the `base_dir`; any path that is not explicitly in the list or not under one of the allowed directories returns `ACCESS DENIED`
- `mcp_client.dispatch(tool_name, tool_input)` is the single dispatch entry point used by both Anthropic and Gemini tool-use loops; `TOOL_NAMES` set now includes all six tool names - `mcp_client.dispatch(tool_name, tool_input)` is the single dispatch entry point used by both Anthropic and Gemini tool-use loops
- Anthropic: MCP tools appear before `run_powershell` in the tools list (no `cache_control` on them; only `run_powershell` carries `cache_control: ephemeral`) - Anthropic: MCP tools appear before `run_powershell` in the tools list (no `cache_control` on them; only `run_powershell` carries `cache_control: ephemeral`)
- Gemini: MCP tools are included in the `FunctionDeclaration` list alongside `run_powershell` - Gemini: MCP tools are included in the `FunctionDeclaration` list alongside `run_powershell`
- `get_file_summary` uses `summarize.summarise_file()` — same heuristic used for the initial `<context>` block, so the AI gets the same compact structural view it already knows - `get_file_summary` uses `summarize.summarise_file()` — same heuristic used for the initial `<context>` block, so the AI gets the same compact structural view it already knows
- `list_directory` sorts dirs before files; shows name, type, and size - `list_directory` sorts dirs before files; shows name, type, and size
- `search_files` uses `Path.glob()` with the caller-supplied pattern (supports `**/*.py` style) - `search_files` uses `Path.glob()` with the caller-supplied pattern (supports `**/*.py` style)
- `read_file` returns raw UTF-8 text; errors (not found, access denied, decode error) are returned as error strings rather than exceptions, so the AI sees them as tool results - `read_file` returns raw UTF-8 text; errors (not found, access denied, decode error) are returned as error strings rather than exceptions, so the AI sees them as tool results
- `web_search(query)` queries DuckDuckGo HTML endpoint and returns the top 5 results (title, URL, snippet) as a formatted string; uses a custom `_DDGParser` (HTMLParser subclass)
- `fetch_url(url)` fetches a URL, strips HTML tags/scripts via `_TextExtractor` (HTMLParser subclass), collapses whitespace, and truncates to 40k chars to prevent context blowup; handles DuckDuckGo redirect links automatically
- `summarize.py` heuristics: `.py` → AST imports + ALL_CAPS constants + classes+methods + top-level functions; `.toml` → table headers + top-level keys; `.md` → h1–h3 headings with indentation; all others → line count + first 8 lines preview - `summarize.py` heuristics: `.py` → AST imports + ALL_CAPS constants + classes+methods + top-level functions; `.toml` → table headers + top-level keys; `.md` → h1–h3 headings with indentation; all others → line count + first 8 lines preview
- Comms log: MCP tool calls log `OUT/tool_call` with `{"name": ..., "args": {...}}` and `IN/tool_result` with `{"name": ..., "output": ...}`; rendered in the Comms History panel via `_render_payload_tool_call` (shows each arg key/value) and `_render_payload_tool_result` (shows output) - Comms log: MCP tool calls log `OUT/tool_call` with `{"name": ..., "args": {...}}` and `IN/tool_result` with `{"name": ..., "output": ...}`; rendered in the Comms History panel via `_render_payload_tool_call` (shows each arg key/value) and `_render_payload_tool_result` (shows output)
@@ -203,9 +199,7 @@ Entry layout: index + timestamp + direction + kind + provider/model header row,
### Gemini Context Management ### Gemini Context Management
- Gemini uses explicit caching via `client.caches.create()` to store the `system_instruction` + tools as an immutable cached prefix with a 1-hour TTL. The cache is created once per chat session. - Gemini uses explicit caching via `client.caches.create()` to store the `system_instruction` + tools as an immutable cached prefix with a 1-hour TTL. The cache is created once per chat session.
- Proactively rebuilds cache at 90% of `_GEMINI_CACHE_TTL = 3600` to avoid stale-reference errors.
- When context changes (detected via `md_content` hash), the old cache is deleted, a new cache is created, and chat history is migrated to a fresh chat session pointing at the new cache. - When context changes (detected via `md_content` hash), the old cache is deleted, a new cache is created, and chat history is migrated to a fresh chat session pointing at the new cache.
- Trims history by dropping oldest pairs if input tokens exceed `_GEMINI_MAX_INPUT_TOKENS = 900_000`.
- If cache creation fails (e.g., content is under the minimum token threshold — 1024 for Flash, 4096 for Pro), the system falls back to inline `system_instruction` in the chat config. Implicit caching may still provide cost savings in this case. - If cache creation fails (e.g., content is under the minimum token threshold — 1024 for Flash, 4096 for Pro), the system falls back to inline `system_instruction` in the chat config. Implicit caching may still provide cost savings in this case.
- The `<context>` block lives inside `system_instruction`, NOT in user messages, preventing history bloat across turns. - The `<context>` block lives inside `system_instruction`, NOT in user messages, preventing history bloat across turns.
- On cleanup/exit, active caches are deleted via `ai_client.cleanup()` to prevent orphaned billing. - On cleanup/exit, active caches are deleted via `ai_client.cleanup()` to prevent orphaned billing.
@@ -251,33 +245,22 @@ Documentation has been completely rewritten matching the strict, structural form
- `docs/Readme.md`: The core interface manual. - `docs/Readme.md`: The core interface manual.
- `docs/guide_tools.md`: Security architecture for `_is_allowed` paths and definitions of the read-only vs destructive tool pipeline. - `docs/guide_tools.md`: Security architecture for `_is_allowed` paths and definitions of the read-only vs destructive tool pipeline.
## Branch Analysis: master vs not_sure (2026-02-22)
### Summary
The `not_sure` branch introduces a static/dynamic context split in the `send()` API signature, separating files+screenshots (cacheable, stable) from discussion history (changes every turn). This improves cache hit rates for both Anthropic and Gemini.
### Current master branch API correctness
- **Anthropic**: Correct. System blocks with cache_control, SDK content block serialisation, history repair, stale file refresh stripping all work properly.
- **Gemini**: Correct after `patch_gemini_history.py` was applied. Uses `_get_gemini_history_list()` for safe SDK access, drops history in pairs to maintain alternating roles, explicit caching via `caches.create()`.
## Updates (2026-02-22 — ai_client.py & aggregate.py) ### not_sure branch improvements
- **Anthropic**: Puts discussion history as a separate uncached system block after the cached static context. Better cache hit rates when discussion changes between turns.
- **Gemini**: Wraps discussion in `<discussion>` tags in user messages and strips old ones from history via regex. Prevents discussion duplication across turns.
### mcp_client.py — Web Tools Added ### not_sure branch bugs (not merged from master)
- `web_search(query)` and `fetch_url(url)` added as two new MCP tools alongside the existing four file tools. - Uses `_gemini_chat.history` directly instead of `_get_gemini_history_list()` — will crash on newer google-genai SDK versions where `.history` was removed.
- `TOOL_NAMES` set updated to include all six tool names for dispatch routing. - Missing the pair-wise history dropping fix (drops single messages, breaking Gemini's alternating role requirement).
- `MCP_TOOL_SPECS` list extended with full JSON schema definitions for both web tools.
- Both tools are declared in `_build_anthropic_tools()` and `_gemini_tool_declaration()` so they are available to both providers.
- Web tools bypass the `_is_allowed` path check (no filesystem access); file tools retain the allowlist enforcement.
### aggregate.py — run() double-I/O elimination ### Recommended merge path
- `run()` now calls `build_file_items()` once, then passes the result to `build_markdown_from_items()` instead of calling `build_files_section()` separately. This avoids reading every file twice per send. Cherry-pick the static/dynamic split from `not_sure` into `master` while keeping master's SDK safety fixes (`_get_gemini_history_list`, pair-wise dropping, `_content_block_to_dict`).
- `build_markdown_from_items()` accepts a `summary_only` flag (default `False`); when `False` it inlines full file content; when `True` it delegates to `summarize.build_summary_markdown()` for compact structural summaries.
- `run()` returns a 3-tuple `(markdown_str, output_path, file_items)` — the `file_items` list is passed through to `gui.py` as `self.last_file_items` for dynamic context refresh after tool calls.
## Updates (2026-02-22 — gui.py [+ Maximize] bug fix)
### Problem
Three `[+ Maximize]` buttons were reading their text content via `dpg.get_value(tag)` at click time:
1. `ConfirmDialog.show()` — passed `f"{self._tag}_script"` as `user_data` and called `dpg.get_value(u)` in the lambda. If the dialog was dismissed before the viewer opened, the item no longer existed and the call would fail silently or crash.
2. `win_script_output` Script `[+ Maximize]` — used `user_data="last_script_text"` and `dpg.get_value(u)`. When word-wrap is ON, `last_script_text` is hidden (`show=False`); in some DPG versions `dpg.get_value` on a hidden `input_text` returns `""`.
3. `win_script_output` Output `[+ Maximize]` — same issue with `"last_script_output"`.
### Fix
- `ConfirmDialog.show()`: changed `user_data` to `self._script` (the actual text string captured at button-creation time) and the callback to `lambda s, a, u: _show_text_viewer("Confirm Script", u)`. The text is now baked in at dialog construction, not read from a potentially-deleted widget.
- `App._append_tool_log()`: added `self._last_script = script` and `self._last_output = result` assignments so the latest values are always available as instance state.
- `win_script_output` buttons: both `[+ Maximize]` buttons now use `lambda s, a, u: _show_text_viewer("...", self._last_script/output)` directly, bypassing DPG widget state entirely.
+21 -39
View File
@@ -98,28 +98,24 @@ def build_file_items(base_dir: Path, files: list[str]) -> list[dict]:
entry : str (original config entry string) entry : str (original config entry string)
content : str (file text, or error string) content : str (file text, or error string)
error : bool error : bool
mtime : float (last modification time, for skip-if-unchanged optimization)
""" """
items = [] items = []
for entry in files: for entry in files:
paths = resolve_paths(base_dir, entry) paths = resolve_paths(base_dir, entry)
if not paths: if not paths:
items.append({"path": None, "entry": entry, "content": f"ERROR: no files matched: {entry}", "error": True, "mtime": 0.0}) items.append({"path": None, "entry": entry, "content": f"ERROR: no files matched: {entry}", "error": True})
continue continue
for path in paths: for path in paths:
try: try:
content = path.read_text(encoding="utf-8") content = path.read_text(encoding="utf-8")
mtime = path.stat().st_mtime
error = False error = False
except FileNotFoundError: except FileNotFoundError:
content = f"ERROR: file not found: {path}" content = f"ERROR: file not found: {path}"
mtime = 0.0
error = True error = True
except Exception as e: except Exception as e:
content = f"ERROR: {e}" content = f"ERROR: {e}"
mtime = 0.0
error = True error = True
items.append({"path": path, "entry": entry, "content": content, "error": error, "mtime": mtime}) items.append({"path": path, "entry": entry, "content": content, "error": error})
return items return items
def build_summary_section(base_dir: Path, files: list[str]) -> str: def build_summary_section(base_dir: Path, files: list[str]) -> str:
@@ -130,39 +126,23 @@ def build_summary_section(base_dir: Path, files: list[str]) -> str:
items = build_file_items(base_dir, files) items = build_file_items(base_dir, files)
return summarize.build_summary_markdown(items) return summarize.build_summary_markdown(items)
def _build_files_section_from_items(file_items: list[dict]) -> str: def build_static_markdown(base_dir: Path, files: list[str], screenshot_base_dir: Path, screenshots: list[str], summary_only: bool = False) -> str:
"""Build the files markdown section from pre-read file items (avoids double I/O).""" """Build the static (cacheable) portion of the context: files + screenshots."""
sections = []
for item in file_items:
path = item.get("path")
entry = item.get("entry", "unknown")
content = item.get("content", "")
if path is None:
sections.append(f"### `{entry}`\n\n```text\n{content}\n```")
continue
suffix = path.suffix.lstrip(".") if hasattr(path, "suffix") else "text"
lang = suffix if suffix else "text"
original = entry if "*" not in entry else str(path)
sections.append(f"### `{original}`\n\n```{lang}\n{content}\n```")
return "\n\n---\n\n".join(sections)
def build_markdown_from_items(file_items: list[dict], screenshot_base_dir: Path, screenshots: list[str], history: list[str], summary_only: bool = False) -> str:
"""Build markdown from pre-read file items instead of re-reading from disk."""
parts = [] parts = []
# STATIC PREFIX: Files and Screenshots must go first to maximize Cache Hits if files:
if file_items:
if summary_only: if summary_only:
parts.append("## Files (Summary)\n\n" + summarize.build_summary_markdown(file_items)) parts.append("## Files (Summary)\n\n" + build_summary_section(base_dir, files))
else: else:
parts.append("## Files\n\n" + _build_files_section_from_items(file_items)) parts.append("## Files\n\n" + build_files_section(base_dir, files))
if screenshots: if screenshots:
parts.append("## Screenshots\n\n" + build_screenshots_section(screenshot_base_dir, screenshots)) parts.append("## Screenshots\n\n" + build_screenshots_section(screenshot_base_dir, screenshots))
# DYNAMIC SUFFIX: History changes every turn, must go last
if history:
parts.append("## Discussion History\n\n" + build_discussion_section(history))
return "\n\n---\n\n".join(parts) return "\n\n---\n\n".join(parts)
def build_dynamic_markdown(history: list[str]) -> str:
"""Build the dynamic (changes every turn) portion: discussion history."""
if history:
return "## Discussion History\n\n" + build_discussion_section(history)
return ""
def build_markdown(base_dir: Path, files: list[str], screenshot_base_dir: Path, screenshots: list[str], history: list[str], summary_only: bool = False) -> str: def build_markdown(base_dir: Path, files: list[str], screenshot_base_dir: Path, screenshots: list[str], history: list[str], summary_only: bool = False) -> str:
parts = [] parts = []
@@ -179,7 +159,7 @@ def build_markdown(base_dir: Path, files: list[str], screenshot_base_dir: Path,
parts.append("## Discussion History\n\n" + build_discussion_section(history)) parts.append("## Discussion History\n\n" + build_discussion_section(history))
return "\n\n---\n\n".join(parts) return "\n\n---\n\n".join(parts)
def run(config: dict) -> tuple[str, Path, list[dict]]: def run(config: dict) -> tuple[str, Path, list]:
namespace = config.get("project", {}).get("name") namespace = config.get("project", {}).get("name")
if not namespace: if not namespace:
namespace = config.get("output", {}).get("namespace", "project") namespace = config.get("output", {}).get("namespace", "project")
@@ -193,18 +173,20 @@ def run(config: dict) -> tuple[str, Path, list[dict]]:
output_dir.mkdir(parents=True, exist_ok=True) output_dir.mkdir(parents=True, exist_ok=True)
increment = find_next_increment(output_dir, namespace) increment = find_next_increment(output_dir, namespace)
output_file = output_dir / f"{namespace}_{increment:03d}.md" output_file = output_dir / f"{namespace}_{increment:03d}.md"
# Build file items once, then construct markdown from them (avoids double I/O) # Build static (files+screenshots) and dynamic (discussion) portions separately for better caching
file_items = build_file_items(base_dir, files) static_md = build_static_markdown(base_dir, files, screenshot_base_dir, screenshots, summary_only=False)
markdown = build_markdown_from_items(file_items, screenshot_base_dir, screenshots, history, dynamic_md = build_dynamic_markdown(history)
summary_only=False) # Write combined markdown to disk for archival
markdown = f"{static_md}\n\n---\n\n{dynamic_md}" if static_md and dynamic_md else static_md or dynamic_md
output_file.write_text(markdown, encoding="utf-8") output_file.write_text(markdown, encoding="utf-8")
return markdown, output_file, file_items file_items = build_file_items(base_dir, files)
return static_md, dynamic_md, output_file, file_items
def main(): def main():
with open("config.toml", "rb") as f: with open("config.toml", "rb") as f:
import tomllib import tomllib
config = tomllib.load(f) config = tomllib.load(f)
markdown, output_file, _ = run(config) static_md, dynamic_md, output_file, _ = run(config)
print(f"Written: {output_file}") print(f"Written: {output_file}")
if __name__ == "__main__": if __name__ == "__main__":
+115 -213
View File
@@ -13,7 +13,6 @@ during chat creation to avoid massive history bloat.
# ai_client.py # ai_client.py
import tomllib import tomllib
import json import json
import time
import datetime import datetime
from pathlib import Path from pathlib import Path
import file_cache import file_cache
@@ -35,12 +34,6 @@ def set_model_params(temp: float, max_tok: int, trunc_limit: int = 8000):
_gemini_client = None _gemini_client = None
_gemini_chat = None _gemini_chat = None
_gemini_cache = None _gemini_cache = None
_gemini_cache_md_hash: int | None = None
_gemini_cache_created_at: float | None = None
# Gemini cache TTL in seconds. Caches are created with this TTL and
# proactively rebuilt at 90% of this value to avoid stale-reference errors.
_GEMINI_CACHE_TTL = 3600
_anthropic_client = None _anthropic_client = None
_anthropic_history: list[dict] = [] _anthropic_history: list[dict] = []
@@ -223,7 +216,6 @@ def cleanup():
def reset_session(): def reset_session():
global _gemini_client, _gemini_chat, _gemini_cache global _gemini_client, _gemini_chat, _gemini_cache
global _gemini_cache_md_hash, _gemini_cache_created_at
global _anthropic_client, _anthropic_history global _anthropic_client, _anthropic_history
global _CACHED_ANTHROPIC_TOOLS global _CACHED_ANTHROPIC_TOOLS
if _gemini_client and _gemini_cache: if _gemini_client and _gemini_cache:
@@ -234,8 +226,6 @@ def reset_session():
_gemini_client = None _gemini_client = None
_gemini_chat = None _gemini_chat = None
_gemini_cache = None _gemini_cache = None
_gemini_cache_md_hash = None
_gemini_cache_created_at = None
_anthropic_client = None _anthropic_client = None
_anthropic_history = [] _anthropic_history = []
_CACHED_ANTHROPIC_TOOLS = None _CACHED_ANTHROPIC_TOOLS = None
@@ -286,53 +276,37 @@ def _list_anthropic_models() -> list[str]:
TOOL_NAME = "run_powershell" TOOL_NAME = "run_powershell"
_agent_tools: dict = {}
def set_agent_tools(tools: dict):
global _agent_tools, _CACHED_ANTHROPIC_TOOLS
_agent_tools = tools
_CACHED_ANTHROPIC_TOOLS = None
def _build_anthropic_tools() -> list[dict]: def _build_anthropic_tools() -> list[dict]:
"""Build the full Anthropic tools list: run_powershell + MCP file tools.""" """Build the full Anthropic tools list: run_powershell + MCP file tools."""
mcp_tools = [] mcp_tools = []
for spec in mcp_client.MCP_TOOL_SPECS: for spec in mcp_client.MCP_TOOL_SPECS:
if _agent_tools.get(spec["name"], True): mcp_tools.append({
mcp_tools.append({ "name": spec["name"],
"name": spec["name"], "description": spec["description"],
"description": spec["description"], "input_schema": spec["parameters"],
"input_schema": spec["parameters"], })
}) powershell_tool = {
"name": TOOL_NAME,
tools_list = mcp_tools "description": (
if _agent_tools.get(TOOL_NAME, True): "Run a PowerShell script within the project base_dir. "
powershell_tool = { "Use this to create, edit, rename, or delete files and directories. "
"name": TOOL_NAME, "The working directory is set to base_dir automatically. "
"description": ( "Always prefer targeted edits over full rewrites where possible. "
"Run a PowerShell script within the project base_dir. " "stdout and stderr are returned to you as the result."
"Use this to create, edit, rename, or delete files and directories. " ),
"The working directory is set to base_dir automatically. " "input_schema": {
"Always prefer targeted edits over full rewrites where possible. " "type": "object",
"stdout and stderr are returned to you as the result." "properties": {
), "script": {
"input_schema": { "type": "string",
"type": "object", "description": "The PowerShell script to execute."
"properties": { }
"script": {
"type": "string",
"description": "The PowerShell script to execute."
}
},
"required": ["script"]
}, },
"cache_control": {"type": "ephemeral"}, "required": ["script"]
} },
tools_list.append(powershell_tool) "cache_control": {"type": "ephemeral"},
elif tools_list: }
# Anthropic requires the LAST tool to have cache_control for the prefix caching to work optimally on tools return mcp_tools + [powershell_tool]
tools_list[-1]["cache_control"] = {"type": "ephemeral"}
return tools_list
_ANTHROPIC_TOOLS = _build_anthropic_tools() _ANTHROPIC_TOOLS = _build_anthropic_tools()
@@ -354,8 +328,6 @@ def _gemini_tool_declaration():
# MCP file tools # MCP file tools
for spec in mcp_client.MCP_TOOL_SPECS: for spec in mcp_client.MCP_TOOL_SPECS:
if not _agent_tools.get(spec["name"], True):
continue
props = {} props = {}
for pname, pdef in spec["parameters"].get("properties", {}).items(): for pname, pdef in spec["parameters"].get("properties", {}).items():
props[pname] = types.Schema( props[pname] = types.Schema(
@@ -373,28 +345,27 @@ def _gemini_tool_declaration():
)) ))
# PowerShell tool # PowerShell tool
if _agent_tools.get(TOOL_NAME, True): declarations.append(types.FunctionDeclaration(
declarations.append(types.FunctionDeclaration( name=TOOL_NAME,
name=TOOL_NAME, description=(
description=( "Run a PowerShell script within the project base_dir. "
"Run a PowerShell script within the project base_dir. " "Use this to create, edit, rename, or delete files and directories. "
"Use this to create, edit, rename, or delete files and directories. " "The working directory is set to base_dir automatically. "
"The working directory is set to base_dir automatically. " "stdout and stderr are returned to you as the result."
"stdout and stderr are returned to you as the result." ),
), parameters=types.Schema(
parameters=types.Schema( type=types.Type.OBJECT,
type=types.Type.OBJECT, properties={
properties={ "script": types.Schema(
"script": types.Schema( type=types.Type.STRING,
type=types.Type.STRING, description="The PowerShell script to execute."
description="The PowerShell script to execute." )
) },
}, required=["script"]
required=["script"] ),
), ))
))
return types.Tool(function_declarations=declarations) if declarations else None return types.Tool(function_declarations=declarations)
def _run_script(script: str, base_dir: str) -> str: def _run_script(script: str, base_dir: str) -> str:
@@ -412,15 +383,12 @@ def _run_script(script: str, base_dir: str) -> str:
# ------------------------------------------------------------------ dynamic file context refresh # ------------------------------------------------------------------ dynamic file context refresh
def _reread_file_items(file_items: list[dict]) -> tuple[list[dict], list[dict]]: def _reread_file_items(file_items: list[dict]) -> list[dict]:
""" """
Re-read file_items from disk, but only files whose mtime has changed. Re-read every file in file_items from disk, returning a fresh list.
Returns (all_items, changed_items) — all_items is the full refreshed list, This is called after tool calls so the AI sees updated file contents.
changed_items contains only the files that were actually modified since
the last read (used to build a minimal [FILES UPDATED] block).
""" """
refreshed = [] refreshed = []
changed = []
for item in file_items: for item in file_items:
path = item.get("path") path = item.get("path")
if path is None: if path is None:
@@ -429,20 +397,11 @@ def _reread_file_items(file_items: list[dict]) -> tuple[list[dict], list[dict]]:
from pathlib import Path as _P from pathlib import Path as _P
p = _P(path) if not isinstance(path, _P) else path p = _P(path) if not isinstance(path, _P) else path
try: try:
current_mtime = p.stat().st_mtime
prev_mtime = item.get("mtime", 0.0)
if current_mtime == prev_mtime:
refreshed.append(item) # unchanged — skip re-read
continue
content = p.read_text(encoding="utf-8") content = p.read_text(encoding="utf-8")
new_item = {**item, "content": content, "error": False, "mtime": current_mtime} refreshed.append({**item, "content": content, "error": False})
refreshed.append(new_item)
changed.append(new_item)
except Exception as e: except Exception as e:
err_item = {**item, "content": f"ERROR re-reading {p}: {e}", "error": True, "mtime": 0.0} refreshed.append({**item, "content": f"ERROR re-reading {p}: {e}", "error": True})
refreshed.append(err_item) return refreshed
changed.append(err_item)
return refreshed, changed
def _build_file_context_text(file_items: list[dict]) -> str: def _build_file_context_text(file_items: list[dict]) -> str:
@@ -506,40 +465,26 @@ def _get_gemini_history_list(chat):
return chat.get_history() return chat.get_history()
return [] return []
def _send_gemini(md_content: str, user_message: str, base_dir: str, file_items: list[dict] | None = None) -> str: def _send_gemini(static_md: str, dynamic_md: str, user_message: str, base_dir: str, file_items: list[dict] | None = None) -> str:
global _gemini_chat, _gemini_cache, _gemini_cache_md_hash, _gemini_cache_created_at global _gemini_chat, _gemini_cache
from google.genai import types from google.genai import types
try: try:
_ensure_gemini_client(); mcp_client.configure(file_items or [], [base_dir]) _ensure_gemini_client(); mcp_client.configure(file_items or [], [base_dir])
sys_instr = f"{_get_combined_system_prompt()}\n\n<context>\n{md_content}\n</context>" sys_instr = f"{_get_combined_system_prompt()}\n\n<context>\n{static_md}\n</context>"
tools_decl = [_gemini_tool_declaration()] tools_decl = [_gemini_tool_declaration()]
# DYNAMIC CONTEXT: Check if files/context changed mid-session # DYNAMIC CONTEXT: Check if files/context changed mid-session
current_md_hash = hash(md_content) current_md_hash = hash(static_md)
old_history = None old_history = None
if _gemini_chat and _gemini_cache_md_hash != current_md_hash: if _gemini_chat and getattr(_gemini_chat, "_last_md_hash", None) != current_md_hash:
old_history = list(_get_gemini_history_list(_gemini_chat)) if _get_gemini_history_list(_gemini_chat) else [] old_history = list(_get_gemini_history_list(_gemini_chat)) if _get_gemini_history_list(_gemini_chat) else []
if _gemini_cache: if _gemini_cache:
try: _gemini_client.caches.delete(name=_gemini_cache.name) try: _gemini_client.caches.delete(name=_gemini_cache.name)
except: pass except: pass
_gemini_chat = None _gemini_chat = None
_gemini_cache = None _gemini_cache = None
_gemini_cache_created_at = None
_append_comms("OUT", "request", {"message": "[CONTEXT CHANGED] Rebuilding cache and chat session..."}) _append_comms("OUT", "request", {"message": "[CONTEXT CHANGED] Rebuilding cache and chat session..."})
# CACHE TTL: Proactively rebuild before the cache expires server-side.
# If we don't, send_message() will reference a deleted cache and fail.
if _gemini_chat and _gemini_cache and _gemini_cache_created_at:
elapsed = time.time() - _gemini_cache_created_at
if elapsed > _GEMINI_CACHE_TTL * 0.9:
old_history = list(_get_gemini_history_list(_gemini_chat)) if _get_gemini_history_list(_gemini_chat) else []
try: _gemini_client.caches.delete(name=_gemini_cache.name)
except: pass
_gemini_chat = None
_gemini_cache = None
_gemini_cache_created_at = None
_append_comms("OUT", "request", {"message": f"[CACHE TTL] Rebuilding cache (expired after {int(elapsed)}s)..."})
if not _gemini_chat: if not _gemini_chat:
chat_config = types.GenerateContentConfig( chat_config = types.GenerateContentConfig(
system_instruction=sys_instr, system_instruction=sys_instr,
@@ -555,10 +500,9 @@ def _send_gemini(md_content: str, user_message: str, base_dir: str, file_items:
config=types.CreateCachedContentConfig( config=types.CreateCachedContentConfig(
system_instruction=sys_instr, system_instruction=sys_instr,
tools=tools_decl, tools=tools_decl,
ttl=f"{_GEMINI_CACHE_TTL}s", ttl="3600s",
) )
) )
_gemini_cache_created_at = time.time()
chat_config = types.GenerateContentConfig( chat_config = types.GenerateContentConfig(
cached_content=_gemini_cache.name, cached_content=_gemini_cache.name,
temperature=_temperature, temperature=_temperature,
@@ -567,38 +511,45 @@ def _send_gemini(md_content: str, user_message: str, base_dir: str, file_items:
) )
_append_comms("OUT", "request", {"message": f"[CACHE CREATED] {_gemini_cache.name}"}) _append_comms("OUT", "request", {"message": f"[CACHE CREATED] {_gemini_cache.name}"})
except Exception as e: except Exception as e:
_gemini_cache = None _gemini_cache = None # Ensure clean state on failure
_gemini_cache_created_at = None
_append_comms("OUT", "request", {"message": f"[CACHE FAILED] {type(e).__name__}: {e} — falling back to inline system_instruction"})
kwargs = {"model": _model, "config": chat_config} kwargs = {"model": _model, "config": chat_config}
if old_history: if old_history:
kwargs["history"] = old_history kwargs["history"] = old_history
_gemini_chat = _gemini_client.chats.create(**kwargs) _gemini_chat = _gemini_client.chats.create(**kwargs)
_gemini_cache_md_hash = current_md_hash _gemini_chat._last_md_hash = current_md_hash
# Build user message: prepend dynamic context (discussion) so it's NOT cached in system_instruction
full_user_msg = f"<discussion>\n{dynamic_md}\n</discussion>\n\n{user_message}" if dynamic_md.strip() else user_message
_append_comms("OUT", "request", {"message": f"[ctx {len(static_md)} static + {len(dynamic_md)} dynamic + msg {len(user_message)}]"})
payload, all_text = full_user_msg, []
_append_comms("OUT", "request", {"message": f"[ctx {len(md_content)} + msg {len(user_message)}]"})
payload, all_text = user_message, []
# Strip stale file refreshes and truncate old tool outputs ONCE before
# entering the tool loop (not per-round — history entries don't change).
if _gemini_chat and _get_gemini_history_list(_gemini_chat):
for msg in _get_gemini_history_list(_gemini_chat):
if msg.role == "user" and hasattr(msg, "parts"):
for p in msg.parts:
if hasattr(p, "function_response") and p.function_response and hasattr(p.function_response, "response"):
r = p.function_response.response
if isinstance(r, dict) and "output" in r:
val = r["output"]
if isinstance(val, str):
if "[SYSTEM: FILES UPDATED]" in val:
val = val.split("[SYSTEM: FILES UPDATED]")[0].strip()
if _history_trunc_limit > 0 and len(val) > _history_trunc_limit:
val = val[:_history_trunc_limit] + "\n\n... [TRUNCATED BY SYSTEM TO SAVE TOKENS.]"
r["output"] = val
for r_idx in range(MAX_TOOL_ROUNDS + 2): for r_idx in range(MAX_TOOL_ROUNDS + 2):
# Strip stale <discussion> blocks from old user messages so they don't accumulate
import re as _re
if _gemini_chat and _get_gemini_history_list(_gemini_chat):
for msg in _get_gemini_history_list(_gemini_chat):
if msg.role == "user" and hasattr(msg, "parts"):
for p in msg.parts:
if hasattr(p, "text") and p.text and "<discussion>" in p.text:
p.text = _re.sub(r"<discussion>.*?</discussion>\n\n", "", p.text, flags=_re.DOTALL)
# Strip stale file refreshes and truncate old tool outputs in Gemini history
if _gemini_chat and _get_gemini_history_list(_gemini_chat):
for msg in _get_gemini_history_list(_gemini_chat):
if msg.role == "user" and hasattr(msg, "parts"):
for p in msg.parts:
if hasattr(p, "function_response") and p.function_response and hasattr(p.function_response, "response"):
r = p.function_response.response
if isinstance(r, dict) and "output" in r:
val = r["output"]
if isinstance(val, str):
if "[SYSTEM: FILES UPDATED]" in val:
val = val.split("[SYSTEM: FILES UPDATED]")[0].strip()
if _history_trunc_limit > 0 and len(val) > _history_trunc_limit:
val = val[:_history_trunc_limit] + "\n\n... [TRUNCATED BY SYSTEM TO SAVE TOKENS.]"
r["output"] = val
resp = _gemini_chat.send_message(payload) resp = _gemini_chat.send_message(payload)
txt = "\n".join(p.text for c in resp.candidates if getattr(c, "content", None) for p in c.content.parts if hasattr(p, "text") and p.text) txt = "\n".join(p.text for c in resp.candidates if getattr(c, "content", None) for p in c.content.parts if hasattr(p, "text") and p.text)
if txt: all_text.append(txt) if txt: all_text.append(txt)
@@ -652,12 +603,12 @@ def _send_gemini(md_content: str, user_message: str, base_dir: str, file_items:
if i == len(calls) - 1: if i == len(calls) - 1:
if file_items: if file_items:
file_items, changed = _reread_file_items(file_items) file_items = _reread_file_items(file_items)
ctx = _build_file_context_text(changed) ctx = _build_file_context_text(file_items)
if ctx: if ctx:
out += f"\n\n[SYSTEM: FILES UPDATED]\n\n{ctx}" out += f"\n\n[SYSTEM: FILES UPDATED]\n\n{ctx}"
if r_idx == MAX_TOOL_ROUNDS: out += "\n\n[SYSTEM: MAX ROUNDS. PROVIDE FINAL ANSWER.]" if r_idx == MAX_TOOL_ROUNDS: out += "\n\n[SYSTEM: MAX ROUNDS. PROVIDE FINAL ANSWER.]"
f_resps.append(types.Part.from_function_response(name=name, response={"output": out})) f_resps.append(types.Part.from_function_response(name=name, response={"output": out}))
log.append({"tool_use_id": name, "content": out}) log.append({"tool_use_id": name, "content": out})
@@ -689,15 +640,7 @@ _FILE_REFRESH_MARKER = "[FILES UPDATED"
def _estimate_message_tokens(msg: dict) -> int: def _estimate_message_tokens(msg: dict) -> int:
""" """Rough token estimate for a single Anthropic message dict."""
Rough token estimate for a single Anthropic message dict.
Caches the result on the dict as '_est_tokens' so repeated calls
(e.g., from _trim_anthropic_history) don't re-scan unchanged messages.
Call _invalidate_token_estimate() when a message's content is modified.
"""
cached = msg.get("_est_tokens")
if cached is not None:
return cached
total_chars = 0 total_chars = 0
content = msg.get("content", "") content = msg.get("content", "")
if isinstance(content, str): if isinstance(content, str):
@@ -715,14 +658,7 @@ def _estimate_message_tokens(msg: dict) -> int:
total_chars += len(_json.dumps(inp, ensure_ascii=False)) total_chars += len(_json.dumps(inp, ensure_ascii=False))
elif isinstance(block, str): elif isinstance(block, str):
total_chars += len(block) total_chars += len(block)
est = max(1, int(total_chars / _CHARS_PER_TOKEN)) return max(1, int(total_chars / _CHARS_PER_TOKEN))
msg["_est_tokens"] = est
return est
def _invalidate_token_estimate(msg: dict):
"""Remove the cached token estimate so the next call recalculates."""
msg.pop("_est_tokens", None)
def _estimate_prompt_tokens(system_blocks: list[dict], history: list[dict]) -> int: def _estimate_prompt_tokens(system_blocks: list[dict], history: list[dict]) -> int:
@@ -734,7 +670,7 @@ def _estimate_prompt_tokens(system_blocks: list[dict], history: list[dict]) -> i
total += max(1, int(len(text) / _CHARS_PER_TOKEN)) total += max(1, int(len(text) / _CHARS_PER_TOKEN))
# Tool definitions (rough fixed estimate — they're ~2k tokens for our set) # Tool definitions (rough fixed estimate — they're ~2k tokens for our set)
total += 2500 total += 2500
# History messages (uses cached estimates for unchanged messages) # History messages
for msg in history: for msg in history:
total += _estimate_message_tokens(msg) total += _estimate_message_tokens(msg)
return total return total
@@ -769,7 +705,6 @@ def _strip_stale_file_refreshes(history: list[dict]):
cleaned.append(block) cleaned.append(block)
if len(cleaned) < len(content): if len(cleaned) < len(content):
msg["content"] = cleaned msg["content"] = cleaned
_invalidate_token_estimate(msg)
def _trim_anthropic_history(system_blocks: list[dict], history: list[dict]): def _trim_anthropic_history(system_blocks: list[dict], history: list[dict]):
@@ -861,28 +796,6 @@ def _strip_cache_controls(history: list[dict]):
if isinstance(block, dict): if isinstance(block, dict):
block.pop("cache_control", None) block.pop("cache_control", None)
def _add_history_cache_breakpoint(history: list[dict]):
"""
Place cache_control:ephemeral on the last content block of the
second-to-last user message. This uses one of the 4 allowed Anthropic
cache breakpoints to cache the conversation prefix so the full history
isn't reprocessed on every request.
"""
user_indices = [i for i, m in enumerate(history) if m.get("role") == "user"]
if len(user_indices) < 2:
return # Only one user message (the current turn) — nothing stable to cache
target_idx = user_indices[-2]
content = history[target_idx].get("content")
if isinstance(content, list) and content:
last_block = content[-1]
if isinstance(last_block, dict):
last_block["cache_control"] = {"type": "ephemeral"}
elif isinstance(content, str):
history[target_idx]["content"] = [
{"type": "text", "text": content, "cache_control": {"type": "ephemeral"}}
]
def _repair_anthropic_history(history: list[dict]): def _repair_anthropic_history(history: list[dict]):
""" """
If history ends with an assistant message that contains tool_use blocks If history ends with an assistant message that contains tool_use blocks
@@ -915,46 +828,36 @@ def _repair_anthropic_history(history: list[dict]):
}) })
def _send_anthropic(md_content: str, user_message: str, base_dir: str, file_items: list[dict] | None = None) -> str: def _send_anthropic(static_md: str, dynamic_md: str, user_message: str, base_dir: str, file_items: list[dict] | None = None) -> str:
try: try:
_ensure_anthropic_client() _ensure_anthropic_client()
mcp_client.configure(file_items or [], [base_dir]) mcp_client.configure(file_items or [], [base_dir])
# Split system into two cache breakpoints: system_text = _get_combined_system_prompt() + f"\n\n<context>\n{static_md}\n</context>"
# 1. Stable system prompt (never changes — always a cache hit) system_blocks = _build_chunked_context_blocks(system_text)
# 2. Dynamic file context (invalidated only when files change) # Dynamic context (discussion history) goes after the cached static prefix, without cache_control
stable_prompt = _get_combined_system_prompt() if dynamic_md.strip():
stable_blocks = [{"type": "text", "text": stable_prompt, "cache_control": {"type": "ephemeral"}}] system_blocks.append({"type": "text", "text": f"<discussion>\n{dynamic_md}\n</discussion>"})
context_text = f"\n\n<context>\n{md_content}\n</context>"
context_blocks = _build_chunked_context_blocks(context_text)
system_blocks = stable_blocks + context_blocks
user_content = [{"type": "text", "text": user_message}] user_content = [{"type": "text", "text": user_message}]
# COMPRESS HISTORY: Truncate massive tool outputs from previous turns # COMPRESS HISTORY: Truncate massive tool outputs from previous turns
for msg in _anthropic_history: for msg in _anthropic_history:
if msg.get("role") == "user" and isinstance(msg.get("content"), list): if msg.get("role") == "user" and isinstance(msg.get("content"), list):
modified = False
for block in msg["content"]: for block in msg["content"]:
if isinstance(block, dict) and block.get("type") == "tool_result": if isinstance(block, dict) and block.get("type") == "tool_result":
t_content = block.get("content", "") t_content = block.get("content", "")
if _history_trunc_limit > 0 and isinstance(t_content, str) and len(t_content) > _history_trunc_limit: if _history_trunc_limit > 0 and isinstance(t_content, str) and len(t_content) > _history_trunc_limit:
block["content"] = t_content[:_history_trunc_limit] + "\n\n... [TRUNCATED BY SYSTEM TO SAVE TOKENS. Original output was too large.]" block["content"] = t_content[:_history_trunc_limit] + "\n\n... [TRUNCATED BY SYSTEM TO SAVE TOKENS. Original output was too large.]"
modified = True
if modified:
_invalidate_token_estimate(msg)
_strip_cache_controls(_anthropic_history) _strip_cache_controls(_anthropic_history)
_repair_anthropic_history(_anthropic_history) _repair_anthropic_history(_anthropic_history)
_anthropic_history.append({"role": "user", "content": user_content}) _anthropic_history.append({"role": "user", "content": user_content})
# Use the 4th cache breakpoint to cache the conversation history prefix.
# This is placed on the second-to-last user message (the last stable one).
_add_history_cache_breakpoint(_anthropic_history)
n_chunks = len(system_blocks) n_chunks = len(system_blocks)
_append_comms("OUT", "request", { _append_comms("OUT", "request", {
"message": ( "message": (
f"[system {n_chunks} chunk(s), {len(md_content)} chars context] " f"[system {n_chunks} chunk(s), {len(static_md)} static + {len(dynamic_md)} dynamic chars] "
f"{user_message[:200]}{'...' if len(user_message) > 200 else ''}" f"{user_message[:200]}{'...' if len(user_message) > 200 else ''}"
), ),
}) })
@@ -974,16 +877,13 @@ def _send_anthropic(md_content: str, user_message: str, base_dir: str, file_item
), ),
}) })
def _strip_private_keys(history):
return [{k: v for k, v in m.items() if not k.startswith("_")} for m in history]
response = _anthropic_client.messages.create( response = _anthropic_client.messages.create(
model=_model, model=_model,
max_tokens=_max_tokens, max_tokens=_max_tokens,
temperature=_temperature, temperature=_temperature,
system=system_blocks, system=system_blocks,
tools=_get_anthropic_tools(), tools=_get_anthropic_tools(),
messages=_strip_private_keys(_anthropic_history), messages=_anthropic_history,
) )
# Convert SDK content block objects to plain dicts before storing in history # Convert SDK content block objects to plain dicts before storing in history
@@ -1066,10 +966,10 @@ def _send_anthropic(md_content: str, user_message: str, base_dir: str, file_item
"content": output, "content": output,
}) })
# Refresh file context after tool calls — only inject CHANGED files # Refresh file context after tool calls and inject into tool result message
if file_items: if file_items:
file_items, changed = _reread_file_items(file_items) file_items = _reread_file_items(file_items)
refreshed_ctx = _build_file_context_text(changed) refreshed_ctx = _build_file_context_text(file_items)
if refreshed_ctx: if refreshed_ctx:
tool_results.append({ tool_results.append({
"type": "text", "type": "text",
@@ -1110,7 +1010,8 @@ def _send_anthropic(md_content: str, user_message: str, base_dir: str, file_item
# ------------------------------------------------------------------ unified send # ------------------------------------------------------------------ unified send
def send( def send(
md_content: str, static_md: str,
dynamic_md: str,
user_message: str, user_message: str,
base_dir: str = ".", base_dir: str = ".",
file_items: list[dict] | None = None, file_items: list[dict] | None = None,
@@ -1118,14 +1019,15 @@ def send(
""" """
Send a message to the active provider. Send a message to the active provider.
md_content : aggregated markdown string from aggregate.run() static_md : cacheable context (files + screenshots) from aggregate.run()
dynamic_md : volatile context (discussion history) that changes every turn
user_message: the user question / instruction user_message: the user question / instruction
base_dir : project base directory (for PowerShell tool calls) base_dir : project base directory (for PowerShell tool calls)
file_items : list of file dicts from aggregate.build_file_items() for file_items : list of file dicts from aggregate.build_file_items() for
dynamic context refresh after tool calls dynamic context refresh after tool calls
""" """
if _provider == "gemini": if _provider == "gemini":
return _send_gemini(md_content, user_message, base_dir, file_items) return _send_gemini(static_md, dynamic_md, user_message, base_dir, file_items)
elif _provider == "anthropic": elif _provider == "anthropic":
return _send_anthropic(md_content, user_message, base_dir, file_items) return _send_anthropic(static_md, dynamic_md, user_message, base_dir, file_items)
raise ValueError(f"unknown provider: {_provider}") raise ValueError(f"unknown provider: {_provider}")
-37
View File
@@ -1,37 +0,0 @@
# Google Python Style Guide Summary
This document summarizes key rules and best practices from the Google Python Style Guide.
## 1. Python Language Rules
- **Linting:** Run `pylint` on your code to catch bugs and style issues.
- **Imports:** Use `import x` for packages/modules. Use `from x import y` only when `y` is a submodule.
- **Exceptions:** Use built-in exception classes. Do not use bare `except:` clauses.
- **Global State:** Avoid mutable global state. Module-level constants are okay and should be `ALL_CAPS_WITH_UNDERSCORES`.
- **Comprehensions:** Use for simple cases. Avoid for complex logic where a full loop is more readable.
- **Default Argument Values:** Do not use mutable objects (like `[]` or `{}`) as default values.
- **True/False Evaluations:** Use implicit false (e.g., `if not my_list:`). Use `if foo is None:` to check for `None`.
- **Type Annotations:** Strongly encouraged for all public APIs.
## 2. Python Style Rules
- **Line Length:** Maximum 80 characters.
- **Indentation:** 4 spaces per indentation level. Never use tabs.
- **Blank Lines:** Two blank lines between top-level definitions (classes, functions). One blank line between method definitions.
- **Whitespace:** Avoid extraneous whitespace. Surround binary operators with single spaces.
- **Docstrings:** Use `"""triple double quotes"""`. Every public module, function, class, and method must have a docstring.
- **Format:** Start with a one-line summary. Include `Args:`, `Returns:`, and `Raises:` sections.
- **Strings:** Use f-strings for formatting. Be consistent with single (`'`) or double (`"`) quotes.
- **`TODO` Comments:** Use `TODO(username): Fix this.` format.
- **Imports Formatting:** Imports should be on separate lines and grouped: standard library, third-party, and your own application's imports.
## 3. Naming
- **General:** `snake_case` for modules, functions, methods, and variables.
- **Classes:** `PascalCase`.
- **Constants:** `ALL_CAPS_WITH_UNDERSCORES`.
- **Internal Use:** Use a single leading underscore (`_internal_variable`) for internal module/class members.
## 4. Main
- All executable files should have a `main()` function that contains the main logic, called from a `if __name__ == '__main__':` block.
**BE CONSISTENT.** When editing code, match the existing style.
*Source: [Google Python Style Guide](https://google.github.io/styleguide/pyguide.html)*
-14
View File
@@ -1,14 +0,0 @@
# Project Context
## Definition
- [Product Definition](./product.md)
- [Product Guidelines](./product-guidelines.md)
- [Tech Stack](./tech-stack.md)
## Workflow
- [Workflow](./workflow.md)
- [Code Style Guides](./code_styleguides/)
## Management
- [Tracks Registry](./tracks.md)
- [Tracks Directory](./tracks/)
-15
View File
@@ -1,15 +0,0 @@
# Product Guidelines: Manual Slop
## Documentation Style
- **Strict & In-Depth:** Documentation must follow an old-school, highly detailed technical breakdown style (similar to VEFontCache-Odin). Focus on architectural design, state management, algorithmic details, and structural formats rather than just surface-level usage.
## UX & UI Principles
- **USA Graphics Company Values:** Embrace high information density and tactile interactions.
- **Arcade Aesthetics:** Utilize arcade game-style visual feedback for state updates (e.g., blinking notifications for tool execution and AI responses) to make the experience fun, visceral, and engaging.
- **Explicit Control & Expert Focus:** The interface should not hold the user's hand. It must prioritize explicit manual confirmation for destructive actions while providing dense, unadulterated access to logs and context.
- **Multi-Viewport Capabilities:** Leverage dockable, floatable panels to allow users to build custom workspaces suitable for multi-monitor setups.
## Code Standards & Architecture
- **Strict State Management:** There must be a rigorous separation between the Main GUI rendering thread and daemon execution threads. The UI should *never* hang during AI communication or script execution. Use lock-protected queues and events for synchronization.
- **Comprehensive Logging:** Aggressively log all actions, API payloads, tool calls, and executed scripts. Maintain timestamped JSON-L and markdown logs to ensure total transparency and debuggability.
- **Dependency Minimalism:** Limit external dependencies where possible. For instance, prefer standard library modules (like `urllib` and `html.parser` for web tools) over heavy third-party packages.
-15
View File
@@ -1,15 +0,0 @@
# Product Guide: Manual Slop
## Vision
To serve as an expert-level utility for personal developer use on small projects, providing full, manual control over vendor API metrics, agent capabilities, and context memory usage.
## Primary Use Cases
- **Full Control over Vendor APIs:** Exposing detailed API metrics and configuring deep agent capabilities directly within the GUI.
- **Context & Memory Management:** Better visualization and management of token usage and context memory, allowing developers to optimize prompt limits manually.
- **Manual "Vibe Coding" Assistant:** Serving as an auxiliary, multi-provider assistant that natively interacts with the codebase via sandboxed PowerShell scripts and MCP-like file tools, emphasizing manual developer oversight and explicit confirmation.
## Key Features
- **Multi-Provider Integration:** Supports both Gemini and Anthropic with seamless switching.
- **Explicit Execution Control:** All AI-generated PowerShell scripts require explicit human confirmation via interactive UI dialogs before execution.
- **Detailed History Management:** Rich discussion history with branching, timestamping, and specific git commit linkage per conversation.
- **In-Depth Toolset Access:** MCP-like file exploration, URL fetching, search, and dynamic context aggregation embedded within a multi-viewport Dear PyGui/ImGui interface.
-1
View File
@@ -1 +0,0 @@
{"last_successful_step": "3.3_initial_track_generated"}
-16
View File
@@ -1,16 +0,0 @@
# Technology Stack: Manual Slop
## Core Language
- **Python 3.11+**
## GUI Frameworks
- **Dear PyGui:** For immediate/retained mode GUI rendering and node mapping.
- **ImGui Bundle (`imgui-bundle`):** To provide advanced multi-viewport and dockable panel capabilities on top of Dear ImGui.
## AI Integration SDKs
- **google-genai:** For Google Gemini API interaction and explicit context caching.
- **anthropic:** For Anthropic Claude API interaction, supporting ephemeral prompt caching.
## Configuration & Tooling
- **tomli-w:** For writing TOML configuration files.
- **uv:** An extremely fast Python package and project manager.
-18
View File
@@ -1,18 +0,0 @@
# Project Tracks
This file tracks all major tracks for the project. Each track has its own detailed plan in its respective folder.
---
- [x] **Track: Implement context visualization and memory management improvements**
*Link: [./tracks/context_management_20260223/](./tracks/context_management_20260223/)*
---
- [ ] **Track: Review vendor api usage in regards to conservative context handling**
*Link: [./tracks/api_metrics_20260223/](./tracks/api_metrics_20260223/)*
---
- [ ] **Track: Add full api/hooks so that gemini cli can test, interact, and manipulate the state of the gui & program backend for automated testing.**
*Link: [./tracks/test_hooks_20260223/](./tracks/test_hooks_20260223/)*
@@ -1,5 +0,0 @@
# Track api_metrics_20260223 Context
- [Specification](./spec.md)
- [Implementation Plan](./plan.md)
- [Metadata](./metadata.json)
@@ -1,8 +0,0 @@
{
"track_id": "api_metrics_20260223",
"type": "feature",
"status": "new",
"created_at": "2026-02-23T10:00:00Z",
"updated_at": "2026-02-23T10:00:00Z",
"description": "Review vendor api usage in regards to conservative context handling"
}
@@ -1,19 +0,0 @@
# Implementation Plan
## Phase 1: Metric Extraction and Logic Review
- [ ] Task: Extract explicit cache counts and lifecycle states from Gemini SDK
- [ ] Sub-task: Write Tests
- [ ] Sub-task: Implement Feature
- [ ] Task: Review and expose 'history bleed' (token limit proximity) flags
- [ ] Sub-task: Write Tests
- [ ] Sub-task: Implement Feature
- [ ] Task: Conductor - User Manual Verification 'Phase 1: Metric Extraction and Logic Review' (Protocol in workflow.md)
## Phase 2: GUI Telemetry and Plotting
- [ ] Task: Implement token budget visualizer (e.g., Progress bars for limits) in Dear PyGui
- [ ] Sub-task: Write Tests
- [ ] Sub-task: Implement Feature
- [ ] Task: Implement active caches data display in Provider/Comms panel
- [ ] Sub-task: Write Tests
- [ ] Sub-task: Implement Feature
- [ ] Task: Conductor - User Manual Verification 'Phase 2: GUI Telemetry and Plotting' (Protocol in workflow.md)
@@ -1,22 +0,0 @@
# Specification: Review vendor api usage in regards to conservative context handling
## Overview
This track aims to optimize token efficiency and transparency by reviewing and improving how vendor APIs (Gemini and Anthropic) handle conservative context pruning. The primary focus is on extracting, plotting, and exposing deep metrics to the GUI so developers can intuit how close they are to API limits (e.g., token caps, cache counts, history bleed).
## Scope
- **Gemini Hooks:** Review explicit context caching, cache invalidation, and tools declaration.
- **Global Orchestration:** Review global context boundaries within the main prompt lifecycle.
- **GUI Metrics:** Expose as much metric data as possible to the user interface (e.g., plotting token usage, visual indicators for when "history bleed" occurs, displaying the number of active caches).
## Functional Requirements
- Implement extensive token and cache metric extraction from both Gemini and Anthropic API responses.
- Expose these metrics to the Dear PyGui frontend, potentially utilizing visual plots or progress bars to indicate token budget consumption.
- Implement tests to explicitly verify context rules, ensuring history pruning acts conservatively and predictable without data loss.
## Non-Functional Requirements
- Ensure GUI rendering of new plots or dense metrics does not block the main thread.
- Adhere to the "Strict State Management" product guideline.
## Out of Scope
- Major feature additions unrelated to context token management or telemetry.
- Expanding the AI's agentic capabilities (e.g., new tools).
@@ -1,5 +0,0 @@
# Track context_management_20260223 Context
- [Specification](./spec.md)
- [Implementation Plan](./plan.md)
- [Metadata](./metadata.json)
@@ -1,8 +0,0 @@
{
"track_id": "context_management_20260223",
"type": "feature",
"status": "new",
"created_at": "2026-02-23T10:00:00Z",
"updated_at": "2026-02-23T10:00:00Z",
"description": "Implement context visualization and memory management improvements"
}
@@ -1,19 +0,0 @@
# Implementation Plan
## Phase 1: Context Memory and Token Visualization [checkpoint: a88311b]
- [x] Task: Implement token usage summary widget e34ff7e
- [ ] Sub-task: Write Tests
- [ ] Sub-task: Implement Feature
- [x] Task: Expose history truncation controls in the Discussion panel 94fe904
- [ ] Sub-task: Write Tests
- [ ] Sub-task: Implement Feature
- [x] Task: Conductor - User Manual Verification 'Phase 1: Context Memory and Token Visualization' (Protocol in workflow.md) a88311b
## Phase 2: Agent Capability Configuration [checkpoint: 1ac6eb9]
- [x] Task: Add UI toggles for available tools per-project 1677d25
- [x] Sub-task: Write Tests
- [x] Sub-task: Implement Feature
- [x] Task: Wire tool toggles to AI provider tool declaration payload 92aa33c
- [ ] Sub-task: Write Tests
- [ ] Sub-task: Implement Feature
- [x] Task: Conductor - User Manual Verification 'Phase 2: Agent Capability Configuration' (Protocol in workflow.md) 1ac6eb9
@@ -1,9 +0,0 @@
# Specification: Context Visualization and Memory Management
## Overview
This track implements UI improvements and structural changes to Manual Slop to provide explicit visualization of context memory usage and token consumption, fulfilling the "Expert systems level utility" and "Full control" product goals.
## Core Objectives
1. **Token Visualization:** Expose token usage metrics in real-time within the GUI (e.g., in a dedicated metrics panel or augmented Comms panel).
2. **Context Memory Management:** Provide tools to manually flush, persist, or truncate history to manage token budgets per-discussion.
3. **Agent Capability Toggles:** Expose explicit configuration options for agent capabilities (e.g., toggle MCP tools on/off) from the UI.
@@ -1,5 +0,0 @@
# Track test_hooks_20260223 Context
- [Specification](./spec.md)
- [Implementation Plan](./plan.md)
- [Metadata](./metadata.json)
@@ -1,8 +0,0 @@
{
"track_id": "test_hooks_20260223",
"type": "feature",
"status": "new",
"created_at": "2026-02-23T10:00:00Z",
"updated_at": "2026-02-23T10:00:00Z",
"description": "Add full api/hooks so that gemini cli can test, interact, and manipulate the state of the gui & program backend for automated testing."
}
@@ -1,22 +0,0 @@
# Implementation Plan
## Phase 1: Foundation and Opt-in Mechanisms
- [ ] Task: Implement CLI flag/env-var to enable the hook system
- [ ] Sub-task: Write Tests
- [ ] Sub-task: Implement Feature
- [ ] Task: Set up lightweight local IPC server (e.g., standard library socket/HTTP) for receiving hook commands
- [ ] Sub-task: Write Tests
- [ ] Sub-task: Implement Feature
- [ ] Task: Conductor - User Manual Verification 'Phase 1: Foundation and Opt-in Mechanisms' (Protocol in workflow.md)
## Phase 2: Hook Implementations and Logging
- [ ] Task: Implement project and AI session state manipulation hooks
- [ ] Sub-task: Write Tests
- [ ] Sub-task: Implement Feature
- [ ] Task: Implement GUI state manipulation hooks with thread-safe queueing
- [ ] Sub-task: Write Tests
- [ ] Sub-task: Implement Feature
- [ ] Task: Integrate aggressive logging for all hook invocations
- [ ] Sub-task: Write Tests
- [ ] Sub-task: Implement Feature
- [ ] Task: Conductor - User Manual Verification 'Phase 2: Hook Implementations and Logging' (Protocol in workflow.md)
@@ -1,21 +0,0 @@
# Specification: Add full api/hooks so that gemini cli can test, interact, and manipulate the state of the gui & program backend for automated testing
## Overview
This track introduces a comprehensive suite of API hooks designed specifically for the Gemini CLI and the Conductor framework. These hooks will allow automated agents to manipulate and test the internal state of the application without requiring manual GUI interaction, enabling automated test-driven development and track progression validation.
## Use Cases
- **Automated Testing & Progression:** Expose low-level state manipulation hooks so that the Gemini CLI + Conductor can autonomously verify track completion, test UI logic, and validate backend states.
## Functional Requirements
- **Comprehensive Access:** The hooks must provide full, unrestricted access to the entire program, including:
- GUI state (Dear PyGui nodes, values, layout data).
- AI session state (history, active caches, tool configurations).
- Project configurations and discussion state.
- **Security & Logging:** The hook system MUST be strictly opt-in (e.g., enabled via a specific command-line argument like `--enable-test-hooks` or an environment variable). When enabled, any invocation of these hooks MUST be aggressively logged to ensure transparency.
## Non-Functional Requirements
- **Thread Safety:** Hooks interacting with the GUI state must respect the main render loop locks and threading model defined in the architecture guidelines.
- **Dependency Minimalism:** The hook interface should utilize built-in mechanisms (like sockets, a lightweight local HTTP server, or standard inter-process communication) without introducing heavy external web frameworks.
## Out of Scope
- Building the actual Gemini CLI or Conductor automation logic itself; this track only builds the *hooks* within Manual Slop that those external agents will consume.
-333
View File
@@ -1,333 +0,0 @@
# Project Workflow
## Guiding Principles
1. **The Plan is the Source of Truth:** All work must be tracked in `plan.md`
2. **The Tech Stack is Deliberate:** Changes to the tech stack must be documented in `tech-stack.md` *before* implementation
3. **Test-Driven Development:** Write unit tests before implementing functionality
4. **High Code Coverage:** Aim for >80% code coverage for all modules
5. **User Experience First:** Every decision should prioritize user experience
6. **Non-Interactive & CI-Aware:** Prefer non-interactive commands. Use `CI=true` for watch-mode tools (tests, linters) to ensure single execution.
## Task Workflow
All tasks follow a strict lifecycle:
### Standard Task Workflow
1. **Select Task:** Choose the next available task from `plan.md` in sequential order
2. **Mark In Progress:** Before beginning work, edit `plan.md` and change the task from `[ ]` to `[~]`
3. **Write Failing Tests (Red Phase):**
- Create a new test file for the feature or bug fix.
- Write one or more unit tests that clearly define the expected behavior and acceptance criteria for the task.
- **CRITICAL:** Run the tests and confirm that they fail as expected. This is the "Red" phase of TDD. Do not proceed until you have failing tests.
4. **Implement to Pass Tests (Green Phase):**
- Write the minimum amount of application code necessary to make the failing tests pass.
- Run the test suite again and confirm that all tests now pass. This is the "Green" phase.
5. **Refactor (Optional but Recommended):**
- With the safety of passing tests, refactor the implementation code and the test code to improve clarity, remove duplication, and enhance performance without changing the external behavior.
- Rerun tests to ensure they still pass after refactoring.
6. **Verify Coverage:** Run coverage reports using the project's chosen tools. For example, in a Python project, this might look like:
```bash
pytest --cov=app --cov-report=html
```
Target: >80% coverage for new code. The specific tools and commands will vary by language and framework.
7. **Document Deviations:** If implementation differs from tech stack:
- **STOP** implementation
- Update `tech-stack.md` with new design
- Add dated note explaining the change
- Resume implementation
8. **Commit Code Changes:**
- Stage all code changes related to the task.
- Propose a clear, concise commit message e.g, `feat(ui): Create basic HTML structure for calculator`.
- Perform the commit.
9. **Attach Task Summary with Git Notes:**
- **Step 9.1: Get Commit Hash:** Obtain the hash of the *just-completed commit* (`git log -1 --format="%H"`).
- **Step 9.2: Draft Note Content:** Create a detailed summary for the completed task. This should include the task name, a summary of changes, a list of all created/modified files, and the core "why" for the change.
- **Step 9.3: Attach Note:** Use the `git notes` command to attach the summary to the commit.
```bash
# The note content from the previous step is passed via the -m flag.
git notes add -m "<note content>" <commit_hash>
```
10. **Get and Record Task Commit SHA:**
- **Step 10.1: Update Plan:** Read `plan.md`, find the line for the completed task, update its status from `[~]` to `[x]`, and append the first 7 characters of the *just-completed commit's* commit hash.
- **Step 10.2: Write Plan:** Write the updated content back to `plan.md`.
11. **Commit Plan Update:**
- **Action:** Stage the modified `plan.md` file.
- **Action:** Commit this change with a descriptive message (e.g., `conductor(plan): Mark task 'Create user model' as complete`).
### Phase Completion Verification and Checkpointing Protocol
**Trigger:** This protocol is executed immediately after a task is completed that also concludes a phase in `plan.md`.
1. **Announce Protocol Start:** Inform the user that the phase is complete and the verification and checkpointing protocol has begun.
2. **Ensure Test Coverage for Phase Changes:**
- **Step 2.1: Determine Phase Scope:** To identify the files changed in this phase, you must first find the starting point. Read `plan.md` to find the Git commit SHA of the *previous* phase's checkpoint. If no previous checkpoint exists, the scope is all changes since the first commit.
- **Step 2.2: List Changed Files:** Execute `git diff --name-only <previous_checkpoint_sha> HEAD` to get a precise list of all files modified during this phase.
- **Step 2.3: Verify and Create Tests:** For each file in the list:
- **CRITICAL:** First, check its extension. Exclude non-code files (e.g., `.json`, `.md`, `.yaml`).
- For each remaining code file, verify a corresponding test file exists.
- If a test file is missing, you **must** create one. Before writing the test, **first, analyze other test files in the repository to determine the correct naming convention and testing style.** The new tests **must** validate the functionality described in this phase's tasks (`plan.md`).
3. **Execute Automated Tests with Proactive Debugging:**
- Before execution, you **must** announce the exact shell command you will use to run the tests.
- **Example Announcement:** "I will now run the automated test suite to verify the phase. **Command:** `CI=true npm test`"
- Execute the announced command.
- If tests fail, you **must** inform the user and begin debugging. You may attempt to propose a fix a **maximum of two times**. If the tests still fail after your second proposed fix, you **must stop**, report the persistent failure, and ask the user for guidance.
4. **Propose a Detailed, Actionable Manual Verification Plan:**
- **CRITICAL:** To generate the plan, first analyze `product.md`, `product-guidelines.md`, and `plan.md` to determine the user-facing goals of the completed phase.
- You **must** generate a step-by-step plan that walks the user through the verification process, including any necessary commands and specific, expected outcomes.
- The plan you present to the user **must** follow this format:
**For a Frontend Change:**
```
The automated tests have passed. For manual verification, please follow these steps:
**Manual Verification Steps:**
1. **Start the development server with the command:** `npm run dev`
2. **Open your browser to:** `http://localhost:3000`
3. **Confirm that you see:** The new user profile page, with the user's name and email displayed correctly.
```
**For a Backend Change:**
```
The automated tests have passed. For manual verification, please follow these steps:
**Manual Verification Steps:**
1. **Ensure the server is running.**
2. **Execute the following command in your terminal:** `curl -X POST http://localhost:8080/api/v1/users -d '{"name": "test"}'`
3. **Confirm that you receive:** A JSON response with a status of `201 Created`.
```
5. **Await Explicit User Feedback:**
- After presenting the detailed plan, ask the user for confirmation: "**Does this meet your expectations? Please confirm with yes or provide feedback on what needs to be changed.**"
- **PAUSE** and await the user's response. Do not proceed without an explicit yes or confirmation.
6. **Create Checkpoint Commit:**
- Stage all changes. If no changes occurred in this step, proceed with an empty commit.
- Perform the commit with a clear and concise message (e.g., `conductor(checkpoint): Checkpoint end of Phase X`).
7. **Attach Auditable Verification Report using Git Notes:**
- **Step 7.1: Draft Note Content:** Create a detailed verification report including the automated test command, the manual verification steps, and the user's confirmation.
- **Step 7.2: Attach Note:** Use the `git notes` command and the full commit hash from the previous step to attach the full report to the checkpoint commit.
8. **Get and Record Phase Checkpoint SHA:**
- **Step 8.1: Get Commit Hash:** Obtain the hash of the *just-created checkpoint commit* (`git log -1 --format="%H"`).
- **Step 8.2: Update Plan:** Read `plan.md`, find the heading for the completed phase, and append the first 7 characters of the commit hash in the format `[checkpoint: <sha>]`.
- **Step 8.3: Write Plan:** Write the updated content back to `plan.md`.
9. **Commit Plan Update:**
- **Action:** Stage the modified `plan.md` file.
- **Action:** Commit this change with a descriptive message following the format `conductor(plan): Mark phase '<PHASE NAME>' as complete`.
10. **Announce Completion:** Inform the user that the phase is complete and the checkpoint has been created, with the detailed verification report attached as a git note.
### Quality Gates
Before marking any task complete, verify:
- [ ] All tests pass
- [ ] Code coverage meets requirements (>80%)
- [ ] Code follows project's code style guidelines (as defined in `code_styleguides/`)
- [ ] All public functions/methods are documented (e.g., docstrings, JSDoc, GoDoc)
- [ ] Type safety is enforced (e.g., type hints, TypeScript types, Go types)
- [ ] No linting or static analysis errors (using the project's configured tools)
- [ ] Works correctly on mobile (if applicable)
- [ ] Documentation updated if needed
- [ ] No security vulnerabilities introduced
## Development Commands
**AI AGENT INSTRUCTION: This section should be adapted to the project's specific language, framework, and build tools.**
### Setup
```bash
# Example: Commands to set up the development environment (e.g., install dependencies, configure database)
# e.g., for a Node.js project: npm install
# e.g., for a Go project: go mod tidy
```
### Daily Development
```bash
# Example: Commands for common daily tasks (e.g., start dev server, run tests, lint, format)
# e.g., for a Node.js project: npm run dev, npm test, npm run lint
# e.g., for a Go project: go run main.go, go test ./..., go fmt ./...
```
### Before Committing
```bash
# Example: Commands to run all pre-commit checks (e.g., format, lint, type check, run tests)
# e.g., for a Node.js project: npm run check
# e.g., for a Go project: make check (if a Makefile exists)
```
## Testing Requirements
### Unit Testing
- Every module must have corresponding tests.
- Use appropriate test setup/teardown mechanisms (e.g., fixtures, beforeEach/afterEach).
- Mock external dependencies.
- Test both success and failure cases.
### Integration Testing
- Test complete user flows
- Verify database transactions
- Test authentication and authorization
- Check form submissions
### Mobile Testing
- Test on actual iPhone when possible
- Use Safari developer tools
- Test touch interactions
- Verify responsive layouts
- Check performance on 3G/4G
## Code Review Process
### Self-Review Checklist
Before requesting review:
1. **Functionality**
- Feature works as specified
- Edge cases handled
- Error messages are user-friendly
2. **Code Quality**
- Follows style guide
- DRY principle applied
- Clear variable/function names
- Appropriate comments
3. **Testing**
- Unit tests comprehensive
- Integration tests pass
- Coverage adequate (>80%)
4. **Security**
- No hardcoded secrets
- Input validation present
- SQL injection prevented
- XSS protection in place
5. **Performance**
- Database queries optimized
- Images optimized
- Caching implemented where needed
6. **Mobile Experience**
- Touch targets adequate (44x44px)
- Text readable without zooming
- Performance acceptable on mobile
- Interactions feel native
## Commit Guidelines
### Message Format
```
<type>(<scope>): <description>
[optional body]
[optional footer]
```
### Types
- `feat`: New feature
- `fix`: Bug fix
- `docs`: Documentation only
- `style`: Formatting, missing semicolons, etc.
- `refactor`: Code change that neither fixes a bug nor adds a feature
- `test`: Adding missing tests
- `chore`: Maintenance tasks
### Examples
```bash
git commit -m "feat(auth): Add remember me functionality"
git commit -m "fix(posts): Correct excerpt generation for short posts"
git commit -m "test(comments): Add tests for emoji reaction limits"
git commit -m "style(mobile): Improve button touch targets"
```
## Definition of Done
A task is complete when:
1. All code implemented to specification
2. Unit tests written and passing
3. Code coverage meets project requirements
4. Documentation complete (if applicable)
5. Code passes all configured linting and static analysis checks
6. Works beautifully on mobile (if applicable)
7. Implementation notes added to `plan.md`
8. Changes committed with proper message
9. Git note with task summary attached to the commit
## Emergency Procedures
### Critical Bug in Production
1. Create hotfix branch from main
2. Write failing test for bug
3. Implement minimal fix
4. Test thoroughly including mobile
5. Deploy immediately
6. Document in plan.md
### Data Loss
1. Stop all write operations
2. Restore from latest backup
3. Verify data integrity
4. Document incident
5. Update backup procedures
### Security Breach
1. Rotate all secrets immediately
2. Review access logs
3. Patch vulnerability
4. Notify affected users (if any)
5. Document and update security procedures
## Deployment Workflow
### Pre-Deployment Checklist
- [ ] All tests passing
- [ ] Coverage >80%
- [ ] No linting errors
- [ ] Mobile testing complete
- [ ] Environment variables configured
- [ ] Database migrations ready
- [ ] Backup created
### Deployment Steps
1. Merge feature branch to main
2. Tag release with version
3. Push to deployment service
4. Run database migrations
5. Verify deployment
6. Test critical paths
7. Monitor for errors
### Post-Deployment
1. Monitor analytics
2. Check error logs
3. Gather user feedback
4. Plan next iteration
## Continuous Improvement
- Review workflow weekly
- Update based on pain points
- Document lessons learned
- Optimize for user happiness
- Keep things simple and maintainable
+2 -2
View File
@@ -1,6 +1,6 @@
[ai] [ai]
provider = "gemini" provider = "gemini"
model = "gemini-3.1-pro-preview" model = "gemini-2.5-pro"
temperature = 0.6000000238418579 temperature = 0.6000000238418579
max_tokens = 12000 max_tokens = 12000
history_trunc_limit = 8000 history_trunc_limit = 8000
@@ -10,7 +10,7 @@ system_prompt = "DO NOT EVER make a shell script unless told to. DO NOT EVER mak
palette = "10x Dark" palette = "10x Dark"
font_path = "C:/Users/Ed/AppData/Local/uv/cache/archive-v0/WSthkYsQ82b_ywV6DkiaJ/pygame_gui/data/FiraCode-Regular.ttf" font_path = "C:/Users/Ed/AppData/Local/uv/cache/archive-v0/WSthkYsQ82b_ywV6DkiaJ/pygame_gui/data/FiraCode-Regular.ttf"
font_size = 18.0 font_size = 18.0
scale = 1.0 scale = 1.1
[projects] [projects]
paths = [ paths = [
+2 -3
View File
@@ -29,7 +29,7 @@ Controls what is explicitly fed into the context compiler.
- **Base Dir:** Defines the root for path resolution and tool constraints. - **Base Dir:** Defines the root for path resolution and tool constraints.
- **Paths:** Explicit files or wildcard globs (e.g., src/**/*.rs). - **Paths:** Explicit files or wildcard globs (e.g., src/**/*.rs).
- When generating a request, full file contents are inlined into the context by default (`summary_only=False`). The AI can also call `get_file_summary` via its MCP tools to get a compact structural view of any file on demand. - When generating a request, these files are summarized symbolically (summarize.py) to conserve tokens, unless the AI explicitly decides to read their full contents via its internal tools.
## Interaction Panels ## Interaction Panels
@@ -46,9 +46,8 @@ Switch between API backends (Gemini, Anthropic) on the fly. Clicking "Fetch Mode
### Global Text Viewer & Script Outputs ### Global Text Viewer & Script Outputs
- **Last Script Output:** Whenever the AI executes a background script, this window pops up, flashing blue. It contains both the executed script and the stdout/stderr. The `[+ Maximize]` buttons read directly from stored instance variables (`_last_script`, `_last_output`) rather than DPG widget tags, so they work correctly regardless of word-wrap state. - **Last Script Output:** Whenever the AI executes a background script, this window pops up, flashing blue. It contains both the executed script and the stdout/stderr.
- **Text Viewer:** A large, resizable global popup invoked anytime you click a [+] or [+ Maximize] button in the UI. Used for deep-reading long logs, discussion entries, or script bodies. - **Text Viewer:** A large, resizable global popup invoked anytime you click a [+] or [+ Maximize] button in the UI. Used for deep-reading long logs, discussion entries, or script bodies.
- **Confirm Dialog:** The `[+ Maximize]` button in the script approval modal passes the script text directly as `user_data` at button-creation time, so it remains safe to click even after the dialog has been dismissed.
## System Prompts ## System Prompts
+5 -5
View File
@@ -1,4 +1,4 @@
# Guide: Architecture # Guide: Architecture
Overview of the package design, state management, and code-path layout. Overview of the package design, state management, and code-path layout.
@@ -33,9 +33,10 @@ This occurs inside aggregate.run.
If using the default workflow, aggregate.py hashes through the following process: If using the default workflow, aggregate.py hashes through the following process:
1. **Glob Resolution:** Iterates through config["files"]["paths"] and unpacks any wildcards (e.g., src/**/*.rs) against the designated base_dir. 1. **Glob Resolution:** Iterates through config["files"]["paths"] and unpacks any wildcards (e.g., src/**/*.rs) against the designated base_dir.
2. **File Item Build:** `build_file_items()` reads each resolved file once, storing path, content, and `mtime`. This list is returned alongside the markdown so `ai_client.py` can use it for dynamic context refresh after tool calls without re-reading from disk. 2. **Summarization Pass:** Instead of concatenating raw file bodies (which would quickly overwhelm the ~200k token limit over multiple rounds), the files are passed to summarize.py.
3. **Markdown Generation:** `build_markdown_from_items()` assembles the final `<project>_00N.md` string. By default (`summary_only=False`) it inlines full file contents. If `summary_only=True`, it delegates to `summarize.build_summary_markdown()` which uses AST-based heuristics to produce compact structural summaries instead. 3. **AST Parsing:** summarize.py runs a heuristic pass. For Python files, it uses the standard ast module to read structural nodes (Classes, Methods, Imports, Constants). It outputs a compact Markdown table.
4. The Markdown file is persisted to disk (`./md_gen/` by default) for auditing. `run()` returns a 3-tuple `(markdown_str, output_path, file_items)`. 4. **Markdown Generation:** The final <project>_00N.md string is constructed, comprising the truncated AST summaries, the user's current project system prompt, and the active discussion branch.
5. The Markdown file is persisted to disk (./md_gen/ by default) for auditing.
### AI Communication & The Tool Loop ### AI Communication & The Tool Loop
@@ -84,4 +85,3 @@ All I/O bound session data is recorded sequentially. session_logger.py hooks int
- logs/comms_<ts>.log: A JSON-L structured timeline of every raw payload sent/received. - logs/comms_<ts>.log: A JSON-L structured timeline of every raw payload sent/received.
- logs/toolcalls_<ts>.log: A sequential markdown record detailing every AI tool invocation and its exact stdout result. - logs/toolcalls_<ts>.log: A sequential markdown record detailing every AI tool invocation and its exact stdout result.
- scripts/generated/: Every .ps1 script approved and executed by the shell runner is physically written to disk for version control transparency. - scripts/generated/: Every .ps1 script approved and executed by the shell runner is physically written to disk for version control transparency.
+7 -12
View File
@@ -12,22 +12,17 @@ Implemented in mcp_client.py. These tools allow the AI to selectively expand its
### Security & Scope ### Security & Scope
Every **filesystem** MCP tool passes its arguments through `_resolve_and_check`. This function ensures that the requested path falls under one of the allowed directories defined in the GUI's Base Dir configurations. Every filesystem MCP tool passes its arguments through _resolve_and_check. This function ensures that the requested path falls under one of the allowed directories defined in the GUI's Base Dir configurations.
If the AI attempts to read or search a path outside the project bounds, the tool safely catches the constraint violation and returns ACCESS DENIED. If the AI attempts to read or search a path outside the project bounds, the tool safely catches the constraint violation and returns ACCESS DENIED.
The two **web tools** (`web_search`, `fetch_url`) bypass this check entirely — they have no filesystem access and are unrestricted.
### Supplied Tools: ### Supplied Tools:
**Filesystem tools** (access-controlled via `_resolve_and_check`): * read_file(path): Returns the raw UTF-8 text of a file.
* `read_file(path)`: Returns the raw UTF-8 text of a file. * list_directory(path): Returns a formatted table of a directory's contents, showing file vs dir and byte sizes.
* `list_directory(path)`: Returns a formatted table of a directory's contents, showing file vs dir and byte sizes. * search_files(path, pattern): Executes an absolute glob search (e.g., **/*.py) to find specific files.
* `search_files(path, pattern)`: Executes a glob search (e.g., `**/*.py`) within an allowed directory. * get_file_summary(path): Invokes the local summarize.py heuristic parser to get the AST structure of a file without reading the whole body.
* `get_file_summary(path)`: Invokes the local `summarize.py` heuristic parser to get the AST structure of a file without reading the whole body. * web_search(query): Queries DuckDuckGo's raw HTML endpoint and returns the top 5 results (Titles, URLs, Snippets) using a native HTMLParser to avoid heavy dependencies.
* fetch_url(url): Downloads a target webpage and strips out all scripts, styling, and structural HTML, returning only the raw prose content (clamped to 40,000 characters).
**Web tools** (unrestricted — no filesystem access):
* `web_search(query)`: Queries DuckDuckGo's raw HTML endpoint and returns the top 5 results (title, URL, snippet) using a native `_DDGParser` (HTMLParser subclass) to avoid heavy dependencies.
* `fetch_url(url)`: Downloads a target webpage and strips out all scripts, styling, and structural HTML via `_TextExtractor`, returning only the raw prose content (clamped to 40,000 characters). Automatically resolves DuckDuckGo redirect links.
## 2. Destructive Execution (run_powershell) ## 2. Destructive Execution (run_powershell)
-35
View File
@@ -1,35 +0,0 @@
# gemini.py
import tomllib
from pathlib import Path
from google import genai
from google.genai import types
_client = None
_chat = None
def _load_key() -> str:
with open("credentials.toml", "rb") as f:
return tomllib.load(f)["gemini"]["api_key"]
def _ensure_client():
global _client
if _client is None:
_client = genai.Client(api_key=_load_key())
def _ensure_chat():
global _chat
if _chat is None:
_ensure_client()
_chat = _client.chats.create(model="gemini-2.0-flash")
def send(md_content: str, user_message: str) -> str:
global _chat
_ensure_chat()
full_message = f"<context>\n{md_content}\n</context>\n\n{user_message}"
response = _chat.send_message(full_message)
return response.text
def reset_session():
global _client, _chat
_client = None
_chat = None
+28 -86
View File
@@ -1,4 +1,4 @@
# gui.py # gui.py
""" """
Note(Gemini): Note(Gemini):
The main DearPyGui interface orchestrator. The main DearPyGui interface orchestrator.
@@ -47,30 +47,6 @@ def hide_tk_root() -> Tk:
root.wm_attributes("-topmost", True) root.wm_attributes("-topmost", True)
return root return root
def get_total_token_usage() -> dict:
"""Returns aggregated token usage across the entire session from comms log."""
usage = {
"input_tokens": 0,
"output_tokens": 0,
"cache_read_input_tokens": 0,
"cache_creation_input_tokens": 0
}
for entry in ai_client.get_comms_log():
if entry.get("kind") == "response" and "usage" in entry.get("payload", {}):
u = entry["payload"]["usage"]
for k in usage.keys():
usage[k] += u.get(k, 0) or 0
return usage
def truncate_entries(entries: list[dict], max_pairs: int) -> list[dict]:
"""Truncates history to the last N pairs of User/AI messages."""
if max_pairs <= 0:
return []
target_count = max_pairs * 2
if len(entries) <= target_count:
return entries
return entries[-target_count:]
# ------------------------------------------------------------------ comms rendering helpers # ------------------------------------------------------------------ comms rendering helpers
@@ -325,9 +301,9 @@ class ConfirmDialog:
with dpg.group(horizontal=True): with dpg.group(horizontal=True):
dpg.add_text("Script:") dpg.add_text("Script:")
dpg.add_button( dpg.add_button(
label="[+ Maximize]", label="[+ Maximize]",
user_data=self._script, user_data=f"{self._tag}_script",
callback=lambda s, a, u: _show_text_viewer("Confirm Script", u) callback=lambda s, a, u: _show_text_viewer("Confirm Script", dpg.get_value(u))
) )
dpg.add_input_text( dpg.add_input_text(
tag=f"{self._tag}_script", tag=f"{self._tag}_script",
@@ -456,8 +432,6 @@ class App:
self._pending_dialog_lock = threading.Lock() self._pending_dialog_lock = threading.Lock()
self._tool_log: list[tuple[str, str]] = [] self._tool_log: list[tuple[str, str]] = []
self._last_script: str = ""
self._last_output: str = ""
# Comms log entries queued from background thread for main-thread rendering # Comms log entries queued from background thread for main-thread rendering
self._pending_comms: list[dict] = [] self._pending_comms: list[dict] = []
@@ -589,13 +563,6 @@ class App:
dpg.set_value("auto_add_history", proj.get("discussion", {}).get("auto_add", False)) dpg.set_value("auto_add_history", proj.get("discussion", {}).get("auto_add", False))
if dpg.does_item_exist("project_word_wrap"): if dpg.does_item_exist("project_word_wrap"):
dpg.set_value("project_word_wrap", proj.get("project", {}).get("word_wrap", True)) dpg.set_value("project_word_wrap", proj.get("project", {}).get("word_wrap", True))
agent_tools = proj.get("agent", {}).get("tools", {})
for t_name in ["run_powershell", "read_file", "list_directory", "search_files", "get_file_summary", "web_search", "fetch_url"]:
tag = f"tool_toggle_{t_name}"
if dpg.does_item_exist(tag):
dpg.set_value(tag, agent_tools.get(t_name, True))
self.cb_word_wrap_toggled(app_data=proj.get("project", {}).get("word_wrap", True)) self.cb_word_wrap_toggled(app_data=proj.get("project", {}).get("word_wrap", True))
def _save_active_project(self): def _save_active_project(self):
@@ -744,15 +711,6 @@ class App:
for entry in entries: for entry in entries:
self._comms_entry_count += 1 self._comms_entry_count += 1
self._append_comms_entry(entry, self._comms_entry_count) self._append_comms_entry(entry, self._comms_entry_count)
if entries:
self._update_token_usage()
def _update_token_usage(self):
if not dpg.does_item_exist("ai_token_usage"):
return
usage = get_total_token_usage()
total = usage["input_tokens"] + usage["output_tokens"]
dpg.set_value("ai_token_usage", f"Tokens: {total} (In: {usage['input_tokens']} Out: {usage['output_tokens']})")
def _append_comms_entry(self, entry: dict, idx: int): def _append_comms_entry(self, entry: dict, idx: int):
if not dpg.does_item_exist("comms_scroll"): if not dpg.does_item_exist("comms_scroll"):
@@ -790,8 +748,6 @@ class App:
return output return output
def _append_tool_log(self, script: str, result: str): def _append_tool_log(self, script: str, result: str):
self._last_script = script
self._last_output = result
self._tool_log.append((script, result)) self._tool_log.append((script, result))
self._rebuild_tool_log() self._rebuild_tool_log()
@@ -874,13 +830,6 @@ class App:
if dpg.does_item_exist("project_word_wrap"): if dpg.does_item_exist("project_word_wrap"):
proj["project"]["word_wrap"] = dpg.get_value("project_word_wrap") proj["project"]["word_wrap"] = dpg.get_value("project_word_wrap")
# Agent tools
proj.setdefault("agent", {}).setdefault("tools", {})
for t_name in ["run_powershell", "read_file", "list_directory", "search_files", "get_file_summary", "web_search", "fetch_url"]:
tag = f"tool_toggle_{t_name}"
if dpg.does_item_exist(tag):
proj["agent"]["tools"][t_name] = dpg.get_value(tag)
# Discussion # Discussion
self._flush_disc_entries_to_project() self._flush_disc_entries_to_project()
disc_sec = proj.setdefault("discussion", {}) disc_sec = proj.setdefault("discussion", {})
@@ -1161,8 +1110,16 @@ class App:
def cb_md_only(self): def cb_md_only(self):
try: try:
md, path, _file_items = self._do_generate() result_from_do_generate = self._do_generate()
self.last_md = md if not isinstance(result_from_do_generate, (list, tuple)):
self._update_status(f"generate error: _do_generate returned a non-tuple ({type(result_from_do_generate)})")
return
num_values = len(result_from_do_generate)
if num_values != 4:
self._update_status(f"generate error: _do_generate returned {num_values} values, expected 4")
return
static_md, dynamic_md, path, _file_items = result_from_do_generate
self.last_md = f"{static_md}\n\n---\n\n{dynamic_md}" if static_md and dynamic_md else static_md or dynamic_md
self.last_md_path = path self.last_md_path = path
self._update_status(f"md written: {path.name}") self._update_status(f"md written: {path.name}")
except Exception as e: except Exception as e:
@@ -1185,8 +1142,16 @@ class App:
if self.send_thread and self.send_thread.is_alive(): if self.send_thread and self.send_thread.is_alive():
return return
try: try:
md, path, file_items = self._do_generate() result_from_do_generate = self._do_generate()
self.last_md = md if not isinstance(result_from_do_generate, (list, tuple)):
self._update_status(f"generate error: _do_generate returned a non-tuple ({type(result_from_do_generate)})")
return
num_values = len(result_from_do_generate)
if num_values != 4:
self._update_status(f"generate error: _do_generate returned {num_values} values, expected 4")
return
static_md, dynamic_md, path, file_items = result_from_do_generate
self.last_md = static_md
self.last_md_path = path self.last_md_path = path
self.last_file_items = file_items self.last_file_items = file_items
except Exception as e: except Exception as e:
@@ -1203,7 +1168,6 @@ class App:
if global_sp: combined_sp.append(global_sp.strip()) if global_sp: combined_sp.append(global_sp.strip())
if project_sp: combined_sp.append(project_sp.strip()) if project_sp: combined_sp.append(project_sp.strip())
ai_client.set_custom_system_prompt("\n\n".join(combined_sp)) ai_client.set_custom_system_prompt("\n\n".join(combined_sp))
ai_client.set_agent_tools(self.project.get("agent", {}).get("tools", {}))
temp = dpg.get_value("ai_temperature") if dpg.does_item_exist("ai_temperature") else 0.0 temp = dpg.get_value("ai_temperature") if dpg.does_item_exist("ai_temperature") else 0.0
max_tok = dpg.get_value("ai_max_tokens") if dpg.does_item_exist("ai_max_tokens") else 8192 max_tok = dpg.get_value("ai_max_tokens") if dpg.does_item_exist("ai_max_tokens") else 8192
trunc = dpg.get_value("ai_history_trunc") if dpg.does_item_exist("ai_history_trunc") else 8000 trunc = dpg.get_value("ai_history_trunc") if dpg.does_item_exist("ai_history_trunc") else 8000
@@ -1265,7 +1229,6 @@ class App:
with self._pending_comms_lock: with self._pending_comms_lock:
self._pending_comms.clear() self._pending_comms.clear()
self._comms_entry_count = 0 self._comms_entry_count = 0
self._update_token_usage()
if dpg.does_item_exist("comms_scroll"): if dpg.does_item_exist("comms_scroll"):
dpg.delete_item("comms_scroll", children_only=True) dpg.delete_item("comms_scroll", children_only=True)
@@ -1368,12 +1331,6 @@ class App:
self.disc_entries.clear() self.disc_entries.clear()
self._rebuild_disc_list() self._rebuild_disc_list()
def cb_disc_truncate(self):
pairs = dpg.get_value("disc_truncate_pairs") if dpg.does_item_exist("disc_truncate_pairs") else 2
self.disc_entries = truncate_entries(self.disc_entries, pairs)
self._rebuild_disc_list()
self._update_status(f"history truncated to {pairs} pairs")
def cb_disc_collapse_all(self): def cb_disc_collapse_all(self):
for i, entry in enumerate(self.disc_entries): for i, entry in enumerate(self.disc_entries):
tag = f"disc_content_{i}" tag = f"disc_content_{i}"
@@ -1714,15 +1671,6 @@ class App:
default_value=self.project.get("project", {}).get("word_wrap", True), default_value=self.project.get("project", {}).get("word_wrap", True),
callback=self.cb_word_wrap_toggled callback=self.cb_word_wrap_toggled
) )
dpg.add_separator()
dpg.add_text("Agent Capabilities")
agent_tools = self.project.get("agent", {}).get("tools", {})
for t_name in ["run_powershell", "read_file", "list_directory", "search_files", "get_file_summary", "web_search", "fetch_url"]:
dpg.add_checkbox(
tag=f"tool_toggle_{t_name}",
label=f"Enable {t_name}",
default_value=agent_tools.get(t_name, True)
)
# ---- Files panel ---- # ---- Files panel ----
with dpg.window( with dpg.window(
@@ -1800,9 +1748,6 @@ class App:
dpg.add_button(label="+ Entry", callback=self.cb_disc_append_entry) dpg.add_button(label="+ Entry", callback=self.cb_disc_append_entry)
dpg.add_button(label="-All", callback=self.cb_disc_collapse_all) dpg.add_button(label="-All", callback=self.cb_disc_collapse_all)
dpg.add_button(label="+All", callback=self.cb_disc_expand_all) dpg.add_button(label="+All", callback=self.cb_disc_expand_all)
dpg.add_text("Keep Pairs:", color=(160, 160, 160))
dpg.add_input_int(tag="disc_truncate_pairs", default_value=2, width=120, min_value=1)
dpg.add_button(label="Truncate", callback=self.cb_disc_truncate)
dpg.add_button(label="Clear All", callback=self.cb_disc_clear) dpg.add_button(label="Clear All", callback=self.cb_disc_clear)
dpg.add_button(label="Save", callback=self.cb_disc_save) dpg.add_button(label="Save", callback=self.cb_disc_save)
dpg.add_checkbox( dpg.add_checkbox(
@@ -1931,8 +1876,6 @@ class App:
with dpg.group(horizontal=True): with dpg.group(horizontal=True):
dpg.add_text("Status: idle", tag="ai_status", color=(200, 220, 160)) dpg.add_text("Status: idle", tag="ai_status", color=(200, 220, 160))
dpg.add_spacer(width=16) dpg.add_spacer(width=16)
dpg.add_text("Tokens: 0 (In: 0 Out: 0)", tag="ai_token_usage", color=(180, 255, 180))
dpg.add_spacer(width=16)
dpg.add_button(label="Clear", callback=self.cb_clear_comms) dpg.add_button(label="Clear", callback=self.cb_clear_comms)
dpg.add_separator() dpg.add_separator()
with dpg.group(horizontal=True): with dpg.group(horizontal=True):
@@ -1990,7 +1933,8 @@ class App:
dpg.add_text("Script:") dpg.add_text("Script:")
dpg.add_button( dpg.add_button(
label="[+ Maximize]", label="[+ Maximize]",
callback=lambda s, a, u: _show_text_viewer("Last Script", self._last_script), user_data="last_script_text",
callback=lambda s, a, u: _show_text_viewer("Last Script", dpg.get_value(u))
) )
dpg.add_input_text( dpg.add_input_text(
tag="last_script_text", tag="last_script_text",
@@ -2006,7 +1950,8 @@ class App:
dpg.add_text("Output:") dpg.add_text("Output:")
dpg.add_button( dpg.add_button(
label="[+ Maximize]", label="[+ Maximize]",
callback=lambda s, a, u: _show_text_viewer("Last Output", self._last_output), user_data="last_script_output",
callback=lambda s, a, u: _show_text_viewer("Last Output", dpg.get_value(u))
) )
dpg.add_input_text( dpg.add_input_text(
tag="last_script_output", tag="last_script_output",
@@ -2192,6 +2137,3 @@ def main():
if __name__ == "__main__": if __name__ == "__main__":
main() main()
+39 -46
View File
File diff suppressed because one or more lines are too long
-11
View File
@@ -100,17 +100,6 @@ def default_project(name: str = "unnamed") -> dict:
"output": {"output_dir": "./md_gen"}, "output": {"output_dir": "./md_gen"},
"files": {"base_dir": ".", "paths": []}, "files": {"base_dir": ".", "paths": []},
"screenshots": {"base_dir": ".", "paths": []}, "screenshots": {"base_dir": ".", "paths": []},
"agent": {
"tools": {
"run_powershell": True,
"read_file": True,
"list_directory": True,
"search_files": True,
"get_file_summary": True,
"web_search": True,
"fetch_url": True
}
},
"discussion": { "discussion": {
"roles": ["User", "AI", "Vendor API", "System"], "roles": ["User", "AI", "Vendor API", "System"],
"active": "main", "active": "main",
-5
View File
@@ -10,8 +10,3 @@ dependencies = [
"anthropic", "anthropic",
"tomli-w" "tomli-w"
] ]
[dependency-groups]
dev = [
"pytest>=9.0.2",
]
-1
View File
@@ -1 +0,0 @@
Get-Content .env | ForEach-Object { $name, $value = $_.Split('=', 2); [Environment]::SetEnvironmentVariable($name, $value, "Process") }
-17
View File
@@ -1,17 +0,0 @@
import pytest
def test_agent_capabilities_config():
# A dummy test to fulfill the Red Phase for Agent Capability Configuration.
# The new function in gui.py should be get_active_tools() or we check the project dict.
from project_manager import default_project
proj = default_project("test_proj")
# We expect 'agent' config to exist in a default project and list tools
assert "agent" in proj
assert "tools" in proj["agent"]
# By default, all tools should probably be True or defined
tools = proj["agent"]["tools"]
assert "run_powershell" in tools
assert tools["run_powershell"] is True
-23
View File
@@ -1,23 +0,0 @@
import pytest
from ai_client import set_agent_tools, _build_anthropic_tools
def test_agent_tools_wiring():
# Only enable read_file and run_powershell
agent_tools = {
"run_powershell": True,
"read_file": True,
"list_directory": False,
"search_files": False,
"get_file_summary": False,
"web_search": False,
"fetch_url": False
}
set_agent_tools(agent_tools)
anth_tools = _build_anthropic_tools()
tool_names = [t["name"] for t in anth_tools]
assert "read_file" in tool_names
assert "run_powershell" in tool_names
assert "list_directory" not in tool_names
assert "web_search" not in tool_names
-22
View File
@@ -1,22 +0,0 @@
import pytest
def test_history_truncation():
# A dummy test to fulfill the Red Phase for the history truncation controls.
# The new function in gui.py should be cb_disc_truncate_history or a related utility.
from project_manager import str_to_entry, entry_to_str
entries = [
{"role": "User", "content": "1", "collapsed": False, "ts": "10:00:00"},
{"role": "AI", "content": "2", "collapsed": False, "ts": "10:01:00"},
{"role": "User", "content": "3", "collapsed": False, "ts": "10:02:00"},
{"role": "AI", "content": "4", "collapsed": False, "ts": "10:03:00"}
]
# We expect a new function truncate_entries(entries, max_pairs) to exist
from gui import truncate_entries
truncated = truncate_entries(entries, max_pairs=1)
# Keeping the last pair (user + ai)
assert len(truncated) == 2
assert truncated[0]["content"] == "3"
assert truncated[1]["content"] == "4"
-35
View File
@@ -1,35 +0,0 @@
import pytest
def test_token_usage_aggregation():
# A dummy test to fulfill the Red Phase for the new token usage widget.
# We will implement a function in gui.py or ai_client.py to aggregate tokens.
from ai_client import _comms_log, clear_comms_log, _append_comms
clear_comms_log()
_append_comms("IN", "response", {
"usage": {
"input_tokens": 100,
"output_tokens": 50,
"cache_read_input_tokens": 10,
"cache_creation_input_tokens": 5
}
})
_append_comms("IN", "response", {
"usage": {
"input_tokens": 200,
"output_tokens": 100,
"cache_read_input_tokens": 20,
"cache_creation_input_tokens": 0
}
})
# We expect a new function get_total_token_usage() to exist
from gui import get_total_token_usage
totals = get_total_token_usage()
assert totals["input_tokens"] == 300
assert totals["output_tokens"] == 150
assert totals["cache_read_input_tokens"] == 30
assert totals["cache_creation_input_tokens"] == 5