From 5e6a38a7903592646aacfce39d703f433bbfd33b Mon Sep 17 00:00:00 2001 From: Ed_ Date: Sun, 8 Mar 2026 12:53:42 -0400 Subject: [PATCH] chore(conductor): Add new track 'Advanced Log Management and Session Restoration' --- conductor/tracks.md | 40 +++++++++------- .../log_session_overhaul_20260308/index.md | 5 ++ .../metadata.json | 8 ++++ .../log_session_overhaul_20260308/plan.md | 38 +++++++++++++++ .../log_session_overhaul_20260308/spec.md | 47 +++++++++++++++++++ 5 files changed, 120 insertions(+), 18 deletions(-) create mode 100644 conductor/tracks/log_session_overhaul_20260308/index.md create mode 100644 conductor/tracks/log_session_overhaul_20260308/metadata.json create mode 100644 conductor/tracks/log_session_overhaul_20260308/plan.md create mode 100644 conductor/tracks/log_session_overhaul_20260308/spec.md diff --git a/conductor/tracks.md b/conductor/tracks.md index 5ad44d7..8c08042 100644 --- a/conductor/tracks.md +++ b/conductor/tracks.md @@ -1,4 +1,4 @@ -# Project Tracks +# Project Tracks This file tracks all major tracks for the project. Each track has its own detailed plan in its respective folder. @@ -12,7 +12,7 @@ This file tracks all major tracks for the project. Each track has its own detail --- -## Phase 3: Future Horizons (Tracks 1-28) +## Phase 3: Future Horizons (Tracks 1-31) *Initialized: 2026-03-06* *Updated: 2026-03-08* @@ -64,52 +64,56 @@ This file tracks all major tracks for the project. Each track has its own detail 14. [x] **Track: On-Demand Definition Lookup** *Link: [./tracks/on_demand_def_lookup_20260306/](./tracks/on_demand_def_lookup_20260306/)* +15. [ ] **Track: Advanced Log Management and Session Restoration** + *Link: [./tracks/log_session_overhaul_20260308/](./tracks/log_session_overhaul_20260308/)* + *Goal: Centralize log management, improve session restoration reliability with full-UI replay mode, and optimize log size via external script/output referencing. Implement transient diagnostic logging for system warnings.* + --- ### Manual UX Controls -15. [x] **Track: Manual Ticket Queue Management** +16. [x] **Track: Manual Ticket Queue Management** *Link: [./tracks/ticket_queue_mgmt_20260306/](./tracks/ticket_queue_mgmt_20260306/)* -16. [x] **Track: Kill/Abort Running Workers** +17. [x] **Track: Kill/Abort Running Workers** *Link: [./tracks/kill_abort_workers_20260306/](./tracks/kill_abort_workers_20260306/)* -17. [x] **Track: Manual Block/Unblock Control** +18. [x] **Track: Manual Block/Unblock Control** *Link: [./tracks/manual_block_control_20260306/](./tracks/manual_block_control_20260306/)* -18. [x] **Track: Pipeline Pause/Resume** +19. [x] **Track: Pipeline Pause/Resume** *Link: [./tracks/pipeline_pause_resume_20260306/](./tracks/pipeline_pause_resume_20260306/)* -19. [x] **Track: Per-Ticket Model Override** +20. [x] **Track: Per-Ticket Model Override** *Link: [./tracks/per_ticket_model_20260306/](./tracks/per_ticket_model_20260306/)* -20. [ ] **Track: Manual UX Validation & Review** +21. [ ] **Track: Manual UX Validation & Review** *Link: [./tracks/manual_ux_validation_20260302/](./tracks/manual_ux_validation_20260302/)* ### Misc Side-tracks -21. [x] **Track: Enhanced Context Control & Cache Awareness** +22. [x] **Track: Enhanced Context Control & Cache Awareness** *Link: [./tracks/enhanced_context_control_20260307/](./tracks/enhanced_context_control_20260307/)* -22. [x] **Track: GUI Performance Profiling & Optimization** +23. [x] **Track: GUI Performance Profiling & Optimization** *Link: [./tracks/gui_performance_profiling_20260307/](./tracks/gui_performance_profiling_20260307/)* -23. [x] **Track: Test Integrity Audit & Intent Documentation** +24. [x] **Track: Test Integrity Audit & Intent Documentation** *Link: [./tracks/test_integrity_audit_20260307/](./tracks/test_integrity_audit_20260307/)* *Goal: Audit tests simplified by AI agents. Add intent documentation comments to prevent future simplification. Covers simulation tests (test_sim_*.py), live workflow tests, and major feature tests.* -24. [x] **Track: Test Regression Verification** +25. [x] **Track: Test Regression Verification** *Link: [./tracks/test_regression_verification_20260307/](./tracks/test_regression_verification_20260307/)* --- ### C/C++ Language Support -25. [ ] **Track: Tree-Sitter C/C++ MCP Tools** +26. [ ] **Track: Tree-Sitter C/C++ MCP Tools** *Link: [./tracks/ts_cpp_tree_sitter_20260308/](./tracks/ts_cpp_tree_sitter_20260308/)* *Goal: Add tree-sitter C and C++ grammars. Extend ASTParser to support C/C++ skeleton and outline extraction. Add MCP tools ts_c_get_skeleton, ts_cpp_get_skeleton, ts_c_get_code_outline, ts_cpp_get_code_outline.* -26. [ ] **Track: Bootstrap gencpp Python Bindings** +27. [ ] **Track: Bootstrap gencpp Python Bindings** *Link: [./tracks/gencpp_python_bindings_20260308/](./tracks/gencpp_python_bindings_20260308/)* *Goal: Bootstrap standalone Python project with CFFI bindings for gencpp C library. Provides foundation for richer C++ AST parsing in future (beyond tree-sitter syntax).* @@ -117,19 +121,19 @@ This file tracks all major tracks for the project. Each track has its own detail ### Path Configuration -27. [ ] **Track: Project-Specific Conductor Directory** +28. [ ] **Track: Project-Specific Conductor Directory** *Link: [./tracks/project_conductor_dir_20260308/](./tracks/project_conductor_dir_20260308/)* *Goal: Make conductor directory per-project. Each project TOML can specify custom conductor dir for isolated track/state management.* -28. [ ] **Track: GUI Path Configuration in Context Hub** +29. [ ] **Track: GUI Path Configuration in Context Hub** *Link: [./tracks/gui_path_config_20260308/](./tracks/gui_path_config_20260308/)* *Goal: Add path configuration UI to Context Hub. Allow users to view and edit configurable paths directly from the GUI.* -29. [ ] **Track: Saved System Prompt Presets** +30. [ ] **Track: Saved System Prompt Presets** *Link: [./tracks/saved_presets_20260308/](./tracks/saved_presets_20260308/)* *Goal: Ability to have saved presets for global and project system prompts. Includes full AI profiles with temperature and top_p settings, managed via a dedicated GUI modal.* -30. [ ] **Track: Saved Tool Presets** +31. [ ] **Track: Saved Tool Presets** *Link: [./tracks/saved_tool_presets_20260308/](./tracks/saved_tool_presets_20260308/)* *Goal: Make agent tools have presets. Add flags for tools related to their level of approval (auto, ask). Move tools to ai settings. Put tools in dynamic TOML-defined categories (Python, General, etc.). Tool Presets added to mma agent role options.* diff --git a/conductor/tracks/log_session_overhaul_20260308/index.md b/conductor/tracks/log_session_overhaul_20260308/index.md new file mode 100644 index 0000000..5ef0eeb --- /dev/null +++ b/conductor/tracks/log_session_overhaul_20260308/index.md @@ -0,0 +1,5 @@ +# Track log_session_overhaul_20260308 Context + +- [Specification](./spec.md) +- [Implementation Plan](./plan.md) +- [Metadata](./metadata.json) diff --git a/conductor/tracks/log_session_overhaul_20260308/metadata.json b/conductor/tracks/log_session_overhaul_20260308/metadata.json new file mode 100644 index 0000000..ffc7bb3 --- /dev/null +++ b/conductor/tracks/log_session_overhaul_20260308/metadata.json @@ -0,0 +1,8 @@ +{ + "track_id": "log_session_overhaul_20260308", + "type": "feature", + "status": "new", + "created_at": "2026-03-08T12:53:00Z", + "updated_at": "2026-03-08T12:53:00Z", + "description": "Move comms log's load log button to log management. Make it load an entire session's log instead of just comms. Rework loading implementation for reliability. Handle and filter MMA agent logs in comms log. Offload generated scripts and tool output to separate files with ID referencing. Relocate performance warnings from discussion to transient diagnostic logs." +} diff --git a/conductor/tracks/log_session_overhaul_20260308/plan.md b/conductor/tracks/log_session_overhaul_20260308/plan.md new file mode 100644 index 0000000..52135a0 --- /dev/null +++ b/conductor/tracks/log_session_overhaul_20260308/plan.md @@ -0,0 +1,38 @@ +# Implementation Plan: Advanced Log Management and Session Restoration + +## Phase 1: Storage Optimization (Offloading Data) +- [ ] Task: Implement file-based offloading for scripts and tool outputs. + - [ ] Update `src/session_logger.py` to include `log_tool_output(session_id, output)` which saves output to a unique file in the session directory and returns the filename. + - [ ] Modify `src/session_logger.py:log_tool_call` to ensure scripts are consistently saved and return a unique filename/ID. + - [ ] Update `src/app_controller.py` to use these unique IDs/filenames in the `payload` of comms and tool logs instead of raw content. +- [ ] Task: Verify that logs are smaller and scripts/outputs are correctly saved to the session directory. +- [ ] Task: Conductor - User Manual Verification 'Phase 1: Storage Optimization' (Protocol in workflow.md) + +## Phase 2: Session-Level Restoration & UI Relocation +- [ ] Task: Relocate the "Load Log" button. + - [ ] Remove the "Load Log" button from `_render_comms_history_panel` in `src/gui_2.py`. + - [ ] Add the "Load Log" button to the "Log Management" panel in `src/gui_2.py`. +- [ ] Task: Rework `cb_load_prior_log` for session-level loading. + - [ ] Update `src/app_controller.py:cb_load_prior_log` to allow selecting a session directory or the main session log file. + - [ ] Implement logic to load all related logs (comms, mma, tools) for that session. + - [ ] Ensure that for entries referencing external files (scripts/outputs), the content is loaded on-demand or during the restoration process. +- [ ] Task: Implement "Historical Replay" UI mode. + - [ ] In `src/gui_2.py`, implement logic to tint the UI (as already partially done for comms) when `is_viewing_prior_session` is True. + - [ ] Populate `disc_entries`, `_comms_log`, and MMA Dashboard states from the loaded session logs. +- [ ] Task: Conductor - User Manual Verification 'Phase 2: Session-Level Restoration' (Protocol in workflow.md) + +## Phase 3: Diagnostic Log & Discussion Cleanup +- [ ] Task: Clean up discussion history and implement Diagnostic Tab. + - [ ] Add `self.diagnostic_log` (a list of transient messages) to `AppController`. + - [ ] Update `src/app_controller.py:_on_performance_alert` to append to `self.diagnostic_log` instead of `disc_entries`. + - [ ] Update `src/ai_client.py` (and other areas) to redirect "SYSTEM WARNING" and similar performance-related messages to the diagnostic log via a new event type. + - [ ] Add a "Diagnostics" tab to the Log Management panel in `src/gui_2.py` to render `self.diagnostic_log`. + - [ ] Ensure `diagnostic_log` is NOT persisted to `manual_slop.toml` or restored during session loads. +- [ ] Task: Conductor - User Manual Verification 'Phase 3: Diagnostic Log & Cleanup' (Protocol in workflow.md) + +## Phase 4: MMA Log Integration & Filtering +- [ ] Task: Improve MMA log visibility and filtering. + - [ ] Ensure MMA sub-agent `log_comms` calls include sufficient metadata (tier, role) for filtering. + - [ ] Update `_render_comms_history_panel` in `src/gui_2.py` to ensure MMA logs are clearly distinct and correctly filtered based on existing UI toggles. +- [ ] Task: Final end-to-end verification of session restoration and log management. +- [ ] Task: Conductor - User Manual Verification 'Phase 4: MMA Log Integration' (Protocol in workflow.md) diff --git a/conductor/tracks/log_session_overhaul_20260308/spec.md b/conductor/tracks/log_session_overhaul_20260308/spec.md new file mode 100644 index 0000000..6b3f6f4 --- /dev/null +++ b/conductor/tracks/log_session_overhaul_20260308/spec.md @@ -0,0 +1,47 @@ +# Specification: Advanced Log Management and Session Restoration + +## Overview +This track focuses on centralizing log management, improving the reliability and scope of session restoration, and optimizing log storage by offloading large data blobs (scripts and tool outputs) to external files. It also aims to "clean" the discussion history by moving transient system warnings to a dedicated diagnostic log. + +## Functional Requirements + +### 1. Centralized Log Management +- Move the "Load Log" functionality from the Comms Log panel to the **Log Management** panel. +- Update the "Load Log" action to load an **entire session** (Comms, MMA Agent logs, and Tool Execution logs) instead of just the Comms log. + +### 2. Session Replay Mode +- When a previous session is loaded, the UI should enter a "Historical/Replay" mode: + - Apply a visual tint to the UI to clearly distinguish it from an active session. + - Populate all respective panels (Discussion, MMA Dashboard, Operation Logs) with the data from the loaded session logs as if they were live. +- Fix the existing broken implementation for loading and parsing historical comms logs. + +### 3. Log Storage Optimization +- **Script Offloading:** AI-generated PowerShell scripts must be saved into the session's directory with a unique identifier naming scheme. +- **Output Offloading:** Output from tool executions (e.g., shell command results) must be saved to separate files within the session directory. +- **ID Referencing:** Log entries in the `.jsonl` files should reference these external files by their filenames instead of embedding the full content. + +### 4. MMA Agent Log Integration +- Ensure MMA sub-agent communications are correctly captured and filterable within the main Comms Log view. +- Utilize existing filter criteria (e.g., role, status) to manage the display of these logs. + +### 5. Diagnostic Log Rework +- Remove system performance warnings and transient notifications from the **Discussion History**. +- Relocate these warnings to a new **Diagnostic Tab** within the Log Management panel. +- **Transient Diagnostics:** Diagnostic logs should not be restored or re-populated during session history loads; they are relevant only to the active runtime environment. + +## Non-Functional Requirements +- **Efficiency:** Significantly reduce the size of individual log files by offloading large content strings. +- **Maintainability:** Decouple log visualization from the main execution logic to ensure robust session replay. + +## Acceptance Criteria +- [ ] The "Load Log" button is successfully relocated and initiates a full session restoration. +- [ ] Loaded sessions correctly tint the UI and populate historical data into their original panels. +- [ ] Comms logs contain filename references for scripts and outputs rather than the raw text. +- [ ] Discussion history remains clear of performance-related system messages. +- [ ] MMA sub-agent logs are visible and correctly filtered in the Comms view. +- [ ] The new Diagnostic Tab correctly displays real-time system warnings. + +## Out of Scope +- Real-time "playback" (animating the session as it happened); only static restoration is required. +- Editing or modifying historical session logs. +- Exporting sessions to formats other than the native log structure.