docs(conductor): Synchronize docs for track 'Advanced Log Management and Session Restoration'

conductor/product.md

@@ -46,6 +46,10 @@ For deep implementation details when planning or implementing tracks, consult `d
 - **Parallel Tool Execution:** Executes independent tool calls (e.g., parallel file reads) concurrently within a single agent turn using an asynchronous execution engine, significantly reducing end-to-end latency.
 - **Automated Tier 4 QA:** Integrates real-time error interception in the shell runner, automatically forwarding technical failures to cheap sub-agents for 20-word diagnostic summaries injected back into the worker history.
 - **Detailed History Management:** Rich discussion history with branching, timestamping, and specific git commit linkage per conversation.
+- **Advanced Log Management:** Optimizes log storage by offloading large data (AI-generated scripts and tool outputs) to unique files within the session directory, using compact `[REF:filename]` pointers in JSON-L logs to minimize token overhead during analysis.
+- **Full Session Restoration:** Allows users to load and reconstruct entire historical sessions from their log directories. Includes a dedicated, tinted **'Historical Replay' mode** that populates discussion history and provides a read-only view of prior agent activities.
+- **Dedicated Diagnostic Logging:** Consolidates transient system warnings and performance alerts into a separate **System Diagnostics** tab within the Log Management panel, ensuring the persistent Discussion History remains focused on high-signal content while providing deep visibility into runtime health.
+- **Improved MMA Observability:** Enhances sub-agent logging by injecting precise ticket IDs and descriptive roles into communication metadata, enabling granular filtering and tracking of parallel worker activities within the Comms History.
 - **In-Depth Toolset Access:** MCP-like file exploration, URL fetching, search, and dynamic context aggregation embedded within a multi-viewport Dear PyGui/ImGui interface.
 - **Integrated Workspace:** A consolidated Hub-based layout (Context, AI Settings, Discussion, Operations) designed for expert multi-monitor workflows.
 - **Session Analysis:** Ability to load and visualize historical session logs with a dedicated tinted "Prior Session" viewing mode.

conductor/tracks.md

@@ -26,13 +26,13 @@ This file tracks all major tracks for the project. Each track has its own detail
    *Goal: Maximize internal state exposure and provide comprehensive control endpoints (worker spawn/kill, pipeline pause/resume, DAG mutation) via the Hook API. Implement WebSocket-based real-time event streaming.*
 
 5. [ ] **Track: Codebase Audit and Cleanup**
-*Link: [./tracks/codebase_audit_20260308/](./tracks/codebase_audit_20260308/)*
+    *Link: [./tracks/codebase_audit_20260308/](./tracks/codebase_audit_20260308/)*
 
 ---
 
 ### GUI Overhauls & Visualizations
 
-1. [~] **Track: Advanced Log Management and Session Restoration**
+3. [x] **Track: Advanced Log Management and Session Restoration**
    *Link: [./tracks/log_session_overhaul_20260308/](./tracks/log_session_overhaul_20260308/)*
    *Goal: Centralize log management, improve session restoration reliability with full-UI replay mode, and optimize log size via external script/output referencing. Implement transient diagnostic logging for system warnings.*
 

conductor/tracks/log_session_overhaul_20260308/plan.md

@@ -22,7 +22,7 @@
 - [ ] Task: Conductor - User Manual Verification 'Phase 2: Session-Level Restoration' (Protocol in workflow.md)
 
 ## Phase 3: Diagnostic Log & Discussion Cleanup
-- [ ] Task: Clean up discussion history and implement Diagnostic Tab.
+- [x] Task: Clean up discussion history and implement Diagnostic Tab. 8e02c1e
     - [ ] Add `self.diagnostic_log` (a list of transient messages) to `AppController`.
     - [ ] Update `src/app_controller.py:_on_performance_alert` to append to `self.diagnostic_log` instead of `disc_entries`.
     - [ ] Update `src/ai_client.py` (and other areas) to redirect "SYSTEM WARNING" and similar performance-related messages to the diagnostic log via a new event type.
@@ -30,9 +30,9 @@
     - [ ] Ensure `diagnostic_log` is NOT persisted to `manual_slop.toml` or restored during session loads.
 - [ ] Task: Conductor - User Manual Verification 'Phase 3: Diagnostic Log & Cleanup' (Protocol in workflow.md)
 
-## Phase 4: MMA Log Integration & Filtering
+## Phase 4: MMA Log Integration & Filtering [checkpoint: c3e0cb3]
-- [ ] Task: Improve MMA log visibility and filtering.
+- [x] Task: Improve MMA log visibility and filtering. c3e0cb3
-    - [ ] Ensure MMA sub-agent `log_comms` calls include sufficient metadata (tier, role) for filtering.
+    - [x] Ensure MMA sub-agent `log_comms` calls include sufficient metadata (tier, role) for filtering.
-    - [ ] Update `_render_comms_history_panel` in `src/gui_2.py` to ensure MMA logs are clearly distinct and correctly filtered based on existing UI toggles.
+    - [x] Update `_render_comms_history_panel` in `src/gui_2.py` to ensure MMA logs are clearly distinct and correctly filtered based on existing UI toggles.
-- [ ] Task: Final end-to-end verification of session restoration and log management.
+- [x] Task: Final end-to-end verification of session restoration and log management. c3e0cb3
 - [ ] Task: Conductor - User Manual Verification 'Phase 4: MMA Log Integration' (Protocol in workflow.md)

src/app_controller.py

@@ -130,7 +130,7 @@ class AppController:
  PROVIDERS: list[str] = ["gemini", "anthropic", "gemini_cli", "deepseek", "minimax"]
 
  def __init__(self):
- # Initialize locks first to avoid initialization order issues
+  # Initialize locks first to avoid initialization order issues
   self._send_thread_lock: threading.Lock = threading.Lock()
   self._disc_entries_lock: threading.Lock = threading.Lock()
   self._pending_comms_lock: threading.Lock = threading.Lock()
@@ -297,6 +297,7 @@ class AppController:
   self._inject_mode: str = "skeleton"
   self._inject_preview: str = ""
   self._show_inject_modal: bool = False
+  self.diagnostic_log: List[Dict[str, Any]] = []
   self._settable_fields: Dict[str, str] = {
    'ai_input': 'ui_ai_input',
    'project_git_dir': 'ui_project_git_dir',
@@ -798,7 +799,6 @@ class AppController:
    "Tool Calls": False,
    "Theme": True,
    "Log Management": False,
-   "Diagnostics": False,
   }
   saved = self.config.get("gui", {}).get("show_windows", {})
   self.show_windows = {k: saved.get(k, v) for k, v in _default_windows.items()}
@@ -1212,13 +1212,11 @@ class AppController:
     self._api_event_queue.append({"type": event_name, "payload": payload})
 
  def _on_performance_alert(self, message: str) -> None:
-  alert_text = f"[PERFORMANCE ALERT] {message}. Please consider optimizing recent changes or reducing load."
+  self.diagnostic_log.append({
-  with self._pending_history_adds_lock:
+    "ts": project_manager.now_ts(),
-   self._pending_history_adds.append({
+    "message": message,
-     "role": "System",
+    "type": "performance"
-     "content": alert_text,
+   })
-     "ts": project_manager.now_ts()
-    })
 
  def _confirm_and_run(self, script: str, base_dir: str, qa_callback: Optional[Callable[[str], str]] = None, patch_callback: Optional[Callable[[str, str], Optional[str]]] = None) -> Optional[str]:
   sys.stderr.write(f"[DEBUG] _confirm_and_run called. test_hooks={self.test_hooks_enabled}, manual_approve={getattr(self, 'ui_manual_approve', False)}\n")

src/gui_2.py

@@ -285,7 +285,7 @@ class App:
      self._flush_to_config()
      models.save_config(self.config)
     except Exception:
-     pass  # silent — don't disrupt the GUI loop
+     pass  # silent — don't disrupt the GUI loop
      # Sync pending comms
    with self._pending_comms_lock:
     if self._pending_comms:
@@ -307,7 +307,7 @@ class App:
     else:
      log_raw = list(self._comms_log)
      if self.ui_focus_agent:
-      self._comms_log_cache = [e for e in log_raw if e.get("source_tier") == self.ui_focus_agent]
+      self._comms_log_cache = [e for e in log_raw if e.get("source_tier", "").startswith(self.ui_focus_agent)]
      else:
       self._comms_log_cache = log_raw
     self._comms_log_dirty = False
@@ -315,7 +315,7 @@ class App:
    if self._tool_log_dirty:
     log_raw = list(self._tool_log)
     if self.ui_focus_agent:
-     self._tool_log_cache = [e for e in log_raw if e.get("source_tier") == self.ui_focus_agent]
+     self._tool_log_cache = [e for e in log_raw if e.get("source_tier", "").startswith(self.ui_focus_agent)]
     else:
      self._tool_log_cache = log_raw
     self._tool_log_dirty = False
@@ -491,81 +491,7 @@ class App:
     if self.perf_profiling_enabled: self.perf_monitor.start_component("_render_log_management")
     self._render_log_management()
     if self.perf_profiling_enabled: self.perf_monitor.end_component("_render_log_management")
-   if self.show_windows["Diagnostics"]:
+
-    exp, opened = imgui.begin("Diagnostics", self.show_windows["Diagnostics"])
-    self.show_windows["Diagnostics"] = bool(opened)
-    if exp:
-     metrics = self.perf_monitor.get_metrics()
-     imgui.text("Performance Telemetry")
-     imgui.same_line()
-     _, self.perf_profiling_enabled = imgui.checkbox("Enable Profiling", self.perf_profiling_enabled)
-     imgui.separator()
-     if imgui.begin_table("perf_table", 3, imgui.TableFlags_.borders_inner_h):
-      imgui.table_setup_column("Metric")
-      imgui.table_setup_column("Value")
-      imgui.table_setup_column("Graph")
-      imgui.table_headers_row()
-      
-      # Core Metrics
-      for label, key, format_str in [
-       ("FPS", "fps", "%.1f"),
-       ("Frame Time (ms)", "frame_time_ms", "%.2f"),
-       ("CPU %", "cpu_percent", "%.1f"),
-       ("Input Lag (ms)", "input_lag_ms", "%.1f")
-      ]:
-       imgui.table_next_row()
-       imgui.table_next_column()
-       imgui.text(label)
-       imgui.table_next_column()
-       if key == "fps":
-        avg_val = imgui.get_io().framerate
-       else:
-        avg_val = metrics.get(f"{key}_avg", metrics.get(key, 0.0))
-       imgui.text(format_str % avg_val)
-       imgui.table_next_column()
-       self.perf_show_graphs.setdefault(key, False)
-       _, self.perf_show_graphs[key] = imgui.checkbox(f"##g_{key}", self.perf_show_graphs[key])
-      
-      imgui.end_table()
-     
-     if self.perf_profiling_enabled:
-      imgui.separator()
-      imgui.text("Detailed Component Timings (Moving Average)")
-      if imgui.begin_table("comp_timings", 3, imgui.TableFlags_.borders):
-       imgui.table_setup_column("Component")
-       imgui.table_setup_column("Avg (ms)")
-       imgui.table_setup_column("Graph")
-       imgui.table_headers_row()
-       # Show all components found in metrics
-       for key, val in metrics.items():
-        if key.startswith("time_") and key.endswith("_ms") and not key.endswith("_avg"):
-         comp_name = key[5:-3]
-         avg_val = metrics.get(f"{key}_avg", val)
-         imgui.table_next_row()
-         imgui.table_next_column()
-         imgui.text(comp_name)
-         imgui.table_next_column()
-         if avg_val > 10.0:
-          imgui.text_colored(imgui.ImVec4(1.0, 0.2, 0.2, 1.0), f"{avg_val:.2f}")
-         else:
-          imgui.text(f"{avg_val:.2f}")
-         imgui.table_next_column()
-         self.perf_show_graphs.setdefault(comp_name, False)
-         _, self.perf_show_graphs[comp_name] = imgui.checkbox(f"##g_{comp_name}", self.perf_show_graphs[comp_name])
-       imgui.end_table()
-      
-     # Render all enabled graphs (core + components)
-     imgui.separator()
-     imgui.text("Performance Graphs")
-     for key, show in self.perf_show_graphs.items():
-      if show:
-       imgui.text(f"History: {key}")
-       hist_data = self.perf_monitor.get_history(key)
-       if hist_data:
-        imgui.plot_lines(f"##plot_{key}", np.array(hist_data, dtype=np.float32), graph_size=imgui.ImVec2(-1, 60))
-       else:
-        imgui.text_disabled(f"(no history data for {key})")
-    imgui.end()
    self.perf_monitor.end_frame()
    # ---- Modals / Popups
    with self._pending_dialog_lock:
@@ -1053,78 +979,168 @@ class App:
   if not exp:
    imgui.end()
    return
-  
-  if self._log_registry is None:
-   self._log_registry = log_registry.LogRegistry(str(paths.get_logs_dir() / "log_registry.toml"))
-  else:
-   # Refresh data occasionally or on demand? For now let's just use the cached object.
-   # The LogRegistry object loads data into self.data upon __init__.
-   # We might want a refresh button or to reload every few seconds.
-   if imgui.button("Refresh Registry"):
-    self._log_registry = log_registry.LogRegistry(str(paths.get_logs_dir() / "log_registry.toml"))
-   imgui.same_line()
-  if imgui.button("Load Log"):
-   self.cb_load_prior_log()
-  imgui.same_line()
-  if imgui.button("Force Prune Logs"):
-   self.controller.event_queue.put("gui_task", {"action": "click", "item": "btn_prune_logs"})
 
-  registry = self._log_registry
+  if imgui.begin_tab_bar("log_mgmt_tabs"):
-  sessions = registry.data
+   if imgui.begin_tab_item("Sessions")[0]:
-  if imgui.begin_table("sessions_table", 7, imgui.TableFlags_.borders | imgui.TableFlags_.row_bg | imgui.TableFlags_.resizable):
+    if self._log_registry is None:
-   imgui.table_setup_column("Session ID")
+     self._log_registry = log_registry.LogRegistry(str(paths.get_logs_dir() / "log_registry.toml"))
-   imgui.table_setup_column("Start Time")
-   imgui.table_setup_column("Star")
-   imgui.table_setup_column("Reason")
-   imgui.table_setup_column("Size (KB)")
-   imgui.table_setup_column("Msgs")
-   imgui.table_setup_column("Actions")
-   imgui.table_headers_row()
-   for session_id, s_data in sessions.items():
-    imgui.table_next_row()
-    imgui.table_next_column()
-    imgui.text(session_id)
-    imgui.table_next_column()
-    imgui.text(s_data.get("start_time", ""))
-    imgui.table_next_column()
-    whitelisted = s_data.get("whitelisted", False)
-    if whitelisted:
-     imgui.text_colored(vec4(255, 215, 0), "YES")
     else:
-     imgui.text("NO")
+     if imgui.button("Refresh Registry"):
-    metadata = s_data.get("metadata") or {}
+      self._log_registry = log_registry.LogRegistry(str(paths.get_logs_dir() / "log_registry.toml"))
-    imgui.table_next_column()
+     imgui.same_line()
-    imgui.text(metadata.get("reason", ""))
+    if imgui.button("Load Log"):
-    imgui.table_next_column()
+     self.cb_load_prior_log()
-    imgui.text(str(metadata.get("size_kb", "")))
-    imgui.table_next_column()
-    imgui.text(str(metadata.get("message_count", "")))
-    imgui.table_next_column()
-    if imgui.button(f"Load##{session_id}"):
-     self.cb_load_prior_log(s_data.get("path"))
     imgui.same_line()
-    if whitelisted:
+    if imgui.button("Force Prune Logs"):
-     if imgui.button(f"Unstar##{session_id}"):
+     self.controller.event_queue.put("gui_task", {"action": "click", "item": "btn_prune_logs"})
-      registry.update_session_metadata(
+
-       session_id,
+    registry = self._log_registry
-       message_count=int(metadata.get("message_count") or 0),
+    sessions = registry.data
-       errors=int(metadata.get("errors") or 0),
+    if imgui.begin_table("sessions_table", 7, imgui.TableFlags_.borders | imgui.TableFlags_.row_bg | imgui.TableFlags_.resizable):
-       size_kb=int(metadata.get("size_kb") or 0),
+     imgui.table_setup_column("Session ID")
-       whitelisted=False,
+     imgui.table_setup_column("Start Time")
-       reason=str(metadata.get("reason") or "")
+     imgui.table_setup_column("Star")
-      )
+     imgui.table_setup_column("Reason")
-    else:
+     imgui.table_setup_column("Size (KB)")
-     if imgui.button(f"Star##{session_id}"):
+     imgui.table_setup_column("Msgs")
-      registry.update_session_metadata(
+     imgui.table_setup_column("Actions")
-       session_id,
+     imgui.table_headers_row()
-       message_count=int(metadata.get("message_count") or 0),
+     for session_id, s_data in sessions.items():
-       errors=int(metadata.get("errors") or 0),
+      imgui.table_next_row()
-       size_kb=int(metadata.get("size_kb") or 0),
+      imgui.table_next_column()
-       whitelisted=True,
+      imgui.text(session_id)
-       reason="Manually whitelisted"
+      imgui.table_next_column()
-      )
+      imgui.text(s_data.get("start_time", ""))
-   imgui.end_table()
+      imgui.table_next_column()
-  
+      whitelisted = s_data.get("whitelisted", False)
+      if whitelisted:
+       imgui.text_colored(vec4(255, 215, 0), "YES")
+      else:
+       imgui.text("NO")
+      metadata = s_data.get("metadata") or {}
+      imgui.table_next_column()
+      imgui.text(metadata.get("reason", ""))
+      imgui.table_next_column()
+      imgui.text(str(metadata.get("size_kb", "")))
+      imgui.table_next_column()
+      imgui.text(str(metadata.get("message_count", "")))
+      imgui.table_next_column()
+      if imgui.button(f"Load##{session_id}"):
+       self.cb_load_prior_log(s_data.get("path"))
+      imgui.same_line()
+      if whitelisted:
+       if imgui.button(f"Unstar##{session_id}"):
+        registry.update_session_metadata(
+         session_id,
+         message_count=int(metadata.get("message_count") or 0),
+         errors=int(metadata.get("errors") or 0),
+         size_kb=int(metadata.get("size_kb") or 0),
+         whitelisted=False,
+         reason=str(metadata.get("reason") or "")
+        )
+      else:
+       if imgui.button(f"Star##{session_id}"):
+        registry.update_session_metadata(
+         session_id,
+         message_count=int(metadata.get("message_count") or 0),
+         errors=int(metadata.get("errors") or 0),
+         size_kb=int(metadata.get("size_kb") or 0),
+         whitelisted=True,
+         reason="Manually whitelisted"
+        )
+     imgui.end_table()
+    imgui.end_tab_item()
+
+   if imgui.begin_tab_item("System Diagnostics")[0]:
+    metrics = self.perf_monitor.get_metrics()
+    imgui.text("Performance Telemetry")
+    imgui.same_line()
+    _, self.perf_profiling_enabled = imgui.checkbox("Enable Profiling", self.perf_profiling_enabled)
+    imgui.separator()
+    
+    if imgui.begin_table("perf_table", 3, imgui.TableFlags_.borders_inner_h):
+     imgui.table_setup_column("Metric")
+     imgui.table_setup_column("Value")
+     imgui.table_setup_column("Graph")
+     imgui.table_headers_row()
+     
+     for label, key, format_str in [
+      ("FPS", "fps", "%.1f"),
+      ("Frame Time (ms)", "frame_time_ms", "%.2f"),
+      ("CPU %", "cpu_percent", "%.1f"),
+      ("Input Lag (ms)", "input_lag_ms", "%.1f")
+     ]:
+      imgui.table_next_row()
+      imgui.table_next_column()
+      imgui.text(label)
+      imgui.table_next_column()
+      if key == "fps":
+       avg_val = imgui.get_io().framerate
+      else:
+       avg_val = metrics.get(f"{key}_avg", metrics.get(key, 0.0))
+      imgui.text(format_str % avg_val)
+      imgui.table_next_column()
+      self.perf_show_graphs.setdefault(key, False)
+      _, self.perf_show_graphs[key] = imgui.checkbox(f"##g_{key}", self.perf_show_graphs[key])
+     imgui.end_table()
+
+    if self.perf_profiling_enabled:
+     imgui.separator()
+     imgui.text("Detailed Component Timings (Moving Average)")
+     if imgui.begin_table("comp_timings", 3, imgui.TableFlags_.borders):
+      imgui.table_setup_column("Component")
+      imgui.table_setup_column("Avg (ms)")
+      imgui.table_setup_column("Graph")
+      imgui.table_headers_row()
+      for key, val in metrics.items():
+       if key.startswith("time_") and key.endswith("_ms") and not key.endswith("_avg"):
+        comp_name = key[5:-3]
+        avg_val = metrics.get(f"{key}_avg", val)
+        imgui.table_next_row()
+        imgui.table_next_column()
+        imgui.text(comp_name)
+        imgui.table_next_column()
+        if avg_val > 10.0:
+         imgui.text_colored(imgui.ImVec4(1.0, 0.2, 0.2, 1.0), f"{avg_val:.2f}")
+        else:
+         imgui.text(f"{avg_val:.2f}")
+        imgui.table_next_column()
+        self.perf_show_graphs.setdefault(comp_name, False)
+        _, self.perf_show_graphs[comp_name] = imgui.checkbox(f"##g_{comp_name}", self.perf_show_graphs[comp_name])
+      imgui.end_table()
+
+    imgui.separator()
+    imgui.text("Performance Graphs")
+    for key, show in self.perf_show_graphs.items():
+     if show:
+      imgui.text(f"History: {key}")
+      hist_data = self.perf_monitor.get_history(key)
+      if hist_data:
+       import numpy as np
+       imgui.plot_lines(f"##plot_{key}", np.array(hist_data, dtype=np.float32), graph_size=imgui.ImVec2(-1, 60))
+      else:
+       imgui.text_disabled(f"(no history data for {key})")
+
+    imgui.separator()
+    imgui.text("Diagnostic Log")
+    if imgui.begin_table("diag_log_table", 3, imgui.TableFlags_.borders | imgui.TableFlags_.row_bg | imgui.TableFlags_.resizable):
+     imgui.table_setup_column("Timestamp", imgui.TableColumnFlags_.width_fixed, 150)
+     imgui.table_setup_column("Type", imgui.TableColumnFlags_.width_fixed, 100)
+     imgui.table_setup_column("Message")
+     imgui.table_headers_row()
+     for entry in reversed(self.controller.diagnostic_log):
+      imgui.table_next_row()
+      imgui.table_next_column()
+      imgui.text(entry.get("ts", ""))
+      imgui.table_next_column()
+      imgui.text(entry.get("type", ""))
+      imgui.table_next_column()
+      imgui.text_wrapped(entry.get("message", ""))
+     imgui.end_table()
+
+    imgui.end_tab_item()
+   imgui.end_tab_bar()
+
   if self.perf_profiling_enabled: self.perf_monitor.end_component("_render_log_management")
   imgui.end()
 
@@ -1852,6 +1868,10 @@ class App:
     imgui.text_colored(C_LBL, f"#{i_display}")
     imgui.same_line()
     imgui.text_colored(vec4(160, 160, 160), ts)
+    ticket_id = entry.get("mma_ticket_id")
+    if ticket_id:
+     imgui.same_line()
+     imgui.text_colored(vec4(255, 120, 120), f"[{ticket_id}]")
     imgui.same_line()
     d_col = DIR_COLORS.get(direction, C_VAL)
     imgui.text_colored(d_col, direction)

src/multi_agent_conductor.py

@@ -498,6 +498,7 @@ def run_worker_lifecycle(ticket: Ticket, context: WorkerContext, context_files:
 
  old_comms_cb = ai_client.comms_log_callback
  def worker_comms_callback(entry: dict) -> None:
+  entry["mma_ticket_id"] = ticket.id
   if event_queue:
    kind = entry.get("kind")
    payload = entry.get("payload", {})
@@ -515,7 +516,7 @@ def run_worker_lifecycle(ticket: Ticket, context: WorkerContext, context_files:
    old_comms_cb(entry)
 
  ai_client.comms_log_callback = worker_comms_callback
- ai_client.set_current_tier("Tier 3")
+ ai_client.set_current_tier(f"Tier 3 (Worker): {ticket.id}")
  try:
   comms_baseline = len(ai_client.get_comms_log())
   response = ai_client.send(