chore(conductor): Add new track 'Optimization pass for Data-Oriented Python heuristics'

conductor/tracks.md

@@ -214,3 +214,9 @@ This file tracks all major tracks for the project. Each track has its own detail
 - [x] **Track: Robust Live Simulation Verification**
 
 ---
+
+---
+
+- [ ] **Track: Optimization pass for Data-Oriented Python heuristics**
+*Link: [./tracks/data_oriented_optimization_20260312/](./tracks/data_oriented_optimization_20260312/)*
+

conductor/tracks/data_oriented_optimization_20260312/index.md

@@ -0,0 +1,5 @@
+# Track data_oriented_optimization_20260312 Context
+
+- [Specification](./spec.md)
+- [Implementation Plan](./plan.md)
+- [Metadata](./metadata.json)

conductor/tracks/data_oriented_optimization_20260312/metadata.json

@@ -0,0 +1,8 @@
+{
+  "track_id": "data_oriented_optimization_20260312",
+  "type": "chore",
+  "status": "new",
+  "created_at": "2026-03-12T00:00:00Z",
+  "updated_at": "2026-03-12T00:00:00Z",
+  "description": "Optimization pass. I want to update the product guidlines to take into account with data-oriented appraoch the more performant way to semantically define procedrual code in python so executes almost entirely heavy operations optimally. I know there is a philosophy of 'the less python does the better' which is problably why the imgui lib is so performant because all python really does is define the ui's DAG via an imgui interface procedurally along with what state the dag may modify within its constraints of interactions the user may do. This problably can be reflected in the way the rest of the codebase is done. I want to go over the ./src and ./simulation to make sure this insight and related herustics are properly enfroced. Worst case I want to identify what code I should consider lower down to C maybe and making python bindings to if there is a significant bottleneck identified via profiling and testing that cannot be resolved otherwise."
+}

conductor/tracks/data_oriented_optimization_20260312/plan.md

@@ -0,0 +1,27 @@
+# Implementation Plan: Data-Oriented Python Optimization Pass
+
+## Phase 1: Guidelines and Instrumentation
+- [ ] Task: Update `conductor/product-guidelines.md` with Data-Oriented Python heuristics and the "less Python does the better" philosophy.
+- [ ] Task: Review existing profiling instrumentation in `src/performance_monitor.py` or diagnostic hooks.
+- [ ] Task: Expand profiling instrumentation to capture more detailed execution times for non-GUI data structures/processes if necessary.
+- [ ] Task: Conductor - User Manual Verification 'Phase 1: Guidelines and Instrumentation' (Protocol in workflow.md)
+
+## Phase 2: Audit and Profiling (`src/` and `simulation/`)
+- [ ] Task: Run profiling scenarios (especially utilizing simulations) to generate baseline metrics.
+- [ ] Task: Audit `src/` (e.g., `dag_engine.py`, `multi_agent_conductor.py`, `aggregate.py`) against the new guidelines, cross-referencing with profiling data to identify bottlenecks.
+- [ ] Task: Audit `simulation/` files against the new guidelines to ensure the test harness is performant and non-blocking.
+- [ ] Task: Compile a list of identified bottleneck targets to refactor.
+- [ ] Task: Conductor - User Manual Verification 'Phase 2: Audit and Profiling (`src/` and `simulation/`)' (Protocol in workflow.md)
+
+## Phase 3: Targeted Optimization and Refactoring
+- [ ] Task: Write/update tests for the first identified bottleneck to establish a performance or structural baseline (Red Phase).
+- [ ] Task: Refactor the first identified bottleneck to align with data-oriented guidelines (Green Phase).
+- [ ] Task: Write/update tests for remaining identified bottlenecks.
+- [ ] Task: Refactor remaining identified bottlenecks.
+- [ ] Task: Conductor - User Manual Verification 'Phase 3: Targeted Optimization and Refactoring' (Protocol in workflow.md)
+
+## Phase 4: Final Evaluation and Documentation
+- [ ] Task: Re-run all profiling scenarios to compare against the baseline metrics.
+- [ ] Task: Analyze remaining bottlenecks that did not reach performance thresholds and document them as candidates for C/C++ bindings (Last Resort).
+- [ ] Task: Generate a final summary report of the optimizations applied and the C extension evaluation.
+- [ ] Task: Conductor - User Manual Verification 'Phase 4: Final Evaluation and Documentation' (Protocol in workflow.md)

conductor/tracks/data_oriented_optimization_20260312/spec.md

@@ -0,0 +1,35 @@
+# Specification: Data-Oriented Python Optimization Pass
+
+## Overview
+Perform an optimization pass and audit across the codebase (`./src` and `./simulation`), aligning the implementation with the Data-Oriented Design philosophy and the "less Python does the better" heuristic. Update the `product-guidelines.md` to formally document this approach for procedural Python code.
+
+## Functional Requirements
+1. **Update Product Guidelines:**
+   - Formalize the heuristic that Python should act primarily as a procedural semantic definer (similar to how ImGui defines a UI DAG), delegating heavy lifting.
+   - Enforce data-oriented guidelines for Python code structure, focusing on minimizing Python JIT overhead.
+2. **Codebase Audit (`./src` and `./simulation`):**
+   - Review global `src/` files and simulation logic against the new guidelines.
+   - Identify bottlenecks that violate these heuristics (e.g., heavy procedural state manipulation in Python).
+3. **Profiling & Instrumentation Expansion:**
+   - Expand existing profiling instrumentation (e.g., `performance_monitor.py` or diagnostic hooks) if currently insufficient for identifying real structural bottlenecks.
+4. **Optimization Execution:**
+   - Refactor identified bottlenecks to align with the new data-oriented Python heuristics.
+   - Re-evaluate performance post-refactor.
+5. **C Extension Evaluation (Last Resort):**
+   - If Python optimizations fail to meet performance thresholds, specifically identify and document routines that must be lowered to C/C++ with Python bindings. Only proceed with bindings if absolutely necessary.
+
+## Non-Functional Requirements
+- Maintain existing test coverage and strict type-hinting requirements.
+- Ensure 1-space indentation and ultra-compact style rules are not violated during refactoring.
+- Ensure the main GUI rendering thread is never blocked.
+
+## Acceptance Criteria
+- `product-guidelines.md` is updated with data-oriented procedural Python guidelines.
+- `src/` and `simulation/` undergo a documented profiling audit.
+- Identified bottlenecks are refactored to reduce Python overhead.
+- No regressions in automated simulation or unit tests.
+- A final report is provided detailing optimizations made and any candidates for future C extension porting.
+
+## Out of Scope
+- Actually implementing C/C++ bindings in this track (this track only identifies/evaluates them as a last resort; if needed, they get a separate track).
+- Major UI visual theme changes.