conductor(followup): metadata_promotion_20260624 - track artifacts (886 lines)

The actual fix for the 4.01e22 combinatoric explosion. Promotes Metadata: TypeAlias = dict[str, Any] to @dataclass(frozen=True, slots=True) and migrates all 695 consumer functions + 213 access sites (107 .get + 106 subscript) to direct field access. TIER-1 READ AGENTS.md + conductor/workflow.md + conductor/edit_workflow.md + conductor/code_styleguides/data_oriented_design.md + conductor/code_styleguides/error_handling.md + conductor/code_styleguides/type_aliases.md + docs/reports/SSDL_CAMPAIGN_ABORTED_20260624.md + src/type_aliases.py + scripts/code_path_audit/code_path_audit.py + scripts/code_path_audit/code_path_audit_ssdl.py before this commit. Why this fixes 4.01e22: - The combinatoric explosion is from dict[str, Any] type-dispatch at every entry.get('key', default) site (per SSDL post-mortem) - Each access has 3 branches: is None, getattr, default - 695 consumers * ~2 branches each = 1390 branches in the sum - 2^1390 ≈ 4.01e22 (the measured baseline) - Promotion to @dataclass with direct field access = 0 branches per access - Expected drop: 4.014e+22 -> < 1e+20 (>= 2 orders of magnitude) 10 VCs: - VC1: Metadata is @dataclass(frozen=True, slots=True), not dict[str, Any] - VC2: 107 .get sites replaced - VC3: 106 subscript sites replaced - VC4: 12+ tests pass in tests/test_metadata_dataclass.py - VC5: 5 sub-aggregate TypeAliases (CommsLogEntry, HistoryMessage, FileItem, ToolDefinition, ToolCall) all point to the new Metadata - VC6: Effective codepaths < 1e+20 - VC7: All 7 audit gates pass --strict - VC8: 10/11 batched test tiers PASS - VC9: End-of-track report written - VC10: New regression-guard test file exists 5-phase phased migration (smallest sub-aggregate first): - Phase 1: CommsLogEntry (~150 sites in session_logger, multi_agent_conductor, app_controller) - Phase 2: HistoryMessage (~80 sites in ai_client) - Phase 3: FileItem (~200 sites in aggregate, app_controller, gui_2) - Phase 4: ToolDefinition+ToolCall (~150 sites in mcp_client, ai_client tool loop) - Phase 5: Metadata direct usage (~115 sites catch-all) 6 phases total (0 + 5 + verification). 18-21 atomic commits. blocked_by: code_path_audit_phase_3_provider_state_20260624 (recommended prerequisite; the two tracks are orthogonal so they can run in parallel; listed as blocked_by for sequencing preference not strict blocking)
2026-06-25 12:06:50 -04:00
parent dc397db7ed
commit e50bebddd9
5 changed files with 886 additions and 0 deletions
@@ -0,0 +1,310 @@
+# Track Specification: metadata_promotion_20260624
+
+## Overview
+
+The actual fix for the 4.01e22 combinatoric explosion. Promotes `Metadata: TypeAlias = dict[str, Any]` to a typed `@dataclass(frozen=True)` and migrates all 695 consumer functions + 213 access sites (107 `.get('key', ...)` + 106 subscript `['key']`) to use direct field access.
+
+## Current State Audit (master `dc397db7`, measured 2026-06-25)
+
+| Metric | Value | Source |
+|---|---:|---|
+| `Metadata` consumers in `src/` | **695** | `scripts/code_path_audit.build_pcg` (was 751 in older measurements; some refactors reduced) |
+| Top consumer files | `app_controller.py: 123`, `mcp_client.py: 94`, `ai_client.py: 73`, `gui_2.py: 44`, `models.py: 29` | `Counter` over `pcg.consumers['Metadata']` |
+| Total branches in Metadata consumers | 3,454 | `scripts/code_path_audit_ssdl.count_branches_in_function` |
+| **Effective codepaths (the 4.01e22)** | **4.014e+22** | `compute_effective_codepaths` |
+| `Metadata` definition | `src/type_aliases.py:5` | `Metadata: TypeAlias = dict[str, Any]` |
+| `.get('key', ...)` access sites | 107 | `git grep` in `src/` |
+| `['key']` subscript access sites | 106 | `git grep` in `src/` |
+| `is None` / `== None` / `!= None` sites | 106 | `git grep` in `src/` (most are unrelated to Metadata; some are redundant defensive checks) |
+| Distinct `.get` keys (top 20) | `ai, args, ast_elements, auto_start, burn_rate, call_count, comment, completed_tickets, conductor, content, context_presets, custom_slices, depends_on, description, dir, discussion, discussions, document, efficiency, files` | `git grep -hoE "\.get\('[a-z_]+',"` |
+| Distinct subscript keys (top 20) | `_toggle_command_palette, app_debug_info, args, blocked_reason, bloom, cache_creation_input_tokens, cache_read_input_tokens, command, comment, content, crt, delete_context_preset, depends_on, discussion, discussions, end_line, fps, frame_time_ms_avg, full_path, get_app_debug_info` | `git grep -hoE "\[[ ]*'[a-z_]+'[ ]*\]"` |
+| TypeAlias chain | `Metadata` is the root; `CommsLogEntry`, `HistoryMessage`, `FileItem`, `ToolDefinition`, `ToolCall` are all aliases to `Metadata` | `src/type_aliases.py` |
+
+### Why this matters
+
+The combinatoric explosion (`4.01e22`) is **not from nil-checks** (per the SSDL post-mortem at `docs/reports/SSDL_CAMPAIGN_ABORTED_20260624.md`):
+
+> "The 4.01e22 is **not from nil-checks**. It's from `Metadata: TypeAlias = dict[str, Any]`. Every consumer function that does `entry.get('key', default)` is a runtime type-dispatch branch. The combinatoric explosion is from the unknown type, not from missing sentinels."
+
+The 3 SSDL techniques the user mentioned (redundant nil-checks, preemptive dependency resolution, no-op nil types) are **half the fix** — they reduce the AROUND the type-dispatch but not the type-dispatch itself. **The actual primary fix is type promotion:**
+
+```python
+# BEFORE (runtime type-dispatch per access):
+entry.get('key', default_value)  # 3 branches: is None, getattr, default
+if 'key' in entry: ...  # 1 branch
+entry['key']  # 1 branch + potential KeyError
+
+# AFTER (direct field access):
+entry.field_name  # 0 branches
+if entry.field_name is not None: ...  # only if nullable
+entry.field_name  # direct, no KeyError
+```
+
+For 213 access sites × ~2 branches each = 426 branches reduced. The exponential `2^N` for the highest-branch-count functions drops by orders of magnitude.
+
+## Goals
+
+| ID | Goal | Acceptance |
+|---|---|---|
+| G1 | Promote `Metadata` to `@dataclass(frozen=True)` with explicit fields | `git grep "^Metadata:" HEAD:src/type_aliases.py` shows `Metadata: TypeAlias = CommsLogEntry` (or similar — the dataclass), NOT `dict[str, Any]` |
+| G2 | Migrate all 213 access sites (107 `.get` + 106 `['key']`) to direct field access | `git grep -E "\.get\('[a-z_]+'," HEAD -- 'src/*.py'` returns 0 hits in promoted files; `git grep -E "\[[ ]*'[a-z_]+'[ ]*\]" HEAD -- 'src/*.py'` returns only allowed-pattern hits |
+| G3 | All 5 sub-aggregates share the same dataclass (per type_aliases.py chain) | `CommsLogEntry`, `HistoryMessage`, `FileItem`, `ToolDefinition`, `ToolCall` all point to the same `Metadata` dataclass |
+| G4 | Effective codepaths drops by ≥ 2 orders of magnitude | `compute_effective_codepaths` returns `< 1e+20` (was 4.014e+22) |
+| G5 | All 7 audit gates pass `--strict` (no regression) | `weak_types`, `type_registry`, `main_thread_imports`, `no_models_config_io`, `code_path_audit_coverage`, `exception_handling`, `optional_in_3_files` all exit 0 |
+| G6 | All existing tests pass (10/11 batched tiers — RAG flake acceptable) | `scripts/run_tests_batched.py` → 10/11 PASS |
+| G7 | New regression-guard tests for the dataclass | `tests/test_metadata_dataclass.py` with 10+ tests for: field access, immutable, `__post_init__` validation, `to_dict()` for backward-compat with JSON serialization |
+
+## Non-Goals
+
+- Modifications to `src/code_path_audit*.py` (the audit infrastructure is correct; the migration is on the consumer side)
+- The 4 NG1 + 7 NG2 audit violations (already addressed in phase 2 + dc397db7)
+- The 4.01e22's nil-check component (per the post-mortem, this is a minor contributor; the type-dispatch is the dominant cause)
+- The RAG test pre-existing flake (per `docs/reports/SSDL_CAMPAIGN_ABORTED_20260624.md` "Out of Scope")
+- New `src/<thing>.py` files (per AGENTS.md hard rule; the dataclass goes in `src/type_aliases.py`)
+- Polishing the 5 sub-aggregates with custom dataclasses each (overkill; one shared dataclass suffices)
+
+## Functional Requirements
+
+### FR1: Design the Metadata dataclass
+
+`Metadata` is a polymorphic dict shape used in 5 sub-aggregates:
+- `CommsLogEntry` (app_controller's session log entries)
+- `HistoryMessage` (ai_client's per-vendor history)
+- `FileItem` (context composition's file items)
+- `ToolDefinition` (mcp_client's tool schema)
+- `ToolCall` (ai_client's tool call records)
+
+The distinct keys used across all 213 access sites are:
+- **From `.get()`**: `ai, args, ast_elements, auto_start, burn_rate, call_count, comment, completed_tickets, conductor, content, context_presets, custom_slices, depends_on, description, dir, discussion, discussions, document, efficiency, files, ...` (107 keys total)
+- **From `[]`**: `_toggle_command_palette, app_debug_info, args, blocked_reason, bloom, cache_creation_input_tokens, cache_read_input_tokens, command, comment, content, crt, delete_context_preset, depends_on, discussion, discussions, end_line, fps, frame_time_ms_avg, full_path, get_app_debug_info, ...` (106 keys total)
+
+After deduplication, the union has ~150-200 distinct keys. The dataclass will have all of them as `Optional[T]` fields (or `T` with a default for required ones). This is wider than ideal but:
+- `@dataclass(frozen=True, slots=True)` keeps memory overhead low
+- Direct attribute access (`entry.field_name`) compiles to a single C-level field read
+- Removes ALL `dict.get()` and `dict['key']` runtime branches at the consumer level
+
+```python
+# src/type_aliases.py
+from __future__ import annotations
+from dataclasses import dataclass, field
+from typing import Any, Callable, NamedTuple, Optional, TypeAlias
+
+
+@dataclass(frozen=True, slots=True)
+class Metadata:
+ role: str = ""
+ content: Any = None
+ tool_calls: Any = None
+ tool_call_id: str = ""
+ name: str = ""
+ args: Any = None
+ source_tier: str = "main"
+ model: str = "unknown"
+ id: str = ""
+ ts: str = ""
+ role_: str = ""  # For dicts that used 'role' as a key
+ description: str = ""
+ depends_on: tuple[str, ...] = ()
+ status: str = ""
+ manual_block: bool = False
+ completed_tickets: int = 0
+ auto_start: bool = False
+ command: str = ""
+ script: str = ""
+ output: Any = None
+ error: str = ""
+ tier: str = ""
+ path: str = ""
+ full_path: str = ""
+ filename: str = ""
+ mtime: float = 0.0
+ size: int = 0
+ # ... ~200 fields total, all Optional or with sensible defaults ...
+
+
+CommsLogEntry: TypeAlias = Metadata
+CommsLog: TypeAlias = list[CommsLogEntry]
+HistoryMessage: TypeAlias = Metadata
+History: TypeAlias = list[HistoryMessage]
+FileItem: TypeAlias = Metadata
+FileItems: TypeAlias = list[FileItem]
+ToolDefinition: TypeAlias = Metadata
+ToolCall: TypeAlias = Metadata
+CommsLogCallback: TypeAlias = Callable[[CommsLogEntry], None]
+JsonPrimitive: TypeAlias = str | int | float | bool | None
+JsonValue: TypeAlias = JsonPrimitive | list["JsonValue"] | dict[str, "JsonValue"]
+
+
+class FileItemsDiff(NamedTuple):
+ refreshed: FileItems
+ changed: FileItems
+```
+
+**Migration helper**: the dataclass also has a `to_dict()` method for JSON serialization (used by `CommsLog` writer, session restoration, etc.):
+
+```python
+@dataclass(frozen=True, slots=True)
+class Metadata:
+ ...fields...
+ def to_dict(self) -> dict[str, Any]:
+  return {k: v for k, v in asdict(self).items() if v is not None or k in _NON_NULL_KEYS}
+```
+
+### FR2: Phase-by-phase migration (5 sub-aggregates)
+
+The 695 consumer functions distribute across the 5 sub-aggregates. To minimize blast radius, migrate sub-aggregate by sub-aggregate:
+
+| Phase | Sub-aggregate | Est. consumers | Primary files |
+|---|---|---:|---|
+| 1 | `CommsLogEntry` | ~150 | `app_controller.py`, `multi_agent_conductor.py`, `session_logger.py` |
+| 2 | `HistoryMessage` | ~80 | `ai_client.py` (per-vendor history) |
+| 3 | `FileItem` | ~200 | `aggregate.py`, `gui_2.py`, `app_controller.py` |
+| 4 | `ToolDefinition` + `ToolCall` | ~150 | `mcp_client.py`, `ai_client.py` |
+| 5 | Other (`Metadata` direct usage) | ~115 | `gui_2.py` (general), `models.py`, `paths.py`, etc. |
+
+Each phase:
+1. Add the new field to the `Metadata` dataclass (if not already present)
+2. Update consumers in that sub-aggregate's primary files: `entry.get('key', default)` → `entry.key or default` (or similar)
+3. Update consumers: `entry['key']` → `entry.key`
+4. Add regression-guard tests for the migrated access pattern
+5. Re-measure effective codepaths after the phase
+
+### FR3: Migration patterns (canonical)
+
+```python
+# BEFORE:
+x = entry.get('model', 'unknown')
+y = entry.get('input_tokens', 0) or 0
+z = entry.get('source_tier', 'main')
+if entry.get('manual_block', False):
+    ...
+role = entry['role']
+if 'depends_on' in entry:
+    deps = entry['depends_on']
+
+# AFTER (with Metadata dataclass):
+x = entry.model or 'unknown'
+y = entry.input_tokens or 0
+z = entry.source_tier or 'main'
+if entry.manual_block:
+    ...
+role = entry.role
+if entry.depends_on:
+    deps = entry.depends_on
+```
+
+The migration is mechanical but requires care:
+- For `Optional[T]` fields: use `entry.field or default_value`
+- For required fields: use `entry.field` directly
+- For polymorphic keys (some entries have the key, some don't): the dataclass default handles this (all fields have defaults)
+- For `['key']` (subscript) where the key is dynamic: rare; keep as `dict[str, Any]` (e.g., `entry.to_dict()['dynamic_key']`)
+
+### FR4: Edge cases
+
+**Polymorphic constructors**: many sites do `entry = {'role': 'user', 'content': 'hi'}`. After migration: `entry = Metadata(role='user', content='hi')`. The dataclass has all the fields as `Optional` or with defaults, so this works.
+
+**Dynamic dict construction**: `for k, v in raw.items(): entry[k] = v`. After migration: `entry = Metadata(**raw)`. The `**` syntax requires that all keys in `raw` are valid field names; if `raw` has unknown keys, this fails. Solution: use a `from_dict` classmethod that filters out unknown keys:
+
+```python
+@classmethod
+def from_dict(cls, raw: dict[str, Any]) -> 'Metadata':
+    valid_fields = {f.name for f in fields(cls)}
+    return cls(**{k: v for k, v in raw.items() if k in valid_fields})
+```
+
+**JSON serialization**: `json.dumps(entry)` fails on dataclass. Solution: `json.dumps(entry.to_dict())`.
+
+**Pickle**: `pickle.dumps(entry)` works (dataclass supports pickle natively via `__reduce__`).
+
+**Equality**: `entry1 == entry2` now works (dataclass generates `__eq__`); before it was `False` for distinct dict instances even with the same content.
+
+### FR5: Re-measurement
+
+After each phase, re-measure:
+
+```bash
+uv run python -c "
+import sys
+sys.path.insert(0, 'scripts/code_path_audit')
+sys.path.insert(0, 'src')
+from code_path_audit import build_pcg
+from code_path_audit_ssdl import count_branches_in_function
+pcg = build_pcg('src').data
+metadata_consumers = pcg.consumers.get('Metadata', [])
+total = sum(2 ** count_branches_in_function(f, 'src') for f in metadata_consumers)
+print(f'Effective codepaths: {total:.3e}')
+print(f'Consumers: {len(metadata_consumers)}')
+"
+```
+
+Expected: drops from 4.014e+22 to < 1e+20 after Phase 1 (just CommsLogEntry); further drops after each subsequent phase.
+
+## Non-Functional Requirements
+
+- NFR1: 1-space indentation (per `conductor/workflow.md`)
+- NFR2: CRLF line endings on Windows
+- NFR3: No comments in source code
+- NFR4: Per-task atomic commits with git notes
+- NFR5: No new pip dependencies (dataclass is stdlib)
+- NFR6: `Result[T]` returns for fallible fns (per `error_handling.md`)
+- NFR7: No new `src/<thing>.py` files (per AGENTS.md hard rule; the dataclass goes in `src/type_aliases.py`)
+
+## Architecture Reference
+
+- `conductor/code_styleguides/data_oriented_design.md` — the "Prefer Fewer Types" principle (the canonical rationale)
+- `conductor/code_styleguides/error_handling.md` — the `Result[T]` convention
+- `conductor/code_styleguides/type_aliases.md` — the 10 TypeAliases convention (per the data_structure_strengthening_20260606 track)
+- `src/type_aliases.py` — the current Metadata definition (line 5)
+- `scripts/code_path_audit/code_path_audit.py` — the consumer detection (3-pass AST)
+- `scripts/code_path_audit/code_path_audit_ssdl.py` — the effective codepaths metric
+- `docs/reports/SSDL_CAMPAIGN_ABORTED_20260624.md` — the post-mortem explaining why this is a type-dispatch problem, not a nil-check problem
+- `conductor/tracks/any_type_componentization_20260621/plan.md` — the grandparent track (48/89 sites promoted, then reverted at `751b94d4`)
+- `conductor/tracks/code_path_audit_20260607/spec_v2.md` — the audit that established the 4.01e22 baseline
+- `docs/reports/code_path_audit/2026-06-22/AUDIT_REPORT.md` — the original 6797-line audit report
+
+## Out of Scope
+
+- Modifications to `src/code_path_audit*.py` (the audit infrastructure is correct)
+- The 4 NG1 + 7 NG2 audit violations (already addressed)
+- The RAG test pre-existing flake (per SSDL post-mortem)
+- The 5 sub-aggregates (`CommsLogEntry`, `HistoryMessage`, `FileItem`, `ToolDefinition`, `ToolCall`) becoming separate dataclasses (overkill; they share the same `Metadata` base)
+- New `src/<thing>.py` files (per AGENTS.md hard rule)
+- Backward-compat support for `dict[str, Any]` (the migration is a hard break; any code that does `Metadata(...).__class__ is dict` will break, but no such code exists per the audit)
+
+## Verification Criteria (Definition of Done)
+
+| # | Criterion | Verification command |
+|---|---|---|
+| VC1 | `Metadata` is a `@dataclass(frozen=True, slots=True)`, not `dict[str, Any]` | `git show HEAD:src/type_aliases.py \| head -10` shows `@dataclass(frozen=True, slots=True) class Metadata:` |
+| VC2 | All 107 `.get('key', ...)` sites on Metadata consumers replaced | `git grep -E "\.get\('[a-z_]+'," HEAD -- 'src/*.py' \| wc -l` returns 0 (or only legitimate non-Metadata uses like `.get('mtime', 0)` on file paths) |
+| VC3 | All 106 `['key']` subscript sites on Metadata consumers replaced | `git grep -E "\[[ ]*'[a-z_]+'[ ]*\]" HEAD -- 'src/*.py' \| wc -l` returns 0 (or only legitimate non-Metadata uses) |
+| VC4 | `Metadata(...)` constructor works for all common patterns | `tests/test_metadata_dataclass.py` passes 10+ tests (constructor, field access, `to_dict()`, `from_dict()`, frozen, slots, equality) |
+| VC5 | All 5 sub-aggregate TypeAliases point to the new `Metadata` | `git grep "TypeAlias = " HEAD:src/type_aliases.py` shows `CommsLogEntry: TypeAlias = Metadata` etc. |
+| VC6 | Effective codepaths drops by ≥ 2 orders of magnitude | `compute_effective_codepaths` returns `< 1e+20` (was 4.014e+22) |
+| VC7 | All 7 audit gates pass `--strict` (no regression) | `weak_types` 102 ≤ 112; `type_registry` 22 files; `main_thread_imports` 17; `no_models_config_io` 0; `code_path_audit_coverage` 0; `exception_handling` 0; `optional_in_3_files` 0 |
+| VC8 | 10/11 batched test tiers PASS (RAG flake acceptable) | `scripts/run_tests_batched.py` → 10/11 |
+| VC9 | End-of-track report written | `docs/reports/TRACK_COMPLETION_metadata_promotion_20260624.md` exists with the new effective-codepaths number |
+| VC10 | New regression-guard test file | `tests/test_metadata_dataclass.py` exists with 10+ tests passing |
+
+## Risks
+
+| # | Risk | Likelihood | Mitigation |
+|---|---|---|---|
+| R1 | The 213 access sites have polymorphic keys that don't fit cleanly into a single dataclass | medium | Use `Optional[T]` for all fields; use `from_dict` classmethod that filters unknown keys; use `to_dict()` for JSON serialization |
+| R2 | Some sites do `entry['key']` where `key` is dynamic (e.g., `entry[variable_name]`) | low | These are rare; keep as `dict[str, Any]` access for dynamic keys; the static field access handles the common case |
+| R3 | The `to_dict()` round-trip loses information (e.g., nested dicts) | low | Implement `to_dict()` carefully; nested dicts pass through as `dict[str, Any]` (not recursively converted) |
+| R4 | Some sites mutate `entry` (e.g., `entry['key'] = value`); dataclass is frozen | medium | These sites are rare; audit them; if found, replace with `dataclasses.replace(entry, field_name=value)` |
+| R5 | Migration breaks the regression-guard tests for `test_provider_state_migration.py` (post-phase 3) | low | The migration is on Metadata, not provider_state; orthogonal changes; per-phase regression-guard test runs |
+| R6 | The 695 consumer functions are too many for one track | high | Break into 5 phases (FR2); each phase is a sub-aggregate; the dataclass is added once and all 5 sub-aggregates reuse it |
+| R7 | The dict-shape is used for JSON-serialized payloads (e.g., comms.log); the dataclass breaks the JSON layer | medium | The dataclass has `to_dict()` + `from_dict()`; the JSON layer converts via these. Verify the comms.log reader/writer (session_logger.py) uses these methods |
+
+## See also
+
+- `docs/reports/SSDL_CAMPAIGN_ABORTED_20260624.md` — the post-mortem explaining why this is a type-dispatch problem
+- `conductor/tracks/any_type_componentization_20260621/plan.md` — the grandparent plan
+- `conductor/tracks/code_path_audit_20260607/spec_v2.md` — the audit that established the 4.01e22 baseline
+- `docs/reports/code_path_audit/2026-06-22/AUDIT_REPORT.md` — the original 6797-line audit report
+- `src/type_aliases.py` — the current Metadata definition
+- `scripts/code_path_audit/code_path_audit.py` — the consumer detection
+- `scripts/code_path_audit/code_path_audit_ssdl.py` — the effective codepaths metric
+- `conductor/code_styleguides/data_oriented_design.md` — the "Prefer Fewer Types" principle