ed/manual_slop
Private
Public Access
0
0
Fork 0
Code Activity
Files
master
manual_slop/conductor/archive/metadata_promotion_20260624/spec.md
T

32 KiB
Raw Permalink Blame History

Track Specification: metadata_promotion_20260624

Status: ACTIVE — corrected 2026-06-25 (Tier 1 audit). The original spec (commit e50bebdd, 2026-06-25) proposed a single @dataclass(frozen=True, slots=True) Metadata with ~200 fields shared across all 5 sub-aggregates. That proposal was REJECTED on 2026-06-25 (user direction): the 5 sub-aggregates are distinct concepts with distinct field sets; lifting them into one mega-dataclass hides the type information that direct field access is supposed to reveal. The corrected design promotes each sub-aggregate to its OWN dataclass with its OWN fields. See docs/reports/PLANNING_CORRECTION_metadata_promotion_20260625.md for the full rationale.

Overview

Promotes the 5 distinct sub-aggregates (CommsLogEntry, HistoryMessage, FileItem, ToolDefinition, ToolCall) to their own typed @dataclass(frozen=True, slots=True) classes (or reuses the existing typed dataclasses where they already exist: models.FileItem, openai_schemas.ToolCall), then migrates the 107 .get('key', ...) + 106 subscript ['key'] access sites on those aggregates to direct field access (entry.ts, t.depends_on, chunk.document). Metadata: TypeAlias = dict[str, Any] is preserved as the catch-all for truly collapsed codepaths (generic JSON parsing at wire boundaries, manual_slop.toml project config, polymorphic containers where the element type is genuinely unknown) and is NOT promoted to a shared mega-dataclass.

The combinatoric explosion (4.01e22 effective codepaths) is addressed by per-aggregate type promotion: each known concept gets its own dataclass with its own fields, the .get() / [] runtime type-dispatch collapses at the source, and the audit's branch count drops per consumer function.

Current State Audit (master dc397db7, measured 2026-06-25)

Metric Value Source
Metadata consumers in src/ 695 scripts/code_path_audit.build_pcg
Top consumer files app_controller.py: 123, mcp_client.py: 94, ai_client.py: 73, gui_2.py: 44, models.py: 29 Counter over pcg.consumers['Metadata']
Total branches in Metadata consumers 3,454 scripts/code_path_audit_ssdl.count_branches_in_function
Effective codepaths (the 4.01e22) 4.014e+22 compute_effective_codepaths
.get('key', ...) access sites (all sub-aggregates) 107 git grep in src/
['key'] subscript access sites 106 git grep in src/
is None / == None / != None sites 106 git grep in src/ (mostly unrelated to Metadata)
TypeAlias chain (current state, before this track) Metadata: dict[str, Any]; CommsLogEntry: Metadata; HistoryMessage: Metadata; FileItem: "models.FileItem"; ToolDefinition: Metadata; ToolCall: "openai_schemas.ToolCall" src/type_aliases.py
Existing per-aggregate dataclasses models.Ticket (15 fields), models.FileItem (10 fields), models.Track (3 fields), openai_schemas.ToolCall (3 fields), openai_schemas.ChatMessage (5 fields), openai_schemas.UsageStats (4 fields), openai_schemas.ToolCallFunction (2 fields), openai_schemas.NormalizedResponse (4 fields), vendor_capabilities.VendorCapabilities (22 fields) git grep "^class .*(dataclass|frozen=True)" src/
Missing per-aggregate dataclasses CommsLogEntry, HistoryMessage, ToolDefinition, RAGChunk, SessionInsights, DiscussionSettings, CustomSlice, MMAUsageStats, ProviderPayload, UIPanelConfig, ContextPreset (full schema), PathInfo actual access patterns from git grep on src/

Why the corrected design (per-aggregate dataclasses) — not one mega-dataclass

The 107 .get('key', default) and 106 ['key'] access sites in src/ span at least 12 distinct aggregates, not 5. A sampling of the actual access patterns:

Access pattern Site Aggregate it actually represents
item.get('custom_slices', []), item.get('content', '') src/aggregate.py:418,421 FileItem (per-file curation)
fi.get('path', 'attachment') src/ai_client.py:2565,2807,2898 FileItem
chunk.get('document', '') src/aggregate.py:3259, src/app_controller.py:251,4162 RAGChunk (RAG retrieval result)
entry.get('source_tier', 'main'), entry.get('model', 'unknown') src/app_controller.py:2277,2302,2310 CommsLogEntry (AI comms log)
u.get('input_tokens', 0), u.get('output_tokens', 0) src/app_controller.py:2304-2309 UsageStats (per-call token usage)
t.get('id', ''), t.get('depends_on', []), t.get('manual_block', False), t.get('status') src/gui_2.py:1366-1438 Ticket (MMA ticket — already a dataclass)
stats.get('model', 'unknown'), stats.get('input', 0), stats.get('output', 0) src/gui_2.py:2199-2201,2216 MMAUsageStats (per-tier rollup)
insights.get('total_tokens', 0), insights.get('call_count', 0), insights.get('burn_rate', 0), insights.get('session_cost', 0), insights.get('completed_tickets', 0), insights.get('efficiency', 0) src/gui_2.py:4926-4931 SessionInsights (overall session stats)
entry.get('temperature', 0.7), entry.get('top_p', 1.0), entry.get('max_output_tokens', 0) src/gui_2.py:3535 DiscussionSettings (per-turn settings)
slc.get('tag', ''), slc.get('comment', '') src/gui_2.py:4048-4054 CustomSlice (visual slice editor)
preset.get('files', []), preset.get('screenshots', []) src/gui_2.py:4184-4185 ContextPreset (file composition)
payload.get('script'), payload.get('args', {}), payload.get('output', ''), payload.get('content', '') src/app_controller.py:2274,2287 ProviderPayload (script-execution payload)
self.project.get('paths', {}), self.project.get('conductor', {}), self.project.get('context_presets', {}) src/app_controller.py:1972,2016,2033; src/gui_2.py:820,4181,4333,4448 ProjectConfig (manual_slop.toml — TRUE catch-all dict; uses Metadata)
gui_cfg.get('separate_message_panel', False), gui_cfg.get('separate_response_panel', False), gui_cfg.get('separate_tool_calls_panel', False) src/app_controller.py:2068-2070 UIPanelConfig
self.project.get('discussion', {}).get('discussions', {}) src/gui_2.py:5036,5046 DiscussionStore
path_info['logs_dir']['path'] src/app_controller.py:1984 PathInfo (nested)

There is no single "Metadata" shape. The 107 .get() sites access ~12 distinct aggregates, each with its own field set. The original spec (commit e50bebdd) proposed a single @dataclass(frozen=True, slots=True) Metadata with ~200 fields merging all 12 aggregates into one polymorphic mega-struct. That is the wrong direction:

  • It hides the type distinctions that direct field access is supposed to reveal.
  • A consumer that has a Ticket can read .source_tier (a CommsLogEntry field) — silently get the empty default — and ship a bug that no type checker will catch.
  • It is "less defined" than the current dict[str, Any]: today, reading .source_tier on a Ticket raises AttributeError immediately; after the mega-dataclass, it silently returns "".

The corrected design is per-aggregate dataclasses: each known concept gets its own typed dataclass with its own fields. Metadata: TypeAlias = dict[str, Any] is preserved for the truly collapsed codepaths where the shape is genuinely unknown (TOML project config, generic JSON parsing, polymorphic log dumping).

Goals

ID Goal Acceptance
G1 Each known sub-aggregate is its OWN @dataclass(frozen=True, slots=True) with its OWN fields (or reuses the existing typed dataclass where one already exists) git grep "^@dataclass|^class .*dataclass" src/ shows CommsLogEntry, HistoryMessage, RAGChunk, SessionInsights, DiscussionSettings, CustomSlice, MMAUsageStats, ProviderPayload, UIPanelConfig, DiscussionStore, ContextPreset (full), PathInfo, ToolDefinition each as its own class; the existing FileItem, ToolCall, Ticket, ChatMessage, UsageStats are reused unchanged
G2 Metadata: TypeAlias = dict[str, Any] is preserved as the catch-all for collapsed codepaths; NOT promoted to a shared mega-dataclass git grep "^Metadata:" src/type_aliases.py shows Metadata: TypeAlias = dict[str, Any] (unchanged); the type is not a dataclass
G3 Migrate the 107 .get('key', ...) + 106 ['key'] access sites on the KNOWN sub-aggregates to direct field access on the per-aggregate dataclass git grep -E "\.get\('[a-z_]+'," HEAD -- 'src/*.py' returns only legitimate non-aggregate uses (e.g., .get('mtime', 0) on file paths, .get('auto_start', False) on config dicts); the per-aggregate sites are gone
G4 Effective codepaths drops by ≥ 2 orders of magnitude compute_effective_codepaths returns < 1e+20 (was 4.014e+22)
G5 All 7 audit gates pass --strict (no regression) weak_types, type_registry, main_thread_imports, no_models_config_io, code_path_audit_coverage, exception_handling, optional_in_3_files all exit 0
G6 All existing tests pass (10/11 batched tiers — RAG flake acceptable) scripts/run_tests_batched.py → 10/11 PASS
G7 New regression-guard tests for each new per-aggregate dataclass tests/test_metadata_dataclass.py is split into tests/test_comms_log_entry.py, tests/test_history_message.py, tests/test_tool_definition.py, tests/test_rag_chunk.py, tests/test_session_insights.py, etc.; each has 5+ tests for: constructor, field access, to_dict()/from_dict() round-trip, frozen, equality
G8 Metadata (the catch-all dict) is used ONLY at the genuinely collapsed codepaths — never as a stand-in for a known sub-aggregate Code review confirms: every .get('key', default) site has been classified as either (a) a known sub-aggregate → migrated to direct field access, or (b) a genuinely collapsed codepath (TOML project config, generic JSON parsing, polymorphic log dumping) → keeps Metadata

Non-Goals

  • Modifications to src/code_path_audit*.py (the audit infrastructure is correct; the migration is on the consumer side)
  • The 4 NG1 + 7 NG2 audit violations (already addressed in phase 2 + dc397db7)
  • The 4.01e22's nil-check component (per the post-mortem at docs/reports/SSDL_CAMPAIGN_ABORTED_20260624.md, this is a minor contributor; the per-aggregate type-dispatch collapse is the dominant cause)
  • The RAG test pre-existing flake (per the SSDL post-mortem "Out of Scope")
  • New src/<thing>.py files (per AGENTS.md hard rule; new dataclasses go in src/type_aliases.py for type-system aggregates, or in the existing module for the aggregate — models.FileItem stays in models.py, openai_schemas.ToolCall stays in openai_schemas.py, etc.)
  • Promoting Metadata: TypeAlias = dict[str, Any] to a shared mega-dataclass (this is the original spec's bad inference; rejected 2026-06-25)
  • The collapsed-codepath sites (self.project.get('paths', {}), self.project.get('conductor', {}), etc.) — these read manual_slop.toml and the shape is genuinely unknown at type level; they keep Metadata as dict[str, Any]

Functional Requirements

FR1: Per-aggregate dataclasses (not one mega-dataclass)

Each known sub-aggregate becomes its OWN dataclass. The design follows the existing pattern at src/openai_schemas.py (ToolCall, ChatMessage, UsageStats, ToolCallFunction, NormalizedResponse — all separate frozen dataclasses with their own fields).

Existing dataclasses — REUSED UNCHANGED

Class Location Fields Consumers that need migration
Ticket src/models.py:302 id, description, target_symbols, context_requirements, depends_on, status, assigned_to, priority, target_file, blocked_reason, step_mode, retry_count, manual_block, model_override, persona_id (15 fields) src/gui_2.py:1366-1438,1682,4810,4820,4868; src/conductor_tech_lead.py:125; src/app_controller.py:4810-4868
FileItem src/models.py:533 path, auto_aggregate, force_full, view_mode, selected, ast_signatures, ast_definitions, ast_mask, custom_slices, injected_at (10 fields) src/aggregate.py:418,421; src/ai_client.py:2565,2807,2898; src/app_controller.py:3508
ToolCall src/openai_schemas.py:32 id, function (ToolCallFunction), type (3 fields) src/mcp_client.py (tool loop section)
ChatMessage src/openai_schemas.py:48 role, content, tool_calls, tool_call_id, name (5 fields) provider-side history (will replace the per-vendor _X_history aliases that were removed in code_path_audit_phase_3_provider_state_20260624)
UsageStats src/openai_schemas.py:68 input_tokens, output_tokens, cache_read_tokens, cache_creation_tokens (4 fields) per-call token usage in src/app_controller.py:2299-2309

NEW dataclasses — to be added

Class Module Fields Consumers that need migration
CommsLogEntry src/type_aliases.py ts, role, kind, direction, model, source_tier, content, error (8 fields) src/app_controller.py:2277,2302,2310; src/session_logger.py; src/multi_agent_conductor.py
HistoryMessage src/type_aliases.py role, content, tool_calls, tool_call_id, name, ts (6 fields) UI-layer discussion history (the per-turn editable list, NOT the provider-side ChatMessage — these are distinct layers per data_structure_strengthening_20260606 §3.1)
ToolDefinition src/type_aliases.py name, description, parameters, auto_start (4 fields) src/mcp_client.py:_build_anthropic_tools and equivalent per-vendor tool builders
RAGChunk src/rag_engine.py document, path, score, metadata (4 fields) src/aggregate.py:3259; src/app_controller.py:251,4162
SessionInsights src/type_aliases.py total_tokens, call_count, burn_rate, session_cost, completed_tickets, efficiency (6 fields) src/gui_2.py:4926-4931
DiscussionSettings src/type_aliases.py temperature, top_p, max_output_tokens (3 fields) src/gui_2.py:3535
CustomSlice src/type_aliases.py tag, comment, start_line, end_line (4 fields) src/gui_2.py:4048-4054,1301-1302
MMAUsageStats src/type_aliases.py model, input, output (3 fields) src/gui_2.py:2199-2201,2216
ProviderPayload src/type_aliases.py script, args, output, source_tier (4 fields) src/app_controller.py:2274,2287
UIPanelConfig src/type_aliases.py separate_message_panel, separate_response_panel, separate_tool_calls_panel (3 fields) src/app_controller.py:2068-2070
PathInfo src/type_aliases.py logs_dir, scripts_dir, project_root (3 fields, nested) src/app_controller.py:1984-1985
ContextPreset src/models.py (full schema) name, files (FileItems), screenshots (list[str]) (3 fields minimum) src/gui_2.py:4184-4185,4333,4448

Why per-aggregate dataclasses, not one shared mega-dataclass

  • Each aggregate has its own field set. A Ticket has depends_on: List[str], manual_block: bool. A CommsLogEntry has source_tier: str, model: str. A RAGChunk has document: str, score: float. They share NO common fields beyond id. There is no "common Metadata base" to extract.
  • A shared mega-dataclass defeats the type system. A consumer that has a Ticket can read .source_tier (a CommsLogEntry field) — silently get the empty default — and ship a bug that no type checker will catch. Today, with dict[str, Any], reading .source_tier on a Ticket raises AttributeError immediately. The mega-dataclass is less defined than the current state.
  • The original convention anticipated per-concept promotion. Per data_structure_strengthening_20260606 §3.3: "Phase 2 can convert Metadata to a TypedDict (or split into per-concept TypedDicts) and the aliases continue to work without breaking changes. The aliases are STABLE NAMES; the underlying type can evolve." The original 2026-06-06 design intent was per-concept promotion, NOT a mega-dataclass. The original 2026-06-25 metadata_promotion_20260624 spec reversed this direction; the corrected spec restores the original intent.

FR2: Metadata stays as the catch-all for collapsed codepaths

Metadata: TypeAlias = dict[str, Any] is preserved unchanged. It is used at sites where the shape is genuinely unknown at type level:

  • manual_slop.toml project config loading (self.project.get('paths', {}), self.project.get('conductor', {}), self.project.get('context_presets', {}), self.project.get('discussion', {})) — these are top-level TOML keys; the aggregator doesn't know which key it's about to read.
  • Generic JSON parsing at the wire boundary (REST API payloads, WebSocket messages) — the body shape is defined by the producer, not the consumer.
  • Polymorphic log dumping — a function that serializes a list of mixed-aggregate entries to JSON without caring about their individual types.

These sites keep Metadata and .get('key', default) because there is no per-aggregate type to promote to. The audit MUST classify every remaining .get('key', default) site as one of: (a) "promoted to per-aggregate dataclass → migrated" or (b) "collapsed codepath → keeps Metadata with documented justification in code comment or commit message."

FR3: Phase-by-phase migration (12+ sub-aggregates, 1 phase per aggregate)

The migration is per-aggregate: each aggregate gets its own phase. Phases are ordered to maximize early feedback:

Phase Sub-aggregate Est. consumers Primary files
0 Design the new dataclasses + add regression-guard test stubs 0 (design only) src/type_aliases.py (and the existing modules for in-place additions)
1 Ticket (already a dataclass; migrate consumers only) ~30 sites src/gui_2.py, src/conductor_tech_lead.py, src/app_controller.py
2 FileItem (already a dataclass; migrate consumers only) ~10 sites src/aggregate.py, src/ai_client.py, src/app_controller.py
3 CommsLogEntry (NEW dataclass + migrate consumers) ~30 sites src/type_aliases.py, src/session_logger.py, src/multi_agent_conductor.py, src/app_controller.py
4 HistoryMessage (NEW dataclass + migrate UI-layer consumers) ~20 sites src/type_aliases.py, src/gui_2.py
5 ChatMessage (already in openai_schemas.py; wire it into the per-vendor send paths) ~27 sites src/ai_client.py
6 UsageStats (already in openai_schemas.py; wire into the per-call usage aggregation) ~10 sites src/app_controller.py
7 ToolCall (already in openai_schemas.py; wire into the tool loop section) ~56 sites src/ai_client.py, src/mcp_client.py
8 ToolDefinition (NEW dataclass + migrate per-vendor tool builders) ~94 sites src/type_aliases.py, src/mcp_client.py
9 RAGChunk (NEW dataclass + migrate consumers) ~5 sites src/rag_engine.py, src/aggregate.py, src/app_controller.py
10 SessionInsights, DiscussionSettings, CustomSlice, MMAUsageStats, ProviderPayload, UIPanelConfig, PathInfo, ContextPreset (small aggregates, batched) ~25 sites src/type_aliases.py, src/models.py, src/gui_2.py, src/app_controller.py
11 Metadata collapsed-codepath audit + classification (per FR2) ~80 sites every .get('key', default) site that is NOT promoted to a per-aggregate dataclass
12 Verification + end-of-track (1 task, 3 commits) 0 terminal + docs/reports/TRACK_COMPLETION_metadata_promotion_20260624.md (NEW)

Each phase:

  1. For NEW dataclasses: define the dataclass in the appropriate module; add regression-guard test
  2. For ALL phases: migrate the consumer sites from .get('key', default) → .field_name (or .field_name or default for nullable fields)
  3. Per-phase regression-guard test runs
  4. Re-measure effective codepaths after the phase

FR4: Migration patterns (canonical)

# BEFORE:
x = entry.get('model', 'unknown')
y = entry.get('input_tokens', 0) or 0
z = entry.get('source_tier', 'main')
if entry.get('manual_block', False):
    ...
role = entry['role']
if 'depends_on' in entry:
    deps = entry['depends_on']

# AFTER (with per-aggregate dataclass):
x = entry.model or 'unknown'         # CommsLogEntry
y = entry.input_tokens or 0           # UsageStats
z = entry.source_tier or 'main'      # CommsLogEntry
if entry.manual_block:                # Ticket
    ...
role = entry.role                     # HistoryMessage / CommsLogEntry
if entry.depends_on:                  # Ticket
    deps = entry.depends_on

The migration is mechanical but requires care:

  • For nullable fields: use entry.field or default_value
  • For required fields: use entry.field directly
  • For polymorphic keys (some entries have the key, some don't): the dataclass default handles this (all fields have defaults; frozen=True, slots=True ensures immutability)
  • For ['key'] (subscript) where the key is dynamic: rare; keep as dict[str, Any] access (e.g., entry.to_dict()['dynamic_key']) — but ONLY if the entry is genuinely a dict, not a dataclass

FR5: Edge cases

Polymorphic constructors: many sites do entry = {'role': 'user', 'content': 'hi'}. After migration: entry = HistoryMessage(role='user', content='hi'). The dataclass has all the fields as Optional or with defaults, so this works.

Dynamic dict construction: for k, v in raw.items(): entry[k] = v. After migration: entry = HistoryMessage(**raw). The ** syntax requires that all keys in raw are valid field names; if raw has unknown keys, this fails. Solution: use a from_dict classmethod that filters out unknown keys (the canonical pattern, already used by models.FileItem.from_dict at src/models.py:600-619 and openai_schemas.NormalizedResponse.from_dict):

@classmethod
def from_dict(cls, raw: dict[str, Any]) -> 'HistoryMessage':
    valid_fields = {f.name for f in fields(cls)}
    return cls(**{k: v for k, v in raw.items() if k in valid_fields})

JSON serialization: json.dumps(entry) fails on dataclass. Solution: json.dumps(entry.to_dict()) (per the canonical to_dict() pattern at src/models.py:567-579 and src/openai_schemas.py:36-43).

Pickle: pickle.dumps(entry) works (dataclass supports pickle natively via __reduce__).

Equality: entry1 == entry2 now works (dataclass generates __eq__); before it was False for distinct dict instances even with the same content.

JSON round-trip preservation: every dataclass in this track has a paired to_dict() + from_dict() (no information loss). This is enforced by the per-dataclass regression-guard test.

FR6: Metadata collapsed-codepath classification (per FR2)

For every remaining .get('key', default) site after all phases:

  1. The site is classified as either (a) "promoted to per-aggregate dataclass" (migrated) or (b) "collapsed codepath" (keeps Metadata).
  2. For (b), the justification is documented in the commit message (one line: "this site reads manual_slop.toml; the shape is unknown until the TOML is parsed").
  3. The audit scripts/audit_weak_types.py --strict continues to flag anonymous dict accesses; the gate is the per-aggregate dataclass promotion, NOT the elimination of all .get().

FR7: Re-measurement

After each phase, re-measure:

uv run python -c "
import sys
sys.path.insert(0, 'scripts/code_path_audit')
sys.path.insert(0, 'src')
from code_path_audit import build_pcg
from code_path_audit_ssdl import count_branches_in_function
pcg = build_pcg('src').data
metadata_consumers = pcg.consumers.get('Metadata', [])
total = sum(2 ** count_branches_in_function(f, 'src') for f in metadata_consumers)
print(f'Effective codepaths: {total:.3e}')
print(f'Consumers: {len(metadata_consumers)}')
"

Expected: drops from 4.014e+22 to < 1e+20 after the aggregate-promotion phases (each phase drops it further as more consumers migrate to direct field access).

Non-Functional Requirements

  • NFR1: 1-space indentation (per conductor/workflow.md)
  • NFR2: CRLF line endings on Windows
  • NFR3: No comments in source code
  • NFR4: Per-task atomic commits with git notes
  • NFR5: No new pip dependencies (dataclass is stdlib)
  • NFR6: Result[T] returns for fallible fns (per error_handling.md)
  • NFR7: No new src/<thing>.py files (per AGENTS.md hard rule; new type-system aggregates go in src/type_aliases.py, in-module aggregates stay in their parent module)

Architecture Reference

  • conductor/code_styleguides/data_oriented_design.md — the canonical DOD reference ("Prefer Fewer Types" — but the types are still distinct)
  • conductor/code_styleguides/error_handling.md — the Result[T] convention
  • conductor/code_styleguides/type_aliases.md — the alias convention (preserved; Metadata: dict[str, Any] stays as the catch-all)
  • src/openai_schemas.py — the canonical per-aggregate dataclass pattern (ToolCall, ChatMessage, UsageStats); the reference implementation for the NEW dataclasses in this track
  • src/models.py:533 — FileItem (the canonical in-module dataclass pattern with to_dict() / from_dict() round-trip)
  • src/models.py:302 — Ticket (the canonical dataclass with get() legacy-compat method, used during migration)
  • docs/reports/SSDL_CAMPAIGN_ABORTED_20260624.md — the post-mortem: the 4.01e22 is from type-dispatch, not nil-checks; the fix is type promotion
  • docs/reports/PLANNING_CORRECTION_metadata_promotion_20260625.md — the corrected-design rationale (this track's correction)
  • conductor/tracks/any_type_componentization_20260621/spec.md — the grandparent track (89 sites promoted to dataclasses across 5 candidates); the per-aggregate pattern this track follows
  • conductor/tracks/data_structure_strengthening_20260606/spec.md §3.3 — the original 2026-06-06 design intent: "Phase 2 can convert Metadata to a TypedDict (or split into per-concept TypedDicts) and the aliases continue to work without breaking changes. The aliases are STABLE NAMES; the underlying type can evolve."
  • scripts/code_path_audit/code_path_audit.py — the consumer detection (3-pass AST)
  • scripts/code_path_audit/code_path_audit_ssdl.py — the effective codepaths metric

Out of Scope

  • Modifications to src/code_path_audit*.py (the audit infrastructure is correct)
  • The 4 NG1 + 7 NG2 audit violations (already addressed in dc397db7)
  • The 4.01e22's nil-check component (per SSDL post-mortem; minor contributor)
  • The RAG test pre-existing flake (per SSDL post-mortem)
  • New src/<thing>.py files (per AGENTS.md hard rule)
  • A shared mega-dataclass across the 5+ sub-aggregates (the original spec's bad inference; rejected 2026-06-25)
  • Promoting Metadata: TypeAlias = dict[str, Any] itself to a dataclass (it's the catch-all for collapsed codepaths; not a known sub-aggregate)
  • Migration of the collapsed-codepath sites (self.project.get('paths', {}), etc.) — these read manual_slop.toml; the shape is genuinely unknown
  • Pydantic migration (the canonical pattern in this codebase is stdlib @dataclass(frozen=True, slots=True); Pydantic is for input validation, not for the data structures used internally)

Verification Criteria (Definition of Done)

# Criterion Verification command
VC1 Metadata: TypeAlias = dict[str, Any] is UNCHANGED in src/type_aliases.py git grep "^Metadata:" src/type_aliases.py shows Metadata: TypeAlias = dict[str, Any]
VC2 Each new sub-aggregate is its OWN @dataclass(frozen=True, slots=True) in the appropriate module git grep -A 2 "^class CommsLogEntry|^class HistoryMessage|^class ToolDefinition|^class RAGChunk|^class SessionInsights|^class DiscussionSettings|^class CustomSlice|^class MMAUsageStats|^class ProviderPayload|^class UIPanelConfig|^class PathInfo" src/ shows each as a separate frozen dataclass
VC3 Existing per-aggregate dataclasses (Ticket, FileItem, ToolCall, ChatMessage, UsageStats) are REUSED unchanged git grep "class Ticket|class FileItem|class ToolCall|class ChatMessage|class UsageStats" src/ shows the existing classes; consumers migrate to direct field access on them
VC4 All 107 .get('key', ...) access sites on KNOWN sub-aggregates replaced git grep -E "\.get\('[a-z_]+'," HEAD -- 'src/*.py' returns only the FR2 collapsed-codepath sites (documented in the per-site classification)
VC5 All 106 ['key'] subscript access sites on KNOWN sub-aggregates replaced git grep -E "\[[ ]*'[a-z_]+'[ ]*\]" HEAD -- 'src/*.py' returns only legitimate non-aggregate uses
VC6 Per-aggregate regression-guard tests exist and pass uv run pytest tests/test_comms_log_entry.py tests/test_history_message.py tests/test_tool_definition.py tests/test_rag_chunk.py tests/test_session_insights.py -v → all pass (5+ tests per file)
VC7 Effective codepaths drops by ≥ 2 orders of magnitude compute_effective_codepaths returns < 1e+20 (was 4.014e+22)
VC8 All 7 audit gates pass --strict (no regression) weak_types ≤ 112; type_registry 22 files; main_thread_imports 17; no_models_config_io 0; code_path_audit_coverage 0; exception_handling 0; optional_in_3_files 0
VC9 10/11 batched test tiers PASS (RAG flake acceptable) scripts/run_tests_batched.py → 10/11
VC10 End-of-track report written docs/reports/TRACK_COMPLETION_metadata_promotion_20260624.md exists with the new effective-codepaths number and the per-aggregate classification of the remaining .get() sites

Risks

# Risk Likelihood Mitigation
R1 Some sub-aggregate has fields that don't fit cleanly into a frozen dataclass (e.g., mutability needed) low The canonical reference is src/openai_schemas.py; all 5 existing dataclasses there are frozen=True. If a field needs mutability, refactor to use dataclasses.replace() instead of mutating in place
R2 Some sites mutate entry (e.g., entry['key'] = value); dataclass is frozen medium Audit these sites; if found, replace with dataclasses.replace(entry, field_name=value)
R3 The dynamic-key subscript sites (entry[variable_name]) are not covered by direct field access low These sites are rare and already classified as collapsed-codepath per FR2; keep them as entry.to_dict()[var_name] if the entry is a dataclass, or entry[var_name] if the entry is a dict
R4 to_dict() round-trip loses information for nested dicts (e.g., custom_slices: list[dict] in FileItem) low FileItem.to_dict() already handles this (passes nested dicts through as dict[str, Any]); mirror the pattern in the new dataclasses
R5 The 695 consumer functions are too many for one track high The track is broken into 12 phases (FR3); each phase is independent and per-aggregate; the per-phase regression-guard test catches regressions early
R6 A collapsed-codepath site is misclassified as a known sub-aggregate (or vice versa) medium The FR6 classification is auditable: every remaining .get() site is either (a) "promoted" or (b) "collapsed with documented justification"; the audit --strict gate catches drift
R7 The dataclass names collide with existing names (e.g., Metadata exists in both src/type_aliases.py and src/models.py) medium Use module-qualified imports: from src.type_aliases import Metadata for the dict alias; from src.models import Metadata for the small dataclass. Document the collision in the per-aggregate test file

See also

  • docs/reports/SSDL_CAMPAIGN_ABORTED_20260624.md — the post-mortem: type promotion fixes the 4.01e22, not nil-checks
  • docs/reports/PLANNING_CORRECTION_metadata_promotion_20260625.md — the corrected-design rationale
  • conductor/code_styleguides/type_aliases.md — the alias convention (preserved; Metadata: dict[str, Any] stays as the catch-all)
  • conductor/code_styleguides/data_oriented_design.md — the canonical DOD reference
  • conductor/tracks/any_type_componentization_20260621/spec.md — the grandparent track (89 sites already promoted to dataclasses)
  • conductor/tracks/data_structure_strengthening_20260606/spec.md §3.3 — the original 2026-06-06 design intent: per-concept promotion
  • src/openai_schemas.py — the canonical per-aggregate dataclass pattern
  • src/models.py:533 — FileItem (canonical in-module dataclass with to_dict() / from_dict())
  • src/models.py:302 — Ticket (canonical dataclass with legacy get() compat)
  • conductor/tracks/code_path_audit_20260607/spec_v2.md — the audit that established the 4.01e22 baseline
  • docs/reports/code_path_audit/2026-06-22/AUDIT_REPORT.md — the original 6797-line audit report
View Git Blame Copy Permalink