From 495882e704e3862cf2ac16d59046cc89a6c40345 Mon Sep 17 00:00:00 2001 From: Ed_ Date: Thu, 25 Jun 2026 14:29:24 -0400 Subject: [PATCH] conductor(plan): correct metadata_promotion_20260624 plan to 13 per-aggregate phases --- .../metadata_promotion_20260624/plan.md | 384 ++++++++++++------ 1 file changed, 257 insertions(+), 127 deletions(-) diff --git a/conductor/tracks/metadata_promotion_20260624/plan.md b/conductor/tracks/metadata_promotion_20260624/plan.md index 9b2ba812..ea1e8b46 100644 --- a/conductor/tracks/metadata_promotion_20260624/plan.md +++ b/conductor/tracks/metadata_promotion_20260624/plan.md @@ -1,116 +1,230 @@ # Plan: metadata_promotion_20260624 -5 phases, 12-15 tasks, 12+ atomic commits. Per-task TDD red-first. Tier 3 workers execute; Tier 2 reviews per phase. +> **CORRECTED 2026-06-25 (Tier 1 audit).** The original plan (commit `e50bebdd`, 2026-06-25) proposed a single shared `@dataclass(frozen=True, slots=True) Metadata` with ~200 fields for all 5 sub-aggregates. That proposal was REJECTED on 2026-06-25 (user direction): each sub-aggregate is its OWN dataclass with its OWN fields. The corrected plan has 12 phases (one per sub-aggregate), uses existing dataclasses where they exist (`Ticket`, `FileItem`, `ToolCall`, `ChatMessage`, `UsageStats`), and adds new per-aggregate dataclasses for the 8 aggregates that don't have one yet. See `docs/reports/PLANNING_CORRECTION_metadata_promotion_20260625.md` for the full rationale. -## Phase 0: Design the dataclass + add regression-guard test (2 tasks, 2 commits) +13 phases, 30-35 tasks, 30+ atomic commits. Per-task TDD red-first. Tier 3 workers execute; Tier 2 reviews per phase. -**Focus:** Create the `@dataclass(frozen=True, slots=True) Metadata` in `src/type_aliases.py` + add the test file. No consumer migration yet. +## Phase 0: Design the per-aggregate dataclasses + add regression-guard test stubs (5 tasks, 5 commits) -- [x] **Task 0.1** [Tier 3]: Design the dataclass. +**Focus:** Add the NEW dataclasses to `src/type_aliases.py` (the type-system aggregates that don't have a parent module); reuse the existing dataclasses in `src/models.py` and `src/openai_schemas.py`. No consumer migration yet. + +- [ ] **Task 0.1** [Tier 3]: Add NEW dataclasses to `src/type_aliases.py`. - WHERE: `src/type_aliases.py` (current 30 lines) - WHAT: - - Replace `Metadata: TypeAlias = dict[str, Any]` with `@dataclass(frozen=True, slots=True) class Metadata: ...` - - Add the canonical fields (from the spec §FR1): role, content, tool_calls, tool_call_id, name, args, source_tier, model, id, ts, description, depends_on, status, manual_block, completed_tickets, auto_start, command, script, output, error, tier, path, full_path, filename, mtime, size + the other ~150-180 distinct keys from the `.get` and `[]` site analysis - - Add `to_dict()` method (for JSON serialization) + `from_dict()` classmethod (filters unknown keys) - - Add `__post_init__` for any derived value validation - - KEEP the 5 sub-aggregate TypeAliases (`CommsLogEntry: TypeAlias = Metadata` etc.) — they all point to the new dataclass + - Add `@dataclass(frozen=True, slots=True) class CommsLogEntry` with `ts, role, kind, direction, model, source_tier, content, error` (8 fields, all with defaults) + - Add `@dataclass(frozen=True, slots=True) class HistoryMessage` with `role, content, tool_calls, tool_call_id, name, ts` (6 fields) + - Add `@dataclass(frozen=True, slots=True) class ToolDefinition` with `name, description, parameters, auto_start` (4 fields) + - Add `@dataclass(frozen=True, slots=True) class SessionInsights` with `total_tokens, call_count, burn_rate, session_cost, completed_tickets, efficiency` (6 fields) + - Add `@dataclass(frozen=True, slots=True) class DiscussionSettings` with `temperature, top_p, max_output_tokens` (3 fields) + - Add `@dataclass(frozen=True, slots=True) class CustomSlice` with `tag, comment, start_line, end_line` (4 fields) + - Add `@dataclass(frozen=True, slots=True) class MMAUsageStats` with `model, input, output` (3 fields) + - Add `@dataclass(frozen=True, slots=True) class ProviderPayload` with `script, args, output, source_tier` (4 fields) + - Add `@dataclass(frozen=True, slots=True) class UIPanelConfig` with `separate_message_panel, separate_response_panel, separate_tool_calls_panel` (3 fields) + - Add `@dataclass(frozen=True, slots=True) class PathInfo` with `logs_dir, scripts_dir, project_root` (3 nested fields) + - Each dataclass has a paired `to_dict()` (for JSON serialization) and `from_dict()` classmethod (filters unknown keys, per FR5) + - KEEP `Metadata: TypeAlias = dict[str, Any]` UNCHANGED (the catch-all for collapsed codepaths) + - KEEP `CommsLog: TypeAlias = list[CommsLogEntry]`, `History: TypeAlias = list[HistoryMessage]`, `FileItems: TypeAlias = list[FileItem]` (the list aliases still work; the element types are now per-aggregate dataclasses) - KEEP `JsonValue`, `JsonPrimitive`, `CommsLogCallback`, `FileItemsDiff` unchanged - - HOW: `manual-slop_edit_file` for surgical edits - - SAFETY: `ast.parse` OK; `from src.type_aliases import Metadata` OK; `Metadata()` constructor works -- [x] **COMMIT:** `refactor(type_aliases): promote Metadata to @dataclass(frozen=True, slots=True)` (Tier 3) -- [x] **GIT NOTE:** Metadata is now a typed dataclass. The 5 sub-aggregate TypeAliases all point to the same class. The consumer migration is in subsequent phases. + - HOW: `manual-slop_edit_file` for surgical edits (or `write_file` if the file is being substantially restructured) + - SAFETY: `ast.parse` OK; `from src.type_aliases import CommsLogEntry, HistoryMessage, ToolDefinition, SessionInsights, DiscussionSettings, CustomSlice, MMAUsageStats, ProviderPayload, UIPanelConfig, PathInfo` OK; constructors work +- [ ] **COMMIT:** `refactor(type_aliases): add per-aggregate dataclasses (CommsLogEntry, HistoryMessage, ToolDefinition, ...)` (Tier 3) +- [ ] **GIT NOTE:** NEW dataclasses added to `src/type_aliases.py`. `Metadata: TypeAlias = dict[str, Any]` is UNCHANGED (the catch-all for collapsed codepaths). No consumer migration yet. -- [x] **Task 0.2** [Tier 3]: Create `tests/test_metadata_dataclass.py`. - - WHERE: NEW FILE `tests/test_metadata_dataclass.py` - - WHAT: 12+ tests: - - `test_empty_constructor`: `Metadata()` returns an instance with all fields as default values - - `test_constructor_with_kwargs`: `Metadata(role='user', content='hi')` works - - `test_field_access`: `entry.role` works - - `test_frozen`: trying to mutate a field raises `dataclasses.FrozenInstanceError` - - `test_slots`: `__slots__` is set (no `__dict__`) - - `test_to_dict`: `entry.to_dict()` returns the same dict as the old `dict[str, Any]` shape - - `test_from_dict`: `Metadata.from_dict({'role': 'user'})` works; unknown keys are silently filtered - - `test_from_dict_preserves_all_fields`: full round-trip - - `test_equality`: two `Metadata(role='user')` instances are equal - - `test_hashable`: `Metadata(role='user')` can be in a set/dict - - `test_type_aliases_resolve_to_metadata`: `CommsLogEntry is Metadata`, `HistoryMessage is Metadata`, etc. - - `test_pickle`: `pickle.dumps(Metadata(...))` works - - HOW: `write_file` to create the new test file (with all 12 tests) - - SAFETY: `uv run python -m pytest tests/test_metadata_dataclass.py -v` shows 12/12 pass -- [x] **COMMIT:** `test(type_aliases): add Metadata dataclass regression-guard suite` (Tier 3) -- [x] **GIT NOTE:** 12 tests cover the dataclass behavior. The consumer migration is in subsequent phases; this commit only adds the dataclass + tests. +- [ ] **Task 0.2** [Tier 3]: Add `RAGChunk` dataclass to `src/rag_engine.py`. + - WHERE: `src/rag_engine.py` (the parent module for RAG) + - WHAT: `@dataclass(frozen=True, slots=True) class RAGChunk` with `document, path, score, metadata` (4 fields, all with defaults); paired `to_dict()` / `from_dict()` + - HOW: `manual-slop_edit_file` + - SAFETY: `from src.rag_engine import RAGChunk` OK; constructor works +- [ ] **COMMIT:** `feat(rag_engine): add RAGChunk dataclass` (Tier 3) +- [ ] **GIT NOTE:** NEW dataclass added to `src/rag_engine.py`. No consumer migration yet. -## Phase 1: Migrate `CommsLogEntry` consumers (~150 sites, 1 commit per file) +- [ ] **Task 0.3** [Tier 3]: Audit and complete `ContextPreset` schema in `src/models.py`. + - WHERE: `src/models.py` (the parent module for ContextPreset) + - WHAT: `ContextPreset` exists at `src/models.py:932` but is partial. Add missing fields based on access patterns: `name, files (FileItems), screenshots (list[str])` minimum; audit the actual usage and add any other required fields; ensure paired `to_dict()` / `from_dict()` + - HOW: `manual-slop_edit_file` + - SAFETY: existing `ContextPreset` consumers continue to work; the `to_dict()` round-trip is lossless +- [ ] **COMMIT:** `refactor(models): complete ContextPreset schema with missing fields` (Tier 3) +- [ ] **GIT NOTE:** `ContextPreset` schema extended. Existing consumers unchanged. -**Focus:** The smallest sub-aggregate first. `CommsLogEntry` is used in `app_controller.py` + `multi_agent_conductor.py` + `session_logger.py`. The migration is mechanical: `entry.get('key', default)` → `entry.key or default`. +- [ ] **Task 0.4** [Tier 3]: Create `tests/test_metadata_dataclass.py` (split into per-aggregate test files per FR G7). + - WHERE: NEW FILES: `tests/test_comms_log_entry.py`, `tests/test_history_message.py`, `tests/test_tool_definition.py`, `tests/test_rag_chunk.py`, `tests/test_session_insights.py`, `tests/test_discussion_settings.py`, `tests/test_custom_slice.py`, `tests/test_mma_usage_stats.py`, `tests/test_provider_payload.py`, `tests/test_ui_panel_config.py`, `tests/test_path_info.py`, `tests/test_context_preset_schema.py` + - WHAT: 5+ tests per file: constructor with kwargs, field access, frozen (raises `FrozenInstanceError`), `to_dict()` / `from_dict()` round-trip, equality, hashability, default values + - HOW: `write_file` per file + - SAFETY: `uv run pytest tests/test_comms_log_entry.py -v` shows 5/5 pass (and similarly for the other 11 files) +- [ ] **COMMIT:** `test(type_aliases): add per-aggregate dataclass regression-guard suite` (Tier 3) +- [ ] **GIT NOTE:** 12 test files, 5+ tests each. The consumer migration is in subsequent phases; this commit only adds the new dataclasses + tests. -- [x] **Task 1.1** [Tier 3]: Migrate `src/session_logger.py` (the smallest, the writer-side). - - WHERE: `src/session_logger.py` (~218 lines; ~30 access sites) - - WHAT: For each `entry.get('key', default)` and `entry['key']` where `entry` is `CommsLogEntry`, replace with `entry.key or default` (or `entry.key` for required fields) +- [ ] **Task 0.5** [Tier 2]: Document the FR6 collapsed-codepath classification rule. + - WHERE: `conductor/code_styleguides/type_aliases.md` (small clarification, NOT a rewrite) + - WHAT: Add a one-paragraph "When to promote to a per-aggregate dataclass" rule: when a sub-aggregate has stable distinct fields, promote it to its OWN dataclass; do NOT share one mega-dataclass across concepts; `Metadata: TypeAlias = dict[str, Any]` is preserved for collapsed codepaths (TOML config, generic JSON parsing, polymorphic log dumping) only. Reference this track's correction as the canonical example. + - HOW: `manual-slop_edit_file` + - SAFETY: styleguide is consistent with the corrected design +- [ ] **COMMIT:** `docs(styleguides): clarify when to promote to per-aggregate dataclass` (Tier 2) +- [ ] **GIT NOTE:** Styleguide clarification. The corrected design is: per-aggregate dataclasses for known sub-aggregates; `Metadata: dict[str, Any]` for collapsed codepaths only. + +## Phase 1: Migrate `Ticket` consumers (~30 sites, 2 commits) + +**Focus:** `Ticket` is already a dataclass (`src/models.py:302`); just migrate the consumers from `t.get('id', '')` to `t.id`. The legacy `Ticket.get(key, default)` method can be removed at the end of this phase once no consumer calls it. + +- [ ] **Task 1.1** [Tier 3]: Migrate `src/gui_2.py` Ticket consumers. + - WHERE: `src/gui_2.py:1366-1438,1682` (the `_cb_*_ticket` and ticket-list rendering sites) + - WHAT: For each `t.get('id', '')`, `t.get('depends_on', [])`, `t.get('manual_block', False)`, `t.get('status')` → `t.id`, `t.depends_on`, `t.manual_block`, `t.status` - HOW: `manual-slop_edit_file` per site - - SAFETY: Run `tests/test_session_logger_optimization.py` + `tests/test_session_logger_reset.py` + `tests/test_session_logging.py` + `tests/test_logging_e2e.py` + the new `tests/test_metadata_dataclass.py` -- [x] **COMMIT:** `refactor(session_logger): migrate CommsLogEntry access sites to Metadata dataclass` (Tier 3) -- [x] **GIT NOTE:** Migrated ~30 access sites in session_logger.py. Verified by the 4 session_logger test files + 12 metadata dataclass tests. + - SAFETY: Run `tests/test_ticket_queue.py` + `tests/test_per_ticket_model.py` + `tests/test_manual_block.py` + the new per-aggregate test files +- [ ] **COMMIT:** `refactor(gui_2): migrate Ticket access sites to direct field access` (Tier 3) +- [ ] **GIT NOTE:** Migrated ~15 Ticket access sites in `src/gui_2.py`. Verified by the ticket test files. -- [x] **Task 1.2** [Tier 3]: Migrate `src/multi_agent_conductor.py` (~70 access sites) -- [x] **Task 1.3** [Tier 3]: Migrate `src/app_controller.py` (the bulk — ~50 access sites that are CommsLogEntry-specific) -- [x] **COMMIT (1.2):** `refactor(multi_agent_conductor): migrate CommsLogEntry access sites` (Tier 3) -- [x] **COMMIT (1.3):** `refactor(app_controller): migrate CommsLogEntry access sites` (Tier 3) -- [x] **GIT NOTES (1.2, 1.3):** Per-file counts. Verified by the full batched test suite (no regression). -- [x] **Task 1.4** [Tier 2]: Re-measure effective codepaths after Phase 1. - - EXPECTED: drops from 4.014e+22 to ~4e+19 (CommsLogEntry has the most consumers; their branch counts drop significantly) - - Document in `docs/reports/metadata_promotion_progress.md` (new file) - -## Phase 2: Migrate `HistoryMessage` consumers (~80 sites, 1 commit per file) - -**Focus:** `ai_client.py` per-vendor history. The 27 call sites in phase 3 just got migrated to `provider_state.get_history("...")`; this phase migrates the `entry.get('role', ...)` and `entry.get('content', ...)` calls inside those functions. - -- [x] **Task 2.1** [Tier 3]: Migrate `src/ai_client.py` (the bulk — ~80 access sites in `_send_anthropic`, `_send_deepseek`, `_send_grok`, etc.) +- [ ] **Task 1.2** [Tier 3]: Migrate `src/conductor_tech_lead.py` and `src/app_controller.py` Ticket consumers. + - WHERE: `src/conductor_tech_lead.py:125`; `src/app_controller.py:4810-4868` (the ticket-list mutation sites) + - WHAT: Same pattern as 1.1 - HOW: `manual-slop_edit_file` per site - - SAFETY: Run the 5 per-provider test files + the 12 metadata dataclass tests + the 7 per-provider migration tests -- [x] **COMMIT:** `refactor(ai_client): migrate HistoryMessage access sites to Metadata dataclass` (Tier 3) -- [x] **GIT NOTE:** Migrated ~80 access sites in ai_client.py. The HistoryMessage aggregate now uses direct field access. -- [x] **Task 2.2** [Tier 2]: Re-measure. EXPECTED: drops further. Document. + - SAFETY: Same as 1.1 +- [ ] **COMMIT:** `refactor(app_controller,conductor_tech_lead): migrate Ticket access sites` (Tier 3) +- [ ] **GIT NOTE:** Migrated ~15 Ticket access sites across 2 files. Verified. +- [ ] **Task 1.3** [Tier 2]: Remove the legacy `Ticket.get(key, default)` method. + - WHERE: `src/models.py` (the `get` method on `Ticket`) + - WHAT: After all consumers have migrated, remove the `get` method + - HOW: `manual-slop_py_remove_def` + - SAFETY: Re-run the full batched test suite; no remaining `.get(key, default)` on Ticket consumers +- [ ] **COMMIT:** `refactor(models): remove legacy Ticket.get() method` (Tier 2) +- [ ] **GIT NOTE:** Legacy compat method removed. Direct field access is now the only path. -## Phase 3: Migrate `FileItem` consumers (~200 sites, 1 commit per file) +## Phase 2: Migrate `FileItem` consumers (~10 sites, 2 commits) -**Focus:** `aggregate.py` + `gui_2.py` + `app_controller.py` (the rest of it). This is the largest phase. `FileItem` is the most polymorphic — many distinct keys. +**Focus:** `FileItem` is already a dataclass (`src/models.py:533`); migrate the consumers. -- [x] **Task 3.1** [Tier 3]: Migrate `src/aggregate.py` (~50 access sites) -- [x] **Task 3.2** [Tier 3]: Migrate `src/app_controller.py` (the remaining ~50 access sites; some overlap with phase 1 CommsLogEntry) -- [x] **Task 3.3** [Tier 3]: Migrate `src/gui_2.py` (~100 access sites; the largest) -- [x] **COMMIT (3.1, 3.2, 3.3):** 3 atomic commits, one per file -- [x] **GIT NOTES:** Per-file counts. Verified. -- [x] **Task 3.4** [Tier 2]: Re-measure. EXPECTED: significant drop. Document. +- [ ] **Task 2.1** [Tier 3]: Migrate `src/aggregate.py` FileItem consumers. + - WHERE: `src/aggregate.py:418,421` + - WHAT: `item.get('custom_slices', [])` → `item.custom_slices`; `item.get('content', '')` → `item.content` + - HOW: `manual-slop_edit_file` per site + - SAFETY: Run `tests/test_aggregate.py` + `tests/test_file_item_model.py` + the new per-aggregate test files +- [ ] **COMMIT:** `refactor(aggregate): migrate FileItem access sites` (Tier 3) +- [ ] **GIT NOTE:** Migrated ~5 FileItem access sites. -## Phase 4: Migrate `ToolDefinition` + `ToolCall` consumers (~150 sites, 2 commits) +- [ ] **Task 2.2** [Tier 3]: Migrate `src/ai_client.py` and `src/app_controller.py` FileItem consumers. + - WHERE: `src/ai_client.py:2565,2807,2898`; `src/app_controller.py:3508` + - WHAT: `fi.get('path', 'attachment')` → `fi.path`; `f['path'] for f in file_items` → `f.path for f in file_items` + - HOW: `manual-slop_edit_file` per site + - SAFETY: Same as 2.1 +- [ ] **COMMIT:** `refactor(ai_client,app_controller): migrate FileItem access sites` (Tier 3) +- [ ] **GIT NOTE:** Migrated ~5 FileItem access sites across 2 files. -**Focus:** `mcp_client.py` + `ai_client.py` (the tool loop section). These are the most typed-shaped; should be clean. +## Phase 3: Migrate `CommsLogEntry` consumers (~30 sites, 3 commits) -- [x] **Task 4.1** [Tier 3]: Migrate `src/mcp_client.py` (~94 access sites — the bulk) -- [x] **Task 4.2** [Tier 3]: Migrate `src/ai_client.py` (the tool loop section only — ~56 access sites) -- [x] **COMMIT (4.1, 4.2):** 2 atomic commits -- [x] **GIT NOTES:** Per-file counts. Verified. -- [x] **Task 4.3** [Tier 2]: Re-measure. EXPECTED: another drop. Document. +**Focus:** New dataclass added in Phase 0; now wire it into the consumers. -## Phase 5: Migrate remaining `Metadata` direct usage (~115 sites, multiple commits) +- [ ] **Task 3.1** [Tier 3]: Migrate `src/session_logger.py`. + - WHERE: `src/session_logger.py` (~30 access sites; the writer-side) + - WHAT: `entry.get('source_tier', 'main')` → `entry.source_tier`; `entry.get('model', 'unknown')` → `entry.model`; etc. + - HOW: `manual-slop_edit_file` per site + - SAFETY: Run `tests/test_session_logger_optimization.py` + `tests/test_session_logger_reset.py` + `tests/test_session_logging.py` + `tests/test_logging_e2e.py` + `tests/test_comms_log_entry.py` +- [ ] **COMMIT:** `refactor(session_logger): migrate CommsLogEntry access sites` (Tier 3) +- [ ] **GIT NOTE:** Migrated ~30 access sites. -**Focus:** The 115 consumer functions that use `Metadata` directly (not via a sub-aggregate alias). This is the catch-all. Many of these are in `gui_2.py` (already partly migrated in phase 3) + `models.py` + `paths.py` + others. +- [ ] **Task 3.2** [Tier 3]: Migrate `src/multi_agent_conductor.py` (~20 sites) +- [ ] **Task 3.3** [Tier 3]: Migrate `src/app_controller.py` CommsLogEntry section (~10 sites) +- [ ] **COMMIT (3.2, 3.3):** 2 atomic commits +- [ ] **Task 3.4** [Tier 2]: Re-measure effective codepaths after Phase 3. -- [x] **Task 5.1** [Tier 3]: Audit remaining `Metadata` direct-usage sites. - - WHICH: `git grep -nE "Metadata\b" -- 'src/*.py'` filtered to NON-sub-aggregate usages - - HOW: `git grep -lE "Metadata\b" -- 'src/*.py'` then per-file count - - EXPECTED: ~115 sites across 5-8 files -- [x] **Task 5.2-5.N** [Tier 3]: Per-file migration (1 commit per file, in decreasing order of access site count) - - For each file: `manual-slop_edit_file` per site - - SAFETY: Run the affected test file + `tests/test_metadata_dataclass.py` -- [x] **COMMIT (5.2-5.N):** 1 per file. All atomic. +## Phase 4: Migrate `HistoryMessage` consumers (~20 sites, 1 commit) -## Phase 6: Verification + end-of-track (1 task, 3 commits) +**Focus:** UI-layer discussion history (NOT provider-side `ChatMessage`; these are distinct layers per `data_structure_strengthening_20260606` §3.1). + +- [ ] **Task 4.1** [Tier 3]: Migrate `src/gui_2.py` discussion UI sites. + - WHERE: `src/gui_2.py` (~20 sites; the editable per-turn message list) + - WHAT: `entry['role']` → `entry.role`; etc. + - HOW: `manual-slop_edit_file` per site + - SAFETY: Run the per-aggregate test files +- [ ] **COMMIT:** `refactor(gui_2): migrate HistoryMessage access sites` (Tier 3) +- [ ] **GIT NOTE:** Migrated ~20 HistoryMessage access sites. +- [ ] **Task 4.2** [Tier 2]: Re-measure. + +## Phase 5: Wire `ChatMessage` into per-vendor send paths (~27 sites, 3 commits) + +**Focus:** `ChatMessage` is already in `src/openai_schemas.py:48`; wire it into the per-vendor send paths that were migrated to `provider_state.get_history("...")` in `code_path_audit_phase_3_provider_state_20260624`. + +- [ ] **Task 5.1** [Tier 3]: Migrate `_send_anthropic` and `_send_deepseek` (~9 sites) +- [ ] **Task 5.2** [Tier 3]: Migrate `_send_grok` and `_send_qwen` (~9 sites) +- [ ] **Task 5.3** [Tier 3]: Migrate `_send_minimax` and `_send_llama` (~9 sites) +- [ ] **COMMIT (5.1, 5.2, 5.3):** 3 atomic commits +- [ ] **Task 5.4** [Tier 2]: Re-measure. + +## Phase 6: Wire `UsageStats` into per-call usage aggregation (~10 sites, 1 commit) + +**Focus:** `UsageStats` is already in `src/openai_schemas.py:68`; wire it into the per-call usage aggregation in `app_controller.py`. + +- [ ] **Task 6.1** [Tier 3]: Migrate `src/app_controller.py:2299-2309`. + - WHERE: `src/app_controller.py:2299-2309` (the `mma_tier_usage` aggregation sites) + - WHAT: `u.get('input_tokens', 0) or 0` → `u.input_tokens or 0`; etc. + - HOW: `manual-slop_edit_file` + - SAFETY: Run `tests/test_token_usage.py` + `tests/test_usage_analytics_popout_sim.py` + `tests/test_openai_schemas.py` +- [ ] **COMMIT:** `refactor(app_controller): migrate UsageStats access sites` (Tier 3) +- [ ] **GIT NOTE:** Migrated ~10 UsageStats access sites. + +## Phase 7: Wire `ToolCall` into the tool loop section (~56 sites, 2 commits) + +**Focus:** `ToolCall` is already in `src/openai_schemas.py:32`; wire it into the tool loop section in `ai_client.py` and `mcp_client.py`. + +- [ ] **Task 7.1** [Tier 3]: Migrate `src/ai_client.py` tool loop section (~56 sites) +- [ ] **Task 7.2** [Tier 3]: Verify `src/mcp_client.py` tool loop section (the small subset) +- [ ] **COMMIT (7.1, 7.2):** 2 atomic commits + +## Phase 8: Migrate `ToolDefinition` consumers (~94 sites, 2 commits) + +**Focus:** New dataclass added in Phase 0; now wire it into the per-vendor tool builders. + +- [ ] **Task 8.1** [Tier 3]: Migrate `src/mcp_client.py` (~70 sites; the bulk) +- [ ] **Task 8.2** [Tier 3]: Migrate `src/ai_client.py` per-vendor tool builders (~24 sites) +- [ ] **COMMIT (8.1, 8.2):** 2 atomic commits + +## Phase 9: Migrate `RAGChunk` consumers (~5 sites, 1 commit) + +**Focus:** New dataclass added in Phase 0; migrate the RAG result consumers. + +- [ ] **Task 9.1** [Tier 3]: Migrate `src/rag_engine.py`, `src/aggregate.py`, `src/app_controller.py` RAG chunk consumers. + - WHERE: `src/aggregate.py:3259`; `src/app_controller.py:251,4162` + - WHAT: `chunk.get('document', '')` → `chunk.document`; etc. + - HOW: `manual-slop_edit_file` per site + - SAFETY: Run `tests/test_rag_engine.py` + `tests/test_rag_*.py` + `tests/test_rag_chunk.py` (new) +- [ ] **COMMIT:** `refactor(rag_engine,aggregate,app_controller): migrate RAGChunk access sites` (Tier 3) +- [ ] **GIT NOTE:** Migrated ~5 RAGChunk access sites across 3 files. + +## Phase 10: Migrate small-batch aggregates (~25 sites, 2 commits) + +**Focus:** `SessionInsights`, `DiscussionSettings`, `CustomSlice`, `MMAUsageStats`, `ProviderPayload`, `UIPanelConfig`, `PathInfo`. These are small aggregates with few sites; batch them. + +- [ ] **Task 10.1** [Tier 3]: Migrate `src/gui_2.py` small-batch consumers. + - WHERE: `src/gui_2.py:2199-2201,2216,3535,4048-4054,4926-4931` (SessionInsights, DiscussionSettings, CustomSlice, MMAUsageStats) + - WHAT: `insights.get('total_tokens', 0)` → `insights.total_tokens`; `entry.get('temperature', 0.7)` → `entry.temperature`; `slc.get('tag', '')` → `slc.tag`; `stats.get('model', 'unknown')` → `stats.model` + - HOW: `manual-slop_edit_file` per site + - SAFETY: Run the per-aggregate test files + the GUI tests +- [ ] **COMMIT:** `refactor(gui_2): migrate SessionInsights, DiscussionSettings, CustomSlice, MMAUsageStats` (Tier 3) +- [ ] **GIT NOTE:** Migrated ~20 small-aggregate access sites. + +- [ ] **Task 10.2** [Tier 3]: Migrate `src/app_controller.py` ProviderPayload, UIPanelConfig, PathInfo consumers. + - WHERE: `src/app_controller.py:1972-2033,2068-2070,2274-2310` (the project config + UI panel config + provider payload sites) + - WHAT: `payload.get('script')` → `payload.script`; `gui_cfg.get('separate_message_panel', False)` → `gui_cfg.separate_message_panel`; `path_info['logs_dir']['path']` → `path_info.logs_dir.path` (nested access) + - HOW: `manual-slop_edit_file` per site + - SAFETY: Run the per-aggregate test files + the app_controller tests +- [ ] **COMMIT:** `refactor(app_controller): migrate ProviderPayload, UIPanelConfig, PathInfo` (Tier 3) +- [ ] **GIT NOTE:** Migrated ~5 small-aggregate access sites. + +## Phase 11: `Metadata` collapsed-codepath audit (FR6, 1 task, 1 commit) + +**Focus:** Every remaining `.get('key', default)` site is classified as either (a) "promoted to per-aggregate dataclass → migrated" or (b) "collapsed codepath → keeps Metadata with documented justification." + +- [ ] **Task 11.1** [Tier 2]: Audit remaining `.get('key', default)` sites. + - WHERE: `git grep -nE "\.get\('[a-z_]+'," HEAD -- 'src/*.py'` + - WHAT: Per-site classification: (a) promoted + migrated (drop from the report), (b) collapsed-codepath (document the justification in the commit message). The expected collapsed-codepath sites are: `self.project.get('paths', {})`, `self.project.get('conductor', {})`, `self.project.get('context_presets', {})`, `self.project.get('discussion', {})`, `gui_cfg.get(...)` (if `UIPanelConfig` doesn't cover it), etc. + - HOW: Manual review + commit message +- [ ] **COMMIT:** `docs(audit): classify remaining .get() sites as promoted or collapsed-codepath` (Tier 2) +- [ ] **GIT NOTE:** Per-site classification. The remaining `.get()` sites are all justified collapsed-codepaths. + +## Phase 12: Verification + end-of-track (1 task, 3 commits) **Focus:** Run all 10 VCs; write `TRACK_COMPLETION`; update `state.toml` + `tracks.md`. -- [x] **Task 6.1** [Tier 2]: +- [ ] **Task 12.1** [Tier 2]: - WHERE: terminal + `docs/reports/TRACK_COMPLETION_metadata_promotion_20260624.md` (NEW) - WHAT: - VC1-VC10 verification (see spec.md §Verification Criteria) @@ -122,54 +236,68 @@ - COMMIT: 3 commits: state, TRACK_COMPLETION, tracks.md update - VERIFY: All 10 VCs pass -## Commit Log (Expected, 12-15 atomic commits) +## Commit Log (Expected, 30-35 atomic commits) -1. (Phase 0) `refactor(type_aliases): promote Metadata to @dataclass(frozen=True, slots=True)` (Tier 3) -2. (Phase 0) `test(type_aliases): add Metadata dataclass regression-guard suite` (Tier 3) -3. (Phase 1) `refactor(session_logger): migrate CommsLogEntry access sites to Metadata dataclass` (Tier 3) -4. (Phase 1) `refactor(multi_agent_conductor): migrate CommsLogEntry access sites` (Tier 3) -5. (Phase 1) `refactor(app_controller): migrate CommsLogEntry access sites` (Tier 3) -6. (Phase 1) [docs] `audit: re-measure effective codepaths after Phase 1` (Tier 2) -7. (Phase 2) `refactor(ai_client): migrate HistoryMessage access sites to Metadata dataclass` (Tier 3) -8. (Phase 2) [docs] `audit: re-measure after Phase 2` (Tier 2) -9. (Phase 3) `refactor(aggregate): migrate FileItem access sites` (Tier 3) -10. (Phase 3) `refactor(app_controller): migrate FileItem access sites` (Tier 3) -11. (Phase 3) `refactor(gui_2): migrate FileItem access sites` (Tier 3) -12. (Phase 3) [docs] `audit: re-measure after Phase 3` (Tier 2) -13. (Phase 4) `refactor(mcp_client): migrate ToolDefinition + ToolCall access sites` (Tier 3) -14. (Phase 4) `refactor(ai_client): migrate ToolDefinition + ToolCall access sites (tool loop section)` (Tier 3) -15. (Phase 4) [docs] `audit: re-measure after Phase 4` (Tier 2) -16. (Phase 5) N commits, 1 per file (varies) -17. (Phase 6) `conductor(state): metadata_promotion_20260624 SHIPPED` (Tier 2) -18. (Phase 6) `docs(reports): TRACK_COMPLETION_metadata_promotion_20260624` (Tier 2) -19. (Phase 6) `conductor(tracks): add metadata_promotion_20260624 row` (Tier 2) +1. (Phase 0) `refactor(type_aliases): add per-aggregate dataclasses (CommsLogEntry, HistoryMessage, ToolDefinition, ...)` +2. (Phase 0) `feat(rag_engine): add RAGChunk dataclass` +3. (Phase 0) `refactor(models): complete ContextPreset schema with missing fields` +4. (Phase 0) `test(type_aliases): add per-aggregate dataclass regression-guard suite` +5. (Phase 0) `docs(styleguides): clarify when to promote to per-aggregate dataclass` +6. (Phase 1) `refactor(gui_2): migrate Ticket access sites to direct field access` +7. (Phase 1) `refactor(app_controller,conductor_tech_lead): migrate Ticket access sites` +8. (Phase 1) `refactor(models): remove legacy Ticket.get() method` +9. (Phase 2) `refactor(aggregate): migrate FileItem access sites` +10. (Phase 2) `refactor(ai_client,app_controller): migrate FileItem access sites` +11. (Phase 3) `refactor(session_logger): migrate CommsLogEntry access sites` +12. (Phase 3) `refactor(multi_agent_conductor): migrate CommsLogEntry access sites` +13. (Phase 3) `refactor(app_controller): migrate CommsLogEntry access sites` +14. (Phase 4) `refactor(gui_2): migrate HistoryMessage access sites` +15. (Phase 5) `refactor(ai_client): migrate ChatMessage access sites in _send_anthropic/_send_deepseek` +16. (Phase 5) `refactor(ai_client): migrate ChatMessage access sites in _send_grok/_send_qwen` +17. (Phase 5) `refactor(ai_client): migrate ChatMessage access sites in _send_minimax/_send_llama` +18. (Phase 6) `refactor(app_controller): migrate UsageStats access sites` +19. (Phase 7) `refactor(ai_client): migrate ToolCall access sites in tool loop section` +20. (Phase 7) `refactor(mcp_client): migrate ToolCall access sites in tool loop section` +21. (Phase 8) `refactor(mcp_client): migrate ToolDefinition access sites` +22. (Phase 8) `refactor(ai_client): migrate ToolDefinition access sites` +23. (Phase 9) `refactor(rag_engine,aggregate,app_controller): migrate RAGChunk access sites` +24. (Phase 10) `refactor(gui_2): migrate SessionInsights, DiscussionSettings, CustomSlice, MMAUsageStats` +25. (Phase 10) `refactor(app_controller): migrate ProviderPayload, UIPanelConfig, PathInfo` +26. (Phase 11) `docs(audit): classify remaining .get() sites as promoted or collapsed-codepath` +27. (Phase 12) `conductor(state): metadata_promotion_20260624 SHIPPED` +28. (Phase 12) `docs(reports): TRACK_COMPLETION_metadata_promotion_20260624` +29. (Phase 12) `conductor(tracks): update metadata_promotion_20260624 row` Plus per-task plan-update commits per the workflow. -## Verification Commands (run at end of each phase + Phase 6) +## Verification Commands (run at end of each phase + Phase 12) ```bash -# VC1: Metadata is a @dataclass(frozen=True, slots=True) -git show HEAD:src/type_aliases.py | head -20 -# Expect: @dataclass(frozen=True, slots=True) class Metadata: +# VC1: Metadata is unchanged +git grep "^Metadata:" src/type_aliases.py +# Expect: Metadata: TypeAlias = dict[str, Any] -# VC2: 107 .get('key', ...) sites replaced +# VC2: Each new sub-aggregate is its OWN @dataclass(frozen=True, slots=True) +git grep -A 1 "^class CommsLogEntry\|^class HistoryMessage\|^class ToolDefinition\|^class RAGChunk\|^class SessionInsights\|^class DiscussionSettings\|^class CustomSlice\|^class MMAUsageStats\|^class ProviderPayload\|^class UIPanelConfig\|^class PathInfo" src/ +# Expect: each followed by @dataclass(frozen=True, slots=True) + +# VC3: Existing dataclasses reused +git grep "class Ticket\|class FileItem\|class ToolCall\|class ChatMessage\|class UsageStats" src/ +# Expect: existing classes unchanged + +# VC4: 107 .get('key', ...) sites on known aggregates replaced git grep -E "\.get\('[a-z_]+'," HEAD -- 'src/*.py' | wc -l -# Expect: <20 (only legitimate non-Metadata uses) +# Expect: only collapsed-codepath sites (FR2; documented in Phase 11 commit) -# VC3: 106 ['key'] subscript sites replaced +# VC5: 106 ['key'] subscript sites on known aggregates replaced git grep -E "\[[ ]*'[a-z_]+'[ ]*\]" HEAD -- 'src/*.py' | wc -l -# Expect: <20 (only legitimate non-Metadata uses) +# Expect: only legitimate non-aggregate uses -# VC4: 12+ tests pass -uv run python -m pytest tests/test_metadata_dataclass.py -v -# Expect: 12/12 pass +# VC6: 60+ tests pass (5+ per new dataclass, 12 dataclasses) +uv run pytest tests/test_comms_log_entry.py tests/test_history_message.py tests/test_tool_definition.py tests/test_rag_chunk.py tests/test_session_insights.py tests/test_discussion_settings.py tests/test_custom_slice.py tests/test_mma_usage_stats.py tests/test_provider_payload.py tests/test_ui_panel_config.py tests/test_path_info.py tests/test_context_preset_schema.py -v +# Expect: all pass -# VC5: 5 sub-aggregate TypeAliases point to Metadata -git grep "TypeAlias = " HEAD:src/type_aliases.py -# Expect: CommsLogEntry: TypeAlias = Metadata, etc. - -# VC6: Effective codepaths drops by >= 2 orders of magnitude +# VC7: Effective codepaths drops by >= 2 orders of magnitude uv run python -c " import sys sys.path.insert(0, 'scripts/code_path_audit') @@ -183,7 +311,7 @@ print(f'Effective codepaths: {total:.3e} (baseline: 4.014e+22)') " # Expect: < 1e+20 -# VC7: 7 audit gates pass +# VC8: 7 audit gates pass uv run python scripts/audit_weak_types.py --strict uv run python scripts/generate_type_registry.py --check uv run python scripts/audit_main_thread_imports.py @@ -193,7 +321,7 @@ uv run python scripts/audit_exception_handling.py --strict uv run python scripts/audit_optional_in_3_files.py --strict # All exit 0 -# VC8: 10/11 batched tiers +# VC9: 10/11 batched tiers uv run python scripts/run_tests_batched.py # Expect: 10/11 PASS ``` @@ -201,16 +329,18 @@ uv run python scripts/run_tests_batched.py ## Notes for Tier 3 workers - **Pattern consistency**: For each access site, the canonical pattern is `entry.field_name or default_value` for nullable fields, `entry.field_name` for required fields. +- **Per-aggregate dataclass reference**: `src/openai_schemas.py` (the canonical pattern for `ToolCall`, `ChatMessage`, `UsageStats`, `ToolCallFunction`, `NormalizedResponse`); `src/models.py:533` (`FileItem` with `to_dict()` / `from_dict()` round-trip). - **Dynamic keys** (e.g., `entry[variable_name]` where the key is not a static string): keep as `entry.to_dict()[variable_name]` for those rare cases. The dataclass handles the common case. -- **Polymorphic construction** (e.g., `entry = {'role': 'user', 'content': 'hi'}`): replace with `entry = Metadata(role='user', content='hi')`. If the dict is dynamic, use `entry = Metadata.from_dict(raw_dict)`. +- **Polymorphic construction** (e.g., `entry = {'role': 'user', 'content': 'hi'}`): replace with `entry = HistoryMessage(role='user', content='hi')`. If the dict is dynamic, use `entry = HistoryMessage.from_dict(raw_dict)`. - **JSON serialization**: `json.dumps(entry.to_dict())` (not `json.dumps(entry)` which would fail on dataclass). - **Indentation**: 1-space per level. - **No comments** in source code (per AGENTS.md). -- **Per-phase regression-guard test runs**: after each phase, run the full batched test suite. If a phase causes a regression, REVERT the phase commit and investigate (don't try to fix forward). +- **Per-phase regression-guard test runs**: after each phase, run the per-aggregate test files + the full batched test suite. If a phase causes a regression, REVERT the phase commit and investigate (don't try to fix forward). ## Notes for Tier 2 reviewer -- The dataclass is the central artifact. After Phase 0, `Metadata()` constructor works. Each subsequent phase migrates consumers in a specific file. -- The 4.01e22 metric drops per phase. Document the drop in `docs/reports/metadata_promotion_progress.md` (new file). +- The per-aggregate dataclasses are the central artifacts. After Phase 0, every new dataclass is importable. Each subsequent phase migrates the consumers in a specific file. +- The 4.01e22 metric drops per phase. Document the drop in the TRACK_COMPLETION report. - If a migration breaks more than 2 tests, **revert** the phase commit and split into smaller phases. Don't accumulate broken state. - The RAG test pre-existing flake is acceptable. Document it but don't try to fix. +- The classification in Phase 11 (collapsed-codepath vs promoted) is auditable; every remaining `.get()` site must have a justification in the commit message. \ No newline at end of file