Private
Public Access
conductor(plan): correct metadata_promotion_20260624 plan to 13 per-aggregate phases
This commit is contained in:
@@ -1,116 +1,230 @@
|
||||
# Plan: metadata_promotion_20260624
|
||||
|
||||
5 phases, 12-15 tasks, 12+ atomic commits. Per-task TDD red-first. Tier 3 workers execute; Tier 2 reviews per phase.
|
||||
> **CORRECTED 2026-06-25 (Tier 1 audit).** The original plan (commit `e50bebdd`, 2026-06-25) proposed a single shared `@dataclass(frozen=True, slots=True) Metadata` with ~200 fields for all 5 sub-aggregates. That proposal was REJECTED on 2026-06-25 (user direction): each sub-aggregate is its OWN dataclass with its OWN fields. The corrected plan has 12 phases (one per sub-aggregate), uses existing dataclasses where they exist (`Ticket`, `FileItem`, `ToolCall`, `ChatMessage`, `UsageStats`), and adds new per-aggregate dataclasses for the 8 aggregates that don't have one yet. See `docs/reports/PLANNING_CORRECTION_metadata_promotion_20260625.md` for the full rationale.
|
||||
|
||||
## Phase 0: Design the dataclass + add regression-guard test (2 tasks, 2 commits)
|
||||
13 phases, 30-35 tasks, 30+ atomic commits. Per-task TDD red-first. Tier 3 workers execute; Tier 2 reviews per phase.
|
||||
|
||||
**Focus:** Create the `@dataclass(frozen=True, slots=True) Metadata` in `src/type_aliases.py` + add the test file. No consumer migration yet.
|
||||
## Phase 0: Design the per-aggregate dataclasses + add regression-guard test stubs (5 tasks, 5 commits)
|
||||
|
||||
- [x] **Task 0.1** [Tier 3]: Design the dataclass.
|
||||
**Focus:** Add the NEW dataclasses to `src/type_aliases.py` (the type-system aggregates that don't have a parent module); reuse the existing dataclasses in `src/models.py` and `src/openai_schemas.py`. No consumer migration yet.
|
||||
|
||||
- [ ] **Task 0.1** [Tier 3]: Add NEW dataclasses to `src/type_aliases.py`.
|
||||
- WHERE: `src/type_aliases.py` (current 30 lines)
|
||||
- WHAT:
|
||||
- Replace `Metadata: TypeAlias = dict[str, Any]` with `@dataclass(frozen=True, slots=True) class Metadata: ...`
|
||||
- Add the canonical fields (from the spec §FR1): role, content, tool_calls, tool_call_id, name, args, source_tier, model, id, ts, description, depends_on, status, manual_block, completed_tickets, auto_start, command, script, output, error, tier, path, full_path, filename, mtime, size + the other ~150-180 distinct keys from the `.get` and `[]` site analysis
|
||||
- Add `to_dict()` method (for JSON serialization) + `from_dict()` classmethod (filters unknown keys)
|
||||
- Add `__post_init__` for any derived value validation
|
||||
- KEEP the 5 sub-aggregate TypeAliases (`CommsLogEntry: TypeAlias = Metadata` etc.) — they all point to the new dataclass
|
||||
- Add `@dataclass(frozen=True, slots=True) class CommsLogEntry` with `ts, role, kind, direction, model, source_tier, content, error` (8 fields, all with defaults)
|
||||
- Add `@dataclass(frozen=True, slots=True) class HistoryMessage` with `role, content, tool_calls, tool_call_id, name, ts` (6 fields)
|
||||
- Add `@dataclass(frozen=True, slots=True) class ToolDefinition` with `name, description, parameters, auto_start` (4 fields)
|
||||
- Add `@dataclass(frozen=True, slots=True) class SessionInsights` with `total_tokens, call_count, burn_rate, session_cost, completed_tickets, efficiency` (6 fields)
|
||||
- Add `@dataclass(frozen=True, slots=True) class DiscussionSettings` with `temperature, top_p, max_output_tokens` (3 fields)
|
||||
- Add `@dataclass(frozen=True, slots=True) class CustomSlice` with `tag, comment, start_line, end_line` (4 fields)
|
||||
- Add `@dataclass(frozen=True, slots=True) class MMAUsageStats` with `model, input, output` (3 fields)
|
||||
- Add `@dataclass(frozen=True, slots=True) class ProviderPayload` with `script, args, output, source_tier` (4 fields)
|
||||
- Add `@dataclass(frozen=True, slots=True) class UIPanelConfig` with `separate_message_panel, separate_response_panel, separate_tool_calls_panel` (3 fields)
|
||||
- Add `@dataclass(frozen=True, slots=True) class PathInfo` with `logs_dir, scripts_dir, project_root` (3 nested fields)
|
||||
- Each dataclass has a paired `to_dict()` (for JSON serialization) and `from_dict()` classmethod (filters unknown keys, per FR5)
|
||||
- KEEP `Metadata: TypeAlias = dict[str, Any]` UNCHANGED (the catch-all for collapsed codepaths)
|
||||
- KEEP `CommsLog: TypeAlias = list[CommsLogEntry]`, `History: TypeAlias = list[HistoryMessage]`, `FileItems: TypeAlias = list[FileItem]` (the list aliases still work; the element types are now per-aggregate dataclasses)
|
||||
- KEEP `JsonValue`, `JsonPrimitive`, `CommsLogCallback`, `FileItemsDiff` unchanged
|
||||
- HOW: `manual-slop_edit_file` for surgical edits
|
||||
- SAFETY: `ast.parse` OK; `from src.type_aliases import Metadata` OK; `Metadata()` constructor works
|
||||
- [x] **COMMIT:** `refactor(type_aliases): promote Metadata to @dataclass(frozen=True, slots=True)` (Tier 3)
|
||||
- [x] **GIT NOTE:** Metadata is now a typed dataclass. The 5 sub-aggregate TypeAliases all point to the same class. The consumer migration is in subsequent phases.
|
||||
- HOW: `manual-slop_edit_file` for surgical edits (or `write_file` if the file is being substantially restructured)
|
||||
- SAFETY: `ast.parse` OK; `from src.type_aliases import CommsLogEntry, HistoryMessage, ToolDefinition, SessionInsights, DiscussionSettings, CustomSlice, MMAUsageStats, ProviderPayload, UIPanelConfig, PathInfo` OK; constructors work
|
||||
- [ ] **COMMIT:** `refactor(type_aliases): add per-aggregate dataclasses (CommsLogEntry, HistoryMessage, ToolDefinition, ...)` (Tier 3)
|
||||
- [ ] **GIT NOTE:** NEW dataclasses added to `src/type_aliases.py`. `Metadata: TypeAlias = dict[str, Any]` is UNCHANGED (the catch-all for collapsed codepaths). No consumer migration yet.
|
||||
|
||||
- [x] **Task 0.2** [Tier 3]: Create `tests/test_metadata_dataclass.py`.
|
||||
- WHERE: NEW FILE `tests/test_metadata_dataclass.py`
|
||||
- WHAT: 12+ tests:
|
||||
- `test_empty_constructor`: `Metadata()` returns an instance with all fields as default values
|
||||
- `test_constructor_with_kwargs`: `Metadata(role='user', content='hi')` works
|
||||
- `test_field_access`: `entry.role` works
|
||||
- `test_frozen`: trying to mutate a field raises `dataclasses.FrozenInstanceError`
|
||||
- `test_slots`: `__slots__` is set (no `__dict__`)
|
||||
- `test_to_dict`: `entry.to_dict()` returns the same dict as the old `dict[str, Any]` shape
|
||||
- `test_from_dict`: `Metadata.from_dict({'role': 'user'})` works; unknown keys are silently filtered
|
||||
- `test_from_dict_preserves_all_fields`: full round-trip
|
||||
- `test_equality`: two `Metadata(role='user')` instances are equal
|
||||
- `test_hashable`: `Metadata(role='user')` can be in a set/dict
|
||||
- `test_type_aliases_resolve_to_metadata`: `CommsLogEntry is Metadata`, `HistoryMessage is Metadata`, etc.
|
||||
- `test_pickle`: `pickle.dumps(Metadata(...))` works
|
||||
- HOW: `write_file` to create the new test file (with all 12 tests)
|
||||
- SAFETY: `uv run python -m pytest tests/test_metadata_dataclass.py -v` shows 12/12 pass
|
||||
- [x] **COMMIT:** `test(type_aliases): add Metadata dataclass regression-guard suite` (Tier 3)
|
||||
- [x] **GIT NOTE:** 12 tests cover the dataclass behavior. The consumer migration is in subsequent phases; this commit only adds the dataclass + tests.
|
||||
- [ ] **Task 0.2** [Tier 3]: Add `RAGChunk` dataclass to `src/rag_engine.py`.
|
||||
- WHERE: `src/rag_engine.py` (the parent module for RAG)
|
||||
- WHAT: `@dataclass(frozen=True, slots=True) class RAGChunk` with `document, path, score, metadata` (4 fields, all with defaults); paired `to_dict()` / `from_dict()`
|
||||
- HOW: `manual-slop_edit_file`
|
||||
- SAFETY: `from src.rag_engine import RAGChunk` OK; constructor works
|
||||
- [ ] **COMMIT:** `feat(rag_engine): add RAGChunk dataclass` (Tier 3)
|
||||
- [ ] **GIT NOTE:** NEW dataclass added to `src/rag_engine.py`. No consumer migration yet.
|
||||
|
||||
## Phase 1: Migrate `CommsLogEntry` consumers (~150 sites, 1 commit per file)
|
||||
- [ ] **Task 0.3** [Tier 3]: Audit and complete `ContextPreset` schema in `src/models.py`.
|
||||
- WHERE: `src/models.py` (the parent module for ContextPreset)
|
||||
- WHAT: `ContextPreset` exists at `src/models.py:932` but is partial. Add missing fields based on access patterns: `name, files (FileItems), screenshots (list[str])` minimum; audit the actual usage and add any other required fields; ensure paired `to_dict()` / `from_dict()`
|
||||
- HOW: `manual-slop_edit_file`
|
||||
- SAFETY: existing `ContextPreset` consumers continue to work; the `to_dict()` round-trip is lossless
|
||||
- [ ] **COMMIT:** `refactor(models): complete ContextPreset schema with missing fields` (Tier 3)
|
||||
- [ ] **GIT NOTE:** `ContextPreset` schema extended. Existing consumers unchanged.
|
||||
|
||||
**Focus:** The smallest sub-aggregate first. `CommsLogEntry` is used in `app_controller.py` + `multi_agent_conductor.py` + `session_logger.py`. The migration is mechanical: `entry.get('key', default)` → `entry.key or default`.
|
||||
- [ ] **Task 0.4** [Tier 3]: Create `tests/test_metadata_dataclass.py` (split into per-aggregate test files per FR G7).
|
||||
- WHERE: NEW FILES: `tests/test_comms_log_entry.py`, `tests/test_history_message.py`, `tests/test_tool_definition.py`, `tests/test_rag_chunk.py`, `tests/test_session_insights.py`, `tests/test_discussion_settings.py`, `tests/test_custom_slice.py`, `tests/test_mma_usage_stats.py`, `tests/test_provider_payload.py`, `tests/test_ui_panel_config.py`, `tests/test_path_info.py`, `tests/test_context_preset_schema.py`
|
||||
- WHAT: 5+ tests per file: constructor with kwargs, field access, frozen (raises `FrozenInstanceError`), `to_dict()` / `from_dict()` round-trip, equality, hashability, default values
|
||||
- HOW: `write_file` per file
|
||||
- SAFETY: `uv run pytest tests/test_comms_log_entry.py -v` shows 5/5 pass (and similarly for the other 11 files)
|
||||
- [ ] **COMMIT:** `test(type_aliases): add per-aggregate dataclass regression-guard suite` (Tier 3)
|
||||
- [ ] **GIT NOTE:** 12 test files, 5+ tests each. The consumer migration is in subsequent phases; this commit only adds the new dataclasses + tests.
|
||||
|
||||
- [x] **Task 1.1** [Tier 3]: Migrate `src/session_logger.py` (the smallest, the writer-side).
|
||||
- WHERE: `src/session_logger.py` (~218 lines; ~30 access sites)
|
||||
- WHAT: For each `entry.get('key', default)` and `entry['key']` where `entry` is `CommsLogEntry`, replace with `entry.key or default` (or `entry.key` for required fields)
|
||||
- [ ] **Task 0.5** [Tier 2]: Document the FR6 collapsed-codepath classification rule.
|
||||
- WHERE: `conductor/code_styleguides/type_aliases.md` (small clarification, NOT a rewrite)
|
||||
- WHAT: Add a one-paragraph "When to promote to a per-aggregate dataclass" rule: when a sub-aggregate has stable distinct fields, promote it to its OWN dataclass; do NOT share one mega-dataclass across concepts; `Metadata: TypeAlias = dict[str, Any]` is preserved for collapsed codepaths (TOML config, generic JSON parsing, polymorphic log dumping) only. Reference this track's correction as the canonical example.
|
||||
- HOW: `manual-slop_edit_file`
|
||||
- SAFETY: styleguide is consistent with the corrected design
|
||||
- [ ] **COMMIT:** `docs(styleguides): clarify when to promote to per-aggregate dataclass` (Tier 2)
|
||||
- [ ] **GIT NOTE:** Styleguide clarification. The corrected design is: per-aggregate dataclasses for known sub-aggregates; `Metadata: dict[str, Any]` for collapsed codepaths only.
|
||||
|
||||
## Phase 1: Migrate `Ticket` consumers (~30 sites, 2 commits)
|
||||
|
||||
**Focus:** `Ticket` is already a dataclass (`src/models.py:302`); just migrate the consumers from `t.get('id', '')` to `t.id`. The legacy `Ticket.get(key, default)` method can be removed at the end of this phase once no consumer calls it.
|
||||
|
||||
- [ ] **Task 1.1** [Tier 3]: Migrate `src/gui_2.py` Ticket consumers.
|
||||
- WHERE: `src/gui_2.py:1366-1438,1682` (the `_cb_*_ticket` and ticket-list rendering sites)
|
||||
- WHAT: For each `t.get('id', '')`, `t.get('depends_on', [])`, `t.get('manual_block', False)`, `t.get('status')` → `t.id`, `t.depends_on`, `t.manual_block`, `t.status`
|
||||
- HOW: `manual-slop_edit_file` per site
|
||||
- SAFETY: Run `tests/test_session_logger_optimization.py` + `tests/test_session_logger_reset.py` + `tests/test_session_logging.py` + `tests/test_logging_e2e.py` + the new `tests/test_metadata_dataclass.py`
|
||||
- [x] **COMMIT:** `refactor(session_logger): migrate CommsLogEntry access sites to Metadata dataclass` (Tier 3)
|
||||
- [x] **GIT NOTE:** Migrated ~30 access sites in session_logger.py. Verified by the 4 session_logger test files + 12 metadata dataclass tests.
|
||||
- SAFETY: Run `tests/test_ticket_queue.py` + `tests/test_per_ticket_model.py` + `tests/test_manual_block.py` + the new per-aggregate test files
|
||||
- [ ] **COMMIT:** `refactor(gui_2): migrate Ticket access sites to direct field access` (Tier 3)
|
||||
- [ ] **GIT NOTE:** Migrated ~15 Ticket access sites in `src/gui_2.py`. Verified by the ticket test files.
|
||||
|
||||
- [x] **Task 1.2** [Tier 3]: Migrate `src/multi_agent_conductor.py` (~70 access sites)
|
||||
- [x] **Task 1.3** [Tier 3]: Migrate `src/app_controller.py` (the bulk — ~50 access sites that are CommsLogEntry-specific)
|
||||
- [x] **COMMIT (1.2):** `refactor(multi_agent_conductor): migrate CommsLogEntry access sites` (Tier 3)
|
||||
- [x] **COMMIT (1.3):** `refactor(app_controller): migrate CommsLogEntry access sites` (Tier 3)
|
||||
- [x] **GIT NOTES (1.2, 1.3):** Per-file counts. Verified by the full batched test suite (no regression).
|
||||
- [x] **Task 1.4** [Tier 2]: Re-measure effective codepaths after Phase 1.
|
||||
- EXPECTED: drops from 4.014e+22 to ~4e+19 (CommsLogEntry has the most consumers; their branch counts drop significantly)
|
||||
- Document in `docs/reports/metadata_promotion_progress.md` (new file)
|
||||
|
||||
## Phase 2: Migrate `HistoryMessage` consumers (~80 sites, 1 commit per file)
|
||||
|
||||
**Focus:** `ai_client.py` per-vendor history. The 27 call sites in phase 3 just got migrated to `provider_state.get_history("...")`; this phase migrates the `entry.get('role', ...)` and `entry.get('content', ...)` calls inside those functions.
|
||||
|
||||
- [x] **Task 2.1** [Tier 3]: Migrate `src/ai_client.py` (the bulk — ~80 access sites in `_send_anthropic`, `_send_deepseek`, `_send_grok`, etc.)
|
||||
- [ ] **Task 1.2** [Tier 3]: Migrate `src/conductor_tech_lead.py` and `src/app_controller.py` Ticket consumers.
|
||||
- WHERE: `src/conductor_tech_lead.py:125`; `src/app_controller.py:4810-4868` (the ticket-list mutation sites)
|
||||
- WHAT: Same pattern as 1.1
|
||||
- HOW: `manual-slop_edit_file` per site
|
||||
- SAFETY: Run the 5 per-provider test files + the 12 metadata dataclass tests + the 7 per-provider migration tests
|
||||
- [x] **COMMIT:** `refactor(ai_client): migrate HistoryMessage access sites to Metadata dataclass` (Tier 3)
|
||||
- [x] **GIT NOTE:** Migrated ~80 access sites in ai_client.py. The HistoryMessage aggregate now uses direct field access.
|
||||
- [x] **Task 2.2** [Tier 2]: Re-measure. EXPECTED: drops further. Document.
|
||||
- SAFETY: Same as 1.1
|
||||
- [ ] **COMMIT:** `refactor(app_controller,conductor_tech_lead): migrate Ticket access sites` (Tier 3)
|
||||
- [ ] **GIT NOTE:** Migrated ~15 Ticket access sites across 2 files. Verified.
|
||||
- [ ] **Task 1.3** [Tier 2]: Remove the legacy `Ticket.get(key, default)` method.
|
||||
- WHERE: `src/models.py` (the `get` method on `Ticket`)
|
||||
- WHAT: After all consumers have migrated, remove the `get` method
|
||||
- HOW: `manual-slop_py_remove_def`
|
||||
- SAFETY: Re-run the full batched test suite; no remaining `.get(key, default)` on Ticket consumers
|
||||
- [ ] **COMMIT:** `refactor(models): remove legacy Ticket.get() method` (Tier 2)
|
||||
- [ ] **GIT NOTE:** Legacy compat method removed. Direct field access is now the only path.
|
||||
|
||||
## Phase 3: Migrate `FileItem` consumers (~200 sites, 1 commit per file)
|
||||
## Phase 2: Migrate `FileItem` consumers (~10 sites, 2 commits)
|
||||
|
||||
**Focus:** `aggregate.py` + `gui_2.py` + `app_controller.py` (the rest of it). This is the largest phase. `FileItem` is the most polymorphic — many distinct keys.
|
||||
**Focus:** `FileItem` is already a dataclass (`src/models.py:533`); migrate the consumers.
|
||||
|
||||
- [x] **Task 3.1** [Tier 3]: Migrate `src/aggregate.py` (~50 access sites)
|
||||
- [x] **Task 3.2** [Tier 3]: Migrate `src/app_controller.py` (the remaining ~50 access sites; some overlap with phase 1 CommsLogEntry)
|
||||
- [x] **Task 3.3** [Tier 3]: Migrate `src/gui_2.py` (~100 access sites; the largest)
|
||||
- [x] **COMMIT (3.1, 3.2, 3.3):** 3 atomic commits, one per file
|
||||
- [x] **GIT NOTES:** Per-file counts. Verified.
|
||||
- [x] **Task 3.4** [Tier 2]: Re-measure. EXPECTED: significant drop. Document.
|
||||
- [ ] **Task 2.1** [Tier 3]: Migrate `src/aggregate.py` FileItem consumers.
|
||||
- WHERE: `src/aggregate.py:418,421`
|
||||
- WHAT: `item.get('custom_slices', [])` → `item.custom_slices`; `item.get('content', '')` → `item.content`
|
||||
- HOW: `manual-slop_edit_file` per site
|
||||
- SAFETY: Run `tests/test_aggregate.py` + `tests/test_file_item_model.py` + the new per-aggregate test files
|
||||
- [ ] **COMMIT:** `refactor(aggregate): migrate FileItem access sites` (Tier 3)
|
||||
- [ ] **GIT NOTE:** Migrated ~5 FileItem access sites.
|
||||
|
||||
## Phase 4: Migrate `ToolDefinition` + `ToolCall` consumers (~150 sites, 2 commits)
|
||||
- [ ] **Task 2.2** [Tier 3]: Migrate `src/ai_client.py` and `src/app_controller.py` FileItem consumers.
|
||||
- WHERE: `src/ai_client.py:2565,2807,2898`; `src/app_controller.py:3508`
|
||||
- WHAT: `fi.get('path', 'attachment')` → `fi.path`; `f['path'] for f in file_items` → `f.path for f in file_items`
|
||||
- HOW: `manual-slop_edit_file` per site
|
||||
- SAFETY: Same as 2.1
|
||||
- [ ] **COMMIT:** `refactor(ai_client,app_controller): migrate FileItem access sites` (Tier 3)
|
||||
- [ ] **GIT NOTE:** Migrated ~5 FileItem access sites across 2 files.
|
||||
|
||||
**Focus:** `mcp_client.py` + `ai_client.py` (the tool loop section). These are the most typed-shaped; should be clean.
|
||||
## Phase 3: Migrate `CommsLogEntry` consumers (~30 sites, 3 commits)
|
||||
|
||||
- [x] **Task 4.1** [Tier 3]: Migrate `src/mcp_client.py` (~94 access sites — the bulk)
|
||||
- [x] **Task 4.2** [Tier 3]: Migrate `src/ai_client.py` (the tool loop section only — ~56 access sites)
|
||||
- [x] **COMMIT (4.1, 4.2):** 2 atomic commits
|
||||
- [x] **GIT NOTES:** Per-file counts. Verified.
|
||||
- [x] **Task 4.3** [Tier 2]: Re-measure. EXPECTED: another drop. Document.
|
||||
**Focus:** New dataclass added in Phase 0; now wire it into the consumers.
|
||||
|
||||
## Phase 5: Migrate remaining `Metadata` direct usage (~115 sites, multiple commits)
|
||||
- [ ] **Task 3.1** [Tier 3]: Migrate `src/session_logger.py`.
|
||||
- WHERE: `src/session_logger.py` (~30 access sites; the writer-side)
|
||||
- WHAT: `entry.get('source_tier', 'main')` → `entry.source_tier`; `entry.get('model', 'unknown')` → `entry.model`; etc.
|
||||
- HOW: `manual-slop_edit_file` per site
|
||||
- SAFETY: Run `tests/test_session_logger_optimization.py` + `tests/test_session_logger_reset.py` + `tests/test_session_logging.py` + `tests/test_logging_e2e.py` + `tests/test_comms_log_entry.py`
|
||||
- [ ] **COMMIT:** `refactor(session_logger): migrate CommsLogEntry access sites` (Tier 3)
|
||||
- [ ] **GIT NOTE:** Migrated ~30 access sites.
|
||||
|
||||
**Focus:** The 115 consumer functions that use `Metadata` directly (not via a sub-aggregate alias). This is the catch-all. Many of these are in `gui_2.py` (already partly migrated in phase 3) + `models.py` + `paths.py` + others.
|
||||
- [ ] **Task 3.2** [Tier 3]: Migrate `src/multi_agent_conductor.py` (~20 sites)
|
||||
- [ ] **Task 3.3** [Tier 3]: Migrate `src/app_controller.py` CommsLogEntry section (~10 sites)
|
||||
- [ ] **COMMIT (3.2, 3.3):** 2 atomic commits
|
||||
- [ ] **Task 3.4** [Tier 2]: Re-measure effective codepaths after Phase 3.
|
||||
|
||||
- [x] **Task 5.1** [Tier 3]: Audit remaining `Metadata` direct-usage sites.
|
||||
- WHICH: `git grep -nE "Metadata\b" -- 'src/*.py'` filtered to NON-sub-aggregate usages
|
||||
- HOW: `git grep -lE "Metadata\b" -- 'src/*.py'` then per-file count
|
||||
- EXPECTED: ~115 sites across 5-8 files
|
||||
- [x] **Task 5.2-5.N** [Tier 3]: Per-file migration (1 commit per file, in decreasing order of access site count)
|
||||
- For each file: `manual-slop_edit_file` per site
|
||||
- SAFETY: Run the affected test file + `tests/test_metadata_dataclass.py`
|
||||
- [x] **COMMIT (5.2-5.N):** 1 per file. All atomic.
|
||||
## Phase 4: Migrate `HistoryMessage` consumers (~20 sites, 1 commit)
|
||||
|
||||
## Phase 6: Verification + end-of-track (1 task, 3 commits)
|
||||
**Focus:** UI-layer discussion history (NOT provider-side `ChatMessage`; these are distinct layers per `data_structure_strengthening_20260606` §3.1).
|
||||
|
||||
- [ ] **Task 4.1** [Tier 3]: Migrate `src/gui_2.py` discussion UI sites.
|
||||
- WHERE: `src/gui_2.py` (~20 sites; the editable per-turn message list)
|
||||
- WHAT: `entry['role']` → `entry.role`; etc.
|
||||
- HOW: `manual-slop_edit_file` per site
|
||||
- SAFETY: Run the per-aggregate test files
|
||||
- [ ] **COMMIT:** `refactor(gui_2): migrate HistoryMessage access sites` (Tier 3)
|
||||
- [ ] **GIT NOTE:** Migrated ~20 HistoryMessage access sites.
|
||||
- [ ] **Task 4.2** [Tier 2]: Re-measure.
|
||||
|
||||
## Phase 5: Wire `ChatMessage` into per-vendor send paths (~27 sites, 3 commits)
|
||||
|
||||
**Focus:** `ChatMessage` is already in `src/openai_schemas.py:48`; wire it into the per-vendor send paths that were migrated to `provider_state.get_history("...")` in `code_path_audit_phase_3_provider_state_20260624`.
|
||||
|
||||
- [ ] **Task 5.1** [Tier 3]: Migrate `_send_anthropic` and `_send_deepseek` (~9 sites)
|
||||
- [ ] **Task 5.2** [Tier 3]: Migrate `_send_grok` and `_send_qwen` (~9 sites)
|
||||
- [ ] **Task 5.3** [Tier 3]: Migrate `_send_minimax` and `_send_llama` (~9 sites)
|
||||
- [ ] **COMMIT (5.1, 5.2, 5.3):** 3 atomic commits
|
||||
- [ ] **Task 5.4** [Tier 2]: Re-measure.
|
||||
|
||||
## Phase 6: Wire `UsageStats` into per-call usage aggregation (~10 sites, 1 commit)
|
||||
|
||||
**Focus:** `UsageStats` is already in `src/openai_schemas.py:68`; wire it into the per-call usage aggregation in `app_controller.py`.
|
||||
|
||||
- [ ] **Task 6.1** [Tier 3]: Migrate `src/app_controller.py:2299-2309`.
|
||||
- WHERE: `src/app_controller.py:2299-2309` (the `mma_tier_usage` aggregation sites)
|
||||
- WHAT: `u.get('input_tokens', 0) or 0` → `u.input_tokens or 0`; etc.
|
||||
- HOW: `manual-slop_edit_file`
|
||||
- SAFETY: Run `tests/test_token_usage.py` + `tests/test_usage_analytics_popout_sim.py` + `tests/test_openai_schemas.py`
|
||||
- [ ] **COMMIT:** `refactor(app_controller): migrate UsageStats access sites` (Tier 3)
|
||||
- [ ] **GIT NOTE:** Migrated ~10 UsageStats access sites.
|
||||
|
||||
## Phase 7: Wire `ToolCall` into the tool loop section (~56 sites, 2 commits)
|
||||
|
||||
**Focus:** `ToolCall` is already in `src/openai_schemas.py:32`; wire it into the tool loop section in `ai_client.py` and `mcp_client.py`.
|
||||
|
||||
- [ ] **Task 7.1** [Tier 3]: Migrate `src/ai_client.py` tool loop section (~56 sites)
|
||||
- [ ] **Task 7.2** [Tier 3]: Verify `src/mcp_client.py` tool loop section (the small subset)
|
||||
- [ ] **COMMIT (7.1, 7.2):** 2 atomic commits
|
||||
|
||||
## Phase 8: Migrate `ToolDefinition` consumers (~94 sites, 2 commits)
|
||||
|
||||
**Focus:** New dataclass added in Phase 0; now wire it into the per-vendor tool builders.
|
||||
|
||||
- [ ] **Task 8.1** [Tier 3]: Migrate `src/mcp_client.py` (~70 sites; the bulk)
|
||||
- [ ] **Task 8.2** [Tier 3]: Migrate `src/ai_client.py` per-vendor tool builders (~24 sites)
|
||||
- [ ] **COMMIT (8.1, 8.2):** 2 atomic commits
|
||||
|
||||
## Phase 9: Migrate `RAGChunk` consumers (~5 sites, 1 commit)
|
||||
|
||||
**Focus:** New dataclass added in Phase 0; migrate the RAG result consumers.
|
||||
|
||||
- [ ] **Task 9.1** [Tier 3]: Migrate `src/rag_engine.py`, `src/aggregate.py`, `src/app_controller.py` RAG chunk consumers.
|
||||
- WHERE: `src/aggregate.py:3259`; `src/app_controller.py:251,4162`
|
||||
- WHAT: `chunk.get('document', '')` → `chunk.document`; etc.
|
||||
- HOW: `manual-slop_edit_file` per site
|
||||
- SAFETY: Run `tests/test_rag_engine.py` + `tests/test_rag_*.py` + `tests/test_rag_chunk.py` (new)
|
||||
- [ ] **COMMIT:** `refactor(rag_engine,aggregate,app_controller): migrate RAGChunk access sites` (Tier 3)
|
||||
- [ ] **GIT NOTE:** Migrated ~5 RAGChunk access sites across 3 files.
|
||||
|
||||
## Phase 10: Migrate small-batch aggregates (~25 sites, 2 commits)
|
||||
|
||||
**Focus:** `SessionInsights`, `DiscussionSettings`, `CustomSlice`, `MMAUsageStats`, `ProviderPayload`, `UIPanelConfig`, `PathInfo`. These are small aggregates with few sites; batch them.
|
||||
|
||||
- [ ] **Task 10.1** [Tier 3]: Migrate `src/gui_2.py` small-batch consumers.
|
||||
- WHERE: `src/gui_2.py:2199-2201,2216,3535,4048-4054,4926-4931` (SessionInsights, DiscussionSettings, CustomSlice, MMAUsageStats)
|
||||
- WHAT: `insights.get('total_tokens', 0)` → `insights.total_tokens`; `entry.get('temperature', 0.7)` → `entry.temperature`; `slc.get('tag', '')` → `slc.tag`; `stats.get('model', 'unknown')` → `stats.model`
|
||||
- HOW: `manual-slop_edit_file` per site
|
||||
- SAFETY: Run the per-aggregate test files + the GUI tests
|
||||
- [ ] **COMMIT:** `refactor(gui_2): migrate SessionInsights, DiscussionSettings, CustomSlice, MMAUsageStats` (Tier 3)
|
||||
- [ ] **GIT NOTE:** Migrated ~20 small-aggregate access sites.
|
||||
|
||||
- [ ] **Task 10.2** [Tier 3]: Migrate `src/app_controller.py` ProviderPayload, UIPanelConfig, PathInfo consumers.
|
||||
- WHERE: `src/app_controller.py:1972-2033,2068-2070,2274-2310` (the project config + UI panel config + provider payload sites)
|
||||
- WHAT: `payload.get('script')` → `payload.script`; `gui_cfg.get('separate_message_panel', False)` → `gui_cfg.separate_message_panel`; `path_info['logs_dir']['path']` → `path_info.logs_dir.path` (nested access)
|
||||
- HOW: `manual-slop_edit_file` per site
|
||||
- SAFETY: Run the per-aggregate test files + the app_controller tests
|
||||
- [ ] **COMMIT:** `refactor(app_controller): migrate ProviderPayload, UIPanelConfig, PathInfo` (Tier 3)
|
||||
- [ ] **GIT NOTE:** Migrated ~5 small-aggregate access sites.
|
||||
|
||||
## Phase 11: `Metadata` collapsed-codepath audit (FR6, 1 task, 1 commit)
|
||||
|
||||
**Focus:** Every remaining `.get('key', default)` site is classified as either (a) "promoted to per-aggregate dataclass → migrated" or (b) "collapsed codepath → keeps Metadata with documented justification."
|
||||
|
||||
- [ ] **Task 11.1** [Tier 2]: Audit remaining `.get('key', default)` sites.
|
||||
- WHERE: `git grep -nE "\.get\('[a-z_]+'," HEAD -- 'src/*.py'`
|
||||
- WHAT: Per-site classification: (a) promoted + migrated (drop from the report), (b) collapsed-codepath (document the justification in the commit message). The expected collapsed-codepath sites are: `self.project.get('paths', {})`, `self.project.get('conductor', {})`, `self.project.get('context_presets', {})`, `self.project.get('discussion', {})`, `gui_cfg.get(...)` (if `UIPanelConfig` doesn't cover it), etc.
|
||||
- HOW: Manual review + commit message
|
||||
- [ ] **COMMIT:** `docs(audit): classify remaining .get() sites as promoted or collapsed-codepath` (Tier 2)
|
||||
- [ ] **GIT NOTE:** Per-site classification. The remaining `.get()` sites are all justified collapsed-codepaths.
|
||||
|
||||
## Phase 12: Verification + end-of-track (1 task, 3 commits)
|
||||
|
||||
**Focus:** Run all 10 VCs; write `TRACK_COMPLETION`; update `state.toml` + `tracks.md`.
|
||||
|
||||
- [x] **Task 6.1** [Tier 2]:
|
||||
- [ ] **Task 12.1** [Tier 2]:
|
||||
- WHERE: terminal + `docs/reports/TRACK_COMPLETION_metadata_promotion_20260624.md` (NEW)
|
||||
- WHAT:
|
||||
- VC1-VC10 verification (see spec.md §Verification Criteria)
|
||||
@@ -122,54 +236,68 @@
|
||||
- COMMIT: 3 commits: state, TRACK_COMPLETION, tracks.md update
|
||||
- VERIFY: All 10 VCs pass
|
||||
|
||||
## Commit Log (Expected, 12-15 atomic commits)
|
||||
## Commit Log (Expected, 30-35 atomic commits)
|
||||
|
||||
1. (Phase 0) `refactor(type_aliases): promote Metadata to @dataclass(frozen=True, slots=True)` (Tier 3)
|
||||
2. (Phase 0) `test(type_aliases): add Metadata dataclass regression-guard suite` (Tier 3)
|
||||
3. (Phase 1) `refactor(session_logger): migrate CommsLogEntry access sites to Metadata dataclass` (Tier 3)
|
||||
4. (Phase 1) `refactor(multi_agent_conductor): migrate CommsLogEntry access sites` (Tier 3)
|
||||
5. (Phase 1) `refactor(app_controller): migrate CommsLogEntry access sites` (Tier 3)
|
||||
6. (Phase 1) [docs] `audit: re-measure effective codepaths after Phase 1` (Tier 2)
|
||||
7. (Phase 2) `refactor(ai_client): migrate HistoryMessage access sites to Metadata dataclass` (Tier 3)
|
||||
8. (Phase 2) [docs] `audit: re-measure after Phase 2` (Tier 2)
|
||||
9. (Phase 3) `refactor(aggregate): migrate FileItem access sites` (Tier 3)
|
||||
10. (Phase 3) `refactor(app_controller): migrate FileItem access sites` (Tier 3)
|
||||
11. (Phase 3) `refactor(gui_2): migrate FileItem access sites` (Tier 3)
|
||||
12. (Phase 3) [docs] `audit: re-measure after Phase 3` (Tier 2)
|
||||
13. (Phase 4) `refactor(mcp_client): migrate ToolDefinition + ToolCall access sites` (Tier 3)
|
||||
14. (Phase 4) `refactor(ai_client): migrate ToolDefinition + ToolCall access sites (tool loop section)` (Tier 3)
|
||||
15. (Phase 4) [docs] `audit: re-measure after Phase 4` (Tier 2)
|
||||
16. (Phase 5) N commits, 1 per file (varies)
|
||||
17. (Phase 6) `conductor(state): metadata_promotion_20260624 SHIPPED` (Tier 2)
|
||||
18. (Phase 6) `docs(reports): TRACK_COMPLETION_metadata_promotion_20260624` (Tier 2)
|
||||
19. (Phase 6) `conductor(tracks): add metadata_promotion_20260624 row` (Tier 2)
|
||||
1. (Phase 0) `refactor(type_aliases): add per-aggregate dataclasses (CommsLogEntry, HistoryMessage, ToolDefinition, ...)`
|
||||
2. (Phase 0) `feat(rag_engine): add RAGChunk dataclass`
|
||||
3. (Phase 0) `refactor(models): complete ContextPreset schema with missing fields`
|
||||
4. (Phase 0) `test(type_aliases): add per-aggregate dataclass regression-guard suite`
|
||||
5. (Phase 0) `docs(styleguides): clarify when to promote to per-aggregate dataclass`
|
||||
6. (Phase 1) `refactor(gui_2): migrate Ticket access sites to direct field access`
|
||||
7. (Phase 1) `refactor(app_controller,conductor_tech_lead): migrate Ticket access sites`
|
||||
8. (Phase 1) `refactor(models): remove legacy Ticket.get() method`
|
||||
9. (Phase 2) `refactor(aggregate): migrate FileItem access sites`
|
||||
10. (Phase 2) `refactor(ai_client,app_controller): migrate FileItem access sites`
|
||||
11. (Phase 3) `refactor(session_logger): migrate CommsLogEntry access sites`
|
||||
12. (Phase 3) `refactor(multi_agent_conductor): migrate CommsLogEntry access sites`
|
||||
13. (Phase 3) `refactor(app_controller): migrate CommsLogEntry access sites`
|
||||
14. (Phase 4) `refactor(gui_2): migrate HistoryMessage access sites`
|
||||
15. (Phase 5) `refactor(ai_client): migrate ChatMessage access sites in _send_anthropic/_send_deepseek`
|
||||
16. (Phase 5) `refactor(ai_client): migrate ChatMessage access sites in _send_grok/_send_qwen`
|
||||
17. (Phase 5) `refactor(ai_client): migrate ChatMessage access sites in _send_minimax/_send_llama`
|
||||
18. (Phase 6) `refactor(app_controller): migrate UsageStats access sites`
|
||||
19. (Phase 7) `refactor(ai_client): migrate ToolCall access sites in tool loop section`
|
||||
20. (Phase 7) `refactor(mcp_client): migrate ToolCall access sites in tool loop section`
|
||||
21. (Phase 8) `refactor(mcp_client): migrate ToolDefinition access sites`
|
||||
22. (Phase 8) `refactor(ai_client): migrate ToolDefinition access sites`
|
||||
23. (Phase 9) `refactor(rag_engine,aggregate,app_controller): migrate RAGChunk access sites`
|
||||
24. (Phase 10) `refactor(gui_2): migrate SessionInsights, DiscussionSettings, CustomSlice, MMAUsageStats`
|
||||
25. (Phase 10) `refactor(app_controller): migrate ProviderPayload, UIPanelConfig, PathInfo`
|
||||
26. (Phase 11) `docs(audit): classify remaining .get() sites as promoted or collapsed-codepath`
|
||||
27. (Phase 12) `conductor(state): metadata_promotion_20260624 SHIPPED`
|
||||
28. (Phase 12) `docs(reports): TRACK_COMPLETION_metadata_promotion_20260624`
|
||||
29. (Phase 12) `conductor(tracks): update metadata_promotion_20260624 row`
|
||||
|
||||
Plus per-task plan-update commits per the workflow.
|
||||
|
||||
## Verification Commands (run at end of each phase + Phase 6)
|
||||
## Verification Commands (run at end of each phase + Phase 12)
|
||||
|
||||
```bash
|
||||
# VC1: Metadata is a @dataclass(frozen=True, slots=True)
|
||||
git show HEAD:src/type_aliases.py | head -20
|
||||
# Expect: @dataclass(frozen=True, slots=True) class Metadata:
|
||||
# VC1: Metadata is unchanged
|
||||
git grep "^Metadata:" src/type_aliases.py
|
||||
# Expect: Metadata: TypeAlias = dict[str, Any]
|
||||
|
||||
# VC2: 107 .get('key', ...) sites replaced
|
||||
# VC2: Each new sub-aggregate is its OWN @dataclass(frozen=True, slots=True)
|
||||
git grep -A 1 "^class CommsLogEntry\|^class HistoryMessage\|^class ToolDefinition\|^class RAGChunk\|^class SessionInsights\|^class DiscussionSettings\|^class CustomSlice\|^class MMAUsageStats\|^class ProviderPayload\|^class UIPanelConfig\|^class PathInfo" src/
|
||||
# Expect: each followed by @dataclass(frozen=True, slots=True)
|
||||
|
||||
# VC3: Existing dataclasses reused
|
||||
git grep "class Ticket\|class FileItem\|class ToolCall\|class ChatMessage\|class UsageStats" src/
|
||||
# Expect: existing classes unchanged
|
||||
|
||||
# VC4: 107 .get('key', ...) sites on known aggregates replaced
|
||||
git grep -E "\.get\('[a-z_]+'," HEAD -- 'src/*.py' | wc -l
|
||||
# Expect: <20 (only legitimate non-Metadata uses)
|
||||
# Expect: only collapsed-codepath sites (FR2; documented in Phase 11 commit)
|
||||
|
||||
# VC3: 106 ['key'] subscript sites replaced
|
||||
# VC5: 106 ['key'] subscript sites on known aggregates replaced
|
||||
git grep -E "\[[ ]*'[a-z_]+'[ ]*\]" HEAD -- 'src/*.py' | wc -l
|
||||
# Expect: <20 (only legitimate non-Metadata uses)
|
||||
# Expect: only legitimate non-aggregate uses
|
||||
|
||||
# VC4: 12+ tests pass
|
||||
uv run python -m pytest tests/test_metadata_dataclass.py -v
|
||||
# Expect: 12/12 pass
|
||||
# VC6: 60+ tests pass (5+ per new dataclass, 12 dataclasses)
|
||||
uv run pytest tests/test_comms_log_entry.py tests/test_history_message.py tests/test_tool_definition.py tests/test_rag_chunk.py tests/test_session_insights.py tests/test_discussion_settings.py tests/test_custom_slice.py tests/test_mma_usage_stats.py tests/test_provider_payload.py tests/test_ui_panel_config.py tests/test_path_info.py tests/test_context_preset_schema.py -v
|
||||
# Expect: all pass
|
||||
|
||||
# VC5: 5 sub-aggregate TypeAliases point to Metadata
|
||||
git grep "TypeAlias = " HEAD:src/type_aliases.py
|
||||
# Expect: CommsLogEntry: TypeAlias = Metadata, etc.
|
||||
|
||||
# VC6: Effective codepaths drops by >= 2 orders of magnitude
|
||||
# VC7: Effective codepaths drops by >= 2 orders of magnitude
|
||||
uv run python -c "
|
||||
import sys
|
||||
sys.path.insert(0, 'scripts/code_path_audit')
|
||||
@@ -183,7 +311,7 @@ print(f'Effective codepaths: {total:.3e} (baseline: 4.014e+22)')
|
||||
"
|
||||
# Expect: < 1e+20
|
||||
|
||||
# VC7: 7 audit gates pass
|
||||
# VC8: 7 audit gates pass
|
||||
uv run python scripts/audit_weak_types.py --strict
|
||||
uv run python scripts/generate_type_registry.py --check
|
||||
uv run python scripts/audit_main_thread_imports.py
|
||||
@@ -193,7 +321,7 @@ uv run python scripts/audit_exception_handling.py --strict
|
||||
uv run python scripts/audit_optional_in_3_files.py --strict
|
||||
# All exit 0
|
||||
|
||||
# VC8: 10/11 batched tiers
|
||||
# VC9: 10/11 batched tiers
|
||||
uv run python scripts/run_tests_batched.py
|
||||
# Expect: 10/11 PASS
|
||||
```
|
||||
@@ -201,16 +329,18 @@ uv run python scripts/run_tests_batched.py
|
||||
## Notes for Tier 3 workers
|
||||
|
||||
- **Pattern consistency**: For each access site, the canonical pattern is `entry.field_name or default_value` for nullable fields, `entry.field_name` for required fields.
|
||||
- **Per-aggregate dataclass reference**: `src/openai_schemas.py` (the canonical pattern for `ToolCall`, `ChatMessage`, `UsageStats`, `ToolCallFunction`, `NormalizedResponse`); `src/models.py:533` (`FileItem` with `to_dict()` / `from_dict()` round-trip).
|
||||
- **Dynamic keys** (e.g., `entry[variable_name]` where the key is not a static string): keep as `entry.to_dict()[variable_name]` for those rare cases. The dataclass handles the common case.
|
||||
- **Polymorphic construction** (e.g., `entry = {'role': 'user', 'content': 'hi'}`): replace with `entry = Metadata(role='user', content='hi')`. If the dict is dynamic, use `entry = Metadata.from_dict(raw_dict)`.
|
||||
- **Polymorphic construction** (e.g., `entry = {'role': 'user', 'content': 'hi'}`): replace with `entry = HistoryMessage(role='user', content='hi')`. If the dict is dynamic, use `entry = HistoryMessage.from_dict(raw_dict)`.
|
||||
- **JSON serialization**: `json.dumps(entry.to_dict())` (not `json.dumps(entry)` which would fail on dataclass).
|
||||
- **Indentation**: 1-space per level.
|
||||
- **No comments** in source code (per AGENTS.md).
|
||||
- **Per-phase regression-guard test runs**: after each phase, run the full batched test suite. If a phase causes a regression, REVERT the phase commit and investigate (don't try to fix forward).
|
||||
- **Per-phase regression-guard test runs**: after each phase, run the per-aggregate test files + the full batched test suite. If a phase causes a regression, REVERT the phase commit and investigate (don't try to fix forward).
|
||||
|
||||
## Notes for Tier 2 reviewer
|
||||
|
||||
- The dataclass is the central artifact. After Phase 0, `Metadata()` constructor works. Each subsequent phase migrates consumers in a specific file.
|
||||
- The 4.01e22 metric drops per phase. Document the drop in `docs/reports/metadata_promotion_progress.md` (new file).
|
||||
- The per-aggregate dataclasses are the central artifacts. After Phase 0, every new dataclass is importable. Each subsequent phase migrates the consumers in a specific file.
|
||||
- The 4.01e22 metric drops per phase. Document the drop in the TRACK_COMPLETION report.
|
||||
- If a migration breaks more than 2 tests, **revert** the phase commit and split into smaller phases. Don't accumulate broken state.
|
||||
- The RAG test pre-existing flake is acceptable. Document it but don't try to fix.
|
||||
- The classification in Phase 11 (collapsed-codepath vs promoted) is auditable; every remaining `.get()` site must have a justification in the commit message.
|
||||
Reference in New Issue
Block a user