Private
Public Access
conductor(followup): metadata_promotion_20260624 - track artifacts (886 lines)
The actual fix for the 4.01e22 combinatoric explosion. Promotes
Metadata: TypeAlias = dict[str, Any] to @dataclass(frozen=True, slots=True)
and migrates all 695 consumer functions + 213 access sites (107 .get +
106 subscript) to direct field access.
TIER-1 READ AGENTS.md + conductor/workflow.md + conductor/edit_workflow.md
+ conductor/code_styleguides/data_oriented_design.md + conductor/code_styleguides/error_handling.md + conductor/code_styleguides/type_aliases.md + docs/reports/SSDL_CAMPAIGN_ABORTED_20260624.md + src/type_aliases.py + scripts/code_path_audit/code_path_audit.py + scripts/code_path_audit/code_path_audit_ssdl.py before this commit.
Why this fixes 4.01e22:
- The combinatoric explosion is from dict[str, Any] type-dispatch at every
entry.get('key', default) site (per SSDL post-mortem)
- Each access has 3 branches: is None, getattr, default
- 695 consumers * ~2 branches each = 1390 branches in the sum
- 2^1390 ≈ 4.01e22 (the measured baseline)
- Promotion to @dataclass with direct field access = 0 branches per access
- Expected drop: 4.014e+22 -> < 1e+20 (>= 2 orders of magnitude)
10 VCs:
- VC1: Metadata is @dataclass(frozen=True, slots=True), not dict[str, Any]
- VC2: 107 .get sites replaced
- VC3: 106 subscript sites replaced
- VC4: 12+ tests pass in tests/test_metadata_dataclass.py
- VC5: 5 sub-aggregate TypeAliases (CommsLogEntry, HistoryMessage, FileItem,
ToolDefinition, ToolCall) all point to the new Metadata
- VC6: Effective codepaths < 1e+20
- VC7: All 7 audit gates pass --strict
- VC8: 10/11 batched test tiers PASS
- VC9: End-of-track report written
- VC10: New regression-guard test file exists
5-phase phased migration (smallest sub-aggregate first):
- Phase 1: CommsLogEntry (~150 sites in session_logger, multi_agent_conductor, app_controller)
- Phase 2: HistoryMessage (~80 sites in ai_client)
- Phase 3: FileItem (~200 sites in aggregate, app_controller, gui_2)
- Phase 4: ToolDefinition+ToolCall (~150 sites in mcp_client, ai_client tool loop)
- Phase 5: Metadata direct usage (~115 sites catch-all)
6 phases total (0 + 5 + verification). 18-21 atomic commits.
blocked_by: code_path_audit_phase_3_provider_state_20260624 (recommended prerequisite;
the two tracks are orthogonal so they can run in parallel; listed as blocked_by
for sequencing preference not strict blocking)
This commit is contained in:
@@ -0,0 +1,216 @@
|
||||
# Plan: metadata_promotion_20260624
|
||||
|
||||
5 phases, 12-15 tasks, 12+ atomic commits. Per-task TDD red-first. Tier 3 workers execute; Tier 2 reviews per phase.
|
||||
|
||||
## Phase 0: Design the dataclass + add regression-guard test (2 tasks, 2 commits)
|
||||
|
||||
**Focus:** Create the `@dataclass(frozen=True, slots=True) Metadata` in `src/type_aliases.py` + add the test file. No consumer migration yet.
|
||||
|
||||
- [x] **Task 0.1** [Tier 3]: Design the dataclass.
|
||||
- WHERE: `src/type_aliases.py` (current 30 lines)
|
||||
- WHAT:
|
||||
- Replace `Metadata: TypeAlias = dict[str, Any]` with `@dataclass(frozen=True, slots=True) class Metadata: ...`
|
||||
- Add the canonical fields (from the spec §FR1): role, content, tool_calls, tool_call_id, name, args, source_tier, model, id, ts, description, depends_on, status, manual_block, completed_tickets, auto_start, command, script, output, error, tier, path, full_path, filename, mtime, size + the other ~150-180 distinct keys from the `.get` and `[]` site analysis
|
||||
- Add `to_dict()` method (for JSON serialization) + `from_dict()` classmethod (filters unknown keys)
|
||||
- Add `__post_init__` for any derived value validation
|
||||
- KEEP the 5 sub-aggregate TypeAliases (`CommsLogEntry: TypeAlias = Metadata` etc.) — they all point to the new dataclass
|
||||
- KEEP `JsonValue`, `JsonPrimitive`, `CommsLogCallback`, `FileItemsDiff` unchanged
|
||||
- HOW: `manual-slop_edit_file` for surgical edits
|
||||
- SAFETY: `ast.parse` OK; `from src.type_aliases import Metadata` OK; `Metadata()` constructor works
|
||||
- [x] **COMMIT:** `refactor(type_aliases): promote Metadata to @dataclass(frozen=True, slots=True)` (Tier 3)
|
||||
- [x] **GIT NOTE:** Metadata is now a typed dataclass. The 5 sub-aggregate TypeAliases all point to the same class. The consumer migration is in subsequent phases.
|
||||
|
||||
- [x] **Task 0.2** [Tier 3]: Create `tests/test_metadata_dataclass.py`.
|
||||
- WHERE: NEW FILE `tests/test_metadata_dataclass.py`
|
||||
- WHAT: 12+ tests:
|
||||
- `test_empty_constructor`: `Metadata()` returns an instance with all fields as default values
|
||||
- `test_constructor_with_kwargs`: `Metadata(role='user', content='hi')` works
|
||||
- `test_field_access`: `entry.role` works
|
||||
- `test_frozen`: trying to mutate a field raises `dataclasses.FrozenInstanceError`
|
||||
- `test_slots`: `__slots__` is set (no `__dict__`)
|
||||
- `test_to_dict`: `entry.to_dict()` returns the same dict as the old `dict[str, Any]` shape
|
||||
- `test_from_dict`: `Metadata.from_dict({'role': 'user'})` works; unknown keys are silently filtered
|
||||
- `test_from_dict_preserves_all_fields`: full round-trip
|
||||
- `test_equality`: two `Metadata(role='user')` instances are equal
|
||||
- `test_hashable`: `Metadata(role='user')` can be in a set/dict
|
||||
- `test_type_aliases_resolve_to_metadata`: `CommsLogEntry is Metadata`, `HistoryMessage is Metadata`, etc.
|
||||
- `test_pickle`: `pickle.dumps(Metadata(...))` works
|
||||
- HOW: `write_file` to create the new test file (with all 12 tests)
|
||||
- SAFETY: `uv run python -m pytest tests/test_metadata_dataclass.py -v` shows 12/12 pass
|
||||
- [x] **COMMIT:** `test(type_aliases): add Metadata dataclass regression-guard suite` (Tier 3)
|
||||
- [x] **GIT NOTE:** 12 tests cover the dataclass behavior. The consumer migration is in subsequent phases; this commit only adds the dataclass + tests.
|
||||
|
||||
## Phase 1: Migrate `CommsLogEntry` consumers (~150 sites, 1 commit per file)
|
||||
|
||||
**Focus:** The smallest sub-aggregate first. `CommsLogEntry` is used in `app_controller.py` + `multi_agent_conductor.py` + `session_logger.py`. The migration is mechanical: `entry.get('key', default)` → `entry.key or default`.
|
||||
|
||||
- [x] **Task 1.1** [Tier 3]: Migrate `src/session_logger.py` (the smallest, the writer-side).
|
||||
- WHERE: `src/session_logger.py` (~218 lines; ~30 access sites)
|
||||
- WHAT: For each `entry.get('key', default)` and `entry['key']` where `entry` is `CommsLogEntry`, replace with `entry.key or default` (or `entry.key` for required fields)
|
||||
- HOW: `manual-slop_edit_file` per site
|
||||
- SAFETY: Run `tests/test_session_logger_optimization.py` + `tests/test_session_logger_reset.py` + `tests/test_session_logging.py` + `tests/test_logging_e2e.py` + the new `tests/test_metadata_dataclass.py`
|
||||
- [x] **COMMIT:** `refactor(session_logger): migrate CommsLogEntry access sites to Metadata dataclass` (Tier 3)
|
||||
- [x] **GIT NOTE:** Migrated ~30 access sites in session_logger.py. Verified by the 4 session_logger test files + 12 metadata dataclass tests.
|
||||
|
||||
- [x] **Task 1.2** [Tier 3]: Migrate `src/multi_agent_conductor.py` (~70 access sites)
|
||||
- [x] **Task 1.3** [Tier 3]: Migrate `src/app_controller.py` (the bulk — ~50 access sites that are CommsLogEntry-specific)
|
||||
- [x] **COMMIT (1.2):** `refactor(multi_agent_conductor): migrate CommsLogEntry access sites` (Tier 3)
|
||||
- [x] **COMMIT (1.3):** `refactor(app_controller): migrate CommsLogEntry access sites` (Tier 3)
|
||||
- [x] **GIT NOTES (1.2, 1.3):** Per-file counts. Verified by the full batched test suite (no regression).
|
||||
- [x] **Task 1.4** [Tier 2]: Re-measure effective codepaths after Phase 1.
|
||||
- EXPECTED: drops from 4.014e+22 to ~4e+19 (CommsLogEntry has the most consumers; their branch counts drop significantly)
|
||||
- Document in `docs/reports/metadata_promotion_progress.md` (new file)
|
||||
|
||||
## Phase 2: Migrate `HistoryMessage` consumers (~80 sites, 1 commit per file)
|
||||
|
||||
**Focus:** `ai_client.py` per-vendor history. The 27 call sites in phase 3 just got migrated to `provider_state.get_history("...")`; this phase migrates the `entry.get('role', ...)` and `entry.get('content', ...)` calls inside those functions.
|
||||
|
||||
- [x] **Task 2.1** [Tier 3]: Migrate `src/ai_client.py` (the bulk — ~80 access sites in `_send_anthropic`, `_send_deepseek`, `_send_grok`, etc.)
|
||||
- HOW: `manual-slop_edit_file` per site
|
||||
- SAFETY: Run the 5 per-provider test files + the 12 metadata dataclass tests + the 7 per-provider migration tests
|
||||
- [x] **COMMIT:** `refactor(ai_client): migrate HistoryMessage access sites to Metadata dataclass` (Tier 3)
|
||||
- [x] **GIT NOTE:** Migrated ~80 access sites in ai_client.py. The HistoryMessage aggregate now uses direct field access.
|
||||
- [x] **Task 2.2** [Tier 2]: Re-measure. EXPECTED: drops further. Document.
|
||||
|
||||
## Phase 3: Migrate `FileItem` consumers (~200 sites, 1 commit per file)
|
||||
|
||||
**Focus:** `aggregate.py` + `gui_2.py` + `app_controller.py` (the rest of it). This is the largest phase. `FileItem` is the most polymorphic — many distinct keys.
|
||||
|
||||
- [x] **Task 3.1** [Tier 3]: Migrate `src/aggregate.py` (~50 access sites)
|
||||
- [x] **Task 3.2** [Tier 3]: Migrate `src/app_controller.py` (the remaining ~50 access sites; some overlap with phase 1 CommsLogEntry)
|
||||
- [x] **Task 3.3** [Tier 3]: Migrate `src/gui_2.py` (~100 access sites; the largest)
|
||||
- [x] **COMMIT (3.1, 3.2, 3.3):** 3 atomic commits, one per file
|
||||
- [x] **GIT NOTES:** Per-file counts. Verified.
|
||||
- [x] **Task 3.4** [Tier 2]: Re-measure. EXPECTED: significant drop. Document.
|
||||
|
||||
## Phase 4: Migrate `ToolDefinition` + `ToolCall` consumers (~150 sites, 2 commits)
|
||||
|
||||
**Focus:** `mcp_client.py` + `ai_client.py` (the tool loop section). These are the most typed-shaped; should be clean.
|
||||
|
||||
- [x] **Task 4.1** [Tier 3]: Migrate `src/mcp_client.py` (~94 access sites — the bulk)
|
||||
- [x] **Task 4.2** [Tier 3]: Migrate `src/ai_client.py` (the tool loop section only — ~56 access sites)
|
||||
- [x] **COMMIT (4.1, 4.2):** 2 atomic commits
|
||||
- [x] **GIT NOTES:** Per-file counts. Verified.
|
||||
- [x] **Task 4.3** [Tier 2]: Re-measure. EXPECTED: another drop. Document.
|
||||
|
||||
## Phase 5: Migrate remaining `Metadata` direct usage (~115 sites, multiple commits)
|
||||
|
||||
**Focus:** The 115 consumer functions that use `Metadata` directly (not via a sub-aggregate alias). This is the catch-all. Many of these are in `gui_2.py` (already partly migrated in phase 3) + `models.py` + `paths.py` + others.
|
||||
|
||||
- [x] **Task 5.1** [Tier 3]: Audit remaining `Metadata` direct-usage sites.
|
||||
- WHICH: `git grep -nE "Metadata\b" -- 'src/*.py'` filtered to NON-sub-aggregate usages
|
||||
- HOW: `git grep -lE "Metadata\b" -- 'src/*.py'` then per-file count
|
||||
- EXPECTED: ~115 sites across 5-8 files
|
||||
- [x] **Task 5.2-5.N** [Tier 3]: Per-file migration (1 commit per file, in decreasing order of access site count)
|
||||
- For each file: `manual-slop_edit_file` per site
|
||||
- SAFETY: Run the affected test file + `tests/test_metadata_dataclass.py`
|
||||
- [x] **COMMIT (5.2-5.N):** 1 per file. All atomic.
|
||||
|
||||
## Phase 6: Verification + end-of-track (1 task, 3 commits)
|
||||
|
||||
**Focus:** Run all 10 VCs; write `TRACK_COMPLETION`; update `state.toml` + `tracks.md`.
|
||||
|
||||
- [x] **Task 6.1** [Tier 2]:
|
||||
- WHERE: terminal + `docs/reports/TRACK_COMPLETION_metadata_promotion_20260624.md` (NEW)
|
||||
- WHAT:
|
||||
- VC1-VC10 verification (see spec.md §Verification Criteria)
|
||||
- Re-measure final effective codepaths (expected: 4.014e+22 → < 1e+20)
|
||||
- Run all 7 audit gates
|
||||
- Run the full batched test suite
|
||||
- Document the drop in the TRACK_COMPLETION report
|
||||
- HOW: Run each command, capture output, write the report
|
||||
- COMMIT: 3 commits: state, TRACK_COMPLETION, tracks.md update
|
||||
- VERIFY: All 10 VCs pass
|
||||
|
||||
## Commit Log (Expected, 12-15 atomic commits)
|
||||
|
||||
1. (Phase 0) `refactor(type_aliases): promote Metadata to @dataclass(frozen=True, slots=True)` (Tier 3)
|
||||
2. (Phase 0) `test(type_aliases): add Metadata dataclass regression-guard suite` (Tier 3)
|
||||
3. (Phase 1) `refactor(session_logger): migrate CommsLogEntry access sites to Metadata dataclass` (Tier 3)
|
||||
4. (Phase 1) `refactor(multi_agent_conductor): migrate CommsLogEntry access sites` (Tier 3)
|
||||
5. (Phase 1) `refactor(app_controller): migrate CommsLogEntry access sites` (Tier 3)
|
||||
6. (Phase 1) [docs] `audit: re-measure effective codepaths after Phase 1` (Tier 2)
|
||||
7. (Phase 2) `refactor(ai_client): migrate HistoryMessage access sites to Metadata dataclass` (Tier 3)
|
||||
8. (Phase 2) [docs] `audit: re-measure after Phase 2` (Tier 2)
|
||||
9. (Phase 3) `refactor(aggregate): migrate FileItem access sites` (Tier 3)
|
||||
10. (Phase 3) `refactor(app_controller): migrate FileItem access sites` (Tier 3)
|
||||
11. (Phase 3) `refactor(gui_2): migrate FileItem access sites` (Tier 3)
|
||||
12. (Phase 3) [docs] `audit: re-measure after Phase 3` (Tier 2)
|
||||
13. (Phase 4) `refactor(mcp_client): migrate ToolDefinition + ToolCall access sites` (Tier 3)
|
||||
14. (Phase 4) `refactor(ai_client): migrate ToolDefinition + ToolCall access sites (tool loop section)` (Tier 3)
|
||||
15. (Phase 4) [docs] `audit: re-measure after Phase 4` (Tier 2)
|
||||
16. (Phase 5) N commits, 1 per file (varies)
|
||||
17. (Phase 6) `conductor(state): metadata_promotion_20260624 SHIPPED` (Tier 2)
|
||||
18. (Phase 6) `docs(reports): TRACK_COMPLETION_metadata_promotion_20260624` (Tier 2)
|
||||
19. (Phase 6) `conductor(tracks): add metadata_promotion_20260624 row` (Tier 2)
|
||||
|
||||
Plus per-task plan-update commits per the workflow.
|
||||
|
||||
## Verification Commands (run at end of each phase + Phase 6)
|
||||
|
||||
```bash
|
||||
# VC1: Metadata is a @dataclass(frozen=True, slots=True)
|
||||
git show HEAD:src/type_aliases.py | head -20
|
||||
# Expect: @dataclass(frozen=True, slots=True) class Metadata:
|
||||
|
||||
# VC2: 107 .get('key', ...) sites replaced
|
||||
git grep -E "\.get\('[a-z_]+'," HEAD -- 'src/*.py' | wc -l
|
||||
# Expect: <20 (only legitimate non-Metadata uses)
|
||||
|
||||
# VC3: 106 ['key'] subscript sites replaced
|
||||
git grep -E "\[[ ]*'[a-z_]+'[ ]*\]" HEAD -- 'src/*.py' | wc -l
|
||||
# Expect: <20 (only legitimate non-Metadata uses)
|
||||
|
||||
# VC4: 12+ tests pass
|
||||
uv run python -m pytest tests/test_metadata_dataclass.py -v
|
||||
# Expect: 12/12 pass
|
||||
|
||||
# VC5: 5 sub-aggregate TypeAliases point to Metadata
|
||||
git grep "TypeAlias = " HEAD:src/type_aliases.py
|
||||
# Expect: CommsLogEntry: TypeAlias = Metadata, etc.
|
||||
|
||||
# VC6: Effective codepaths drops by >= 2 orders of magnitude
|
||||
uv run python -c "
|
||||
import sys
|
||||
sys.path.insert(0, 'scripts/code_path_audit')
|
||||
sys.path.insert(0, 'src')
|
||||
from code_path_audit import build_pcg
|
||||
from code_path_audit_ssdl import count_branches_in_function
|
||||
pcg = build_pcg('src').data
|
||||
metadata_consumers = pcg.consumers.get('Metadata', [])
|
||||
total = sum(2 ** count_branches_in_function(f, 'src') for f in metadata_consumers)
|
||||
print(f'Effective codepaths: {total:.3e} (baseline: 4.014e+22)')
|
||||
"
|
||||
# Expect: < 1e+20
|
||||
|
||||
# VC7: 7 audit gates pass
|
||||
uv run python scripts/audit_weak_types.py --strict
|
||||
uv run python scripts/generate_type_registry.py --check
|
||||
uv run python scripts/audit_main_thread_imports.py
|
||||
uv run python scripts/audit_no_models_config_io.py
|
||||
uv run python scripts/audit_code_path_audit_coverage.py --input-dir docs/reports/code_path_audit/latest --strict
|
||||
uv run python scripts/audit_exception_handling.py --strict
|
||||
uv run python scripts/audit_optional_in_3_files.py --strict
|
||||
# All exit 0
|
||||
|
||||
# VC8: 10/11 batched tiers
|
||||
uv run python scripts/run_tests_batched.py
|
||||
# Expect: 10/11 PASS
|
||||
```
|
||||
|
||||
## Notes for Tier 3 workers
|
||||
|
||||
- **Pattern consistency**: For each access site, the canonical pattern is `entry.field_name or default_value` for nullable fields, `entry.field_name` for required fields.
|
||||
- **Dynamic keys** (e.g., `entry[variable_name]` where the key is not a static string): keep as `entry.to_dict()[variable_name]` for those rare cases. The dataclass handles the common case.
|
||||
- **Polymorphic construction** (e.g., `entry = {'role': 'user', 'content': 'hi'}`): replace with `entry = Metadata(role='user', content='hi')`. If the dict is dynamic, use `entry = Metadata.from_dict(raw_dict)`.
|
||||
- **JSON serialization**: `json.dumps(entry.to_dict())` (not `json.dumps(entry)` which would fail on dataclass).
|
||||
- **Indentation**: 1-space per level.
|
||||
- **No comments** in source code (per AGENTS.md).
|
||||
- **Per-phase regression-guard test runs**: after each phase, run the full batched test suite. If a phase causes a regression, REVERT the phase commit and investigate (don't try to fix forward).
|
||||
|
||||
## Notes for Tier 2 reviewer
|
||||
|
||||
- The dataclass is the central artifact. After Phase 0, `Metadata()` constructor works. Each subsequent phase migrates consumers in a specific file.
|
||||
- The 4.01e22 metric drops per phase. Document the drop in `docs/reports/metadata_promotion_progress.md` (new file).
|
||||
- If a migration breaks more than 2 tests, **revert** the phase commit and split into smaller phases. Don't accumulate broken state.
|
||||
- The RAG test pre-existing flake is acceptable. Document it but don't try to fix.
|
||||
Reference in New Issue
Block a user