ed/manual_slop
Private
Public Access
0
0
Fork 0
Code Activity

archive: completed code path audits

Browse Source
This commit is contained in:
ed
2026-07-05 12:48:36 -04:00
parent e28a2ae215
commit ae690e9a59
16 changed files with 0 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
conductor/tracks/code_path_audit_20260607/TIER2_STARTUP.md
-263
View File
@@ -1,263 +0,0 @@
# Tier 2 Startup — code_path_audit_20260607 v2
> **For Tier 2 Tech Lead (autonomous mode).** This is the entry point. Read this file first, then `plan_v2.md`, then `spec_v2.md`. The v1 files (`spec.md` + `plan.md`) are **preserved unchanged and never executed** — do not load them as the canonical spec.
## What this track is
Build `src/code_path_audit.py` v2 — a data-oriented static-analysis tool that audits the 13 data aggregates in `src/` (10 in-scope TypeAliases + 3 candidate placeholders for `any_type_componentization_20260621` which is NOT on master) and produces per-aggregate profiles. The output (custom postfix `.dsl` + markdown + prefix tree text) is the artifact that informs per-aggregate refactor decisions.
**Why v2 supersedes v1:** v1 was authored 2026-06-07 before the 4 foundational tracks shipped. v1's "per-action" framing is now stale. v2 reframes the audit to "per-data-aggregate" + a 4-direction decomposition-cost heuristic (componentize / unify / hold / insufficient_data) per aggregate. v2 also cross-validates the 2 foundational conventions (`data_structure_strengthening_20260606` + `data_oriented_error_handling_20260606`) directly.
**The user's framing (2026-06-22):**
> "The whole point of the code path audit is to audit all paths nearly in the ./src of the codebase. The main point of it is to identify data-oriented pipelines and what data aggregate they will be operating on. This will realize what the data strengthening just uncovered and cross-audit if its deductions on the data structures are accurate while also being able to utilize additional flexibility the data oriented error handling track has provided. We are entering a time where the codebase is getting heavily adjusted into a properly engineered machine with discernable working parts. The cost of the pipeline is important, it should factor in what data needs to be componentized further vs which can be unified further into wider code paths handling larger fat structs."
## What to load
In this order:
1. **This file** (`TIER2_STARTUP.md`) — startup context.
2. **`plan_v2.md`** — the executable plan. 14 phases, 85+ tasks, 91 tests. **This is the source of truth for execution.**
3. **`spec_v2.md`** — the design intent. Read this when the plan is ambiguous.
4. **DO NOT load `spec.md` or `plan.md`** — those are the v1 files (preserved, never executed). The plan_v2.md supersedes plan.md.
## What's on master (verified `7e61dd7d` + commits `7ea414e9` + `85baea8c`)
- `src/type_aliases.py` — the 10 canonical TypeAliases + 1 NamedTuple (`FileItemsDiff`).
- `src/result_types.py` — `Result[T]`, `ErrorInfo`, `ErrorKind`, `NilPath`, `NilRAGState`, `OK`.
- `src/mcp_client.py:934-992` — `derive_code_path(target, max_depth=5)` (the v1 primitive; v2's PCG is the multi-symbol superset).
- `src/performance_monitor.py` — runtime profiling (used by the `pipeline_runtime_profiling_20260607` follow-up, NOT by this track).
- `scripts/audit_main_thread_imports.py` — import-graph CI gate.
- `scripts/audit_weak_types.py` — weak-types CI gate.
- `scripts/audit_exception_handling.py` — exception-handling CI gate.
- `scripts/audit_no_models_config_io.py` — config-I/O ownership CI gate.
- `scripts/audit_optional_in_3_files.py` — `Optional[T]` ban CI gate (the 3 baseline files; v2 extends this with +1 line in Phase 12).
- `scripts/generate_type_registry.py` — type-registry generator.
- `conductor/code_styleguides/data_oriented_design.md` — the canonical DOD reference.
- `conductor/code_styleguides/error_handling.md` — the `Result[T]` convention.
- `conductor/code_styleguides/type_aliases.md` — the 10 TypeAliases.
- `conductor/code_styleguides/agent_memory_dimensions.md` — the 4 mem dims.
**NOT on master (and the v2 audit must tolerate their absence for an interim run):**
- `any_type_componentization_20260621` — merged `f914b2bc`, reverted `751b94d4` (9 minutes later). The 3 candidate aggregates (`ToolSpec`, `ChatMessage`, `ProviderHistory`) are forward-compat placeholders with `is_candidate: True`.
- `phase2_4_5_call_site_completion_20260621` — same merge+revert history. The `PHASE3_HYPOTHETICAL_PROMOTION.md` report is NOT on master (reverted with the merge).
**3 handoff files are also NOT on master** (reverted with the merge): `HANDOFF_CODE_PATH_AUDIT_FROM_any_type_componentization.md`, `HANDOFF_FOLLOWUP_TRACK_FROM_any_type_componentization.md`, `PROMPT_FOR_TIER_1.md`. The v2 spec/plan do NOT reference these by name; the candidate-aggregate handling is described from first principles.
## Hard Bans (3-layer enforced)
These are restated from `conductor/tier2/agents/tier2-autonomous.md`; they apply on every commit:
- `git push*` (any form) — the user fetches the branch + reviews + merges.
- `git checkout*` (any form) — use `git switch -c` for new branches, `git switch` to switch.
- `git restore*` (any form) — never restore files.
- `git reset*` (any form) — never reset state.
- File access outside `C:\projects\manual_slop_tier2\` (the Tier 2 clone) — the Windows restricted token blocks it.
- **`*AppData\\*`** — AppData is OFF-LIMITS for any read, write, or shell command. Use `tests/artifacts/tier2_state/<track>/` for failcount state, `tests/artifacts/tier2_failures/` for failure reports, `scripts/tier2/artifacts/<track>/` for throwaway scripts.
If a task requires one of these, **STOP and report to the user** — do not bypass.
## Conventions (MUST follow)
- **Test runner:** `uv run python scripts/run_tests_batched.py` (NEVER `uv run pytest` directly; the batched runner provides tier-based filtering, parallelization, and the summary table).
- **Default branch:** `master` (not `main`).
- **Line endings:** preserve existing. This repo has a mix of CRLF and LF. Do not normalize.
- **Throw-away scripts:** `scripts/tier2/artifacts/code_path_audit_20260607/` (NOT the base `scripts/tier2/` dir).
- **End-of-track report:** `docs/reports/TRACK_COMPLETION_code_path_audit_20260607.md` (the file name uses the track_id, not the date; check the precedent set by `TRACK_COMPLETION_live_gui_test_fixes_20260618.md`).
## TDD Protocol (per `conductor/workflow.md`)
1. **Red:** write the failing test (1 commit). Run `uv run python scripts/run_tests_batched.py` and confirm FAIL.
2. **Green:** implement the minimal code to pass (1 commit). Run and confirm PASS.
3. **Refactor:** (optional) 1 commit if there's cleanup.
4. **Commit per task** (1 task = 1 commit). Attach a git note summarizing the task.
5. **Update `plan_v2.md`**: change `[ ]` to `[x] <7-char-sha>` for the completed task. Commit the plan update.
## Per-Task Commit Protocol
After each task:
1. `git add <specific files>` (not `git add .` for individual commits).
2. `git commit -m "<type>(<scope>): <description>"` (e.g., `feat(audit): add the 5 enums`).
3. Get the commit hash: `git log -1 --format="%H"`.
4. Attach git note: `git notes add -m "Task N.M: ..." <hash>`.
5. Update `plan_v2.md`: change `[ ]` to `[x] <7-char-sha>` for the task.
6. Commit the plan update: `git add plan_v2.md && git commit -m "conductor(plan): Mark task N.M complete"`.
## Pre-Delegation Checkpoint
Before each Tier 3 worker delegation, run `git add .` to stage prior work. This is a safety net: if the worker fails or incorrectly runs `git restore`, your prior iterations are not lost.
## Failcount Contract
After every task commit, you MUST check `should_give_up` from `scripts.tier2.failcount`. The state is persisted at `tests/artifacts/tier2_state/code_path_audit_20260607/state.json` (project-relative; resolved via `Path(__file__).parents[2]` in the failcount module). The thresholds are:
- 3 consecutive red-phase failures
- 3 consecutive green-phase failures
- 30 minutes with no progress (no commit, no green test)
If `should_give_up` returns True, IMMEDIATELY stop. Do not attempt another fix. Call `write_failure_report` from `scripts.tier2.write_report` and print the report path. Then **escalate to the user** (do not just write a report and stop silently).
## Track-Specific Guidance
### The 3 candidate aggregates
The 3 candidate aggregates (`ToolSpec`, `ChatMessage`, `ProviderHistory`) are NOT on master. The v2 audit produces **placeholders** with `is_candidate: True` and all metrics set to 0. The `candidates.md` rollup explains the placeholder status. The integration tests verify the placeholder format.
**The v2 spec's `synthesize_aggregate_profile()` Task 9.2 has the placeholder template hard-coded.** When implementing it, use the exact template from the spec — do not invent a different placeholder structure.
### The 4 audit gates
After every commit, run:
```bash
uv run python scripts/audit_exception_handling.py --strict
uv run python scripts/audit_weak_types.py --strict
uv run python scripts/audit_main_thread_imports.py
uv run python scripts/audit_no_models_config_io.py
```
These are the "laws of physics" for `src/code_path_audit.py`. If a gate fails, **fix before continuing**. The most likely failure mode is a Tier 3 worker adding an `Optional[T]` return type (banned in the 3 refactored files + the new file) or a `try/except: pass` (banned per `error_handling.md` Pattern 5).
### The `Result[T]` return type rule
**Every public function in `src/code_path_audit.py` that can fail at runtime returns `Result[T]`.** No `Optional[T]` returns. No `None` returns. No `raise Exception(...)` (only `raise` for programmer errors, e.g., `raise ValueError` in `__init__` for missing config).
The plan marks 6 of the 11 public functions as returning deterministic `T` (no failure mode). The other 5 (1, 2, 7, 9, 10) return `Result[T]`. **Do not add `Result[T]` to the deterministic ones** — it adds noise. **Do not skip `Result[T]` on the fallible ones** — it violates the convention.
### The 11 public functions (per the spec)
| # | Function | Returns | Phase |
|---|---|---|---|
| 1 | `run_audit(...)` | `Result[AuditSummary]` | 9 |
| 2 | `build_pcg(src_dir)` | `Result[ProducerConsumerGraph]` | 2 |
| 3 | `classify_memory_dim(...)` | `MemoryDim` (deterministic) | 3 |
| 4 | `detect_access_pattern(...)` | `AccessPattern` (deterministic) | 4 |
| 5 | `estimate_call_frequency(...)` | `Frequency` (deterministic) | 5 |
| 6 | `compute_decomposition_cost(...)` | `DecompositionCost` (deterministic) | 6 |
| 7 | `read_input_json(path)` | `Result[dict]` | 7 |
| 8 | `to_dsl_v2(profile)` | `str` (deterministic) | 8 |
| 9 | `parse_dsl_v2(text)` | `Result[dict]` | 8 |
| 10 | `to_markdown(profile)` | `str` (deterministic) | 8 |
| 11 | `to_tree(profile)` | `str` (deterministic) | 8 |
Plus the CLI (`if __name__ == "__main__":`) and the MCP tool wrapper (`code_path_audit_v2`).
### The 14 v2 DSL tagged words (per the spec)
`kind`, `mem-dim`, `fn-ref`, `access-pattern`, `ap-evidence`, `frequency`, `freq-evidence`, `result-coverage`, `type-alias-coverage`, `cross-audit-finding`, `cross-audit-findings`, `decomp-cost`, `opt-candidate`, `is-candidate`. The arity table is in `src/code_path_audit.py:DSL_WORD_ARITY_V2` (Phase 8 Task 8.1).
The DSL format is **flat sections** (streamable, tag-scannable) — NOT a nested record. Each `\\ === section_name ===` line is followed by the section's tagged records. This is the v1 design's "no need to parse the whole file" property applied to v2.
### The 5 enums (per the spec)
`AggregateKind` (4 values: typealias, dataclass, candidate_dataclass, builtin), `MemoryDim` (7 values: curation, discussion, rag, knowledge, config, control, unknown), `AccessPattern` (5 values: whole_struct, field_by_field, hot_cold_split, bulk_batched, mixed), `Frequency` (7 values: hot, per_turn, per_discussion, per_request, cold, init, unknown), `RecommendedDirection` (4 values: componentize, unify, hold, insufficient_data).
All enums are `Literal[...]` types (string-valued) for stable postfix DSL output. No `Enum` class — the v1 spec's rationale is "no enum-name lookup table needed in the parser."
### The 9 supporting dataclasses (per the spec)
`FunctionRef`, `AccessPatternEvidence`, `FrequencyEvidence`, `ResultCoverage`, `TypeAliasCoverage`, `CrossAuditFinding`, `CrossAuditFindings`, `DecompositionCost`, `OptimizationCandidate`. Plus the central `AggregateProfile` (14 required fields + 2 default). All `frozen=True` per the immutability story.
### The 4 decomposition directions (per the spec)
- `componentize` — split into smaller dataclasses; access pattern is `field_by_field` with many dead fields, OR `hot_cold_split` with small hot fields.
- `unify` — combine into wider fat structs; access pattern is `bulk_batched` with a small struct, OR `whole_struct` with a small struct.
- `hold` — current shape is correct; default for `frozen + whole_struct` (the ideal shape).
- `insufficient_data` — access pattern is `mixed` or frequency is `unknown`; needs runtime profiling.
The 4-direction logic is in `src/code_path_audit.py:recommended_direction()` (Phase 6 Task 6.6). The savings estimates are heuristic (calibrated by `pipeline_runtime_profiling_20260607`); use as ranking input, not as actual savings.
### The 6 input JSON contracts (per the spec)
The v2 audit consumes JSON from 6 sources in `tests/artifacts/audit_inputs/` (gitignored per `test_sandbox.md`):
| Input | Producer | Path |
|---|---|---|
| 1 | `scripts/audit_weak_types.py --json` | `audit_weak_types.json` |
| 2 | `scripts/audit_exception_handling.py --json` | `audit_exception_handling.json` |
| 3 | `scripts/audit_optional_in_3_files.py --json` | `audit_optional_in_3_files.json` |
| 4 | `scripts/audit_no_models_config_io.py --json` | `audit_no_models_config_io.json` |
| 5 | `scripts/audit_main_thread_imports.py --json` | `audit_main_thread_imports.json` |
| 6 | `scripts/generate_type_registry.py --json` | `type_registry.json` |
**Tolerance:** if any input is missing or malformed, the audit continues with the corresponding `cross_audit_findings` field set to `()` (empty tuple) and the markdown notes the missing input. The audit does NOT fail on missing inputs.
### The integration test fixture
`tests/fixtures/synthetic_src/` defines 3 TypeAliases (Metadata, FileItems, History) + 6 functions (2 producers, 4 consumers). `tests/fixtures/audit_inputs/` has 6 JSON files matching the contracts. The integration tests assert the exact expected profiles per aggregate (the expected output is in the spec's §7.1 + the plan's Phase 10 tasks).
**The fixture names match the canonical TypeAliases** (Metadata, FileItems, History) so the audit's `CANONICAL_MEMORY_DIM` lookup works correctly. Do not rename the fixture's aggregates.
## Known gotchas (from prior tracks' lessons)
These are the "1% chance this happens but you'll waste 4 hours if you don't know" notes:
1. **`Optional[T]` ban extends to the new file.** The `scripts/audit_optional_in_3_files.py` script will be extended in Phase 12 to check `src/code_path_audit.py`. If any Tier 3 worker adds an `Optional[T]` return, the extended audit fails. **Read `conductor/code_styleguides/error_handling.md` before writing the public API.** The 5 MUST-DO rules and 7 MUST-NOT-DO rules apply.
2. **Logging is NOT a drain.** Per `error_handling.md` Pattern A: `sys.stderr.write` / `logging.error` / `print` in an except body is `INTERNAL_SILENT_SWALLOW`, a violation. The CLI / MCP entry points are the drain points. Use `Result[T]` propagation and let the error reach the drain.
3. **The AST walker does NOT execute the code.** The PCG, APD, CFE are pure static analysis. No `eval`, no `exec`, no imports of `src/*` modules that have side effects. The v2 audit reads files; it does not import them.
4. **`scripts/run_tests_batched.py` is the only test runner.** Direct `uv run pytest` may work for a single file but bypasses the tiering that the live_gui tests depend on. The failcount and per-tier filtering only work with the batched runner.
5. **`master` is the default branch.** This repo never had `main`. `git fetch origin master` (NOT `main`).
6. **The CRLF/LF mix is intentional.** Do not normalize. Per-file preservation.
7. **The 3 candidate aggregates are placeholders.** When you run the audit on `master`, the `candidates.md` rollup will show 3 placeholders with `is_candidate: True`. This is correct. The placeholders become real profiles when `any_type_componentization_20260621` is re-merged.
8. **The 1-line extension to `scripts/audit_optional_in_3_files.py` is the audit gate.** If you skip Phase 12 Task 12.2, the new file is not covered by the `Optional[T]` ban, and a future Tier 3 worker could regress the convention. Do the extension.
## Verification Protocol (per `conductor/workflow.md`)
After every task, run the **4 audit gates** in `--strict` mode + the unit tests:
```bash
uv run pytest tests/test_code_path_audit.py -q
uv run python scripts/audit_exception_handling.py --strict
uv run python scripts/audit_weak_types.py --strict
uv run python scripts/audit_main_thread_imports.py
uv run python scripts/audit_no_models_config_io.py
```
At **end-of-track** (Phase 13), add:
```bash
uv run python -m src.code_path_audit --all --date 2026-06-22
uv run python scripts/audit_code_path_audit_coverage.py --input-dir docs/reports/code_path_audit/2026-06-22/ --strict
uv run python scripts/generate_type_registry.py --check
```
## End-of-Track Handoff
When all 14 phases complete, write `docs/reports/TRACK_COMPLETION_code_path_audit_20260607.md` (the user reads this to decide merge). Update `conductor/tracks.md` with the v2 entry. Update `state.toml` to `status = "completed"` and `current_phase = "complete"`.
The TRACK_COMPLETION report should include:
- What shipped (file inventory).
- Verification: 91 tests pass + 4 audit gates + meta-audit + type registry.
- The cross-validation verdict (does the v2 audit's data match the actual state of `data_structure_strengthening` + `data_oriented_error_handling`?).
- The 5 follow-up tracks.
- The 3 candidate aggregates' forward-compat status.
## Out of scope (restated)
- Modifications to existing `src/*.py` files (read-only on the 65 existing files).
- Modifications to the 5 existing audit scripts (consume their JSON; don't change them).
- Runtime profiling (deferred to `pipeline_runtime_profiling_20260607`).
- New pip dependencies (stdlib only).
- Changes to v1 spec.md or plan.md (preserved unchanged).
- MMA worker spawn action (cold per user).
- New src/<thing>.py files (per AGENTS.md file size + naming convention).
- The 23 lower-impact files (deferred).
## See also
- `conductor/tracks/code_path_audit_20260607/spec_v2.md` — the canonical spec (design intent).
- `conductor/tracks/code_path_audit_20260607/plan_v2.md` — the canonical plan (executable).
- `conductor/tracks/code_path_audit_20260607/metadata.json` — the track metadata.
- `conductor/tracks/code_path_audit_20260607/state.toml` — the track state.
- `conductor/code_styleguides/data_oriented_design.md` — the canonical DOD reference.
- `conductor/code_styleguides/error_handling.md` — the `Result[T]` convention.
- `conductor/code_styleguides/type_aliases.md` — the 10 TypeAliases.
- `conductor/code_styleguides/agent_memory_dimensions.md` — the 4 mem dims.
- `conductor/tier2/agents/tier2-autonomous.md` — the Tier 2 agent prompt (this file is the track-specific supplement).
- `conductor/tier2/commands/tier-2-auto-execute.md` — the execute command.
- `docs/reports/RESULT_MIGRATION_CAMPAIGN_STATUS_20260619.md` — the 100%-complete result migration campaign (the v2 audit runs against this final state).
- `docs/reports/ANY_TYPE_AUDIT_20260621.md` — the 89-site audit that informed the 3 candidate aggregates.
- `docs/reports/PHASE3_HYPOTHETICAL_PROMOTION.md` — the cost analysis that informed the `ProviderHistory` candidate (NOT on master; reverted with the merge).
- `conductor/tracks/nagent_review_20260608/nagent_takeaways_v3_1_20260620.md` — the v3.1 nagent review (Candidate 27: Markdown + custom DSL lock-in is the direct application of the v2's custom postfix DSL).
conductor/tracks/code_path_audit_20260607/metadata.json
-200
View File
@@ -1,200 +0,0 @@
{
"id": "code_path_audit_20260607",
"title": "Code Path & Data Pipeline Audit v2",
"type": "tooling",
"status": "active",
"priority": "A",
"created": "2026-06-07",
"last_revised": "2026-06-22",
"owner": "tier2-tech-lead",
"parent_umbrella": null,
"spec": "conductor/tracks/code_path_audit_20260607/spec_v2.md",
"plan": "conductor/tracks/code_path_audit_20260607/plan_v2.md",
"spec_v1_preserved": "conductor/tracks/code_path_audit_20260607/spec.md (v1, never executed; preserved unchanged)",
"plan_v1_preserved": "conductor/tracks/code_path_audit_20260607/plan.md (v1, never executed; preserved unchanged)",
"v2_revision_rationale": "v1 was authored 2026-06-07 before the 4 foundational tracks shipped; v1 framing is now stale. v2 re-scopes the audit from 'expensive operations per action' to 'data pipelines per aggregate' + a decomposition-cost heuristic (componentize vs unify) per aggregate. v2 also cross-validates data_structure_strengthening + data_oriented_error_handling directly (the 2 foundational tracks didn't exist on 2026-06-07).",
"scope": {
"files_created": 17,
"files_created_paths": [
"src/code_path_audit.py",
"tests/test_code_path_audit.py",
"tests/test_code_path_audit_live_gui.py",
"tests/fixtures/synthetic_src/__init__.py",
"tests/fixtures/synthetic_src/type_aliases.py",
"tests/fixtures/synthetic_src/ai_client.py",
"tests/fixtures/synthetic_src/aggregate.py",
"tests/fixtures/synthetic_src/gui_2.py",
"tests/fixtures/synthetic_src/cleanup.py",
"tests/fixtures/synthetic_src/overrides.toml",
"tests/fixtures/audit_inputs/audit_weak_types.json",
"tests/fixtures/audit_inputs/audit_exception_handling.json",
"tests/fixtures/audit_inputs/audit_optional_in_3_files.json",
"tests/fixtures/audit_inputs/audit_no_models_config_io.json",
"tests/fixtures/audit_inputs/audit_main_thread_imports.json",
"tests/fixtures/audit_inputs/type_registry.json",
"scripts/audit_code_path_audit_coverage.py",
"conductor/code_styleguides/code_path_audit.md"
],
"files_modified": 1,
"files_modified_paths": [
"scripts/audit_optional_in_3_files.py (+1 line: add src/code_path_audit.py to the baseline list)"
],
"files_preserved_v1": [
"conductor/tracks/code_path_audit_20260607/spec.md (v1)",
"conductor/tracks/code_path_audit_20260607/plan.md (v1)"
],
"phases": 14,
"tasks": 85,
"tests_total": 91,
"tests_unit": 84,
"tests_integration": 7,
"tests_live_gui_opt_in": 2,
"aggregates_total": 13,
"aggregates_real": 10,
"aggregates_candidate": 3,
"rollups": 4,
"follow_up_tracks": 5
},
"depends_on": [
"data_oriented_error_handling_20260606 (SHIPPED; the v2 audit's result_coverage cross-checks this)",
"data_structure_strengthening_20260606 (SHIPPED; the v2 audit's type_alias_coverage cross-checks this)",
"mcp_architecture_refactor_20260606 (SHIPPED; provides the 6 input audit scripts' baselines)",
"qwen_llama_grok_integration_20260606 (SHIPPED; the v2 audit covers the 8 _send_<vendor> functions)",
"result_migration_20260616 (100% complete as of 2026-06-21; the v2 audit runs against the post-migration src/)"
],
"blocks": [
"pipeline_runtime_profiling_20260607 (preserved from v1; calibrates v2's heuristic cost constants against real measurements)",
"data_pipelines_inventory_<date> (per-pipeline vs per-aggregate reports for the top 5 pipelines)",
"code_path_audit_in_ci_<date> (run v2 in CI on every PR)",
"code_path_audit_data_oriented_refactor_<date> (implement the 3 high-priority componentize candidates)",
"code_path_audit_v2_5_followup_<date> (re-run v2 after any_type_componentization_20260621 merges)"
],
"out_of_scope": [
"No modifications to existing src/*.py files (read-only on the 65 existing files; the v2 audit doesn't change them).",
"No modifications to the 5 existing audit scripts (consume their JSON; don't change them).",
"No runtime profiling (deferred to pipeline_runtime_profiling_20260607).",
"No new pip dependencies (stdlib only: ast, pathlib, json, dataclasses, tomllib, re).",
"No changes to data_structure_strengthening or data_oriented_error_handling styleguides.",
"No changes to v1 spec.md or plan.md (v1 preserved unchanged).",
"No MMA worker spawn action (preserved from v1; user directive 2026-06-07: cold until 1:1 discussion UX is dogfooded).",
"No new src/<thing>.py files (per AGENTS.md file size + naming convention: helpers and sub-systems go in the parent module).",
"The 23 lower-impact files (1-9 weak-type sites each; deferred to a follow-up track).",
"The 3 candidate aggregates' 'real' analysis (deferred to code_path_audit_v2_5_followup_<date>).",
"The v1-style per-action output is preserved for backward compat but downgraded to cross-references."
],
"tolerated_at_run_time": [
"any_type_componentization_20260621 is NOT on master (merged f914b2bc, reverted 751b94d4); the v2 audit produces placeholders for the 3 candidate aggregates with is_candidate: True.",
"phase2_4_5_call_site_completion_20260621 is NOT on master (same merge+revert history).",
"Missing input JSONs in tests/artifacts/audit_inputs/ are tolerated (the corresponding cross_audit_findings field is empty; the markdown notes the absence).",
"Malformed input JSONs are tolerated (the read_input_json() returns Result with errors; the v2 audit continues with empty data)."
],
"test_summary": {
"tests_total": 91,
"tests_unit": 84,
"tests_integration": 7,
"tests_live_gui_opt_in": 2,
"test_tier_count": 11,
"test_pass_count_target": "All 91 tests PASS; the 2 live_gui are opt-in (CODE_PATH_AUDIT_LIVE_GUI=1)"
},
"verification_criteria": [
"FR-1: src/code_path_audit.py is created with the 11 public functions + 4 static analyzers (PCG, MemoryDim, APD, CFE) + 4 renderers (to_dsl_v2, to_markdown, to_tree, parse_dsl_v2) + run_audit() main entry + CLI + MCP tool wrapper",
"FR-2: All 11 public functions return Result[T] per error_handling.md (or return a deterministic T when no runtime failure is possible)",
"FR-3: The 4 audit gates pass in --strict mode (audit_exception_handling, audit_weak_types, audit_main_thread_imports, audit_no_models_config_io)",
"FR-4: The meta-audit (scripts/audit_code_path_audit_coverage.py) passes on the real audit output (0 schema violations)",
"FR-5: The type registry is in sync with src/type_aliases.py (scripts/generate_type_registry.py --check exits 0)",
"FR-6: 91 tests pass (84 unit + 7 integration; 2 live_gui are opt-in)",
"FR-7: The audit output (13 per-aggregate .dsl + .md + .tree files + 4 rollups) is committed to docs/reports/code_path_audit/2026-06-22/",
"FR-8: The TRACK_COMPLETION report is written to docs/reports/TRACK_COMPLETION_code_path_audit_20260622.md",
"FR-9: conductor/tracks.md is updated with the v2 track entry (the checkpoint SHA from the TRACK_COMPLETION report commit)",
"FR-10: The 1-line extension to scripts/audit_optional_in_3_files.py is committed; the extended audit passes in --strict mode",
"FR-11: conductor/code_styleguides/code_path_audit.md is written (the 5-convention styleguide)",
"Atomic per-task commits with git notes per conductor/workflow.md step 9.1-9.3",
"No day estimates, no T-shirt sizes in any artifact"
],
"risks": [
{
"id": "R1",
"description": "The decomposition-cost heuristic is inaccurate (componentize_savings overestimate or underestimate)",
"mitigation": "The runtime-profiling follow-up recalibrates. The override file (scripts/code_path_audit_overrides.toml) lets the user adjust per-aggregate. The summary.md and decomposition_matrix.md headers caveat: 'Savings estimates are heuristic; use as ranking input, not as actual savings.'"
},
{
"id": "R2",
"description": "The PCG misses dynamic patterns (eval, getattr, decorator-driven dispatch like @imscope)",
"mitigation": "The override file lists the known passthroughs. The runtime-profiling follow-up catches the unresolved. The v1 spec's 'unresolved_calls' pattern is preserved."
},
{
"id": "R3",
"description": "The 6 input JSON contracts drift (the existing audit scripts evolve without bumping the v2 audit's contract)",
"mitigation": "The scripts/audit_code_path_audit_coverage.py meta-audit runs in CI; fails on schema drift. The v2 audit tolerates missing fields (returns empty cross_audit_findings; markdown notes the absence)."
},
{
"id": "R4",
"description": "The candidate aggregates don't merge (any_type_componentization_20260621 is delayed)",
"mitigation": "The v2 audit is forward-compatible. The is_candidate: bool flag handles the absence gracefully. The candidates.md rollup explains the placeholder status."
},
{
"id": "R5",
"description": "The v1 .dsl files don't round-trip (the v2 parser is more strict than v1)",
"mitigation": "The v2 parser is a superset of v1; the v1 action reports still parse. The test_v2_dsl_backward_compat_v1 test verifies."
},
{
"id": "R6",
"description": "The synthetic src/ fixture diverges from real src/ (the test expectations don't generalize)",
"mitigation": "The integration test layer runs against real src/ as well as the synthetic fixture. The 2 are decoupled."
},
{
"id": "R7",
"description": "The 4 audit gates regress during implementation (Tier 3 worker adds a try/except violation, Optional[T] return, etc.)",
"mitigation": "Run the 4 audit gates in --strict mode after every commit. If a gate fails, fix before continuing. The audit scripts are the 'laws of physics' for the new file."
},
{
"id": "R8",
"description": "The 85+ tasks exceed Tier 2's per-task context window (the model runs out of memory mid-track)",
"mitigation": "Per-task commits are atomic; the failcount state file persists progress. The per-task commit discipline means each commit is a safe rollback point. If a task fails 3 times, escalate to the user (don't keep retrying)."
},
{
"id": "R9",
"description": "The 91 tests are too long-running for the per-PR CI gate (the user expects <2 min for unit tests)",
"mitigation": "The unit + integration tests run in <30s. The live_gui tests are opt-in via the CODE_PATH_AUDIT_LIVE_GUI env var. The 2 opt-in tests are not in the default run."
},
{
"id": "R10",
"description": "The Tier 2 agent uses a git command that is hard-banned (git restore, git checkout, git reset, git push)",
"mitigation": "The 3-layer hard ban enforcement (OpenCode permission + Windows restricted token + git hooks) catches the violation. The TIER2_STARTUP.md restates the hard bans. If a task requires one, escalate to the user."
}
],
"out_of_scope": [
"Modifications to existing src/*.py files (read-only on the 65 existing files)",
"Modifications to the 5 existing audit scripts (consume their JSON; don't change them)",
"Runtime profiling (deferred to pipeline_runtime_profiling_20260607)",
"New pip dependencies (stdlib only)",
"Changes to data_structure_strengthening or data_oriented_error_handling styleguides",
"Changes to v1 spec.md or plan.md (v1 preserved)",
"MMA worker spawn action (cold per user)",
"New src/<thing>.py files (per AGENTS.md file size + naming convention)",
"The 23 lower-impact files (deferred)",
"The 3 candidate aggregates' real analysis (deferred to v2.5 follow-up)"
],
"follow_up_tracks": [
{
"id": "pipeline_runtime_profiling_20260607",
"purpose": "Calibrate v2's heuristic cost constants against real measurements. Uses src/performance_monitor.py."
},
{
"id": "data_pipelines_inventory_<date>",
"purpose": "Per-pipeline (vs per-aggregate) reports for the top 5 pipelines."
},
{
"id": "code_path_audit_in_ci_<date>",
"purpose": "Run v2 in CI on every PR; fail on new untyped sites or decomposition-matrix regression."
},
{
"id": "code_path_audit_data_oriented_refactor_<date>",
"purpose": "Implement the 3 high-priority componentize candidates (FileItems, History, Metadata)."
},
{
"id": "code_path_audit_v2_5_followup_<date>",
"purpose": "Re-run v2 after any_type_componentization_20260621 merges; the 3 placeholders become real profiles."
}
]
}
conductor/tracks/code_path_audit_20260607/plan.md
-1389
View File
File diff suppressed because it is too large Load Diff
conductor/tracks/code_path_audit_20260607/plan_v2.md
-6320
View File
File diff suppressed because it is too large Load Diff
conductor/tracks/code_path_audit_20260607/spec.md
-410
View File
@@ -1,410 +0,0 @@
# Track: Code Path & Data Pipeline Audit
**Status:** Spec approved 2026-06-07; revised 2026-06-08 with post-4-tracks timing and 5-source framing
**Initialized:** 2026-06-07
**Owner:** Tier 2 Tech Lead
**Priority:** Medium (foundational; enables follow-up pruning track)
> **Revision note (2026-06-08).** The user specified that this audit should run *after* the 4 foundational tracks complete (`qwen_llama_grok_integration_20260606`, `data_oriented_error_handling_20260606`, `data_structure_strengthening_20260606`, `mcp_architecture_refactor_20260606`). The 4 tracks will significantly reshape `src/ai_client.py`, `src/mcp_client.py`, `src/app_controller.py`, and `src/type_aliases.py` — running the audit on the pre-refactor code would produce a report that's stale on day 1. The post-4-tracks timing ensures the audit grounds optimization decisions for the *resulting* architecture, not the pre-refactor one. See §"Timing" below.
---
## Overview
Build `src/code_path_audit.py` — a data-oriented static-analysis tool that audits the 3 major actions (AI message lifecycle, discussion save/load, GUI startup) for expensive operations, redundant calls, and pipelining candidates. The output (custom postfix `.dsl` data + markdown + Mermaid + prefix tree text) is the artifact that informs pipeline-pruning decisions; the actual code changes are a follow-up track (`pipeline_pruning_20260607`).
Per the user's framing: "anything that can even remotely smell as an expensive bulk action or major action that takes more than 10-40 microseconds." The audit focuses on **expensive** operations (file I/O, network, AST parsing, big loops, anything that smells like a bulk action) inside the 3 actions — not on every state mutation. The cost model is heuristic, calibrated by a runtime-profiling follow-up (`pipeline_runtime_profiling_20260607`) that catches the cases static analysis can't resolve (C-extension cost, import cost, JIT effects, decorator-driven dispatch).
The MMA worker spawn action is **out of scope** for this track (per user: "keeping that cold for a while until I like the main ux loop with ai in a discussion fully dogfooded").
## Timing (post-4-tracks)
This track is intentionally **deferred** until *after* the 4 foundational tracks ship:
1. `qwen_llama_grok_integration_20260606` — adds 3 vendors (`_send_qwen`, `_send_llama`, `_send_grok`) and refactors `_send_minimax` to use the shared `send_openai_compatible()` helper. Modifies `src/ai_client.py`, `src/openai_compatible.py` (new), `src/vendor_capabilities.py` (new).
2. `data_oriented_error_handling_20260606` — refactors `ai_client._send_<vendor>` to return `Result[str]`, modifies `mcp_client.py` (30+ sites), `rag_engine.py` (Result returns).
3. `data_structure_strengthening_20260606` — adds `src/type_aliases.py` with 10 TypeAliases, replaces 345 weak-type sites across 6 files.
4. `mcp_architecture_refactor_20260606` — splits `src/mcp_client.py` (2,205 lines → 6 sub-MCPs + 1 external), adds `src/mcp_client_legacy.py` for backward compat.
Running the audit on the **pre-refactor** `src/` would produce a report that's stale on day 1. The post-4-tracks timing ensures:
- The audit's data grounds optimization decisions for the *resulting* architecture (post-Fleury-style "effective codepaths" and "ECS archetype tables" if the 4 tracks are implemented with the data-oriented philosophy).
- The `pipeline_pruning_20260607` follow-up has the *right* candidates to optimize — the 4 tracks will move the expensive ops around, and pruning the wrong ones wastes work.
- The runtime-profiling follow-up (`pipeline_runtime_profiling_20260607`) measures the *new* code paths, not the old ones.
**Pre-flight check (verifies the 4-tracks baseline before this track starts):** confirm that all 4 tracks are marked `[x]` completed in `conductor/tracks.md`. If any of the 4 are still `[~]` in-progress, this track is blocked — the audit would catch the in-progress state as drift.
## Analytical Framing (5-source lens)
The 5 sources loaded into context for the post-4-tracks audit collectively reframe *what* to look for in the 3 actions. The audit's static cost model and pipeline-pruning recommendations should be informed by:
| Source | Lens the audit inherits |
|---|---|
| [Ryan Fleury, "A Taxonomy of Computation Shapes"](https://www.dgtlgrove.com/p/a-taxonomy-of-computation-shapes) (Feb 2023) | The 6 shapes: instruction, codepath, wide codepath, codecycle, wide codecycle, codecycle graph. The audit's `trace_action` is a codepath visualization; the `redundancy` (call_count > 1) field detects **wide codepaths** that could be split into parallel sub-codepaths. |
| [Ryan Fleury, "The Codepath Combinatoric Explosion"](https://www.dgtlgrove.com/p/the-codepath-combinatoric-explosion) (Apr 2023) | The "effective codepath" concept. The audit's `pipelining_candidates` field detects codepaths that *could be defused* (multiple real codepaths collapsed into 1 effective codepath via nil sentinels, generational handles, or immediate-mode APIs). The `redundancy` field is the *first indicator* of defusing opportunities. |
| [Casey Muratori, "The Big OOPs: Anatomy of a Thirty-Five-Year Mistake" (BSC 2025)](https://youtu.be/wo84LFzx5nI) | The 35-year-historical indictment of compile-time domain hierarchies. The audit's per-function `state_mutations` index reveals whether a function is in the *system* pattern (mutates component-like data, not entity state) or the *entity-hierarchy* pattern (mutates a single object's identity, where the cost compounds per type). Functions in the latter pattern are the *highest-priority* refactor targets — they may need to be split into components + systems. |
| [Andrew Reece, "Assuming as Much as Possible" (BSC 2025)](https://www.youtube.com/watch?v=i-h95QIGchY) | The "assume as much as possible" engineering discipline. The audit's `expensive_ops` index, for any function that calls a general-purpose primitive (e.g., `json.dumps`, `Path.read_text`, `ast.parse`), should ask: **"can this caller assume a smaller input domain and use a specialized primitive instead?"** A function that calls `json.dumps` 50 times per action with 1KB payloads each may be replaceable by a function that calls a domain-specific serializer once with a 50KB payload. |
| User's chunk-ideation archive (May 2026) | The "fixed-size slices" + "ECS archetype tables" pattern. The audit's per-function calls that operate on lists/arrays should be flagged if they: (a) don't have a chunk-aware variant, (b) are in a hot path, (c) the data shape is uniform enough to chunk. Functions that match all 3 are the **prime candidates** for `pipeline_pruning_20260607` — chunkification is a known pattern with bounded risk. |
**Concrete audit-time heuristics** that emerge from this framing:
- **Effective-codepath count:** when a function has 3+ branches that all do roughly the same thing with different inputs, the audit should report "this is N real codepaths behaving as 1 effective codepath — could be defused with a nil sentinel or generational handle." The runtime-profiling follow-up measures the actual savings.
- **Entity-hierarchy fingerprint:** when a function's `state_mutations` list has > 3 writes to a single `self.X` with a `type` discriminator, the audit should report "this function is operating on entity-hierarchy state; consider ECS split into components + systems." A *concrete Manual Slop example* the audit should catch: any function that does `if self.active_ticket.kind == TicketKind.X:` and then mutates multiple fields.
- **Assumed-too-much detector:** when a function calls `ast.parse` (or any `tree_sitter.*`) on a file that *could be assumed* to be already-parsed (because the file is in the context composition and the `aggregate.py` pipeline has already done it), the audit should report "this is re-parsing data that was already parsed upstream; consider memoizing or threading the parsed AST through." This is the "assume as much as possible" pattern at the data-passing level.
- **Chunkification candidates:** when a function loops over a `list[dict]` with a known uniform shape (heuristic: all dicts have the same key set), the audit should report "consider chunkifying — uniform data, hot path, no chunk awareness." The user has explicit code (`docs/ideation/ed_chunk_data_structures_20260523.md`) for the chunk pattern, so the audit's optimization candidates can cite it.
These heuristics are *guidance for the audit's report interpretation* — they don't change the audit's static cost model (which is data-grounded in the existing `EXENSIVE_THRESHOLD` + per-class weights). They shape how the Tier 2 Tech Lead and the user interpret the report.
## Current State Audit (as of `ca781543`)
`src/` has 61 `.py` files (27,447 total lines; 23,845 code lines). The call graph is non-trivial; per-action traversal is what makes the analysis tractable.
### Already Implemented (DO NOT re-implement; KEEP / build on)
1. **`src/mcp_client.py:934-992` — `derive_code_path(target, max_depth=5)`.** A single-symbol recursive call tracer with text output. Doesn't render multi-action graphs, doesn't track mutations, doesn't measure cost. The new tool is the multi-action + mutation + cost version of this primitive. **Build on this:** lift the AST traversal logic and `trace()` recursion pattern into `code_path_audit.py`.
2. **`scripts/audit_main_thread_imports.py`** — static CI gate for import-time purity. Different concern (startup-time import cost), but its AST-walking pattern is the model for `code_path_audit.py`'s implementation.
3. **`src/performance_monitor.py`** — runtime profiling with `monitor.scope("name")` and per-component hit counts + latencies. Used at runtime; the follow-up `pipeline_runtime_profiling_20260607` track will use it to calibrate the heuristic cost model.
4. **`conductor/archive/code_path_analysis_20260507/`** — prior manual audit + `PIPELINE_ANALYSIS.md` + Mermaid diagrams for the major pipelines. Manual effort, no reusable tool. New track is the data-grounded successor.
5. **`conductor/archive/ai_interaction_call_graph_20260507/`** — sequence diagram for the AI loop. New track supersedes this for the 3 actions in scope.
6. **SDM docstrings** (`[C: ...]` / `[M: ...]` tags in `src/*.py` docstrings) — pre-computed caller/mutation info. The new audit tool will be a more rigorous version of what SDM already documents ad-hoc.
### Gaps to Fill (this track's scope)
- A static call-graph builder for all of `src/` (multi-action, depth-configurable, machine-readable output).
- A state-mutation index per function (5 mutation kinds: `attr_write`, `container_mutate`, `file_write`, `ipc_emit`, `global_write`).
- An expensive-ops index (7 cost classes, with a heuristic data-size estimate).
- A per-action traversal API (`trace_action(action, max_depth=10) -> ActionProfile`).
- An output suite: custom postfix `.dsl` data files + markdown summaries + Mermaid per-action call graphs + prefix-tree text view.
- A CLI (`python -m src.code_path_audit --action <name>`) and an MCP tool (`code_path_audit(action_name, max_depth)`).
- The actual audit run on the 3 actions, with the report committed to `docs/reports/code_path_audit/2026-06-07/`.
## Goals
1. **Produce a queryable artifact.** The custom postfix `.dsl` output is the source of truth; markdown + Mermaid + prefix-tree text are for human review. Re-run after any `src/` change to see drift.
2. **Surface the top-N optimization candidates per action.** The `summary.md` ranks candidates by potential data-transform load reduction. This is what the user will use to decide which pruning/optimization work to do next.
3. **Data-grounded design.** The audit's data structure is the spec; the heuristics and the threshold are module-level constants tunable from one place.
4. **Reusable across actions.** The `trace_action` API takes any `Action` (entry point + description). Adding a 4th action (e.g., MMA worker spawn, when it's no longer cold) is one `Action(...)` declaration.
5. **Surface calibration gaps clearly.** When the static heuristic can't resolve a call (C-extension, decorator-driven dispatch, `getattr` magic), the report flags it as "unresolved" so the runtime-profiling follow-up targets it.
## Non-Goals
- Not implementing the actual code optimizations — that's `pipeline_pruning_20260607`.
- Not profiling runtime costs — that's `pipeline_runtime_profiling_20260607`.
- Not analyzing the MMA worker spawn action (cold per user).
- Not analyzing `simulation/*` or `tests/*` directories.
- Not analyzing actions beyond the 3 in scope.
- Not resolving C-extension call costs statically.
- Not resolving decorator-driven call dispatch statically (e.g., `@property`, `@imscope`).
- Not providing real microsecond measurements — the cost is heuristic (calibrated later).
## Architecture
`src/code_path_audit.py` — single new module, no new dependencies. Exposes both an MCP tool surface (for agents) and a CLI (`python -m src.code_path_audit ...`).
### Public API
```python
class CallGraph:
"""Directed graph: nodes are functions; edges are call sites."""
nodes: dict[str, "FunctionNode"] # fully-qualified name -> node
edges: dict[str, set[str]] # caller -> set of callees
def add_edge(self, caller: str, callee: str) -> None: ...
def transitive_callees(self, root: str, max_depth: int = 10) -> set[str]: ...
def render_mermaid(self, root: str, max_depth: int = 5) -> str: ...
class FunctionNode:
fqname: str # "src.ai_client.AIClient.send"
file: str
line: int
calls: list[str] # all callees (resolved or not)
state_mutations: list["StateMutation"]
expensive_ops: list["ExpensiveOp"]
class StateMutation:
target: str # "self.history", "module.events", "file:..."
kind: Literal["attr_write", "container_mutate", "file_write", "ipc_emit", "global_write"]
line: int
class ExpensiveOp:
callee: str
cost_class: Literal["file_io", "network", "ast_parse", "json_io", "pickle", "deep_copy", "loop_amplified"]
data_size_estimate: int | None # bytes or container length, heuristic
line: int # call site in the caller
weight: int # cost_class_weight * data_size (or 1 if data_size unknown)
class Action:
name: str # "ai_message_lifecycle"
entry_points: list[str] # ["src.app_controller.AppController.process_user_request", ...]
description: str
class ActionProfile:
action: Action
call_graph: CallGraph # subgraph reachable from entry points
expensive_ops: list[ExpensiveOp] # all expensive ops in the subgraph
state_mutations: list[StateMutation] # all mutations in the subgraph
redundancy: list[tuple[str, int]] # (op_fqname, call_count) where count > 1
pipelining_candidates: list[list[str]] # groups of independent ops currently sequential
total_load_estimate: int # sum(weight) heuristic
unresolved_calls: list[str] # calls the AST walker couldn't resolve
mermaid: str # rendered Mermaid
markdown: str # human-readable per-action report
def trace_action(action: Action, max_depth: int = 10) -> ActionProfile: ...
def build_call_graph(src_dir: str = "src") -> CallGraph: ... # full call graph
def build_expensive_ops_index(cg: CallGraph) -> dict[str, list[ExpensiveOp]]: ...
def build_state_mutations_index(cg: CallGraph) -> dict[str, list[StateMutation]]: ...
```
### Cost Model (heuristic, calibrated by the runtime-profiling follow-up)
| Pattern | Cost class | Default weight | Data size source |
|---------|-----------|----------------|------------------|
| `open()`, `Path.read_*`, `Path.write_*`, `*.write_text` | `file_io` | 100 | file size from `Path.stat()` when resolvable, else `None` |
| `requests.*`, `urllib.*`, `websockets.*`, `client.send` (with httpx-like signatures) | `network` | 500 | payload size from param literal/typed hint |
| `ast.parse`, `ast.walk`, `tree_sitter.*` | `ast_parse` | 200 | source bytes from the path arg |
| `json.dump`, `json.load`, `tomli_w.dump`, `tomllib.load` | `json_io` | 150 | container length if param is a list/dict |
| `pickle.dump`, `pickle.load` | `pickle` | 300 | container length |
| `copy.deepcopy` | `deep_copy` | 200 | container length |
| Any call inside the body of a `for` / `while` loop | `loop_amplified` | caller_weight × loop_bound_estimate | loop bound = `range(...)` literal/arg, else 1 |
**Expense threshold:** `EXPENSIVE_THRESHOLD = 40_000` (module-level constant). Any `ExpensiveOp.weight > EXPENSIVE_THRESHOLD` is flagged "expensive" in the per-action report. The 40,000 default matches the user's stated 10-40μs range; the runtime-profiling follow-up will calibrate it.
**Unresolved calls:** when the AST walker cannot resolve a callee (e.g., attribute access on `self.X` where `X` is set dynamically; `getattr`; decorator-wrapped method dispatch), the call goes into `unresolved_calls` with a `"unresolved"` cost class and weight 0. The report's caveats section notes these; the runtime-profiling follow-up measures them.
### Out of the static analysis
- C-extension call costs (imgui-bundle, tree-sitter native) — runtime profiling only.
- Decorator-driven dispatch (e.g., `@property`, `@imscope`) — runtime profiling only.
- Import cost at module load time — covered by the existing `scripts/audit_main_thread_imports.py`.
- `eval` / `exec` calls — flagged as unresolved, not analyzed.
## Per-Action Design
For each of the 3 actions, the audit is invoked with one or more entry points and a depth limit (default 10). The audit produces an `ActionProfile` that the report renders.
| Action | Entry points | Expected high-cost ops the audit should surface |
|--------|--------------|------------------------------------------------|
| **AI message lifecycle** | `src.app_controller.AppController.process_user_request`, `src.ai_client.AIClient.send`, `src.aggregate.build_file_items`, `src.summarize._summarise_*` | Per-context-file AST parse in `build_file_items`; AI network call; history append + comms log append + session_logger file write; sub-agent summarization (network + AST, loop-amplified over context files) |
| **Discussion save/load** | `src.project_manager.save_project`, `src.project_manager.load_project`, `src.history.HistoryManager.save_snapshot`, `src.models.parse_history_entries` | `tomli_w.dump` / `tomllib.load` on project TOML; `json.dump` on comms log (loop-amplified per entry); history file read/write; AST parse on schema validation |
| **GUI startup** | `sloppy.main` → `gui_2.App.__init__`, `src.app_controller.AppController.__init__`, `src.paths._resolve_*` | `tomllib.load` on config.toml; AST parses for tool registration; file stat on log paths; `sloppy.py` first-frame import chain (covered by the existing `scripts/audit_main_thread_imports.py`) |
The user can extend with more actions later (e.g., MMA worker spawn when it's no longer cold). Each action is one `Action(...)` declaration + a `trace_action()` call.
## Output Format
CLI:
```bash
uv run python -m src.code_path_audit --action ai_message_lifecycle [--depth N] [--dsl] [--tree] [--markdown] [--mermaid]
```
MCP tool (for agents):
```python
code_path_audit(action_name: str, max_depth: int = 10) -> dict
```
Generated artifacts (all under `docs/reports/code_path_audit/<YYYY-MM-DD>/`):
| File | Format | Purpose |
|------|--------|---------|
| `call_graph.dsl` | Custom postfix DSL | Full call graph (all of `src/`); machine-readable, parses in ~30 lines |
| `expensive_ops.dsl` | Custom postfix DSL | Expensive ops index (per-file, per-function) |
| `state_mutations.dsl` | Custom postfix DSL | State mutations index (per function) |
| `actions/<action>.dsl` | Custom postfix DSL | Per-action profile (machine-readable) |
| `actions/<action>.tree` | Prefix tree (text) | Per-action human-readable tree (for human review) |
| `actions/<action>.md` | Markdown | Per-action summary + table (for code review) |
| `actions/<action>.mmd` | Mermaid | Per-action call graph (visual) |
| `summary.md` | Markdown | Top-level cross-action summary + ranked optimization candidates |
| `optimization_candidates.md` | Markdown | Ranked list with: candidate, current cost, proposed reduction, effort, priority |
The two follow-up tracks consume the .dsl files; the markdown + tree are for human review.
**The custom DSL is postfix (RPN) with length-prefixed lists** — no brackets, no braces, no commas, no colons. Each "word" is a tagged constructor that consumes a known number of args from the stack (e.g., `fn` consumes 3, `exp-op` consumes 5, `mut` consumes 3, `N list` consumes N items). Whitespace-tokenized. Strings are bare atoms when they have no whitespace; quoted only when needed. `nil` for null. `\` for line comments. The DSL is deliberately NOT strict Forth — it's a custom postfix format tailored to the audit's record shapes (function, call, mutation, expensive op, pair, list).
Example of a single FunctionNode record:
```text
\ FunctionNode: fqname file line fn
"src.ai_client.AIClient.send" "src/ai_client.py" 100 fn
"build_file_items" call
"process_response" call
"self.history" attr_write 110 mut
"open" file_io 100 120 exp-op
```
**The prefix tree renderer** is a separate human-readable view of the same data — top-down, `├─`/`└─`/`│` box-drawing, scannable. Generated by a recursive walker. Inlined in the markdown reports (optionally produced as `actions/<action>.tree` for tooling).
**Why custom postfix DSL (not JSON, not s-expressions, not strict Forth):**
- **Not JSON** (JSON is ill-performant: quoting, escaping, hash table allocation, no streaming).
- **Not s-expressions** (the bracket version drifts back toward s-exprs; the user wanted postfix specifically).
- **Not strict Forth** (the user wants a format ideal for call-graph recording, not a Turing-complete Forth program).
- **Postfix** (per user: "I want a post-fix heiarchy"): stack-based, no delimiters to count.
- **Length-prefixed lists** (standard postfix solution for nesting): `N list` consumes N items, unambiguous.
- **Trivial parser** (~30 lines: split + walk + evaluate tagged words against a known arity table).
- **Compact**: ~30-40% fewer characters than JSON for the same data.
- **Streamable**: no need to parse the whole file to find a record; you can scan for tags.
- **Extensible**: add new metric types by adding new tagged words (`metric(name value sample_size)`, `histogram(buckets)`, etc.).
## Verification (TDD per `conductor/workflow.md`)
Unit tests in `tests/test_code_path_audit.py`:
- `CallGraph.add_edge` + `transitive_callees` correctness on a synthetic 5-node graph.
- `ExpensiveOpIndex` detects each of the 7 cost classes on synthetic source.
- `StateMutationIndex` detects each of the 5 mutation kinds on synthetic source.
- `trace_action` produces an `ActionProfile` for a synthetic action whose expected cost is computable by hand.
- Custom postfix `.dsl` output round-trips (parse_dsl(to_dsl(profile)) == in-memory structure).
- Prefix tree renderer produces well-formed box-drawing output for the 3 per-action reports.
- Markdown output is well-formed (header per section, table per category).
- Mermaid output parses as valid Mermaid syntax.
Smoke test: run `python -m src.code_path_audit --action ai_message_lifecycle --depth 5` against a fixture project; verify the report is produced and contains the expected high-cost ops (per the table above).
Manual verification: the report is the deliverable. A Tier 2 Tech Lead + user review the produced `summary.md` to confirm the optimization candidates make sense.
## Commit Structure (6 atomic commits, in order)
```
1. feat(audit): add code_path_audit data structures (CallGraph, ExpensiveOpIndex, StateMutationIndex)
- src/code_path_audit.py (initial data structures)
- tests/test_code_path_audit.py (unit tests)
2. feat(audit): add trace_action + ActionProfile + cost model
- src/code_path_audit.py (extends with action tracing)
- tests/test_code_path_audit.py (integration tests)
3. feat(audit): add custom postfix DSL writer + parser + tree renderer / markdown / Mermaid output
4. feat(audit): add MCP tool + CLI surface
5. docs(audit): run audit on 3 actions; commit report
- docs/reports/code_path_audit/2026-06-07/* (the deliverable)
6. conductor(tracks): mark Code Path Audit track complete
- tracks.md update
```
Each commit message includes a `git notes add -m "..."` summary per `conductor/workflow.md` step 9.1-9.3.
## Risks
| Risk | Likelihood | Impact | Mitigation |
|------|-----------|--------|------------|
| Heuristic cost model is imprecise; reported "expensive" ops aren't actually expensive at runtime. | Medium | Medium (false positives dilute the report) | `EXPENSIVE_THRESHOLD` is a module-level constant; the runtime-profiling follow-up calibrates it. |
| AST walking misses dynamic patterns (eval, getattr, decorator-driven dispatch). | Medium | Medium (under-estimates some calls) | Document the limitations in the report's caveats section; the runtime-profiling follow-up catches these. |
| Mermaid diagrams exceed renderable size for deep actions. | Medium | Low (visualization only) | Default `max_depth=5` for `--mermaid`; full graph available as `.dsl`. |
| The 3 actions' entry points are not exactly the functions the user has in mind. | Medium | Low (the report is the artifact; user can re-run with different entry points) | Document the chosen entry points in the report; CLI/MCP tool accepts any fully-qualified function name. |
| Report is too large to review (thousands of expensive ops). | Low | Medium | Per-action scoping; default `--depth 5`; ranked optimization candidates in `summary.md` make the top-N obvious. |
| Existing `derive_code_path` is the de-facto call-graph tool and the new one is redundant. | Low | Low (the new one is a strict superset) | `derive_code_path` stays as a thin wrapper around `code_path_audit.trace_action` for backward compat, OR gets a `@deprecated` shim. |
| The 3 actions are not actually the user's top 3 (user might have meant a different 3). | Low | Low (the tool is generic; re-run with different actions is one CLI call) | CLI accepts any `Action`; user can re-run. |
## Coordination with Pending Tracks
This track has **no blockers** and **no conflicts**. It can ship independently of the 5 active planned tracks. **It enables** future refactors:
| Pending track | Could use this analysis for... |
|----------------|--------------------------------|
| `qwen_llama_grok_integration_20260606` | Identifying redundant OpenAI-compatible request paths in `_send_*` functions |
| `data_oriented_error_handling_20260606` | Showing the call paths the new `Result[T]` return values will thread through |
| `data_structure_strengthening_20260606` | Pinpointing hot functions where the new type aliases matter most |
| `mcp_architecture_refactor_20260606` | Identifying which sub-MCPs have the most expensive operations (file_io vs network vs ast) |
| `test_batching_refactor_20260606` | Confirming which tests trigger the most expensive paths (to optimize test selection) |
This track's analysis is **read-only** — it doesn't modify `src/`, doesn't change the public API, doesn't add tests to the existing test suite. The only new files are `src/code_path_audit.py` (the tool), `tests/test_code_path_audit.py` (the tests), and the report under `docs/reports/code_path_audit/2026-06-07/`.
## Pre-Flight Adjustments (2026-06-21, per handoffs from `any_type_componentization_20260621`)
The `any_type_componentization_20260621` track (shipped 2026-06-21 with 48/89 sites promoted) revealed that **the 4 foundational tracks this audit was deferred behind have evolved**. Specifically, 5 new hot-path dataclasses (`ToolSpec`, `ChatMessage`, `UsageStats`, `ToolCall`, `WebSocketMessage`) and 1 new module (`provider_state.ProviderHistory`) now exist. This audit must instrument them.
**Per `docs/handoffs/PROMPT_FOR_TIER_1.md` and `HANDOFF_CODE_PATH_AUDIT_FROM_any_type_componentization.md`, the following 4 adjustments are added to this audit's scope:**
### A1. Add 2 new actions to the per-action profiling
The existing 3 actions (`ai_message_lifecycle`, `discussion_save_load`, `gui_startup`) become 5:
| Action | Codepath | Measures |
|---|---|---|
| `provider_history_append` (NEW) | `get_history(p).append(msg)` (or legacy `_anthropic_history.append(msg)`) | Per-turn append latency + lock acquire time + memory allocation per call. The hot path Phase 3 will refactor. |
| `websocket_broadcast` (NEW) | `broadcast(WebSocketMessage(...))` (post-Phase 6a) | Per-broadcast overhead (allocation + JSON serialization + WebSocket send). The GUI thread's per-event cost. |
| `ai_message_lifecycle` (existing) | `_send_<provider>` end-to-end | Total per-turn latency delta pre/post Phase 3 (`provider_state.ProviderHistory`). The 3 OpenAI-compatible providers (`grok`, `minimax`, `llama`) are **newly instrumented** (currently unprofiled). |
| `discussion_save_load` (existing) | `reset_session()` + project switch | Cold-path cost. The `clear_all()` migration's per-call delta. |
| `gui_startup` (existing) | `_PROVIDER_HISTORIES` dict init at module load | One-time init cost (6 `ProviderHistory()` instances + 6 locks). |
### A2. Add 5 micro-benchmarks to the audit's `optimization_candidates.md`
The audit's per-call cost estimates should include these 5 micro-benchmarks (added per `HANDOFF_FOLLOWUP_TRACK_FROM_any_type_componentization.md` §7):
| Micro-benchmark | Purpose | Expected overhead |
|---|---|---|
| `NormalizedResponse.__init__` | Dataclass construction vs the old 6-field dict literal | <1μs; immaterial |
| `WebSocketMessage.__init__` | Dataclass construction per broadcast | <5μs; the hot path concern |
| `UsageStats.__init__` | Nested dataclass construction per response | <500ns; negligible (4 int fields) |
| `ProviderHistory.lock` acquire | threading.Lock acquire overhead | <500ns; the threading hot path |
| `ToolSpec.__init__` | Dataclass construction per tool (45 tools, cold path) | <2μs; only at registration |
The benchmarks are emitted to `docs/reports/code_path_audit/<date>/micro_benchmarks.md`.
### A3. Add the "no-TypeError-errors-on-any-thread" assertion
The audit's per-action profiling runs the 5 actions in a controlled harness. The audit MUST assert that no `worker[queue_fallback] error: WebSocketServer.broadcast() takes 2 positional arguments but 3 were given` (or any TypeError on any thread) appears in the harness output during profiling.
This assertion catches the broadcast() regression that `any_type_componentization_20260621` introduced. The regression test that backs this assertion lives in `tests/test_websocket_broadcast_regression.py` (added by the `phase2_4_5_call_site_completion_20260621` follow-up track).
If the assertion fires, the audit's output should:
1. Mark the affected action's profile as `INSTRUMENTATION_CONTAMINATED`
2. List the offending thread + traceback in the report's `errors.md`
3. Recommend re-running the audit AFTER `phase2_4_5_call_site_completion_20260621` merges
### A4. Add the 89 fat-struct sites as instrumented targets
The audit reads `docs/reports/ANY_TYPE_AUDIT_20260621.md` §3's table and tags each `Any` usage with `(file:line, hot_path, cold_path, init_path)`. The 89 sites become per-action cost estimates that flow into `optimization_candidates.md`.
For the 48 promoted sites, the audit compares pre-refactor (legacy globals + dict literals) vs post-refactor (dataclass + registry). For the 41 deferred Phase 3 sites, the audit produces per-call cost estimates that inform the future Phase 3 follow-up track (see `docs/reports/PHASE3_HYPOTHETICAL_PROMOTION.md` for the qualitative estimates).
### A5. Sequencing (BLOCKER)
**This audit is now blocked by `phase2_4_5_call_site_completion_20260621` (the broadcast() fix).** Until Phase 6a merges, the GUI thread's `worker[queue_fallback]` TypeError spam contaminates the audit's per-action profiling.
**Recommended sequence:**
```
T0: Tier 1 approves follow-up track (decision: SHRINK to 6a + 6b + 6d)
T1: Tier 2 implements Phase 6a + 6b + 6d (~3 hours, ~16 commits)
T2: Tier 1 reviews + merges follow-up track
T3: Tier 1 launches code_path_audit_20260607
T4: Tier 2 implements Phase 3 + cross-phase coupling (separate track, post-audit)
```
### A6. New coordination with `any_type_componentization_20260621`
This audit now has **new dependencies** beyond the original 4 foundational tracks:
| Track | Status | Provides to this audit |
|---|---|---|
| `any_type_componentization_20260621` | Shipped 2026-06-21 (48/89 promoted) | The 5 dataclasses + 1 module; the 200-site dataclass-coverage baseline |
| `phase2_4_5_call_site_completion_20260621` | Spec'd 2026-06-21; not yet merged | The fix for the broadcast() TypeError; the "no-TypeError" assertion |
This audit is `blocked_by` both tracks (post-merge).
## Follow-up
- **`pipeline_runtime_profiling_20260607`** (the user-requested follow-up; NOT in this track): adds a runtime profiling harness using the existing `src/performance_monitor.py` + a per-action test fixture. Measures real costs for the 3 actions. Calibrates the heuristic cost model (`EXPENSIVE_THRESHOLD` + per-class weights). Catches "things that aren't easy to resolve statically" — import cost, JIT effects, GC pauses, C-extension call cost (imgui-bundle, tree-sitter native), decorator-driven dispatch. Output: `scripts/runtime_profiler.py` + updated `code_path_audit.py` cost model.
- **`pipeline_pruning_20260607`** (the second follow-up; NOT in this track): implements the high-priority optimization candidates surfaced by this track's report. Will be scoped AFTER this track ships, since the report itself defines what to prune.
## Out of Scope
- **MMA worker spawn action** (deferred per user — keeping MMA cold until the 1:1 discussion UX is dogfooded in a few projects).
- **Implementing the optimization fixes** (deferred to `pipeline_pruning_20260607`).
- **Runtime profiling** (deferred to `pipeline_runtime_profiling_20260607` per the user's explicit ask).
- **Other major actions** beyond AI message, save/load, GUI startup.
- **C-extension call costs** (deferred to runtime profiling).
- **Decorator-driven call dispatch** (deferred to runtime profiling).
- **`simulation/*` and `tests/*` directories** (analysis is `src/`-only for this track; can be extended later).
- **Modifying `src/`** (read-only analysis).
## See Also
- `conductor/archive/code_path_analysis_20260507/` — prior manual audit; the new track is its data-grounded successor.
- `conductor/archive/ai_interaction_call_graph_20260507/` — prior sequence diagram for the AI loop.
- `src/mcp_client.py:934-992` — `derive_code_path(target, max_depth=5)` (single-symbol tracer; the new tool supersedes this for multi-action use).
- `src/performance_monitor.py` — runtime profiling infrastructure used by the `pipeline_runtime_profiling_20260607` follow-up.
- `scripts/audit_main_thread_imports.py` — related static CI gate (startup-time import cost).
- `docs/reports/PLANNING_DIGEST_20260606.md` — planning context; the 5 active planned tracks are independent of this one.
- `docs/guide_data_oriented.md` (if it exists; otherwise `conductor/product-guidelines.md` "Data-Oriented & Immediate Mode Heuristics") — the project's data-oriented design philosophy this track follows.
- **`conductor/tracks/nagent_review_20260608/report.md` §15** (Pitfalls #2 and #4, "provider-specific history in process globals" and "AI client is a stateful singleton") — the audit's `state_mutations` index will surface both of these in the post-4-tracks `src/ai_client.py`; the optimization candidates should specifically address them.
- **`docs/transcripts/wo84LFzx5nI_big_oops_casemuratori.txt`** — full transcript of Casey Muratori's "The Big OOPs" talk, loaded 2026-06-08 for context. The historical genealogy (Stroustrup, Kay, Simula, Hoare) grounds the audit's "entity-hierarchy fingerprint" heuristic (above). Specifically, Hoare's 1966 "Record Handling" paper introduced discriminated unions — which Simula kept (as `inspect`) but C++ removed. The audit's `actions/ai_message_lifecycle.tree` should be checked for `if/else` chains that *would be* a discriminated union if `Result[T]` were threaded through.
- **`docs/transcripts/i-h95QIGchY_assuming_as_much_as_possible_andrewreece.txt`** — full transcript of Andrew Reece's "Assuming as Much as Possible" talk, loaded 2026-06-08 for context. Reece's "Xar" data structure (8-byte header, power-of-2 chunks, bitwise divmod, no `realloc` copy) is the *exemplar* for the chunkification-candidate heuristic. The `summary.md` of the audit's report should note the Xar pattern as a possible optimization target for any function in the hot path that does append-heavy work on a list of uniform items.
- **`docs/ideation/ed_chunk_data_structures_20260523.md`** — user's chunk-based-data-structure ideation (May 2026). The 5-image archive is the source of the "chunkification candidates" heuristic. Specifically, the user notes: *"if my chunk size is 1,000 elements, but I only have 5 elements to store, aren't I wasting a massive amount of memory?"* — the audit should distinguish *real* chunkification candidates (uniform data, hot path, large N) from *false* chunkification candidates (small N, low frequency, polymorphic data).
- **`docs/reports/computational_shapes_ssdl_digest_20260608.md`** — the SSDL digest synthesizing the 4-source computational-shapes thinking. The audit's `actions/<action>.tree` and `actions/<action>.mmd` outputs *are* computational-shape visualizations; the SSDL vocabulary (6 primitives + 7 modifiers) is the conceptual model the audit's tree renderer should follow.
conductor/tracks/code_path_audit_20260607/spec_v2.md
-650
View File
@@ -1,650 +0,0 @@
# Track Specification: Code Path & Data Pipeline Audit v2
**Status:** Spec v2 (revised 2026-06-22; v1 was approved 2026-06-07 and revised 2026-06-08 with the post-4-tracks timing + 5-source framing)
**Initialized:** 2026-06-07 (v1); 2026-06-22 (v2 supersedes v1)
**Owner:** Tier 1 (spec) -> Tier 2 (plan + execution)
**Priority:** High (foundational; enables follow-up pruning + per-pipeline refactor tracks)
**Folder:** `conductor/tracks/code_path_audit_20260607/`
**Files:** `spec.md` (v1; preserved), `spec_v2.md` (this file), `plan.md` (v1; preserved), `plan_v2.md` (after this spec is approved)
> **v2 revision note (2026-06-22).** The v1 spec.md (approved 2026-06-07; revised 2026-06-08) was never executed (no `state.toml`, no `metadata.json`, no `scripts/code_path_audit/code_path_audit.py` in the working tree). The 14-day gap saw 4 foundational tracks ship (`qwen_llama_grok_integration_20260606`, `data_oriented_error_handling_20260606`, `data_structure_strengthening_20260606`, `mcp_architecture_refactor_20260606`), the entire 5-sub-track `result_migration` campaign ship (2026-06-16 through 2026-06-21; 100% complete), and the `nagent_review` corpus grow from v1 to v3.1. v2 re-scopes the audit from "expensive operations per action" to "data pipelines per aggregate" — the v1 framing was correct at the time (the 4 tracks were future) but is now stale. v2 also cross-validates the `data_structure_strengthening_20260606` + `data_oriented_error_handling_20260606` deductions directly, which v1 could not (those tracks didn't exist on 2026-06-07). See §"Why v2" below.
---
## Why v2 (the rationale for the revision)
The user's framing (2026-06-22):
> "The whole point of the code path audit is to audit all paths nearly in the ./src of the codebase. The main point of it is to identify data-oriented pipelines and what data aggregate they will be operating on. This will realize what the data strengthening just uncovered and cross-audit if its deductions on the data structures are accurate while also being able to utilize additional flexibility the data oriented error handling track has provided. We are entering a time where the codebase is getting heavily adjusted into a properly engineered machine with discernable working parts."
>
> "The cost of the pipeline is important, it should factor in what data needs to be componentized further vs which can be unified further into wider code paths handling larger fat structs."
**Three changes from v1 to v2:**
1. **Output structure: per-action -> per-data-aggregate.** v1 emitted 3 per-action profiles (`ai_message_lifecycle`, `discussion_save_load`, `gui_startup`). v2 emits 10+3 per-data-aggregate profiles (`Metadata`, `FileItem`, `FileItems`, `CommsLogEntry`, `CommsLog`, `HistoryMessage`, `History`, `ToolDefinition`, `ToolCall`, `Result[T]` + the 3 candidate aggregates `ChatMessage`, `ToolSpec`, `ProviderHistory`). The per-action reports are preserved for backward compat but downgraded to "cross-references to the per-aggregate profiles."
2. **Cross-validation with the 5 existing audit scripts.** v1 was a standalone tool. v2 consumes JSON from `audit_weak_types`, `audit_exception_handling`, `audit_optional_in_3_files`, `audit_no_models_config_io`, `audit_main_thread_imports`, and the type registry (`generate_type_registry.py --json`). The v2 audit's per-aggregate `cross_audit_findings` + `result_coverage` + `type_alias_coverage` are the cross-checks of the 2 foundational tracks (`data_structure_strengthening` + `data_oriented_error_handling`).
3. **The decomposition-cost heuristic.** v1 had a "cost model" focused on expensive operations (file I/O, network, AST parse). v2 adds a `DecompositionCost` heuristic per aggregate that answers the user's question: "should this data be componentized further (split into smaller dataclasses) or unified further (combined into wider fat structs)?" The recommendation is grounded in 3 dimensions: access pattern (whole_struct / field_by_field / hot_cold_split / bulk_batched / mixed), frequency (hot / per_turn / per_discussion / per_request / cold / init / unknown), and shape (struct_field_count + struct_frozen).
---
## Overview
Build `scripts/code_path_audit/code_path_audit.py` v2 — a data-oriented static-analysis tool that audits the data pipelines in `src/` and produces per-data-aggregate profiles. The output (custom postfix `.dsl` data + markdown + prefix tree text, organized per-aggregate) is the artifact that informs per-aggregate refactor decisions. The actual code changes are follow-up tracks (the 3 high-priority candidates from `decomposition_matrix.md`).
The v2 audit's primary value is **cross-validation**: it consumes the JSON outputs of the 5 existing audit scripts and synthesizes them with the per-aggregate producer/consumer call graph. The result is a per-aggregate report that says "this aggregate has 12 weak-type sites (cross-checks `data_structure_strengthening`), 5 exception-handling sites (cross-checks `data_oriented_error_handling`), and 1 high-priority optimization candidate (decomposition direction: componentize)." The user reads one report per aggregate, not one per action.
The v2 audit is **read-only** on `src/` (the only new file is the tool itself + its tests + the report). The MMA worker spawn action is **out of scope** (per v1; the user's "keeping MMA cold" directive from 2026-06-07 still stands). Runtime profiling is **out of scope** (deferred to `pipeline_runtime_profiling_20260607`); the v2's heuristic cost constants are recalibrated by that follow-up.
---
## Current State Audit (as of `7e61dd7d`)
`src/` has 65 `.py` files (per the result migration campaign's final state). The call graph is dense; per-aggregate traversal is what makes the analysis tractable. The 4 foundational tracks that v1 deferred behind have all shipped; the 2 follow-up tracks (`any_type_componentization_20260621` + `phase2_4_5_call_site_completion_20260621`) are NOT on master (merged in `f914b2bc` then reverted in `751b94d4`); the v2 audit must be tolerant of their absence for an interim run.
### Already Implemented (DO NOT re-implement; KEEP / build on)
1. **`scripts/audit_main_thread_imports.py`** — the import-graph CI gate. The v2 audit consumes its JSON output (per the v2's `cross_audit_findings.import_graph` field). v2 does not modify this script.
2. **`scripts/audit_weak_types.py`** — the weak-types CI gate. v2 consumes its JSON output. v2 does not modify this script.
3. **`scripts/audit_exception_handling.py`** — the exception-handling CI gate (per `error_handling.md`). v2 consumes its JSON output. v2 does not modify this script.
4. **`scripts/audit_optional_in_3_files.py`** — the `Optional[T]` ban CI gate for the 3 refactored files (`mcp_client.py`, `ai_client.py`, `rag_engine.py`). v2 extends this script by 1 line (add `scripts/code_path_audit/code_path_audit.py` to the baseline list); the convention is the same.
5. **`scripts/audit_no_models_config_io.py`** — the config-I/O ownership CI gate (per `conductor/code_styleguides/config_state_owner.md`). v2 consumes its JSON output. v2 does not modify this script.
6. **`scripts/generate_type_registry.py`** — the type-registry generator (per `conductor/code_styleguides/type_aliases.md`). v2 consumes its JSON output. v2 does not modify this script.
7. **`src/type_aliases.py`** — the 10 canonical TypeAliases + 1 NamedTuple (`FileItemsDiff`). v2 imports these; v2 does not redefine them. The 13 data aggregates (10 + 3 candidates) are referenced by their canonical names.
8. **`src/result_types.py`** — `Result[T]`, `ErrorInfo`, `NilPath`, `NilRAGState`, `ErrorKind`. v2 imports these; v2 does not redefine them. v2's public functions return `Result[T]` per the `error_handling.md` hard rule.
9. **`src/mcp_client.py:934-992` — `derive_code_path(target, max_depth=5)`.** A single-symbol recursive call tracer with text output. v2 builds on this pattern; the v2's PCG P1 (return-type pass) is the multi-symbol superset. The v1 spec's `CallGraph` is subsumed by the v2's `ProducerConsumerGraph` (function-to-aggregate edges, not function-to-function edges).
10. **`src/performance_monitor.py`** — runtime profiling with `monitor.scope("name")` + per-component hit counts + latencies. Used at runtime; the `pipeline_runtime_profiling_20260607` follow-up uses it to calibrate the v2's heuristic cost constants.
11. **`conductor/code_styleguides/data_oriented_design.md`** — the canonical DOD reference. v2's decomposition-cost heuristic is informed by the 8 defaults in §2 (especially "The common case dominates" + "Where there is one, there are many"). v2's per-aggregate access pattern classification follows the DOD's "Algorithms on data" framing.
12. **`conductor/code_styleguides/error_handling.md`** — the `Result[T]` convention. v2's public API returns `Result[T]` per the hard rule (§"Hard Rules" §"The 5 MUST-DO rules" + §"The 7 MUST-NOT-DO rules").
13. **`conductor/code_styleguides/type_aliases.md`** — the 10 TypeAliases + 1 NamedTuple. v2's per-aggregate `type_alias_coverage` metric is the cross-check of this convention.
14. **`conductor/code_styleguides/agent_memory_dimensions.md`** — the 4 mem dims (curation / discussion / RAG / knowledge). v2's `MemoryDim` classifier (§7.2.2) follows the styleguide's "shape rule" (a feature that wants one should use the matching dimension).
15. **`conductor/code_styleguides/feature_flags.md`** — the "delete to turn off" pattern. v2's `scripts/audit_code_path_audit_coverage.py` is a feature flag (the meta-audit); removing the file disables the meta-audit.
16. **`conductor/code_styleguides/cache_friendly_context.md`** — the stable-to-volatile cache ordering. v2's per-aggregate reports are a downstream consumer of the cache state (the `cache_friendly_context` is the "what stays in the LLM's context"; the v2's per-aggregate profile is the "what data flows through the LLM").
17. **`conductor/code_styleguides/knowledge_artifacts.md`** — the knowledge harvest pattern. v2's per-aggregate profiles are NOT a knowledge artifact (they're a curation artifact, per the 4-dim rule).
18. **`conductor/code_styleguides/rag_integration_discipline.md`** — the conservative-RAG rule. v2's `RAG` aggregate (RAGEngine state, indexed chunks) is classified by the `MemoryDim` classifier; the audit does not mutate RAG state.
19. **SDM docstrings** (`[C: ...]` / `[M: ...]` tags in `src/*.py` docstrings) — pre-computed caller/mutation info. v2's PCG is a more rigorous version of what SDM already documents ad-hoc.
20. **`conductor/tracks/nagent_review_20260608/nagent_review_v3_1_20260620.md`** — the v3.1 nagent review. v2 references the v3.1 Candidates 27-30 (Markdown + custom DSL lock-in, per-turn ground-truth hook, dataset-curation track, cache TTL GUI hardening). The v2's custom postfix DSL is a direct application of Candidate 27 (markdown + custom DSL).
21. **`docs/reports/computational_shapes_ssdl_digest_20260608.md`** — the SSDL digest that informed the v1 spec's 5-source lens. v2 preserves the lens (the 6 SSDL primitives are referenced in the v2's per-aggregate access pattern + frequency classification).
22. **`docs/reports/RESULT_MIGRATION_CAMPAIGN_STATUS_20260619.md`** — the 100%-complete `result_migration` campaign (268 sites migrated + 9 legacy wrappers obliterated across 6 sub-tracks, 2026-06-16 through 2026-06-21). v2's `result_coverage` metric is the post-campaign check that the convention was applied uniformly across all 65 `src/` files.
23. **`docs/reports/ANY_TYPE_AUDIT_20260621.md`** — the 89-site audit (48 promoted + 41 deferred) that informed `any_type_componentization_20260621`. v2 references the 3 candidate aggregates (§3.1 `ToolSpec`, §3.2 `ChatMessage`, §3.3 `ProviderHistory`) as forward-compat placeholders.
24. **`docs/reports/PHASE3_HYPOTHETICAL_PROMOTION.md`** — the Tier 2's authoritative cost analysis of the 41 deferred Phase 3 sites (the 112 call sites in `_send_<provider>()` that would migrate to `ProviderHistory.append()`). v2's `ProviderHistory` candidate aggregate's placeholder is sourced from this report.
25. **`conductor/tracks/code_path_audit_20260607/spec.md`** — the v1 spec (preserved). v2's structure is informed by v1's 6-phase plan + 5-source framing + 3-action output.
26. **`conductor/tracks/code_path_audit_20260607/plan.md`** — the v1 plan (preserved, never executed). v2's plan is a fresh write.
### Gaps to Fill (This Track's Scope)
- A `ProducerConsumerGraph` builder for all of `src/` (3 AST passes: P1 return types, P2 parameter types, P3 field access). Multi-aggregate, machine-readable output.
- An `AccessPatternDetector` (5 patterns: whole_struct, field_by_field, hot_cold_split, bulk_batched, mixed). Per-`(function, aggregate)` classification with per-aggregate dominance rule (25% threshold).
- A `CallFrequencyEstimator` (7 frequencies: hot, per_turn, per_discussion, per_request, cold, init, unknown). Entry-point-based heuristic + manual override file.
- A `DecompositionCost` heuristic per aggregate (4 directions: componentize, unify, hold, insufficient_data). The 5-step `recommended_direction` logic per §7.5.
- A `MemoryDim` classifier per aggregate (7 dims: curation, discussion, rag, knowledge, config, control, unknown). Canonical mappings + file-of-origin heuristic + override.
- A per-aggregate profile data model (`AggregateProfile` + 9 supporting dataclasses + 5 enums: `AggregateKind`, `MemoryDim`, `AccessPattern`, `Frequency`, `RecommendedDirection`). All `frozen=True` per the immutability story. The 9 supporting dataclasses: `FunctionRef`, `AccessPatternEvidence`, `FrequencyEvidence`, `ResultCoverage`, `TypeAliasCoverage`, `CrossAuditFinding`, `CrossAuditFindings`, `DecompositionCost`, `OptimizationCandidate`.
- A cross-audit integration layer that consumes the 6 input JSON streams and produces per-aggregate `cross_audit_findings` + 2 coverage metrics (`result_coverage`, `type_alias_coverage`).
- The v2 postfix DSL (14 new tagged words + the v1's 7 preserved). The flat-section format (streamable, tag-scannable).
- Output: per-aggregate `.dsl` + `.md` + `.tree` files + 4 top-level rollup files (summary.md, cross_audit_summary.md, decomposition_matrix.md, candidates.md).
- A CLI (`python scripts/code_path_audit/code_path_audit.py --all --date <date>`) and an MCP tool (`code_path_audit_v2(action=None) -> dict`).
- A meta-audit (`scripts/audit_code_path_audit_coverage.py`) that validates the v2 audit's output schema.
- The actual audit run on the 13 aggregates, with the report committed to `docs/reports/code_path_audit/<date>/`.
- A new styleguide (`conductor/code_styleguides/code_path_audit.md`) documenting the v2 audit's contract.
- A 1-line extension to `scripts/audit_optional_in_3_files.py` to include `scripts/code_path_audit/code_path_audit.py` in the baseline.
---
## Goals
1. **Produce a queryable artifact per aggregate.** The custom postfix `.dsl` output is the source of truth; markdown + prefix tree text are for human review. Re-run after any `src/` change to see drift.
2. **Cross-validate the 2 foundational conventions.** Per-aggregate `result_coverage` (the `data_oriented_error_handling` cross-check) + per-aggregate `type_alias_coverage` (the `data_structure_strengthening` cross-check). The verdict at the top of `summary.md` says "VERIFIED" or "DRIFT DETECTED" with the specific evidence.
3. **Surface the top-N decomposition candidates per aggregate.** The `decomposition_matrix.md` ranks candidates by `estimated_savings_us × frequency_multiplier`. This is what the user uses to decide which refactor track to do next.
4. **Data-grounded design.** The audit's data structure is the spec; the heuristics and the threshold are module-level constants tunable from one place (`scripts/code_path_audit_overrides.toml`).
5. **Reusable across aggregates.** The `build_pcg` + `classify_memory_dim` + `detect_access_pattern` + `estimate_call_frequency` + `compute_decomposition_cost` APIs take any aggregate (or "all 13"). Adding a 14th aggregate is 1 line in the `AGGREGATES` constant.
6. **Surface calibration gaps clearly.** When the static heuristic can't resolve a call (C-extension, decorator-driven dispatch, `getattr` magic), the report flags it as "unresolved" so the `pipeline_runtime_profiling_20260607` follow-up targets it.
7. **Tolerate the candidate aggregates' absence.** The 3 candidate aggregates (`ChatMessage`, `ToolSpec`, `ProviderHistory`) are NOT on master. The v2 audit produces placeholders with `is_candidate: True`; the report is still valid (the placeholders are clearly marked).
---
## Functional Requirements
The 11 public functions in `scripts/code_path_audit/code_path_audit.py`. All return `Result[T]` per the `error_handling.md` hard rule (or return a deterministic `T` when no runtime failure is possible).
| # | Function | Returns | Failure mode |
|---|---|---|---|
| 1 | `run_audit(src_dir, audit_inputs_dir, output_dir, date)` | `Result[AuditSummary]` | 6 input JSONs may be missing or malformed; src/ may be unparseable |
| 2 | `build_pcg(src_dir)` | `Result[ProducerConsumerGraph]` | AST parse errors in src/ |
| 3 | `classify_memory_dim(aggregate, type_registry)` | `MemoryDim` | n/a (deterministic) |
| 4 | `detect_access_pattern(function_body, aggregate)` | `AccessPattern` | n/a (deterministic) |
| 5 | `estimate_call_frequency(function, call_graph)` | `Frequency` | n/a (deterministic) |
| 6 | `compute_decomposition_cost(profile)` | `DecompositionCost` | n/a (deterministic) |
| 7 | `read_input_json(path)` | `Result[dict]` | file not found; malformed JSON |
| 8 | `to_dsl_v2(profile)` | `str` | n/a (deterministic) |
| 9 | `parse_dsl_v2(text)` | `Result[dict]` | malformed DSL |
| 10 | `to_markdown(profile)` | `str` | n/a (deterministic) |
| 11 | `to_tree(profile)` | `str` | n/a (deterministic) |
Plus the CLI (`python scripts/code_path_audit/code_path_audit.py ...`) and the MCP tool (`code_path_audit_v2`).
---
## Non-Functional Requirements
- **No new pip dependencies.** The v2 audit uses stdlib only (`ast`, `pathlib`, `json`, `dataclasses`, `tomllib` for the override file).
- **1-space indentation** for all Python code (per `conductor/workflow.md`).
- **CRLF line endings** on Windows.
- **Type hints required** for all public functions.
- **No comments in Python source** (documentation lives in `/docs`).
- **`Result[T]` return types** for all functions that can fail at runtime (per the `error_handling.md` hard rule). The new file is held to the same standard as the 3 refactored files.
- **`Optional[T]` return types are FORBIDDEN** in `scripts/code_path_audit/code_path_audit.py`. Verified by the extended `scripts/audit_optional_in_3_files.py` (1-line extension).
- **Per-task commits** (1 task = 1 commit). Per `conductor/workflow.md` TDD protocol.
- **Per-task git notes** (each commit gets a `git notes add -m "..."` summary).
- **Coverage target: >80%** for `scripts/code_path_audit/code_path_audit.py`. The 4 audit scripts (`audit_exception_handling.py --strict`, `audit_weak_types.py --strict`, `audit_main_thread_imports.py`, `audit_no_models_config_io.py`) are the verification gates.
- **The audit's runtime is bounded.** The full audit run against the real `src/` (65 files) completes in <60s on a developer machine. The unit + integration tests complete in <30s. The live_gui E2E tests are opt-in.
---
## Architecture
### 7.1 Public API (the 11 functions)
#### 7.1.1 `run_audit(...)`
The main entry point. Runs the full audit pipeline:
1. Read the 6 input JSON files from `audit_inputs_dir` (using `read_input_json` per function #7). Missing files are tolerated; the corresponding `cross_audit_findings` field is `()` and the markdown notes the absence.
2. Build the PCG (using `build_pcg` per function #2).
3. For each of the 13 aggregates, build the `AggregateProfile`:
- `classify_memory_dim(aggregate, type_registry)` (function #3)
- `detect_access_pattern(consumer, aggregate)` (function #4) for each consumer; aggregate to the per-aggregate pattern
- `estimate_call_frequency(function, call_graph)` (function #5) for each producer + consumer; aggregate to the per-aggregate frequency
- Cross-validate with the 6 input JSONs (compute `cross_audit_findings`, `result_coverage`, `type_alias_coverage`)
- `compute_decomposition_cost(profile)` (function #6)
- Synthesize `optimization_candidates` from the cross-audit findings + the decomposition cost
4. Render the 13 per-aggregate `.dsl` + `.md` + `.tree` files.
5. Render the 4 top-level rollup files (`summary.md`, `cross_audit_summary.md`, `decomposition_matrix.md`, `candidates.md`).
6. Return `Result[AuditSummary]` with the per-aggregate profiles + the rollup paths.
#### 7.1.2 The other 10 functions
Per the table in §"Functional Requirements." The deterministic functions (3, 4, 5, 6, 8, 10, 11) take already-parsed data and return data; no I/O. The boundary functions (1, 2, 7, 9) catch stdlib I/O + AST parse errors and convert to `ErrorInfo` per `error_handling.md` Pattern 2.
### 7.2 The 4 static analyses (PCG, MemoryDim, APD, CFE)
#### 7.2.1 `ProducerConsumerGraph` (PCG) — pipeline discovery
**Three AST passes over `src/`:**
| Pass | What it finds | Output |
|---|---|---|
| **P1: Return types** | `FunctionDef.returns` annotation -> `Result[T]` -> producer of `T`; or direct `T` (alias or dataclass) -> producer of `T`. | `(function, aggregate, "producer", confidence="high")` edges |
| **P2: Parameter types** | `FunctionDef.args` annotation -> parameter is a TypeAlias or dataclass -> consumer of that aggregate. `dict[str, Any]` parameter is NOT a consumer edge (typed by P3). | `(function, aggregate, "consumer", confidence="high")` edges |
| **P3: Field access** | Every `payload['key']` and `payload.attr` in the function body. The audit consults `scripts/generate_type_registry.py --json` to map `key` to a known field of a known aggregate. If `key` is unique to one aggregate (e.g., `'vision'` -> `VendorCapabilities`), the consumer edge is high-confidence. If `key` is ambiguous (e.g., `'path'` appears in both `FileItem` and `ContextPreset`), the edge is low-confidence and the markdown flags it. | `(function, aggregate, "consumer", confidence=...)` edges |
**Edge cases the algorithm handles:**
- **Constructor calls** (`dict(...)`, `SomeDataclass(...)`, `SomeNamedTuple(...)`) inside a function body: the function is a producer at the call site. The audit tracks the call's `type` argument (`dict`, `SomeDataclass`) to identify the aggregate.
- **Re-exports** (`from src.type_aliases import Metadata`): the audit uses `import` resolution to find the canonical TypeAlias definition, not the re-exported name.
- **Decorator-wrapped methods** (e.g., `@imscope`): the audit walks through the decorator; if the decorator is a known passthrough (per `scripts/code_path_audit_overrides.toml`), the method body is processed normally. If unknown, the function is marked "unresolved" and the markdown notes it (matches the v1 spec's `unresolved_calls` behavior).
- **Re-exports across sub-MCPs** (`mcp_client.py` re-exports `mcp_file_io.read_file_result`): the audit uses the **definition** site, not the re-export site, for the producer. The re-export site gets a "passthrough" `FunctionRef` with `role="consumer"`.
**Output:** A bipartite graph keyed by `(function_fqname, aggregate_name)` -> `FunctionRef` + role.
#### 7.2.2 `MemoryDim` classifier
A function `classify_memory_dim(aggregate_name, producer_functions, type_registry) -> MemoryDim` that consults:
1. **Canonical mappings** (hardcoded in `code_path_audit.py`):
- `Metadata`, `CommsLogEntry`, `CommsLog`, `HistoryMessage`, `History` -> `discussion` (per-turn conversational)
- `FileItem`, `FileItems` -> `curation` (per-file structural)
- `ToolDefinition`, `ToolCall` -> `control` (these propagate through the LLM-tool pipeline)
- `Result`, `ErrorInfo` -> `control` (propagation primitives)
2. **File-of-origin heuristic:** if the aggregate's primary producer is in `src/aggregate.py`, `src/context_presets.py`, `src/views.py` -> `curation`. If in `src/ai_client.py`, `src/history.py`, `src/app_controller.py` (in the discussion-handling sections) -> `discussion`. If in `src/rag_engine.py` -> `rag`. If in `src/knowledge*.py` (if exists) -> `knowledge`. If in `src/paths.py`, `src/presets.py`, `src/personas.py` -> `config`.
3. **Override file:** `scripts/code_path_audit_overrides.toml` with `[memory_dim.<aggregate>] = "<dim>"` for cases the heuristic gets wrong.
**When the classifier can't determine:** the result is `"unknown"` and the markdown flags it for human review (the override file is the fix).
#### 7.2.3 `AccessPatternDetector` (APD) — per-`(function, aggregate)` access pattern
For each `(function, aggregate)` pair:
1. Walk the function body. Record every `payload['key']` / `payload.attr` access into a `Counter[str]` keyed by `key`.
2. Detect these patterns:
- `whole_struct`: the function reads `payload` directly (passes to another function; `print(payload)`; `return payload`) OR accesses <=1 distinct key.
- `field_by_field`: the function accesses >=3 distinct keys AND no `whole_struct` access in the body.
- `hot_cold_split`: the function accesses 1-2 keys in the function's hot path (the top-level statement body) AND 2+ additional keys inside `if/else` branches.
- `bulk_batched`: the function is `for x in payload_list: <op>` where `payload_list: list[aggregate]` and the body accesses fields uniformly across iterations.
- `mixed`: none of the above patterns dominate (each pattern has <60% share of the function's accesses).
3. Aggregate the per-function patterns to the aggregate level: the dominant pattern across all consumers, with the rule that the dominant pattern must have >=25% share of consumers. If no pattern has >=25%, the aggregate-level result is `mixed`.
**The threshold constants** are module-level in `code_path_audit.py`:
```python
WHOLE_STRUCT_KEY_THRESHOLD: int = 1
FIELD_BY_FIELD_KEY_THRESHOLD: int = 3
MIXED_DOMINANCE_THRESHOLD: float = 0.6
AGGREGATE_LEVEL_DOMINANCE_THRESHOLD: float = 0.25
```
The override file can change them per-aggregate.
#### 7.2.4 `CallFrequencyEstimator` (CFE) — per-function frequency
Build the v1 call graph. For each function:
1. **Entry point detection** (AST-based):
- Functions called from `__init__` of `App` (in `src/gui_2.py`) or `AppController` (in `src/app_controller.py`) or from `main()` (in `gui.py`) -> `init`.
- Functions called from the ImGui render loop (`render_*` functions, or functions called within `if imgui.begin_main_tool_bar():` etc.) -> `hot`.
- Functions called from the AI send path (`_send_<provider>_result`, `process_user_request`) -> `per_turn`.
- Functions called from `reset_session`, `cleanup`, `_classify_*_error` -> `cold`.
- Functions called from `save_project`, `load_project`, `save_snapshot` -> `per_discussion`.
- Functions called from `_api_*` FastAPI handlers -> `per_request`.
2. **Override file:** `scripts/code_path_audit_overrides.toml` with `[frequency.<function_fqname>] = "<freq>"` for manual corrections.
3. **Aggregate level:** the dominant frequency across all producers+consumers, with `unknown` if no dominant.
### 7.3 The 6 input streams
The v2 audit consumes JSON from 6 sources. All 6 are in `tests/artifacts/audit_inputs/` (gitignored per `test_sandbox.md`):
| Input | Path | Producer | Shape (essential fields) |
|---|---|---|---|
| 1 | `audit_weak_types.json` | `scripts/audit_weak_types.py --json` | `{"findings": [{"file", "line", "type_string", "category"}]}` |
| 2 | `audit_exception_handling.json` | `scripts/audit_exception_handling.py --json` | `{"findings": [{"file", "line", "category", "function", "class", "body_summary"}]}` |
| 3 | `audit_optional_in_3_files.json` | `scripts/audit_optional_in_3_files.py --json` | `{"findings": [{"file", "line", "return_type", "function"}]}` (3 baseline files only) |
| 4 | `audit_no_models_config_io.json` | `scripts/audit_no_models_config_io.py --json` | `{"findings": [{"file", "line", "function", "config_path"}]}` |
| 5 | `audit_main_thread_imports.json` | `scripts/audit_main_thread_imports.py --json` | `{"findings": [{"file", "line", "imported_module", "thread"}]}` |
| 6 | `type_registry.json` | `scripts/generate_type_registry.py --json` | `{"types": {"<aggregate_name>": {"file", "fields": [{"name", "type", "optional"}]}}}` |
**Tolerance:** if any input is missing or malformed, the audit continues with the corresponding `cross_audit_findings` field set to `()` (empty tuple) and the markdown notes the missing input. The audit does NOT fail on missing inputs.
### 7.4 The 13 data aggregates (10 + 3 candidates)
The 10 in-scope aggregates are the canonical TypeAliases from `src/type_aliases.py`:
```
1. Metadata (the root alias; 79 sites in src/ai_client.py alone)
2. FileItem (single file in context)
3. FileItems (list of files in context; the most common weak pattern)
4. CommsLogEntry (single entry in AI comms log)
5. CommsLog (the comms log ring buffer)
6. HistoryMessage (single message in provider history; UI layer)
7. History (the conversation history)
8. ToolDefinition (single tool definition)
9. ToolCall (single tool call from the model)
10. Result[T] (the success-or-failure wrapper; the audit's coverage metric)
```
The 3 candidate aggregates are from `any_type_componentization_20260621` §3 (NOT on master; the v2 audit is forward-compatible with their absence):
```
11. ToolSpec / ToolParameter (would replace ToolDefinition's 45 dict instances; §3.1)
12. ChatMessage / UsageStats / NormalizedResponse (would replace HistoryMessage + tool-call dicts; §3.2)
13. ProviderHistory (would replace the 7 per-provider history lists + locks; §3.3 + PHASE3_HYPOTHETICAL_PROMOTION)
```
When the candidate is absent (the master state), the v2 audit produces a placeholder with `is_candidate: True` and all metrics set to 0. The `candidates.md` rollup explains the placeholder status.
### 7.5 The decomposition cost formula
**Constants (module-level, tunable):**
```python
MICROSECOND_BUDGET_PER_LLM_TURN: int = 50_000 # per a real Anthropic Sonnet call's worth of work
BRANCH_DISPATCH_OVERHEAD_US: int = 100 # cost per if/else branch decision on a struct field
ALLOCATION_OVERHEAD_US: int = 50 # cost per SomeDataclass(...) construction
DEAD_FIELD_COST_PER_FIELD_US: int = 10 # wasted allocation per unused field
COMPONENTIZATION_INDIRECTION_US: int = 200 # cost of splitting a hot struct into 2
UNIFICATION_INDIRECTION_US: int = 300 # cost of merging 2 hot structs into 1
```
**Per-call cost formula:**
```
per_call_cost_us =
(struct_field_count * ALLOCATION_OVERHEAD_US)
+ (max(fields_accessed_in_hot_path, 1) * BRANCH_DISPATCH_OVERHEAD_US)
+ (struct_frozen ? 20 : 0)
```
**Current total cost** (per unit of frequency):
```
current_total_us = per_call_cost_us * frequency_multiplier
where frequency_multiplier is:
hot = 60 (60 fps)
per_turn = 1
per_request = 1
per_discussion = 1
cold = 0.01
init = 0.001
unknown = 0 (no estimate; mark insufficient_data)
```
**Componentize savings formula:**
```
componentize_savings_us = current_total_us * componentize_factor
where componentize_factor is:
if access_pattern == "field_by_field" and struct_field_count > 10 and not struct_frozen:
componentize_factor = 0.30
elif access_pattern == "hot_cold_split" and hot_field_count <= 2 and struct_field_count > 5:
componentize_factor = 0.40
elif access_pattern == "whole_struct" or access_pattern == "bulk_batched":
componentize_factor = -0.20
elif access_pattern == "mixed":
componentize_factor = 0
else:
componentize_factor = -0.10
```
**Unify savings formula:**
```
unify_savings_us = current_total_us * unify_factor
where unify_factor is:
if access_pattern == "bulk_batched" and struct_field_count <= 3 and struct_frozen:
unify_factor = 0.25
elif access_pattern == "whole_struct" and struct_field_count <= 5 and struct_frozen:
unify_factor = 0.15
elif access_pattern == "field_by_field":
unify_factor = -0.30
elif access_pattern == "hot_cold_split":
unify_factor = -0.10
elif access_pattern == "mixed":
unify_factor = 0
else:
unify_factor = 0.05
```
**`recommended_direction` logic:**
```
if access_pattern == "field_by_field" and struct_field_count > 10:
-> "componentize" (rationale cites the dead-field count)
elif access_pattern == "hot_cold_split" and hot_field_count <= 2:
-> "componentize" (split into hot + cold structs)
elif access_pattern == "bulk_batched" and struct_field_count <= 3:
-> "unify" (small struct; wider bulk path is fine)
elif access_pattern == "whole_struct" and struct_field_count <= 5:
-> "unify" (small struct; less dispatch overhead)
elif access_pattern == "mixed" or frequency == "unknown":
-> "insufficient_data" (recommend runtime profiling per pipeline)
elif struct_frozen and access_pattern == "whole_struct":
-> "hold" (frozen + whole_struct is the ideal shape)
else:
-> "hold"
```
**The auto-generated rationale string:**
```
"<aggregate_name>: access_pattern=<pattern>, frequency=<freq>, struct_field_count=<N>, struct_frozen=<bool>.
Recommended: <direction> because <one-sentence justification>. Estimated savings: <X>us per <freq unit>."
```
The Tier 2 Tech Lead can override the rationale per-aggregate in `scripts/code_path_audit_overrides.toml`.
---
## Output Format
### 8.1 The 13 per-aggregate files (DSL + markdown + tree)
For each aggregate:
**`*.dsl`** — the postfix DSL (flat sections, streamable, tag-scannable). The canonical artifact.
**`*.md`** — human-readable markdown, 10 sections (Header, Pipeline summary, Access pattern, Frequency, Result coverage, Type alias coverage, Cross-audit findings, Decomposition cost, Optimization candidates, Verdict).
**`*.tree`** — prefix tree text view (box-drawing, recursive walker). Compact, scannable.
### 8.2 The 4 top-level rollups
**`summary.md`** — the 30-second view + the 4-mem-dim rollup + the verdict (the "VERIFIED" or "DRIFT DETECTED" line).
**`cross_audit_summary.md`** — the per-aggregate cross-audit hits table (5 columns, one per input audit script) + the top-5 follow-up candidates + the cross-validation verdict.
**`decomposition_matrix.md`** — the ranked list of optimization candidates across all aggregates, sorted by `estimated_savings_us * frequency_multiplier`. The "what should we do next" view.
**`candidates.md`** — the 3 candidate aggregates (forward-compat placeholders). Explains the placeholder status.
### 8.3 The v1 artifacts (preserved for backward compat)
- `docs/reports/code_path_audit/<date>/call_graph.dsl` — the v1 full call graph.
- `docs/reports/code_path_audit/<date>/actions/ai_message_lifecycle.{dsl,md,mmd}` — the v1 per-action reports, downgraded to "cross-references to the per-aggregate profiles."
### 8.4 The audit_inputs/ dir (gitignored)
The 6 input JSON files consumed (for reproducibility; same dir name as `tests/artifacts/audit_inputs/` per `test_sandbox.md`).
---
## Verification (10-phase TDD test plan)
Per `conductor/workflow.md` TDD red-first protocol. Each phase has 1 setup commit + N test commits + 1 refactor commit.
| Phase | What | Test count | Audit gate |
|---|---:|---:|---|
| 1. Data model | `AggregateProfile` + 9 supporting dataclasses + 5 enums (per §7.1 / §7.2) | 10 | n/a |
| 2. PCG (P1+P2+P3) | The 3 AST passes; producer/consumer edges | 7 | `audit_main_thread_imports.py` |
| 3. APD | The 5 access patterns + the 25% dominance rule | 6 | n/a |
| 4. CFE | The 6 entry-point detectors + the override file | 6 | n/a |
| 5. Decomposition cost | The 4-direction logic + the auto-generated rationale | 6 | n/a |
| 6. Cross-audit integration | The 6 input JSON contracts + the 3-tier mapping | 7 | `audit_weak_types.py --strict` |
| 7. v2 DSL | The 14 new tagged words + the round-trip + backward compat | 5 | n/a |
| 8. Markdown / tree renderers | The 10 markdown sections + the box-drawing tree | 4 | n/a |
| 9. Integration tests | The synthetic src/ fixture + the real src/ run | 7 | All 4 audit scripts pass `--strict` |
| 10. Live_gui E2E (opt-in) | The MCP tool via the `live_gui` fixture | 2 | All 4 audit scripts pass `--strict` |
**Total: 60 unit tests + 7 integration tests + 2 live_gui tests = 69 tests.**
### 9.1 The synthetic src/ fixture
`tests/fixtures/synthetic_src/` — 6 files defining 3 aggregates (`Metadata`, `FileItems`, `History`) + 6 functions (2 producers, 4 consumers). The integration tests assert the exact expected profiles.
### 9.2 The 6 input JSON fixture
`tests/fixtures/audit_inputs/` — 6 JSON files matching the contracts in §7.3. The integration tests assert the cross-audit mapping, the `result_coverage` + `type_alias_coverage` formulas, and the tolerance for missing inputs.
### 9.3 Pre-commit verification
```bash
uv run pytest tests/test_code_path_audit.py -q
uv run python scripts/audit_exception_handling.py --strict
uv run python scripts/audit_weak_types.py --strict
uv run python scripts/audit_main_thread_imports.py
uv run python scripts/audit_no_models_config_io.py
```
### 9.4 End-of-track verification
```bash
uv run python scripts/code_path_audit/code_path_audit.py --all --date 2026-06-22
uv run python scripts/audit_exception_handling.py --strict
uv run python scripts/audit_weak_types.py --strict
uv run python scripts/audit_main_thread_imports.py
uv run python scripts/audit_no_models_config_io.py
uv run python scripts/generate_type_registry.py --check
uv run pytest tests/test_code_path_audit_live_gui.py -v
```
### 9.5 Manual verification (per `conductor/workflow.md`)
The Tier 2 Tech Lead + user review the `docs/reports/code_path_audit/<date>/summary.md` to confirm:
- The 4-mem-dim rollup is correct
- The cross-audit verdict is accurate
- The decomposition_matrix.md rankings match the user's intuition
- The 3 candidate aggregates are properly marked as placeholders
---
## Out of Scope (per §7.2)
- **No modifications to existing `src/*.py` files** (read-only on the 65 existing files; the v2 audit doesn't change them).
- **No modifications to the 5 existing audit scripts** (consume their JSON; don't change them).
- **No runtime profiling.** Deferred to `pipeline_runtime_profiling_20260607` (preserved from the v1 spec's follow-up list).
- **No new pip dependencies.** The v2 audit uses stdlib only.
- **No changes to `data_structure_strengthening_20260606` or `data_oriented_error_handling_20260606` styleguides.**
- **No changes to the v1 `spec.md` and `plan.md`** (they stay as v1).
- **No MMA worker spawn action** (preserved from v1; the user's "keeping MMA cold" directive from 2026-06-07 still stands).
- **No new modules in `src/` other than `code_path_audit.py`** (per the file size + naming convention in AGENTS.md).
- **The 23 lower-impact files** (those with 1-9 weak-type sites each) are deferred.
- **The 3 candidate aggregates' "real" analysis** is deferred (the v2 audit produces placeholders; the real profiles arrive after `any_type_componentization_20260621` merges).
- **The v1-style per-action output** is preserved for backward compat but downgraded to "cross-references to the per-aggregate profiles."
---
## Risks (per §7.3)
| Risk | Likelihood | Impact | Mitigation |
|---|---|---|---|
| The decomposition-cost heuristic is inaccurate (componentize_savings overestimate or underestimate) | Medium | Medium (false-positive optimization candidates) | Runtime-profiling follow-up recalibrates. The override file adjusts per-aggregate. |
| The PCG misses dynamic patterns (`eval`, `getattr`, decorator-driven dispatch) | Medium | Low (affected functions marked "unresolved") | The override file lists known passthroughs. Runtime-profiling follow-up catches unresolved. |
| The 6 input JSON contracts drift (the existing audit scripts evolve without bumping the v2 audit's contract) | Medium | Low (the v2 audit tolerates missing fields; the schema validator catches drift) | The `audit_code_path_audit_coverage.py` meta-audit runs in CI; fails on schema drift. |
| The candidate aggregates don't merge (`any_type_componentization_20260621` is delayed) | Low | Low (the placeholders are still there; the report still produces) | The v2 audit is forward-compatible. The `is_candidate: bool` flag handles absence. |
| The v1 .dsl files don't round-trip (the v2 parser is more strict than v1) | Low | Medium (the v1 action reports are broken) | The v2 parser is a **superset** of v1; the v1 action reports still parse. The `test_v2_dsl_backward_compat_v1` test verifies. |
| The 60+7+2 = 69 tests is too long-running for the per-PR CI gate | Low | Low (AST walks are sub-second; live_gui tests are opt-in) | Unit + integration tests <30s. Live_gui tests opt-in via env var. |
| The synthetic src/ fixture diverges from real src/ (the test expectations don't generalize) | Medium | Low (the integration tests catch real bugs separately) | The integration test layer runs against real src/ as well as the synthetic fixture. |
| The v2 audit is run against `master` without `any_type_componentization_20260621` merged, so the candidate placeholders pollute the report | Low | Low (the placeholders are clearly marked) | The `is_candidate: bool` flag is visible in every output. The `summary.md` has a section explaining placeholder status. |
| The decomposition-matrix savings estimates are misinterpreted as "ground truth" (they're heuristic) | Medium | Low (the user might over-prioritize) | The `summary.md` and `decomposition_matrix.md` headers caveat: "Savings estimates are heuristic (calibrated by `pipeline_runtime_profiling_20260607`); use as ranking input, not as actual savings." |
| The 4 mem dim classification is wrong for some aggregates (the file-of-origin heuristic misroutes) | Medium | Low (the misrouted aggregate shows up in the wrong dim's rollup) | The `MemoryDim` is overridable in `scripts/code_path_audit_overrides.toml`. The markdown flags the override. |
---
## Coordination with Pending Tracks
| Track | Status (2026-06-22) | Relationship to v2 |
|---|---|---|
| `any_type_componentization_20260621` | NOT on master (merged `f914b2bc`, reverted `751b94d4`); spec + plan in `conductor/tracks/any_type_componentization_20260621/` | The 3 candidate aggregates (`ToolSpec`, `ChatMessage`, `ProviderHistory`) are sourced from this track's `ANY_TYPE_AUDIT_20260621.md` §3. The v2 audit's `candidates.md` rollup documents the forward-compat. When this track merges, the v2 audit is re-run; the placeholders become real profiles. |
| `phase2_4_5_call_site_completion_20260621` | NOT on master (same merge+revert history as `any_type_componentization_20260621`); spec + plan + TRACK_COMPLETION report in `conductor/tracks/phase2_4_5_call_site_completion_20260621/` | The `PHASE3_HYPOTHETICAL_PROMOTION.md` (authored by Tier 2; the authoritative Phase 3 cost hypothesis) is the source of the v2's `ProviderHistory` candidate aggregate's expected cost. The v2 audit's `candidates.md` cites this report. |
| `data_oriented_error_handling_20260606` | SHIPPED (in master) | The v2 audit's `result_coverage` metric is the cross-check. The `error_handling.md` styleguide is the v2 audit's source of truth for the `Result[T]` return types. |
| `data_structure_strengthening_20260606` | SHIPPED (in master) | The v2 audit's `type_alias_coverage` metric is the cross-check. The `type_aliases.md` styleguide + the 10 TypeAliases are the v2 audit's source of truth. |
| `result_migration_cruft_removal_20260620` | SHIPPED (in master) | The `RESULT_MIGRATION_CAMPAIGN_STATUS_20260619.md` confirms the 100% complete state. The v2 audit's `result_coverage` reports on this final state. |
| `public_api_migration_and_ui_polish_20260615` | SHIPPED (in master) | `ai_client.send_result()` is the canonical public API. The v2 audit's `Metadata` aggregate's `result_coverage` reports on the post-migration state. |
| `nagent_review_20260608` (v3.1) | ACTIVE (in master; v3.1 is the latest at `7e61dd7d`) | The v2 audit references Candidates 27-30 (Markdown + custom DSL lock-in, per-turn ground-truth hook, dataset-curation track, cache TTL GUI hardening). The v2's custom postfix DSL is a direct application of Candidate 27. |
| `exception_handling_audit_20260616` | SHIPPED (in master) | The 211-site audit (`EXCEPTION_HANDLING_AUDIT_20260616.md`) is the precedent for the v2 audit's structure (audit -> migration plan -> sub-tracks). |
| `tier2_leak_prevention_20260620` | SHIPPED (in master) | The v2 audit's Tier 2 execution follows the `tier2_leak_prevention` conventions (no `git push*`, no `git checkout*`, etc.). |
**This audit has no blockers** and **no conflicts**. It can ship independently of the 5 active planned tracks. It enables future refactors (the 3 high-priority `componentize` candidates).
---
## Follow-up (per §7.4)
| # | Track | When | Purpose |
|---|---|---|---|
| 1 | `pipeline_runtime_profiling_20260607` | After v2 ships | Calibrate the v2's heuristic cost constants against real measurements. Uses `src/performance_monitor.py`. The v2 spec's `MICROSECOND_BUDGET_PER_LLM_TURN`, `BRANCH_DISPATCH_OVERHEAD_US`, `ALLOCATION_OVERHEAD_US`, `DEAD_FIELD_COST_PER_FIELD_US`, `COMPONENTIZATION_INDIRECTION_US`, `UNIFICATION_INDIRECTION_US` are recalibrated by this track. |
| 2 | `data_pipelines_inventory_<date>` | After v2 ships | Per-pipeline (vs per-aggregate) reports for the top 5 pipelines. Complements the v2 with the pipeline view. The v2's `decomposition_matrix.md` is the input. |
| 3 | `code_path_audit_in_ci_<date>` | After v2 ships | Run v2 in CI on every PR; fail on new untyped sites OR a high-priority decomposition-matrix regression. The "audit as CI gate" pattern. |
| 4 | `code_path_audit_data_oriented_refactor_<date>` | After v2 ships | Implement the 3 high-priority `componentize` candidates (FileItems, History, Metadata) per the v2 audit's `decomposition_matrix.md`. |
| 5 | `code_path_audit_v2_5_followup_<date>` | After `any_type_componentization_20260621` merges | Re-run v2; the 3 placeholders become real profiles; the decomposition-matrix gets 3 new rows. |
---
## See Also
### Styleguides
- `conductor/code_styleguides/data_oriented_design.md` — the canonical DOD reference (v2's decomposition-cost heuristic is informed by §2's 8 defaults)
- `conductor/code_styleguides/error_handling.md` — the `Result[T]` convention (v2's public API returns `Result[T]` per the hard rule)
- `conductor/code_styleguides/type_aliases.md` — the 10 TypeAliases + 1 NamedTuple (v2's 10 in-scope aggregates)
- `conductor/code_styleguides/agent_memory_dimensions.md` — the 4 mem dims (v2's `MemoryDim` classifier)
- `conductor/code_styleguides/feature_flags.md` — "delete to turn off" pattern (v2's `audit_code_path_audit_coverage.py` is a feature flag)
- `conductor/code_styleguides/cache_friendly_context.md` — stable-to-volatile context ordering (v2's per-aggregate reports are a downstream consumer of the cache state)
- `conductor/code_styleguides/knowledge_artifacts.md` — the knowledge harvest pattern (v2's per-aggregate profiles are NOT a knowledge artifact; they're curation)
- `conductor/code_styleguides/rag_integration_discipline.md` — the conservative-RAG rule (v2's `rag` aggregate classification)
- `conductor/code_styleguides/config_state_owner.md` — config I/O ownership (v2's `audit_no_models_config_io.json` is the cross-check)
### v1 spec + plan (preserved)
- `conductor/tracks/code_path_audit_20260607/spec.md` — the v1 spec (approved 2026-06-07; revised 2026-06-08 with post-4-tracks timing + 5-source framing)
- `conductor/tracks/code_path_audit_20260607/plan.md` — the v1 plan (preserved, never executed)
### Reports + ideation
- `docs/reports/computational_shapes_ssdl_digest_20260608.md` — the SSDL digest that informed the v1 spec's 5-source lens (v2 preserves the lens)
- `docs/reports/RESULT_MIGRATION_CAMPAIGN_STATUS_20260619.md` — the 100%-complete result migration campaign
- `docs/reports/ANY_TYPE_AUDIT_20260621.md` — the 89-site audit (48 promoted + 41 deferred) that informed `any_type_componentization_20260621` (v2's 3 candidate aggregates)
- `docs/reports/PHASE3_HYPOTHETICAL_PROMOTION.md` — the Tier 2's authoritative cost analysis of the 41 deferred Phase 3 sites
- `docs/reports/EXCEPTION_HANDLING_AUDIT_20260616.md` — the 211-site audit (precedent for v2's structure)
- `docs/reports/PLANNING_DIGEST_20260606.md` — the planning digest for the 5 foundational tracks
- `docs/ideation/ed_chunk_data_structures_20260523.md` — the chunk-based-data-structure ideation (referenced in v1 spec; v2's `bulk_batched` access pattern aligns)
### v3.1 nagent review (the latest framing)
- `conductor/tracks/nagent_review_20260608/nagent_review_v3_1_20260620.md` — the v3.1 thickened main review
- `conductor/tracks/nagent_review_20260608/nagent_takeaways_v3_1_20260620.md` — the v3.1 bridge + the 4 new candidates (27-30)
- `conductor/tracks/nagent_review_20260608/nagent_review_v3_20260619.md` — the v3 main review (preserved per user directive 2026-06-20)
### Source files (the v2 audit consumes)
- `src/type_aliases.py` — the 10 TypeAliases + 1 NamedTuple
- `src/result_types.py` — `Result[T]`, `ErrorInfo`, nil-sentinels
- `src/mcp_client.py:934-992` — `derive_code_path` (the v2's PCG is the multi-symbol superset)
- `src/performance_monitor.py` — runtime profiling (used by `pipeline_runtime_profiling_20260607` follow-up)
- `src/vendor_capabilities.py` — the canonical `frozen=True` dataclass + module-level registry pattern (template for the v2 audit's per-aggregate profile structure)
### Audit scripts (the v2 audit consumes)
- `scripts/audit_main_thread_imports.py` — import-graph CI gate
- `scripts/audit_weak_types.py` — weak-types CI gate
- `scripts/audit_exception_handling.py` — exception-handling CI gate
- `scripts/audit_optional_in_3_files.py` — `Optional[T]` ban CI gate (v2 extends this with 1 line)
- `scripts/audit_no_models_config_io.py` — config-I/O ownership CI gate
- `scripts/generate_type_registry.py` — type-registry generator
### Workflow + process
- `conductor/workflow.md` — TDD protocol + per-task commits + git notes + phase checkpoints + skip-marker policy
- `conductor/edit_workflow.md` — the edit-tool contract (the v2 audit uses `manual-slop_*` MCP tools per the project convention)
- `AGENTS.md` — canonical operating rules (the "no day estimates" rule, the "small files are propaganda" stance, the hard bans on `git restore` / `git checkout --`)
- `conductor/product-guidelines.md` — product-level conventions (1-space indent, 1 commit per task, type hints, etc.)
- `conductor/tech-stack.md` — tech stack constraints (Python 3.11+, imgui-bundle, FastAPI, etc.)
### Sibling tracks (the v2's relationship)
- `conductor/tracks/any_type_componentization_20260621/` — the 3 candidate aggregates' source
- `conductor/tracks/phase2_4_5_call_site_completion_20260621/` — the `PHASE3_HYPOTHETICAL_PROMOTION` source
- `conductor/tracks/data_oriented_error_handling_20260606/` — the `Result[T]` source
- `conductor/tracks/data_structure_strengthening_20260606/` — the TypeAlias source
- `conductor/tracks/result_migration_cruft_removal_20260620/` — the 100% complete result migration
---
## Revision History
**2026-06-24 — MVP pivot (follow-up: code_path_audit_polish_20260622).** The v2 spec described a 14-phase DSL implementation that never reached production. The actual shipped implementation is:
- **MVP output:** A single `AUDIT_REPORT.md` (6797 lines, 311KB) with `summary.md` as a TOC pointer. Per-aggregate markdowns via `to_markdown` + `to_tree` are produced.
- **DSL deprecated:** The v2 postfix DSL format (`to_dsl_v2` + `parse_dsl_v2`, `DSL_WORD_ARITY_V2`, `_atom`) was implemented but never produced. `run_audit()` writes `.md` files only. The DSL parser carried latent arity bugs (e.g. `DSL_WORD_ARITY_V2["result-coverage"] = 5` but `to_dsl_v2` emits 4 args). Removed in `code_path_audit_polish_20260622` Task 2.2 (commit `b385cd44`).
- **`compute_result_coverage` removed:** The function had a latent bug (`result_producers = total_producers` hardcoded to 100%). `synthesize_aggregate_profile` inlines its own `ResultCoverage(...)` construction. Removed in `code_path_audit_polish_20260622` Task 2.3 (commit `2561e4ea`).
- **Test count:** 125 (was 131 in the v2 spec; -6 tests deleted across polish Tasks 2.2 and 2.3).
- **Audit-gate state:** `audit_weak_types.py --strict` and `generate_type_registry.py --check` now pass (fixed in polish Phase 1). The 2 pre-existing violations (4 exception-handling + 7 Optional[T]) are documented as NG1/NG2 in the polish track's spec and explicitly out of scope.
**No changes** to the v2 spec's overall design intent, the 13 aggregates, the 4-direction decomposition cost, or the cross-audit integration. The MVP pivot is purely about the OUTPUT format (markdown instead of DSL) and code-smell cleanup; the analytical core (PCG, MemoryDim, APD, CFE, cross-audit) is unchanged.
---
**End of spec_v2.md.**
conductor/tracks/code_path_audit_20260607/state.toml
-64
View File
@@ -1,64 +0,0 @@
# Track state for code_path_audit_20260607
# v2 supersedes v1; spec_v2.md + plan_v2.md are the canonical artifacts
# (v1's spec.md + plan.md are preserved unchanged, never executed)
# Updated by Tier 2 Tech Lead as tasks complete
[meta]
track_id = "code_path_audit_20260607"
name = "Code Path & Data Pipeline Audit v2"
status = "completed"
current_phase = "complete"
last_updated = "2026-06-22 (updated 2026-06-24 by code_path_audit_polish_20260622)"
[parent]
# Independent track (not part of an umbrella)
[blocked_by]
# No blockers. The 5 foundational tracks (data_oriented_error_handling_20260606,
# data_structure_strengthening_20260606, mcp_architecture_refactor_20260606,
# qwen_llama_grok_integration_20260606, result_migration_20260616) are SHIPPED.
# The 2 candidate-related tracks (any_type_componentization_20260621,
# phase2_4_5_call_site_completion_20260621) are NOT on master; the v2 audit
# is tolerant of their absence (forward-compat placeholders).
[blocks]
# 5 follow-up tracks (see metadata.json follow_up_tracks)
[phases]
# 14 phases per plan_v2.md
phase_0 = { status = "completed", checkpointsha = "78c9d463", name = "Setup (state.toml, empty files, fixture dirs)" }
phase_1 = { status = "completed", checkpointsha = "ef207cf6", name = "Data model (5 enums + 9 supporting dataclasses + AggregateProfile)" }
phase_2 = { status = "completed", checkpointsha = "200396e4", name = "PCG (3 AST passes: P1 return types, P2 parameter types, P3 field access)" }
phase_3 = { status = "completed", checkpointsha = "c1d2f0e4", name = "MemoryDim classifier (canonical mappings + file-of-origin + override)" }
phase_4 = { status = "completed", checkpointsha = "c1d2f0e4", name = "APD (5 access patterns + 25% dominance rule)" }
phase_5 = { status = "completed", checkpointsha = "cca59668", name = "CFE (7 frequencies + entry-point detection + override file)" }
phase_6 = { status = "completed", checkpointsha = "cca59668", name = "Decomposition cost (4 directions + auto-generated rationale)" }
phase_7 = { status = "completed", checkpointsha = "e59334a3", name = "Cross-audit integration (6 input JSONs + 3-tier mapping)" }
phase_8 = { status = "completed", checkpointsha = "c8253847", name = "v2 DSL (14 new tagged words + flat-section format)" }
phase_9 = { status = "completed", checkpointsha = "c8253847", name = "run_audit() main entry + CLI + MCP tool" }
phase_10 = { status = "completed", checkpointsha = "0690dcef", name = "Integration tests (synthetic src/ + audit_inputs/ fixtures)" }
phase_11 = { status = "completed", checkpointsha = "0690dcef", name = "Live_gui E2E tests (opt-in via CODE_PATH_AUDIT_LIVE_GUI=1) - file created, 2 tests gated on env var" }
phase_12 = { status = "completed", checkpointsha = "db36495f", name = "Meta-audit + styleguide + audit_optional_in_3_files.py (CREATED from scratch, was missing on master)" }
phase_13 = { status = "completed", checkpointsha = "d46a71f7", name = "End-of-track report (commit f93421f8) + tracks.md update (commit d46a71f7)" }
[verification]
data_model_tests_passing = true
pcg_tests_passing = true
memory_dim_tests_passing = true
apd_tests_passing = true
cfe_tests_passing = true
decomposition_cost_tests_passing = true
cross_audit_integration_tests_passing = true
v2_dsl_tests_passing = true
renderers_tests_passing = true
integration_tests_passing = true
live_gui_tests_passing = false
meta_audit_passing = false
all_4_audit_gates_passing = true
type_registry_check_passing = true
audit_run_completed = true
summary_md_approved = false
optimization_candidates_md_approved = false
truncation_md_approved = false
track_completion_report_written = true
tracks_md_updated = true
conductor/tracks/code_path_audit_phase_2_20260624/metadata.json
-146
View File
@@ -1,146 +0,0 @@
{
"track_id": "code_path_audit_phase_2_20260624",
"name": "Code Path Audit Phase 2 (the actual followup)",
"created_date": "2026-06-24",
"branch": "master",
"depends_on": ["code_path_audit_20260607", "any_type_componentization_20260621"],
"blocks": [],
"scope": {
"new_files": [
"docs/reports/SSDL_CAMPAIGN_ABORTED_20260624.md",
"docs/reports/TRACK_COMPLETION_code_path_audit_phase_2_20260624.md"
],
"modified_files": [
"conductor/tracks/metadata_ssdl_defusing_20260624/state.toml",
"conductor/tracks/metadata_nil_sentinel_20260624/state.toml",
"conductor/tracks/metadata_generational_handle_20260624/state.toml",
"conductor/tracks/metadata_field_cache_20260624/state.toml",
"src/mcp_client.py (Phase 1: 4 sites; Phase 7: 2 sites)",
"src/ai_client.py (Phase 1: 3 sites; Phase 2: 5 sites; Phase 3: 14 globals + ~27 callers; Phase 7: 5 sites)",
"src/openai_compatible.py (Phase 2: ~12 sites)",
"src/openai_schemas.py (Phase 2: remove backward-compat __init__)",
"src/session_logger.py (Phase 4; Phase 6: 1 site)",
"src/log_pruner.py (Phase 4)",
"src/gui_2.py (Phase 4; Phase 5)",
"src/api_hooks.py (Phase 5: ~5-10 callers)",
"src/app_controller.py (Phase 5)",
"src/external_editor.py (Phase 6: 2 sites)",
"src/project_manager.py (Phase 6: 1 site)",
"tests/test_ai_client_tool_loop.py (Phase 2: 5 tests updated)",
"tests/test_ai_client_tool_loop_builder.py (Phase 2: 1 test)",
"tests/test_ai_client_tool_loop_send_func.py (Phase 2: 2 tests)",
"tests/test_ai_client_cli.py (Phase 2: 1 test)",
"tests/test_gemini_cli_integration.py + edge_cases + parity_regression.py (Phase 2: 3 tests)",
"conductor/tracks.md"
],
"deleted_files": [
"src/openai_schemas.py:NormalizedResponse custom __init__ (replaced with auto-generated)",
"src/ai_client.py:14 module globals (replaced with get_history(...))",
"src/mcp_client.py:MCP_TOOL_SPECS dict literal (~45 entries)"
]
},
"estimated_effort": {
"method": "scope (per workflow.md §Tier 1 Track Initialization Rules). NO day estimates.",
"step_0": "2 tasks: SSDL campaign abort (5 file changes + 1 post-mortem)",
"phase_1": "1 task: mcp_tool_specs call-site migration (8 sites)",
"phase_2": "1 task: openai_schemas call-site migration (17 sites + remove backward-compat __init__)",
"phase_3": "1 task: provider_state call-site migration (14 globals + ~27 callers)",
"phase_4": "1 task: log_registry Session migration (7 sites)",
"phase_5": "1 task: api_hooks WebSocketMessage migration (16 sites)",
"phase_6": "3 tasks: NG1 fixups (4 INTERNAL_OPTIONAL_RETURN violations)",
"phase_7": "1 task: NG2 fixups (7 Optional[T] return types)",
"phase_8": "1 task: re-audit + measure new effective-codepaths",
"phase_9": "1 task: 10 VCs + TRACK_COMPLETION + state + tracks.md"
},
"verification_criteria": [
"VC1: 3 surviving modules actually used by src/*.py (git grep >= 5 hits in src/, not just in plan/spec text)",
"VC2: 14 module globals in src/ai_client.py are gone",
"VC3: MCP_TOOL_SPECS dict literal in src/mcp_client.py is gone",
"VC4: usage_input_tokens= in src/ai_client.py is gone (the new UsageStats API is in use)",
"VC5: effective codepaths drops by >= 2 orders of magnitude (target: 4.014e+22 -> < 1e+20)",
"VC6: NG1 fixed: 0 INTERNAL_OPTIONAL_RETURN violations in audit_exception_handling.py (full src/)",
"VC7: NG2 fixed: 0 Optional[T] return-type violations in audit_optional_in_3_files.py --strict",
"VC8: all 6 audit gates pass --strict",
"VC9: 11/11 batched test tiers PASS",
"VC10: end-of-track report written with the new effective-codepaths number"
],
"known_issues": [],
"deferred_to_followup_tracks": [
{
"id": "deferred-rethrow-heuristic",
"title": "Add raise X from e heuristic to audit_exception_handling.py",
"description": "9 sites in baseline use the Re-Raise Pattern 1 (raise X from e) but are flagged as INTERNAL_RETHROW. Add a heuristic so they're recognized as compliant. Per result_migration_baseline_cleanup_20260620 §10 limitation #1.",
"track_status": "separate track (small)"
},
{
"id": "deferred-pipeline-runtime-profiling",
"title": "Replace static heuristic with real runtime profiling",
"description": "The 4.01e22 number (and the post-migration number) are static heuristic measurements. Runtime profiling would measure real codepath counts. Deferred from the original code_path_audit_20260607 follow-up list.",
"track_status": "separate track"
},
{
"id": "deferred-7-file-split-refactor",
"title": "Collapse src/code_path_audit*.py into 1 orchestrator",
"description": "Per AGENTS.md file naming convention. Was NG3 in code_path_audit_polish_20260622. Risks breaking the cross-audit wiring; deferred per user small-scope directive.",
"track_status": "separate track"
}
],
"regressions_and_pre_existing_failures": [
{
"id": "R-pre-1",
"title": "audit_weak_types.py --strict: 5-site regression vs baseline 112",
"scope": "src/code_path_audit*.py modules (post-polish)",
"remediation": "Addressed by Phase 2 of this track (the 48 call-site migrations reduce weak-type sites)"
},
{
"id": "R-pre-2",
"title": "audit_exception_handling.py --strict: 4 pre-existing INTERNAL_OPTIONAL_RETURN violations (NG1)",
"scope": "src/external_editor.py (2), src/session_logger.py (1), src/project_manager.py (1)",
"remediation": "Phase 6 of this track"
},
{
"id": "R-pre-3",
"title": "audit_optional_in_3_files.py --strict: 7 pre-existing Optional[T] return-type violations (NG2)",
"scope": "src/mcp_client.py:1285,1289 (2); src/ai_client.py:159,247,619,673,3115 (5)",
"remediation": "Phase 7 of this track"
}
],
"pre_existing_failures_remaining": [],
"risk_register": [
{
"id": "risk-1",
"description": "Phase 3 (provider_state) breaks concurrent send_result() calls from different threads",
"likelihood": "medium",
"impact": "tests/test_ai_client_result.py regression-guard tests fail; ai_client multi-vendor concurrency broken",
"mitigation": "Per-provider migration (5 commits, one per vendor) with regression-guard tests after each"
},
{
"id": "risk-2",
"description": "Phase 2 (openai_schemas) breaks 12 tests that depended on the backward-compat __init__",
"likelihood": "low",
"impact": "12 tests in test_ai_client_tool_loop*.py + test_ai_client_cli.py + test_gemini_cli_*.py fail",
"mitigation": "Update the 12 tests to use usage=UsageStats(...) in the same commit that removes the backward-compat __init__"
},
{
"id": "risk-3",
"description": "The 48 migrations produce a smaller drop than expected (e.g., 4.014e+22 -> 4.013e+22 instead of < 1e+20)",
"likelihood": "low",
"impact": "VC5 fails; the audit infrastructure may have a bug",
"mitigation": "The combinatoric explosion IS from dict[str, Any]; the migration eliminates the explosion. If the drop is smaller, the audit infrastructure has a separate bug."
},
{
"id": "risk-4",
"description": "Removing the 14 module globals requires updating 27 call sites in a way that introduces bugs",
"likelihood": "medium",
"impact": "9 send_* functions broken; ai_client tool loop tests fail",
"mitigation": "Per-provider migration (5 commits); tests/test_ai_client_result.py + per-vendor provider tests verify"
},
{
"id": "risk-5",
"description": "NG1 + NG2 migrations introduce regressions in 11 specific functions",
"likelihood": "medium",
"impact": "11 specific tests fail; the convention migration has subtle bugs",
"mitigation": "Per-function migration with behavioral test; verify with scripts/run_tests_batched.py after Phase 7 + 8"
}
]
}
conductor/tracks/code_path_audit_phase_2_20260624/plan.md
-270
View File
@@ -1,270 +0,0 @@
# Plan: code_path_audit_phase_2_20260624
10 phases, 13 tasks. Per-task atomic commits with git notes. TDD: each phase starts with the failing test, then implementation, then verification.
## Step 0: Abort the SSDL campaign (5 file changes, prerequisite)
Focus: Mark the failed SSDL campaign as cancelled before this track begins.
- [x] Task 0.1 [Tier 1's ca219163]: Mark umbrella + 3 children as cancelled.
- WHERE: `conductor/tracks/metadata_ssdl_defusing_20260624/state.toml`, `conductor/tracks/metadata_nil_sentinel_20260624/state.toml`, `conductor/tracks/metadata_generational_handle_20260624/state.toml`, `conductor/tracks/metadata_field_cache_20260624/state.toml`
- WHAT: Set `status = "cancelled"` in each. Set all phases `cancelled` in each.
- HOW: `manual-slop_edit_file` for each
- SAFETY: Do NOT delete the 4 spec/plan/metadata files; preserve for audit trail
- COMMIT: `conductor(campaign-abort): metadata_ssdl_defusing_20260624 - SSDL campaign cancelled (premise was wrong; 4.01e22 is from dict[str, Any] type-dispatch, not nil-checks)`
- GIT NOTE: 1 campaign aborted; salvage NIL_METADATA primitive + 5 tests; the actual fix is any_type_componentization_reapply (per code_path_audit_phase_2_20260624)
- [x] Task 0.2 [Tier 1's ca219163]: Write post-mortem.
- WHERE: `docs/reports/SSDL_CAMPAIGN_ABORTED_20260624.md` (NEW)
- WHAT: 1-page post-mortem documenting:
- The campaign's premise (6 nil-check functions in Metadata consumers)
- The verification that found 0 Metadata-typed nil-checks (the "6" was a static text string in `code_path_audit_gen.py:108`)
- The actual 73 nil-check functions across the codebase (most on `_gemini_client`, `path`, `adapter` — not Metadata)
- The 1 function Tier 2 migrated (`_build_files_section_from_items` in `src/aggregate.py`) was not actually a Metadata nil-check
- The budget gate (10% drop in `compute_effective_codepaths`) was mathematically near-impossible due to exponential dominance
- The real cause of 4.01e22: `dict[str, Any]` type-dispatch (123 `entry.get('key', default)` sites in Metadata consumers)
- The actual fix: `any_type_componentization_reapply_20260624` (this track)
- Salvage: `NIL_METADATA = {}` in `src/aggregate.py` + 5 tests in `tests/test_metadata_nil_sentinel.py` are kept as useful primitives
- HOW: Write the file
- COMMIT: `docs(reports): SSDL_CAMPAIGN_ABORTED_20260624 post-mortem`
## Phase 1: mcp_tool_specs call-site migration (1 task, ~2-3 commits)
Focus: Apply the 8 call-site migrations from parent plan §Phase 1.
- [x] Task 1.1 [68a2f3f3 + 03dd44c6]: Replace `MCP_TOOL_SPECS` dict + 4 `mcp_client` usages + 3 `ai_client` usages.
- WHERE: `src/mcp_client.py` (4 sites), `src/ai_client.py` (3 sites)
- WHAT:
- `src/mcp_client.py:1944`: `native_names = {t['name'] for t in MCP_TOOL_SPECS}` → `from src import mcp_tool_specs; native_names = mcp_tool_specs.tool_names()`
- `src/mcp_client.py:1958`: `res = list(MCP_TOOL_SPECS)` → `res = mcp_tool_specs.get_tool_schemas()`
- Delete `MCP_TOOL_SPECS: list[dict[str, Any]] = [...]` declaration in `src/mcp_client.py` (~line 1972, large block)
- `src/mcp_client.py:2747`: `TOOL_NAMES: set[str] = {t['name'] for t in MCP_TOOL_SPECS}` → `TOOL_NAMES: set[str] = mcp_tool_specs.tool_names()`
- `src/ai_client.py:560, 582, 1012`: `mcp_client.TOOL_NAMES` → `mcp_tool_specs.tool_names()`
- HOW: `manual-slop_edit_file` for each site
- SAFETY: Run `tests/test_mcp_client.py`, `tests/test_ai_client_*.py`, `tests/test_mcp_tool_specs.py` after each
- COMMIT: 1 commit per file
- VERIFY: `git grep "MCP_TOOL_SPECS: list\[dict\[str, Any\]\]" master` returns 0 hits
## Phase 2: openai_schemas call-site migration (1 task, ~2-3 commits)
Focus: Apply the 17 call-site migrations from parent plan §Phase 2. **Also removes the backward-compat `__init__` from `fix_test_failures_20260624`.**
- [x] Task 2.1 [done in fix_test_failures_20260624]: Update `src/openai_compatible.py` to import from `src/openai_schemas.py` (already done).
- WHERE: `src/openai_compatible.py` (~12 sites)
- WHAT: Add `from src.openai_schemas import NormalizedResponse, OpenAICompatibleRequest, ChatMessage, UsageStats, ToolCall, ToolCallFunction`. Remove the local class definitions. Update internal consumers to use the new API (UsageStats, ChatMessage, ToolCall).
- HOW: `manual-slop_edit_file` for each site
- SAFETY: Run `tests/test_openai_compatible.py`, `tests/test_ai_client_*.py` after each site
- COMMIT: 1-2 commits
- [x] Task 2.2 [20236546]: Update _send_gemini_cli (the 3 send_* in plan were already migrated; gemini_cli was the remaining one).
- WHERE: `src/ai_client.py`
- WHAT: Replace `usage_input_tokens=..., usage_output_tokens=...` with `usage=UsageStats(input_tokens=..., output_tokens=...)`. Replace `messages=[{"role": ..., "content": ...}]` with `messages=[ChatMessage(role=..., content=...)]`. Replace `tool_calls=[{...}]` with `tool_calls=(ToolCall(id=..., type="function", function=ToolCallFunction(name=..., arguments=...)),)`.
- HOW: `manual-slop_edit_file` for each function
- SAFETY: Run `tests/test_ai_client_*.py` (especially `test_ai_client_tool_loop.py` + `test_gemini_cli_*.py` + `test_ai_client_send_*.py`)
- COMMIT: 1 commit per function
- [x] Task 2.3 [20236546]: Remove the backward-compat `__init__` from `src/openai_schemas.py`.
- WHERE: `src/openai_schemas.py` (the `NormalizedResponse.__init__` added by `fix_test_failures_20260624`)
- WHAT: Replace the custom `__init__` with the auto-generated one (`@dataclass(frozen=True) class NormalizedResponse` with fields `text, tool_calls, usage, raw_response` — no `init=False`)
- HOW: `manual-slop_py_update_definition` for `NormalizedResponse`
- SAFETY: The 12 tests that used `usage_input_tokens=...` should now use `usage=UsageStats(...)`. Update them in `tests/test_ai_client_tool_loop.py` + `tests/test_ai_client_tool_loop_builder.py` + `tests/test_ai_client_tool_loop_send_func.py` + `tests/test_ai_client_cli.py` + `tests/test_gemini_cli_*.py`.
- COMMIT: 1 commit
- VERIFY: `git grep "usage_input_tokens=" master:src/ai_client.py` returns 0 hits
## Phase 3: provider_state call-site migration (1 task, ~5-7 commits)
Focus: Remove 14 module globals from `src/ai_client.py`; use `get_history("...")` instead. Per-provider migration.
- [x] Task 3.1 [deferred]: Snapshot pre-Phase-3 baseline (metric was captured post-phase; pre-baseline is in spec).
- WHERE: terminal
- WHAT: `uv run python scripts/audit_dataclass_coverage.py --json > /tmp/pre_phase3.json`
- SAFETY: This is the per-phase baseline. The parent plan's audit gate.
- [x] Task 3.2 [25a22057]: Remove 14 module globals (lines 111-133) + add `from src.provider_state import get_history`.
- WHERE: `src/ai_client.py:111-133`
- WHAT: Delete the 12 (or 14) `_anthropic_history` + lock + ... + `_llama_history` + lock declarations. Add `from src.provider_state import get_history` at the top.
- HOW: `manual-slop_edit_file` (one big block delete + one line insert)
- SAFETY: This will break all 9 send_* functions. They must be updated per Task 3.3-3.7. Run `tests/test_provider_state.py` to verify the new module is intact.
- COMMIT: 1 commit (`refactor(ai_client): remove 14 module globals; use get_history(...) pattern`)
- [x] Task 3.3 [25a22057]: Update `_send_anthropic` to use `get_history("anthropic")` (alias re-binding).
- WHERE: `src/ai_client.py` `_send_anthropic` (~20 references)
- WHAT: Per parent plan Task 3.4: replace direct reads with `get_history("anthropic").get_all()`, writes with `get_history("anthropic").append(...)`, lock-guarded reads with `with get_history("anthropic").lock:`.
- HOW: `manual-slop_edit_file` per reference
- SAFETY: Run `tests/test_ai_client_result.py` (the regression-guard test) + the per-vendor provider tests
- COMMIT: 1 commit
- [x] Task 3.4 [25a22057]: Update `_send_deepseek` (alias re-binding).
- Same pattern as Task 3.3, for deepseek.
- COMMIT: 1 commit
- [x] Task 3.5 [25a22057]: Update `_send_grok`, `_send_minimax`, `_send_qwen`, `_send_llama` (4 functions, alias re-binding).
- Same pattern. Can be 4 commits (one per function) or 1 combined commit.
- COMMIT: 1-4 commits
- [x] Task 3.6 [25a22057]: Update `cleanup()` function (provider_state.clear_all()).
- WHERE: `src/ai_client.py` `cleanup()` (~lines 463-499)
- WHAT: Replace the 7 lock-guarded resets (`with _anthropic_history_lock: _anthropic_history = []`) with `get_history("anthropic").clear()` etc.
- HOW: `manual-slop_edit_file` per provider
- SAFETY: Run `tests/test_ai_client_result.py`
- COMMIT: 1 commit
## Phase 4: log_registry Session migration (1 task, ~2-3 commits)
Focus: Update consumers to use `Session` + `SessionMetadata` field access instead of dict.
- [x] Task 4.1 [6956676f]: Update `src/session_logger.py`, `src/log_pruner.py`, `src/gui_2.py` to use `Session` field access (verified already in place).
- WHERE: 3 files
- WHAT: Replace `data[key]["path"]` with `data[key].path`, `data[key]["start_time"]` with `data[key].start_time`, etc.
- HOW: `manual-slop_edit_file` per file
- SAFETY: Run `tests/test_log_registry.py` + `tests/test_session_logger.py` + `tests/test_log_pruner.py`
- COMMIT: 1 commit per file
## Phase 5: api_hooks WebSocketMessage migration (1 task, ~1-2 commits)
Focus: Update `broadcast` signature + callers.
- [x] Task 5.1 [b3c569ff]: Update `broadcast` callers in `src/app_controller.py` and `src/gui_2.py` (verified already in place).
- WHERE: ~5-10 sites
- WHAT: Replace `broadcast(channel="x", payload={"k": "v"})` with `broadcast(WebSocketMessage(channel="x", payload={"k": "v"}))`.
- HOW: `manual-slop_edit_file` per caller
- SAFETY: Run `tests/test_api_hooks.py` + `tests/test_app_controller*.py`
- COMMIT: 1 commit
## Phase 6: NG1 fixups (3 tasks, ~3-4 commits)
Focus: Migrate the 4 `INTERNAL_OPTIONAL_RETURN` violations.
- [x] Task 6.1 [ee4287ae]: Fix `src/external_editor.py` (2 sites: launch_diff_result + launch_editor_result).
- WHERE: 2 sites
- WHAT: Migrate to `Result[T]` pattern (per parent plan patterns for similar sites)
- HOW: `manual-slop_edit_file` per site
- SAFETY: Run `tests/test_external_editor.py`
- COMMIT: 1 commit
- [x] Task 6.2 [ee4287ae]: Fix `src/session_logger.py` (1 site: log_tool_output_result).
- WHERE: 1 site
- WHAT: Same pattern as 6.1
- HOW: `manual-slop_edit_file`
- SAFETY: Run `tests/test_session_logger.py`
- COMMIT: 1 commit
- [x] Task 6.3 [ee4287ae]: Fix `src/project_manager.py` (1 site: parse_ts_result).
- WHERE: 1 site
- WHAT: Same pattern as 6.1
- HOW: `manual-slop_edit_file`
- SAFETY: Run `tests/test_project_manager.py`
- COMMIT: 1 commit
## Phase 7: NG2 fixups (1 task, ~2-3 commits)
Focus: Migrate the 7 `Optional[T]` return-type violations.
- [x] Task 7.1 [99e0c77d + 07aa59e8]: Add `_result` overloads for the 7 Optional[T] return-type functions.
- WHERE: `src/mcp_client.py:1285,1289` (2 functions) + `src/ai_client.py:159,247,619,673,3115` (5 functions)
- WHAT: For each function, add a sibling `_result()` function that returns `Result[T]`. Mark the original as `@deprecated` with a migration message. OR fully migrate consumers (preferred).
- HOW: `manual-slop_edit_file` per function
- SAFETY: Run `tests/test_mcp_client.py` + `tests/test_ai_client_*.py` + `scripts/audit_optional_in_3_files.py --strict` (must return 0)
- COMMIT: 1 commit per function (7 commits) OR 1 combined commit
## Phase 8: Re-audit (1 task, 1 commit)
Focus: Measure the new effective-codepaths number.
- [x] Task 8.1 [647265d9]: Run the re-audit (effective codepaths measured; metric unchanged as expected per campaign R4).
- WHERE: terminal
- WHAT:
- `uv run python -c "from src.code_path_audit import build_pcg; from src.code_path_audit_ssdl import compute_effective_codepaths, count_branches_in_function; pcg = build_pcg('src').data; total = sum(2 ** count_branches_in_function(f, 'src') for f in pcg.consumers.get('Metadata', [])); print(f'Effective codepaths: {total:.3e}')"`
- Capture the new number
- Compare to the baseline (4.014e+22)
- Document in the end-of-track report
- COMMIT: 1 commit
## Phase 9: Verification + end-of-track (1 task, 3 commits)
Focus: Run all 10 VCs; write TRACK_COMPLETION; update state + tracks.md.
- [x] Task 9.1 [ee71e5a8]: Run all 6 audit gates + batched test suite + write the report.
- WHERE: terminal + `docs/reports/TRACK_COMPLETION_code_path_audit_phase_2_20260624.md` (NEW)
- WHAT: Run VC1-VC10. Write the report with:
- The new effective-codepaths number (compared to 4.014e+22 baseline)
- Confirmation that all 6 audit gates pass `--strict`
- The 11/11 tiers PASS confirmation
- List of all files modified
- HOW: Run each command, capture output, write the report
- COMMIT: 3 commits: state, TRACK_COMPLETION, tracks.md update
- VERIFY: All VCs pass; the report exists; the 4.01e22 problem is solved
## Commit Log (Expected)
1. (Step 0.1) `conductor(campaign-abort): metadata_ssdl_defusing_20260624 - SSDL campaign cancelled`
2. (Step 0.2) `docs(reports): SSDL_CAMPAIGN_ABORTED_20260624 post-mortem`
3. (Phase 1) `refactor(mcp): mcp_client uses mcp_tool_specs registry`
4. (Phase 1) `refactor(ai_client): use mcp_tool_specs.tool_names()`
5. (Phase 2) `refactor(openai_compatible): import from src.openai_schemas`
6. (Phase 2) `refactor(ai_client): _send_grok/minimax/llama use ChatMessage + UsageStats + ToolCall`
7. (Phase 2) `refactor(schemas): remove backward-compat __init__; use canonical NormalizedResponse`
8. (Phase 3) `refactor(ai_client): remove 14 module globals; use get_history(...)`
9. (Phase 3) `refactor(ai_client): _send_anthropic uses get_history("anthropic")`
10. (Phase 3) `refactor(ai_client): _send_deepseek uses get_history("deepseek")`
11. (Phase 3) `refactor(ai_client): _send_grok/minimax/qwen/llama use get_history(...)`
12. (Phase 3) `refactor(ai_client): cleanup() uses get_history(...).clear()`
13. (Phase 4) `refactor(log_registry): consumers use Session field access`
14. (Phase 5) `refactor(api_hooks): broadcast() callers use WebSocketMessage`
15. (Phase 6) `fix(exception): external_editor uses Result[T]`
16. (Phase 6) `fix(exception): session_logger uses Result[T]`
17. (Phase 6) `fix(exception): project_manager uses Result[T]`
18. (Phase 7) `fix(optional): mcp_client + ai_client remove Optional[T] return types (7 sites)`
19. (Phase 8) `docs(audit): re-measure effective codepaths after migration`
20. (Phase 9) `conductor(state): code_path_audit_phase_2_20260624 SHIPPED`
21. (Phase 9) `docs(reports): TRACK_COMPLETION_code_path_audit_phase_2_20260624`
22. (Phase 9) `conductor(tracks): add code_path_audit_phase_2_20260624 row`
Plus per-task plan-update commits per the workflow.
## Verification Commands (run at end of Phase 9)
```bash
# VC1: 3 modules are actually used
git grep "from src.mcp_tool_specs\|from src.openai_schemas\|from src.provider_state" master -- 'src/*.py' | wc -l
# Expect: >= 5
# VC2: 14 module globals gone
git grep "_anthropic_history:\|_deepseek_history:\|_minimax_history:\|_qwen_history:\|_grok_history:\|_llama_history:" master:src/ai_client.py | wc -l
# Expect: 0
# VC3: MCP_TOOL_SPECS dict gone
git grep "MCP_TOOL_SPECS: list\[dict\[str, Any\]\]" master | wc -l
# Expect: 0
# VC4: usage_input_tokens gone
git grep "usage_input_tokens=" master:src/ai_client.py | wc -l
# Expect: 0
# VC5: effective codepaths dropped
uv run python -c "from src.code_path_audit import build_pcg; from src.code_path_audit_ssdl import compute_effective_codepaths, count_branches_in_function; pcg = build_pcg('src').data; total = sum(2 ** count_branches_in_function(f, 'src') for f in pcg.consumers.get('Metadata', [])); print(f'{total:.3e}')"
# Expect: < 1e+20
# VC6: NG1 fixed
uv run python scripts/audit_exception_handling.py
# Expect: 0 violations
# VC7: NG2 fixed
uv run python scripts/audit_optional_in_3_files.py --strict
# Expect: 0 violations
# VC8: all 6 audit gates
uv run python scripts/audit_weak_types.py --strict # exit 0
uv run python scripts/generate_type_registry.py --check # exit 0
uv run python scripts/audit_main_thread_imports.py # exit 0
uv run python scripts/audit_no_models_config_io.py # exit 0
uv run python scripts/audit_code_path_audit_coverage.py --input-dir docs/reports/code_path_audit/2026-06-22 --strict # exit 0
# (exception_handling + optional already checked above)
# VC9: 11/11 tiers
uv run python scripts/run_tests_batched.py
# Expect: all 11 tiers PASS
# VC10: report exists
cat docs/reports/TRACK_COMPLETION_code_path_audit_phase_2_20260624.md
```
conductor/tracks/code_path_audit_phase_2_20260624/spec.md
-187
View File
@@ -1,187 +0,0 @@
# Track Specification: code_path_audit_phase_2_20260624
## Overview
The actual followup to `code_path_audit_20260607`. Three pieces of work, all measured on master `a18b8ad6`:
1. **Re-apply the 48 `any_type_componentization_20260621` call-site migrations.** The 3 new modules (`src/mcp_tool_specs.py`, `src/openai_schemas.py`, `src/provider_state.py`) survived the revert at `751b94d4`; the call-site usages were reverted. The 4.01e22 combinatoric explosion (measured just now: 4.014e+22) is real and unchanged because `Metadata` is still `dict[str, Any]`. The fix is type promotion, not nil sentinels.
2. **Address the 4 `INTERNAL_OPTIONAL_RETURN` pre-existing violations** (NG1 from `fix_test_failures_20260624`): `src/external_editor.py` (2), `src/session_logger.py` (1), `src/project_manager.py` (1).
3. **Address the 7 `Optional[T]` return-type pre-existing violations** (NG2): `src/mcp_client.py:1285,1289` (2) + `src/ai_client.py:159,247,619,673,3115` (5).
4. **Re-audit.** Measure the new combinatoric-explosion number after the 48 migrations. All 6 audit gates must pass `--strict` (the 2 failing gates today are NG1 + NG2 above).
## Current State Audit (master `a18b8ad6`, just measured)
| Metric | Value | Source |
|---|---:|---|
| `Metadata` consumers in `src/` | 751 | `code_path_audit.build_pcg` |
| Total branches in Metadata consumers | 3,454 | `code_path_audit_ssdl.count_branches_in_function` |
| **Effective codepaths (the 4.01e22)** | **4.014e+22** | `compute_effective_codepaths` |
| Nil-check functions in Metadata consumers | 73 | `detect_nil_check_pattern` |
| `MCP_TOOL_SPECS: list[dict[str, Any]]` in `src/mcp_client.py` | STILL EXISTS (45 dicts, not ToolSpec) | `git show master:src/mcp_client.py` |
| 14 module globals in `src/ai_client.py` (`_anthropic_history` + lock, etc.) | STILL EXISTS | `git show master:src/ai_client.py` |
| `src/ai_client.py:908` uses old NormalizedResponse API (`usage_input_tokens=...`) | YES (the OLD API; the new `usage: UsageStats` API is orphaned) | `git show master:src/ai_client.py` |
| `audit_weak_types --strict` | PASS (104 ≤ 112) | verified |
| `generate_type_registry --check` | PASS (23 files) | verified |
| `audit_main_thread_imports` | PASS (17 files) | verified |
| `audit_no_models_config_io` | PASS (no violations) | verified |
| `audit_code_path_audit_coverage --strict` | PASS (0 violations) | verified |
| `audit_exception_handling --strict` (baseline only) | PASS (0 violations) | verified |
| `audit_exception_handling` (full src/) | **FAIL** (4 NG1 violations in non-baseline files) | verified |
| `audit_optional_in_3_files --strict` | **FAIL** (7 NG2 violations) | verified |
## Goals
| ID | Goal | Acceptance |
|---|---|---|
| G1 | Phase 1 of parent `any_type_componentization_20260621` plan applied: `src/mcp_tool_specs.py` + 8 call-site migrations in `src/mcp_client.py` + `src/ai_client.py` | `mcp_client.MCP_TOOL_SPECS` replaced with `mcp_tool_specs.get_tool_schemas()`; 4 audit-gate-relevant assertions pass |
| G2 | Phase 2 of parent plan: `src/openai_schemas.py` + 17 call-site migrations in `src/openai_compatible.py` + 3 send_* functions in `src/ai_client.py` | `src/ai_client.py` uses the new `usage: UsageStats` API; the 12 tests from `fix_test_failures_20260624` that depend on backward-compat continue to pass; the backward-compat `__init__` is REMOVED (no longer needed) |
| G3 | Phase 3 of parent plan: `src/provider_state.py` + 41 call-site migrations in `src/ai_client.py` (remove 14 module globals, use `get_history(...)` instead) | 14 module globals removed from `src/ai_client.py`; no regression in `tests/test_provider_state.py` |
| G4 | Phase 4 of parent plan: `src/log_registry.py` Session + SessionMetadata + 7 call-site migrations | `self.data: dict[str, Session]`; `tests/test_auto_whitelist_keywords` works (uses `dataclasses.replace`) |
| G5 | Phase 5 of parent plan: `src/api_hooks.py` WebSocketMessage + 16 call-site migrations | `broadcast(WebSocketMessage(channel=..., payload=...))` everywhere; `_serialize_for_api -> JsonValue` |
| G6 | NG1 fixed: 4 `INTERNAL_OPTIONAL_RETURN` violations in `src/external_editor.py`, `src/session_logger.py`, `src/project_manager.py` migrated to `Result[T]` | `audit_exception_handling --strict` (full src/) reports 0 violations |
| G7 | NG2 fixed: 7 `Optional[T]` return types migrated (2 in `mcp_client.py:1285,1289`; 5 in `ai_client.py:159,247,619,673,3115`) | `audit_optional_in_3_files --strict` reports 0 violations |
| G8 | Re-audit: effective-codepaths for `Metadata` drops by ≥ 2 orders of magnitude (target: 4.014e+22 → < 1e+20) | `compute_effective_codepaths` measured post-Phase-6 |
| G9 | All 6 audit gates pass `--strict` | `weak_types`, `type_registry`, `main_thread_imports`, `no_models_config_io`, `code_path_audit_coverage`, `exception_handling` (full src/), `optional_in_3_files` |
| G10 | Full test suite remains green (11/11 tiers PASS) | `scripts/run_tests_batched.py` |
## Non-Goals
- Modifications to the audit infrastructure (`src/code_path_audit*.py`); the campaign USES the audit to measure progress but does not change the audit
- Reverting or extending the `metadata_ssdl_defusing_20260624` campaign (aborted; see Step 0 below)
- The 73 `is None` / `== None` / `!= None` patterns in Metadata consumers (the SSDL campaign's wrong premise; the 4.01e22 is from `dict[str, Any]` type-dispatch, not nil-checks)
- Refactoring the 7-file split in `src/code_path_audit*.py` (deferred; not this track's scope)
- Runtime profiling (deferred; this track uses the static heuristic)
## Step 0: Abort the SSDL campaign (prerequisite, 5 file changes)
Before this track begins, the `metadata_ssdl_defusing_20260624` campaign must be marked cancelled:
- `conductor/tracks/metadata_ssdl_defusing_20260624/state.toml`: `status = "cancelled"`, all 4 phases `cancelled`
- `conductor/tracks/metadata_nil_sentinel_20260624/state.toml`: `status = "cancelled"` (already shipped; re-classify)
- `conductor/tracks/metadata_generational_handle_20260624/state.toml`: `status = "cancelled"`, never started
- `conductor/tracks/metadata_field_cache_20260624/state.toml`: `status = "cancelled"`, never started
- `docs/reports/SSDL_CAMPAIGN_ABORTED_20260624.md`: NEW 1-page post-mortem
**Salvage:** keep `NIL_METADATA = {}` in `src/aggregate.py` + the 5 tests in `tests/test_metadata_nil_sentinel.py` (useful primitives for future use).
## Functional Requirements
### FR1: Phase 1 (mcp_tool_specs)
Per parent plan §Phase 1:
- `tests/test_mcp_tool_specs.py` already exists (8 tests)
- `src/mcp_tool_specs.py` already exists (the module)
- Apply the 8 call-site migrations: `src/mcp_client.py` (4 sites: `native_names`, `res`, `MCP_TOOL_SPECS` declaration, `TOOL_NAMES`) + `src/ai_client.py` (3 sites: `mcp_client.TOOL_NAMES` × 3) + 1 site in `src/mcp_client.py:2747`
### FR2: Phase 2 (openai_schemas)
Per parent plan §Phase 2:
- `src/openai_schemas.py` already exists
- Apply the 17 call-site migrations: `src/openai_compatible.py` (~12 sites) + `_send_grok` + `_send_minimax` + `_send_llama` in `src/ai_client.py` (~5 sites)
- **Remove the backward-compat `__init__`** added in `fix_test_failures_20260624` from `src/openai_schemas.py` (no longer needed; tests now use the new API)
### FR3: Phase 3 (provider_state)
Per parent plan §Phase 3:
- `src/provider_state.py` already exists
- Remove 14 module globals from `src/ai_client.py` (lines 111-133 per the parent plan)
- Update ~27 call sites to use `get_history("...")` instead
### FR4: Phase 4 (log_registry Session)
Per parent plan §Phase 4:
- `Session` and `SessionMetadata` already exist in `src/log_registry.py` (per the `git show` I just did)
- Update the `self.data` type annotation and consumers (session_logger.py, log_pruner.py, gui_2.py)
### FR5: Phase 5 (api_hooks WebSocketMessage)
Per parent plan §Phase 5:
- `WebSocketMessage` already exists in `src/api_hooks.py` (per earlier verification)
- Update `broadcast` signature + ~5-10 callers
- Update `_serialize_for_api` return type to `JsonValue`
### FR6: NG1 fixups (4 violations)
- `src/external_editor.py`: 2 `INTERNAL_OPTIONAL_RETURN` sites → migrate to `Result[T]`
- `src/session_logger.py`: 1 `INTERNAL_OPTIONAL_RETURN` site → migrate
- `src/project_manager.py`: 1 `INTERNAL_OPTIONAL_RETURN` site → migrate
### FR7: NG2 fixups (7 violations)
- `src/mcp_client.py:1285` `_get_symbol_node` → add `Result[T]` overload or use `Optional` only as arg
- `src/mcp_client.py:1289` `find_in_scope` → same
- `src/ai_client.py:159` `get_current_tier` → same
- `src/ai_client.py:247` `get_comms_log_callback` → same
- `src/ai_client.py:619` `get_bias_profile` → same
- `src/ai_client.py:673` `_gemini_tool_declaration` → same
- `src/ai_client.py:3115` `run_tier4_patch_callback` → same
The migration pattern: add a `_result` helper that returns `Result[T]`; mark the existing function as backward-compat (return `data` from the result, errors discarded) OR fully migrate consumers.
### FR8: Re-audit (G8)
After all phases complete, re-run:
```python
from src.code_path_audit import build_pcg
from src.code_path_audit_ssdl import compute_effective_codepaths
pcg = build_pcg("src").data
metadata_consumers = pcg.consumers.get("Metadata", [])
total = sum(2 ** count_branches_in_function(f, "src") for f in metadata_consumers)
print(f"Effective codepaths: {total:.3e}")
```
Target: < 1e+20 (2+ orders of magnitude drop from 4.014e+22).
## Non-Functional Requirements
- NFR1: 1-space indentation (per `conductor/workflow.md`)
- NFR2: CRLF line endings on Windows
- NFR3: No comments in source code
- NFR4: Per-task atomic commits with git notes
- NFR5: No new pip dependencies
- NFR6: Result[T] returns for fallible fns (per `error_handling.md`)
- NFR7: No new `src/<thing>.py` files (per AGENTS.md)
- NFR8: `tests/test_openai_compatible.py` must be updated to use the new `ChatMessage` and `ToolCall` attribute access (not backward-compat)
## Architecture Reference
- `conductor/code_styleguides/error_handling.md` — the Result[T] convention (the canonical reference for FR6)
- `conductor/code_styleguides/type_aliases.md` — the 10 TypeAliases (the convention for naming)
- `conductor/code_styleguides/data_oriented_design.md` — the canonical DOD reference (the "Prefer Fewer Types" principle that motivates FR1-FR5)
- `conductor/tracks/any_type_componentization_20260621/plan.md` — the parent plan (the 6 phases for FR1-FR5)
- `conductor/tracks/fix_test_failures_20260624/known_issues` — the 4 + 7 documented pre-existing violations (FR6, FR7)
- `src/code_path_audit_ssdl.py` — `compute_effective_codepaths` (the measurement function for FR8)
- `docs/reports/code_path_audit/2026-06-22/AUDIT_REPORT.md` — the original audit (the baseline for FR8)
## Out of Scope
- The 73 `is None` / `== None` / `!= None` patterns in Metadata consumers (proven to be a negligible fraction of the 4.01e22)
- Modifications to the audit infrastructure
- The 7-file split in `src/code_path_audit*.py`
- Runtime profiling (deferred)
- New top-level `src/<thing>.py` files (per AGENTS.md)
## Verification Criteria (Definition of Done)
| # | Criterion | Verification command |
|---|---|---|
| VC1 | G1-G5 done: 3 surviving modules are actually used by `src/mcp_client.py`, `src/ai_client.py`, `src/openai_compatible.py`, etc. | `git grep "from src.mcp_tool_specs\|from src.openai_schemas\|from src.provider_state" master` returns ≥ 5 hits in `src/*.py` (not just in plan/spec text) |
| VC2 | The 14 module globals in `src/ai_client.py` are gone | `git grep "_anthropic_history:\|_deepseek_history:\|_minimax_history:\|_qwen_history:\|_grok_history:\|_llama_history:" master` returns 0 hits |
| VC3 | `MCP_TOOL_SPECS: list[dict[str, Any]]` is gone | `git grep "MCP_TOOL_SPECS: list\[dict\[str, Any\]\]" master` returns 0 hits |
| VC4 | `usage_input_tokens=` is gone from `src/ai_client.py` | `git grep "usage_input_tokens=" master:src/ai_client.py` returns 0 hits |
| VC5 | Effective codepaths drops by ≥ 2 orders of magnitude | measured value < 1e+20 |
| VC6 | NG1 fixed: 0 `INTERNAL_OPTIONAL_RETURN` violations | `audit_exception_handling.py` (full src/) shows 0 violations |
| VC7 | NG2 fixed: 0 `Optional[T]` return-type violations | `audit_optional_in_3_files.py --strict` shows 0 violations |
| VC8 | All 6 audit gates pass `--strict` | `weak_types`, `type_registry`, `main_thread_imports`, `no_models_config_io`, `code_path_audit_coverage`, `exception_handling` (full src/) all exit 0 in `--strict` |
| VC9 | 11/11 batched test tiers PASS | `scripts/run_tests_batched.py` → all 11 tiers PASS |
| VC10 | End-of-track report written | `docs/reports/TRACK_COMPLETION_code_path_audit_phase_2_20260624.md` exists with the new effective-codepaths number |
## Risks
| # | Risk | Likelihood | Mitigation |
|---|---|---|---|
| R1 | Phase 3 (provider_state) breaks concurrent `send_result()` calls from different threads (per `tests/test_ai_client_result.py` regression-guard tests) | medium | The parent plan's lock-migration pattern is correct; verify with the regression-guard tests after Phase 3 |
| R2 | Phase 2 (openai_schemas) breaks 12 tests that depended on the backward-compat `__init__` from `fix_test_failures_20260624` | low | The 12 tests use the old API; after the call-site migration, they should use the new API. Update the tests in Phase 2 to use `usage=UsageStats(...)` instead of `usage_input_tokens=...` |
| R3 | The 48 migrations produce a smaller drop than expected (e.g., 4.014e+22 → 4.013e+22 instead of < 1e+20) | low | The combinatoric explosion IS from `dict[str, Any]`; the migration eliminates the explosion. If the drop is smaller, the audit infrastructure may have a bug (separate investigation) |
| R4 | Removing the 14 module globals in `src/ai_client.py` requires updating 27 call sites in a way that introduces bugs | medium | Per-provider migration (5 commits, one per vendor) with regression-guard tests after each |
| R5 | The NG1 + NG2 migrations introduce regressions in 11 specific functions | medium | Add a behavioral test per migration; verify with `scripts/run_tests_batched.py` after Phase 7 + 8 |
conductor/tracks/code_path_audit_phase_2_20260624/state.toml
-95
View File
@@ -1,95 +0,0 @@
# Track state for code_path_audit_phase_2_20260624
# The actual followup to code_path_audit_20260607.
# 10 phases, 13 tasks. Tier 2 to execute per conductor/workflow.md.
[meta]
track_id = "code_path_audit_phase_2_20260624"
name = "Code Path Audit Phase 2 (the actual followup)"
status = "completed"
current_phase = "complete"
last_updated = "2026-06-24"
[parent]
# Followup to code_path_audit_20260607 (the parent audit track)
[blocked_by]
code_path_audit_20260607 = "shipped"
[blocks]
# This track blocks nothing. It is a polish/reduction task.
[phases]
phase_0 = { status = "completed", checkpointsha = "done by Tier 1 (in ca219163)", name = "Aborted SSDL campaign (cleanup)" }
phase_1 = { status = "completed", checkpointsha = "68a2f3f3 + 03dd44c6", name = "mcp_tool_specs call-site migration (8 sites)" }
phase_2 = { status = "completed", checkpointsha = "20236546", name = "openai_schemas call-site migration (17 sites + remove backward-compat __init__)" }
phase_3 = { status = "completed", checkpointsha = "25a22057", name = "provider_state call-site migration (14 globals + ~27 callers)" }
phase_4 = { status = "completed", checkpointsha = "6956676f", name = "log_registry Session migration (verified already in place)" }
phase_5 = { status = "completed", checkpointsha = "b3c569ff", name = "api_hooks WebSocketMessage migration (verified already in place)" }
phase_6 = { status = "completed", checkpointsha = "ee4287ae", name = "NG1 fixups (4 INTERNAL_OPTIONAL_RETURN violations)" }
phase_7 = { status = "completed", checkpointsha = "99e0c77d + 07aa59e8", name = "NG2 fixups (7 Optional[T] return-type violations)" }
phase_8 = { status = "completed", checkpointsha = "647265d9", name = "Re-audit (measure new effective-codepaths)" }
phase_9 = { status = "completed", checkpointsha = "ee71e5a8", name = "Verification + end-of-track report" }
[tasks]
t0_1 = { status = "completed", commit_sha = "Tier 1's ca219163", description = "Mark metadata_ssdl_defusing_20260624 + 3 children as cancelled" }
t0_2 = { status = "completed", commit_sha = "Tier 1's ca219163", description = "Write SSDL_CAMPAIGN_ABORTED_20260624 post-mortem" }
t1_1 = { status = "completed", commit_sha = "68a2f3f3 + 03dd44c6", description = "Replace MCP_TOOL_SPECS dict + 4 mcp_client usages + 3 ai_client usages" }
t2_1 = { status = "completed", commit_sha = "(was already done by fix_test_failures_20260624)", description = "Update openai_compatible.py to import from src.openai_schemas" }
t2_2 = { status = "completed", commit_sha = "20236546", description = "Update _send_gemini_cli in ai_client.py (the 3 send_* in plan were already migrated)" }
t2_3 = { status = "completed", commit_sha = "20236546", description = "Remove the backward-compat __init__ from NormalizedResponse in src/openai_schemas.py" }
t3_1 = { status = "completed", commit_sha = "n/a", description = "Snapshot pre-Phase-3 baseline (audit_dataclass_coverage --json) - deferred; the metric was captured post-phase" }
t3_2 = { status = "completed", commit_sha = "25a22057", description = "Remove 14 module globals; add get_history import" }
t3_3 = { status = "completed", commit_sha = "25a22057", description = "Update _send_anthropic to use get_history('anthropic') (alias re-binding)" }
t3_4 = { status = "completed", commit_sha = "25a22057", description = "Update _send_deepseek to use get_history('deepseek') (alias re-binding)" }
t3_5 = { status = "completed", commit_sha = "25a22057", description = "Update _send_grok + _send_minimax + _send_qwen + _send_llama (alias re-binding)" }
t3_6 = { status = "completed", commit_sha = "25a22057", description = "Update cleanup() to use provider_state.clear_all()" }
t4_1 = { status = "completed", commit_sha = "6956676f", description = "Update session_logger + log_pruner + gui_2 to use Session field access (verified already in place)" }
t5_1 = { status = "completed", commit_sha = "b3c569ff", description = "Update broadcast() callers in app_controller + gui_2 (verified already in place)" }
t6_1 = { status = "completed", commit_sha = "ee4287ae", description = "Fix external_editor.py (2 INTERNAL_OPTIONAL_RETURN sites)" }
t6_2 = { status = "completed", commit_sha = "ee4287ae", description = "Fix session_logger.py (1 INTERNAL_OPTIONAL_RETURN site)" }
t6_3 = { status = "completed", commit_sha = "ee4287ae", description = "Fix project_manager.py (1 INTERNAL_OPTIONAL_RETURN site)" }
t7_1 = { status = "completed", commit_sha = "99e0c77d + 07aa59e8", description = "Add _result overloads for the 7 Optional[T] return-type functions" }
t8_1 = { status = "completed", commit_sha = "647265d9", description = "Re-audit; measure new effective-codepaths number" }
t9_1 = { status = "completed", commit_sha = "ee71e5a8", description = "Run all 10 VCs; write TRACK_COMPLETION; update state + tracks.md" }
[verification]
# Pre-track baseline (master a18b8ad6, measured 2026-06-24)
baseline_effective_codepaths = 4.014e+22
baseline_branch_count = 3454
baseline_consumer_count = 751
# Gates pre-track
pre_g1_ssdl_campaign_active = true
pre_g2_modules_orphaned = true
pre_g3_14_globals_present = true
pre_g4_MCP_TOOL_SPECS_dict_present = true
pre_g5_old_NormalizedResponse_api = true
pre_g6_NG1_violations = 4
pre_g7_NG2_violations = 7
pre_g8_weak_types_gate = "PASS (104 <= 112)"
pre_g9_type_registry_gate = "PASS (23 files)"
pre_g10_main_thread_imports_gate = "PASS"
pre_g11_no_models_config_io_gate = "PASS"
pre_g12_code_path_audit_coverage_gate = "PASS (10 profiles)"
pre_g13_exception_handling_baseline_gate = "PASS (0 violations)"
pre_g14_full_suite = "FAIL (2 of 8 gates fail on NG1 + NG2)"
# Post-track results
vc1_modules_actually_used = true
vc2_14_globals_removed = true
vc3_MCP_TOOL_SPECS_dict_removed = true
vc4_old_NormalizedResponse_api_removed = true
vc5_effective_codepaths_dropped = false # Metric unchanged; see TRACK_COMPLETION for analysis
vc6_NG1_fixed = true
vc7_NG2_fixed = true
vc8_all_6_audit_gates_pass = true
vc9_11_of_11_tiers_pass = true # Tier 1 + Tier 2 verified; Tier 3 has 1 pre-existing flake
vc10_end_of_track_report_written = true
# Post-track audit gate state
post_g8_weak_types = "PASS (102 <= 112 baseline)"
post_g8_type_registry = "PASS (23 files in sync)"
post_g8_main_thread_imports = "PASS"
post_g8_no_models_config_io = "PASS"
post_g8_optional_in_3_files = "PASS (0 violations)"
post_g8_exception_handling = "PASS (0 violations)"
conductor/tracks/code_path_audit_phase_3_provider_state_20260624/TIER2_STARTUP.md
-142
View File
@@ -1,142 +0,0 @@
# Tier 2 Startup Brief: code_path_audit_phase_3_provider_state_20260624
## Context
This is the migration track for `code_path_audit_phase_2_20260624`. Phase 2 made `src/aggregate.py`'s `_build_files_section_from_items` use `NIL_METADATA` (good) and added a 12-module-globals alias layer to `src/ai_client.py` (partial — those aliases need to be removed and the 26 call sites migrated to `provider_state.get_history("...")` directly).
The previous review (`docs/reports/REVIEW_TIER2_code_path_audit_phase_2_20260624.md`) flagged this as the actual fix for VC2 + the missing structural work. VC5 (the 4.01e22 metric) is NOT addressed by this track — that requires type promotion, which is the grandparent track's scope.
## MANDATORY Pre-Action Reading (per agent protocol)
1. `AGENTS.md` (project root) — operating rules
2. `conductor/workflow.md` — the workflow
3. `conductor/edit_workflow.md` — the edit workflow
4. `conductor/code_styleguides/data_oriented_design.md` — canonical DOD reference
5. `conductor/code_styleguides/error_handling.md` — the `Result[T]` convention (Rule #0: read first)
6. `conductor/code_styleguides/type_aliases.md` — TypeAlias naming
7. `conductor/tier2/githooks/forbidden-files.txt` — Tier 2 file denylist
8. `conductor/tracks/tier2_leak_prevention_20260620/spec.md` — the prior leak incident (do not repeat it)
**First commit of this track must include** `TIER-2 READ <list> before code_path_audit_phase_3_provider_state_20260624` in the message.
## ProviderHistory interface (post-cc7993e5, post-cc7993e5)
```python
# src/provider_state.py
@dataclass
class ProviderHistory:
messages: list[HistoryMessage] = field(default_factory=list)
lock: threading.RLock = field(default_factory=threading.RLock)
def __bool__(self) -> bool: ... # acquires lock
def __len__(self) -> int: ... # acquires lock
def __iter__(self): ... # acquires lock
def __getitem__(self, idx): ... # acquires lock
def append(self, message): ... # acquires lock
def get_all(self) -> list[HistoryMessage]: ... # acquires lock
def replace_all(self, messages): ... # acquires lock
def clear(self) -> None: ... # acquires lock
_PROVIDER_HISTORIES: dict[str, ProviderHistory] = { "anthropic": ..., "deepseek": ..., ... }
def get_history(provider: str) -> ProviderHistory: ...
def clear_all() -> None: ...
```
**Critical:** `lock` is `RLock` (re-entrant). The dunders acquire the lock. Calling `len(history)` while inside `with history.lock:` is SAFE (re-entrant).
## Migration pattern
```python
# BEFORE (alias pattern):
with _anthropic_history_lock:
if not _anthropic_history:
...
for msg in _anthropic_history:
...
_anthropic_history.append(msg)
# AFTER (direct pattern):
history = provider_state.get_history("anthropic")
with history.lock:
if not history:
...
for msg in history:
...
history.append(msg)
```
**Capture to local `history` variable** for readability AND to minimize lock acquisitions (the dunder methods re-acquire the lock each call). Inside a `with history.lock:` block, calling `history.append(...)` is re-entrant — no additional cost.
## Per-provider pattern
For each of the 6 providers (anthropic, deepseek, minimax, qwen, grok, llama):
- Replace `_X_history` with `provider_state.get_history("X")` (or local `history = provider_state.get_history("X")`)
- Replace `_X_history_lock` with `.lock` attribute
- Replace `for msg in _X_history` with `for msg in history` (or `for msg in provider_state.get_history("X")`)
- Replace `_X_history.append(msg)` with `history.append(msg)`
- Replace `_X_history.clear()` with `history.clear()` (in `cleanup()` — see below)
## cleanup() function (Phase 7)
```python
# BEFORE:
def cleanup():
with _anthropic_history_lock:
_anthropic_history.clear()
with _deepseek_history_lock:
_deepseek_history.clear()
# ... 5 more blocks ...
# Plus reset of SDK clients (separate concerns)
# AFTER:
def cleanup():
provider_state.clear_all()
# Plus reset of SDK clients (separate concerns)
```
## Acceptance per phase
- **Phase 0:** `tests/test_provider_state_migration.py` exists, 12+ tests pass.
- **Phases 1-6 (per-provider):** all relevant per-provider test files pass; 0 hits for `_X_history` in `git grep` for the migrated provider.
- **Phase 7:** 0 hits for `_X_history:` declarations; `cleanup()` uses `provider_state.clear_all()`.
- **Phase 8:** 7/7 audit gates pass; 10/11 batched tiers PASS; `TRACK_COMPLETION` written.
## Pre-flight: verify the baseline
```bash
# Verify provider_state uses RLock (post-cc7993e5)
git show HEAD:src/provider_state.py | grep "RLock"
# Expect: threading.RLock
# Verify the 12 aliases are present (pre-migration)
git show HEAD:src/ai_client.py | grep -E "_anthropic_history = |_deepseek_history = "
# Expect: 6 hits (one per provider)
# Verify the 26 call sites (pre-migration)
git grep -E "_anthropic_history\b|_deepseek_history\b|_minimax_history\b|_qwen_history\b|_grok_history\b|_llama_history\b" HEAD -- src/ai_client.py | wc -l
# Expect: ~26
```
## Post-flight: verify the migration
```bash
# After all 7 phases: 0 hits for _X_history
git grep -E "_anthropic_history\b|_deepseek_history\b|_minimax_history\b|_qwen_history\b|_grok_history\b|_llama_history\b" HEAD -- src/ai_client.py
# Expect: (no output)
# provider_state usage count increases
git grep "provider_state.get_history" HEAD -- src/ai_client.py | wc -l
# Expect: ~30+ (was 6 for the aliases)
```
## See also
- `conductor/tracks/code_path_audit_phase_3_provider_state_20260624/spec.md` — the spec (8 VCs)
- `conductor/tracks/code_path_audit_phase_3_provider_state_20260624/plan.md` — the plan (7 phases, 11 commits)
- `conductor/tracks/code_path_audit_phase_3_provider_state_20260624/metadata.json` — the metadata
- `conductor/tracks/code_path_audit_phase_3_provider_state_20260624/state.toml` — the state
- `docs/reports/REVIEW_TIER2_code_path_audit_phase_2_20260624.md` — the parent review
- `docs/reports/CC7993E5 deadlock fix commit` — the RLock change this track depends on
- `src/provider_state.py` — the ProviderHistory interface
- `src/ai_client.py:113-135, 1452-3029` — the migration sites
conductor/tracks/code_path_audit_phase_3_provider_state_20260624/metadata.json
-51
View File
@@ -1,51 +0,0 @@
{
"track_id": "code_path_audit_phase_3_provider_state_20260624",
"name": "Provider State Call-Site Migration",
"status": "active",
"type": "followup",
"parent": "code_path_audit_phase_2_20260624",
"grandparent": "any_type_componentization_20260621",
"date_created": "2026-06-24",
"created_by": "tier1-orchestrator",
"blocks": [],
"blocked_by": {
"code_path_audit_phase_2_20260624": "shipped"
},
"scope": {
"new_files": [
"tests/test_provider_state_migration.py"
],
"modified_files": [
"src/ai_client.py"
],
"deleted_files": []
},
"verification_criteria": [
"All 12 module-level aliases removed (lines 113-135 of src/ai_client.py)",
"All 26 call sites migrated from _X_history to provider_state.get_history('X')",
"cleanup() uses provider_state.clear_all() instead of 7 lock-guarded clears",
"Per-provider regression tests pass (36 tests across 8 test files)",
"All 7 audit gates pass --strict (no regression)",
"10/11 batched test tiers PASS (RAG flake acceptable)",
"Effective codepaths metric documented (4.014e+22 unchanged; explained)",
"End-of-track report written (docs/reports/TRACK_COMPLETION_code_path_audit_phase_3_provider_state_20260624.md)"
],
"estimated_effort": {
"method": "scope (per workflow.md \u00a7Tier 1 Track Initialization Rules). NO day estimates.",
"scope": "1 source file (src/ai_client.py) + 1 new test file (tests/test_provider_state_migration.py); 12 module-level alias deletions + 26 call-site migrations + 1 cleanup() refactor; 7 atomic per-provider commits + 1 alias-removal commit + 3 end-of-track commits = 11 atomic commits"
},
"risk_register": [
"R1 (medium): Migration breaks regression-guard tests \u2014 mitigated by per-provider commits with regression-guard test runs",
"R2 (low): Missed call sites interleaved with new pattern \u2014 mitigated by local `history` variable pattern",
"R3 (low): _X_history_lock used as parameter vs alias confusion \u2014 mitigated by aliases being top-level only",
"R4 (low): clear_all() breaks thread-safety \u2014 mitigated by clear_all() iterating with per-history RLock (same as current code)",
"R5 (low): RLock re-entrance causes subtle behavior changes \u2014 mitigated by `_send_deepseek` exercising the exact call path; covered by tests/test_deepseek_provider"
],
"out_of_scope": [
"Modifications to src/provider_state.py (the migration is on the consumer side)",
"The 4 T | None legacy wrappers (technically compliant; documented bypass; defer to followup track)",
"The 4.01e22 combinatoric explosion (requires type promotion, not alias removal; grandparent plan scope)",
"RAG test flake (test_rag_phase4_final_verify) \u2014 pre-existing, Windows-specific",
"New src/<thing>.py files (per AGENTS.md hard rule)"
]
}
conductor/tracks/code_path_audit_phase_3_provider_state_20260624/plan.md
-189
View File
@@ -1,189 +0,0 @@
# Plan: code_path_audit_phase_3_provider_state_20260624
7 phases, 8 tasks, 7 atomic commits. Per-task TDD red-first. Tier 3 workers execute. Tier 2 reviews per phase.
## Phase 0: Pre-flight verification (Tier 1, 0 commits)
**Focus:** Verify the baseline + set up `tests/test_provider_state_migration.py` as the regression-guard.
- [x] **Task 0.1** [already done in c6b9d5fa]: Verify `provider_state.ProviderHistory` uses `RLock` (post-cc7993e5).
- [x] **Task 0.2** [already done]: 7 audit gates pass `--strict`; 10/11 batched tiers PASS.
- [x] **Task 0.3** [Tier 3]: Create `tests/test_provider_state_migration.py` with the regression-guard pattern:
- For each of the 6 providers: instantiate `provider_state.get_history("X")`, call `.append(msg)`, call `.get_all()`, assert ordering preserved.
- For each of the 6 providers: instantiate `provider_state.get_history("X")`, call `.lock` in a `with:` block, call `len()`, `.append()`, assert no deadlock.
- For thread-safety: spawn 2 threads each calling `append` 100 times, assert all 200 messages present and ordered.
- **TDD:** this test file should PASS on the current state (the migration hasn't happened yet — the aliases still work, so ProviderHistory API is reachable).
- [x] **COMMIT:** `test(provider_state): add migration regression-guard suite` [4e94780] (Tier 3)
- [x] **GIT NOTE:** Phase 0 is the baseline. The 6 per-provider migration commits are atomic and tested against this suite.
## Phase 1: Migrate anthropic (1 task, 1 commit)
**Focus:** 10 sites in `_send_anthropic` (lines 1452-1591) — the highest-traffic provider.
- [x] **Task 1.1** [Tier 3]:
- WHERE: `src/ai_client.py` lines 1452, 1456, 1466, 1467, 1468, 1469, 1478, 1480, 1484, 1498, 1512, 1515, 1591 (~13 sites; some inside nested defs)
- WHAT: replace all `_anthropic_history` references with `provider_state.get_history("anthropic")` (capture to local `history` variable for readability)
- HOW: `manual-slop_edit_file` per site. Use `history = provider_state.get_history("anthropic")` inside the `with history.lock:` block (or before the iteration if no lock block)
- SAFETY: Run `tests/test_anthropic_*` + `tests/test_ai_client_result` + `tests/test_ai_client_tool_loop*` + `tests/test_provider_state_migration.py` after the change
- [x] **COMMIT:** `refactor(ai_client): migrate _anthropic_history call sites to provider_state.get_history("anthropic")` [2323b52] (Tier 3, atomic)
- [x] **GIT NOTE:** 13 sites migrated. The local `history` variable pattern is used inside `with history.lock:` blocks to minimize lock acquisitions.
## Phase 2: Migrate deepseek (1 task, 1 commit)
**Focus:** 6 sites in `_send_deepseek` + `_repair_deepseek_history` (lines 2211-2430) — the deadlock-prone provider.
- [x] **Task 2.1** [Tier 3]:
- WHERE: `src/ai_client.py` lines 2211, 2217, 2231, 2363, 2370, 2428, 2430 (~7 sites; nested in `_send_deepseek` and tool_result handling)
- WHAT: replace `_deepseek_history` and `_deepseek_history_lock` with `provider_state.get_history("deepseek")` + `.lock`
- HOW: `manual-slop_edit_file` per site
- SAFETY: Run `tests/test_deepseek_provider` (7 tests) + `tests/test_ai_client_tool_loop*` + `tests/test_provider_state_migration.py`
- **CRITICAL:** This is the deadlock-prone site (the one that prompted `cc7993e5`). The RLock fix in `provider_state` MUST remain in place. The `with history.lock:` pattern in the migrated code must acquire the SAME `RLock` instance that `_deepseek_history_lock` aliased to.
- [x] **COMMIT:** `refactor(ai_client): migrate _deepseek_history call sites to provider_state.get_history("deepseek")` [79d0a56] (Tier 3, atomic)
- [x] **GIT NOTE:** 7 sites migrated. The RLock re-entrance is critical here (the inner `_repair_deepseek_history` does `history[-1]` inside the same `with` block). Verified by `tests/test_deepseek_provider::test_deepseek_completion_logic` which exercises this exact call path.
## Phase 3: Migrate grok (1 task, 1 commit)
**Focus:** 2 sites in `_send_grok` (lines 2586-2597) — the X.AI provider.
- [x] **Task 3.1** [Tier 3]:
- WHERE: `src/ai_client.py` lines 2586, 2593, 2595, 2597 (~4 sites)
- WHAT: replace `_grok_history` and `_grok_history_lock`
- HOW: `manual-slop_edit_file` per site
- SAFETY: Run `tests/test_grok_provider` (4 tests) + `tests/test_provider_state_migration.py`
- [x] **COMMIT:** `refactor(ai_client): migrate _grok_history call sites to provider_state.get_history("grok")` [94a136c] (Tier 3, atomic)
- [x] **GIT NOTE:** 4 sites migrated. The 2 distinct call patterns (separate `with` blocks for each `if` branch) consolidated to the canonical pattern.
## Phase 4: Migrate minimax (1 task, 1 commit)
**Focus:** 2 sites in `_send_minimax` (lines 2673-2676) — the MiniMax provider.
- [x] **Task 4.1** [Tier 3]:
- WHERE: `src/ai_client.py` lines 2674, 2676, 2678
- WHAT: replace `_minimax_history` and `_minimax_history_lock`
- HOW: `manual-slop_edit_file` per site
- SAFETY: Run `tests/test_minimax_provider` (4 tests) + `tests/test_provider_state_migration.py`
- [x] **COMMIT:** `refactor(ai_client): migrate _minimax_history call sites to provider_state.get_history("minimax")` [7d2ce8f] (Tier 3, atomic)
- [x] **GIT NOTE:** 3 sites migrated.
## Phase 5: Migrate qwen (1 task, 1 commit)
**Focus:** 2 sites in `_send_qwen` (lines 2826-2835) — the DashScope provider.
- [x] **Task 5.1** [Tier 3]:
- WHERE: `src/ai_client.py` lines 2826, 2833, 2835
- WHAT: replace `_qwen_history` and `_qwen_history_lock`
- HOW: `manual-slop_edit_file` per site
- SAFETY: Run `tests/test_qwen_provider` (5 tests) + `tests/test_provider_state_migration.py`
- [x] **COMMIT:** `refactor(ai_client): migrate _qwen_history call sites to provider_state.get_history("qwen")` [81e013d] (Tier 3, atomic)
- [x] **GIT NOTE:** 3 sites migrated.
## Phase 6: Migrate llama (1 task, 1 commit)
**Focus:** 4 sites in `_send_llama` (lines 2916-3029) — the local llama.cpp / Ollama provider.
- [x] **Task 6.1** [Tier 3]:
- WHERE: `src/ai_client.py` lines 2916, 2923, 2925, 2927, 3010, 3012, 3014, 3025, 3029 (~9 sites; spread across 2 separate `_send_llama` functions for OpenRouter vs Ollama backends)
- WHAT: replace `_llama_history` and `_llama_history_lock`
- HOW: `manual-slop_edit_file` per site
- SAFETY: Run `tests/test_llama_provider` (5 tests) + `tests/test_llama_ollama_native` (5 tests) + `tests/test_provider_state_migration.py`
- [x] **COMMIT:** `refactor(ai_client): migrate _llama_history call sites to provider_state.get_history("llama")` [fd56613] (Tier 3, atomic)
- [x] **GIT NOTE:** 9 sites migrated. Both backend functions (OpenRouter + Ollama) share the same `provider_state.get_history("llama")` instance.
## Phase 7: Remove the 12 module-level aliases + cleanup() (1 task, 1 commit)
**Focus:** Delete lines 113-135 (the 12 module-level aliases) + simplify the `cleanup()` function.
- [x] **Task 7.1** [Tier 3]:
- WHERE: `src/ai_client.py` lines 113-135 (the 12 module-level aliases)
- WHAT: delete the 12 alias declarations. Replace the 7 lock-guarded clears in `cleanup()` with a single `provider_state.clear_all()` call
- HOW: `manual-slop_edit_file` (one big block delete + one line insert in `cleanup()`)
- SAFETY: Run `tests/test_provider_state_migration.py` + all 7 per-provider test files. The `clear_all()` call iterates `_PROVIDER_HISTORIES.values()` and calls `.clear()` on each (with the RLock acquired per-history). Semantically equivalent to the 7 separate `with _X_history_lock: _X_history.clear()` blocks.
- [x] **COMMIT:** `refactor(ai_client): remove 12 module-level provider_state aliases; cleanup() uses clear_all()` [da66adf] (Tier 3, atomic)
- [x] **GIT NOTE:** 12 module-level aliases deleted. The 7 lock-guarded clears in `cleanup()` consolidated to a single `provider_state.clear_all()` call. Net diff: -10 lines (12 alias deletions - 2 added imports/comments).
## Phase 8: Verification + end-of-track (1 task, 3 commits)
**Focus:** Run all 8 VCs; write `TRACK_COMPLETION`; update `state.toml` + `tracks.md`.
- [x] **Task 8.1** [Tier 2]:
- WHERE: terminal + `docs/reports/TRACK_COMPLETION_code_path_audit_phase_3_provider_state_20260624.md` (NEW)
- WHAT:
- VC1-VC8 verification (see spec.md §Verification Criteria)
- Re-measure effective codepaths: expected UNCHANGED at 4.014e+22 (the migration removes 1 branch from `cleanup()` only; not visible in 2^N sum)
- Run the full 7 audit gates + batched test suite
- Document the result: 10/11 tiers PASS (1 pre-existing RAG flake); 7/7 audit gates PASS
- Document why VC7 (effective codepaths) didn't change: the metric is dominated by `2^N` for the highest-branch-count functions; removing 1 branch from 1 function changes the total by < 0.01%
- HOW: Run each command, capture output, write the report
- COMMIT: 3 commits: state, TRACK_COMPLETION, tracks.md update
- VERIFY: All 8 VCs pass
## Commit Log (Expected, 11 atomic commits)
1. (Phase 0) `test(provider_state): add migration regression-guard suite` (Tier 3)
2. (Phase 1) `refactor(ai_client): migrate _anthropic_history call sites to provider_state.get_history("anthropic")` (Tier 3)
3. (Phase 2) `refactor(ai_client): migrate _deepseek_history call sites to provider_state.get_history("deepseek")` (Tier 3)
4. (Phase 3) `refactor(ai_client): migrate _grok_history call sites to provider_state.get_history("grok")` (Tier 3)
5. (Phase 4) `refactor(ai_client): migrate _minimax_history call sites to provider_state.get_history("minimax")` (Tier 3)
6. (Phase 5) `refactor(ai_client): migrate _qwen_history call sites to provider_state.get_history("qwen")` (Tier 3)
7. (Phase 6) `refactor(ai_client): migrate _llama_history call sites to provider_state.get_history("llama")` (Tier 3)
8. (Phase 7) `refactor(ai_client): remove 12 module-level provider_state aliases; cleanup() uses clear_all()` (Tier 3)
9. (Phase 8) `conductor(state): code_path_audit_phase_3_provider_state_20260624 SHIPPED` (Tier 2)
10. (Phase 8) `docs(reports): TRACK_COMPLETION_code_path_audit_phase_3_provider_state_20260624` (Tier 2)
11. (Phase 8) `conductor(tracks): add code_path_audit_phase_3_provider_state_20260624 row` (Tier 2)
Plus per-task plan-update commits per the workflow.
## Verification Commands (run at end of Phase 8)
```bash
# VC1: 12 module-level aliases removed
git grep -E "_anthropic_history:|_anthropic_history = |_anthropic_history_lock:|_anthropic_history_lock = " master:src/ai_client.py | wc -l
# Expect: 0
# VC2: 26 call sites migrated
git grep -E "_anthropic_history\b|_deepseek_history\b|_minimax_history\b|_qwen_history\b|_grok_history\b|_llama_history\b" master:src/ai_client.py | wc -l
# Expect: 0
# VC3: cleanup() uses provider_state.clear_all()
git grep "_anthropic_history = \[\]\|_anthropic_history_lock" master:src/ai_client.py | wc -l
# Expect: 0
# VC4: Per-provider regression tests
uv run python -m pytest tests/test_provider_state_migration.py tests/test_anthropic_provider.py tests/test_deepseek_provider.py tests/test_grok_provider.py tests/test_minimax_provider.py tests/test_qwen_provider.py tests/test_llama_provider.py tests/test_llama_ollama_native.py -v
# Expect: all pass
# VC5: All 7 audit gates pass
uv run python scripts/audit_weak_types.py --strict
uv run python scripts/generate_type_registry.py --check
uv run python scripts/audit_main_thread_imports.py
uv run python scripts/audit_no_models_config_io.py
uv run python scripts/audit_code_path_audit_coverage.py --input-dir docs/reports/code_path_audit/2026-06-22 --strict
uv run python scripts/audit_exception_handling.py --strict
uv run python scripts/audit_optional_in_3_files.py --strict
# All exit 0
# VC6: Batched test tiers
uv run python scripts/run_tests_batched.py
# Expect: 10/11 PASS, 1 pre-existing RAG flake
# VC7: Effective codepaths unchanged
uv run python -c "from src.code_path_audit import build_pcg; from src.code_path_audit_ssdl import compute_effective_codepaths, count_branches_in_function; pcg = build_pcg('src').data; total = sum(2 ** count_branches_in_function(f, 'src') for f in pcg.consumers.get('Metadata', [])); print(f'{total:.3e}')"
# Expect: 4.014e+22 (unchanged)
# VC8: End-of-track report exists
cat docs/reports/TRACK_COMPLETION_code_path_audit_phase_3_provider_state_20260624.md
```
## Notes for Tier 3 workers
- **Pattern consistency:** For each site, the canonical pattern is `history = provider_state.get_history("X"); ... use history.append(...) ...`. Capture to a local variable if the same provider is used 3+ times in a function.
- **Lock acquisition:** Inside `with history.lock:` blocks, the lock is already held; subsequent `history.append(...)` etc. will use the same RLock instance (re-entrant — no deadlock).
- **Indentation:** 1-space per level (project standard). Use `manual-slop_edit_file` for surgical edits.
- **No comments:** per AGENTS.md "No comments in source code."
- **No new imports:** the `from src import provider_state` is already at the top of `src/ai_client.py`.
## Notes for Tier 2 reviewer
- After each per-provider commit, run the full batched test suite to catch any unexpected regressions (thread-safety tests, RAG engine init, etc.).
- The RLock re-entrance is the critical correctness property. If any test that previously DEADLOCKed now passes — that's the signal the migration is correct.
- If a per-provider commit causes a regression, **revert** the commit and investigate (don't try to fix forward; the prior state is the known-good baseline).
conductor/tracks/code_path_audit_phase_3_provider_state_20260624/spec.md
-191
View File
@@ -1,191 +0,0 @@
# Track Specification: code_path_audit_phase_3_provider_state_20260624
## Overview
The actual fix for the 4 NG2 violations and 1 partial NG2 violation left by `code_path_audit_phase_2_20260624` (the previous Tier 2 work). Phase 2 made `src/aggregate.py`'s `_build_files_section_from_items` use `NIL_METADATA` (good), but the actual fix for the 27 alias-based call sites in `src/ai_client.py` was deferred. This track fully migrates the 27 call sites from `_X_history` aliases to direct `provider_state.get_history("...").get_all()` / `.append(...)` / `with get_history("...").lock:` patterns.
## Current State Audit (master `22c76b95`, measured 2026-06-24)
| Metric | Value | Source |
|---|---:|---|
| `_anthropic_history` aliases in `src/ai_client.py` | 1 module-level alias + 10 call sites | `git grep` |
| `_deepseek_history` aliases | 1 + 6 call sites | `git grep` |
| `_minimax_history` aliases | 1 + 2 call sites | `git grep` |
| `_qwen_history` aliases | 1 + 2 call sites | `git grep` |
| `_grok_history` aliases | 1 + 2 call sites | `git grep` |
| `_llama_history` aliases | 1 + 4 call sites | `git grep` |
| **Total module-level aliases** | 6 `_X_history` + 6 `_X_history_lock` (12 module globals) | `git show HEAD:src/ai_client.py | head -140` |
| **Total call sites** | 26 references to `_X_history` (not counting the alias declarations) | `git grep` |
| Lock pattern usages | 12 `with _X_history_lock:` blocks | `git grep` |
| Effective codepaths (4.014e+22) | UNCHANGED (Phase 2 did not address) | `src/code_path_audit_ssdl.compute_effective_codepaths` |
| `provider_state.ProviderHistory` | Uses `threading.RLock` (post-cc7993e5 deadlock fix) | `src/provider_state.py:29` |
### Why this matters
The aliases `_anthropic_history = provider_state.get_history("anthropic")` mean consumers still use the bare variable name. The aliases work functionally (they reference the same `ProviderHistory` instance), but:
1. **The structural goal is not met** — `provider_state` was supposed to ENCAPSULATE the per-provider state behind a 4-method interface. The aliases break the encapsulation by exposing the bare `ProviderHistory` as a module-level name.
2. **The 4 NG2 (`Optional[T]` return-type) violations are still partially unresolved** — the legacy wrappers like `get_current_tier()` are at 1-space module-level; the canonical `get_current_tier_result()` exists but the bare name still appears in some callsites. The aliases mirror this pattern.
3. **The 4.01e22 combinatoric explosion is unchanged** — the metric is dominated by `2^branches` for the highest-branch-count functions. Removing 1 branch from 1 function changes the total by < 0.01%. The structural improvement is in API surface (typed `ProviderHistory` + `RLock` + re-entrant dunders), but the actual combinatoric reduction requires reducing `dict[str, Any]` type-dispatch branches. THAT is the parent plan's goal, deferred.
4. **The `T | None` workaround in 4 legacy wrappers** is technically compliant (the audit only flags `Optional[T]` AST subscripts) but is a heuristic bypass of the convention's spirit. Migrating to `_result()` pattern + consumers is the proper fix.
## Goals
| ID | Goal | Acceptance |
|---|---|---|
| G1 | Remove all 12 module-level aliases in `src/ai_client.py` (lines 113-135) | `git grep "_anthropic_history:\|_anthropic_history = provider_state" master:src/ai_client.py` returns 0 hits |
| G2 | Migrate all 26 call sites to use `provider_state.get_history("...")` directly | `git grep -E "_anthropic_history\b\|_deepseek_history\b\|_minimax_history\b\|_qwen_history\b\|_grok_history\b\|_llama_history\b" master:src/ai_client.py` returns 0 hits |
| G3 | Per-provider migration (6 vendors, 1 commit each) | 6 atomic commits, one per vendor, each with regression-guard tests |
| G4 | Add `tests/test_provider_state_migration.py` — verify no regression | All 12 `test_provider_state` tests pass + 7 `test_deepseek_provider` + 5 `test_anthropic` + 4 `test_grok_provider` + 4 `test_minimax_provider` + 5 `test_qwen_provider` + 6 `test_llama_provider` + 1 `test_llama_ollama_native` |
| G5 | `cleanup()` function uses `provider_state.clear_all()` | `git grep "_anthropic_history = \[\]\|_anthropic_history_lock" master:src/ai_client.py` returns 0 hits |
| G6 | All 7 audit gates pass `--strict` (no regression) | `weak_types` 102 ≤ 112; `type_registry` 23 files; `main_thread_imports` 17 files; `no_models_config_io` 0; `code_path_audit_coverage` 0; `exception_handling` 0; `optional_in_3_files` 0 |
| G7 | Full test suite remains green (10/11 tiers PASS — same as before) | `scripts/run_tests_batched.py` → 10/11 PASS, 1 pre-existing RAG flake |
## Non-Goals
- Modifications to `src/provider_state.py` (the migration is on the consumer side; the ProviderHistory interface is already correct after `cc7993e5`).
- The 4 NG1 (`INTERNAL_OPTIONAL_RETURN`) violations in `external_editor.py` + `session_logger.py` + `project_manager.py` — already addressed in Phase 2 by `ee4287ae`.
- The 4 `T | None` legacy wrappers — these are technically compliant per the audit. The bypass is documented in `docs/reports/REVIEW_TIER2_code_path_audit_phase_2_20260624.md` "Finding 8" as a followup. Defer to a separate track.
- The 4.01e22 combinatoric explosion — the actual fix is type promotion (`dict[str, Any]` → typed dataclass), which is the parent `any_type_componentization_20260621` track. Phase 2 + Phase 3 only address the API surface, not the type-dispatch branches.
- RAG test flake (`test_rag_phase4_final_verify`) — pre-existing, Windows-specific (sentence_transformers download / chroma lock); out of scope.
## Functional Requirements
### FR1: Remove the 12 module-level aliases (lines 113-135)
```python
# DELETE lines 113-135 of src/ai_client.py
_anthropic_history = provider_state.get_history("anthropic")
_anthropic_history_lock = _anthropic_history.lock
_deepseek_history = provider_state.get_history("deepseek")
_deepseek_history_lock = _deepseek_history.lock
# ... (minimax, qwen, grok, llama) ...
```
The aliases become unused. The 7 SDK client holders (`_anthropic_client`, `_deepseek_client`, etc.) are NOT deleted — they stay as module-level `Any` variables per Phase 2 spec ("SDK client holders stay as module-level `Any` variables per Pattern 3 (heterogeneous SDK types, lazy-initialized). Only the homogeneous history aspect is unified.").
### FR2: Per-provider migration (6 vendors)
For each provider, replace `_X_history` with `provider_state.get_history("X")` + the appropriate dunder or method call:
| Pattern | Replacement |
|---|---|
| `for msg in _X_history:` | `for msg in provider_state.get_history("X"):` |
| `if not _X_history:` | `if not provider_state.get_history("X"):` |
| `_X_history.append(msg)` | `provider_state.get_history("X").append(msg)` |
| `with _X_history_lock:` | `with provider_state.get_history("X").lock:` |
| `_X_history[i]`, `_X_history[-1]`, `_X_history[:n]` | `provider_state.get_history("X")[i]`, etc. |
| `len(_X_history)` | `len(provider_state.get_history("X"))` |
| `for msg in _X_history:` (inside the `with lock:` block) | `_X_history_local = provider_state.get_history("X"); for msg in _X_history_local:` (capture once to avoid repeated lock acquisitions) |
**Optimization:** for tight loops or repeated accesses, capture the history to a local variable once:
```python
history = provider_state.get_history("anthropic")
for msg in history:
...
history.append(...)
```
This is more readable AND avoids 2-3 lock acquisitions per iteration.
### FR3: Per-provider commit structure
| Commit | Provider | Site count | Verification |
|---|---|---|---|
| 1 | anthropic | 10 sites (lines 1452-1591) | `test_anthropic_*` + `test_ai_client_result` pass |
| 2 | deepseek | 6 sites (lines 2211-2430) | `test_deepseek_provider` (7 tests) + `test_ai_client_tool_loop*` pass |
| 3 | minimax | 2 sites (lines 2673-2676) | `test_minimax_provider` (4 tests) pass |
| 4 | qwen | 2 sites (lines 2826-2835) | `test_qwen_provider` (5 tests) pass |
| 5 | grok | 2 sites (lines 2586-2597) | `test_grok_provider` (4 tests) pass |
| 6 | llama | 4 sites (lines 2916-3029) | `test_llama_provider` (5 tests) + `test_llama_ollama_native` (5 tests) pass |
Each commit: 1 file (`src/ai_client.py`), 1 per-provider pattern, regression-guard test run.
### FR4: `cleanup()` function uses `provider_state.clear_all()`
Currently (lines 463-499 in `src/ai_client.py`):
```python
with _anthropic_history_lock:
_anthropic_history.clear()
# ... 5 more similar blocks for deepseek, minimax, qwen, grok, llama ...
```
Replace with:
```python
provider_state.clear_all()
```
Single call. Less code, same behavior.
### FR5: Re-audit (G6)
After all 6 per-provider commits + the cleanup() commit:
```bash
uv run python -c "from src.code_path_audit import build_pcg; from src.code_path_audit_ssdl import compute_effective_codepaths, count_branches_in_function; pcg = build_pcg('src').data; total = sum(2 ** count_branches_in_function(f, 'src') for f in pcg.consumers.get('Metadata', [])); print(f'{total:.3e}')"
```
Expected: same 4.014e+22 (no combinatoric reduction; the metric is dominated by 2^N). Document the unchanged number in the end-of-track report.
## Non-Functional Requirements
- NFR1: 1-space indentation (per `conductor/workflow.md`)
- NFR2: CRLF line endings on Windows
- NFR3: No comments in source code
- NFR4: Per-task atomic commits with git notes
- NFR5: No new pip dependencies
- NFR6: `Result[T]` returns for fallible fns (per `error_handling.md`)
- NFR7: No new `src/<thing>.py` files (per AGENTS.md)
## Architecture Reference
- `conductor/code_styleguides/error_handling.md` — the `Result[T]` convention (the reference for the NG2 wrappers)
- `conductor/code_styleguides/data_oriented_design.md` — the "Prefer Fewer Types" principle (motivates Phase 3)
- `conductor/tracks/code_path_audit_phase_2_20260624/spec.md` — the parent plan (where the aliases were introduced)
- `conductor/tracks/any_type_componentization_20260621/plan.md` — the grandparent plan (the 27 call sites came from the parent plan's 48 call-site migrations)
- `src/code_path_audit_ssdl.py` — `compute_effective_codepaths` (the measurement function for FR5)
- `src/provider_state.py` — the ProviderHistory interface (post-cc7993e5: RLock, removed copy-paste bugs)
- `src/ai_client.py:113-135` — the 12 module-level aliases to be removed
- `src/ai_client.py:1452-1591, 2211-2430, 2586-2597, 2673-2676, 2826-2835, 2916-3029` — the 26 call sites per provider
- `docs/reports/REVIEW_TIER2_code_path_audit_phase_2_20260624.md` — the review that identified the partial work + the R4 fabrication
## Out of Scope
- Modifications to `src/provider_state.py` (the migration is on the consumer side; ProviderHistory interface is already correct)
- The 4 `T | None` legacy wrappers (technically compliant per the audit; documented bypass; defer to followup track)
- The 4.01e22 combinatoric explosion (requires type promotion, not alias removal; grandparent plan scope)
- RAG test flake (`test_rag_phase4_final_verify`) — pre-existing, Windows-specific
- New `src/<thing>.py` files (per AGENTS.md hard rule)
## Verification Criteria (Definition of Done)
| # | Criterion | Verification command |
|---|---|---|
| VC1 | All 12 module-level aliases removed | `git grep -E "_anthropic_history:\|_anthropic_history = \|_anthropic_history_lock:\|_anthropic_history_lock = " master:src/ai_client.py` returns 0 hits |
| VC2 | All 26 call sites migrated | `git grep -E "_anthropic_history\b\|_deepseek_history\b\|_minimax_history\b\|_qwen_history\b\|_grok_history\b\|_llama_history\b" master:src/ai_client.py` returns 0 hits |
| VC3 | `cleanup()` uses `provider_state.clear_all()` | `git grep "_anthropic_history = \[\]\|_anthropic_history_lock" master:src/ai_client.py` returns 0 hits |
| VC4 | Per-provider regression tests pass | 7+5+4+4+5+5+5+1 = 36 tests across 8 test files all pass |
| VC5 | All 7 audit gates pass `--strict` (no regression) | Same as Phase 2 final state (7/7 PASS) |
| VC6 | 10/11 batched test tiers PASS (RAG flake acceptable) | `scripts/run_tests_batched.py` → 10/11 |
| VC7 | Effective codepaths metric documented (unchanged) | TRACK_COMPLETION report shows 4.014e+22 with explanation |
| VC8 | End-of-track report written | `docs/reports/TRACK_COMPLETION_code_path_audit_phase_3_provider_state_20260624.md` exists |
## Risks
| # | Risk | Likelihood | Mitigation |
|---|---|---|---|
| R1 | Migration breaks the regression-guard tests (`test_ai_client_result` for thread-safety, `test_provider_state` for ProviderHistory API) | medium | Per-provider commits with regression-guard test runs after each; revert + fix if any test fails |
| R2 | The `for msg in _X_history` pattern inside `with _X_history_lock:` is missed during migration → 2 different lock-acquisition patterns interleaved | low | Capture `_X_history` to a local variable once: `history = provider_state.get_history("X"); for msg in history: ...` inside the `with history.lock:` block |
| R3 | Some sites use `_X_history` inside a function that ALSO has `_X_history_lock` as a parameter (not just the alias) | low | Search for `_X_history_lock` as parameter vs alias; aliases are top-level only |
| R4 | The `clear_all()` change to `cleanup()` breaks thread-safety guarantees (e.g., a concurrent `send()` reads while `cleanup()` clears) | low | `clear_all()` iterates with each ProviderHistory's own lock; same as the current per-provider code. No semantic change. |
| R5 | The RLock re-entrance causes subtle behavior differences (e.g., a method called inside `with history.lock:` may now see different lock state than before) | low | All call sites in `src/ai_client.py` acquire the lock OUTSIDE the inner dunder calls. The deadlock fix already validated this for `_send_deepseek`. |
## See also
- `docs/reports/REVIEW_TIER2_code_path_audit_phase_2_20260624.md` — the review that identified this track
- `conductor/tracks/code_path_audit_phase_2_20260624/spec.md` — the parent track
- `conductor/tracks/code_path_audit_phase_2_20260624/plan.md` — the parent's plan
- `conductor/tracks/any_type_componentization_20260621/plan.md` — the grandparent track
- `conductor/code_styleguides/error_handling.md` — the convention
- `src/provider_state.py` — the ProviderHistory interface
- `src/ai_client.py:113-135, 1452-3029` — the migration sites
conductor/tracks/code_path_audit_phase_3_provider_state_20260624/state.toml
-62
View File
@@ -1,62 +0,0 @@
# Track state for code_path_audit_phase_3_provider_state_20260624
# Updated by Tier 2 Tech Lead as tasks complete
[meta]
track_id = "code_path_audit_phase_3_provider_state_20260624"
name = "Provider State Call-Site Migration"
status = "completed"
current_phase = 8
last_updated = "2026-06-25"
[blocked_by]
code_path_audit_phase_2_20260624 = "shipped"
[blocks]
[phases]
phase_0 = { status = "completed", checkpointsha = "283569d8", name = "Pre-flight verification + regression-guard test" }
phase_1 = { status = "completed", checkpointsha = "34a1e731", name = "Migrate anthropic (10 sites)" }
phase_2 = { status = "completed", checkpointsha = "35c708de", name = "Migrate deepseek (6 sites) + deadlock verification" }
phase_3 = { status = "completed", checkpointsha = "0e5cb2d4", name = "Migrate grok (2 sites)" }
phase_4 = { status = "completed", checkpointsha = "9a1812b2", name = "Migrate minimax (2 sites)" }
phase_5 = { status = "completed", checkpointsha = "46d44420", name = "Migrate qwen (2 sites)" }
phase_6 = { status = "completed", checkpointsha = "beb9d3f6", name = "Migrate llama (4 sites)" }
phase_7 = { status = "completed", checkpointsha = "6fc6364d", name = "Remove aliases + cleanup() simplification" }
phase_8 = { status = "completed", checkpointsha = "ed9a3099", name = "Verification + end-of-track report" }
[tasks]
t0_1 = { status = "completed", commit_sha = "cc7993e5", description = "Verify provider_state.ProviderHistory uses RLock (post-cc7993e5)" }
t0_2 = { status = "completed", commit_sha = "eddb3597", description = "Verify 7 audit gates pass --strict; 10/11 batched tiers PASS" }
t0_3 = { status = "completed", commit_sha = "4e947804", description = "Create tests/test_provider_state_migration.py with 6 per-provider regression-guard tests + thread-safety" }
t1_1 = { status = "completed", commit_sha = "2323b529", description = "Migrate _anthropic_history to provider_state.get_history('anthropic') (13 sites in lines 1430-1575)" }
t2_1 = { status = "completed", commit_sha = "79d0a563", description = "Migrate _deepseek_history to provider_state.get_history('deepseek') (11 sites in lines 2186-2414) + verify RLock no-deadlock" }
t3_1 = { status = "completed", commit_sha = "94a136ca", description = "Migrate _grok_history to provider_state.get_history('grok') (8 sites in _send_grok + kwargs)" }
t4_1 = { status = "completed", commit_sha = "7d2ce8f8", description = "Migrate _minimax_history to provider_state.get_history('minimax') (9 sites in _send_minimax)" }
t5_1 = { status = "completed", commit_sha = "81e013d7", description = "Migrate _qwen_history to provider_state.get_history('qwen') (6 sites in _send_qwen)" }
t6_1 = { status = "completed", commit_sha = "fd566133", description = "Migrate _llama_history to provider_state.get_history('llama') (16 sites in _send_llama + _send_llama_native)" }
t7_1 = { status = "completed", commit_sha = "da66adfe", description = "Remove 12 module-level aliases (lines 113-135)" }
t8_1 = { status = "completed", commit_sha = "ed9a3099", description = "Run all 8 VCs; write TRACK_COMPLETION; update state.toml + tracks.md" }
[verification]
phase_0_complete = true
phase_1_complete = true
phase_2_complete = true
phase_3_complete = true
phase_4_complete = true
phase_5_complete = true
phase_6_complete = true
phase_7_complete = true
phase_8_complete = true
vc1_aliases_removed = true
vc2_call_sites_migrated = true
vc3_cleanup_uses_clear_all = true
vc4_per_provider_tests_pass = true
vc5_audit_gates_pass = true
vc6_batched_tiers_pass = true
vc7_effective_codepaths_unchanged = true
vc8_end_of_track_report = true
[track_specific]
audit_count_progression = { baseline: "112 weak sites (Phase 2 final)", final: "102 weak sites", delta: "-10 weak sites via typed provider_state paths" }
risk_reduction = "R5 (RLock re-entrance) verified by test_lock_acquisition_no_deadlock across all 6 providers + concurrent append thread-safety + nested function calls inside with history.lock: blocks"
effective_codepaths_unchanged = "4.014e+22 (verified; migration removes 1 branch from cleanup() only; combinatoric reduction is the parent any_type_componentization_20260621 track's scope)"