|
|
|
@@ -0,0 +1,237 @@
|
|
|
|
|
# Track: Meta-Tooling Duration Analysis
|
|
|
|
|
|
|
|
|
|
**Status:** Spec drafted 2026-07-05; awaiting user review
|
|
|
|
|
**Initialized:** 2026-07-05
|
|
|
|
|
**Owner:** Tier 1 Orchestrator (sole author of spec; Tier 2 Tech Lead executes the audit script per plan)
|
|
|
|
|
**Priority:** Informational (one-shot analysis; no app behavior change)
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
## Overview
|
|
|
|
|
|
|
|
|
|
Produce an empirical duration analysis of the meta-tooling that built this codebase from Feb 21 → Jul 5, 2026 (~133 days; 244 tracks; 1,946 track-folder commits; 1,005 sub-agent task logs; 317 session logs; 1,719 work-prefix commits globally).
|
|
|
|
|
|
|
|
|
|
The deliverable is a **read-only analysis** in the Meta-Tooling domain (per `docs/guide_meta_boundary.md §0`): an audit script that consumes git history + on-disk log files and emits a Markdown report + JSON source-of-truth. **No Application-domain changes.** No `src/`, no `gui_2.py`, no MMA engine, no GUI panel.
|
|
|
|
|
|
|
|
|
|
The user has ~3-6 months of LLM-assisted dev history and wants hard numbers to set realistic expectations for future tracks. The fundamental question: **"Given a future track with N files to modify and M scope, how long should I expect it to take, accounting for the era (model quality) it's happening in?"**
|
|
|
|
|
|
|
|
|
|
## Current State Audit (as of `4c9fc99c`)
|
|
|
|
|
|
|
|
|
|
### Already Exists (DO NOT re-implement; build on these)
|
|
|
|
|
|
|
|
|
|
1. **`scripts/audit/generate_chronology.py:329-411`** — already extracts per-track `init_sha`, `end_sha`, `commit_count`, `date`, `folder_link` by walking `conductor/tracks/` + `conductor/archive/`. **Build on this**: import `_count_work_commits`, `_git_log`, `walk_track_folders`, `_repo_root` from this module instead of duplicating the git plumbing.
|
|
|
|
|
2. **`conductor/chronology.md`** (248 lines, 244 rows, regenerated by `generate_chronology.py`) — the canonical per-track index. The new analyzer consumes this rather than re-walking the filesystem.
|
|
|
|
|
3. **`docs/reports/`** directory — established convention for analysis output. Existing reports include `TRACK_COMPLETION_*`, `TRACK_ABORTED_*`, `PLANNING_DIGEST_*`, `SESSION_REPORT_*`. The new report joins this set as `META_TOOLING_DURATION_ANALYSIS_<date>.md`.
|
|
|
|
|
4. **`logs/agents/mma_tier3-worker_task_*.log`, `mma_tier4-qa_task_*.log`, `mma_tier1_task_*.log`** — 1,005 sub-agent task logs with parseable `TIMESTAMP:` + `ROLE:` + `RESULT:` blocks. Established format per `scripts/audit/generate_chronology.py:43-58` which already references these.
|
|
|
|
|
5. **`logs/<session>/`** directories (317 of them) — the higher-level chat session containers; each session wraps multiple sub-agent tasks. The session name pattern is `YYYYMMDD_HHMMSS[_<suffix>]` per the existing naming.
|
|
|
|
|
6. **`config.toml`** — the runtime configuration. Its git history is the strongest single signal for era boundaries (provider/model line changes).
|
|
|
|
|
7. **`conductor/workflow.md §0` and §"Conductor Token Firewalling"** — the canonical tier-model assignments (Tier 1 = `gemini-3.1-pro-preview`, Tier 2 = `gemini-3-flash-preview`, Tier 3/4 = `gemini-2.5-flash-lite`). This is the model-assumption baseline.
|
|
|
|
|
|
|
|
|
|
### Gaps to Fill (this track's scope)
|
|
|
|
|
|
|
|
|
|
- An era-detection algorithm using multi-signal weighted-vote (config.toml + workflow.md + track-spec content + agent-log content). No single source is authoritative.
|
|
|
|
|
- A per-track duration model (init_sha → end_sha in calendar days, in work-commits, in files-touched).
|
|
|
|
|
- A per-op duration model across 3 op layers:
|
|
|
|
|
- **Op Layer A: Sub-agent task logs** (`logs/agents/mma_tier*_task_*.log`) — parsed per-task duration (prompt received → result returned)
|
|
|
|
|
- **Op Layer B: Work-prefix commits** (`feat|fix|refactor|perf|test`) cross-cut by commit-prefix family
|
|
|
|
|
- **Op Layer C: Session logs** (`logs/<session>/`) — the top-level chat session durations
|
|
|
|
|
- Track-attribution: which sub-agent tasks / commits land inside a known track folder vs which are "simple todos" (no track context).
|
|
|
|
|
- Era-bucketed statistics: every metric broken down by detected era with the evidence chain.
|
|
|
|
|
- A use-case prediction table: "for a track with N files in subsystem S, median duration is X" — derived from the per-track dataset.
|
|
|
|
|
- The Markdown report (10 sections, ~300-500 lines) and JSON source-of-truth artifact.
|
|
|
|
|
|
|
|
|
|
## Goals
|
|
|
|
|
|
|
|
|
|
1. **Empirical, reproducible numbers.** Every duration in the report traces back to a specific JSON record; every era boundary traces back to a named git evidence chain.
|
|
|
|
|
2. **Multi-dimensional slicing.** Track-level, op-level (3 layers), era-level — the same data sliced multiple ways so the user can answer different questions.
|
|
|
|
|
3. **Self-documenting methodology.** §1 of the report explains exactly how eras were detected, what signals were weighted how, and what the failure modes are. Re-runs of the script (if any future agent runs it) produce the same output.
|
|
|
|
|
4. **No app changes.** This is strictly Meta-Tooling analysis per `docs/guide_meta_boundary.md §0`. Zero edits to `src/`, `gui_2.py`, `ai_client.py`, the MMA engine, or the GUI.
|
|
|
|
|
5. **Useful for expectation-setting.** §8 of the report has the use-case prediction table — the actual user-facing artifact for setting expectations.
|
|
|
|
|
|
|
|
|
|
## Non-Goals
|
|
|
|
|
|
|
|
|
|
- Not a re-runnable ongoing system. One-shot only (per user direction 2026-07-05).
|
|
|
|
|
- Not a live dashboard. No GUI changes.
|
|
|
|
|
- Not a predictive ML model. The use-case predictions are statistical aggregates (median + IQR), not a regression.
|
|
|
|
|
- Not measuring LLM "thinking" time or token costs (those are not in the data we have).
|
|
|
|
|
- Not measuring idle time between commits (we measure calendar days and work-commits, not active human/agent minutes).
|
|
|
|
|
- Not tracking which specific model was used per commit (the per-task model is not logged; we can only era-bucket).
|
|
|
|
|
- Not producing a CSV. JSON + Markdown only (per Q2 answer 2b).
|
|
|
|
|
- Not regenerating `conductor/chronology.md`. We consume it; we don't rewrite it.
|
|
|
|
|
|
|
|
|
|
## Architecture
|
|
|
|
|
|
|
|
|
|
### Data Flow
|
|
|
|
|
|
|
|
|
|
```
|
|
|
|
|
git log (entire repo, 4,997 commits)
|
|
|
|
|
↓
|
|
|
|
|
git log -- config.toml (era detection)
|
|
|
|
|
git log -- conductor/workflow.md (era detection)
|
|
|
|
|
git log -- conductor/tracks/*/spec.md (era detection, model mentions)
|
|
|
|
|
↓
|
|
|
|
|
generate_chronology.py:walk_track_folders() (reused) → track index
|
|
|
|
|
↓
|
|
|
|
|
parse logs/agents/mma_tier*_task_*.log (Op Layer A) → task index
|
|
|
|
|
↓
|
|
|
|
|
git log --grep='^feat|^fix|^refactor|^perf|^test' (Op Layer B) → commit-prefix index
|
|
|
|
|
↓
|
|
|
|
|
parse logs/<session>/* (Op Layer C) → session index
|
|
|
|
|
↓
|
|
|
|
|
[era detector: multi-signal weighted-vote on date candidates]
|
|
|
|
|
↓
|
|
|
|
|
[track-attributor: bucket each op to a track or 'track-less']
|
|
|
|
|
↓
|
|
|
|
|
[statistician: per-track, per-op, per-era, per-use-case aggregations]
|
|
|
|
|
↓
|
|
|
|
|
[renderer: 10-section Markdown report]
|
|
|
|
|
↓
|
|
|
|
|
[writer: docs/reports/META_TOOLING_DURATION_ANALYSIS_<date>.md]
|
|
|
|
|
[writer: tests/artifacts/meta_tooling_stats_<date>.json]
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
### Per-File Design
|
|
|
|
|
|
|
|
|
|
#### NEW: `scripts/audit/analyze_meta_tooling_durations.py`
|
|
|
|
|
|
|
|
|
|
A single-file audit script (~600-900 lines per `data_oriented_design.md §8.5` styleguide). Sections:
|
|
|
|
|
|
|
|
|
|
1. **Era detection** (`detect_eras()` — pure function)
|
|
|
|
|
- Inputs: `Path` to repo root
|
|
|
|
|
- Signal A: `git log --diff-filter=AM -G "provider = " -- config.toml` → list of `(date, evidence)` candidates
|
|
|
|
|
- Signal B: `git log --diff-filter=AM -G "gemini-3|gemini-2\.5|MiniMax|claude|grok" -- conductor/workflow.md conductor/tech-stack.md` → list of `(date, evidence)`
|
|
|
|
|
- Signal C: `grep -l "model:" logs/agents/mma_tier3-worker_task_*.log` (per-file mtime + content model mentions) → first-appearance dates
|
|
|
|
|
- Signal D: `git log -G "gemini-3-flash-preview|gemini-3.1-pro-preview|MiniMax" -- conductor/tracks/*/spec.md` → per-track model mentions grouped by track init date
|
|
|
|
|
- Algorithm: cluster candidates within 7-day windows; cluster centroid becomes era boundary; record all evidence in the era's record
|
|
|
|
|
- Output: `list[EraBoundary]` dataclass instances
|
|
|
|
|
|
|
|
|
|
2. **Track index** (`build_track_index()`)
|
|
|
|
|
- Imports `walk_track_folders`, `_git_log`, `_count_work_commits` from `scripts.audit.generate_chronology`
|
|
|
|
|
- Extends each row with: `init_date` (ISO), `end_date` (ISO), `duration_days`, `files_touched` (`git log --name-only --pretty=format: | sort -u | wc -l` per track folder), `era` (assigned by EraBoundary lookup)
|
|
|
|
|
|
|
|
|
|
3. **Op Layer A: sub-agent task index** (`build_subagent_task_index()`)
|
|
|
|
|
- Walks `logs/agents/mma_tier*_task_*.log` (1,005 files)
|
|
|
|
|
- Parses header: `TIMESTAMP:`, `ROLE:` (tier3-worker / tier4-qa / tier1), `RESULT:` block
|
|
|
|
|
- `in_track` attribution: if the task's timestamp falls inside any track's `[init_date, end_date]` window → that track; else `None` (track-less)
|
|
|
|
|
- Outputs: `list[SubAgentTaskOp]` dataclass
|
|
|
|
|
|
|
|
|
|
4. **Op Layer B: commit-prefix index** (`build_commit_prefix_index()`)
|
|
|
|
|
- `git log --grep='^(feat|fix|refactor|perf|test)(|\([^)]+\)):' --pretty=format:'%h|%ad|%s' --date=short`
|
|
|
|
|
- Same in-track attribution
|
|
|
|
|
- Outputs: `list[CommitPrefixOp]`
|
|
|
|
|
|
|
|
|
|
5. **Op Layer C: session log index** (`build_session_index()`)
|
|
|
|
|
- Walks `logs/` (excludes `logs/agents/`); 317 dirs
|
|
|
|
|
- Session start = earliest mtime of any file in `logs/<session>/`
|
|
|
|
|
- Session end = latest mtime
|
|
|
|
|
- Output: `list[SessionOp]`
|
|
|
|
|
|
|
|
|
|
6. **Statistician** (`compute_stats(track_idx, op_idx_a, op_idx_b, op_idx_c, eras)`)
|
|
|
|
|
- Per-track: median/p25/p75 of duration_days, commits, files touched
|
|
|
|
|
- Per-op-layer-per-era: median/p25/p75 of duration
|
|
|
|
|
- Per-week: commit count, task count, session count (moving avg)
|
|
|
|
|
- Per-use-case: tracks grouped by `files_touched` bucket → median duration per bucket
|
|
|
|
|
- All results in typed `@dataclass(frozen=True, slots=True)` per `data_oriented_design.md §8.5`
|
|
|
|
|
|
|
|
|
|
7. **Renderer** (10 functions, one per Markdown section)
|
|
|
|
|
- `render_methodology()`, `render_era_timeline()`, `render_track_stats()`, `render_op_stats()`, `render_subagent_stats()`, `render_trackless_ops()`, `render_trends()`, `render_use_case_predictions()`, `render_limitations()`, `render_appendix()`
|
|
|
|
|
- Each returns a `str`; concatenated into the final report
|
|
|
|
|
|
|
|
|
|
8. **Writer** (`write_artifacts(report_md, stats_json, output_dir)`)
|
|
|
|
|
- Writes `docs/reports/META_TOOLING_DURATION_ANALYSIS_<YYYY-MM-DD>.md`
|
|
|
|
|
- Writes `tests/artifacts/meta_tooling_stats_<YYYY-MM-DD>.json`
|
|
|
|
|
|
|
|
|
|
#### NEW: `tests/test_analyze_meta_tooling_durations.py`
|
|
|
|
|
|
|
|
|
|
Unit tests for the pure-function modules (era detection, statistician, parsers). Skips the git-process-spawning functions via mocks. ~150 lines.
|
|
|
|
|
|
|
|
|
|
#### NEW: `docs/reports/META_TOOLING_DURATION_ANALYSIS_<YYYY-MM-DD>.md`
|
|
|
|
|
|
|
|
|
|
The report. ~300-500 lines, 10 sections per design.
|
|
|
|
|
|
|
|
|
|
#### NEW: `tests/artifacts/meta_tooling_stats_<YYYY-MM-DD>.json`
|
|
|
|
|
|
|
|
|
|
The raw data. Schema per design §Q2.
|
|
|
|
|
|
|
|
|
|
### Convention Compliance
|
|
|
|
|
|
|
|
|
|
- **Python Type Promotion Mandate** (`data_oriented_design.md §8.5`): every dataclass uses `@dataclass(frozen=True, slots=True)`. No `dict[str, Any]`. No `Any`. No `Optional[T]` returns (use `Result[T]` + `NIL_T` per `error_handling.md`).
|
|
|
|
|
- **1-space indentation** per `conductor/code_styleguides/python.md §1`.
|
|
|
|
|
- **CRLF line endings** per `conductor/workflow.md §"Code Style"`.
|
|
|
|
|
- **ImGui scope discipline**: N/A (no GUI in this track).
|
|
|
|
|
- **Audit Script Policy** (`conductor/workflow.md §"Audit Script Policy"`): the script has a `--help`, supports `--json` (JSON-only output mode for re-analysis), and exits 0 on success.
|
|
|
|
|
- **No new `src/<thing>.py` file** (per `AGENTS.md §"File Size and Naming Convention"`).
|
|
|
|
|
|
|
|
|
|
## Functional Requirements
|
|
|
|
|
|
|
|
|
|
1. **FR1**: The script MUST be invokable as `uv run python scripts/audit/analyze_meta_tooling_durations.py` with no arguments and produce both artifacts.
|
|
|
|
|
2. **FR2**: The script MUST have `--help` output that documents the data sources, era detection algorithm, and limitations.
|
|
|
|
|
3. **FR3**: The script MUST support `--json-only` to emit only the JSON artifact (no Markdown), for downstream re-analysis.
|
|
|
|
|
4. **FR4**: The script MUST accept `--output-dir <path>` to override the default `docs/reports/` + `tests/artifacts/` destinations (CI/headless use).
|
|
|
|
|
5. **FR5**: Every era boundary in the report MUST cite the git evidence chain (commit SHA + the diff line that triggered the boundary).
|
|
|
|
|
6. **FR6**: The JSON artifact MUST be schema-versioned (`schema_version: "1"`) and contain `eras`, `tracks`, `ops` (3 lists: `subagent_tasks`, `commits`, `sessions`), `track_less_ops`, `metadata` (run timestamp + repo HEAD).
|
|
|
|
|
7. **FR7**: Per-era stats MUST include: track count, median duration_days, p25/p75 duration_days, median commits, median files_touched, op counts per layer.
|
|
|
|
|
8. **FR8**: Use-case predictions (§8) MUST group tracks by `files_touched` buckets (1-5, 6-15, 16-30, 31+) and report median + IQR duration_days per bucket per era.
|
|
|
|
|
9. **FR9**: The script MUST be deterministic — same input repo state produces same output (sorted, no timestamps in the analytical content).
|
|
|
|
|
10. **FR10**: The script MUST handle the case where some sub-agent task logs are malformed (missing `RESULT:` block) without crashing — log a warning, skip the entry, continue.
|
|
|
|
|
|
|
|
|
|
## Non-Functional Requirements
|
|
|
|
|
|
|
|
|
|
- **NFR1**: Script runtime: <120 seconds on the current repo state (4,997 commits, 1,005 log files, 317 sessions).
|
|
|
|
|
- **NFR2**: Script memory: <500MB peak.
|
|
|
|
|
- **NFR3**: The Markdown report MUST be human-readable without external tools (no Mermaid, no external CSS).
|
|
|
|
|
- **NFR4**: The Markdown report MUST contain a "Limitations" section enumerating what the analysis cannot tell the user (e.g., LLM "thinking" time, idle time, per-task model attribution).
|
|
|
|
|
- **NFR5**: The script MUST emit progress to stderr (line-buffered) so the user sees what's being computed (no silent 60-second stall).
|
|
|
|
|
- **NFR6**: Test coverage for the pure-function modules (era detection, statistician, parsers) MUST be ≥80% lines.
|
|
|
|
|
|
|
|
|
|
## Architecture Reference
|
|
|
|
|
|
|
|
|
|
- **Meta-Tooling vs Application domain distinction**: `docs/guide_meta_boundary.md §0` — confirms this is Meta-Tooling-only.
|
|
|
|
|
- **Per-source-file code audit pattern**: `conductor/code_styleguides/data_oriented_design.md §8.5` (Python Type Promotion Mandate).
|
|
|
|
|
- **Audit script convention**: `conductor/workflow.md §"Audit Script Policy"` — the audit + CI gate pair pattern.
|
|
|
|
|
- **Reused analysis primitives**: `scripts/audit/generate_chronology.py:329-411` (per-track walker).
|
|
|
|
|
- **Standard convention**: `conductor/code_styleguides/python.md §1` (1-space indent); `conductor/code_styleguides/error_handling.md` (Result[T] + NIL_T).
|
|
|
|
|
|
|
|
|
|
## Out of Scope
|
|
|
|
|
|
|
|
|
|
- **No app behavior change** — strictly Meta-Tooling.
|
|
|
|
|
- **No new `src/<thing>.py` file** — `scripts/audit/` is the only new namespace.
|
|
|
|
|
- **No re-runnability / CI integration** — one-shot; not a CI gate.
|
|
|
|
|
- **No CSV output** — Markdown + JSON only.
|
|
|
|
|
- **No predictive ML model** — statistical aggregates only.
|
|
|
|
|
- **No per-task model attribution** — we era-bucket by git-history evidence, not per-task logs.
|
|
|
|
|
- **No measurement of LLM "thinking" time or token costs** — data not present.
|
|
|
|
|
- **No GUI panel or live dashboard**.
|
|
|
|
|
|
|
|
|
|
## Risks
|
|
|
|
|
|
|
|
|
|
| Risk | Likelihood | Impact | Mitigation |
|
|
|
|
|
|---|---|---|---|
|
|
|
|
|
| Era detection produces too many/few eras (over/under-clustering) | Medium | Medium | 7-day cluster window is tunable via `--era-window-days`; report §1 documents the chosen window. |
|
|
|
|
|
| Sub-agent task log parsing breaks on malformed files | Medium | Low | FR10 + warning log. |
|
|
|
|
|
| Script runtime exceeds 120s on large repos | Low | Low | NFR1 budget; profile with `--profile` flag if needed. |
|
|
|
|
|
| Use-case predictions in §8 have too few samples per bucket per era (high variance) | Medium | Medium | §8 reports bucket sample sizes; warns when N < 5. |
|
|
|
|
|
| The user disagrees with the era boundaries I detect | Medium | Low | §1 documents the algorithm + evidence; user can rerun with `--era-window-days` or `--override-eras <csv>`. |
|
|
|
|
|
| Session log parsing is brittle (folder naming varies) | Medium | Low | Parse by `YYYYMMDD_HHMMSS[_<suffix>]` pattern; skip dirs that don't match. |
|
|
|
|
|
|
|
|
|
|
## Verification
|
|
|
|
|
|
|
|
|
|
1. **V1**: `uv run python scripts/audit/analyze_meta_tooling_durations.py --help` exits 0 with the documented help text.
|
|
|
|
|
2. **V2**: `uv run python scripts/audit/analyze_meta_tooling_durations.py` exits 0, produces `docs/reports/META_TOOLING_DURATION_ANALYSIS_2026-07-05.md` (≥300 lines), and produces `tests/artifacts/meta_tooling_stats_2026-07-05.json`.
|
|
|
|
|
3. **V3**: `uv run python scripts/audit/analyze_meta_tooling_durations.py --json-only` exits 0, produces only the JSON, and the JSON validates against the documented schema.
|
|
|
|
|
4. **V4**: `uv run pytest tests/test_analyze_meta_tooling_durations.py` passes with ≥80% line coverage on the pure-function modules.
|
|
|
|
|
5. **V5**: The Markdown report contains all 10 sections in the documented order.
|
|
|
|
|
6. **V6**: The JSON artifact contains the documented top-level keys (`schema_version`, `generated_at`, `eras`, `tracks`, `ops`, `track_less_ops`, `metadata`).
|
|
|
|
|
7. **V7**: Script runtime is <120s on the current repo state (timed via `Measure-Command`).
|
|
|
|
|
|
|
|
|
|
## Definition of Done
|
|
|
|
|
|
|
|
|
|
- [ ] All 7 verification criteria pass.
|
|
|
|
|
- [ ] All 10 functional requirements implemented.
|
|
|
|
|
- [ ] All 6 non-functional requirements met.
|
|
|
|
|
- [ ] The audit script is committed to `scripts/audit/analyze_meta_tooling_durations.py`.
|
|
|
|
|
- [ ] The unit tests are committed to `tests/test_analyze_meta_tooling_durations.py`.
|
|
|
|
|
- [ ] The Markdown report is committed to `docs/reports/META_TOOLING_DURATION_ANALYSIS_2026-07-05.md`.
|
|
|
|
|
- [ ] The JSON artifact is committed to `tests/artifacts/meta_tooling_stats_2026-07-05.json`.
|
|
|
|
|
- [ ] A chronology.md row added: `docs(chronology): add row for meta_tooling_duration_analysis_20260705 archive` (when the track folder is archived after shipping).
|
|
|
|
|
- [ ] This spec is reviewed and approved by the user before plan.md is written.
|
