download_media now shells out to gallery-dl (handles X.com auth/403/video->mp4) instead of urllib, returns the downloaded files, + media_files_for_post glob helper + --cookies CLI. render_markdown embeds images with  and videos with an HTML <video controls> tag (were plain links). extract_corpus uses the package download_media (one download path). Tests rewritten for the gallery-dl API. Corpus: 4 threads, 33 media (32 img embeds + 1 video), 0 missing.
Replaced convert_cookies.py + run_corpus.py + dedupe_corpus.py (+ the scratch merge_corpus.py) with a single extract_corpus.py that: converts cookies, fetches each URL's full conversation, MERGES overlapping conversations into one thread (union of posts, no duplicate threads/media), downloads ALL media (images + video/mp4) via gallery-dl itself (plain urllib 403'd on pbs.twimg.com and never fetched videos), and renders. Result: 4 merged threads, 33 media files (incl mp4), 0 missing links. cookies_netscape.txt gitignored.
fetch_thread_from_url now passes -o conversations=true (full threads, not single tweets), sorts posts chronologically, and sets root_post_id to the URL's target tweet. render_markdown builds front-matter from the target (root_post_id) post, not the earliest conversation tweet (fixes wrong @handle). Added dedupe_corpus.py: keeps deepest thread per conversation, deletes duplicates, keeps+marks unique branches (branch_of) + writes threads_index.json. 33 tests pass.
Script used calendar-day duration (end_date - init_date), which
inflated tracks that sat dormant between work and archive-move.
Example: external_editor_integration_20260308 reported 60 days
(init 2026-03-08, last archive-move 2026-05-07) when actual work
was 2 active days.
Corrected metric: active days = distinct calendar days with at
least one commit touching the track folder. All 30 top tracks
now show 2-7 active days, matching the user's expectation that
no track lasts more than a week.
Also fixed commit_count + files_touched for all 30 tracks (the
script's work-prefix-only count was always 0-1 for these).
Top 3 examples:
- external_editor_integration_20260308: 60d/1c/4f -> 2d/4c/8f
- code_path_audit_20260607: 18d/1c/7f -> 7d/23c/14f
- nagent_review_20260608: 12d/0c/20f -> 3d/53c/20f
Still stale (need recompute, not hand-fix):
- 3.1 per-era aggregates
- 8 use-case predictions
- 218 other tracks not in top-30
The script's commit_count and files_touched columns only counted
work-prefix commits (feat|fix|refactor|perf|test) with pathspecs
matching the track folder. Track folders contain only metadata
commits (conductor(*), chore(conductor), docs(track)), so the
script reported 0 commits for any track that did real work.
Corrected rows (now using total commits + total unique files
touching the track folder, any prefix):
- code_path_audit_20260607: 1->23 commits, 7->14 files
- qwen_llama_grok_integration_20260606: 0->25 commits, 4->8 files
- nagent_review_20260608: 0->53 commits, 4->20 files
- data_oriented_error_handling_20260606: 0->21 commits, 4->8 files
- superpowers_review_20260619: 0->63 commits, 8->16 files
Added 9.1 caveat explaining the manual corrections and the
methodology error that caused them.
thread_from_dict (JSON wire boundary) round-trips thread_data.json; download_media/render_markdown gain argparse main() + __main__ so the plan Task 5.3 pipeline runs via -m; dual-import added so all modules run standalone or via -m (G5). Pipeline integration test (html->names->render) + CLI tests cover the glue without network. TDD: red -> green (4 pipeline + 29 regression = 33 passed).
- Move conductor/tracks/meta_tooling_duration_analysis_20260705/ to
conductor/archive/meta_tooling_duration_analysis_20260705/ (preserves
history as a directory rename since the spec/plan/metadata files were
uncommitted; they're added to the archive in this commit).
- Prepend the track's row to conductor/chronology.md (newest-first).
- Note: spec.md, plan.md, metadata.json, state.toml are being committed
for the first time in this archive commit. Per project convention,
spec + plan are committed at archive time per workflow.md chronology
maintenance rules.
Added standalone Result[T] (frozen+slots, ok/err classmethods, is_ok) to error_types.py per canonical AND-over-OR error_handling.md (not the video_analysis _Ok|_Err sum type). render_markdown renders YAML front-matter from root post, per-post ## sections with reply markers, quote blockquotes, and ./media/<name> links; media_names mapping keeps render decoupled from download_media. TDD: red (no Result) -> green (7 render + 9 types passed).
The original build_track_index spawned 5+ git subprocesses per track
folder (244 tracks * 5 calls = 1200+ subprocesses). That exceeded the
120s NFR1 budget. Replaced with a single 'git log --all --name-only'
pass + a mutable _TrackAccumulator; reduces git subprocess count from
1200+ to 1.
Also adds 3 smoke tests for build_subagent_task_index,
build_session_index, build_track_index (covering previously untested
pure-function indexers). 23 tests pass; coverage up from 39% to 61%.
Scripts + workflow for extracting Twitter/X posts and threads into
Markdown with associated media. Mirrors the scripts/video_analysis/
pattern. Standalone requirement: zero imports from src/, conductor/,
or scripts.video_analysis — copy-pasteable to another repo with only
gallery-dl as the external dep.
5 modules: __init__.py, error_types.py (Result[T, ErrorInfo] +
ThreadData/PostData typed dataclasses), fetch_thread.py (gallery-dl
subprocess for URLs + html.parser fallback for local HTML),
download_media.py (stdlib urllib, idempotent), render_markdown.py
(YAML front-matter + per-post sections + ./media/ links).
Reference project: C:\projects\forth\bootslop — the corpus feeds
bootslop's scripts and reference-generation pipeline. Acceptance
corpus: 8 threads (@NOTimothyLottes x6 + @VPCOMPRESSB x2) extracted
to tests/artifacts/twitter_threads_corpus/. The ?s=20 quote-share
suffix on the @VPCOMPRESSB URLs must be stripped by fetch_thread.py
before acquisition (added to FR2 as URL normalization).
5 phases / 23 tasks. 8 verification criteria (VC1-VC8). TDD red-first
on the pure-function modules (render_markdown, types, media naming).
Deleted scripts/audit/generate_chronology.py + chronology_quality_gate.py
after repeated corruption incidents (auto-classifier drifted from
user intent, silently rewrote rows). chronology.md is now maintained
by hand: add a row at the top when a track ships/abandons/is archived.
Updated conductor/workflow.md §Chronology Maintenance and
conductor/tracks.md §Archiving a track to reflect manual maintenance.
Added 40 rows to conductor/chronology.md for the 2026-07-05 archive
batch, with archive-folder paths and deferral notes where applicable.
Removed 12 archived-track rows from conductor/tracks.md Active Tracks
table (rows 2, 3, 7c, 16, 17, 23b, 23c, 23d, 25, 26, 29, 29c) and
their track-detail anchors.
Wrote docs/reports/ARCHIVE_REVIEW_20260705.md: a categorized index
of unfinished work items from the archived tracks' state.toml files
that remain relevant to the codebase, with a ranked follow-up list.
Per agent_directives_consolidation_20260705 §3.4.
The 14-line §Data Structure Conventions section (the 'names for shapes'
pattern intro + the 16-alias list + canonical reference) is replaced
with a 1-line pointer to conductor/code_styleguides/type_aliases.md,
the canonical home for the 10 TypeAlias definitions + 11 per-aggregate
dataclasses + 5-pattern decision tree.
Net: 14 lines reduced to 1 line of pointer. The type-registry
auto-generation note (docs/type_registry/) is preserved as unique
project information.
Phase 3.2, 3.3, 3.4 all complete in this single commit batch.
Per agent_directives_consolidation_20260705 §3.4.
The 30-line §Data-Oriented Error Handling section (key principles +
incremental rollout plan + audit-script enforcement) is replaced with
a 1-line pointer to conductor/code_styleguides/error_handling.md, the
canonical home for the Result[T] + NIL_T convention.
Net: 30 lines reduced to 1 line. Duplicate content removed; the
canonical styleguide remains the source of truth.
Per agent_directives_consolidation_20260705 §3.4.
The 2 duplicate bullets (Indentation, Newlines) are replaced with a
1-line pointer to conductor/code_styleguides/python.md §1, the
canonical home for Python style. The 4 project-specific bullets
(Vertical Compaction, Region Blocks, Type Hinting, SDM) are
preserved — they are unique to this project.
Net: 6 lines reduced to 1 line of pointer + 4 preserved bullets.
Per agent_directives_consolidation_20260705 §3.3.
The 9-line §9 'No Diagnostic Noise in Production Code' section is
replaced with a 1-line pointer to the canonical home in
conductor/code_styleguides/python.md §AI-Agent Specific Conventions.
Net: 9 lines reduced to 1 line. Duplicate content removed; the
canonical styleguide remains the source of truth.
Per agent_directives_consolidation_20260705 §3.2.
The 14-line abridged §Process Anti-Patterns section (1-line summaries per
pattern) is promoted to the full canonical home: each of the 8 anti-patterns
now has the full Symptom + Rule content matching what was previously in
AGENTS.md. The 9th item (Workspace-Path Drift Pattern) was already
present here and is preserved.
AGENTS.md §Process Anti-Patterns is now a thin index pointing here
(per the previous commit in Phase 1.3). The canonical home for the full
process anti-pattern content is now conductor/workflow.md.
Net: 14 lines expanded to 80 lines (canonical expansion). 9 items
with full Symptom + Rule content. The 8 anti-patterns are now
locally accessible from the operational workflow file where they're
most actionable.
Per agent_directives_consolidation_20260705 §3.2.
The 25-line §Known Pitfalls section (the git restore/git checkout/git
reset hard ban with full rationale + correct non-destructive inspection
pattern) is replaced with a thin pointer to AGENTS.md §Critical Anti-Patterns,
the canonical project-wide home for HARD BANs.
Net: 25 lines reduced to 7 lines (72% reduction). The full content
remains in AGENTS.md where the 4 canonical HARD BANs are documented.
Per agent_directives_consolidation_20260705 §3.1.
The 8 anti-patterns (Deduction Loop, Report-Instead-of-Fix, Scope-Creep
Track-Doc, Inherited-Cruft, No Diagnostic Noise, Surrender, Verbose-Commit-
Message, Isolated Pass Verification Fallacy) are now 1-line summaries
with a pointer to conductor/workflow.md §Process Anti-Patterns, which
becomes the canonical home for these rules.
Net: 70 lines reduced to 13 lines (81% reduction). The full Symptom + Rule
content remains in conductor/workflow.md where it will be promoted to
canonical in Phase 2.2 of this track.
Per agent_directives_consolidation_20260705 §3.1.
The 5 items (ALWAYS use proper edit tool, decorator-orphan pitfall,
ast.parse is not enough, git status trap, small edits beat big scripts)
are now 1-line pointers to conductor/edit_workflow.md, which is the
canonical home for edit-tool lessons-learned (per its §1, §6, §7).
Net: 30 lines reduced to 7 lines (76% reduction in section size).
Duplicate content removed; conductor/edit_workflow.md remains the
canonical source for these rules.
Per agent_directives_consolidation_20260705 §3.1.
The 9 duplicated items (read full files, modify tech stack, skip TDD, skip
markers, batch commits, no comments, set_file_slice, git restore caveat)
are now 1-line summaries + pointers to the canonical homes:
- conductor/code_styleguides/python.md (LLM Default Anti-Patterns §17)
- conductor/code_styleguides/data_oriented_design.md §8.5 (Python
Type Promotion Mandate)
- conductor/edit_workflow.md (the edit tool contract)
- conductor/workflow.md §Known Pitfalls + Skip-Marker Policy
The 4 canonical HARD BANs (git restore / git checkout / git reset;
git stash*; day estimates; opaque types) remain inline as they are the
project-wide rules with no single canonical home. Each HARD BAN now
points to its technical mandate for the full rationale.
Net effect: AGENTS.md §Critical Anti-Patterns reduced from 14-line
section with 13 mixed items to 19-line section with 4 HARD BANs + a thin
index. The duplicated content is now in the canonical styleguides where it
belongs; the 4 HARD BANs are deduplicated to their one proper home here.
ed