Replaced convert_cookies.py + run_corpus.py + dedupe_corpus.py (+ the scratch merge_corpus.py) with a single extract_corpus.py that: converts cookies, fetches each URL's full conversation, MERGES overlapping conversations into one thread (union of posts, no duplicate threads/media), downloads ALL media (images + video/mp4) via gallery-dl itself (plain urllib 403'd on pbs.twimg.com and never fetched videos), and renders. Result: 4 merged threads, 33 media files (incl mp4), 0 missing links. cookies_netscape.txt gitignored.
fetch_thread_from_url now passes -o conversations=true (full threads, not single tweets), sorts posts chronologically, and sets root_post_id to the URL's target tweet. render_markdown builds front-matter from the target (root_post_id) post, not the earliest conversation tweet (fixes wrong @handle). Added dedupe_corpus.py: keeps deepest thread per conversation, deletes duplicates, keeps+marks unique branches (branch_of) + writes threads_index.json. 33 tests pass.
Scripts + workflow for extracting Twitter/X posts and threads into
Markdown with associated media. Mirrors the scripts/video_analysis/
pattern. Standalone requirement: zero imports from src/, conductor/,
or scripts.video_analysis — copy-pasteable to another repo with only
gallery-dl as the external dep.
5 modules: __init__.py, error_types.py (Result[T, ErrorInfo] +
ThreadData/PostData typed dataclasses), fetch_thread.py (gallery-dl
subprocess for URLs + html.parser fallback for local HTML),
download_media.py (stdlib urllib, idempotent), render_markdown.py
(YAML front-matter + per-post sections + ./media/ links).
Reference project: C:\projects\forth\bootslop — the corpus feeds
bootslop's scripts and reference-generation pipeline. Acceptance
corpus: 8 threads (@NOTimothyLottes x6 + @VPCOMPRESSB x2) extracted
to tests/artifacts/twitter_threads_corpus/. The ?s=20 quote-share
suffix on the @VPCOMPRESSB URLs must be stripped by fetch_thread.py
before acquisition (added to FR2 as URL normalization).
5 phases / 23 tasks. 8 verification criteria (VC1-VC8). TDD red-first
on the pure-function modules (render_markdown, types, media naming).
ed