ed/manual_slop
Private
Public Access
0
0
Fork 0
Code Activity

refactor(twitter_threads): consolidate corpus extraction into one script; gallery-dl downloads media

Browse Source 
Replaced convert_cookies.py + run_corpus.py + dedupe_corpus.py (+ the scratch merge_corpus.py) with a single extract_corpus.py that: converts cookies, fetches each URL's full conversation, MERGES overlapping conversations into one thread (union of posts, no duplicate threads/media), downloads ALL media (images + video/mp4) via gallery-dl itself (plain urllib 403'd on pbs.twimg.com and never fetched videos), and renders. Result: 4 merged threads, 33 media files (incl mp4), 0 missing links. cookies_netscape.txt gitignored.
This commit is contained in:
ed
2026-07-05 18:50:55 -04:00
parent ae9cc1d494
commit 7e828cbd61
5 changed files with 139 additions and 207 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.gitignore
+1
View File
@@ -34,3 +34,4 @@ conductor/archive/analysis/video_analysis_*/artifacts/*.vtt
# video.log intentionally committed (small text, useful for debugging)
conductor/archive/analysis/video_analysis_deob_warmup_20260621/samples
scripts/twitter_threads/cookies.txt
conductor/tracks/twitter_threads_extraction_20260705/cookies_netscape.txt