manual_slop/scripts/twitter_threads/README.md

twitter_threads — standalone Twitter/X thread extraction tooling

Acquire Twitter/X posts + threads + their media and emit a self-contained Markdown document per thread (a thread.md plus a sibling media/ directory). The output is archivable, readable offline, and consumable by downstream analysis pipelines without any dependency on the live X.com service.

This package is the Twitter/X analog of scripts/video_analysis/: a small, dependency-light pipeline that acquires source content and emits Markdown. It is standalone — it imports nothing from src/ or conductor/ and can be copied to another repo with only gallery-dl as an external dependency.

Modules

File	Responsibility
fetch_thread.py	Acquire a post/thread into a ThreadData. URL path wraps gallery-dl (subprocess); local-HTML path uses stdlib html.parser. Also the pipeline entry CLI (writes thread_data.json).
download_media.py	Download the images/videos/GIFs referenced by a ThreadData via stdlib urllib into media/. Idempotent (skips existing files).
render_markdown.py	Emit the Markdown document (YAML front-matter + per-post sections + ./media/ links) from a ThreadData.
error_types.py	Shared Result[T] + ErrorInfo (data-oriented error handling) and the ThreadData / PostData / PostMetrics dataclasses.

Prerequisites

• Python 3.11+
• gallery-dl for the URL acquisition path: pip install gallery-dl
• cookies.txt exported from your browser (for auth-gated content). Export via a browser extension (e.g. "Get cookies.txt") while logged in to x.com.

The script internals use stdlib only (urllib, html.parser, json, pathlib, dataclasses, subprocess). gallery-dl is invoked as a subprocess, never imported.

Usage

Run each stage in order. From this repo (module form):

# 1. Fetch the thread -> writes <output>/<post_id>/thread_data.json
uv run python -m scripts.twitter_threads.fetch_thread \
  "https://x.com/NOTimothyLottes/status/1757198624818168210" \
  --output ./thread_output/ --cookies ./cookies.txt

# 2. Download the media -> writes <post_id>/media/<post_id>_img1.jpg etc.
uv run python -m scripts.twitter_threads.download_media \
  --input ./thread_output/1757198624818168210/thread_data.json \
  --output ./thread_output/1757198624818168210/media/

# 3. Render Markdown -> writes thread.md linking ./media/<file>
uv run python -m scripts.twitter_threads.render_markdown \
  --input ./thread_output/1757198624818168210/thread_data.json \
  --media-dir ./thread_output/1757198624818168210/media/ \
  --output ./thread_output/1757198624818168210/thread.md

Standalone (after copying the directory to another repo — run the files directly):

python fetch_thread.py "https://x.com/<user>/status/<id>" --output ./out/ --cookies ./cookies.txt
python download_media.py --input ./out/<id>/thread_data.json --output ./out/<id>/media/
python render_markdown.py --input ./out/<id>/thread_data.json --media-dir ./out/<id>/media/ --output ./out/<id>/thread.md

Each module also supports --help.

Local-HTML fallback (no network / gallery-dl broken)

If gallery-dl fails (auth wall, rate limit, X.com breakage), save the thread page from your browser (Ctrl+S, "Webpage, Complete") and pass the local .html file to fetch_thread.py instead of a URL:

python fetch_thread.py ./saved_thread.html --output ./out/

The parser reads <article> elements carrying data-post-id / data-author / data-handle / data-reply-to / data-quote-of / metric attributes, a <time datetime> element, a <div data-testid="tweetText"> and <img>/<source> media elements.

Output layout

thread_output/
  <post_id>/
    thread_data.json     # intermediate ThreadData (JSON)
    thread.md            # the rendered document
    media/
      <post_id>_img1.jpg
      <post_id>_vid1.mp4

Output Markdown schema

A YAML front-matter block (author, handle, post_url, post_id, timestamp, post_count, reply/repost/like/view counts — view_count: null when unavailable), followed by an # @<handle> — <title> heading and one ## Post N (<timestamp>) section per post (in chronological order, with — reply to Post M markers on replies). Media are linked via relative ./media/<filename> paths. Quote-tweets render as a > blockquote linking the quoted post.

URL normalization

Quote-share URLs carry a ?s=20 (or similar) suffix that is quote-share context, not part of the post identity. fetch_thread.py strips the query string + fragment before acquisition.

Acquisition strategies (and their tradeoffs)

• Strategy A — gallery-dl (default, implemented). Mature, actively maintained; handles auth via --cookies cookies.txt. Wrapped as a subprocess.
• Strategy B — snscrape (legacy fallback, not bundled). Was the standard pre-2023 tool; increasingly broken against X.com. Usable for some older content; not a dependency here.
• Strategy C — local HTML parse (implemented, no-network fallback). Save the page in the browser; parse it with stdlib html.parser. Most resilient when gallery-dl breaks.
• Strategy D — tweepy / Twitter API v2 (paid, not implemented). If you have developer credentials, swap the acquisition backend in fetch_thread_from_url. Not the default because it requires paid API access.

Copy to another repo

Copy the entire scripts/twitter_threads/ directory. The only external dependency is gallery-dl (pip install gallery-dl). The modules use a dual-import so they run both as a package (python -m ...) and as bare scripts (python fetch_thread.py ...). There are no imports from src/, conductor/, or scripts.video_analysis — the package is self-contained.

Conventions

Per conductor/code_styleguides/python.md: 1-space indentation, type hints, no inline comments. Per conductor/code_styleguides/error_handling.md: functions return Result[T] with ErrorInfo as data (no exceptions across the public API).