diff --git a/scripts/twitter_threads/README.md b/scripts/twitter_threads/README.md new file mode 100644 index 00000000..861c32cc --- /dev/null +++ b/scripts/twitter_threads/README.md @@ -0,0 +1,132 @@ +# twitter_threads — standalone Twitter/X thread extraction tooling + +Acquire Twitter/X posts + threads + their media and emit a self-contained Markdown +document per thread (a `thread.md` plus a sibling `media/` directory). The output is +archivable, readable offline, and consumable by downstream analysis pipelines without +any dependency on the live X.com service. + +This package is the Twitter/X analog of `scripts/video_analysis/`: a small, +dependency-light pipeline that acquires source content and emits Markdown. It is +**standalone** — it imports nothing from `src/` or `conductor/` and can be copied to +another repo with only `gallery-dl` as an external dependency. + +## Modules + +| File | Responsibility | +|---|---| +| `fetch_thread.py` | Acquire a post/thread into a `ThreadData`. URL path wraps `gallery-dl` (subprocess); local-HTML path uses stdlib `html.parser`. Also the pipeline entry CLI (writes `thread_data.json`). | +| `download_media.py` | Download the images/videos/GIFs referenced by a `ThreadData` via stdlib `urllib` into `media/`. Idempotent (skips existing files). | +| `render_markdown.py` | Emit the Markdown document (YAML front-matter + per-post sections + `./media/` links) from a `ThreadData`. | +| `error_types.py` | Shared `Result[T]` + `ErrorInfo` (data-oriented error handling) and the `ThreadData` / `PostData` / `PostMetrics` dataclasses. | + +## Prerequisites + +- **Python 3.11+** +- **`gallery-dl`** for the URL acquisition path: `pip install gallery-dl` +- **`cookies.txt`** exported from your browser (for auth-gated content). Export via a + browser extension (e.g. "Get cookies.txt") while logged in to x.com. + +The script internals use **stdlib only** (`urllib`, `html.parser`, `json`, `pathlib`, +`dataclasses`, `subprocess`). `gallery-dl` is invoked as a subprocess, never imported. + +## Usage + +Run each stage in order. From this repo (module form): + +```bash +# 1. Fetch the thread -> writes //thread_data.json +uv run python -m scripts.twitter_threads.fetch_thread \ + "https://x.com/NOTimothyLottes/status/1757198624818168210" \ + --output ./thread_output/ --cookies ./cookies.txt + +# 2. Download the media -> writes /media/_img1.jpg etc. +uv run python -m scripts.twitter_threads.download_media \ + --input ./thread_output/1757198624818168210/thread_data.json \ + --output ./thread_output/1757198624818168210/media/ + +# 3. Render Markdown -> writes thread.md linking ./media/ +uv run python -m scripts.twitter_threads.render_markdown \ + --input ./thread_output/1757198624818168210/thread_data.json \ + --media-dir ./thread_output/1757198624818168210/media/ \ + --output ./thread_output/1757198624818168210/thread.md +``` + +Standalone (after copying the directory to another repo — run the files directly): + +```bash +python fetch_thread.py "https://x.com//status/" --output ./out/ --cookies ./cookies.txt +python download_media.py --input ./out//thread_data.json --output ./out//media/ +python render_markdown.py --input ./out//thread_data.json --media-dir ./out//media/ --output ./out//thread.md +``` + +Each module also supports `--help`. + +### Local-HTML fallback (no network / gallery-dl broken) + +If `gallery-dl` fails (auth wall, rate limit, X.com breakage), save the thread page +from your browser (Ctrl+S, "Webpage, Complete") and pass the local `.html` file to +`fetch_thread.py` instead of a URL: + +```bash +python fetch_thread.py ./saved_thread.html --output ./out/ +``` + +The parser reads `
` elements carrying `data-post-id` / `data-author` / +`data-handle` / `data-reply-to` / `data-quote-of` / metric attributes, a +`