docs(twitter_threads): README — standalone usage + strategies + copy-to-another-repo
FR7/VC2: prerequisites (gallery-dl, cookies.txt, py3.11+), usage for all 3 stages (-m and bare-script), local-HTML fallback, output layout, Markdown schema, URL normalization, strategies A-D, copy-to-another-repo instructions.
This commit is contained in:
@@ -0,0 +1,132 @@
|
||||
# twitter_threads — standalone Twitter/X thread extraction tooling
|
||||
|
||||
Acquire Twitter/X posts + threads + their media and emit a self-contained Markdown
|
||||
document per thread (a `thread.md` plus a sibling `media/` directory). The output is
|
||||
archivable, readable offline, and consumable by downstream analysis pipelines without
|
||||
any dependency on the live X.com service.
|
||||
|
||||
This package is the Twitter/X analog of `scripts/video_analysis/`: a small,
|
||||
dependency-light pipeline that acquires source content and emits Markdown. It is
|
||||
**standalone** — it imports nothing from `src/` or `conductor/` and can be copied to
|
||||
another repo with only `gallery-dl` as an external dependency.
|
||||
|
||||
## Modules
|
||||
|
||||
| File | Responsibility |
|
||||
|---|---|
|
||||
| `fetch_thread.py` | Acquire a post/thread into a `ThreadData`. URL path wraps `gallery-dl` (subprocess); local-HTML path uses stdlib `html.parser`. Also the pipeline entry CLI (writes `thread_data.json`). |
|
||||
| `download_media.py` | Download the images/videos/GIFs referenced by a `ThreadData` via stdlib `urllib` into `media/`. Idempotent (skips existing files). |
|
||||
| `render_markdown.py` | Emit the Markdown document (YAML front-matter + per-post sections + `./media/` links) from a `ThreadData`. |
|
||||
| `error_types.py` | Shared `Result[T]` + `ErrorInfo` (data-oriented error handling) and the `ThreadData` / `PostData` / `PostMetrics` dataclasses. |
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- **Python 3.11+**
|
||||
- **`gallery-dl`** for the URL acquisition path: `pip install gallery-dl`
|
||||
- **`cookies.txt`** exported from your browser (for auth-gated content). Export via a
|
||||
browser extension (e.g. "Get cookies.txt") while logged in to x.com.
|
||||
|
||||
The script internals use **stdlib only** (`urllib`, `html.parser`, `json`, `pathlib`,
|
||||
`dataclasses`, `subprocess`). `gallery-dl` is invoked as a subprocess, never imported.
|
||||
|
||||
## Usage
|
||||
|
||||
Run each stage in order. From this repo (module form):
|
||||
|
||||
```bash
|
||||
# 1. Fetch the thread -> writes <output>/<post_id>/thread_data.json
|
||||
uv run python -m scripts.twitter_threads.fetch_thread \
|
||||
"https://x.com/NOTimothyLottes/status/1757198624818168210" \
|
||||
--output ./thread_output/ --cookies ./cookies.txt
|
||||
|
||||
# 2. Download the media -> writes <post_id>/media/<post_id>_img1.jpg etc.
|
||||
uv run python -m scripts.twitter_threads.download_media \
|
||||
--input ./thread_output/1757198624818168210/thread_data.json \
|
||||
--output ./thread_output/1757198624818168210/media/
|
||||
|
||||
# 3. Render Markdown -> writes thread.md linking ./media/<file>
|
||||
uv run python -m scripts.twitter_threads.render_markdown \
|
||||
--input ./thread_output/1757198624818168210/thread_data.json \
|
||||
--media-dir ./thread_output/1757198624818168210/media/ \
|
||||
--output ./thread_output/1757198624818168210/thread.md
|
||||
```
|
||||
|
||||
Standalone (after copying the directory to another repo — run the files directly):
|
||||
|
||||
```bash
|
||||
python fetch_thread.py "https://x.com/<user>/status/<id>" --output ./out/ --cookies ./cookies.txt
|
||||
python download_media.py --input ./out/<id>/thread_data.json --output ./out/<id>/media/
|
||||
python render_markdown.py --input ./out/<id>/thread_data.json --media-dir ./out/<id>/media/ --output ./out/<id>/thread.md
|
||||
```
|
||||
|
||||
Each module also supports `--help`.
|
||||
|
||||
### Local-HTML fallback (no network / gallery-dl broken)
|
||||
|
||||
If `gallery-dl` fails (auth wall, rate limit, X.com breakage), save the thread page
|
||||
from your browser (Ctrl+S, "Webpage, Complete") and pass the local `.html` file to
|
||||
`fetch_thread.py` instead of a URL:
|
||||
|
||||
```bash
|
||||
python fetch_thread.py ./saved_thread.html --output ./out/
|
||||
```
|
||||
|
||||
The parser reads `<article>` elements carrying `data-post-id` / `data-author` /
|
||||
`data-handle` / `data-reply-to` / `data-quote-of` / metric attributes, a
|
||||
`<time datetime>` element, a `<div data-testid="tweetText">` and `<img>`/`<source>`
|
||||
media elements.
|
||||
|
||||
## Output layout
|
||||
|
||||
```
|
||||
thread_output/
|
||||
<post_id>/
|
||||
thread_data.json # intermediate ThreadData (JSON)
|
||||
thread.md # the rendered document
|
||||
media/
|
||||
<post_id>_img1.jpg
|
||||
<post_id>_vid1.mp4
|
||||
```
|
||||
|
||||
## Output Markdown schema
|
||||
|
||||
A YAML front-matter block (author, handle, post_url, post_id, timestamp, post_count,
|
||||
reply/repost/like/view counts — `view_count: null` when unavailable), followed by an
|
||||
`# @<handle> — <title>` heading and one `## Post N (<timestamp>)` section per post (in
|
||||
chronological order, with `— reply to Post M` markers on replies). Media are linked via
|
||||
relative `./media/<filename>` paths. Quote-tweets render as a `>` blockquote linking the
|
||||
quoted post.
|
||||
|
||||
## URL normalization
|
||||
|
||||
Quote-share URLs carry a `?s=20` (or similar) suffix that is quote-share context, not
|
||||
part of the post identity. `fetch_thread.py` strips the query string + fragment before
|
||||
acquisition.
|
||||
|
||||
## Acquisition strategies (and their tradeoffs)
|
||||
|
||||
- **Strategy A — `gallery-dl` (default, implemented).** Mature, actively maintained;
|
||||
handles auth via `--cookies cookies.txt`. Wrapped as a subprocess.
|
||||
- **Strategy B — `snscrape` (legacy fallback, not bundled).** Was the standard pre-2023
|
||||
tool; increasingly broken against X.com. Usable for some older content; not a
|
||||
dependency here.
|
||||
- **Strategy C — local HTML parse (implemented, no-network fallback).** Save the page in
|
||||
the browser; parse it with stdlib `html.parser`. Most resilient when `gallery-dl`
|
||||
breaks.
|
||||
- **Strategy D — `tweepy` / Twitter API v2 (paid, not implemented).** If you have
|
||||
developer credentials, swap the acquisition backend in `fetch_thread_from_url`. Not
|
||||
the default because it requires paid API access.
|
||||
|
||||
## Copy to another repo
|
||||
|
||||
Copy the entire `scripts/twitter_threads/` directory. The only external dependency is
|
||||
`gallery-dl` (`pip install gallery-dl`). The modules use a dual-import so they run both
|
||||
as a package (`python -m ...`) and as bare scripts (`python fetch_thread.py ...`). There
|
||||
are **no imports from `src/`, `conductor/`, or `scripts.video_analysis`** — the package
|
||||
is self-contained.
|
||||
|
||||
## Conventions
|
||||
|
||||
Per `conductor/code_styleguides/python.md`: 1-space indentation, type hints, no inline
|
||||
comments. Per `conductor/code_styleguides/error_handling.md`: functions return
|
||||
`Result[T]` with `ErrorInfo` as data (no exceptions across the public API).
|
||||
Reference in New Issue
Block a user