# twitter_threads — standalone Twitter/X thread extraction tooling

Acquire Twitter/X posts + threads + their media and emit a self-contained Markdown
document per thread (a `thread.md` plus a sibling `media/` directory). The output is
archivable, readable offline, and consumable by downstream analysis pipelines without
any dependency on the live X.com service.

This package is the Twitter/X analog of `scripts/video_analysis/`: a small,
dependency-light pipeline that acquires source content and emits Markdown. It is
**standalone** — it imports nothing from `src/` or `conductor/` and can be copied to
another repo with only `gallery-dl` as an external dependency.

## Modules

| File | Responsibility |
|---|---|
| `fetch_thread.py` | Acquire a post/thread into a `ThreadData`. URL path wraps `gallery-dl` (subprocess); local-HTML path uses stdlib `html.parser`. Also the pipeline entry CLI (writes `thread_data.json`). |
| `download_media.py` | Download the images/videos/GIFs referenced by a `ThreadData` via stdlib `urllib` into `media/`. Idempotent (skips existing files). |
| `render_markdown.py` | Emit the Markdown document (YAML front-matter + per-post sections + `./media/` links) from a `ThreadData`. |
| `error_types.py` | Shared `Result[T]` + `ErrorInfo` (data-oriented error handling) and the `ThreadData` / `PostData` / `PostMetrics` dataclasses. |

## Prerequisites

- **Python 3.11+**
- **`gallery-dl`** for the URL acquisition path: `pip install gallery-dl`
- **`cookies.txt`** exported from your browser (for auth-gated content). Export via a
  browser extension (e.g. "Get cookies.txt") while logged in to x.com.

The script internals use **stdlib only** (`urllib`, `html.parser`, `json`, `pathlib`,
`dataclasses`, `subprocess`). `gallery-dl` is invoked as a subprocess, never imported.

## Usage

Run each stage in order. From this repo (module form):

```bash
# 1. Fetch the thread -> writes <output>/<post_id>/thread_data.json
uv run python -m scripts.twitter_threads.fetch_thread \
  "https://x.com/NOTimothyLottes/status/1757198624818168210" \
  --output ./thread_output/ --cookies ./cookies.txt

# 2. Download the media -> writes <post_id>/media/<post_id>_img1.jpg etc.
uv run python -m scripts.twitter_threads.download_media \
  --input ./thread_output/1757198624818168210/thread_data.json \
  --output ./thread_output/1757198624818168210/media/

# 3. Render Markdown -> writes thread.md linking ./media/<file>
uv run python -m scripts.twitter_threads.render_markdown \
  --input ./thread_output/1757198624818168210/thread_data.json \
  --media-dir ./thread_output/1757198624818168210/media/ \
  --output ./thread_output/1757198624818168210/thread.md
```

Standalone (after copying the directory to another repo — run the files directly):

```bash
python fetch_thread.py "https://x.com/<user>/status/<id>" --output ./out/ --cookies ./cookies.txt
python download_media.py --input ./out/<id>/thread_data.json --output ./out/<id>/media/
python render_markdown.py --input ./out/<id>/thread_data.json --media-dir ./out/<id>/media/ --output ./out/<id>/thread.md
```

Each module also supports `--help`.

### Local-HTML fallback (no network / gallery-dl broken)

If `gallery-dl` fails (auth wall, rate limit, X.com breakage), save the thread page
from your browser (Ctrl+S, "Webpage, Complete") and pass the local `.html` file to
`fetch_thread.py` instead of a URL:

```bash
python fetch_thread.py ./saved_thread.html --output ./out/
```

The parser reads `<article>` elements carrying `data-post-id` / `data-author` /
`data-handle` / `data-reply-to` / `data-quote-of` / metric attributes, a
`<time datetime>` element, a `<div data-testid="tweetText">` and `<img>`/`<source>`
media elements.

## Output layout

```
thread_output/
  <post_id>/
    thread_data.json     # intermediate ThreadData (JSON)
    thread.md            # the rendered document
    media/
      <post_id>_img1.jpg
      <post_id>_vid1.mp4
```

## Output Markdown schema

A YAML front-matter block (author, handle, post_url, post_id, timestamp, post_count,
reply/repost/like/view counts — `view_count: null` when unavailable), followed by an
`# @<handle> — <title>` heading and one `## Post N (<timestamp>)` section per post (in
chronological order, with `— reply to Post M` markers on replies). Media are linked via
relative `./media/<filename>` paths. Quote-tweets render as a `>` blockquote linking the
quoted post.

## URL normalization

Quote-share URLs carry a `?s=20` (or similar) suffix that is quote-share context, not
part of the post identity. `fetch_thread.py` strips the query string + fragment before
acquisition.

## Acquisition strategies (and their tradeoffs)

- **Strategy A — `gallery-dl` (default, implemented).** Mature, actively maintained;
  handles auth via `--cookies cookies.txt`. Wrapped as a subprocess.
- **Strategy B — `snscrape` (legacy fallback, not bundled).** Was the standard pre-2023
  tool; increasingly broken against X.com. Usable for some older content; not a
  dependency here.
- **Strategy C — local HTML parse (implemented, no-network fallback).** Save the page in
  the browser; parse it with stdlib `html.parser`. Most resilient when `gallery-dl`
  breaks.
- **Strategy D — `tweepy` / Twitter API v2 (paid, not implemented).** If you have
  developer credentials, swap the acquisition backend in `fetch_thread_from_url`. Not
  the default because it requires paid API access.

## Copy to another repo

Copy the entire `scripts/twitter_threads/` directory. The only external dependency is
`gallery-dl` (`pip install gallery-dl`). The modules use a dual-import so they run both
as a package (`python -m ...`) and as bare scripts (`python fetch_thread.py ...`). There
are **no imports from `src/`, `conductor/`, or `scripts.video_analysis`** — the package
is self-contained.

## Conventions

Per `conductor/code_styleguides/python.md`: 1-space indentation, type hints, no inline
comments. Per `conductor/code_styleguides/error_handling.md`: functions return
`Result[T]` with `ErrorInfo` as data (no exceptions across the public API).