manual_slop/conductor/tracks/twitter_threads_extraction_20260705/spec.md

Track Specification: Twitter/X Thread Extraction Tooling

Overview

Build standalone, reusable Python scripts + a workflow for extracting Twitter/X posts and threads into Markdown, with their associated media (images, videos, GIFs) downloaded and linked alongside the post text. The output should be a self-contained Markdown document per thread (or single post) plus a sibling media/ directory holding the downloaded assets, so the resulting artifact can be archived, read offline, or fed into downstream analysis pipelines without any dependency on the live X.com service.

This is the Twitter/X analog of the existing scripts/video_analysis/ campaign tooling: a small, dependency-light pipeline that acquires source content losslessly and emits Markdown. It is NOT integrated into the Manual Slop application runtime — it is standalone tooling that lives under scripts/twitter_threads/ and can be copied to other repos with minimal deps.

Why standalone: the user explicitly requested these scripts "be standalone tooling so they don't have lots of dependencies with the repo's codebase and could be utilized in others." The scripts must not import from src/ and must not depend on the application's config, paths, or runtime. The only shared dependency is the project's Python toolchain (uv + pyproject.toml extras) for optional convenience.

Reference Project + Test Corpus (the @NOTimothyLottes threads)

Reference project: C:\projects\forth\bootslop — a sibling project the user is building. The Twitter thread extraction tooling produced by this track is intended to feed reference material into bootslop's scripts and reference-generation pipeline. The 6 threads below (all from @NOTimothyLottes) are the initial test corpus the tooling must handle. They are the first batch the user wants extracted, and they double as the concrete acceptance test for the scripts: if the tooling can extract all 6 cleanly, the track ships.

The threads cover a mix of single posts and multi-post threads (the @NOTimothyLottes account frequently posts long technical threads on graphics, low-level systems, and performance), so they exercise both the single-post path and the thread-chain path of fetch_thread.py.

#	URL	Notes
1	https://x.com/NOTimothyLottes/status/1757198624818168210	Earlier thread (2024-02); older X.com URL format. Tests the URL path against a pre-rename-era post.
2	https://x.com/NOTimothyLottes/status/1653570742762479620	Earliest in the set (2023-05); tests legacy content acquisition.
3	https://x.com/NOTimothyLottes/status/1917646466417381426	Recent (2025-04); tests the current X.com URL format + auth-wall behavior.
4	https://x.com/NOTimothyLottes/status/1917645859791200562	Recent; likely a sibling post in the same thread as #3 (sequential post IDs). Tests thread-chain detection.
5	https://x.com/NOTimothyLottes/status/1917644904055910502	Recent; likely a sibling in the same thread as #3-4.
6	https://x.com/NOTimothyLottes/status/1917642786804785230	Recent; likely the root of the #3-4-5 thread (lowest post ID in the cluster). Tests root-post detection + the thread-chain walk from root.
7	https://x.com/VPCOMPRESSB/status/1991383117571957052	Recent (2025); different author (@VPCOMPRESSB). Tests the tooling against a second account. The ?s=20 query param (tweet quote-share suffix) must be stripped before passing to gallery-dl.
8	https://x.com/VPCOMPRESSB/status/1987744335333622188	Recent; second author thread. Tests corpus extraction across multiple authors in a single batch.

Acceptance criterion (added): the end-to-end smoke test (Plan Task 5.3) must run the pipeline against all 8 URLs and produce 8 thread.md + media/ pairs in tests/artifacts/twitter_threads_corpus/. The corpus is the deliverable that gets handed off to bootslop. The ?s=20 query-share suffix on the @VPCOMPRESSB URLs must be stripped by fetch_thread.py before acquisition (the suffix is a quote-share context param, not part of the post identity).

Current State Audit (as of master 4c9fc99c)

Already Implemented (DO NOT re-implement)

• scripts/video_analysis/ — the precedent pipeline (7 modules: download_video.py, extract_transcript.py, extract_keyframes.py, ocr_frames.py, synthesize_report.py, error_types.py, __init__.py). This track mirrors that pattern: a __init__.py documenting the namespace, an error_types.py for shared Result[T, ErrorInfo], and per-stage modules.
• scripts/video_analysis/error_types.py — the canonical ErrorInfo + make_error shape. The Twitter scripts reuse the same shape (copied, not imported — standalone requirement).
• conductor/code_styleguides/error_handling.md — the Result[T] + NIL_T sentinel pattern. The Twitter scripts follow it for their internal error handling.
• conductor/code_styleguides/python.md §1 — 1-space indentation, type hints, no comments in implementation code.

Gaps to Fill (This Track's Scope)

• GAP-1: No Twitter/X extraction scripts exist. There is no scripts/twitter_threads/ directory. The user wants the equivalent of scripts/video_analysis/ for Twitter/X content.
• GAP-2: No defined workflow for acquiring thread + media. Twitter/X has no public transcript API like YouTube does; acquisition requires either (a) authenticated API access via tweepy/twitter-api-python (requires developer credentials), (b) guest/anonymous scraping via gallery-dl or snscrape (increasingly broken since the X.com rename + auth walls), or (c) manual export via the user's browser followed by script-driven post-processing. The track must pick a strategy and document the tradeoffs.
• GAP-3: No Markdown emission format. The output Markdown schema (how a thread renders, how media is linked, how quote-tweets and replies are represented, how metadata is captured) is undefined. The track must define it.
• GAP-4: No README/workflow doc for the standalone tooling. Per the standalone requirement, the scripts must ship with a README explaining how to use them outside this repo.

Goals

• G1. A scripts/twitter_threads/ package with a small pipeline: fetch_thread.py (acquire post + thread + quote-tweets), download_media.py (download images/videos/GIFs referenced by the posts), render_markdown.py (emit the Markdown document + media links), error_types.py (shared Result[T, ErrorInfo]), __init__.py (namespace docstring).
• G2. A defined Markdown output schema: one .md file per thread (or single post), with a YAML front-matter block capturing metadata (author, post ID, post URL, timestamp, reply count, repost count, like count, view count if available), followed by the thread body rendered in chronological order, with media referenced via relative ./media/<filename> links.
• G3. A media/ directory per thread holding the downloaded assets, named by post ID + index (e.g., media/<post_id>_img1.jpg, media/<post_id>_vid1.mp4).
• G4. A README.md in scripts/twitter_threads/ documenting: prerequisites (which acquisition strategy, which optional deps), usage (CLI invocation), output layout, and the "copy to another repo" instructions (which files to copy, which deps to install).
• G5. The scripts must be runnable both as python scripts/twitter_threads/<module>.py (standalone) and python -m scripts.twitter_threads.<module> (from this repo). No imports from src/ or conductor/.

Functional Requirements

FR1: Acquisition Strategy (the core design decision)

The scripts must support at least one acquisition strategy that works without paid X.com developer credentials. The track must implement and document the tradeoffs of:

• Strategy A — gallery-dl (recommended default): gallery-dl is a mature, actively-maintained tool that can download Twitter/X posts + threads + media. It handles auth via cookies exported from the browser (the cookies.txt pattern) or via guest access where available. It produces JSON metadata + downloaded media files. The scripts wrap gallery-dl as a subprocess (same pattern as scripts/video_analysis/download_video.py wraps yt-dlp).
• Strategy B — snscrape (legacy fallback): snscrape was the standard pre-2023 tool but is increasingly broken against X.com. Document it as a fallback for older content (pre-rename Twitter) but do not depend on it as the primary path.
• Strategy C — manual export + post-processing (the no-network path): the user saves the thread HTML via the browser (Ctrl+S, or a single-page archive), and the scripts parse the local HTML file to extract post text + media URLs, then download the media via urllib (no X.com auth needed for the media CDN URLs, which are public). This is the most resilient path if gallery-dl breaks.
• Strategy D — tweepy / twitter-api-python (the paid-API path): documented as the "if you have developer credentials" option, but NOT implemented as the default. The track notes it in the README as an alternative acquisition backend.

Decision: implement Strategy A (gallery-dl subprocess wrapper) as the primary path, with Strategy C (local HTML parse) as a fallback for when gallery-dl fails. Document Strategy B and Strategy D in the README as alternatives.

FR2: fetch_thread.py — Thread Acquisition

• Input: a single tweet/X post URL (e.g., https://x.com/<user>/status/<id>) OR a local HTML file path.
• URL normalization (added): strip any query-string suffix before acquisition. X.com quote-share URLs carry a ?s=20 (or similar) param that is quote-share context, not part of the post identity. fetch_thread.py must parse the URL with urllib.parse.urlparse + reconstruct without the query string before passing to gallery-dl. This is a one-line urlunparse((scheme, netloc, path, params, "", fragment)) call.
• If URL: invoke gallery-dl with the URL + --write-metadata + --dump-json flags to acquire the post + its thread context + media. Capture the JSON output. If gallery-dl fails (auth wall, rate limit), emit a Result.err with an ErrorInfo and document the fallback (Strategy C).
• If local HTML: parse the HTML to extract the post text, the post ID, the author handle, the timestamp, and the media URLs. Use html.parser from the stdlib (no BeautifulSoup dependency — standalone requirement).
• Output: a ThreadData dataclass (typed, frozen, slots) holding: the root post + the chain of replies (the thread), each post being a PostData dataclass (id, author, handle, text, timestamp, media_urls, reply_to_id, quote_of_id, metrics).
• Returns Result[ThreadData, ErrorInfo].

FR3: download_media.py — Media Download

• Input: a ThreadData instance + an output directory path.
• For each PostData in the thread, for each media_url in post.media_urls: download the file via urllib.request.urlopen (stdlib; no requests dependency). Name the file <post_id>_<index><ext> where ext is derived from the URL or Content-Type header.
• Handle the 4 media types X.com serves: images (.jpg, .png), videos (.mp4), GIFs (served as .mp4 after the GIF-to-mp4 conversion X.com does), and inline images in quote-tweets.
• Returns Result[list[Path], ErrorInfo] — the list of downloaded file paths.
• Idempotent: if a file already exists at the target path with the same byte size, skip the download.

FR4: render_markdown.py — Markdown Emission

• Input: a ThreadData instance + the list of downloaded media paths + an output .md path.
• Output schema:

  ---
  title: "<thread title or first post's first line>"
  author: "<display name>"
  handle: "@<handle>"
  post_url: "<root post URL>"
  post_id: "<root post ID>"
  timestamp: "<ISO 8601>"
  post_count: <N>
  reply_count: <N>
  repost_count: <N>
  like_count: <N>
  view_count: <N or null>
  ---
  
  # @<handle> — <thread title>
  
  ## Post 1 (<ISO timestamp>)
  
  <post text, preserving line breaks>
  
  [Media 1](./media/<post_id>_img1.jpg)
  [Media 2](./media/<post_id>_vid1.mp4)
  
  ## Post 2 (<ISO timestamp>) — reply to Post 1
  
  <post text>
  
  ...
  

• Quote-tweets: rendered as a nested blockquote with a link to the quoted post.
• Reply chains: each post in the thread is a ## Post N section, in chronological order.
• Media: linked via relative ./media/<filename> paths so the Markdown renders correctly when the .md + media/ directory are moved together.
• Returns Result[Path, ErrorInfo] — the output .md path.

FR5: error_types.py — Shared Error Type

• Copies the shape of scripts/video_analysis/error_types.py: ErrorInfo dataclass (frozen, slots) + make_error factory. Standalone — no import from scripts.video_analysis.

FR6: __init__.py — Namespace Docstring

• Mirrors scripts/video_analysis/__init__.py: a docstring documenting the namespace, the per-module responsibilities, and the standalone-usage note.

FR7: README.md — Standalone Usage Doc

• Prerequisites: gallery-dl (install via pip install gallery-dl), a cookies.txt exported from the browser (for auth-gated content), Python 3.11+.
• Usage:

  # From this repo:
  uv run python -m scripts.twitter_threads.fetch_thread "https://x.com/<user>/status/<id>" --output ./thread_output/
  
  # Standalone (copied to another repo):
  python scripts/twitter_threads/fetch_thread.py "https://x.com/<user>/status/<id>" --output ./thread_output/
  

• Output layout:

  thread_output/
    <post_id>/
      thread.md
      media/
        <post_id>_img1.jpg
        <post_id>_vid1.mp4
  

• "Copy to another repo" instructions: copy the entire scripts/twitter_threads/ directory + the error_types.py dependency (which is self-contained). No src/ imports. The only external dep is gallery-dl (installable via pip).

FR8: Tests

• tests/test_twitter_threads_render.py — tests for render_markdown.py (the pure-function module; no network). Construct a ThreadData fixture, render to Markdown, assert the YAML front-matter + the per-post sections + the media links.
• tests/test_twitter_threads_fetch.py — tests for fetch_thread.py's local-HTML-parse path (Strategy C). Feed a small fixture HTML file, assert the parsed ThreadData.
• tests/test_twitter_threads_media.py — tests for download_media.py's naming + idempotency logic. Mock the HTTP fetch (no network in tests); assert file naming + skip-if-exists behavior.
• Per conductor/code_styleguides/error_handling.md, all tests assert on Result.ok / Result.err rather than raising.

Non-Functional Requirements

• Standalone: zero imports from src/, conductor/, scripts.video_analysis, or any project-internal module. The scripts/twitter_threads/ directory must be copy-pasteable to another repo with only gallery-dl as an external dep.
• Dependency-light: stdlib only for the script internals (urllib, html.parser, json, pathlib, dataclasses, subprocess). gallery-dl is invoked as a subprocess, not imported. No requests, no BeautifulSoup, no tweepy in the default path.
• 1-space indentation per conductor/code_styleguides/python.md §1.
• Type hints on all public functions.
• Result[T, ErrorInfo] return types per conductor/code_styleguides/error_handling.md. No bare exceptions in the public API.
• No comments in implementation code per conductor/code_styleguides/python.md (docstrings allowed; inline comments banned).
• CRLF line endings on Windows (the repo default).
• File size convention per AGENTS.md: the scripts go in scripts/twitter_threads/ (scripts are namespace-isolated by directory). No new src/ files.

Architecture Reference

• scripts/video_analysis/ — the precedent pipeline (the pattern this track mirrors).
• scripts/video_analysis/download_video.py — the subprocess-wrapper pattern (yt-dlp → gallery-dl analog).
• scripts/video_analysis/error_types.py — the ErrorInfo shape (copied, not imported).
• conductor/code_styleguides/error_handling.md — the Result[T] + NIL_T sentinel pattern.
• conductor/code_styleguides/python.md §1 — 1-space indentation + type hints + no comments.
• conductor/code_styleguides/data_oriented_design.md §8.5 — typed @dataclass(frozen=True, slots=True) for ThreadData / PostData. No dict[str, Any] in the script internals.

Out of Scope

• Integration with the Manual Slop GUI. The scripts are standalone; no gui_2.py or app_controller.py changes.
• Integration with the RAG subsystem. The output Markdown could be RAG-indexed, but that's a future track.
• Integration with the MMA engine. The scripts are not MMA tools; they're standalone analysis tooling.
• Real-time streaming / polling. The scripts are one-shot extractors, not stream listeners.
• Twitter API v2 paid endpoints. Documented in the README as an alternative; not implemented as the default.
• snscrape as a hard dependency. Documented as a legacy fallback; not installed by default.
• Browser automation (Selenium/Playwright). Not used. The local-HTML-parse fallback (Strategy C) uses the user's manually-saved HTML, not a headless browser.
• Cross-post aggregation (Twitter + Mastodon + Bluesky). Twitter/X only for this track.

Track Structure

This is a single track (no children). The work is small enough to fit in one plan. If the user later wants a "Twitter thread analysis campaign" (the analog of the video analysis campaign — many threads, cross-thread synthesis), that's a separate future track that consumes this tooling.

Verification

• VC1: scripts/twitter_threads/ exists with 5 files: __init__.py, error_types.py, fetch_thread.py, download_media.py, render_markdown.py.
• VC2: scripts/twitter_threads/README.md exists and documents prerequisites, usage, output layout, and the "copy to another repo" instructions.
• VC3: tests/test_twitter_threads_render.py passes: given a ThreadData fixture, render_markdown.py produces the expected Markdown (YAML front-matter + per-post sections + media links).
• VC4: tests/test_twitter_threads_fetch.py passes: given a small fixture HTML file, fetch_thread.py (Strategy C path) produces the expected ThreadData.
• VC5: tests/test_twitter_threads_media.py passes: download_media.py produces correctly-named files and skips existing files.
• VC6: python scripts/twitter_threads/fetch_thread.py --help works (standalone invocation, no src/ imports).
• VC7: The scripts contain zero imports from src/, conductor/, or scripts.video_analysis. Verified via grep -r "from src\." scripts/twitter_threads/ returning nothing + grep -r "import src\." scripts/twitter_threads/ returning nothing.

See Also

• conductor/tracks/video_analysis_campaign_2_20260627/spec.md — the sibling track (video analysis; the pattern this track mirrors).
• scripts/video_analysis/ — the precedent pipeline.
• conductor/code_styleguides/python.md §1 — 1-space indentation.
• conductor/code_styleguides/error_handling.md — Result[T] pattern.
• conductor/code_styleguides/data_oriented_design.md §8.5 — typed dataclasses, no dict[str, Any].