Private
Public Access
0
0
Commit Graph

9 Commits

Author SHA1 Message Date
ed 2c680e23e4 feat(twitter_threads): gallery-dl media download + inline media embeds in Markdown
download_media now shells out to gallery-dl (handles X.com auth/403/video->mp4) instead of urllib, returns the downloaded files, + media_files_for_post glob helper + --cookies CLI. render_markdown embeds images with ![](...) and videos with an HTML <video controls> tag (were plain links). extract_corpus uses the package download_media (one download path). Tests rewritten for the gallery-dl API. Corpus: 4 threads, 33 media (32 img embeds + 1 video), 0 missing.
2026-07-05 19:08:43 -04:00
ed ae9cc1d494 feat(twitter_threads): full-thread extraction (conversations) + front-matter from target + corpus dedupe
fetch_thread_from_url now passes -o conversations=true (full threads, not single tweets), sorts posts chronologically, and sets root_post_id to the URL's target tweet. render_markdown builds front-matter from the target (root_post_id) post, not the earliest conversation tweet (fixes wrong @handle). Added dedupe_corpus.py: keeps deepest thread per conversation, deletes duplicates, keeps+marks unique branches (branch_of) + writes threads_index.json. 33 tests pass.
2026-07-05 18:38:43 -04:00
ed 53721a7796 fix(twitter_threads): align gallery-dl parser to live schema + add track extraction scripts
Live Task 5.3 validation against real gallery-dl output: (1) pass -o text-tweets=true so text-only tweets are returned (media-only default yielded []); (2) fix author name/nick swap (gallery-dl name=handle, nick=display). Added convert_cookies.py (JSON cookie export -> Netscape) + run_corpus.py (8-URL pipeline -> docs/twitter/) to the track dir. Extracted 8/8 threads. 33 tests still pass.
2026-07-05 18:28:40 -04:00
ed fe207c1e42 docs(twitter_threads): README — standalone usage + strategies + copy-to-another-repo
FR7/VC2: prerequisites (gallery-dl, cookies.txt, py3.11+), usage for all 3 stages (-m and bare-script), local-HTML fallback, output layout, Markdown schema, URL normalization, strategies A-D, copy-to-another-repo instructions.
2026-07-05 17:50:30 -04:00
ed ed86731701 feat(twitter_threads): thread_from_dict + download_media/render_markdown CLIs (end-to-end -m pipeline)
thread_from_dict (JSON wire boundary) round-trips thread_data.json; download_media/render_markdown gain argparse main() + __main__ so the plan Task 5.3 pipeline runs via -m; dual-import added so all modules run standalone or via -m (G5). Pipeline integration test (html->names->render) + CLI tests cover the glue without network. TDD: red -> green (4 pipeline + 29 regression = 33 passed).
2026-07-05 17:48:07 -04:00
ed 06ff9299e2 feat(twitter_threads): fetch_thread.py — html.parser (local HTML) + gallery-dl subprocess (URL) + URL query-strip + CLI
fetch_thread_from_html parses the data-attr article contract via stdlib html.parser; _normalize_url strips ?s=20 quote-share suffix (spec FR2); fetch_thread_from_url wraps gallery-dl --dump-json (best-effort, manually smoke-tested); dual-import (absolute/relative) makes it runnable via -m AND as a bare script (VC6/G5); CLI writes thread_data.json. TDD: red(no module) -> green(7 fetch + 22 regression = 29 passed).
2026-07-05 17:36:03 -04:00
ed f503eb5de3 feat(twitter_threads): download_media.py — stdlib urllib + idempotent + typed per-kind naming
media_names_for_post derives <post_id>_<kind><index>.<ext> (img/vid/gif, per-kind counters) from media_urls; download_media(thread, output_dir) -> Result[list[Path]] downloads via stdlib urlopen, idempotent skip when target exists with size>0, URLError -> Result.err(HttpError). Authorized HTTP-boundary mock tests. TDD: red(no module) -> green(6 media + 16 regression pass).
2026-07-05 17:24:45 -04:00
ed ef98f6d1a1 feat(twitter_threads): Result[T] + render_markdown.py (YAML front-matter + per-post sections + media links)
Added standalone Result[T] (frozen+slots, ok/err classmethods, is_ok) to error_types.py per canonical AND-over-OR error_handling.md (not the video_analysis _Ok|_Err sum type). render_markdown renders YAML front-matter from root post, per-post ## sections with reply markers, quote blockquotes, and ./media/<name> links; media_names mapping keeps render decoupled from download_media. TDD: red (no Result) -> green (7 render + 9 types passed).
2026-07-05 17:17:18 -04:00
ed 161d8da882 feat(twitter_threads): scaffold package + error types + typed dataclasses
TIER-2 READ AGENTS.md, conductor/workflow.md, conductor/edit_workflow.md, conductor/tier2/githooks/forbidden-files.txt, conductor/tracks/tier2_leak_prevention_20260620/spec.md (archived -> read docs/reports/2026-06-15/TRACK_COMPLETION_tier2_leak_prevention_20260620.md), conductor/code_styleguides/data_oriented_design.md, conductor/code_styleguides/python.md, conductor/code_styleguides/type_aliases.md, conductor/code_styleguides/error_handling.md, conductor/product-guidelines.md (Core Value via aggregate_directives) before Phase 1 (scaffold + error_types + dataclasses).

scripts/twitter_threads/__init__.py (namespace docstring); error_types.py (ErrorInfo + make_error + PostMetrics/PostData/ThreadData, frozen+slots); tests/test_twitter_threads_types.py (9 contract tests).
2026-07-05 17:07:04 -04:00