ed/manual_slop
Private
Public Access
0
0
Fork 0
Code Activity

feat(twitter_threads): full-thread extraction (conversations) + front-matter from target + corpus dedupe

Browse Source 
fetch_thread_from_url now passes -o conversations=true (full threads, not single tweets), sorts posts chronologically, and sets root_post_id to the URL's target tweet. render_markdown builds front-matter from the target (root_post_id) post, not the earliest conversation tweet (fixes wrong @handle). Added dedupe_corpus.py: keeps deepest thread per conversation, deletes duplicates, keeps+marks unique branches (branch_of) + writes threads_index.json. 33 tests pass.
This commit is contained in:
Edward R. Gonzalez
2026-07-05 18:38:43 -04:00
parent 53721a7796
commit ae9cc1d494
4 changed files with 114 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/twitter_threads/fetch_thread.py
+5 -2
View File
@@ -207,7 +207,7 @@ def _posts_from_gallery_dl(data: Any) -> list[PostData]:
def fetch_thread_from_url(url: str, cookies_path: Path | None = None) -> Result[ThreadData]:
clean = _normalize_url(url)
args = ["gallery-dl", "--dump-json", "-o", "text-tweets=true"]
args = ["gallery-dl", "--dump-json", "-o", "text-tweets=true", "-o", "conversations=true"]
if cookies_path is not None:
args += ["--cookies", str(cookies_path)]
args.append(clean)
@@ -224,7 +224,10 @@ def fetch_thread_from_url(url: str, cookies_path: Path | None = None) -> Result[
posts = _posts_from_gallery_dl(data)
if not posts:
return Result.err(make_error("ParseError", "fetch_thread_from_url", "no tweet metadata in gallery-dl output"))
return Result.ok(ThreadData(root_post_id=posts[0].post_id, posts=tuple(posts), source_url=clean))
posts.sort(key=lambda p: int(p.post_id) if p.post_id.isdigit() else 0)
target_id = clean.rstrip("/").split("/")[-1]
root_id = target_id if target_id.isdigit() else posts[0].post_id
return Result.ok(ThreadData(root_post_id=root_id, posts=tuple(posts), source_url=clean))
def thread_to_dict(thread: ThreadData) -> dict[str, Any]: