ed/manual_slop
Private
Public Access
0
0
Fork 0
Code Activity

fix(twitter_threads): align gallery-dl parser to live schema + add track extraction scripts

Browse Source 
Live Task 5.3 validation against real gallery-dl output: (1) pass -o text-tweets=true so text-only tweets are returned (media-only default yielded []); (2) fix author name/nick swap (gallery-dl name=handle, nick=display). Added convert_cookies.py (JSON cookie export -> Netscape) + run_corpus.py (8-URL pipeline -> docs/twitter/) to the track dir. Extracted 8/8 threads. 33 tests still pass.
This commit is contained in:
ed
2026-07-05 18:28:40 -04:00
parent 12bee9aba6
commit 53721a7796
3 changed files with 110 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/twitter_threads/fetch_thread.py
+3 -3
View File
@@ -144,8 +144,8 @@ def _media_url_from_entry(entry: Any) -> str:
def _post_from_meta(meta: dict[str, Any]) -> PostData:
author = meta.get("author") or {}
if isinstance(author, dict):
author_name = str(author.get("name") or author.get("nick") or "")
handle = str(author.get("nick") or author.get("name") or "")
author_name = str(author.get("nick") or author.get("name") or "")
handle = str(author.get("name") or author.get("nick") or "")
else:
author_name = str(author)
handle = str(author)
@@ -207,7 +207,7 @@ def _posts_from_gallery_dl(data: Any) -> list[PostData]:
def fetch_thread_from_url(url: str, cookies_path: Path | None = None) -> Result[ThreadData]:
clean = _normalize_url(url)
args = ["gallery-dl", "--dump-json"]
args = ["gallery-dl", "--dump-json", "-o", "text-tweets=true"]
if cookies_path is not None:
args += ["--cookies", str(cookies_path)]
args.append(clean)