more branch scrape exploration.

This commit is contained in:
ed committed 2026-08-24 23:04:33 -04:00
1 parent c637c393b8
commit a2de38f187
17 files changed
+452 -3

No files matched your search

+4 -1
View File
@@ -8,6 +8,7 @@ From the vault root:
python C:\projects\Threadwell\scripts\twitter\tw.py graph --id <snowflake>
python C:\projects\Threadwell\scripts\twitter\tw.py refresh --id <snowflake> --tip
python C:\projects\Threadwell\scripts\twitter\tw.py refresh --id <spine-tip> --tip --branch <branch-tip>
python C:\projects\Threadwell\scripts\twitter\tw.py add-branch --id <original-spine-tip> --from <reply-node>
python C:\projects\Threadwell\scripts\twitter\tw.py audit-media --id <snowflake>
python C:\projects\Threadwell\scripts\twitter\tw.py publish --id <snowflake>
python C:\projects\Threadwell\scripts\twitter\tw.py sync --handle <handle>
@@ -22,6 +23,8 @@ python C:\projects\Threadwell\scripts\twitter\tw.py migrate-media --all --apply
`refresh` = refetch (cookies) + ingest + emit `--force` + media_merge. `--tip` treats `--id` as the tip and walks back to the OP regardless of handle. Repeatable `--branch <tip-id>` captures only those additional explicit tip paths and merges them into the same `thread_data.json`; it does not discover replies automatically. Captures run sequentially with gallery-dl retries disabled and five-second extractor/request pacing. Each thread emits exactly one archive directory owned by the OP (the post with `reply_to_id == None`); cross-author responders do not get a directory. Does not flip `draft`. Does not commit. Frozen ids abort.
`add-branch` incrementally extends an already emitted thread. `--id` must be the original stored spine tip; repeatable `--from <reply-node>` values are each captured once. The command locally retains each node's missing attachment path and entire visible descendant subtree, merges only previously absent posts into the existing `thread_data.json`, downloads media for the new posts, and re-emits with the original spine unchanged. It preserves existing origin/local/fallback/derived/OCR media records and adds provider-origin plus local-copy records for new branch media. A visible leaf is only the deepest reply returned by that capture; the command does not query descendants individually or claim provider-wide completeness. Fallback upload remains a separate explicit action after origin unavailability is confirmed.
Usual publish path after you like the note:
1. `tw.py graph --id …` — confirm tip vs root.
@@ -43,7 +46,7 @@ Frozen threads (ids listed in `do_not_refetch.txt`, matched against every captur
| Script | Job |
|---|---|
| `tw.py` | locate / graph / refetch / emit / refresh / ocr / merge / sync / publish / migrate-media / backup / fallback / restore-origin |
| `tw.py` | locate / graph / refetch / add-branch / emit / refresh / ocr / merge / sync / publish / migrate-media / backup / fallback / restore-origin |
| `graph_dry_run.py` | tip graph, no vault writes |
| `ingest_gallery.py` | gallery-dl JSON → `thread_data.json` |
| `emit_archive.py` | JSON → notes + assets. Uses canonical media locations. `--force` preserves derived/fallback rows |