2109857584937062679, walker update

This commit is contained in:
ed committed 2026-11-14 22:34:07 -05:00
1 parent 6b9bac9074
commit 2343468bcd
8 files changed
+544 -16

No files matched your search

+17 -1
View File
@@ -21,7 +21,7 @@ python scripts/twitter/tw.py reslug --all
python scripts/twitter/tw.py reslug --all --apply
python scripts/twitter/tw.py relabel --all
python scripts/twitter/tw.py relabel --all --apply
python scripts/twitter/tw.py discover --id <post-id>
python scripts/twitter/tw.py discover --id <post-id> [--cap N] [--max-requests N]
```
`paths.py` finds the vault two parents above this file. Dumps live at `../manual_slop/docs/twitter`. Scratch lives at `../Threadwell-ai/scratch`.
@@ -81,6 +81,22 @@ were found has a subtree not yet reached, so it is walked too. One
`--id` reaches the whole thread. `discover` is read-only against the
vault and prints no cookies and no cursor values.
### Rate limit
The endpoint allows 300 requests per 15-minute window. The walk stays
inside that window:
- requests are paced 5 seconds apart, the same pacing the gallery-dl
captures use;
- `--max-requests` (default 180) caps the whole walk, not one focal;
- a 429 raises instead of sleeping. gallery-dl's default is to wait
out the window and retry, which reads as a hung command; `discover`
stops, reports what it found, and prints the refill time.
Re-run `discover` after the window refills to finish the walk. The
post ids already found are printed before the stop, so a partial run
still feeds `--branch`.
A capture is complete when no post reports more replies than the
capture holds. `refresh` overwrites `thread_data.json` with whatever
gallery-dl returns, so a refresh that drops posts loses them; the way