2109857584937062679, walker update

This commit is contained in:
ed committed 2026-11-14 22:34:07 -05:00
1 parent 6b9bac9074
commit 2343468bcd
8 files changed
+544 -16

No files matched your search

@@ -0,0 +1,37 @@
---
title: "@NOTimothyLottes This is fascinate."
type: archive
source: twitter
source_url: "https://x.com/Colonthreee/status/2109900069486858664"
author: ":3"
handle: Colonthreee
post_id: "2109900069486858664"
date: 2026-10-13
archived: 2026-11-14
draft: false
tags:
- archive
- twitter
- NOTimothyLottes
description: "@NOTimothyLottes This is fascinate."
in_reply_to: ""
parent_post_id: "2109859799454330995"
---
## Source
- URL: https://x.com/Colonthreee/status/2109900069486858664
- Author: :3 (@Colonthreee)
- Posted: 2026-10-13 06:52:31
## Branch
**1/** **@Colonthreee** ^2109900069486858664
**@NOTimothyLottes**
This is fascinate. Is this a bit like SPIR-V/similar binaries?
## Related
- Spine: [[archive/threads/NOTimothyLottes/2026-10-13-im-finding-that-my-zero-effort-compile-language]]
@@ -0,0 +1,43 @@
---
title: "@NOTimothyLottes do you have some source code exemple to share, just to check the kind of \"human friendly\"? Isn't it just \"Timothy\" friendly? :)"
type: archive
source: twitter
source_url: "https://x.com/leonard_coder/status/2109913619697635371"
author: "Arnaud Carré"
handle: leonard_coder
post_id: "2109913619697635371"
date: 2026-10-13
archived: 2026-11-14
draft: false
tags:
- archive
- twitter
- NOTimothyLottes
description: "@NOTimothyLottes do you have some source code exemple to share, just to check the kind of \"human friendly\"? Isn't it just \"Timothy\" friendly? :)"
in_reply_to: ""
parent_post_id: "2109859799454330995"
---
## Source
- URL: https://x.com/leonard_coder/status/2109913619697635371
- Author: Arnaud Carré (@leonard_coder)
- Posted: 2026-10-13 07:46:21
## Branch
**1/** **@leonard_coder** ^2109913619697635371
**@NOTimothyLottes**
do you have some source code exemple to share, just to check the kind of "human friendly"? Isn't it just "Timothy" friendly? :)
**2/** **@NOTimothyLottes** ^2110006918601465949
**@leonard_coder**
When it converges enough that I'm not still changing stuff, I'll post some examples.
## Related
- Spine: [[archive/threads/NOTimothyLottes/2026-10-13-im-finding-that-my-zero-effort-compile-language]]
@@ -0,0 +1,48 @@
---
title: "I'm finding that my zero effort compile language which is effectively precompiled 32-bit words of binary x86-64 code that get taped together interleaved with flexible 32-bit immediates or rip relative addresses, is converging to something 6502 like."
type: archive
source: twitter
source_url: "https://x.com/NOTimothyLottes/status/2109857584937062679"
author: "NOTimothyLottes"
handle: NOTimothyLottes
post_id: "2109857584937062679"
date: 2026-10-13
archived: 2026-11-14
draft: false
tags:
- archive
- twitter
- NOTimothyLottes
description: "I'm finding that my zero effort compile language which is effectively precompiled 32-bit words of binary x86-64 code that get taped together interleaved with flexible 32-bit immediates or rip relative addresses, is converging to something 6502 like."
in_reply_to: ""
---
## Source
- URL: https://x.com/NOTimothyLottes/status/2109857584937062679
- Author: NOTimothyLottes (@NOTimothyLottes)
- Posted: 2026-10-13 04:03:42
## Thread
**1/** **@NOTimothyLottes** ^2109857584937062679
I'm finding that my zero effort compile language which is effectively precompiled 32-bit words of binary x86-64 code that get taped together interleaved with flexible 32-bit immediates or rip relative addresses, is converging to something 6502 like.
**2/** **@NOTimothyLottes** ^2109858227764461593
In that I have an accumulator, and 2 other operands, not {X and Y}, but more like {operand, and data stack pointer}. Where a "dictionary" acts like named variable in the zero-page.
**3/** **@NOTimothyLottes** ^2109858871657545805
At the point where a virtual ISA has a tiny opcode bitwidth, it becomes amenable to factoring out code generation. But it also maps a space (tiny opcodes) similar to the constraints the 6502 engineers played in.
**4/** **@NOTimothyLottes** ^2109859256287531455
The domain becomes, instant compile, slightly slower, but still native like performance for a simple virtual ISA, and then ability to go easily directly to the physical machine for anything that needs raw assembly.
**5/** **@NOTimothyLottes** ^2109859799454330995
Taken a few weeks of not working on the project to progress to a point where the syntax of the language is human friendly enough to be easily usable for simultaneous usage as a virtualized language and a powerful macro assembly (for CPU and GPU) at the same time.
Branches: [[archive/threads/NOTimothyLottes/2026-10-13-im-finding-that-my-zero-effort-compile-language/2026-10-13-Colonthreee-this-is-fascinate-is-this-a-bit-like-spir-v]], [[archive/threads/NOTimothyLottes/2026-10-13-im-finding-that-my-zero-effort-compile-language/2026-10-13-leonard_coder-do-you-have-some-source-code-exemple-to-share]]
+1
View File
@@ -157,5 +157,6 @@ tags:
- [[archive/threads/NOTimothyLottes/2026-10-04-if-the-deck-1-did-4m-units-youd-only-need-a-2-5]] - [[archive/threads/NOTimothyLottes/2026-10-04-if-the-deck-1-did-4m-units-youd-only-need-a-2-5]]
- [[archive/threads/NOTimothyLottes/2026-10-04-thread-serious-talk-on-steamdeck-2-speculation]] - [[archive/threads/NOTimothyLottes/2026-10-04-thread-serious-talk-on-steamdeck-2-speculation]]
- [[archive/threads/NOTimothyLottes/2026-10-06-if-one-used-bc7-for-a-giant-vq-code-book-like-a]] - [[archive/threads/NOTimothyLottes/2026-10-06-if-one-used-bc7-for-a-giant-vq-code-book-like-a]]
- [[archive/threads/NOTimothyLottes/2026-10-13-im-finding-that-my-zero-effort-compile-language]]
- [[archive/threads/NOTimothyLottes/2026-11-07-not-a-joke-im-re-trying-development-through]] - [[archive/threads/NOTimothyLottes/2026-11-07-not-a-joke-im-re-trying-development-through]]
- [[archive/threads/NOTimothyLottes/2026-11-13-thankfully-a-friend-sent-this-to-me-otherwise-id]] - [[archive/threads/NOTimothyLottes/2026-11-13-thankfully-a-friend-sent-this-to-me-otherwise-id]]
+17 -1
View File
@@ -21,7 +21,7 @@ python scripts/twitter/tw.py reslug --all
python scripts/twitter/tw.py reslug --all --apply python scripts/twitter/tw.py reslug --all --apply
python scripts/twitter/tw.py relabel --all python scripts/twitter/tw.py relabel --all
python scripts/twitter/tw.py relabel --all --apply python scripts/twitter/tw.py relabel --all --apply
python scripts/twitter/tw.py discover --id <post-id> python scripts/twitter/tw.py discover --id <post-id> [--cap N] [--max-requests N]
``` ```
`paths.py` finds the vault two parents above this file. Dumps live at `../manual_slop/docs/twitter`. Scratch lives at `../Threadwell-ai/scratch`. `paths.py` finds the vault two parents above this file. Dumps live at `../manual_slop/docs/twitter`. Scratch lives at `../Threadwell-ai/scratch`.
@@ -81,6 +81,22 @@ were found has a subtree not yet reached, so it is walked too. One
`--id` reaches the whole thread. `discover` is read-only against the `--id` reaches the whole thread. `discover` is read-only against the
vault and prints no cookies and no cursor values. vault and prints no cookies and no cursor values.
### Rate limit
The endpoint allows 300 requests per 15-minute window. The walk stays
inside that window:
- requests are paced 5 seconds apart, the same pacing the gallery-dl
captures use;
- `--max-requests` (default 180) caps the whole walk, not one focal;
- a 429 raises instead of sleeping. gallery-dl's default is to wait
out the window and retry, which reads as a hung command; `discover`
stops, reports what it found, and prints the refill time.
Re-run `discover` after the window refills to finish the walk. The
post ids already found are printed before the stop, so a partial run
still feeds `--branch`.
A capture is complete when no post reports more replies than the A capture is complete when no post reports more replies than the
capture holds. `refresh` overwrites `thread_data.json` with whatever capture holds. `refresh` overwrites `thread_data.json` with whatever
gallery-dl returns, so a refresh that drops posts loses them; the way gallery-dl returns, so a refresh that drops posts loses them; the way
+126 -3
View File
@@ -41,6 +41,14 @@ def load(name: str) -> dict:
return json.loads((FIXTURES / name).read_text(encoding="utf-8")) return json.loads((FIXTURES / name).read_text(encoding="utf-8"))
class AbortExtraction(Exception):
"""Stands in for gallery-dl's AbortExtraction.
The walker matches the exception by class name, so the test needs a
class with that name.
"""
class ParsePageTests(unittest.TestCase): class ParsePageTests(unittest.TestCase):
"""The page parser must find cursors at every depth.""" """The page parser must find cursors at every depth."""
@@ -257,7 +265,7 @@ class DiscoverTests(unittest.TestCase):
self.assertGreater(len(walked_focals), len(result.per_focal)) self.assertGreater(len(walked_focals), len(result.per_focal))
def test_capture_dir_records_the_pages(self) -> None: def test_capture_dir_records_the_pages(self) -> None:
"""A run writes its response pages so it can be inspected after.""" """A run writes one response page per request, none overwritten."""
import tempfile import tempfile
class FakeApi: class FakeApi:
@@ -271,17 +279,47 @@ class DiscoverTests(unittest.TestCase):
with tempfile.TemporaryDirectory() as tmp: with tempfile.TemporaryDirectory() as tmp:
target = Path(tmp) / "walker" target = Path(tmp) / "walker"
walker.discover( result = walker.discover(
("2121244702649532667",), ("2121244702649532667",),
request_cap=2, request_cap=2,
extractor=FakeExtractor(), extractor=FakeExtractor(),
capture_dir=target, capture_dir=target,
) )
pages = sorted(target.glob("page-*.json")) pages = sorted(target.glob("page-*.json"))
self.assertEqual(len(pages), 2) self.assertEqual(len(pages), result.calls)
body = json.loads(pages[0].read_text(encoding="utf-8")) body = json.loads(pages[0].read_text(encoding="utf-8"))
self.assertIn("threaded_conversation_with_injections_v2", body["data"]) self.assertIn("threaded_conversation_with_injections_v2", body["data"])
def test_capture_dir_keeps_every_page_of_a_multi_focal_walk(self) -> None:
"""A walk over several focals must not reuse a page file name.
The page number counts the whole walk, not one focal, so the
evidence of one focal cannot overwrite another's.
"""
import tempfile
class FakeApi:
features_pagination = {"feature": True}
def _call(self, endpoint: str, params: dict, method: str = "GET") -> dict:
return load("page_nested_cursor.json")
class FakeExtractor:
api = FakeApi()
with tempfile.TemporaryDirectory() as tmp:
target = Path(tmp) / "walker"
result = walker.discover(
("2121244702649532667",),
request_cap=2,
extractor=FakeExtractor(),
capture_dir=target,
)
names = sorted(p.name for p in target.glob("page-*.json"))
self.assertEqual(len(names), len(set(names)))
self.assertEqual(len(names), result.calls)
self.assertGreater(len(result.per_focal), 1)
def test_posts_from_other_conversations_are_excluded(self) -> None: def test_posts_from_other_conversations_are_excluded(self) -> None:
"""A conversation page can pull in other conversations' posts. """A conversation page can pull in other conversations' posts.
@@ -314,6 +352,91 @@ class DiscoverTests(unittest.TestCase):
self.assertNotIn(post_id, other_posts) self.assertNotIn(post_id, other_posts)
class RateLimitTests(unittest.TestCase):
"""The walk must stay inside the endpoint's rate-limit window."""
def test_budget_stops_the_walk(self) -> None:
"""The whole walk is capped, not just one focal."""
tracker = walker._RateTracker(max_requests=3)
tracker.check_budget()
tracker.check_budget()
tracker.check_budget()
with self.assertRaises(walker.RateLimitReached):
tracker.check_budget()
def test_budget_error_names_the_budget(self) -> None:
tracker = walker._RateTracker(max_requests=1)
tracker.check_budget()
with self.assertRaises(walker.RateLimitReached) as caught:
tracker.check_budget()
self.assertIn("budget of 1", str(caught.exception))
def test_a_nearly_spent_window_stops_the_walk(self) -> None:
"""The walk stops before it spends the last requests in a window."""
tracker = walker._RateTracker(max_requests=100)
class FakeResponse:
headers = {
"x-rate-limit-remaining": "5",
"x-rate-limit-reset": "1794705157",
}
tracker.observe(FakeResponse())
self.assertEqual(tracker.remaining, 5)
with self.assertRaises(walker.RateLimitReached) as caught:
tracker.check_budget()
self.assertIn("nearly spent", str(caught.exception))
self.assertIn("5 left", str(caught.exception))
def test_the_refill_time_is_reported(self) -> None:
"""The operator learns when the window refills."""
tracker = walker._RateTracker(max_requests=100)
class FakeResponse:
headers = {
"x-rate-limit-remaining": "0",
"x-rate-limit-reset": "1794705157",
}
tracker.observe(FakeResponse())
error = tracker.rate_limited_error()
self.assertIn("refills at", str(error))
self.assertIsNotNone(error.reset_at)
def test_gallery_dl_abort_becomes_a_rate_limit_error(self) -> None:
"""gallery-dl's AbortExtraction is translated, not swallowed.
With ``ratelimit=abort`` gallery-dl raises instead of sleeping
until the window refills. The walker turns that into a message
the operator can act on.
"""
tracker = walker._RateTracker(max_requests=100)
class FakeApi:
features_pagination = {"feature": True}
def _call(self, endpoint: str, params: dict, method: str = "GET") -> dict:
raise AbortExtraction("Rate limit exceeded")
class FakeExtractor:
api = FakeApi()
with self.assertRaises(walker.RateLimitReached):
walker._request(
FakeExtractor(),
"2121244702649532667",
None,
walker.request_shape()[0],
walker.request_shape()[1],
tracker,
)
def test_the_walk_reports_the_refill_time(self) -> None:
tracker = walker._RateTracker(max_requests=100)
error = tracker.rate_limited_error()
self.assertIn("rate limit reached", str(error))
class KenpexRegressionTests(unittest.TestCase): class KenpexRegressionTests(unittest.TestCase):
"""The kenpex conversation: the fixture NOTE.md section 5 asked for.""" """The kenpex conversation: the fixture NOTE.md section 5 asked for."""
+23 -1
View File
@@ -30,6 +30,7 @@ if str(_HERE) not in sys.path:
from paths import COOKIES, DUMPS, FROZEN, HERE, SCRATCH, VAULT from paths import COOKIES, DUMPS, FROZEN, HERE, SCRATCH, VAULT
from models import PostData, ThreadData, load_thread from models import PostData, ThreadData, load_thread
from walker import DEFAULT_MAX_REQUESTS
try: try:
from twitter.tree import spine_from_tip, spine_ids, spine_quote_ids from twitter.tree import spine_from_tip, spine_ids, spine_quote_ids
@@ -773,6 +774,10 @@ def cmd_discover(args: argparse.Namespace) -> int:
Read-only against the vault. Prints one post id per line to stdout Read-only against the vault. Prints one post id per line to stdout
so the output pipes straight into ``refresh --branch``. Captured so the output pipes straight into ``refresh --branch``. Captured
response pages go to ``scratch/walker_<id>/``. response pages go to ``scratch/walker_<id>/``.
The walk stays inside the endpoint's rate-limit window: requests are
paced, and the walk stops with a message naming the refill time
rather than sleeping through it.
""" """
focals = tuple(dict.fromkeys(args.ids)) focals = tuple(dict.fromkeys(args.ids))
if not focals: if not focals:
@@ -793,6 +798,7 @@ def cmd_discover(args: argparse.Namespace) -> int:
focals, focals,
request_cap=args.cap, request_cap=args.cap,
capture_dir=SCRATCH / f"walker_{focals[0]}", capture_dir=SCRATCH / f"walker_{focals[0]}",
max_requests=args.max_requests,
) )
except WalkerError as exc: except WalkerError as exc:
raise SystemExit(f"walker failed: {exc}") from exc raise SystemExit(f"walker failed: {exc}") from exc
@@ -800,9 +806,15 @@ def cmd_discover(args: argparse.Namespace) -> int:
print(post_id) print(post_id)
print( print(
f"posts={len(result.post_ids)} focals={len(focals)} " f"posts={len(result.post_ids)} focals={len(focals)} "
f"requests={result.calls} stop={result.stopped_reason}", f"requests={result.calls} http={result.http_requests} "
f"stop={result.stopped_reason}",
file=sys.stderr, file=sys.stderr,
) )
if result.reset_at is not None:
refill = datetime.fromtimestamp(
result.reset_at, tz=timezone.utc
).strftime("%Y-%m-%d %H:%M:%S UTC")
print(f"rate-limit window refills at {refill}", file=sys.stderr)
return 0 return 0
@@ -1120,6 +1132,16 @@ def build_parser() -> argparse.ArgumentParser:
default=20, default=20,
help="maximum requests per focal (default 20)", help="maximum requests per focal (default 20)",
) )
discover_p.add_argument(
"--max-requests",
type=int,
default=DEFAULT_MAX_REQUESTS,
help=(
"maximum requests for the whole walk (default "
f"{DEFAULT_MAX_REQUESTS}); keeps the walk inside one "
"rate-limit window"
),
)
return parser return parser
+249 -11
View File
@@ -22,6 +22,7 @@ import json
import logging import logging
import re import re
import textwrap import textwrap
import time
from collections import deque from collections import deque
from dataclasses import dataclass from dataclasses import dataclass
from pathlib import Path from pathlib import Path
@@ -42,18 +43,165 @@ _gallery_dl: Any = None
# variables template it reads from its own source. # variables template it reads from its own source.
FOCAL_TWEET_ID = "__focal_tweet_id__" FOCAL_TWEET_ID = "__focal_tweet_id__"
#: X allows 300 TweetDetail requests per 15-minute window. Five seconds
#: between requests puts a walk at 180 per window, the same pacing the
#: gallery-dl CLI uses for twitter captures.
DEFAULT_REQUEST_INTERVAL = 5.0
#: Requests one walk may issue. The window holds 300 over 15 minutes, so
#: a walk capped here finishes inside one window without needing a
#: refill.
DEFAULT_MAX_REQUESTS = 180
#: Remaining-request count at which a walk stops and waits for the
#: rate-limit window to refill. gallery-dl treats "remaining < 6" the
#: same way.
RATE_RESERVE = 6
class WalkerError(RuntimeError): class WalkerError(RuntimeError):
"""Raised when the walker cannot build or run a request.""" """Raised when the walker cannot build or run a request."""
class RateLimitReached(WalkerError):
"""Raised when the walk must stop because the request budget ran out.
``reset_at`` is the epoch second the endpoint's rate-limit window
refills, when the endpoint said so.
"""
def __init__(self, message: str, reset_at: float | None = None) -> None:
super().__init__(message)
self.reset_at = reset_at
class _RateTracker:
"""Counts the walk's requests and reads the endpoint's rate-limit window.
``TwitterAPI._call`` runs inside ``while True:``. On a 429, or on a
response whose ``x-rate-limit-remaining`` has nearly run out, it
calls ``_handle_ratelimit`` — which sleeps until the window refills
and then retries. A walk that blocks there looks hung and silently
burns the operator's time.
The walk therefore configures gallery-dl to ``ratelimit=abort`` (so
it raises instead of sleeping) and to ``sleep-request`` (so it paces
itself). This tracker counts the requests that actually go out and
reads the window from every response, so the walk can stop before it
spends the last requests in a window and report when the window
refills.
"""
def __init__(
self,
*,
max_requests: int = DEFAULT_MAX_REQUESTS,
) -> None:
self.max_requests = max_requests
self.calls = 0
self.pages = 0
self.remaining: int | None = None
self.reset_at: float | None = None
def observe(self, response: Any) -> None:
"""Record the rate-limit headers of one response."""
headers = getattr(response, "headers", None)
if not headers:
return
remaining = headers.get("x-rate-limit-remaining")
if remaining is not None:
try:
self.remaining = int(remaining)
except (TypeError, ValueError):
pass
reset = headers.get("x-rate-limit-reset")
if reset is not None:
try:
self.reset_at = float(reset)
except (TypeError, ValueError):
pass
def install(self, extractor: Any) -> None:
"""Watch the extractor's HTTP responses.
The session is wrapped rather than ``Extractor.request`` so that
every request is counted, including the retries gallery-dl makes
on a 5xx.
A 429 is raised here rather than returned. gallery-dl's request
layer retries a 429 with a 60-second sleep, four times over,
before the API layer ever sees it — four minutes of apparent
hang. Raising at the session boundary skips that loop.
"""
session = getattr(extractor, "session", None)
request = getattr(session, "request", None)
if request is None:
return
original = request
def observed(*args: Any, **kwargs: Any) -> Any:
self.check_budget()
response = original(*args, **kwargs)
self.observe(response)
if getattr(response, "status_code", None) == 429:
raise self.rate_limited_error()
return response
session.request = observed
def check_budget(self) -> None:
"""Stop the walk before it spends past its budget or its window."""
if self.calls >= self.max_requests:
raise RateLimitReached(
f"request budget of {self.max_requests} spent"
)
if self.remaining is not None and self.remaining <= RATE_RESERVE:
raise RateLimitReached(
f"rate-limit window nearly spent ({self.remaining} left); "
f"it refills at {self.refill_time()}",
reset_at=self.reset_at,
)
self.calls += 1
def refill_time(self) -> str:
"""Return the refill time as a local timestamp, or a placeholder."""
if self.reset_at is None:
return "an unknown time"
return time.strftime(
"%Y-%m-%d %H:%M:%S", time.localtime(self.reset_at)
)
def rate_limited_error(self) -> RateLimitReached:
"""Return the error to raise when the endpoint refused the walk."""
if self.remaining is not None:
detail = f"{self.remaining} requests left"
else:
detail = "the window is exhausted"
return RateLimitReached(
f"rate limit reached ({detail}); it refills at {self.refill_time()}",
reset_at=self.reset_at,
)
def next_page(self) -> int:
"""Return a page number that no other page in this walk uses."""
self.pages += 1
return self.pages
@dataclass(frozen=True) @dataclass(frozen=True)
class WalkerResult: class WalkerResult:
"""What one walk found.""" """What one walk found.
``calls`` counts the API requests the walk made. ``http_requests``
counts the HTTP requests that reached the endpoint, which is the
number the rate-limit window counts.
"""
post_ids: tuple[str, ...] post_ids: tuple[str, ...]
calls: int calls: int
per_focal: dict[str, tuple[str, ...]] per_focal: dict[str, tuple[str, ...]]
stopped_reason: str stopped_reason: str
reset_at: float | None = None
http_requests: int = 0
@dataclass(frozen=True) @dataclass(frozen=True)
@@ -155,14 +303,30 @@ def request_shape() -> tuple[dict[str, Any], dict[str, Any]]:
return variables, toggles return variables, toggles
def build_extractor() -> Any: def build_extractor(
*,
request_interval: float = DEFAULT_REQUEST_INTERVAL,
) -> Any:
"""Build a gallery-dl extractor that is authenticated with the vault cookies. """Build a gallery-dl extractor that is authenticated with the vault cookies.
Cookies, the CSRF token, and the transaction id all come from Cookies, the CSRF token, and the transaction id all come from
gallery-dl's own request layer, so the walker speaks the same gallery-dl's own request layer, so the walker speaks the same
protocol gallery-dl does. protocol gallery-dl does.
Two settings keep the walk inside the endpoint's rate-limit window:
- ``sleep-request`` paces the requests. The endpoint allows a fixed
number per window, so an unpaced walk exhausts it.
- ``ratelimit`` is set to ``abort``. Left at its default, a 429 makes
gallery-dl sleep until the window refills and then retry, inside a
``while True:`` loop — the walk looks hung. Aborting lets the
walker report the refill time instead.
""" """
twitter = _load_gallery_dl() twitter = _load_gallery_dl()
from gallery_dl import config as gd_config
gd_config.set(("extractor", "twitter"), "sleep-request", request_interval)
gd_config.set(("extractor", "twitter"), "ratelimit", "abort")
url = "https://x.com/i/web/status/1" url = "https://x.com/i/web/status/1"
extractor = twitter.TwitterTweetExtractor.from_url(url) extractor = twitter.TwitterTweetExtractor.from_url(url)
if extractor is None: if extractor is None:
@@ -371,6 +535,7 @@ def _request(
cursor: str | None, cursor: str | None,
variables: dict[str, Any], variables: dict[str, Any],
toggles: dict[str, Any], toggles: dict[str, Any],
tracker: "_RateTracker | None" = None,
) -> dict[str, Any]: ) -> dict[str, Any]:
"""Issue one TweetDetail request through gallery-dl's request layer.""" """Issue one TweetDetail request through gallery-dl's request layer."""
request_variables = dict(variables) request_variables = dict(variables)
@@ -384,12 +549,22 @@ def _request(
), ),
"fieldToggles": json.dumps(toggles, separators=(",", ":"), ensure_ascii=False), "fieldToggles": json.dumps(toggles, separators=(",", ":"), ensure_ascii=False),
} }
body = extractor.api._call(tweet_detail_endpoint(), params, method="GET") try:
body = extractor.api._call(tweet_detail_endpoint(), params, method="GET")
except Exception as exc:
if tracker is not None and _is_rate_limit_abort(exc):
raise tracker.rate_limited_error() from exc
raise
if not isinstance(body, dict): if not isinstance(body, dict):
raise WalkerError("TweetDetail returned a non-object body") raise WalkerError("TweetDetail returned a non-object body")
return body return body
def _is_rate_limit_abort(exc: Exception) -> bool:
"""True when gallery-dl aborted because the rate limit was hit."""
return type(exc).__name__ == "AbortExtraction"
def _walk_focal( def _walk_focal(
extractor: Any, extractor: Any,
focal_id: str, focal_id: str,
@@ -397,9 +572,16 @@ def _walk_focal(
variables: dict[str, Any], variables: dict[str, Any],
toggles: dict[str, Any], toggles: dict[str, Any],
capture_dir: Path | None = None, capture_dir: Path | None = None,
tracker: "_RateTracker | None" = None,
page_cache: dict[str | None, dict[str, Any]] | None = None,
) -> tuple[set[str], dict[str, int], dict[str, str | None], int, str]: ) -> tuple[set[str], dict[str, int], dict[str, str | None], int, str]:
"""Walk every cursor under one focal. """Walk every cursor under one focal.
``page_cache`` holds the pages already fetched in this walk, keyed
by cursor. The walk expands its focal set over posts in the same
conversation, and each new focal would otherwise re-fetch the same
pages from the top.
Returns ``(post_ids, reply_counts, reply_to_ids, calls, reason)``. Returns ``(post_ids, reply_counts, reply_to_ids, calls, reason)``.
""" """
post_ids: set[str] = set() post_ids: set[str] = set()
@@ -409,30 +591,65 @@ def _walk_focal(
pending: deque[str | None] = deque([None]) pending: deque[str | None] = deque([None])
calls = 0 calls = 0
conversation: str | None = None conversation: str | None = None
reason = "exhausted"
productive = 0
while pending and calls < request_cap: while pending and calls < request_cap:
cursor = pending.popleft() cursor = pending.popleft()
calls += 1 cached = None if page_cache is None else page_cache.get(cursor)
body = _request(extractor, focal_id, cursor, variables, toggles) if cached is not None:
if capture_dir is not None: body = cached
_capture_page(capture_dir, calls, body) else:
calls += 1
try:
body = _request(extractor, focal_id, cursor, variables, toggles, tracker)
except RateLimitReached as exc:
# The walk ran out of room. Report what it found so the
# operator can resume with another run.
reason = "rate-limited"
logger.info("walk stopped: %s", exc)
break
if page_cache is not None:
page_cache[cursor] = body
if capture_dir is not None and cached is None:
_capture_page(capture_dir, tracker.next_page() if tracker else calls, body)
page = parse_page(body) page = parse_page(body)
if conversation is None: if conversation is None:
conversation = _focal_conversation(page, focal_id) conversation = _focal_conversation(page, focal_id)
found = 0
for index, tweet_id in enumerate(page.post_ids): for index, tweet_id in enumerate(page.post_ids):
if page.conversation_ids[index] != conversation: if page.conversation_ids[index] != conversation:
continue continue
if tweet_id not in post_ids:
found += 1
post_ids.add(tweet_id) post_ids.add(tweet_id)
reply_counts[tweet_id] = page.reply_counts[index] reply_counts[tweet_id] = page.reply_counts[index]
reply_to[tweet_id] = page.reply_to_ids[index] reply_to[tweet_id] = page.reply_to_ids[index]
if found:
productive += 1
for value in page.cursors: for value in page.cursors:
if value in seen_cursors: if value in seen_cursors:
continue continue
seen_cursors.add(value) seen_cursors.add(value)
pending.append(value) pending.append(value)
reason = "request-cap" if pending else "exhausted" if not found and not productive:
# The page held no posts and no earlier page did either.
# Its cursors lead away from the conversation's posts, so
# following them spends the budget on empty pages.
reason = "no-posts"
break
if reason == "exhausted":
reason = "request-cap" if pending else "exhausted"
return post_ids, reply_counts, reply_to, calls, reason return post_ids, reply_counts, reply_to, calls, reason
def _clear_capture_dir(capture_dir: Path) -> None:
"""Remove the pages of a previous walk from the capture directory."""
if not capture_dir.is_dir():
return
for path in capture_dir.glob("page-*.json"):
path.unlink()
def _capture_page(capture_dir: Path, ordinal: int, body: dict[str, Any]) -> None: def _capture_page(capture_dir: Path, ordinal: int, body: dict[str, Any]) -> None:
"""Write one response page so a run can be inspected afterwards. """Write one response page so a run can be inspected afterwards.
@@ -493,6 +710,7 @@ def discover(
extractor: Any | None = None, extractor: Any | None = None,
max_focals: int = 40, max_focals: int = 40,
capture_dir: Path | None = None, capture_dir: Path | None = None,
max_requests: int = DEFAULT_MAX_REQUESTS,
) -> WalkerResult: ) -> WalkerResult:
"""Walk each focal's conversation and return every post id found. """Walk each focal's conversation and return every post id found.
@@ -503,16 +721,25 @@ def discover(
``request_cap`` bounds the requests issued per focal. ``request_cap`` bounds the requests issued per focal.
``max_focals`` bounds how many focals the expansion may add, so a ``max_focals`` bounds how many focals the expansion may add, so a
conversation that keeps reporting unreached children cannot run conversation that keeps reporting unreached children cannot run
away. Requests are sequential. No cursor value is logged or away. ``max_requests`` bounds the requests the whole walk may
persisted. issue, so it stays inside one rate-limit window. Requests are
sequential. No cursor value is logged or persisted.
""" """
if not focal_ids: if not focal_ids:
return WalkerResult(post_ids=(), calls=0, per_focal={}, stopped_reason="no-focals") return WalkerResult(post_ids=(), calls=0, per_focal={}, stopped_reason="no-focals")
variables, toggles = request_shape() variables, toggles = request_shape()
if extractor is None: if extractor is None:
extractor = build_extractor() extractor = build_extractor()
tracker = _RateTracker(max_requests=max_requests)
tracker.install(extractor)
if capture_dir is not None:
# The capture dir holds one walk's evidence. A previous run's
# pages would otherwise sit beside this run's, with page
# numbers that no longer line up.
_clear_capture_dir(capture_dir)
all_posts: set[str] = set() all_posts: set[str] = set()
per_focal: dict[str, tuple[str, ...]] = {} per_focal: dict[str, tuple[str, ...]] = {}
page_cache: dict[str | None, dict[str, Any]] = {}
calls = 0 calls = 0
stopped = "exhausted" stopped = "exhausted"
walked: set[str] = set() walked: set[str] = set()
@@ -523,12 +750,21 @@ def discover(
continue continue
walked.add(focal) walked.add(focal)
posts, reply_counts, reply_to, focal_calls, reason = _walk_focal( posts, reply_counts, reply_to, focal_calls, reason = _walk_focal(
extractor, focal, request_cap, variables, toggles, capture_dir extractor,
focal,
request_cap,
variables,
toggles,
capture_dir,
tracker,
page_cache,
) )
per_focal[focal] = tuple(sorted(posts)) per_focal[focal] = tuple(sorted(posts))
all_posts.update(posts) all_posts.update(posts)
calls += focal_calls calls += focal_calls
stopped = reason stopped = reason
if reason == "rate-limited":
break
if len(walked) >= max_focals: if len(walked) >= max_focals:
stopped = "focal-cap" stopped = "focal-cap"
break break
@@ -544,4 +780,6 @@ def discover(
calls=calls, calls=calls,
per_focal=per_focal, per_focal=per_focal,
stopped_reason=stopped, stopped_reason=stopped,
reset_at=tracker.reset_at,
http_requests=tracker.calls,
) )