diff --git a/archive/threads/NOTimothyLottes/2023-08-06-the-next-challenge-is-overscan-safety-since-real/index.md b/archive/threads/NOTimothyLottes/2023-08-06-the-next-challenge-is-overscan-safety-since-real/index.md index c2251f6..116a066 100644 --- a/archive/threads/NOTimothyLottes/2023-08-06-the-next-challenge-is-overscan-safety-since-real/index.md +++ b/archive/threads/NOTimothyLottes/2023-08-06-the-next-challenge-is-overscan-safety-since-real/index.md @@ -1,30 +1,46 @@ --- -title: "The next challenge is overscan safety, since real non-VGA CRTs will be used in development." +title: "1st SPC project: to be based on vintage 640x480 render (VGA style), +fallback to 640x240 for 15 KHz arcade CRTs (PGM-esk), and CRT shader scaling on Deck and OLED/LCD." type: archive source: twitter -source_url: "https://x.com/NOTimothyLottes/status/1688022114962374656" +source_url: "https://x.com/NOTimothyLottes/status/1688018382543597568" author: "NOTimothyLottes" handle: NOTimothyLottes -post_id: "1688022114962374656" +post_id: "1688018382543597568" date: 2023-08-06 -archived: 2026-08-23 -draft: true +archived: 2026-08-24 +draft: false tags: - archive - twitter - NOTimothyLottes -description: "The next challenge is overscan safety, since real non-VGA CRTs will be used in development." +description: "1st SPC project: to be based on vintage 640x480 render (VGA style), +fallback to 640x240 for 15 KHz arcade CRTs (PGM-esk), and CRT shader scaling on Deck and OLED/LCD." in_reply_to: "" --- ## Source -- URL: https://x.com/NOTimothyLottes/status/1688022114962374656 +- URL: https://x.com/NOTimothyLottes/status/1688018382543597568 - Author: NOTimothyLottes (@NOTimothyLottes) -- Posted: 2023-08-06 03:00:08 +- Posted: 2023-08-06 02:45:18 ## Thread **1/** +1st SPC project: to be based on vintage 640x480 render (VGA style), +fallback to 640x240 for 15 KHz arcade CRTs (PGM-esk), and CRT shader scaling on Deck and OLED/LCD. Use mix of render pixel aspect changes, running less than 480 lines, and letter-box to keep integer Y scaling. + +![](https://pbs.twimg.com/media/F20LgFtWQAAMtcv?format=png&name=orig) + +**2/** + +My 15 KHz CRT output path when docked is HD Fury Nano (won't do <480p) doing HDMI to VGA, with a sync combiner (to workaround GBS-C bug), into GBS-C which line drops 480p ~60Hz to 240p ~60Hz and outputs component. The 640 width gets maintained even if some CRTs cannot resolve it. + +**3/** + +Presents issues for UI, namely that my bitmap fonts need to look good in both 480p and 240p modes, so for editor (assembler) I'm going with a font that uses right angles and designed around 2x2 stroke width, in a 8x16 (pow2) bitmap, so 640x240p still gets a readable 8x8 (below) + +![](https://pbs.twimg.com/media/F20Ogh8XoAAq7hR?format=png&name=orig) + +**4/** + The next challenge is overscan safety, since real non-VGA CRTs will be used in development. The Y axis has an easy fix, simply keep cursor pinned in the 'safe' zone, but still print lines in the overscan area (so they can be seen when attached to non-overscan CRTs/etc too). diff --git a/archive/threads/NOTimothyLottes/2023-08-06-yeah-perhaps-wildly-alien-but-for-those-who-grew/index.md b/archive/threads/NOTimothyLottes/2023-08-06-yeah-perhaps-wildly-alien-but-for-those-who-grew/index.md index cc1c664..adc9988 100644 --- a/archive/threads/NOTimothyLottes/2023-08-06-yeah-perhaps-wildly-alien-but-for-those-who-grew/index.md +++ b/archive/threads/NOTimothyLottes/2023-08-06-yeah-perhaps-wildly-alien-but-for-those-who-grew/index.md @@ -1,32 +1,100 @@ --- -title: "Yeah, perhaps wildly alien." +title: "SPC: RDNA2 ASSEMBLER [thread] ..." type: archive source: twitter -source_url: "https://x.com/NOTimothyLottes/status/1688286623375454208" +source_url: "https://x.com/NOTimothyLottes/status/1688275737692815360" author: "NOTimothyLottes" handle: NOTimothyLottes -post_id: "1688286623375454208" +post_id: "1688275737692815360" date: 2023-08-06 -archived: 2026-08-23 -draft: true +archived: 2026-08-24 +draft: false tags: - archive - twitter - NOTimothyLottes -description: "Yeah, perhaps wildly alien." +description: "SPC: RDNA2 ASSEMBLER [thread] ..." in_reply_to: "" --- ## Source -- URL: https://x.com/NOTimothyLottes/status/1688286623375454208 +- URL: https://x.com/NOTimothyLottes/status/1688275737692815360 - Author: NOTimothyLottes (@NOTimothyLottes) -- Posted: 2023-08-06 20:31:12 +- Posted: 2023-08-06 19:47:57 ## Thread **1/** +SPC: RDNA2 ASSEMBLER [thread] ... Did initial design plan this weekend. The aim is a GPU-side assembler+editor, building RDNA2 binary images for Steam Deck. Live edit, live data view, etc. Assembler and editor running in massive vectorized shader code. + +**2/** + +Good reason not to use text input for assembler, instead will use a custom binary format, and treat all 32-bit words in the binary the exact same way (so massively parallel assembly step). This is parallel design 101. + +**3/** + +Some basics: RDNA2 instructions are 1-N 32-bit words. Each of the 32-bit components have a bunch of fields. Will typically need 1-6 fields per 32-bit word, as will be joining some fields into one argument (like DLC/GLC) below. + +![](https://pbs.twimg.com/media/F232wWkX0AAZWPh?format=jpg&name=orig) +![](https://pbs.twimg.com/media/F233F_sWMAAqHBo?format=png&name=orig) + +**4/** + +Could argue that MIMG has the most fields, but in practice many of those fields will be joined (see example), and I'm not likely to be using NSA encoding. So still will only need maximum 6 arguments to build a 32-bit value. + +![](https://pbs.twimg.com/media/F233qJyWQAAjmOo?format=jpg&name=orig) +![](https://pbs.twimg.com/media/F2333WKXEAAc2fZ?format=png&name=orig) + +**5/** + +Custom editor, using a custom font, designed for vintage CRTs. Wanted 5-character symbol names for easy packing in 30-bit values. So setup a 6-bit character table. This is using a modified 'right-angle-only' font (like the one below). So some glyphs alias, and only upper case. + +![](https://pbs.twimg.com/media/F234Y1IXcAECWTB?format=png&name=orig) + +**6/** + +Will need some kind of symbol table, what ends up providing the arguments to the rules that build the 32-bit words that comprise the final binary. The symbol indirection provides a way to name registers for example, and then change the register mapping later. + +![](https://pbs.twimg.com/media/F2357mwXUAAC6O6?format=png&name=orig) + +**7/** + +Notice SMEM ops have 21-bit immediate offsets. I'll use 20-bits of that for an absolute positive offset from KART base. 1 MiB of easy data. The editor will relink tagged symbols that represent offsets automatically. + +![](https://pbs.twimg.com/media/F236eA_XkAAc8-_?format=jpg&name=orig) + +**8/** + +Next will need a collection of rules, how to join the 6 possible symbol arguments together. Rules for example for each of the RDNA2 instruction encodings (VOPC, etc). This reduces to just a 6 entry 5-bit left shift amount associated with each argument. + +![](https://pbs.twimg.com/media/F237AEiWgAAkzD0?format=png&name=orig) + +**9/** + +But will need a 7th argument, hard coded to a signed 16-bit *4 relative offset as this is how RDNA2 does all it's relative branches. This will be setup inside the editor to auto relink as source is edited. + +![](https://pbs.twimg.com/media/F237uqaXAAAKAFl?format=jpg&name=orig) + +**10/** + +Source then is simple, two 128-bit values per 32-bit word in the binary. One to select {RULE, SYMBOLS, RELATIVE BRANCH}, and another for {LABEL, COMMENT}. + +![](https://pbs.twimg.com/media/F238Q-zXYAAO1Eo?format=png&name=orig) + +**11/** + +And then the source editor, shows a line per 32-bit value (so VOP3's for example would take 2 lines). So the 'assembler' is deceptively simple, but powerful enough to do everything required to generate GPU binary code. And all the opcode generation is in the data itself :) + +![](https://pbs.twimg.com/media/F238wJ8WoAAtEfw?format=png&name=orig) + +**12/** + +The editor itself will have 3 views {symbol table, rule table, source table}. The editor itself will use 2 copies of everything, so each edit, it's one entry/lane, reading prior state and updating to the new. Dead simple, dead parallel (not necessarily efficient) :) + +**13/** + Yeah, perhaps wildly alien. But for those who grew up with MOD-like trackers (like Impulse Tracker below), the interface will almost feel like home (those had simple {instrument, pattern, arangement} editors, with a very spreadsheet feel). -![](https://pbs.twimg.com/media/F23_OOhXgAUBtlb?format=png&name=orig) \ No newline at end of file +![](https://pbs.twimg.com/media/F23_OOhXgAUBtlb?format=png&name=orig) diff --git a/archive/threads/NOTimothyLottes/2024-11-19-after-vk-device-is-open-i-signal-a-background/index.md b/archive/threads/NOTimothyLottes/2024-11-19-after-vk-device-is-open-i-signal-a-background/index.md index 12e94ec..669ff64 100644 --- a/archive/threads/NOTimothyLottes/2024-11-19-after-vk-device-is-open-i-signal-a-background/index.md +++ b/archive/threads/NOTimothyLottes/2024-11-19-after-vk-device-is-open-i-signal-a-background/index.md @@ -1,30 +1,58 @@ --- -title: "After VK device is open, I signal a background thread to load the SPIR-V module, while building the descriptor set layout, which then unblocks PSO compile on background threads." +title: "Working towards a new bring up of front-buffering on NV via VK." type: archive source: twitter -source_url: "https://x.com/NOTimothyLottes/status/1858715434436227544" +source_url: "https://x.com/NOTimothyLottes/status/1858709400791261630" author: "NOTimothyLottes" handle: NOTimothyLottes -post_id: "1858715434436227544" +post_id: "1858709400791261630" date: 2024-11-19 -archived: 2026-08-23 -draft: true +archived: 2026-08-24 +draft: false tags: - archive - twitter - NOTimothyLottes -description: "After VK device is open, I signal a background thread to load the SPIR-V module, while building the descriptor set layout, which then unblocks PSO compile on background threads." +description: "Working towards a new bring up of front-buffering on NV via VK." in_reply_to: "" --- ## Source -- URL: https://x.com/NOTimothyLottes/status/1858715434436227544 +- URL: https://x.com/NOTimothyLottes/status/1858709400791261630 - Author: NOTimothyLottes (@NOTimothyLottes) -- Posted: 2024-11-19 03:34:27 +- Posted: 2024-11-19 03:10:28 ## Thread **1/** +Working towards a new bring up of front-buffering on NV via VK. My last implementation had been tuned on AMD and didn't work on NV any more. NV does seem to accept a 1-deep swap with IMMEDIATE presentation at least on latest drivers, so that is a good start. + +**2/** + +Since my image resources are static after init time, only swap images change when the driver kills the swap chain (which once upon a time seemed to happen on ALT+TAB maybe). So it's one descriptor set always bound. + +**3/** + +Stopped using VK_DESCRIPTOR_SET_LAYOUT_CREATE_UPDATE_AFTER_BIND_POOL_BIT_EXT (god awful naming length people), because I think it could be a perf hit on NV due to extra indirection. NV driver is free to bake down the single set now before the command buffer is sent over. + +**4/** + +At this stage, to the point of having everything loaded and swap chain created, it's 2700 lines of engine code. That includes embedding headers (no external includes), so it's a one file compile. People claim rolling your own engine is hard? Not really if you keep it focused. + +**5/** + +So far it's a 0.25 sec hot load time on this NV dGPU laptop. Includes mapping a 4 MiB 'cart' file from pagecache, doing a 512 MiB buffer for GPU usage, hits on all PSOs, allocating some images, and kicking the command buffer that copies in the cart, and clears everything. + +**6/** + +Everything at load-time is multi-threaded to try to minimize start to in-game time. This is in sharp contrast to runtime where the only multi-threading being used is to separate things that are blocking. + +**7/** + +I get some utility on parallelizing {vulkan instance creation, mapping the cart file, warming the TLBs by walking all the pages, bringing up the window} it's about 0.08 seconds in at that point + +**8/** + After VK device is open, I signal a background thread to load the SPIR-V module, while building the descriptor set layout, which then unblocks PSO compile on background threads. And the rest of the VK setup runs in parallel. Working towards swap creation. diff --git a/archive/threads/NOTimothyLottes/2024-12-15-texel-buffer-is-needed-for-whatever-formats-one/index.md b/archive/threads/NOTimothyLottes/2024-12-15-texel-buffer-is-needed-for-whatever-formats-one/index.md index d89d8c4..bf5d3f5 100644 --- a/archive/threads/NOTimothyLottes/2024-12-15-texel-buffer-is-needed-for-whatever-formats-one/index.md +++ b/archive/threads/NOTimothyLottes/2024-12-15-texel-buffer-is-needed-for-whatever-formats-one/index.md @@ -1,30 +1,34 @@ --- -title: "TEXEL_BUFFER is needed for whatever formats one cannot load from SSBOs, and both of those require 'indexes', and whenever the IHVs actually correctly optimize the pointer extension, one needs the byte offsets instead." +title: "The real \"buffer zoo\" problem with Vulkan is needing to make your HW instruction emulation macros with overcomplete both {offset, and index} inputs so the emulation can choose the correct one based on whatever crap path is required {SSBO, TEXEL_BUFFER, future pointer}" type: archive source: twitter -source_url: "https://x.com/NOTimothyLottes/status/1868436190447432064" +source_url: "https://x.com/NOTimothyLottes/status/1868435545736782009" author: "NOTimothyLottes" handle: NOTimothyLottes -post_id: "1868436190447432064" +post_id: "1868435545736782009" date: 2024-12-15 -archived: 2026-08-23 -draft: true +archived: 2026-08-24 +draft: false tags: - archive - twitter - NOTimothyLottes -description: "TEXEL_BUFFER is needed for whatever formats one cannot load from SSBOs, and both of those require 'indexes', and whenever the IHVs actually correctly optimize the pointer extension, one needs the byte offsets instead." +description: "The real \"buffer zoo\" problem with Vulkan is needing to make your HW instruction emulation macros with overcomplete both {offset, and index} inputs so the emulation can choose the correct one based on whatever crap path is required {SSBO, TEXEL_BUFFER, future pointer}" in_reply_to: "" --- ## Source -- URL: https://x.com/NOTimothyLottes/status/1868436190447432064 +- URL: https://x.com/NOTimothyLottes/status/1868435545736782009 - Author: NOTimothyLottes (@NOTimothyLottes) -- Posted: 2024-12-15 23:21:16 +- Posted: 2024-12-15 23:18:42 ## Thread **1/** +The real "buffer zoo" problem with Vulkan is needing to make your HW instruction emulation macros with overcomplete both {offset, and index} inputs so the emulation can choose the correct one based on whatever crap path is required {SSBO, TEXEL_BUFFER, future pointer} + +**2/** + TEXEL_BUFFER is needed for whatever formats one cannot load from SSBOs, and both of those require 'indexes', and whenever the IHVs actually correctly optimize the pointer extension, one needs the byte offsets instead. So deadcode removal nightmare land wins today :( diff --git a/archive/threads/NOTimothyLottes/2026-07-22-a-language-that-you-author-in-a-compressed-space/index.md b/archive/threads/NOTimothyLottes/2026-07-22-a-language-that-you-author-in-a-compressed-space/index.md index 63f6e03..ba23b21 100644 --- a/archive/threads/NOTimothyLottes/2026-07-22-a-language-that-you-author-in-a-compressed-space/index.md +++ b/archive/threads/NOTimothyLottes/2026-07-22-a-language-that-you-author-in-a-compressed-space/index.md @@ -1,30 +1,57 @@ --- -title: "A language that you author in a compressed space directly." +title: "Working through rough draft of assembly for a different idea for a micro macro-forth." type: archive source: twitter -source_url: "https://x.com/NOTimothyLottes/status/2079746130309415185" +source_url: "https://x.com/NOTimothyLottes/status/2079742177370841105" author: "NOTimothyLottes" handle: NOTimothyLottes -post_id: "2079746130309415185" +post_id: "2079742177370841105" date: 2026-07-22 -archived: 2026-08-23 -draft: true +archived: 2026-08-24 +draft: false tags: - archive - twitter - NOTimothyLottes -description: "A language that you author in a compressed space directly." +description: "Working through rough draft of assembly for a different idea for a micro macro-forth." in_reply_to: "" --- ## Source -- URL: https://x.com/NOTimothyLottes/status/2079746130309415185 +- URL: https://x.com/NOTimothyLottes/status/2079742177370841105 - Author: NOTimothyLottes (@NOTimothyLottes) -- Posted: 2026-07-22 01:51:31 +- Posted: 2026-07-22 01:35:49 ## Thread **1/** +Working through rough draft of assembly for a different idea for a micro macro-forth. +1. 8-bit/word (color to relay meaning) +2. Roughly 62 useful dictionary entries/page +3. Supports paging (to extend dictionary) +4. Supports call/return - but compiler inlines everything +... + +![](https://pbs.twimg.com/media/HNy5A_uW0AAj1cU?format=png&name=orig) + +**2/** + +5. Compile step is branch-free +6. Compiles to branch-free x86-64 source +7. Which is then executed + +Noticed that I can pre-compile all the dictionary words into a lookup table. So actual compile just writes unaligned 8-bytes but only advances the write pointer by the actual size + +**3/** + +The interesting bits about this micro project was factoring out all branches, using branch-free interpreter loop to inline all calls so later execution is linear. Got messy doing a branch free software return stack. But costs are relatively low (40 ops/character) + +**4/** + +Intended usage would be for tiny-gramming programs in self contained chunks that include their own code generation (effectively a macro-forth used to generate machine code). + +**5/** + A language that you author in a compressed space directly. So Whitney-esk in using lots of single character variables (or double character with page characters) -BUT- very non-Whitney in that it has no higher order array constructs diff --git a/archive/threads/index.md b/archive/threads/index.md index 3f4eb2b..7523c17 100644 --- a/archive/threads/index.md +++ b/archive/threads/index.md @@ -17,5 +17,6 @@ Example of the note shape (single spine, no branches): [[archive/threads/How to - [[archive/threads/VPCOMPRESSB]] - [[archive/threads/VictorTaelin]] - [[archive/threads/kenpex]] +- [[archive/threads/mmalex]] - [[archive/threads/rianflo]] - [[archive/threads/winning_tactic]] diff --git a/archive/threads/mmalex/2025-06-28-i-vibe-coded-something-horrendous-and-i-love-it-a/index.md b/archive/threads/mmalex/2025-06-28-i-vibe-coded-something-horrendous-and-i-love-it-a/index.md new file mode 100644 index 0000000..0b6a375 --- /dev/null +++ b/archive/threads/mmalex/2025-06-28-i-vibe-coded-something-horrendous-and-i-love-it-a/index.md @@ -0,0 +1,42 @@ +--- +title: "i vibe coded something horrendous and I love it: a python c preprocessor that finds blocks like /* python */ in a .c file, runs the python, and dumps the stdout after the comment, replacing any old block if needed." +type: archive +source: twitter +source_url: "https://x.com/mmalex/status/1939051429060018590" +author: "π–’π–’π–†π–‘π–Šπ–" +handle: mmalex +post_id: "1939051429060018590" +date: 2025-06-28 +archived: 2026-08-24 +draft: false +tags: + - archive + - twitter + - mmalex +description: "i vibe coded something horrendous and I love it: a python c preprocessor that finds blocks like /* python */ in a .c file, runs the python, and dumps the stdout after the comment, replacing any old block if needed." +in_reply_to: "" +--- + +## Source + +- URL: https://x.com/mmalex/status/1939051429060018590 +- Author: π–’π–’π–†π–‘π–Šπ– (@mmalex) +- Posted: 2025-06-28 20:01:01 + +## Thread + +**1/** + +i vibe coded something horrendous and I love it: a python c preprocessor that finds blocks like /* python */ in a .c file, runs the python, and dumps the stdout after the comment, replacing any old block if needed. im using it to precompute tables... + +**2/** + +@mmalex There was a python tool for this remember as far back as 2000ish when working on a large c/c++ game codebase. Ah found it, it’s called Cog: https://cog.readthedocs.io/en/latest/ + +**3/** + +@seflless ah yes ! I knew it existed - it wasnt an original idea after all, but like a faint memory stirring in my brain - but couldn't find it. thanks for the link! + +**4/** + +@mmalex @seflless Many of my forth-like languages worked via nested runtime code gen, so it was possible to generate+run code to build either code or data, then repeat. The system would bootstrap its own assembler/etc that way. diff --git a/archive/threads/mmalex/index.md b/archive/threads/mmalex/index.md new file mode 100644 index 0000000..881e683 --- /dev/null +++ b/archive/threads/mmalex/index.md @@ -0,0 +1,12 @@ +--- +title: mmalex +type: note +draft: false +description: Archived threads by mmalex. +tags: + - archive + - twitter + - mmalex +--- + +- [[archive/threads/mmalex/2025-06-28-i-vibe-coded-something-horrendous-and-i-love-it-a]] diff --git a/scripts/twitter/README.md b/scripts/twitter/README.md index ef61956..1e409a6 100644 --- a/scripts/twitter/README.md +++ b/scripts/twitter/README.md @@ -7,6 +7,7 @@ From the vault root: ``` python C:\projects\Threadwell\scripts\twitter\tw.py graph --id python C:\projects\Threadwell\scripts\twitter\tw.py refresh --id --tip +python C:\projects\Threadwell\scripts\twitter\tw.py refresh --id --tip --branch python C:\projects\Threadwell\scripts\twitter\tw.py audit-media --id python C:\projects\Threadwell\scripts\twitter\tw.py publish --id python C:\projects\Threadwell\scripts\twitter\tw.py sync --handle @@ -19,7 +20,7 @@ python C:\projects\Threadwell\scripts\twitter\tw.py migrate-media --all --apply `paths.py` sets vault = two parents above this file, dumps = `../manual_slop/docs/twitter`, scratch = `../Threadwell-ai/scratch`. -`refresh` = refetch (cookies) + ingest + emit `--force` + media_merge. `--tip` treats `--id` as the tip and walks back to the OP regardless of handle. Each thread emits exactly one archive directory owned by the OP (the post with `reply_to_id == None`); cross-author responders do not get a directory. Does not flip `draft`. Does not commit. Frozen ids abort. +`refresh` = refetch (cookies) + ingest + emit `--force` + media_merge. `--tip` treats `--id` as the tip and walks back to the OP regardless of handle. Repeatable `--branch ` captures only those additional explicit tip paths and merges them into the same `thread_data.json`; it does not discover replies automatically. Captures run sequentially with gallery-dl retries disabled and five-second extractor/request pacing. Each thread emits exactly one archive directory owned by the OP (the post with `reply_to_id == None`); cross-author responders do not get a directory. Does not flip `draft`. Does not commit. Frozen ids abort. Usual publish path after you like the note: @@ -62,4 +63,4 @@ Frozen threads (ids listed in `do_not_refetch.txt`, matched against every captur | `do_not_refetch.txt` | frozen id list | | `tests/` | isolated, deterministic unit tests | -Secrets: `secrets/twitter_cookies.txt`, `secrets/credentials.toml` `[catbox] userhash` and `[backup] id`/`root`. The script reads them; never paste into chat, tests, or commits. \ No newline at end of file +Secrets: `secrets/twitter_cookies.txt`, `secrets/credentials.toml` `[catbox] userhash` and `[backup] id`/`root`. The script reads them; never paste into chat, tests, or commits. diff --git a/scripts/twitter/tests/test_tw_media_commands.py b/scripts/twitter/tests/test_tw_media_commands.py index 0e9cd19..da9c6c9 100644 --- a/scripts/twitter/tests/test_tw_media_commands.py +++ b/scripts/twitter/tests/test_tw_media_commands.py @@ -1,11 +1,87 @@ from __future__ import annotations +import json import unittest +from pathlib import Path +from tempfile import TemporaryDirectory -from twitter.tw import build_parser +from twitter.tw import ( + _capture_ids, + _gallery_base_args, + _merge_gallery_files, + _validate_capture_ids, + build_parser, +) class TwMediaCommandTests(unittest.TestCase): + def test_refetch_accepts_repeatable_branch(self) -> None: + args = build_parser().parse_args([ + "refetch", "--id", "100", + "--branch", "200", + "--branch", "300", + ]) + self.assertEqual(args.branch, ["200", "300"]) + + def test_refresh_accepts_repeatable_branch(self) -> None: + args = build_parser().parse_args([ + "refresh", "--id", "100", "--tip", + "--branch", "200", + ]) + self.assertEqual(args.branch, ["200"]) + + def test_capture_ids_deduplicates_in_first_seen_order(self) -> None: + self.assertEqual( + _capture_ids("100", ["200", "100", "300", "200"]), + ("100", "200", "300"), + ) + + def test_gallery_base_args_disable_retries_and_add_pacing(self) -> None: + args = _gallery_base_args() + self.assertEqual(args[args.index("--retries") + 1], "0") + self.assertEqual(args[args.index("--sleep-extractor") + 1], "5") + self.assertEqual(args[args.index("--sleep-request") + 1], "5") + + def test_merge_gallery_files_concatenates_arrays(self) -> None: + with TemporaryDirectory() as tmp: + root = Path(tmp) + first = root / "first.json" + second = root / "second.json" + output = root / "gallery.json" + first.write_text('[{"id": "100"}]', encoding="utf-8") + second.write_text('[{"id": "200"}]', encoding="utf-8") + _merge_gallery_files([first, second], output) + self.assertEqual( + json.loads(output.read_text(encoding="utf-8")), + [{"id": "100"}, {"id": "200"}], + ) + + def test_merge_gallery_files_rejects_non_array(self) -> None: + with TemporaryDirectory() as tmp: + root = Path(tmp) + capture = root / "capture.json" + capture.write_text('{"id": "100"}', encoding="utf-8") + with self.assertRaisesRegex(SystemExit, "JSON array"): + _merge_gallery_files([capture], root / "gallery.json") + + def test_validate_capture_ids_rejects_missing_requested_id(self) -> None: + with TemporaryDirectory() as tmp: + path = Path(tmp) / "thread_data.json" + path.write_text( + json.dumps({ + "root_post_id": "100", + "source_url": "https://x.com/i/status/100", + "posts": [ + {"post_id": "100", "metrics": {}}, + {"post_id": "200", "metrics": {}}, + ], + }), + encoding="utf-8", + ) + _validate_capture_ids(path, ("100", "200")) + with self.assertRaisesRegex(SystemExit, "300"): + _validate_capture_ids(path, ("100", "300")) + def test_migrate_defaults_to_dry_run(self) -> None: args = build_parser().parse_args(["migrate-media", "--id", "100"]) self.assertEqual(args.cmd, "migrate-media") @@ -29,4 +105,4 @@ class TwMediaCommandTests(unittest.TestCase): if __name__ == "__main__": - unittest.main() \ No newline at end of file + unittest.main() diff --git a/scripts/twitter/tw.py b/scripts/twitter/tw.py index 1b208b8..85e5444 100644 --- a/scripts/twitter/tw.py +++ b/scripts/twitter/tw.py @@ -27,6 +27,7 @@ if str(_HERE) not in sys.path: sys.path.insert(0, str(_HERE)) from paths import COOKIES, DUMPS, FROZEN, HERE, SCRATCH, VAULT +from models import load_thread try: from frozen import frozen_match, load_frozen_ids, require_writable @@ -178,46 +179,79 @@ def cmd_graph(args: argparse.Namespace) -> int: return 0 +def _capture_ids(post_id: str, branch_ids: list[str]) -> tuple[str, ...]: + """Return the primary id plus distinct explicit branch tips.""" + return tuple(dict.fromkeys((post_id, *branch_ids))) + + +def _gallery_base_args() -> list[str]: + """Return the shared safe, paced gallery-dl arguments.""" + return [ + "gallery-dl", + "--cookies", + str(COOKIES), + "--retries", + "0", + "--sleep-extractor", + "5", + "--sleep-request", + "5", + "-o", + "text-tweets=true", + "-o", + "conversations=true", + ] + + +def _merge_gallery_files(paths: list[Path], output: Path) -> None: + """Concatenate gallery-dl JSON arrays for local ingest.""" + merged: list[object] = [] + for path in paths: + data: object = json.loads(path.read_text(encoding="utf-8")) + if not isinstance(data, list): + raise SystemExit(f"gallery-dl output is not a JSON array: {path}") + merged.extend(data) + output.write_text( + json.dumps(merged, ensure_ascii=False) + "\n", + encoding="utf-8", + newline="\n", + ) + + +def _validate_capture_ids(path: Path, capture_ids: tuple[str, ...]) -> None: + """Fail if gallery-dl omitted an explicitly requested tip.""" + thread = load_thread(path) + present = {post.post_id for post in thread.posts} + missing = [post_id for post_id in capture_ids if post_id not in present] + if missing: + raise SystemExit(f"requested tips missing from capture: {', '.join(missing)}") + + def cmd_refetch(args: argparse.Namespace) -> int: - """gallery-dl a thread into the scratch dir, then run ``ingest_gallery.py`` to materialize thread_data.json.""" + """Capture one spine tip plus explicit branch tips into one dump.""" post_id = args.id - check_frozen(post_id) + capture_ids = _capture_ids(post_id, args.branch) + for capture_id in capture_ids: + check_frozen(capture_id) if not COOKIES.is_file(): raise SystemExit(f"missing {COOKIES}") + out = scratch_dir(post_id) - media = out / "media" - media.mkdir(parents=True, exist_ok=True) - url = f"https://x.com/i/status/{post_id}" + out.mkdir(parents=True, exist_ok=True) + capture_paths: list[Path] = [] + for capture_id in capture_ids: + capture = out / f"gallery_{capture_id}.json" + url = f"https://x.com/i/status/{capture_id}" + with capture.open("w", encoding="utf-8") as fh: + subprocess.check_call( + [*_gallery_base_args(), "--dump-json", url], + stdout=fh, + ) + capture_paths.append(capture) + gallery = out / "gallery.json" - with gallery.open("w", encoding="utf-8") as fh: - subprocess.check_call( - [ - "gallery-dl", - "--cookies", - str(COOKIES), - "--dump-json", - "-o", - "text-tweets=true", - "-o", - "conversations=true", - url, - ], - stdout=fh, - ) - run( - [ - "gallery-dl", - "--cookies", - str(COOKIES), - "-D", - str(media), - "-o", - "text-tweets=true", - "-o", - "conversations=true", - url, - ] - ) + _merge_gallery_files(capture_paths, gallery) + source_url = f"https://x.com/i/status/{post_id}" run( [ sys.executable, @@ -227,11 +261,19 @@ def cmd_refetch(args: argparse.Namespace) -> int: "--out", str(out / "thread_data.json"), "--source-url", - url, + source_url, "--root", post_id, ] ) + _validate_capture_ids(out / "thread_data.json", capture_ids) + + media = out / "media" + media.mkdir(parents=True, exist_ok=True) + for capture_id in capture_ids: + url = f"https://x.com/i/status/{capture_id}" + run([*_gallery_base_args(), "-D", str(media), url]) + print(f"refetch -> {out}") return 0 @@ -562,6 +604,14 @@ def build_parser() -> argparse.ArgumentParser: ): p = sub.add_parser(name) p.add_argument("--id", required=True) + if name in ("refetch", "refresh"): + p.add_argument( + "--branch", + action="append", + default=[], + metavar="TIP_ID", + help="capture and merge this explicit branch tip; repeatable", + ) if name in ("emit", "refresh"): p.add_argument("--tip", action="store_true", help="use --id as tip") p.add_argument("--slug", default=None)