80 lines
8.0 KiB
Markdown
80 lines
8.0 KiB
Markdown
# twitter
|
|
|
|
Front door: `tw.py`. Individual scripts still work.
|
|
|
|
From the vault root:
|
|
|
|
```
|
|
python C:\projects\Threadwell\scripts\twitter\tw.py graph --id <snowflake>
|
|
python C:\projects\Threadwell\scripts\twitter\tw.py refresh --id <snowflake> --tip
|
|
python C:\projects\Threadwell\scripts\twitter\tw.py refresh --id <spine-tip> --tip --branch <branch-tip>
|
|
python C:\projects\Threadwell\scripts\twitter\tw.py add-branch --id <original-spine-tip> --from <reply-node>
|
|
python C:\projects\Threadwell\scripts\twitter\tw.py emit --id <spine-tip> --tip --preserve-existing
|
|
python C:\projects\Threadwell\scripts\twitter\tw.py audit-media --id <snowflake>
|
|
python C:\projects\Threadwell\scripts\twitter\tw.py publish --id <snowflake>
|
|
python C:\projects\Threadwell\scripts\twitter\tw.py sync --handle <handle>
|
|
python C:\projects\Threadwell\scripts\twitter\tw.py backup --id <snowflake>
|
|
python C:\projects\Threadwell\scripts\twitter\tw.py fallback --id <snowflake> --media-id <media-id> --role <role> --confirm-origin-unavailable
|
|
python C:\projects\Threadwell\scripts\twitter\tw.py restore-origin --id <snowflake> --media-id <media-id>
|
|
python C:\projects\Threadwell\scripts\twitter\tw.py migrate-media --id <snowflake>
|
|
python C:\projects\Threadwell\scripts\twitter\tw.py migrate-media --all --apply
|
|
python C:\projects\Threadwell\scripts\twitter\tw.py reslug --all
|
|
python C:\projects\Threadwell\scripts\twitter\tw.py reslug --all --apply
|
|
```
|
|
|
|
`paths.py` sets vault = two parents above this file, dumps = `../manual_slop/docs/twitter`, scratch = `../Threadwell-ai/scratch`.
|
|
|
|
`refresh` = refetch (cookies) + ingest + emit `--force` + media_merge. `--tip` treats `--id` as the tip and walks back to the OP regardless of handle. Repeatable `--branch <tip-id>` captures only those additional explicit tip paths and merges them into the same `thread_data.json`; it does not discover replies automatically. Captures run sequentially with gallery-dl retries disabled and five-second extractor/request pacing. Each thread emits exactly one archive directory owned by the OP (the post with `reply_to_id == None`); cross-author responders do not get a directory. New notes default to `draft: false` (publish by default). Does not commit. Frozen ids abort.
|
|
|
|
`emit --preserve-existing` is the safe completion step after inspecting a fresh scratch capture. Fresh records win for ids the provider returned; posts present only in the existing emitted `thread_data.json` are retained after the command verifies both captures share a conversation root. This prevents an incomplete gallery-dl refresh from deleting archived replies. The merged scratch dataset keeps the requested `--id` as its stored spine tip before normal emission and media-manifest reconciliation.
|
|
|
|
`add-branch` incrementally extends an already emitted thread. `--id` must be the original stored spine tip; repeatable `--from <reply-node>` values are each captured once. The command locally retains each node's missing attachment path and entire visible descendant subtree, merges only previously absent posts into the existing `thread_data.json`, downloads media for the new posts, and re-emits with the original spine unchanged. It preserves existing origin/local/fallback/derived/OCR media records and adds provider-origin plus local-copy records for new branch media. A visible leaf is only the deepest reply returned by that capture; the command does not query descendants individually or claim provider-wide completeness. Gallery-dl may return only the ancestor chain when replies are behind nested show-more cursors; then the command adds nothing and known reply/stream-tip ids must be supplied explicitly. Fallback upload remains a separate explicit action after origin unavailability is confirmed.
|
|
|
|
New emits publish by default. `tw.py publish --id <snowflake>` is only used to flip a thread that was captured with `draft: true` (rare after this change). The usual review path is:
|
|
|
|
1. `tw.py graph --id …` — confirm tip vs root.
|
|
2. `tw.py refresh --id … --tip` if the dump stored a tip as root.
|
|
3. `tw.py audit-media --id …` — confirm origin URLs are selected.
|
|
|
|
Site wikilinks must be `[[archive/threads/<handle>/<date-slug>]]`. Short `[[slug]]` 404s.
|
|
|
|
## Media policy
|
|
|
|
Published Twitter/X media cites the original provider's HTTPS URL by default. The vault also retains a local, unpublished copy under `assets/threads/<handle>/<date-slug>/` using the same relative taxonomy as the archive. A separately requested backup may mirror those assets to configured storage. Alternative-host uploads are a manual fallback used only after the origin is confirmed unavailable; the original URL and every fallback URL remain recorded in `media.json`.
|
|
|
|
The `lift` command is retired. Use `fallback` for a single media item after explicit confirmation, or `restore-origin` to select the immutable provider URL again. The Catbox userhash and backup destination id/root are read from `secrets/credentials.toml` by the script. Never paste them into chat.
|
|
|
|
Frozen threads (ids listed in `do_not_refetch.txt`, matched against every captured post id) are read-only across refetch, refresh, emit, manifest migration, note rewrite, fallback, publication selection, and tracked backup operations. Read-only `audit-media` may still inspect them.
|
|
|
|
## Canonical thread directories
|
|
|
|
Each thread archive directory is named from the rendered title that appears in `index.md` (`<YYYY-MM-DD>-<slugified rendered title>`). `tw.py reslug --all` reads each existing `index.md` frontmatter date and title, derives the expected directory, and reports every mapping without writing. `tw.py reslug --all --apply` preflights every pair, refuses the entire apply on any non-frozen conflict, and moves assets first with the archive directory immediately after; if the archive move fails, assets roll back. If a destination is occupied, the apply aborts without `-2`, `-3`, or any other suffix. Frozen ids in `do_not_refetch.txt` are reported as `frozen` and skipped.
|
|
|
|
Future emits use the same rendered title for the directory and `index.md` frontmatter, so normal emit does this automatically — no flag is needed on emit, refresh, or add-branch. Hand-move only the archive or only the asset is not supported; both must move as a pair. Each old archive prefix is replaced exactly once in mutable `*.md` and `*.canvas` outside `.git/`, `site/`, `assets/`, `secrets/`, `node_modules/`, and frozen archive dirs, so mappings do not cascade. No aliases, redirect pages, compatibility folders, suffix fallback, or network contact.
|
|
|
|
## Pieces
|
|
|
|
| Script | Job |
|
|
|---|---|
|
|
| `tw.py` | locate / graph / refetch / add-branch / emit / refresh / ocr / merge / sync / publish / migrate-media / backup / fallback / restore-origin |
|
|
| `graph_dry_run.py` | tip graph, no vault writes |
|
|
| `ingest_gallery.py` | gallery-dl JSON → `thread_data.json` |
|
|
| `emit_archive.py` | JSON → notes + assets. Uses canonical media locations. `--force` preserves derived/fallback rows |
|
|
| `media_migrate.py` | legacy `media.json` → canonical locations, dry-run or apply |
|
|
| `media_audit.py` | local integrity, reference counts, mirror freshness, optional origin check |
|
|
| `media_manifest.py` | schema, validation, merge, selection, atomic I/O |
|
|
| `media_refs.py` | media-aware markup and note rewrite planning |
|
|
| `backup_assets.py` | sparse, hash-verified copy of one thread asset dir |
|
|
| `catbox_client.py` | one sanitized Catbox upload; no batching or retry |
|
|
| `fallback_media.py` | confirm-origin fallback activation and restore-origin |
|
|
| `frozen.py` | resolve frozen ids against every captured post id |
|
|
| `lift_catbox.py` | retired entry; points operators to `tw.py fallback` |
|
|
| `ocr_pass.py` | Umi HTTP `:1224` / tesseract / Windows OCR |
|
|
| `media_embed.py` | `--attach-ocr` only; refuses `--show crt` |
|
|
| `media_merge.py` | restore derived rows under canonical API |
|
|
| `crt_pass.py` | stock slang only; ShaderGlass is the look reference |
|
|
| `do_not_refetch.txt` | frozen id list |
|
|
| `tests/` | isolated, deterministic unit tests |
|
|
|
|
Secrets: `secrets/twitter_cookies.txt`, `secrets/credentials.toml` `[catbox] userhash` and `[backup] id`/`root`. The script reads them; never paste into chat, tests, or commits.
|