From 3af011196c463971b396a2cc05a50a4c6632668a Mon Sep 17 00:00:00 2001 From: Ed_ Date: Tue, 23 Jun 2026 16:06:44 -0400 Subject: [PATCH] conductor(deob_pilot): Initialize Phase 2 (pilot) track with full scaffold MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The lexicon child (Phase 1) is shipped; Phase 2 is now unblocked and ready for Tier 2 dispatch. 5 new files in video_analysis_deob_pilot_20260621/: - spec.md: updated to reference the new files (lightweight scaffold) - plan.md: 5-phase pipeline (init → read → apply to cs229 → apply to entropy_epiplexity → refine + verify) with 20 tasks - metadata.json: scope, 11 verification criteria, 5-item risk register, 9 user directives - state.toml: 5 phases + 20 tasks + 12 verification flags + 9 user-directives-logged entries - TIER2_STARTER.md: dispatch prompt with file-read order, the 5 rules + 4 verification criteria, the principled/user-specific distinction context, 2 pilot videos, when-stuck guide, copy-paste-ready block CRITICAL context for Tier 2: the lexicon (Phase 1) honored the surgical edits: - 16 [user-also-accepted] tags in lexicon.md - 4 [principled] + 4 [user-preferred] tags in dedup_map.md - §3.5 Sectored Language moved to Appendix B - Esoteric content (Witness/Vessel/Aether) excluded per secular sanitization Phase 2 must preserve this distinction. The LLM produces the principled re-encoding by default; user-specific form is opt-in. Esoteric content stays in cluster_0_twitter.md only. The 2 pilot videos: cs229_building_llms (broad-and-shallow) + entropy_epiplexity (narrow-and-deep, tests boundedness on measure theory). --- .../TIER2_STARTER.md | 339 ++++++++++++++++++ .../metadata.json | 130 +++++++ .../plan.md | 61 ++++ .../spec.md | 9 +- .../state.toml | 82 +++++ 5 files changed, 619 insertions(+), 2 deletions(-) create mode 100644 conductor/tracks/video_analysis_deob_pilot_20260621/TIER2_STARTER.md create mode 100644 conductor/tracks/video_analysis_deob_pilot_20260621/metadata.json create mode 100644 conductor/tracks/video_analysis_deob_pilot_20260621/plan.md create mode 100644 conductor/tracks/video_analysis_deob_pilot_20260621/state.toml diff --git a/conductor/tracks/video_analysis_deob_pilot_20260621/TIER2_STARTER.md b/conductor/tracks/video_analysis_deob_pilot_20260621/TIER2_STARTER.md new file mode 100644 index 00000000..96bf3196 --- /dev/null +++ b/conductor/tracks/video_analysis_deob_pilot_20260621/TIER2_STARTER.md @@ -0,0 +1,339 @@ +# Tier 2 Starter Prompt: Video Analysis De-obfuscation — Pilot on 2 Videos + +**Purpose.** This file is the dispatch prompt for Tier 2 autonomous agents picking up the Phase 2 (pilot) child track. It supplements the auto-loaded `spec.md` + `plan.md` per `conductor/tier2/commands/tier-2-auto-execute.md` step 2. + +**Track:** `video_analysis_deob_pilot_20260621` (Pass 2 of 3, Phase 2 of 3 within Pass 2) + +--- + +## Track identity + +- **ID:** `video_analysis_deob_pilot_20260621` +- **Type:** Research-only child track (Phase 2 of Pass 2; **the first actual application of the de-obfuscation scheme**) +- **Status:** spec ✓ (lightweight scaffold); plan/metadata/state to be created by Tier 2 +- **Priority:** A (user-blocking; Pass 2 of the 3-pass research campaign) +- **Domain:** Meta-tooling (research deliverable; no `src/` changes) + +## Mission (what this track produces) + +This Phase 2 child **validates the refined lexicon** (Phase 1) by applying it to 2 Pass 1 reports. The pilot produces **7 deliverables**: + +For each of the 2 pilot videos, 3 files in `artifacts//`: +1. **`_translation.md`** — side-by-side table: original expression ↔ re-encoded form, with form anchor per row +2. **`_deobfuscated.md`** — the re-encoded report (replacement; same 8-section structure as Pass 1, but every standard-math expression is replaced with the principled re-encoding) +3. **`_decoder.md`** — per-term decoder: form anchor, etymology, definition history, link to original section + +Plus: +- **`pilot_report.md`** — lexicon refinements discovered, concepts that didn't fit (gaps), process improvements for the apply phase + +The pilot's output is the first empirical evidence that the de-obfuscation scheme works on real Pass 1 material. The `pilot_report.md` informs Phase 3 (apply) by surfacing lexicon gaps + process improvements. + +--- + +## The 2 pilot videos (why these 2) + +1. **`cs229_building_llms`** — Foundational ML/LLM coverage. Wide scope, tests "form projection" across many concepts (sets, functions, optimization, learning). Good test for the lexicon's general applicability. +2. **`entropy_epiplexity`** — Math-heavy, focused on information-theoretic concepts (entropy, mutual information, epiplexity). Good test for "boundedness" + type-theoretic encoding of measure theory (a common bottleneck in de-obfuscation efforts). + +These 2 videos test the lexicon on different shapes: one broad-and-shallow, one narrow-and-deep. The pilot's refinements will inform the apply phase. + +--- + +## Files to read in this order + +### 1. This Phase 2 child track (REQUIRED) + +- `./spec.md` (lightweight scaffold; read first) +- `./plan.md` (5-phase pipeline; 20 tasks) + +### 2. The refined lexicon (PRIMARY INPUT — the operational spec) + +- `/conductor/tracks/video_analysis_deob_lexicon_20260621/lexicon.md` (the codified operational spec; 12 sections + 4 appendices) +- `/conductor/tracks/video_analysis_deob_lexicon_20260621/terms_catalog.md` (machine-readable lexicon; 4-tier table with 9 columns per term) +- `/conductor/tracks/video_analysis_deob_lexicon_20260621/dedup_map.md` (6 noise-dedup maps; 3 principled + 3 user-preferred) + +### 3. The warmup's prompt template (the LLM-direct operational spec) + +- `/conductor/tracks/video_analysis_deob_warmup_20260621/prompt_template.md` (5 rules + 6 noise-dedup maps + 4-layer format + 7 example transformations + verification checklist) + +### 4. The 2 Pass 1 reports (the INPUT to de-obfuscate) + +- `/conductor/tracks/video_analysis_cs229_building_llms_20260621/report.md` (foundational ML/LLM coverage; ~1000-10000 LOC) +- `/conductor/tracks/video_analysis_entropy_epiplexity_20260621/report.md` (math-heavy, information-theoretic; ~1000-10000 LOC) + +### 5. Project conventions (REQUIRED at session start per workflow.md) + +- `/AGENTS.md` (critical anti-patterns, file naming, no day estimates) +- `/conductor/workflow.md` (task workflow, Tier 2 sandbox conventions, failcount contract) +- `/conductor/code_styleguides/python.md` (1-space indent, type hints, no comments — IF code is written; not applicable here) +- `/conductor/code_styleguides/error_handling.md` (Result[T] pattern — not applicable) + +### 6. Reference tracks (consult as needed) + +- `/conductor/tracks/video_analysis_deob_warmup_20260621/TIER2_STARTER.md` (sibling track; the warmup's dispatch prompt; useful precedent) +- `/conductor/tracks/video_analysis_deob_lexicon_20260621/TIER2_STARTER.md` (sibling track; the lexicon child's dispatch prompt; useful precedent) + +--- + +## The 5 rules + the 4 verification criteria (load-bearing) + +### The 5 rules (per `lexicon.md` §1 — every re-encoding must satisfy all 5) + +1. **Boundedness** — every value is a finite form. `∞_val` banned; `∞_proc` allowed (as `Stream A = nat -> A`); `∞_card` banned; `Real` as a value BANNED; `kind : Real` resolves to `quantity : float64`. +2. **Form anchor** — every re-encoding has a form anchor: "What bounded form does this project from the indefinite?" +3. **Etymology** — every new term has a 1-line origin + 1-line definition history. +4. **Lossless** — every Pass 1 concept is represented. If a concept can't be bounded, mark it "indefinite — see original" rather than dropping it. **Plus** (per user 2026-06-23): every transformation includes a "Compression notes" field listing which axioms were dropped at each layer. +5. **Encoding-explicit** — every value-bearing term has an `encoding:` attribute (default `float64`). + +### The 4 verification criteria (per `lexicon.md` §0.1 + `prompt_template.md` verification checklist) + +For each of the 6 deliverables (2 videos × 3 layers), verify: + +- [ ] **Lossless** — no Pass 1 concept dropped; compression history preserved per layer. +- [ ] **Bounded** — no `∞_val` or `∞_card`; the "real number line" as a value is banned. +- [ ] **Encoding-explicit** — every value-bearing term has an `encoding:` attribute. Default: `float64`. +- [ ] **Constructively typed** — every expression has a type. +- [ ] **Etymology-cited** — every new term has the 1-line origin + 1-line definition history. +- [ ] **Form-anchored** — every re-encoding has a form anchor. +- [ ] **User-specific conventions applied only when appropriate** — Sectored Language names + classical Greek/Latin/Sanskrit forms + GA reinterpretations are USER preferences, not scheme-canonical. Apply them only when the reader would prefer them. + +--- + +## The 3-layer deliverable structure (per `prompt_template.md` §"The 3-Layer Output Format") + +For each video, the LLM produces 3 files in `artifacts//`: + +### Layer 1: `_translation.md` (side-by-side table) + +| # | Original Section | Original Expression | Re-encoded Form | Form Anchor | Etymology | +|---|------------------|--------------------|-----------------|-------------|-----------| + +### Layer 2: `_deobfuscated.md` (the re-encoded report) + +Same 8-section structure as Pass 1's report: +1. TL;DR +2. Key Concepts +3. Frame Analysis +4. Transcript Highlights +5. Mathematical / Theoretical Content (re-encoded) +6. Connections to Other Videos in Campaign +7. Open Questions / Follow-up +8. References + +But every standard-math expression in §5 (and anywhere else) is replaced with the principled re-encoding per the lexicon. + +### Layer 3: `_decoder.md` (per-term decoder) + +```markdown +## Term: + +- **Original notation:** ... +- **Re-encoded:** ... +- **Form anchor:** the bounded form is X; the projection is Y +- **Etymology (1-line):** +- **Definition history (1-line):** +- **Source sections in original:** §X.Y +- **Cluster cross-ref:** research/cluster_*.md §X.Y +``` + +For user-also-accepted terms, additionally: +- **User-specific form:** the Sectored Language V1 name, the GA reinterpretation, or the classical Greek/Latin/Sanskrit form +- **4-language etymology:** Greek + Latin + English + Sanskrit + +### The "indefinite — see original" pattern (per Rule 1) + +If a concept can't be bounded: +```markdown +## Term: + +- **Status:** INDEFINITE — see original +- **Reason:** +- **Source sections in original:** §X.Y +- **Cluster cross-ref:** research/cluster_*.md §X.Y +``` + +The user values honest uncertainty over confident guesses. + +--- + +## The principled vs user-specific distinction (the surgical-edits context) + +Per the 2026-06-23 surgical edits + Phase 1's formalization: + +- **The LLM produces the principled re-encoding by default** (the scheme-canonical form, from the 5 rules + the 3 principled noise-dedup maps). +- **The user-specific form is opt-in** (Sectored Language V1, GA reinterpretations, classical Greek/Latin/Sanskrit forms). The LLM produces the user-specific form only when the reader would prefer it. +- **The 4-language pattern is REQUIRED for user-also-accepted terms** (Greek + Latin + English + Sanskrit, per the warmup's `Notiones.txt` reference). +- **The 3 user-preferred noise-dedup maps are optional output conventions** (the GA collapse, the "invent → construct" rename, the "number = quantity = value" collapse). +- **Esoteric content (Witness/Vessel/Aether ontology) is NOT in the public lexicon.** It stays in `cluster_0_twitter.md` for the user's reference. **Do NOT include it in the pilot deliverables.** + +--- + +## The 31 unresolved items (per `lexicon.md` §11.3) + +The warmup's 31 unresolved items (12 from §A.3 + 19 from §11.3) are the lexicon's open questions. Some of these may surface during the pilot. If they do: +- If the answer is clear from the Pass 1 report's content: define the term in the pilot's `pilot_report.md` (a "lexicon v2 addition" entry). +- If unclear: add to the pilot's "gaps" section for the apply phase. + +--- + +## The compression history (per Rule 4 + user 2026-06-23) + +Every transformation in the 3-layer output should include a "**Compression notes**" field that lists which axioms were dropped at each layer. For example: +- Layer 1 (compressed original): "Uses associativity and commutativity to combine terms" +- Layer 2 (fully expanded): "Expands the compressed form; compression note: applied associativity and commutativity; to audit the lossless history, see the original §3.2" +- Layer 3 (executable code): "Implements the expanded form; compression note: same as Layer 2" + +--- + +## Critical user directives (load-bearing) + +The warmup captured these in `state.toml` `[user_directives_logged]`; they apply to Phase 2 unchanged. + +1. **Constructive type theory as foundation.** +2. **Boundedness for direct knowledge.** +3. **Cycles/iteration allowed but expressed explicitly** (`∞_val` banned, `∞_proc` allowed). +4. **Etymology-aware lexicon** (1-line origin + 1-line history; 4-language for user-also-accepted terms). +5. **Lossless preservation with explicit compression history** (per user 2026-06-23). +6. **Encoding-explicit** (every value-bearing term has an `encoding:` attribute; per user 2026-06-23). +7. **Honest epistemic hedging** (per the "Don't know what `<<` here is" pattern). +8. **The principled vs user-specific distinction** (the LLM produces the principled form; user-specific is opt-in). +9. **Secular sanitization** (esoteric content NOT in the pilot deliverables). + +--- + +## Key risks + +- **R1 (medium):** LLM may produce user-specific forms as if they were principled. **Mitigation:** the prompt_template.md's verification checklist explicitly checks for this; the LLM is told to apply user-specific forms only when the reader would prefer them. +- **R2 (medium):** Lexicon can't capture a concept in bounded form. **Mitigation:** Rule 1 explicitly allows "indefinite — see original"; the pilot_report.md captures these gaps for the apply phase. +- **R3 (low):** Decoder has too many terms (verbose). **Mitigation:** decoder format is per-term; the LLM groups by section and skips terms that don't appear in the de-obfuscated.md. +- **R4 (medium):** LLM output is too brief (skips concepts). **Mitigation:** verification checklist includes the Lossless criterion; the 4 verbatim checks per video; pilot_report.md captures skips as gaps. +- **R5 (medium):** Entropy_epiplexity's measure theory defeats the boundedness rule. **Mitigation:** this is WHY the pilot tests this video; if the lexicon can't capture it, capture the gap in pilot_report.md; the apply phase uses the refined lexicon (or a v2). + +--- + +## Hard constraints + +- **No `src/*.py` changes.** This is a research-only track; no production code. +- **No `pyproject.toml` dependencies.** All work is research (markdown files). +- **No `uv pip install` for new packages.** The pilot is markdown only. +- **No day estimates in any artifact.** Scope measured in files/sites per `conductor/workflow.md`. +- **Per-task atomic commits.** Each deliverable (6 per-video + 1 pilot_report) is committed in its own commit with a git note. +- **No comments in code** (if any code is written; not applicable here). +- **The LLM invocation is the heart of this track.** The LLM is invoked with the prompt_template.md + the Pass 1 report. The output is the 3-layer deliverable. The Tier 2 worker delegates the LLM invocation to a Tier 3 worker (per `conductor/workflow.md` "Standard Task Workflow" step 4). +- **Esoteric content NOT in deliverables.** Witness/Vessel/Aether ontology stays in `cluster_0_twitter.md` only. + +--- + +## Tier 2 sandbox conventions (per `conductor/tier2/agents/tier2-autonomous.md`) + +- **Test runner:** `uv run python scripts/run_tests_batched.py` (NEVER `uv run pytest` directly). Not applicable for this research-only track — no tests to run. +- **Default branch:** `master`. Use `origin/master` for `git fetch` and as the base for new branches. +- **Line endings:** preserve existing (CRLF stays CRLF, LF stays LF). +- **Throw-away scripts:** `scripts/tier2/artifacts//` (not the base `scripts/tier2/` directory). Not applicable here. +- **End-of-track report:** `docs/reports/TRACK_COMPLETION_.md` per `conductor/tier2/agents/tier2-autonomous.md` step 42. +- **State update:** `state.toml` → `status = "completed"` at the end. +- **Hard bans:** `git push*`, `git checkout*`, `git restore*`, `git reset*` (3-layer enforced). +- **File access:** Tier 2 clone only. **NEVER USE APPDATA** — denied at the bash level. +- **Failcount contract:** After every task commit, check `should_give_up` from `scripts.tier2.failcount`. Thresholds: 3 consecutive red, 3 consecutive green, 30 min no progress. + +--- + +## Verification criteria (gate for Phase 2 completion) + +- [ ] Both pilot videos have 3-layer deliverables (6 files total in `artifacts/`) +- [ ] All 3 deliverables per video pass the 4 criteria (Lossless, Bounded, Constructively typed, Etymology-cited) +- [ ] All 3 deliverables per video pass the additional 3 criteria (Encoding-explicit, Form-anchored, User-specific conventions applied only when appropriate) +- [ ] `pilot_report.md` captures lexicon refinements + gaps + process improvements +- [ ] `pilot_report.md` includes updates to the 31 unresolved items from the warmup +- [ ] User has reviewed and approved the 2 deliverables + pilot_report.md +- [ ] All 7 deliverables committed atomically (6 per-video + 1 pilot_report) +- [ ] Git notes attached to each commit +- [ ] `state.toml` updated to `status = "completed"` +- [ ] `docs/reports/TRACK_COMPLETION_video_analysis_deob_pilot_20260621.md` exists + +--- + +## Execution plan (per pilot plan.md) + +| Phase | Task | Notes | +|---|---|---| +| 0 | Initialize | Create `state.toml` per Tier 2 conventions | +| 1 | Read the refined lexicon | 6 file reads: lexicon.md + terms_catalog.md + dedup_map.md + prompt_template.md + 2 Pass 1 reports | +| 2 | Apply to `cs229_building_llms` | Delegate to Tier 3 worker: LLM invocation with prompt_template + cs229 report. Produces 3 files in `artifacts/cs229_building_llms/`. Commit per file. | +| 3 | Apply to `entropy_epiplexity` | Same as Phase 2. Produces 3 files in `artifacts/entropy_epiplexity/`. | +| 4 | Refine + pilot report | Write `pilot_report.md` capturing refinements + gaps + process improvements + 31-item updates. | +| 5 | User review + verification | Pause for user feedback. Idempotency check. Audit. End-of-track report. | + +--- + +## When stuck + +- **The LLM output violates the principled/user-specific distinction.** Re-prompt the LLM with an explicit instruction: "Produce the principled re-encoding; the user-specific form is opt-in." Per the warmup's surgical edits, the principled form is the scheme's canonical output. +- **The LLM can't bound a concept.** Apply Rule 1: mark "indefinite — see original" in the decoder. Capture the gap in `pilot_report.md` for the apply phase. +- **The LLM output is too verbose.** The decoder format is per-term; the LLM should group by section and skip terms that don't appear in the de-obfuscated.md. Re-prompt with a "be concise" instruction if needed. +- **The LLM output is too brief (skips concepts).** The Lossless criterion is violated. Re-prompt with "every Pass 1 concept must be represented; use 'indefinite — see original' for concepts that can't be bounded." Check the verbatim 4 criteria per video. +- **The `pilot_report.md` is too short to capture the refinements.** Expand the format: include the original Pass 1 concept, the re-encoded form, the gap (if any), the proposed fix (for the apply phase or a lexicon v2). +- **The lexicon can't capture `entropy_epiplexity`'s measure theory.** This is a known risk (R5); the gap is captured in `pilot_report.md` for the apply phase. The apply phase uses the refined lexicon (or a v2 if the pilot surfaces enough gaps). + +--- + +## Quick reference: dispatch + +``` +/tier-2-auto-execute video_analysis_deob_pilot_20260621 +``` + +Plus this context (paste BEFORE invoking): + +``` +TRACK: video_analysis_deob_pilot_20260621 +TYPE: Research-only child track (Pass 2 Phase 2 of 3) +STATUS: spec ✓ (lightweight); plan/metadata/state to be created +PRIORITY: A (user-blocking) + +MISSION: Validate the refined lexicon (Phase 1) by applying it to 2 Pass 1 reports. +PRODUCES: 7 deliverables (2 videos × 3-layer + 1 pilot_report.md) +CONSUMES: lexicon.md + terms_catalog.md + dedup_map.md + prompt_template.md + 2 Pass 1 reports + +CRITICAL: The 5 rules + 4 verification criteria are NON-NEGOTIABLE. Every re-encoding must satisfy all 9. +- The LLM produces the principled re-encoding by default +- The user-specific form is opt-in (Sectored Language, GA, classical Greek/Latin) +- The 4-language pattern is REQUIRED for user-also-accepted terms +- Esoteric content (Witness/Vessel/Aether) is NOT in the public deliverables + +USER DIRECTIVES (load-bearing): +1. Constructive type theory as foundation +2. Boundedness for direct knowledge +3. Cycles/iteration explicit; no "infinite" values +4. Etymology-aware lexicon (1-line origin + 1-line history; 4-language for user-specific) +5. Lossless with compression history +6. Encoding-explicit (every value-bearing term has encoding: attribute) +7. Honest epistemic hedging +8. Principled vs user-specific distinction (Phase 1 formalized this; Phase 2 must preserve it) +9. Secular sanitization + +THE 2 VIDEOS: +- cs229_building_llms: foundational ML/LLM coverage; wide scope; tests "form projection" across many concepts +- entropy_epiplexity: math-heavy, information-theoretic; tests boundedness + type-theoretic encoding of measure theory + +FILES TO READ FIRST: +1. ./TIER2_STARTER.md (this file) +2. ./spec.md (the lightweight scaffold) +3. /conductor/tracks/video_analysis_deob_lexicon_20260621/lexicon.md (the codified operational spec; PRIMARY input) +4. /conductor/tracks/video_analysis_deob_lexicon_20260621/terms_catalog.md (machine-readable) +5. /conductor/tracks/video_analysis_deob_lexicon_20260621/dedup_map.md (6 noise-dedup maps) +6. /conductor/tracks/video_analysis_deob_warmup_20260621/prompt_template.md (the LLM-direct operational spec) +7. /conductor/tracks/video_analysis_cs229_building_llms_20260621/report.md (Pass 1 report #1) +8. /conductor/tracks/video_analysis_entropy_epiplexity_20260621/report.md (Pass 1 report #2) + +EXECUTION: Read inputs → invoke LLM with prompt_template + cs229 report → 3-layer deliverable → invoke LLM with prompt_template + entropy_epiplexity report → 3-layer deliverable → write pilot_report.md → user review → closeout. +``` + +--- + +## Post-Phase-2 + +After Phase 2 ships, Phase 3 (apply) can start: +- `video_analysis_deob_apply_20260621/` (consumes the pilot's `pilot_report.md` refinements + the (refined-by-pilot) lexicon) +- Applies the lexicon to 10 remaining Pass 1 reports + 1 cross-cutting synthesis +- Produces 11 × 3-layer deliverables (33 files) + `apply_report.md` diff --git a/conductor/tracks/video_analysis_deob_pilot_20260621/metadata.json b/conductor/tracks/video_analysis_deob_pilot_20260621/metadata.json new file mode 100644 index 00000000..e209948c --- /dev/null +++ b/conductor/tracks/video_analysis_deob_pilot_20260621/metadata.json @@ -0,0 +1,130 @@ +{ + "track_id": "video_analysis_deob_pilot_20260621", + "name": "Video Analysis De-obfuscation — Pilot on 2 Videos (Pass 2 Phase 2 of 3)", + "created": "2026-06-21", + "status": "spec_approved", + "blocked_by": [ + "video_analysis_deob_lexicon_20260621" + ], + "blocks": [ + "video_analysis_deob_apply_20260621" + ], + "priority": "A", + "rationale": "User-blocking Phase 2 of Pass 2. Consumes the refined lexicon (Phase 1) and applies it to 2 Pass 1 reports (cs229_building_llms + entropy_epiplexity) to validate the lexicon and capture refinements. Produces 3-layer deobfuscation per video (translation / replacement / decoder) + pilot_report.md with refinements. Research-only; no src/ changes.", + "type": "research-only child track (Pass 2 Phase 2 of 3)", + "domain": "meta-tooling (research deliverable; no manual_slop src/ changes)", + "scope": { + "new_folders": [ + "conductor/tracks/video_analysis_deob_pilot_20260621/" + ], + "new_files": [ + "spec.md (lightweight scaffold)", + "plan.md", + "metadata.json", + "state.toml", + "TIER2_STARTER.md", + "artifacts/cs229_building_llms/translation.md", + "artifacts/cs229_building_llms/deobfuscated.md", + "artifacts/cs229_building_llms/decoder.md", + "artifacts/entropy_epiplexity/translation.md", + "artifacts/entropy_epiplexity/deobfuscated.md", + "artifacts/entropy_epiplexity/decoder.md", + "pilot_report.md (lexicon refinements + gaps + process improvements)" + ], + "modified_files": [], + "deleted_files": [], + "gitignored_patterns": [] + }, + "estimated_effort": { + "method": "scope (per conductor/workflow.md Tier 1 Track Initialization Rules). NO day estimates.", + "phase_0": "1 task: init state.toml", + "phase_1": "6 tasks: read lexicon + terms_catalog + dedup_map + prompt_template + 2 Pass 1 reports", + "phase_2": "3 tasks: apply lexicon to cs229 (3-layer deliverable) + commit + verify", + "phase_3": "3 tasks: apply lexicon to entropy_epiplexity (3-layer deliverable) + commit + verify", + "phase_4": "2 tasks: write pilot_report.md + commit", + "phase_5": "5 tasks: user review + state update + idempotency + audit + end-of-track report", + "summary": "5 phases, 20 tasks, 7 deliverables (6 per-video 3-layer files + 1 pilot report). No day estimates per project convention." + }, + "verification_criteria": [ + "Both pilot videos have 3-layer deliverables (6 files total in artifacts/)", + "All 3 deliverables per video pass the 4 criteria (Lossless, Bounded, Constructively typed, Etymology-cited)", + "pilot_report.md captures lexicon refinements + gaps + process improvements", + "User has reviewed and approved the 2 deliverables + pilot_report.md", + "All 7 deliverables committed atomically (6 per-video + 1 pilot_report)", + "Git notes attached to each commit", + "state.toml updated to status = 'completed'", + "End-of-track report at docs/reports/TRACK_COMPLETION_video_analysis_deob_pilot_20260621.md", + "No new src/*.py files created (per AGENTS.md File Size and Naming Convention)", + "No new pyproject.toml dependencies" + ], + "risk_register": [ + { + "id": "R1", + "title": "LLM produces de-obfuscation that violates the principled/user-specific distinction", + "likelihood": "medium", + "scope_impact": "the 3-layer output may include user-specific forms as if they were principled", + "mitigation": "TIER2_STARTER.md tells Tier 2 the LLM's prompt_template.md already encodes the distinction; the verification checklist explicitly checks that user-specific entries are tagged and the principled spine is preserved" + }, + { + "id": "R2", + "title": "Lexicon can't capture a concept in bounded form", + "likelihood": "medium", + "scope_impact": "some concepts in the Pass 1 reports must be flagged 'indefinite — see original'", + "mitigation": "Rule 1 (Boundedness) explicitly allows 'indefinite — see original' rather than forcing a translation; the pilot_report.md captures these gaps for the apply phase" + }, + { + "id": "R3", + "title": "LLM output is too verbose (decoder has too many terms)", + "likelihood": "low", + "scope_impact": "decoder.md may be 1000+ LOC for a single video", + "mitigation": "decoder format is per-term; the LLM should group by section and skip terms that don't appear in the de-obfuscated.md" + }, + { + "id": "R4", + "title": "LLM output is too brief (skips concepts)", + "likelihood": "medium", + "scope_impact": "violates the lossless directive (Rule 4)", + "mitigation": "verification checklist includes the Lossless criterion; the 4 verbatim checks per video; pilot_report.md captures skips as gaps" + }, + { + "id": "R5", + "title": "Entropy_epiplexity's measure theory defeats the boundedness rule", + "likelihood": "medium", + "scope_impact": "the report's information-theoretic concepts may not have bounded forms in the lexicon", + "mitigation": "this is WHY the pilot tests this video; if the lexicon can't capture it, capture the gap in pilot_report.md; the apply phase uses the refined lexicon (or a v2)" + } + ], + "architecture_reference": { + "primary_documents": [ + "conductor/workflow.md (track convention, per-task commits, git notes)", + "conductor/tracks/video_analysis_deob_20260621/spec.md (umbrella design)", + "conductor/tracks/video_analysis_deob_lexicon_20260621/lexicon.md (the refined operational spec; PRIMARY input)", + "conductor/tracks/video_analysis_deob_lexicon_20260621/terms_catalog.md (machine-readable lexicon)", + "conductor/tracks/video_analysis_deob_lexicon_20260621/dedup_map.md (6 noise-dedup maps refined)" + ], + "related_tracks": [ + "conductor/tracks/video_analysis_deob_warmup_20260621/ (upstream precursor; the prompt_template.md is the LLM-direct operational spec)", + "conductor/tracks/video_analysis_deob_apply_20260621/ (downstream consumer; consumes pilot_report.md refinements)" + ] + }, + "deferred_to_followup_tracks": [ + { + "title": "Phase 3 (apply)", + "description": "Applies the refined lexicon (informed by the pilot) to 10 remaining Pass 1 reports + 1 cross-cutting synthesis.", + "track_status": "blocked by this track" + } + ], + "regressions_and_pre_existing_failures": [], + "pre_existing_failures_remaining": [], + "user_directives": [ + "Constructive type theory as foundation (2026-06-21)", + "Boundedness for direct knowledge; cycles/iteration explicit (2026-06-21)", + "Etymology-aware lexicon (2026-06-21)", + "Lossless with explicit compression history (2026-06-23)", + "Encoding-explicit (every value-bearing term has encoding: attribute) (2026-06-23)", + "Honest epistemic hedging (2026-06-21)", + "Secular sanitization - esoteric content NOT in public lexicon (2026-06-23)", + "Principled vs user-specific distinction - Phase 1 formalized this (2026-06-23); Phase 2 must preserve it", + "No day estimates per conductor/workflow.md Tier 1 Track Initialization Rules (added 2026-06-16). Scope measured in files/sites only." + ] +} diff --git a/conductor/tracks/video_analysis_deob_pilot_20260621/plan.md b/conductor/tracks/video_analysis_deob_pilot_20260621/plan.md new file mode 100644 index 00000000..d5c167e0 --- /dev/null +++ b/conductor/tracks/video_analysis_deob_pilot_20260621/plan.md @@ -0,0 +1,61 @@ +# Plan: Video Analysis De-obfuscation — Pilot on 2 Videos + +This is the Phase 2 (pilot) child plan for Pass 2 of the 3-pass research campaign. Per the Tier 1 Track Initialization Rules, scope is measured in files/sites — no day estimates. + +## Phase 0: Init + +- [ ] **Task 0.1:** Initialize the Phase 2 child track: create `state.toml` (Tier 2). + +## Phase 1: Read the refined lexicon (the operational spec) + +- [ ] **Task 1.1:** Read `lexicon.md` (the codified operational spec; 12 sections + 4 appendices). +- [ ] **Task 1.2:** Read `terms_catalog.md` (the machine-readable form; 4-tier table with 9 columns per term). +- [ ] **Task 1.3:** Read `dedup_map.md` (the 6 noise-dedup maps refined; 3 principled + 3 user-preferred). +- [ ] **Task 1.4:** Read the warmup's `prompt_template.md` (the LLM-direct operational spec; 5 rules + 6 noise-dedup maps + 4-layer format + 7 example transformations). +- [ ] **Task 1.5:** Read both Pass 1 reports: `cs229_building_llms/report.md` (foundational ML/LLM coverage; wide scope) + `entropy_epiplexity/report.md` (math-heavy, information-theoretic; tests boundedness + type-theoretic encoding of measure theory). +- [ ] **Task 1.6:** Honor the surgical-edits formalization: the lexicon's `[user-also-accepted]` and `[user-preferred]` tags are explicit. The LLM produces the principled re-encoding by default; the user-specific form is opt-in. + +## Phase 2: Apply to `cs229_building_llms` + +- [ ] **Task 2.1:** Tier 3 worker invokes an LLM with the `prompt_template.md` + the `cs229_building_llms` Pass 1 report. The LLM produces 3 files in `artifacts/cs229_building_llms/`: + - `translation.md` (side-by-side table; uses the terms from `terms_catalog.md`) + - `deobfuscated.md` (re-encoded; same 8-section structure as Pass 1, but every standard-math expression is replaced with the principled re-encoding) + - `decoder.md` (per-term decoder; form anchor + etymology + definition history) +- [ ] **Task 2.2:** Commit the 3 deliverables (one commit per file with git note) at `artifacts/cs229_building_llms/`. +- [ ] **Task 2.3:** Verify the 4 criteria (Lossless, Bounded, Constructively typed, Etymology-cited) are met. + +## Phase 3: Apply to `entropy_epiplexity` + +- [ ] **Task 3.1:** Tier 3 worker invokes an LLM with the `prompt_template.md` + the `entropy_epiplexity` Pass 1 report. The LLM produces 3 files in `artifacts/entropy_epiplexity/` (same structure as Phase 2). +- [ ] **Task 3.2:** Commit the 3 deliverables at `artifacts/entropy_epiplexity/`. +- [ ] **Task 3.3:** Verify the 4 criteria are met. + +## Phase 4: Refine + pilot report + +- [ ] **Task 4.1:** Write `pilot_report.md` capturing: + - **Lexicon refinements** discovered during the 2 pilot applications (terms that didn't fit; terms that needed new entries; terms that worked but need adjustment) + - **Gaps** — concepts in the Pass 1 reports that the lexicon couldn't capture + - **Process improvements** for the apply phase (e.g., how to scale the lexicon application to 10 + 1 synthesis) + - **The 31 unresolved items from the warmup** — which got defined, which got deferred, which need a v2 +- [ ] **Task 4.2:** Commit `pilot_report.md` with git note. + +## Phase 5: User review + verification + +- [ ] **Task 5.1:** User reviews the 2 × 3-layer deliverables + `pilot_report.md`. Approves the refined lexicon, or iterates (loop back to Phase 2's lexicon child). +- [ ] **Task 5.2:** Update `state.toml` to `status = "completed"`. +- [ ] **Task 5.3:** Idempotency check (re-run the LLM invocation, confirm the 3-layer output is consistent). +- [ ] **Task 5.4:** Audit checklist: 4 criteria met for both videos; lexicon refinements captured in `pilot_report.md`; no esoteric content leaked. +- [ ] **Task 5.5:** Write end-of-track report at `docs/reports/TRACK_COMPLETION_video_analysis_deob_pilot_20260621.md`. + +## Verification (gate per workflow.md) + +- [ ] Both pilot videos have 3-layer deliverables (6 files total in `artifacts/`) +- [ ] All 3 deliverables per video pass the 4 criteria (Lossless, Bounded, Constructively typed, Etymology-cited) +- [ ] `pilot_report.md` captures lexicon refinements + gaps + process improvements +- [ ] User has reviewed and approved +- [ ] All 7 deliverables committed atomically (6 per-video + 1 pilot_report) +- [ ] Git notes attached to each commit +- [ ] `state.toml` updated to `status = "completed"` +- [ ] End-of-track report at `docs/reports/TRACK_COMPLETION_video_analysis_deob_pilot_20260621.md` + +The Phase 2 child is "Pass 2 Phase 2 complete" when the 2 × 3-layer deliverables + pilot report are committed + user-approved. Phase 3 (apply) can then start. diff --git a/conductor/tracks/video_analysis_deob_pilot_20260621/spec.md b/conductor/tracks/video_analysis_deob_pilot_20260621/spec.md index f6dc7135..708bf4fc 100644 --- a/conductor/tracks/video_analysis_deob_pilot_20260621/spec.md +++ b/conductor/tracks/video_analysis_deob_pilot_20260621/spec.md @@ -47,13 +47,13 @@ Plus: |---|---| | `pilot_report.md` | Lexicon refinements discovered, concepts that didn't fit (gaps), process improvements for the apply phase | -**Optional (added per child track execution convention):** `plan.md`, `metadata.json`, `state.toml`. +**Optional (added per child track execution convention):** `plan.md`, `metadata.json`, `state.toml`. (These are now present in the folder, scaffolded at spec time for Tier 2 to consume. Per the lexicon child pattern, `plan.md` enumerates the 5-phase pipeline; `metadata.json` is the scope/risk register; `state.toml` is the task tracker.) ## 4. Pipeline (5 phases) Per the umbrella spec §5 (Phase 3 of the umbrella). Each phase commits atomically. -- [ ] **Phase 1: Init.** Initialize the child track. +- [ ] **Phase 1: Init.** Initialize the child track (Tier 2 reads the scaffolded `plan.md` + `metadata.json` + `state.toml`). - [ ] **Phase 2: Apply to `cs229_building_llms`.** Tier 3 worker (LLM invoked with the prompt template + the report). Produces 3-layer deliverable in `artifacts/cs229_building_llms/`. - [ ] **Phase 3: Apply to `entropy_epiplexity`.** Same as above. Produces 3-layer deliverable in `artifacts/entropy_epiplexity/`. - [ ] **Phase 4: Refine.** Write `pilot_report.md` capturing: lexicon refinements, concepts that didn't fit (gaps), process improvements. @@ -101,4 +101,9 @@ For `_decoder.md`, the per-term format: - [Umbrella spec.md](../../video_analysis_deob_20260621/spec.md) - [Umbrella README.md](../../video_analysis_deob_20260621/README.md) - [Lexicon child spec.md](../../video_analysis_deob_lexicon_20260621/spec.md) +- [Lexicon child lexicon.md](../../video_analysis_deob_lexicon_20260621/lexicon.md) (the refined operational spec; PRIMARY input) +- [Lexicon child terms_catalog.md](../../video_analysis_deob_lexicon_20260621/terms_catalog.md) (machine-readable lexicon) +- [Lexicon child dedup_map.md](../../video_analysis_deob_lexicon_20260621/dedup_map.md) (6 noise-dedup maps) +- [Warmup prompt_template.md](../../video_analysis_deob_warmup_20260621/prompt_template.md) (the LLM-direct operational spec) - Pass 1 reports: `cs229_building_llms` + `entropy_epiplexity` +- [TIER2_STARTER.md](./TIER2_STARTER.md) (the dispatch prompt for Tier 2) diff --git a/conductor/tracks/video_analysis_deob_pilot_20260621/state.toml b/conductor/tracks/video_analysis_deob_pilot_20260621/state.toml new file mode 100644 index 00000000..41dc6444 --- /dev/null +++ b/conductor/tracks/video_analysis_deob_pilot_20260621/state.toml @@ -0,0 +1,82 @@ +# Track state for video_analysis_deob_pilot_20260621 +# Updated by Tier 2 Tech Lead during execution + +[meta] +track_id = "video_analysis_deob_pilot_20260621" +name = "Video Analysis De-obfuscation — Pilot on 2 Videos (Pass 2 Phase 2 of 3)" +status = "active" +current_phase = 0 # Phase 0 = init +last_updated = "2026-06-23" + +[blocked_by] +video_analysis_deob_lexicon_20260621 = "shipped 2026-06-23 (b7988c49)" + +[blocks] +video_analysis_deob_apply_20260621 = "blocked (consumes pilot_report.md refinements)" + +[phases] +phase_0 = { status = "pending", checkpointsha = "", name = "Init (state.toml)" } +phase_1 = { status = "pending", checkpointsha = "", name = "Read the refined lexicon (the operational spec)" } +phase_2 = { status = "pending", checkpointsha = "", name = "Apply to cs229_building_llms" } +phase_3 = { status = "pending", checkpointsha = "", name = "Apply to entropy_epiplexity" } +phase_4 = { status = "pending", checkpointsha = "", name = "Refine + pilot report" } +phase_5 = { status = "pending", checkpointsha = "", name = "User review + verification" } + +[tasks] +# Phase 0 (init) +t0_1 = { status = "pending", commit_sha = "", description = "Initialize Phase 2 child track: create state.toml per Tier 2 conventions" } + +# Phase 1 (read refined lexicon) +t1_1 = { status = "pending", commit_sha = "", description = "Read lexicon.md (the codified operational spec; 12 sections + 4 appendices)" } +t1_2 = { status = "pending", commit_sha = "", description = "Read terms_catalog.md (the machine-readable form; 4-tier table with 9 columns per term)" } +t1_3 = { status = "pending", commit_sha = "", description = "Read dedup_map.md (the 6 noise-dedup maps refined; 3 principled + 3 user-preferred)" } +t1_4 = { status = "pending", commit_sha = "", description = "Read the warmup's prompt_template.md (the LLM-direct operational spec; 5 rules + 6 noise-dedup maps + 4-layer format + 7 example transformations)" } +t1_5 = { status = "pending", commit_sha = "", description = "Read both Pass 1 reports: cs229_building_llms/report.md (foundational ML/LLM) + entropy_epiplexity/report.md (math-heavy, information-theoretic)" } +t1_6 = { status = "pending", commit_sha = "", description = "Honor the surgical-edits formalization: the lexicon's [user-also-accepted] and [user-preferred] tags are explicit. The LLM produces the principled re-encoding by default; the user-specific form is opt-in." } + +# Phase 2 (apply to cs229) +t2_1 = { status = "pending", commit_sha = "", description = "Tier 3 worker invokes an LLM with the prompt_template.md + the cs229_building_llms Pass 1 report. Produces 3 files in artifacts/cs229_building_llms/: translation.md (side-by-side table) + deobfuscated.md (re-encoded) + decoder.md (per-term)" } +t2_2 = { status = "pending", commit_sha = "", description = "Commit the 3 deliverables (one commit per file with git note) at artifacts/cs229_building_llms/" } +t2_3 = { status = "pending", commit_sha = "", description = "Verify the 4 criteria (Lossless, Bounded, Constructively typed, Etymology-cited) are met" } + +# Phase 3 (apply to entropy_epiplexity) +t3_1 = { status = "pending", commit_sha = "", description = "Tier 3 worker invokes an LLM with the prompt_template.md + the entropy_epiplexity Pass 1 report. Produces 3 files in artifacts/entropy_epiplexity/: translation.md + deobfuscated.md + decoder.md" } +t3_2 = { status = "pending", commit_sha = "", description = "Commit the 3 deliverables at artifacts/entropy_epiplexity/" } +t3_3 = { status = "pending", commit_sha = "", description = "Verify the 4 criteria are met" } + +# Phase 4 (refine + pilot report) +t4_1 = { status = "pending", commit_sha = "", description = "Write pilot_report.md capturing: (a) Lexicon refinements discovered during the 2 pilot applications; (b) Gaps — concepts the lexicon couldn't capture; (c) Process improvements for the apply phase; (d) Updates to the 31 unresolved items from the warmup" } +t4_2 = { status = "pending", commit_sha = "", description = "Commit pilot_report.md with git note" } + +# Phase 5 (user review + verification) +t5_1 = { status = "pending", commit_sha = "", description = "User reviews the 2 × 3-layer deliverables + pilot_report.md. Approves the refined lexicon or iterates (loop back to Phase 2's lexicon child)" } +t5_2 = { status = "pending", commit_sha = "", description = "Update state.toml to status = 'completed'" } +t5_3 = { status = "pending", commit_sha = "", description = "Idempotency check (re-run the LLM invocation, confirm the 3-layer output is consistent)" } +t5_4 = { status = "pending", commit_sha = "", description = "Audit checklist: 4 criteria met for both videos; lexicon refinements captured in pilot_report.md; no esoteric content leaked" } +t5_5 = { status = "pending", commit_sha = "", description = "Write end-of-track report at docs/reports/TRACK_COMPLETION_video_analysis_deob_pilot_20260621.md" } + +[verification] +cs229_translation_committed = false +cs229_deobfuscated_committed = false +cs229_decoder_committed = false +entropy_translation_committed = false +entropy_deobfuscated_committed = false +entropy_decoder_committed = false +pilot_report_committed = false +user_approved = false +state_toml_completed = false +end_of_track_report_committed = false +all_4_criteria_cs229 = false +all_4_criteria_entropy = false + +[user_directives_logged] +constructive_type_theory = "Per user 2026-06-21: 'I like Norman Wildberger's work. And I like the constructivist current progress on type theories as a foundational system.'" +bounded_for_knowledge = "Per user 2026-06-21: 'No observer or mechanism or construct can be infinite in resolution or quantification. To have distinction must have a bounds.'" +cycles_iteration_allowed = "Per user 2026-06-21: 'Infinite is okay well handled CORRECTLY... What can be indefinite is that can be subjected upon is that of cycles, that of iteration, that of repetition.'" +etymology_aware = "Per user 2026-06-21: etymology + definitional history documented" +lossless_compression_history = "Per user 2026-06-23: explicit compression notes per layer in the de-obfuscation's 3-layer output" +encoding_explicit = "Per user 2026-06-23: every value-bearing term has an encoding: attribute (default float64)" +honest_epistemic_hedging = "Per user 2026-06-21: 'Don't know what `<<` here is' / 'Me fucking around' style annotations are valid" +secular_sanitization = "Per user 2026-06-23: esoteric content (Witness/Vessel/Aether, classical philosophy) NOT in public lexicon; stays in cluster_0_twitter.md" +principled_vs_user_specific = "Per user 2026-06-23 surgical edits: Phase 1 formalized this; Phase 2 must preserve it (the LLM produces the principled re-encoding; user-specific form is opt-in)" +no_day_estimates = "Per conductor/workflow.md Tier 1 Track Initialization Rules (added 2026-06-16). Scope measured in files/sites only."