ed/manual_slop
Private
Public Access
0
0
Fork 0
Code Activity

conductor(track): nagent_review_v3 §9 Case-study methodology cluster

Browse Source
This commit is contained in:
ed
2026-06-20 08:29:36 -04:00
parent d876744fc5
commit 54e62b1037
1 changed files with 63 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
conductor/tracks/nagent_review_20260608/nagent_review_v3_20260619.md
+63 -1
View File
@@ -489,7 +489,69 @@ The Q9 expansion generalizes v2.3 Pattern 1 ("durable work, disposable workers")
## §9 Case-study methodology
(filled in by Phase 10 — the 5-element pattern + GPT-5.5 note + sibling-review cross-refs)
**Source:** both case-study repos (`macton/pep-copt`, `macton/differentiable-collisions-optc`); both `prompts/create-*.md` files in each; both `prove-optimized-harness.sh` scripts (per §3 cross-refs); both `README.md` files.
**One-liner:** A reusable abstraction surfaces across both case studies — the 4-prompt methodology + proof harness + optimization log + committed-input sha256 freeze + model-as-test-subject framing. Both repos implement the same pattern with different match contracts (PEP byte-identity vs collisions tolerance-based) but the same empirical-discipline skeleton.
**Pattern(s) vs v2.3:** NEW. v2.3 had no case-study methodology (no case-study repos existed). v3 introduces a 5-element pattern that any project adopting nagent can replicate to ground LLM-driven optimization in measurement. EXTENDS v2.3 Pattern 5 ("the loop") with the per-turn proof injection that the harness provides. EXTENDS v2.3 Pattern 7 ("repo history as data") with the optimization log as a per-hypothesis history file.
**Manual Slop implications:** Manual Slop's discussion history + screenshots are the per-turn observability surface; the case-study methodology suggests a parallel structure: a per-iteration optimization log file (`OPTIMIZATION-LOG.md`) that records hypothesis + change + before/after + keep/revert + cost. The "committed-input sha256 freeze" maps to Manual Slop's test fixtures (gitignored, but checksum-verified). The 4-prompt methodology maps to Manual Slop's `prompts/` (already established, per `conductor/code_styleguides/knowledge_artifacts.md`).
**Decision candidate:** NEW Candidate 25 (MEDIUM). "Optimization-log discipline for Manual Slop agent work" — adopt the `OPTIMIZATION-LOG.md` pattern: every agent iteration records hypothesis + change + before/after + keep/revert + cost (wall-clock + tokens). See `decisions.md` Candidate 25.
**Cross-refs:** `conductor/tracks/intent_dsl_survey_20260612/` — the survey's Cluster 4 "Meta-Tooling DSLs" is the closest prior art (the 4-prompt methodology is implicitly an intent-DSL for "drive nagent at an optimization problem"). `conductor/tracks/superpowers_review_20260619/` — the superpowers `brainstorming` skill is a process parallel (structured questions to refine an idea before implementation; the case-study prompts serve the same role). §3 Hooks (the proof harness IS the `--hook-per-run`); §8 Operating rules (the Q9 expansion is invoked when micro-tweaks plateau).
**Source-read citations:**
- `pep-copt/README.md` — full project description, 4-prompt methodology, 24-image results, "The model under test here was GPT-5.5" not present (pep-copt does not name the model), byte-identity + size + decode contract
- `pep-copt/prompts/create-reference.md` — reference pipeline specification
- `pep-copt/prompts/create-optimized-test-harness.md` — test/comparison/measurement scaffold
- `pep-copt/prompts/create-optimized.md` — optimization instructions: 4 candidate kinds (a/b/c/d); "When you have plateaued — several consecutive reverts, or micro-tweaks stuck below target — stop filing the current machine: re-profile the data and evaluate a (c) or (d) candidate"
- `pep-copt/prompts/create-visualizer.md` — quality visualizer specification
- `pep-copt/prove-optimized-harness.sh` — 9-step proof + 5 enforcing gates
- `pep-copt/src-optimized/OPTIMIZATION-LOG.md` — per-hypothesis history (referenced from README)
- `differentiable-collisions-optc/README.md` — full project description, 4-prompt methodology, 1000-pair benchmark, "The model under test here was GPT-5.5. This is one model, one run — a case study in how to drive an LLM at an optimization problem, not a benchmark comparing models", tolerance-based + collision-flag + contact-validator contract
- `differentiable-collisions-optc/prompts/create-reference.md` — reference specification
- `differentiable-collisions-optc/prompts/create-optimized-test-harness.md` — harness specification
- `differentiable-collisions-optc/prompts/create-optimized.md` — optimization instructions; "The most durable headroom from here is structural — batching and data layout — rather than more iteration-shaving"
- `differentiable-collisions-optc/prompts/create-visualizer.md` — visualizer specification
- `differentiable-collisions-optc/prove-optimized-harness.sh` — 10-step proof + 4 enforcing gates
- `differentiable-collisions-optc/src-optimized/OPTIMIZATION-LOG.md` — per-hypothesis history
**Honest gaps in this cluster:**
- **The GPT-5.5 string is unverified.** As of 2026-06-20, the publicly-known GPT families are 4 / 4o / 4.5 / 5; "GPT-5.5" is not a known public model. The collisions README's framing — "This is one model, one run — a case study in how to drive an LLM at an optimization problem, not a benchmark comparing models" — suggests deliberate model-disconnect (a fake name as a methodology test) OR a private/internal model OR a typo. The pep-copt README does not name the model. Without further evidence, the §9 section treats "GPT-5.5" as a model-disconnect placeholder per the README's stated framing.
- The 4-prompt methodology is implicit (the README lists the 4 prompts but does not name the pattern). The §9 cluster surfaces the pattern explicitly; a future track could formalize it as `prompts/create-{phase}.md` template.
- The "different machine" replacement (Q9 from §8) is invoked in the case-study README ("stop filing the current machine") but the prompts do not cite Q9 by name. The connection is implicit; an explicit cross-reference would help.
- The optimization log format (`OPTIMIZATION-LOG.md` schema) is not specified in the prompts; each repo develops its own. A template would help future projects adopt the pattern.
**Pattern deep-dive.** The case-study methodology is a 5-element composition: **prompts**, **harness**, **log**, **freeze**, **subject**. Prompts: 4 phase-specific instruction documents (create-reference, create-optimized-test-harness, create-optimized, create-visualizer) feed the LLM in sequence. Harness: `prove-optimized-harness.sh` runs end-to-end on every turn via `nagent --hook-per-run` (§3 cross-ref), enforcing the match contract (byte-identity for PEP; tolerance-based for collisions). Log: `OPTIMIZATION-LOG.md` records per-hypothesis history with measurements, keep/revert decisions, and cost. Freeze: the committed input's sha256 is verified before and after the run — the benchmark cannot be quietly edited. Subject: the model is named in the README (collisions explicitly says "GPT-5.5") as a methodology-test single-model run, not a benchmark.
The match-contract variation between the two repos is informative. PEP uses byte-identity after decompression (lossless, `.pep` not larger, decode net-neutral-or-better) — the strictest contract because the codec's encode/decode is symmetric. Collisions uses tolerance-based (collision flags identical, distance within `1 mm + 0.1%·|d_ref| + 5e-4·(|c1−c2|/α²)`, contact points certified for validity rather than matched) — a relaxed contract because collision detection has many equally-valid witness points for face/edge contacts. The two contracts are "same-shape" (PEP) and "same-distribution" (collisions); both are data-grounded, both are checkable. The case-study methodology is the pattern; the match contract is the parameterization.
The connection to §8 Q9 is direct. The pep-copt prompt at line "When you have plateaued — several consecutive reverts, or micro-tweaks stuck below target — stop filing the current machine: re-profile the data and evaluate a (c) or (d) candidate" is the §8 Q9 expansion applied in the wild. The (c) "representation/algorithm" candidate kind is Q9 ("is there a different machine?"); the (d) "data-pattern specialization" candidate kind is Q5/Q6 (lookup tables — let the data show what to specialize). The case-study methodology is the empirical harness for Q9's principle.
The connection to `intent_dsl_survey_20260612` is implicit. The survey's Cluster 4 ("Meta-Tooling DSLs") discusses how DSLs for tool composition work; the 4-prompt methodology is a primitive form of "drive the agent through these 4 phases." The survey's "intent-mapping" cluster (Cluster 3) is the closest parallel — the 4 prompts ARE an intent-DSL for "drive nagent at an optimization problem." A future track could lift the 4-prompt methodology to a templated DSL (e.g. `prompts/create-{phase}.md` skeleton with placeholders for domain-specific terminology).
The connection to `superpowers_review_20260619` is process-parallel. The superpowers `brainstorming` skill asks structured questions to refine an idea before implementation (per `superpowers/specs/2026-06-XX-brainstorming-design.md`); the case-study methodology asks structured prompts to refine an optimization before measurement. Both serve "the model should not skip the early work." A future track could document the parallel.
A code-shape sketch using survey grammar:
```
case-study { input, model, target } :: result {ssdl} [B]
// 4-prompt methodology, run in sequence
ref := run(prompts/create-reference, input, model)
harness := run(prompts/create-optimized-test-harness, input, model)
log := []
for iter := 0..N:
hypothesis := pick-candidate(log, ref)
opt := run(prompts/create-optimized, {input, hypothesis}, model)
hook-result := hook-per-run(harness, opt) // per §3
verdict := gate(hook-result, contract) // match contract: byte-identity | tolerance
if verdict.ok:
log.append({hypothesis, opt, hook-result, verdict, cost})
commit(opt, log)
else:
log.append({hypothesis, opt, hook-result, verdict, cost, kept: false})
revert()
if plateau(log) -> replace-machine(log) // per §8 Q9
return opt
```
The `{ssdl}` [B] marker notes the abstraction: the case-study is a boundary where the model's working state meets measurement. The match contract is the parameterization. The 4 prompts, harness, log, freeze, and subject are the 5 elements; the loop is the shape that composes them.
The GPT-5.5 observation is worth a separate note. As of 2026-06-20, public GPT families are 4 / 4o / 4.5 / 5; "GPT-5.5" is not a known public model. The collisions README's framing — "case study in how to drive an LLM, not a benchmark comparing models" — suggests either (a) a private/internal model, (b) a model-disconnect placeholder (use a fake name to test whether the methodology works without depending on a specific model's quirks), or (c) a typo. Without further evidence, the §9 section treats "GPT-5.5" as a model-disconnect placeholder per the README's stated framing. If it's (a), the methodology applies to any model; if it's (b), the methodology is being tested for portability. Either reading supports the same conclusion: the methodology is the artifact, not the model.
## §10 PEP case study