From bb1ddcabc7ed374050f2f11f534d05ea76f2b857 Mon Sep 17 00:00:00 2001 From: Ed_ Date: Sun, 5 Jul 2026 13:18:16 -0400 Subject: [PATCH] =?UTF-8?q?conductor(track):=20superpowers=20review=20sect?= =?UTF-8?q?ion=205=20=E2=80=94=20verification-before-completion=20(deep-di?= =?UTF-8?q?ve)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../superpowers_review_20260619/report.md | 62 ++++++++++++++++++- 1 file changed, 61 insertions(+), 1 deletion(-) diff --git a/conductor/tracks/superpowers_review_20260619/report.md b/conductor/tracks/superpowers_review_20260619/report.md index 8a4cc9e8..3e41ad3a 100644 --- a/conductor/tracks/superpowers_review_20260619/report.md +++ b/conductor/tracks/superpowers_review_20260619/report.md @@ -247,7 +247,67 @@ The project follows the TDD discipline strictly. The four elaborations (tier-3-w ## 5. Verification Before Completion - +## 5. Verification Before Completion + +### 5.1 What the skill prescribes + +The `verification-before-completion` skill is rigid — same status as TDD. The Iron Law: "NO COMPLETION CLAIMS WITHOUT FRESH VERIFICATION EVIDENCE." The Gate Function is a 5-step process: IDENTIFY (what command proves this claim?) → RUN (execute the FULL command fresh) → READ (full output, exit code, failure counts) → VERIFY (does output confirm the claim?) → ONLY THEN (make the claim). The skill's Common Failures table is the canonical list of what "verification" means: Tests pass = test command output (0 failures); Linter clean = linter output (0 errors); Build succeeds = build command (exit 0); Bug fixed = test original symptom (passes); Regression test works = red-green verified; Agent completed = VCS diff shows changes; Requirements met = line-by-line checklist. + +The skill's Red Flags list the linguistic telltales: "should", "probably", "seems to"; expressing satisfaction before verification ("Great!", "Perfect!", "Done!"); about to commit without verification; trusting agent success reports; partial verification; "just this once"; tired and wanting work over; ANY wording implying success without verification. + +The Rationalization Prevention table is exhaustive: "Should work now" → RUN; "I'm confident" → Confidence ≠ evidence; "Linter passed" → Linter ≠ compiler; "Agent said success" → Verify independently; "I'm tired" → Exhaustion ≠ excuse; "Different words so rule doesn't apply" → Spirit over letter. + +### 5.2 Mapping to the project's existing pattern + +The project has a multi-layered verification regime that goes beyond the skill's iron law: + +| Skill rule | Project equivalent | Where | +|---|---|---| +| "Tests pass" requires test command output | `conductor/workflow.md` §"Quality Gates" — "All tests pass"; the batched runner `scripts/run_tests_batched.py` provides tier-based filtering and parallelization | `conductor/workflow.md` + `scripts/run_tests_batched.py` | +| "Build succeeds" requires exit 0 | `conductor/tech-stack.md` + `pyproject.toml` — but the project doesn't have a build step (interpreted Python), so verification is "tests + import-check + smoke run" | Project-wide | +| "Bug fixed" requires test of original symptom | `conductor/workflow.md` "systematic-debugging" reference + the project's "Bug found → write failing test reproducing it" rule (per TDD skill integration) | `conductor/workflow.md` | +| "Regression test works" requires red-green-red cycle | Project follows TDD strictly (per Section 4); red-green is enforced via the workflow's "Write Failing Tests" step | `conductor/workflow.md` §"Task Workflow" step 4 | +| "Agent completed" requires VCS diff verification | `conductor/workflow.md` §"Conductor Token Firewalling" tier-4 QA dispatch — agents report failures, humans check VCS diffs | `conductor/workflow.md` | +| "Requirements met" requires line-by-line checklist | `conductor/workflow.md` §"Phase Completion Verification and Checkpointing Protocol" — automated tests + API hooks + user confirmation | `conductor/workflow.md` | +| "Express satisfaction before verification" | `AGENTS.md` "Critical Anti-Patterns" — implicit (the user calls out "Express satisfaction before verification" patterns; the docs use neutral language) | `AGENTS.md` | +| "About to commit/push/PR without verification" | The project's per-task atomic commits + git notes enforce this: every commit has a verification step in its plan.md task | `conductor/workflow.md` §"Task Workflow" step 9 | +| "Trusting agent success reports" | Tier-4 QA agents are dispatched to summarize errors, not to claim success; the project's tier-3 worker success reports are checked via VCS diff | `conductor/workflow.md` §"Conductor Token Firewalling" | +| "Partial verification" | The Isolated-Pass Verification Fallacy rule (per `conductor/workflow.md` §"Isolated-Pass Verification Fallacy"): "A test that passes in isolation but fails in batch is failing — it's just that the failure is masked by isolation." | `conductor/workflow.md` | +| "Tired and wanting work over" | n/a — no explicit anti-rationalization table for this | — | + +### 5.3 Where the project already follows the discipline + +- **The Phase Completion Verification and Checkpointing Protocol** is the project's verification gate. It runs (1) automated tests in batches (max 4 files at a time, with timeouts), (2) automated API hook verification, (3) presents results to the user for confirmation. This goes BEYOND the skill's "make the claim with evidence" by adding user confirmation. +- **The Skip-Marker Policy** is project-specific: `@pytest.mark.skip` is documentation of a known failure, not an excuse to avoid fixing. The skill doesn't have this discipline; the project's policy is that skip markers must document the underlying issue and the fix path. +- **The Isolated-Pass Verification Fallacy rule** is a project-specific elaboration: tests that pass in isolation but fail in batch are FAILING. The skill's Common Failures table says "Tests pass = Test command output: 0 failures" but doesn't address the isolation-vs-batch distinction. +- **The "Verification before completion" AGENTS.md reference** is implicit in the per-task atomic commits + git notes workflow: every commit has a verification step recorded in its plan.md task. + +### 5.4 Where the project doesn't follow the discipline + +- **"Tired and wanting work over"** is not explicitly addressed. The project's verification is procedural, not anti-rationalization. This is a softer gap; in practice the user catches it. +- **The skill's "Trusting agent success reports" rule** is partially addressed by the tier-4 QA dispatch pattern, but tier-3 workers are trusted to commit code without per-task human review (per `conductor/workflow.md` §"Task Workflow" step 5). The project uses tier-2's per-task git notes as the verification artifact, but the user doesn't approve each task before the next starts. +- **The skill's "ANY wording implying success without verification" rule** is enforced by the project's reporting style (neutral language in commits + git notes), but there's no audit that flags overly-satisfied phrasing. +- **"Verify Fix Worked" (Phase 4 of systematic-debugging)** maps to "run tests after fix" but the project doesn't have a dedicated verification step that says "verify the original symptom is gone" — it relies on the test suite as the proxy. + +### 5.5 Recommendations summary + +The project follows the discipline with significant elaborations (Phase Completion Verification, Skip-Marker Policy, Isolated-Pass Verification Fallacy). The deferred rebuild may want to: + +- **MEDIUM:** Add a "verification ritual" anti-rationalization table to `conductor/workflow.md` §"Task Workflow" step 9 ("Commit Code Changes"). The table would catch the soft gaps: "tired", "should work", "agent said success". The current project relies on user review to catch these; codifying the table would make them explicit. +- **LOW:** Consider adding a `verification-before-completion` SKILL.md to `.agents/skills/` mirroring the superpowers one. The project's discipline is currently embedded in workflow.md prose; a dedicated skill would make it toolable for agents that use the Skill tool. + +**Verdict.** + +| Field | Value | +|---|---| +| **Primary** | `PARITY` | +| **Integration tag** | `INTEGRATED` | +| **Section size** | deep-dive | +| **Cross-refs** | nagent_review_20260608 §10 ("durable verification"); fable_review_20260617 §13 (project's evidence-first approach vs Fable's persona-first approach); intent_dsl_survey_20260612 §6 (verification is one of the project's "AI-agent properties" the survey documents) | + +**Rationale.** The project's verification regime is a faithful re-encoding of the skill with three project-specific elaborations (Phase Completion Verification, Skip-Marker Policy, Isolated-Pass Verification Fallacy). The Iron Law is enforced via the per-task commit + git note + user-confirmation gate. The skill's anti-rationalization table is mostly present in prose form, but the "tired" and "should work" patterns are not explicitly caught. + +**Recommended change.** Add a "verification ritual" anti-rationalization table to `conductor/workflow.md` §"Task Workflow" step 9; consider promoting the verification discipline to a dedicated `.agents/skills/verification-before-completion/SKILL.md` mirroring the superpowers plugin. ## 6. Systematic Debugging