docs(workflow): add Skip-Marker Policy section
Per 2026-06-07 user feedback during test_suite cleanup: "if the intent is to annotate a known failure, fine. But that known failure must be addressed with priority." New section between "Per-Task Decision Protocol" and "Documentation Refresh Protocol" makes the policy explicit: - Skip markers are DOCUMENTATION, not avoidance - They're useful for opt-in integration tests, unimplemented features, or feature-flag-gated code - They're NOT useful for pre-existing failures, "I don't understand this" issues, or racy tests the agent doesn't want to debug - When adding a marker, MUST document the underlying issue AND what the fix would be - When the fix is in-session reachable, FIX IT INSTEAD of skipping — limited context is not an excuse Includes a 4-question review checklist before adding a skip. References the existing AGENTS.md "Use skip markers as excuse to AVOID" rule so the two policies don't drift.
This commit is contained in:
@@ -551,6 +551,37 @@ When the implementing agent encounters a decision not covered by the plan:
|
||||
|
||||
---
|
||||
|
||||
## Skip-Marker Policy: Documentation, Not Avoidance
|
||||
|
||||
`@pytest.mark.skip(reason=...)` is **documentation of a known failure**, not a way to avoid fixing the underlying bug. Skip markers are useful for:
|
||||
|
||||
- **Opt-in integration tests** that require external resources (a real API key, a live provider, a specific env var). Use `@pytest.mark.skipif(...)` with an env-var gate so the test runs when the resource is available and skips by default.
|
||||
- **Tests for features that don't exist yet** (planned but not implemented).
|
||||
- **Tests for features behind a feature flag** that's currently off.
|
||||
|
||||
Skip markers are NOT useful for:
|
||||
|
||||
- **Pre-existing failing tests** (a test that "used to pass" or "was supposed to pass but the underlying code regressed"). The underlying code/test should be fixed in-session.
|
||||
- **Tests that the agent doesn't understand** ("I don't know how to fix this, so I'll skip it"). Escalate to a Tier 4 QA agent for analysis, or ask the user.
|
||||
- **Tests with racy assertions that the agent doesn't want to debug** (e.g., a `time.sleep(0.5)` would fix it). Fix the race, don't skip.
|
||||
|
||||
**When you add a skip marker, you MUST also:**
|
||||
1. Document the underlying issue in the `reason=` string (one or two sentences).
|
||||
2. State what the fix would be (file:line or a one-line description).
|
||||
3. Commit the skip with a follow-up note in the commit body that records the underlying issue, so the next agent (or future self after compaction) can find it via `git log --oneline --grep "skip"`.
|
||||
|
||||
**When the underlying issue is fixable in-session, FIX IT INSTEAD of adding a skip marker.** Limited context is not an excuse: the agent may not know whether the fix is "important" or "easy" until it tries. A skip marker that never gets revisited is a silent test-suite rot.
|
||||
|
||||
**Review checklist before adding a skip marker:**
|
||||
- [ ] Is this a known-bad infrastructure issue (env-var gated)? Use `@pytest.mark.skipif` instead.
|
||||
- [ ] Is this a feature not yet implemented? If so, the feature should be a TODO, not a skip.
|
||||
- [ ] Can the test be fixed in < 30 minutes of investigation? If yes, fix it.
|
||||
- [ ] If the fix is too large, is the underlying issue tracked elsewhere (a conductor track, a TODO in the code)?
|
||||
|
||||
Reference: AGENTS.md "Critical Anti-Patterns" section "Use skip markers as excuse to AVOID" (added 2026-06-07).
|
||||
|
||||
---
|
||||
|
||||
## Documentation Refresh Protocol
|
||||
|
||||
Architectural refactor tracks often change the *shape* of modules the existing docs describe. After a track ships, the affected guides may be partly out of date.
|
||||
|
||||
Reference in New Issue
Block a user