Private
Public Access
feat(directives): scavenge sweep 4/5 (tracks + commands + styleguides + todos): 18 batch-4 directives + concurrent worker batches
This commit is contained in:
@@ -0,0 +1,10 @@
|
||||
# failure_message_actionable_not_vague
|
||||
|
||||
## v1
|
||||
|
||||
**Why this iteration:** Lifted from `conductor/todos/TODO_test_full_live_workflow.md` §5 (defensive state assertions before wait) + `conductor/todos/TODO_test_full_live_workflow_v2.md` §"Verification" (failure message on real regression is clear and actionable) — the test must fail fast with a clear reason ("click was not dispatched within 5s", "API returned error: file not found") rather than a confusing timeout ("Project failed to activate").
|
||||
**Source:** `conductor/todos/TODO_test_full_live_workflow.md` §5 + `conductor/todos/TODO_test_full_live_workflow_v2.md` §"Verification"
|
||||
|
||||
---
|
||||
|
||||
**Lifted:** 2026-07-03 scavenge sweep batch 4/5: tracks + commands + styleguides + todos
|
||||
@@ -0,0 +1,72 @@
|
||||
# Test failure messages MUST be actionable — surface the failure reason, the expected vs. actual state, and the timeout that was hit
|
||||
|
||||
## What it says
|
||||
|
||||
When a test times out or fails on an asynchronous operation, the failure message MUST:
|
||||
|
||||
1. **Surface the failure reason** if the operation returned one (e.g., "click was not dispatched within 5s", "API returned error: <error-message>", "background thread hung on lock acquisition")
|
||||
2. **Show expected vs. actual state** (e.g., "expected path=foo.toml, got path=bar.toml")
|
||||
3. **State the timeout** (e.g., "did not complete within 30s")
|
||||
4. **Not just say "timeout" or "failed"** — those are useless
|
||||
|
||||
## Why
|
||||
|
||||
A test that fails with `"AssertionError"` or `"TimeoutError"` after 30 seconds of waiting gives the engineer no signal:
|
||||
|
||||
- Was the click never dispatched?
|
||||
- Was the click dispatched but the handler crashed?
|
||||
- Did the handler run but the API returned an error?
|
||||
- Did the API return success but the state didn't update?
|
||||
- Did the state update but the test's poll interval was too long?
|
||||
|
||||
Without an actionable message, the engineer has to re-run the test under a debugger, attach a tracer, or read the controller's logs to find out what happened. Each of these steps adds 10-30 minutes. A clear failure message short-circuits this loop.
|
||||
|
||||
## The pattern (the canonical example from test_full_live_workflow)
|
||||
|
||||
WRONG:
|
||||
|
||||
```python
|
||||
def test_live_project_switch(live_gui):
|
||||
client.click("btn_project_new_automated")
|
||||
# wait for the project to be created
|
||||
time.sleep(2) # blind wait
|
||||
assert client.get_value("active_project_path") == expected_path
|
||||
```
|
||||
|
||||
RIGHT:
|
||||
|
||||
```python
|
||||
def test_live_project_switch(live_gui):
|
||||
client.click("btn_project_new_automated")
|
||||
# Defensive check: file should exist before we wait for activation
|
||||
deadline = time.monotonic() + 5
|
||||
while time.monotonic() < deadline:
|
||||
if temp_project_path.exists():
|
||||
break
|
||||
time.sleep(0.5)
|
||||
else:
|
||||
pytest.fail(
|
||||
f"temp_project.toml not created within 5s of click; "
|
||||
f"path={temp_project_path}"
|
||||
)
|
||||
|
||||
# Now wait for activation with bounded timeout + reason
|
||||
client.wait_for_project_switch(expected_path, timeout=30)
|
||||
# If this raises, the message includes the API's error field
|
||||
```
|
||||
|
||||
## What goes in the failure message
|
||||
|
||||
For every async test, the failure path should include:
|
||||
|
||||
- The action taken (which click, which API call)
|
||||
- The expected outcome (state name + value)
|
||||
- The actual outcome (state name + value, or "no response", or "API error: <error>")
|
||||
- The elapsed time (so the engineer knows if it was a quick failure or a slow timeout)
|
||||
- The relevant path or ID (so the engineer can find the resource in the logs)
|
||||
|
||||
## Cross-refs
|
||||
|
||||
- `conductor/todos/TODO_test_full_live_workflow.md` §5 — defensive state assertions before wait
|
||||
- `conductor/todos/TODO_test_full_live_workflow_v2.md` §"Verification" — clear and actionable failure messages on real regression
|
||||
- `docs/guide_testing.md` §"live_gui Test Fragility (Authoring-Side)" — poll-not-sleep pattern (the underlying mechanism)
|
||||
Reference in New Issue
Block a user