1.0 KiB
Test failures must not be re-run more than twice in a single investigation
1. The Deduction Loop (kill it)
Symptom: Run test → fail → read log → form hypothesis → run again → fail differently → add diag → run again → fail again → loop. You end up running the same test 4+ times in one session, each run reading partial log output.
Rule: You are allowed to run a failing test at most 2 times in a single investigation. After the 2nd failure, STOP running the test. Read the relevant source code (get_file_slice or py_get_skeleton), predict the failure mode from the code, and instrument ALL the relevant state in one pass before the next run. If the test still fails after 1 instrumented run, report to the user — do not loop.
Worst case captured upfront. Before running the test, ask: "what is the worst-case information I will need if this fails?" Add the diag for that, then run. The diag lines themselves are wasteful in production — see "No Diagnostic Noise in Production" below.