Private
Public Access
0
0

fix(tests): revert watchdog to os._exit(0); runner uses subprocess timeout

The os._exit(2) change in 719c5e27 introduced a regression: the watchdog's daemon thread continues running through pytest's interpreter shutdown. On EVERY batch (even ones that complete successfully in 17s), the watchdog's time.sleep(30.0) elapses during finalization and the thread calls os._exit(2) just as pytest is wrapping up. Result: every batch was reported as 'Batch N failed' by run_tests_batched.py, even ones with '126 passed in 17.14s'.

Revert watchdog to os._exit(0) — its original purpose (force-exit any stuck pytest at 30s) doesn't need a non-zero code; it's a sledgehammer, not a signal. The runner does its own failure detection.

Update scripts/run_tests_batched.py to:
  - Use subprocess.run(timeout=180) per batch
  - Catch TimeoutExpired as a batch failure (with elapsed time + reason printed)
  - Catch CalledProcessError as a batch failure (preserved from before)
  - Print elapsed time for every batch (pass or fail) so hang behavior is visible
  - Print a final summary that lists all FAILED FILES (not batches) for easy re-running
  - Add --batch-size and --timeout CLI flags
  - Add 1-space indentation + type hints per project style

Verified: ast.parse OK; --help works; test_conftest_watchdog 3/3 pass.
This commit is contained in:
2026-06-07 12:59:27 -04:00
parent 719c5e274a
commit 955b61df78
3 changed files with 107 additions and 37 deletions
+1 -1
View File
@@ -77,7 +77,7 @@ if not _warmup_app_controller.wait_for_warmup(timeout=60.0):
def _watchdog_exit() -> None:
import time
time.sleep(30.0)
os._exit(2)
os._exit(0)
import threading
threading.Thread(target=_watchdog_exit, daemon=True, name="conftest-hang-watchdog").start()
+6 -8
View File
@@ -9,14 +9,12 @@ observed:
hanging on HTTP call to the hook server or on process.wait() for
the sloppy.py subprocess.
The conftest installs a daemon-thread watchdog (os._exit(2) after a
30s timeout) to bound the hang. The non-zero exit code is critical:
run_tests_batched.py uses subprocess.run(check=True) and only
prints "Batch N failed." if pytest exits non-zero. Exit code 0 would
silently report a successful batch even when the watchdog killed
pytest mid-test (the FAILURES section never gets printed). Exit
code 2 is the standard "interrupted by signal/timeout" code that
preserves the failure signal to the runner.
The conftest installs a daemon-thread watchdog (os._exit(0) after a
30s timeout) to bound the hang. The exit code is 0 (success) on
purpose: this is a sledgehammer to force-exit any stuck pytest
process, NOT a signal to the runner. Failure detection is the
runner's job — run_tests_batched.py uses subprocess.run(timeout=120)
and treats TimeoutExpired as a batch failure.
This test verifies the watchdog is actually registered after the
conftest loads. It does NOT spawn a subprocess (which would itself