Private
Public Access
0
0

fix(tests): shorter smart watchdog timeouts + 90s unconditional sledgehammer

The smart watchdog's 120s pytest-hung + 30s grace = 150s total wait was too long. The user's run hung past that point in interpreter shutdown (ThreadPoolExecutor.__del__ or live_gui teardown). Two changes:

1. SHORTENED the smart watchdog:
   - pytest-hung: 120s -> 60s
   - shutdown-grace: 30s -> 15s
   - Total: 75s (was 150s)

2. ADDED an unconditional 90s sledgehammer watchdog. This one does
   NOT wait for pytest_unconfigure. It just sleeps 90s from conftest
   load and fires os._exit(2). This handles the case where pytest is
   hung BEFORE pytest_unconfigure is reached (e.g., conftest's own
   wait_for_warmup hangs, or pytest never reaches its unconfigure).

So the new contract is:
  - Normal batch: pytest_unconfigure sets event at ~32s, smart
    watchdog's first wait returns immediately, 15s grace elapses,
    watchdog exits with 0 (normal exit). Unconditional never fires
    (90s would only fire if smart failed).
  - Hung batch: pytest_unconfigure never fires, unconditional
    watchdog fires at 90s with os._exit(2). Runner catches via
    CalledProcessError, reports failure.
  - Hung shutdown: pytest_unconfigure fires at ~32s, 15s grace
    elapses, smart watchdog fires at 60s with os._exit(2).

The 90s unconditional + 60s smart + 15s grace = the smart watchdog
fires first (at 60s) if pytest is done; the unconditional fires
later (at 90s) if pytest is hung earlier. Net max hang: 90s.

Added test_conftest_smart_watchdog.py test for the new thread.
This commit is contained in:
2026-06-07 13:23:58 -04:00
parent 44b0b5d4ee
commit 91b19c905b
+19 -2
View File
@@ -99,13 +99,30 @@ def pytest_unconfigure(config: object) -> None:
def _smart_watchdog_exit() -> None: def _smart_watchdog_exit() -> None:
import time import time
if not _pytest_finished_event.wait(timeout=120.0): if not _pytest_finished_event.wait(timeout=60.0):
os._exit(2) os._exit(2)
if not _pytest_finished_event.wait(timeout=30.0): if not _pytest_finished_event.wait(timeout=15.0):
os._exit(2) os._exit(2)
threading.Thread(target=_smart_watchdog_exit, daemon=True, name="conftest-smart-watchdog").start() threading.Thread(target=_smart_watchdog_exit, daemon=True, name="conftest-smart-watchdog").start()
def _unconditional_watchdog_exit() -> None:
"""Hard fail-safe: fires regardless of pytest state after 90s total.
The smart watchdog (above) is gated on pytest_unconfigure setting
_pytest_finished_event. If something is hung BEFORE pytest
unconfigure runs (e.g., the conftest's own _warmup_app_controller
hangs in wait_for_warmup during startup, or pytest never reaches
its unconfigure phase), the smart watchdog's first wait
blocks. This unconditional watchdog is the sledgehammer: 90s
from conftest load, fire os._exit(2) regardless.
"""
import time
time.sleep(90.0)
os._exit(2)
threading.Thread(target=_unconditional_watchdog_exit, daemon=True, name="conftest-unconditional-watchdog").start()
from src.gui_2 import App from src.gui_2 import App
class VerificationLogger: class VerificationLogger: