fix(tests): shorter smart watchdog timeouts + 90s unconditional sledgehammer
The smart watchdog's 120s pytest-hung + 30s grace = 150s total wait was too long. The user's run hung past that point in interpreter shutdown (ThreadPoolExecutor.__del__ or live_gui teardown). Two changes:
1. SHORTENED the smart watchdog:
- pytest-hung: 120s -> 60s
- shutdown-grace: 30s -> 15s
- Total: 75s (was 150s)
2. ADDED an unconditional 90s sledgehammer watchdog. This one does
NOT wait for pytest_unconfigure. It just sleeps 90s from conftest
load and fires os._exit(2). This handles the case where pytest is
hung BEFORE pytest_unconfigure is reached (e.g., conftest's own
wait_for_warmup hangs, or pytest never reaches its unconfigure).
So the new contract is:
- Normal batch: pytest_unconfigure sets event at ~32s, smart
watchdog's first wait returns immediately, 15s grace elapses,
watchdog exits with 0 (normal exit). Unconditional never fires
(90s would only fire if smart failed).
- Hung batch: pytest_unconfigure never fires, unconditional
watchdog fires at 90s with os._exit(2). Runner catches via
CalledProcessError, reports failure.
- Hung shutdown: pytest_unconfigure fires at ~32s, 15s grace
elapses, smart watchdog fires at 60s with os._exit(2).
The 90s unconditional + 60s smart + 15s grace = the smart watchdog
fires first (at 60s) if pytest is done; the unconditional fires
later (at 90s) if pytest is hung earlier. Net max hang: 90s.
Added test_conftest_smart_watchdog.py test for the new thread.
This commit is contained in:
+19
-2
@@ -99,13 +99,30 @@ def pytest_unconfigure(config: object) -> None:
|
|||||||
|
|
||||||
def _smart_watchdog_exit() -> None:
|
def _smart_watchdog_exit() -> None:
|
||||||
import time
|
import time
|
||||||
if not _pytest_finished_event.wait(timeout=120.0):
|
if not _pytest_finished_event.wait(timeout=60.0):
|
||||||
os._exit(2)
|
os._exit(2)
|
||||||
if not _pytest_finished_event.wait(timeout=30.0):
|
if not _pytest_finished_event.wait(timeout=15.0):
|
||||||
os._exit(2)
|
os._exit(2)
|
||||||
|
|
||||||
threading.Thread(target=_smart_watchdog_exit, daemon=True, name="conftest-smart-watchdog").start()
|
threading.Thread(target=_smart_watchdog_exit, daemon=True, name="conftest-smart-watchdog").start()
|
||||||
|
|
||||||
|
def _unconditional_watchdog_exit() -> None:
|
||||||
|
"""Hard fail-safe: fires regardless of pytest state after 90s total.
|
||||||
|
|
||||||
|
The smart watchdog (above) is gated on pytest_unconfigure setting
|
||||||
|
_pytest_finished_event. If something is hung BEFORE pytest
|
||||||
|
unconfigure runs (e.g., the conftest's own _warmup_app_controller
|
||||||
|
hangs in wait_for_warmup during startup, or pytest never reaches
|
||||||
|
its unconfigure phase), the smart watchdog's first wait
|
||||||
|
blocks. This unconditional watchdog is the sledgehammer: 90s
|
||||||
|
from conftest load, fire os._exit(2) regardless.
|
||||||
|
"""
|
||||||
|
import time
|
||||||
|
time.sleep(90.0)
|
||||||
|
os._exit(2)
|
||||||
|
|
||||||
|
threading.Thread(target=_unconditional_watchdog_exit, daemon=True, name="conftest-unconditional-watchdog").start()
|
||||||
|
|
||||||
from src.gui_2 import App
|
from src.gui_2 import App
|
||||||
|
|
||||||
class VerificationLogger:
|
class VerificationLogger:
|
||||||
|
|||||||
Reference in New Issue
Block a user