diff --git a/docs/guide_testing.md b/docs/guide_testing.md index 3abfeff6..fc59b465 100644 --- a/docs/guide_testing.md +++ b/docs/guide_testing.md @@ -581,14 +581,40 @@ The `live_gui` session fixture runs once at the start of the test session and te ## Known Gotchas (2026-06-05) -### Live_gui Non-Determinism +### Authoring Robust `live_gui` Tests (Don't Assume Clean State) -`live_gui` is a **session-scoped** fixture. All tests in a session share the same `sloppy.py` subprocess. As a result: +`live_gui` is a **session-scoped** fixture. All tests in a session share the same `sloppy.py` subprocess. The subprocess is **not** restarted between tests; its internal state (Fonts, DisplaySize, internal caches, current theme, current workspace profile, current discussion, current MMA track) **accumulates** from the previous test. -- A test that triggers deep render paths early in the session "warms up" ImGui state (Fonts, DisplaySize, internal caches) for later tests. The same test run in a different order or in isolation may fail with a crash that wouldn't happen in the full run. -- `time.sleep(1)` between test operations is **not** enough for ImGui to stabilize in the first few render frames. Reproducers that need stable render state should sleep 3+ seconds before triggering the call. -- Bisect by running the failing test both **with** and **without** the rest of the suite to distinguish "needs warmup" from "real bug". -- When writing a new live_gui test, do NOT assume the first render frame is "ready". Use a `wait_for_event` or multi-poll pattern that tolerates the first few frames being broken. +**This is a test-authoring contract, not a fixture bug.** A test that "passes when run after test X" but "fails when run in isolation" is a fragile test. Robust `live_gui` tests must: + +1. **Not assume clean state.** Before invoking an operation, explicitly verify the precondition via the Hook API (e.g. `client.get_value("show_my_window")`, `client.get_mma_status()`, `client.get_session()`). Do not assume a previous test set the state. +2. **Use the wait-for-ready pattern, not fixed sleeps.** `time.sleep(1)` is **not** enough for ImGui to stabilize in the first few render frames (use 3+ seconds, but better: use `wait_for_event` with a generous timeout, or poll `client.get_status()` until ImGui reports `ready`). Fixed sleeps are a code smell; if you reach for one, the right answer is almost always "poll a gettable field instead". +3. **Reset state explicitly if the test depends on it.** For tests that mutate state (e.g. "click button X"), reset the relevant state via Hook API in a `try/finally` so the next test starts from a known baseline. Alternatively, use a function-scoped helper that issues a `reset_session` callback before the test body. +4. **Test both in the full suite AND in isolation before merging.** If a test passes in the full suite but fails in isolation, the test is fragile — fix the test, don't add a "warmup" comment. Bisecting by `pytest path::test -k "filter"` or `pytest --collect-only --quiet` helps. +5. **Use `get_value`/`wait_for_event` to assert ready, not just to assert success.** Example: + ```python + def test_open_settings_modal(live_gui): + client.push_event("custom_callback", {"callback": "_toggle_settings", "args": []}) + # Wait for the modal to actually appear, not just for the click to dispatch + assert client.get_value("show_settings_modal"), "settings modal did not open" + ``` + The `get_value` poll doubles as a wait-for-ready AND a correctness assertion. + +**Anti-pattern (fragile):** +```python +def test_open_settings_modal(live_gui): + client.push_event("custom_callback", {"callback": "_toggle_settings", "args": []}) + time.sleep(1) # hope the modal opened + assert some_cached_value["settings_open"] is True # may be stale from a prior test +``` + +**Pattern (robust):** +```python +def test_open_settings_modal(live_gui): + client.reset_session() # function-scoped helper; Hook API reset callback + client.push_event("custom_callback", {"callback": "_toggle_settings", "args": []}) + assert client.get_value("show_settings_modal"), "settings modal did not open" +``` ### Early-Render C-Level Crashes (Defer-Not-Catch Pattern)