Private
Public Access
0
0

docs(testing): reframe live_gui gotcha as test-authoring contract, not fixture bug

This commit is contained in:
2026-06-05 18:39:33 -04:00
parent 71b0082bbf
commit 0fec0f4f56
+32 -6
View File
@@ -581,14 +581,40 @@ The `live_gui` session fixture runs once at the start of the test session and te
## Known Gotchas (2026-06-05)
### Live_gui Non-Determinism
### Authoring Robust `live_gui` Tests (Don't Assume Clean State)
`live_gui` is a **session-scoped** fixture. All tests in a session share the same `sloppy.py` subprocess. As a result:
`live_gui` is a **session-scoped** fixture. All tests in a session share the same `sloppy.py` subprocess. The subprocess is **not** restarted between tests; its internal state (Fonts, DisplaySize, internal caches, current theme, current workspace profile, current discussion, current MMA track) **accumulates** from the previous test.
- A test that triggers deep render paths early in the session "warms up" ImGui state (Fonts, DisplaySize, internal caches) for later tests. The same test run in a different order or in isolation may fail with a crash that wouldn't happen in the full run.
- `time.sleep(1)` between test operations is **not** enough for ImGui to stabilize in the first few render frames. Reproducers that need stable render state should sleep 3+ seconds before triggering the call.
- Bisect by running the failing test both **with** and **without** the rest of the suite to distinguish "needs warmup" from "real bug".
- When writing a new live_gui test, do NOT assume the first render frame is "ready". Use a `wait_for_event` or multi-poll pattern that tolerates the first few frames being broken.
**This is a test-authoring contract, not a fixture bug.** A test that "passes when run after test X" but "fails when run in isolation" is a fragile test. Robust `live_gui` tests must:
1. **Not assume clean state.** Before invoking an operation, explicitly verify the precondition via the Hook API (e.g. `client.get_value("show_my_window")`, `client.get_mma_status()`, `client.get_session()`). Do not assume a previous test set the state.
2. **Use the wait-for-ready pattern, not fixed sleeps.** `time.sleep(1)` is **not** enough for ImGui to stabilize in the first few render frames (use 3+ seconds, but better: use `wait_for_event` with a generous timeout, or poll `client.get_status()` until ImGui reports `ready`). Fixed sleeps are a code smell; if you reach for one, the right answer is almost always "poll a gettable field instead".
3. **Reset state explicitly if the test depends on it.** For tests that mutate state (e.g. "click button X"), reset the relevant state via Hook API in a `try/finally` so the next test starts from a known baseline. Alternatively, use a function-scoped helper that issues a `reset_session` callback before the test body.
4. **Test both in the full suite AND in isolation before merging.** If a test passes in the full suite but fails in isolation, the test is fragile — fix the test, don't add a "warmup" comment. Bisecting by `pytest path::test -k "filter"` or `pytest --collect-only --quiet` helps.
5. **Use `get_value`/`wait_for_event` to assert ready, not just to assert success.** Example:
```python
def test_open_settings_modal(live_gui):
client.push_event("custom_callback", {"callback": "_toggle_settings", "args": []})
# Wait for the modal to actually appear, not just for the click to dispatch
assert client.get_value("show_settings_modal"), "settings modal did not open"
```
The `get_value` poll doubles as a wait-for-ready AND a correctness assertion.
**Anti-pattern (fragile):**
```python
def test_open_settings_modal(live_gui):
client.push_event("custom_callback", {"callback": "_toggle_settings", "args": []})
time.sleep(1) # hope the modal opened
assert some_cached_value["settings_open"] is True # may be stale from a prior test
```
**Pattern (robust):**
```python
def test_open_settings_modal(live_gui):
client.reset_session() # function-scoped helper; Hook API reset callback
client.push_event("custom_callback", {"callback": "_toggle_settings", "args": []})
assert client.get_value("show_settings_modal"), "settings modal did not open"
```
### Early-Render C-Level Crashes (Defer-Not-Catch Pattern)