Private
Public Access
docs(track): add test suite audit context to test_engine_integration spec
Appends the full audit findings to the spec's new 'Test Suite Audit Context' section: 27 test-engine upgrade candidates (with per-test classification), ~44 tests fine as-is, ~10 new capabilities enabled, the 3-dimension ordering taxonomy proposal (criticality x fixture x subsystem), and the 4-track campaign sequence informed by the audit. Source: docs/reports/test_suite_audit_20260627.md
This commit is contained in:
@@ -184,4 +184,68 @@ The GIL-transfer caveat (documented at the top of `test_engine.pyi`) is handled
|
||||
- **The test engine's interactive UI panel (`show_test_engine_windows`).** Not shown by default. Can be added as a debug toggle in a follow-up.
|
||||
- **Test engine license audit.** Per the stub: "free for individuals, educational, open-source, and small businesses. Paid for larger businesses." This project is personal-use; no audit needed. Flagged for awareness only.
|
||||
- **CI wiring of the test engine.** The `live_gui` fixture already runs in CI via the batched runner. The `--enable-test-engine` flag is additive. No CI config changes needed.
|
||||
- **Touching `src/models.py` or any taxonomy files.** Zero overlap with the running `tier2/post_module_taxonomy_de_cruft_20260627` branch or the `enforcement_gap_closure_20260627` track.
|
||||
- **Touching `src/models.py` or any taxonomy files.** Zero overlap with the running `tier2/post_module_taxonomy_de_cruft_20260627` branch or the `enforcement_gap_closure_20260627` track.
|
||||
|
||||
## Test Suite Audit Context (added 2026-06-27)
|
||||
|
||||
A full audit of the test suite was conducted on 2026-06-27 (`docs/reports/test_suite_audit_20260627.md`). The findings directly inform the test engine campaign's scope and sequencing:
|
||||
|
||||
### Cruft findings (the upgrade surface)
|
||||
|
||||
- **393 test files** total, run by `run_tests_batched.py` with a 2-level sort (fixture class → batch group). No assertion-criticality ordering exists.
|
||||
- **6 skip markers** — 4 of which are the same root cause (Gemini 503 in `summarize.summarise_file`). One track mocking the Gemini API eliminates all 4.
|
||||
- **60 files use `time.sleep`** (38 of them live_gui) — the anti-pattern explicitly banned in `workflow.md`. Each is a latent race condition. The test engine's `wait_for_test_results(timeout)` replaces these.
|
||||
- **~12-14 one-shot phase tests** are cruft (verifying completed phases like `test_phase_3_final_verify.py`, `test_code_path_audit_phase78.py`).
|
||||
- **3 redundant clusters**: history (5 files), theme (6 files), markdown tables (5 files) — likely overlapping coverage.
|
||||
- **The `core` batch is 245 files (62% of the suite)** in a single xdist run — the bottleneck for targeted verification.
|
||||
|
||||
### Test engine upgrade candidates (27 of 58 live_gui tests)
|
||||
|
||||
These tests exercise interactions the Hook API cannot express well (docking, focus, panel visibility, pop-out, keyboard). The test engine's `ctx.dock_into`, `ctx.window_focus`, `ctx.window_resize`, `ctx.key_press`, `ctx.capture_screenshot_window` would upgrade them:
|
||||
|
||||
- **Docking/layout**: `test_workspace_profiles_sim.py`, `test_auto_switch_sim.py`, `test_preset_windows_layout.py`, `test_gui_text_viewer.py`
|
||||
- **Pop-out panels**: `test_task_dag_popout_sim.py`, `test_usage_analytics_popout_sim.py`
|
||||
- **Command palette + keyboard**: `test_command_palette_sim.py`, `test_undo_redo_sim.py`
|
||||
- **MMA UI flows**: `test_mma_step_mode_sim.py`, `test_mma_concurrent_tracks_sim.py`, `test_visual_mma.py`, `test_visual_sim_mma_v2.py`
|
||||
- **Visual regression candidates**: `test_visual_orchestration.py`, `test_visual_sim_gui_ux.py`, `test_live_markdown_render.py`, `test_gui_stress_performance.py`
|
||||
- **Hook API integration**: `test_hooks.py`, `test_reset_session_clears_mma_and_rag.py`, `test_live_workflow.py`, `test_extended_sims.py`
|
||||
- **Other UI interactions**: `test_gui_context_presets.py`, `test_tool_management_layout.py`, `test_selectable_ui.py`, `test_saved_presets_sim.py`, `test_system_prompt_sim.py`, `test_z_negative_flows.py`
|
||||
|
||||
**~44 live_gui tests are fine as-is** (provider tests, API endpoint tests, model/logic tests) — the test engine adds no value for pure-logic tests.
|
||||
|
||||
### New test capabilities enabled ONLY by the test engine
|
||||
|
||||
- Drag-and-drop docking (`ctx.dock_into`)
|
||||
- Window focus order (`ctx.window_focus`)
|
||||
- Window resize (`ctx.window_resize`)
|
||||
- Keyboard shortcuts (`ctx.key_press` — Ctrl+Z, Ctrl+Shift+P, etc.)
|
||||
- Tab close (`ctx.tab_close`)
|
||||
- Screenshot visual regression (`ctx.capture_screenshot_window` + baseline diff)
|
||||
- Tree open/close (`ctx.item_open_all`)
|
||||
- Multi-step input (`ctx.key_chars` + `ctx.key_press(Enter)`)
|
||||
- Item hover + tooltip (`ctx.item_hold`)
|
||||
- Table column resize (`ctx.table_resize_column`)
|
||||
|
||||
### Proposed ordering taxonomy (assertion-criticality-based)
|
||||
|
||||
The audit proposes a 3-dimension sort: **(criticality, fixture_class, subsystem)** with 6 criticality levels:
|
||||
|
||||
| Level | Name | Description | Approx count |
|
||||
|---|---|---|---|
|
||||
| C0 | Smoke | "Does the app start and respond?" | ~3 |
|
||||
| C1 | Structural | "Do core subsystems exist and have the right shape?" | ~45 |
|
||||
| C2 | Behavioral | "Do subsystems work in isolation?" | ~200 |
|
||||
| C3 | Integration | "Do subsystems compose correctly?" | ~50 |
|
||||
| C4 | UI/Visual | "Does the GUI render + respond to user input?" | 27 (test engine candidates) |
|
||||
| C5 | Stress/Perf | "Does it hold under load?" | ~8 |
|
||||
|
||||
The key insight: the current `live_gui` tier (58 tests) is a monolithic batch mixing C0/C3/C4/C5. Splitting by criticality enables fast-fail (C0 runs first; if it fails, skip the rest) + targeted verification (run only C4-ui when testing a GUI change).
|
||||
|
||||
### Recommended campaign sequence (informed by the audit)
|
||||
|
||||
1. **`test_engine_integration_20260627`** (this track) — build the bridge
|
||||
2. **`test_suite_cruft_cleanup_<date>`** (new, not yet initialized) — delete one-shot cruft, fix Gemini 503 skips, consolidate redundant clusters, replace `time.sleep` with poll loops
|
||||
3. **`test_ordering_taxonomy_<date>`** (new, not yet initialized) — add the criticality dimension to the batched runner (`categorizer.py` + `batcher.py` + `test_categories.toml`)
|
||||
4. **`test_engine_migration_<date>`** (Campaign A Track 2) — migrate the 27 high-value live_gui tests to the test engine, re-classifying them as C4-ui in the new ordering
|
||||
|
||||
Full audit at: `docs/reports/test_suite_audit_20260627.md`
|
||||
Reference in New Issue
Block a user