diff --git a/conductor/tracks/test_engine_integration_20260627/spec.md b/conductor/tracks/test_engine_integration_20260627/spec.md index f1e9f508..daf9040a 100644 --- a/conductor/tracks/test_engine_integration_20260627/spec.md +++ b/conductor/tracks/test_engine_integration_20260627/spec.md @@ -184,4 +184,68 @@ The GIL-transfer caveat (documented at the top of `test_engine.pyi`) is handled - **The test engine's interactive UI panel (`show_test_engine_windows`).** Not shown by default. Can be added as a debug toggle in a follow-up. - **Test engine license audit.** Per the stub: "free for individuals, educational, open-source, and small businesses. Paid for larger businesses." This project is personal-use; no audit needed. Flagged for awareness only. - **CI wiring of the test engine.** The `live_gui` fixture already runs in CI via the batched runner. The `--enable-test-engine` flag is additive. No CI config changes needed. -- **Touching `src/models.py` or any taxonomy files.** Zero overlap with the running `tier2/post_module_taxonomy_de_cruft_20260627` branch or the `enforcement_gap_closure_20260627` track. \ No newline at end of file +- **Touching `src/models.py` or any taxonomy files.** Zero overlap with the running `tier2/post_module_taxonomy_de_cruft_20260627` branch or the `enforcement_gap_closure_20260627` track. + +## Test Suite Audit Context (added 2026-06-27) + +A full audit of the test suite was conducted on 2026-06-27 (`docs/reports/test_suite_audit_20260627.md`). The findings directly inform the test engine campaign's scope and sequencing: + +### Cruft findings (the upgrade surface) + +- **393 test files** total, run by `run_tests_batched.py` with a 2-level sort (fixture class → batch group). No assertion-criticality ordering exists. +- **6 skip markers** — 4 of which are the same root cause (Gemini 503 in `summarize.summarise_file`). One track mocking the Gemini API eliminates all 4. +- **60 files use `time.sleep`** (38 of them live_gui) — the anti-pattern explicitly banned in `workflow.md`. Each is a latent race condition. The test engine's `wait_for_test_results(timeout)` replaces these. +- **~12-14 one-shot phase tests** are cruft (verifying completed phases like `test_phase_3_final_verify.py`, `test_code_path_audit_phase78.py`). +- **3 redundant clusters**: history (5 files), theme (6 files), markdown tables (5 files) — likely overlapping coverage. +- **The `core` batch is 245 files (62% of the suite)** in a single xdist run — the bottleneck for targeted verification. + +### Test engine upgrade candidates (27 of 58 live_gui tests) + +These tests exercise interactions the Hook API cannot express well (docking, focus, panel visibility, pop-out, keyboard). The test engine's `ctx.dock_into`, `ctx.window_focus`, `ctx.window_resize`, `ctx.key_press`, `ctx.capture_screenshot_window` would upgrade them: + +- **Docking/layout**: `test_workspace_profiles_sim.py`, `test_auto_switch_sim.py`, `test_preset_windows_layout.py`, `test_gui_text_viewer.py` +- **Pop-out panels**: `test_task_dag_popout_sim.py`, `test_usage_analytics_popout_sim.py` +- **Command palette + keyboard**: `test_command_palette_sim.py`, `test_undo_redo_sim.py` +- **MMA UI flows**: `test_mma_step_mode_sim.py`, `test_mma_concurrent_tracks_sim.py`, `test_visual_mma.py`, `test_visual_sim_mma_v2.py` +- **Visual regression candidates**: `test_visual_orchestration.py`, `test_visual_sim_gui_ux.py`, `test_live_markdown_render.py`, `test_gui_stress_performance.py` +- **Hook API integration**: `test_hooks.py`, `test_reset_session_clears_mma_and_rag.py`, `test_live_workflow.py`, `test_extended_sims.py` +- **Other UI interactions**: `test_gui_context_presets.py`, `test_tool_management_layout.py`, `test_selectable_ui.py`, `test_saved_presets_sim.py`, `test_system_prompt_sim.py`, `test_z_negative_flows.py` + +**~44 live_gui tests are fine as-is** (provider tests, API endpoint tests, model/logic tests) — the test engine adds no value for pure-logic tests. + +### New test capabilities enabled ONLY by the test engine + +- Drag-and-drop docking (`ctx.dock_into`) +- Window focus order (`ctx.window_focus`) +- Window resize (`ctx.window_resize`) +- Keyboard shortcuts (`ctx.key_press` — Ctrl+Z, Ctrl+Shift+P, etc.) +- Tab close (`ctx.tab_close`) +- Screenshot visual regression (`ctx.capture_screenshot_window` + baseline diff) +- Tree open/close (`ctx.item_open_all`) +- Multi-step input (`ctx.key_chars` + `ctx.key_press(Enter)`) +- Item hover + tooltip (`ctx.item_hold`) +- Table column resize (`ctx.table_resize_column`) + +### Proposed ordering taxonomy (assertion-criticality-based) + +The audit proposes a 3-dimension sort: **(criticality, fixture_class, subsystem)** with 6 criticality levels: + +| Level | Name | Description | Approx count | +|---|---|---|---| +| C0 | Smoke | "Does the app start and respond?" | ~3 | +| C1 | Structural | "Do core subsystems exist and have the right shape?" | ~45 | +| C2 | Behavioral | "Do subsystems work in isolation?" | ~200 | +| C3 | Integration | "Do subsystems compose correctly?" | ~50 | +| C4 | UI/Visual | "Does the GUI render + respond to user input?" | 27 (test engine candidates) | +| C5 | Stress/Perf | "Does it hold under load?" | ~8 | + +The key insight: the current `live_gui` tier (58 tests) is a monolithic batch mixing C0/C3/C4/C5. Splitting by criticality enables fast-fail (C0 runs first; if it fails, skip the rest) + targeted verification (run only C4-ui when testing a GUI change). + +### Recommended campaign sequence (informed by the audit) + +1. **`test_engine_integration_20260627`** (this track) — build the bridge +2. **`test_suite_cruft_cleanup_`** (new, not yet initialized) — delete one-shot cruft, fix Gemini 503 skips, consolidate redundant clusters, replace `time.sleep` with poll loops +3. **`test_ordering_taxonomy_`** (new, not yet initialized) — add the criticality dimension to the batched runner (`categorizer.py` + `batcher.py` + `test_categories.toml`) +4. **`test_engine_migration_`** (Campaign A Track 2) — migrate the 27 high-value live_gui tests to the test engine, re-classifying them as C4-ui in the new ordering + +Full audit at: `docs/reports/test_suite_audit_20260627.md` \ No newline at end of file