manual_slop/conductor/tracks/test_engine_integration_20260627/spec.md

Track Specification: ImGui Test Engine Integration (Bridge via API Hooks)

Overview

Integrate the Dear ImGui Test Engine (imgui_bundle.imgui.test_engine) into Manual Slop's test infrastructure to enable high-fidelity simulation of user interactions — docking, window focus, panel visibility, drag-and-drop, keyboard input — that the current Hook API cannot express.

The design principle: the API hooks layer (HookServer on :8999 + ApiHookClient) remains the single communication boundary between the test process (pytest) and the GUI subprocess. The test engine is integrated behind the API hooks, not alongside them. New /api/test_engine/* endpoints bridge the test process to the engine's queue_test / get_result_summary API. The engine's test_func closures run on the engine's scenario thread (GIL-transferred by hello_imgui/immapp); they use ctx.item_click("**/Label"), ctx.dock_into(src, dst, dir), ctx.window_focus(ref) etc. to post simulated input events to the GUI render thread. The existing _pending_gui_tasks queue and the engine's input simulation are two separate event injection paths into the same GUI thread; they compose without conflict.

This is Track 1 of 3 in the test engine campaign. Track 1 = enable the engine + build the bridge + smoke test. Track 2 (follow-up) = migrate docking/focus/panel tests. Track 3 (follow-up) = visual regression via screenshot capture.

Current State Audit (as of master 77b70226)

Already Implemented (DO NOT re-implement)

• imgui_bundle v1.92.5 (pinned in pyproject.toml:7) ships the test engine compiled into the nanobind binary. Verified: from imgui_bundle import imgui; imgui.test_engine.TestEngine is a live class; imgui.test_engine.register_test, imgui.test_engine.queue_test, imgui.test_engine.get_result_summary, imgui.test_engine.TestContext with dock_into, window_focus, item_click, capture_screenshot_window, etc. are all present (verified via dir() enumeration — ~95 TestContext methods + ~35 module-level functions). The .pyi stub at .venv/Lib/site-packages/imgui_bundle/imgui/test_engine.pyi documents the full API.

• hello_imgui.RunnerParams.use_imgui_test_engine: bool = False (.venv/Lib/site-packages/imgui_bundle/hello_imgui.pyi:2969) — the flag that enables the engine. When True, hello_imgui/immapp compiles the engine into the runner and provides the GIL-transfer mechanism for the scenario thread. The engine is already compiled into the wheel (the C++ build flag -DHELLOIMGUI_WITH_TEST_ENGINE=ON was set for the published wheel); the Python-side flag just activates it.

• hello_imgui.get_imgui_test_engine() (.venv/Lib/site-packages/imgui_bundle/hello_imgui.pyi:3355) — returns the live TestEngine instance after use_imgui_test_engine = True. Verified callable.

• RunnerCallbacks.register_tests: VoidFunction (.venv/Lib/site-packages/imgui_bundle/hello_imgui.pyi:1809) — the callback that hello_imgui invokes at startup to let the app register tests via imgui.test_engine.register_test(engine, group, name). The demo at .venv/Lib/site-packages/imgui_bundle/demos_python/demos_immapp/demo_testengine.py shows the full pattern.

• imgui_bundle.imgui.test_engine_checks.CHECK(result: bool) — the assertion primitive that emits pass/fail to the engine's result log with file:line traceback. Verified importable.

• The app already uses hello_imgui.RunnerParams + immapp.run() — the exact integration path the test engine requires:

  • src/gui_2.py:641: self.runner_params = _hi.RunnerParams()
  • src/gui_2.py:684-688: self.runner_params.callbacks.show_gui/show_menus/load_additional_fonts/setup_imgui_style/post_init are set
  • src/gui_2.py:1486: immapp.run(app.runner_params, ...) — the main loop entry point
  • The GIL-transfer mechanism is built into immapp.run when use_imgui_test_engine = True; no additional threading code is needed on the Python side.

• HookServer (src/api_hooks.py:857) — the HTTP server on 127.0.0.1:8999, started when --enable-test-hooks is passed. The do_GET method (line 157) and do_POST method (line 490) use a flat if/elif self.path == "/api/..." dispatch. The server holds self.app (the App instance) and accesses it via _get_app_attr(app, ...) helpers. The _pending_gui_tasks queue (app_controller.py:900) + _pending_gui_tasks_lock (app_controller.py:822) + _process_pending_gui_tasks() (app_controller.py:1844, called per-frame from gui_2.py:1759) is the existing thread-safe command queue from HTTP handler thread → main render thread.

• ApiHookClient (src/api_hook_client.py) — the Python client with retry logic, health-check polling, wait_for_server(timeout), push_event(action, payload), get_value(item), set_value(item, value), click(item), wait_for_event(event_type, timeout), etc. Used by all live_gui tests.

• live_gui fixture (tests/conftest.py:641) — session-scoped; spawns sloppy.py --enable-test-hooks --config=<temp> as a subprocess; polls http://127.0.0.1:8999/status until ready; yields a _LiveGuiHandle with .client (an ApiHookClient), .process, .workspace. The fixture's subprocess args are at conftest.py:792: gui_args = ["uv", "run", "python", "-u", gui_script, "--enable-test-hooks"].

• sloppy.py (79 lines) — the entry point. CLI flags at lines 31-36: --headless, --web-host, --web-port, --enable-test-hooks, --config. The else branch at line 75 (the normal GUI mode) calls from src.gui_2 import main; main().

• AppController.test_hooks_enabled (src/app_controller.py:1042) — set via "--enable-test-hooks" in sys.argv or SLOP_TEST_HOOKS=1 env var. Same pattern works for --enable-test-engine.

Gaps to Fill (This Track's Scope)

• GAP-1: The test engine is not enabled. runner_params.use_imgui_test_engine is never set to True. No callbacks.register_tests callback exists. The engine's scenario thread + GIL-transfer mechanism are dormant.

• GAP-2: No /api/test_engine/* bridge endpoints. The HookServer has no way for the test process to queue a test, poll results, or abort a running test. The test engine API (queue_test, get_result_summary, is_test_queue_empty, abort_current_test) is only accessible from inside the GUI process — not from the HTTP boundary.

• GAP-3: No ApiHookClient methods for test engine operations. The client has click, set_value, push_event, wait_for_event — but no queue_test, wait_for_test_results, get_test_results.

• GAP-4: live_gui fixture doesn't pass --enable-test-engine. The subprocess at conftest.py:792 only passes --enable-test-hooks. Without the engine flag, the engine won't activate even after GAP-1 is fixed.

• GAP-5: No smoke test proving the end-to-end threading model works. The test engine's scenario thread + GIL transfer is the highest-risk piece. A minimal smoke test (register a trivial test that clicks a known button + asserts a state change, queue it via the API, poll for results, assert pass) is needed to prove the bridge works before Track 2 migrates real tests.

Architecture: Why the API hooks + test engine compose

pytest test process
  └── ApiHookClient (HTTP :8999)          ← single communication boundary (KEPT)
       └── HookServer.do_POST             ← new /api/test_engine/* endpoints
            └── imgui.test_engine.queue_test(engine, test)  ← schedules on engine
                 └── TestContext.test_func(ctx)  ← runs on engine scenario thread
                      └── ctx.item_click("**/Label")  ← posts simulated input to GUI thread
                           └── GUI render thread processes the simulated event
                                └── _process_pending_gui_tasks() still runs per-frame
                                     (existing queue; unaffected; two separate injection paths)

The test engine's test_func runs on its own thread (the scenario thread). The ctx.* primitives post simulated input events to the ImGui input queue on the GUI render thread. This is the same destination as real user input and the same destination as _pending_gui_tasks — but a different injection mechanism. The two paths are independent; they don't share state, locks, or queues. The test engine doesn't touch _pending_gui_tasks and vice versa.

The GIL-transfer caveat (documented at the top of test_engine.pyi) is handled by hello_imgui/immapp when use_imgui_test_engine = True — the C++ layer transfers the GIL between the main thread and the scenario thread. No additional Python-side threading code is needed. The test_func callback runs with the GIL held; it can safely call ctx.* primitives (which are C++ nanobind calls that release the GIL during the simulated input wait).

Goals

• G1. sloppy.py accepts --enable-test-engine CLI flag; when set, App.run() sets runner_params.use_imgui_test_engine = True + assigns runner_params.callbacks.register_tests to a method that registers tests.

• G2. App has a _register_imgui_tests(self) method (called by hello_imgui at startup via the register_tests callback) that registers at least one smoke test ("Smoke Tests", "Click Increment Button") via imgui.test_engine.register_test(engine, group, name). The smoke test's test_func(ctx) calls ctx.set_ref("...") + ctx.item_click("**/...") + CHECK(...).

• G3. HookServer (in src/api_hooks.py) has 4 new endpoints:

  • POST /api/test_engine/queue — body {"group": "...", "name": "..."}; finds the test by group+name via imgui.test_engine.find_test_by_name(engine, group, name); calls queue_test(engine, test); responds {"status": "queued"}.
  • GET /api/test_engine/status — calls is_test_queue_empty(engine); responds {"queue_empty": true/false}.
  • GET /api/test_engine/results — calls get_result_summary(engine, out_results); responds {"count_tested": N, "count_success": N, "count_in_queue": N}.
  • POST /api/test_engine/abort — calls abort_current_test(engine); responds {"status": "aborted"}.

• G4. ApiHookClient (in src/api_hook_client.py) has 4 new methods:

  • queue_test(group: str, name: str) -> dict — POST to /api/test_engine/queue.
  • get_test_status() -> dict — GET /api/test_engine/status.
  • get_test_results() -> dict — GET /api/test_engine/results.
  • wait_for_test_results(timeout: float = 30.0) -> dict — polls get_test_status() until queue_empty == True or timeout; then returns get_test_results().

• G5. The live_gui fixture passes --enable-test-engine in addition to --enable-test-hooks in the subprocess args (conftest.py:792). The engine activates on every live_gui test run.

• G6. A smoke test in tests/test_test_engine_smoke.py that:

  1. Uses the live_gui fixture.
  2. Queues the smoke test via client.queue_test("Smoke Tests", "Click Increment Button").
  3. Polls via client.wait_for_test_results(timeout=30).
  4. Asserts results["count_success"] >= 1 and results["count_tested"] >= 1. This proves the full bridge works: pytest → HTTP → HookServer → engine → scenario thread → ctx.item_click → GUI thread → state change → CHECK → result log → get_result_summary → HTTP → pytest assert.

• G7. End-of-track report at docs/reports/TRACK_COMPLETION_test_engine_integration_20260627.md documenting: what shipped, the threading model verification, any GIL-transfer issues encountered, and the handoff to Track 2 (docking test migration).

Functional Requirements

FR1: --enable-test-engine CLI flag

• sloppy.py: add parser.add_argument("--enable-test-engine", action="store_true", help="Enable the Dear ImGui Test Engine for automated UI testing") alongside the existing --enable-test-hooks flag (line 35).
• src/app_controller.py: add self.test_engine_enabled: bool = ("--enable-test-engine" in sys.argv) near line 1042 (same pattern as test_hooks_enabled).
• src/gui_2.py App.run() (line 619): between the RunnerParams() construction (line 641) and the callbacks.show_gui = ... assignments (line 684), add:

  if getattr(self.controller, "test_engine_enabled", False):
   self.runner_params.use_imgui_test_engine = True
   self.runner_params.callbacks.register_tests = self._register_imgui_tests
  

  This is guarded by the flag so normal runs are unaffected.

FR2: App._register_imgui_tests(self) method

• New method on App in src/gui_2.py (near the other callback registrations, ~line 700):

  def _register_imgui_tests(self) -> None:
   """Called by hello_imgui at startup to register ImGui Test Engine tests.
   Reads the live engine via hello_imgui.get_imgui_test_engine().
   [C: src/gui_2.py:App.run (via callbacks.register_tests)]
   """
   from imgui_bundle import hello_imgui
   from imgui_bundle.imgui import test_engine
   engine = hello_imgui.get_imgui_test_engine()
   # Smoke test: click a known button and verify state change
   test = test_engine.register_test(engine, "Smoke Tests", "Click Increment Button")
   def smoke_func(ctx) -> None:
    from imgui_bundle.imgui.test_engine_checks import CHECK
    ctx.set_ref("...")  # TODO: set to a known window
    ctx.item_click("**/...")  # TODO: click a known button
    CHECK(True)  # TODO: verify state change
   test.test_func = smoke_func
  

  The exact button + state to click + verify is determined during implementation by inspecting the running GUI's item tree (use ctx.window_info / imgui.show_id_stack_tool_window to find labels). The smoke test should click something harmless (e.g., a tab switch, a checkbox toggle) and verify the state changed.

FR3: /api/test_engine/* endpoints in HookServer

• In src/api_hooks.py do_POST (line 490): add 2 new elif branches for POST /api/test_engine/queue and POST /api/test_engine/abort.
• In src/api_hooks.py do_GET (line 157): add 2 new elif branches for GET /api/test_engine/status and GET /api/test_engine/results.
• All 4 endpoints guard on test_engine_enabled — if the engine is not active, respond {"error": "test engine not enabled", "enabled": false} with HTTP 503.
• The engine instance is obtained via hello_imgui.get_imgui_test_engine() inside the handler (lazy import; the handler runs on the HTTP thread, but get_imgui_test_engine() is a C++ accessor that returns a pointer — safe to call from any thread).

FR4: ApiHookClient methods

• In src/api_hook_client.py: add 4 methods per G4. Follow the existing method pattern (e.g., get_status, push_event): construct the URL, requests.post/get, retry on connection error, parse JSON, return the dict.

FR5: live_gui fixture update

• In tests/conftest.py:792: change gui_args to include "--enable-test-engine" when the fixture spawns the subprocess. The flag flows through to AppController.test_engine_enabled → App.run() → runner_params.use_imgui_test_engine = True.

FR6: Smoke test

• tests/test_test_engine_smoke.py (NEW) — 2-3 tests:
  • test_engine_enabled: client.get_value("test_engine_enabled") returns True (or verify via a new gettable field).
  • test_queue_and_run_smoke_test: queue the smoke test, poll for results, assert success.
  • test_engine_results_shape: get_test_results() returns the expected dict shape.

Non-Functional Requirements

• 1-space indentation for all Python code.
• No comments in body per AGENTS.md.
• CRLF line endings preserved.
• Atomic per-task commits.
• Thread safety: the test_func runs on the engine scenario thread. It must NOT directly mutate App or AppController state — it must use ctx.* primitives (which post simulated input to the GUI thread). Reading state via hello_imgui.get_imgui_test_engine() or engine queries (ctx.item_info, ctx.window_info) is safe. The CHECK() assertion runs on the scenario thread but only writes to the engine's result log (thread-safe C++ structure).
• No live_gui regression: the --enable-test-engine flag must not affect normal GUI behavior when live_gui tests are NOT using the engine. The engine's scenario thread is idle when no tests are queued. The show_test_engine_windows panel is NOT shown by default (only via explicit call).
• Performance: the engine adds a per-frame overhead when active. The fps_idling settings in runner_params remain unchanged. The engine's overhead is sub-millisecond per frame when no tests are running.

Architecture Reference

• docs/guide_testing.md — the live_gui fixture, the structural testing contract, the Puppeteer pattern.
• docs/guide_api_hooks.md — the Hook API surface, the /api/ask protocol, the ApiHookClient method reference.
• docs/guide_gui_2.md — the App class lifecycle, the runner_params construction, the callbacks system.
• .venv/Lib/site-packages/imgui_bundle/demos_python/demos_immapp/demo_testengine.py — the canonical demo for the test engine integration pattern (register_tests callback + test_func closures + CHECK).
• .venv/Lib/site-packages/imgui_bundle/imgui/test_engine.pyi — the full API stub (2644 lines). Key sections: TestContext methods (lines 1445-2096), module-level functions (lines 433-500, 2639+), TestEngineResultSummary (3 fields: count_tested, count_success, count_in_queue).
• .venv/Lib/site-packages/imgui_bundle/imgui/test_engine_checks.py — the CHECK(result: bool) assertion primitive.
• conductor/workflow.md "Live_gui Test Fragility" + "Async Setters Need Poll-For-State" — the existing patterns for live_gui tests; the test engine's wait_for_test_results replaces time.sleep + get_value polling with a single engine-side poll.

Out of Scope

• Migrating existing live_gui tests to the test engine. That's Track 2 (test_engine_docking_tests_<date>). This track only builds the bridge + proves it works with 1 smoke test.
• Visual regression via screenshot capture. That's Track 3 (test_engine_capture_regression_<date>). The ctx.capture_screenshot_window API is available but not wired in this track.
• Headless test execution (no GUI window). The test engine requires a live GLFW window (the scenario thread drives the actual ImGui render loop). Headless mode is a future research item, not this track.
• The test engine's interactive UI panel (show_test_engine_windows). Not shown by default. Can be added as a debug toggle in a follow-up.
• Test engine license audit. Per the stub: "free for individuals, educational, open-source, and small businesses. Paid for larger businesses." This project is personal-use; no audit needed. Flagged for awareness only.
• CI wiring of the test engine. The live_gui fixture already runs in CI via the batched runner. The --enable-test-engine flag is additive. No CI config changes needed.
• Touching src/models.py or any taxonomy files. Zero overlap with the running tier2/post_module_taxonomy_de_cruft_20260627 branch or the enforcement_gap_closure_20260627 track.

Test Suite Audit Context (added 2026-06-27)

A full audit of the test suite was conducted on 2026-06-27 (docs/reports/test_suite_audit_20260627.md). The findings directly inform the test engine campaign's scope and sequencing:

Cruft findings (the upgrade surface)

• 393 test files total, run by run_tests_batched.py with a 2-level sort (fixture class → batch group). No assertion-criticality ordering exists.
• 6 skip markers — 4 of which are the same root cause (Gemini 503 in summarize.summarise_file). One track mocking the Gemini API eliminates all 4.
• 60 files use time.sleep (38 of them live_gui) — the anti-pattern explicitly banned in workflow.md. Each is a latent race condition. The test engine's wait_for_test_results(timeout) replaces these.
• ~12-14 one-shot phase tests are cruft (verifying completed phases like test_phase_3_final_verify.py, test_code_path_audit_phase78.py).
• 3 redundant clusters: history (5 files), theme (6 files), markdown tables (5 files) — likely overlapping coverage.
• The core batch is 245 files (62% of the suite) in a single xdist run — the bottleneck for targeted verification.

Test engine upgrade candidates (27 of 58 live_gui tests)

These tests exercise interactions the Hook API cannot express well (docking, focus, panel visibility, pop-out, keyboard). The test engine's ctx.dock_into, ctx.window_focus, ctx.window_resize, ctx.key_press, ctx.capture_screenshot_window would upgrade them:

• Docking/layout: test_workspace_profiles_sim.py, test_auto_switch_sim.py, test_preset_windows_layout.py, test_gui_text_viewer.py
• Pop-out panels: test_task_dag_popout_sim.py, test_usage_analytics_popout_sim.py
• Command palette + keyboard: test_command_palette_sim.py, test_undo_redo_sim.py
• MMA UI flows: test_mma_step_mode_sim.py, test_mma_concurrent_tracks_sim.py, test_visual_mma.py, test_visual_sim_mma_v2.py
• Visual regression candidates: test_visual_orchestration.py, test_visual_sim_gui_ux.py, test_live_markdown_render.py, test_gui_stress_performance.py
• Hook API integration: test_hooks.py, test_reset_session_clears_mma_and_rag.py, test_live_workflow.py, test_extended_sims.py
• Other UI interactions: test_gui_context_presets.py, test_tool_management_layout.py, test_selectable_ui.py, test_saved_presets_sim.py, test_system_prompt_sim.py, test_z_negative_flows.py

~44 live_gui tests are fine as-is (provider tests, API endpoint tests, model/logic tests) — the test engine adds no value for pure-logic tests.

New test capabilities enabled ONLY by the test engine

• Drag-and-drop docking (ctx.dock_into)
• Window focus order (ctx.window_focus)
• Window resize (ctx.window_resize)
• Keyboard shortcuts (ctx.key_press — Ctrl+Z, Ctrl+Shift+P, etc.)
• Tab close (ctx.tab_close)
• Screenshot visual regression (ctx.capture_screenshot_window + baseline diff)
• Tree open/close (ctx.item_open_all)
• Multi-step input (ctx.key_chars + ctx.key_press(Enter))
• Item hover + tooltip (ctx.item_hold)
• Table column resize (ctx.table_resize_column)

Proposed ordering taxonomy (assertion-criticality-based)

The audit proposes a 3-dimension sort: (criticality, fixture_class, subsystem) with 6 criticality levels:

Level	Name	Description	Approx count
C0	Smoke	"Does the app start and respond?"	~3
C1	Structural	"Do core subsystems exist and have the right shape?"	~45
C2	Behavioral	"Do subsystems work in isolation?"	~200
C3	Integration	"Do subsystems compose correctly?"	~50
C4	UI/Visual	"Does the GUI render + respond to user input?"	27 (test engine candidates)
C5	Stress/Perf	"Does it hold under load?"	~8

The key insight: the current live_gui tier (58 tests) is a monolithic batch mixing C0/C3/C4/C5. Splitting by criticality enables fast-fail (C0 runs first; if it fails, skip the rest) + targeted verification (run only C4-ui when testing a GUI change).

Recommended campaign sequence (informed by the audit)

1. test_engine_integration_20260627 (this track) — build the bridge
2. test_suite_cruft_cleanup_<date> (new, not yet initialized) — delete one-shot cruft, fix Gemini 503 skips, consolidate redundant clusters, replace time.sleep with poll loops
3. test_ordering_taxonomy_<date> (new, not yet initialized) — add the criticality dimension to the batched runner (categorizer.py + batcher.py + test_categories.toml)
4. test_engine_migration_<date> (Campaign A Track 2) — migrate the 27 high-value live_gui tests to the test engine, re-classifying them as C4-ui in the new ordering

Full audit at: docs/reports/test_suite_audit_20260627.md