Phase 5B of startup_speedup_20260606 track.
src/theme_2.py had 3 top-level NERV imports:
from src import theme_nerv
from src.theme_nerv import DATA_GREEN
from src.theme_nerv_fx import CRTFilter, AlertPulsing, StatusFlicker
And 3 module-level FX object instantiations:
_crt_filter = CRTFilter()
_alert_pulsing = AlertPulsing()
_status_flicker = StatusFlicker()
ALL removed. The 3 use sites now lookup via _require_warmed:
- apply() NERV branch: theme_nerv = _require_warmed('src.theme_nerv')
- ai_text_color(): theme_nerv = _require_warmed('src.theme_nerv')
(then uses theme_nerv.DATA_GREEN)
- render_post_fx(): theme_nerv_fx = _require_warmed('src.theme_nerv_fx')
(then creates FX objects locally per-call)
The _status_flicker was instantiated but never used (dead code path;
the StatusFlicker class is still importable via theme_nerv_fx but not
auto-constructed in theme_2.py).
TESTS:
- tests/test_theme_2_no_top_level_nerv.py: 4/4 PASS (all RED -> GREEN)
- tests/test_theme.py, test_theme_nerv.py, test_theme_nerv_fx.py,
test_theme_models.py: 21/21 PASS (no breakage)
EFFECTIVENESS: import src.theme_2 no longer triggers src.theme_nerv or
src.theme_nerv_fx (~485ms combined). For users on default theme, these
are NEVER loaded. For NERV users, the warmup pre-loads on _io_pool and
the lookup is O(1).
NEXT: Phase 5C (markdown table) follows same TDD pattern.
Phase 5A T5A.1-T5A.4 of startup_speedup_20260606 track.
src/commands.py was importing src.command_palette at module load to
create the CommandRegistry singleton. The 32 @registry.register
decorators on the command functions needed this registry at import time.
Approach: lazy registry proxy. The @registry.register decorator now
just queues the function in a list; the real CommandRegistry is built
on first access to any other registry attribute (.all, .get, etc.).
By that time, all 32 decorators have run and the pending list is
populated, so the real registration is complete in one pass.
src/commands.py changes:
- Removed 'from src.command_palette import CommandRegistry'
- Added 'from src.module_loader import _require_warmed'
- Added _LazyCommandRegistry class (proxy)
- Added _get_real_registry() function (initializes on first access)
- Replaced 'registry = CommandRegistry()' with 'registry = _LazyCommandRegistry()'
- The 32 @registry.register decorators are unchanged (the proxy's
register method returns the function unchanged after queueing it)
EFFECTIVENESS:
- 'import src.commands' no longer triggers src.command_palette (~244ms)
- The warmup on AppController's _io_pool pre-loads src.command_palette
on a background thread during startup
- First access to registry.all() (e.g. from gui_2.py at palette open
time) is O(1) - the warmup module is already in sys.modules
TESTS:
- tests/test_commands_no_top_level_command_palette.py: 4/4 PASS (3 RED, 1 green; now all green)
- tests/test_command_palette.py: 13/13 PASS (no breakage)
- tests/test_command_palette_sim.py: 7/7 PASS (live_gui tests, the
full palette flow works end-to-end with the lazy proxy)
ARCHITECTURAL NOTE: The lazy proxy is a minimal-change solution that
preserves the public API. The 32 decorated functions don't need any
changes; gui_2.py's 'from src.commands import registry' still works
unchanged. The deferral is invisible to consumers.
NEXT: Phase 5B (NERV theme) and 5C (markdown table) follow the same
TDD pattern. 5D is the bulk refactor of src/gui_2.py feature-gated
imports via the audit_gui2_imports.py script.
Phase 4 T4.1-T4.4 of startup_speedup_20260606 track.
DEVIATION FROM ORIGINAL SPEC: spec.md said fastapi was in src/api_hooks.py
but it was actually in src/app_controller.py (lines 17, 21). api_hooks.py
uses stdlib http.server. Phase 4 target corrected to app_controller.
LIFTED _require_warmed TO SHARED MODULE: created src/module_loader.py to
avoid duplicating the lookup logic and the cross-module import smell
(app_controller -> ai_client). src/ai_client.py re-exports it so the
T3.1 test (which asserts hasattr(src.ai_client, '_require_warmed'))
continues to work.
src/app_controller.py changes:
- Added 'from __future__ import annotations' (enables lazy type annotations;
-> FastAPI return type now a forward reference)
- Removed 'from fastapi import FastAPI, Depends, HTTPException' (line 17)
- Removed 'from fastapi.security.api_key import APIKeyHeader' (line 21)
- Added 'from src.module_loader import _require_warmed' (cross-module via
shared utility, not via ai_client)
- create_api(): added lookups at top of function body
- 7 _api_* helper functions (_api_get_key, _api_generate, _api_stream,
_api_confirm_action, _api_get_session, _api_delete_session,
_api_get_context): added 'HTTPException = _require_warmed(...).HTTPException'
at top of each function body
EFFECTIVENESS:
- import src.app_controller no longer triggers fastapi import (saves ~470ms
in main thread; only loaded when --enable-test-hooks is set)
- When --enable-test-hooks is set, the AppController's warmup pre-loads
fastapi on the _io_pool, so create_api()'s lookup is O(1)
TESTS:
- tests/test_app_controller_no_top_level_fastapi.py: 4/4 PASS (was 3 RED + 1 pass)
- tests/test_ai_client_no_top_level_sdk_imports.py: 9/9 still PASS (re-export works)
- tests/test_app_controller_mcp.py, test_app_controller_offloading.py: pass
- tests/test_headless_service.py: 10/11 PASS (1 pre-existing failure
test_generate_endpoint is a circular-import issue in google.genai,
reproduces identically on stashed pre-Phase-4 state - NOT a regression
from this change)
- tests/test_hooks.py: pass
NEXT: Phase 5 (feature-gated GUI module imports - command palette, NERV
theme, markdown table), then Phase 6 (ad-hoc threads -> _io_pool).
Phase 3 T3.2 + T3.3 of startup_speedup_20260606 track.
The 5 heavy SDKs (anthropic, google.genai, openai, google.genai.types,
requests) are no longer imported at module level. Each function that
needs them now calls _require_warmed(name) to get the module from
sys.modules (populated by AppController's warmup on _io_pool).
This is the load-bearing wall of the Main Thread Purity Invariant:
heavy modules are never in the main thread's import chain.
run_discussion_compression now uses _require_warmed for both
google.genai.types (gemini branch) and requests (deepseek branch).
Tests/test_tier4_patch_generation.py adapted: the 2 tests that
mocked 'src.ai_client.types' (no longer a module-level attr)
now mock 'src.ai_client._require_warmed' (the new public mechanism).
T3.1 tests now pass (9/9). T3.3 breakage fixed.
All 25 ai_client + tier4 tests pass.
Phase 2 Task T2.5 of the startup_speedup_20260606 track.
In AppController.__init__, right after the lock init (and before the
heavy subsystem construction that follows), create the shared _io_pool
and WarmupManager, then submit the warmup list. The warmup runs
concurrently with the rest of __init__, so by the time __init__
returns, the heavy modules are loaded (or in flight).
Changes:
- Add imports: from src.io_pool import make_io_pool,
from src.warmup import WarmupManager
- In __init__, after the locks block, add:
self._io_pool = make_io_pool()
self._warmup = WarmupManager(self._io_pool)
self._warmup.submit(self._compute_warmup_list())
- Add _compute_warmup_list() method: returns ['google.genai',
'anthropic', 'openai', 'requests', 'src.command_palette',
'src.theme_nerv', 'src.theme_nerv_fx', 'src.markdown_table',
'numpy'] always, plus ['fastapi', 'fastapi.security.api_key']
if self.test_hooks_enabled
- Add public delegation methods: warmup_status(), is_warmup_done(),
wait_for_warmup(timeout), on_warmup(callback)
- In shutdown(), add self._io_pool.shutdown(wait=False)
The warmup currently is a no-op for the heavy modules already imported
at the top of app_controller.py (fastapi, requests, etc. are
already in sys.modules). The infrastructure is in place; Phase 3 will
remove the top-level imports so the warmup actually does work.
Verified: all 18 tests pass (test_io_pool + test_warmup + existing
test_app_controller_mcp + test_app_controller_offloading).
Phase 2 Tasks T2.1-T2.4 of the startup_speedup_20260606 track.
NEW: src/io_pool.py
make_io_pool() factory: 4-worker ThreadPoolExecutor with
thread_name_prefix='controller-io'. The sanctioned way for any
background work. Replaces ad-hoc threading.Thread() calls per
the 'no new threads' rule.
NEW: src/warmup.py
WarmupManager: manages a list of modules to import on the shared
pool. Public API:
.submit(modules) - start warmup (call once)
.status() - {pending, completed, failed}
.is_done() - bool
.wait(timeout) - block until done
.on_complete(callback) - register completion callback
.reset() - clear state
Thread-safe (lock-guarded). 10 tests cover all paths.
NEW: tests/test_io_pool.py (4 tests):
- ThreadPoolExecutor returned
- 4 workers
- Threads named 'controller-io-*'
- Jobs run in parallel (barrier test)
NEW: tests/test_warmup.py (10 tests):
- One job per module submitted
- Initial pending list correct
- Failed imports tracked
- Done event set after all complete
- wait() blocks until done
- on_complete callback fires (and immediately if already done)
- Modules actually end up in sys.modules
- reset() clears state
- Jobs run concurrently (not serially)
All 14 tests pass. AppController integration is the next commit.
Lightweight, in-memory profiler for AppController init phases. Used by
the startup_speedup_20260606 track to measure where the time goes
during boot (config hydration, hook server start, subsystem init, etc.).
The profiler is exposed via /api/startup_profile (Phase 8 work) and
the Diagnostics panel so the user can see the exact per-phase cost.
Public API:
StartupProfiler() - create
.phase(name) - context manager
.snapshot() - {phases: {name: {start_ts, duration_ms}}, total_ms, count}
.reset() - clear recorded phases
.enable() / .disable() - toggle recording
Implementation:
- dataclass with list of _Phase(name, start_ts, end_ts)
- @contextmanager records wall-clock via time.perf_counter
- records duration even if the body raises (try/finally)
- snapshot is a copy, so consumers can't mutate the live state
TDD: 5 tests in tests/test_startup_profiler.py cover: basic
recording, total math, snapshot isolation, exception safety, empty
state.
Track.get_executable_tickets (in models.py) called TrackDAG at
runtime, forcing a top-level import of src.dag_engine into models.py
and creating a 2-cycle that broke whichever module loaded second
(Ticket was not yet defined when models.py loaded first; TrackDAG
was not yet defined when dag_engine.py loaded first).
Fix: hoist the method out of the Track dataclass and into a free
function get_executable_tickets(track) in dag_engine.py. models.py
no longer needs TrackDAG at all, so the cycle is one-directional
(models -> dag_engine) and resolves cleanly in any import order.
Tests updated:
- tests/test_mma_models.py: import get_executable_tickets and call
it instead of track.get_executable_tickets() (4 call sites)
- tests/test_conductor_engine_v2.py: comment update
Verified both import orders resolve cleanly:
forward: import src.models; import src.dag_engine -> OK
reverse: import src.dag_engine; import src.models -> OK
34 tests pass (test_mma_models, test_dag_engine, test_execution_engine,
test_arch_boundary_phase3, test_track_state_schema).
When switching projects, the previous implementation ran the entire
save/load/refresh sequence on the main thread. With large project files
or slow disks, this caused the UI to freeze for several seconds.
Fix:
- _switch_project now returns immediately after setting flags; the
actual work runs in a daemon thread (_do_project_switch)
- New is_project_stale() property returns True while a switch is queued
or running; the GUI renders an amber/yellow tint overlay to signal
the controller state lags the user's last click
- AI ops are gated: _api_generate returns HTTP 409, _handle_generate_send
and _handle_md_only early-return with ai_status feedback, all when
is_project_stale() is true
- Queued switches (clicking project A then B in rapid succession) are
coalesced: B replaces A as the target; once A completes, B is
triggered automatically via the finally branch in _do_project_switch
- New state fields: _project_switch_in_progress, _project_switch_pending_path,
_project_switch_thread, _project_switch_lock
- AppController state class attributes use hasattr guard for _app to
keep the controller usable standalone in tests/headless mode
UX:
- Render loop keeps drawing during the switch
- User can still scroll, switch tabs, browse files
- Amber tint + popup explains what's happening and that AI ops are paused
- ai_status shows the target project name
Tests:
- _wait_for_switch helper added for the new async switch flow
- All 7 existing switch tests updated to call _wait_for_switch
- 2 new tests:
- test_switch_project_non_blocking: verifies _switch_project returns
in <0.2s and is_project_stale() is True during the switch
- test_api_generate_blocked_while_stale: verifies _api_generate
raises HTTPException(409) while a switch is in progress
All 33 related tests pass.
When switching projects, the previous project's context_files remained
visible in the Context Composition panel because the controller's
self.context_files list was not reloaded from the new project's TOML
files.paths entry.
Fix in _refresh_from_project:
- After loading self.files from the project TOML, populate
self.context_files with deep copies of those FileItem objects
- Reset self._app.ui_selected_context_files to match the new project's
auto_aggregate set
- Guard the _app access with hasattr so the controller is usable
standalone (in tests, headless mode, etc.) without an attached App
Test: 1 new test in tests/test_project_switch_persona_preset.py
- test_switch_project_resets_context_files: switches from project_a
(forth + gte_hello files) to project_b (gencpp timing files) and
asserts context_files contains ONLY project_b's files
Two fixes for the regression introduced in b92daef3 (and an additional
hardening for the persona->context_preset stale-reference class of bug):
1. Regression: persona_manager was missing on first project load.
_load_active_project creates preset_manager and tool_preset_manager
but did not create persona_manager, so the new
self.personas = self.persona_manager.load_all() line in
_refresh_from_project raised AttributeError on app startup before
the post-_load_active_project persona_manager creation could run.
Fix: create self.persona_manager in _load_active_project alongside
the other managers, so the manager is available when
_refresh_from_project runs.
2. Stale reference: persona's context_preset field pointed to a
preset (e.g. 'GTE') that no longer exists in the project, causing
load_context_preset to raise KeyError and crash the persona
selector panel (which triggered the cascading 'Missing End()' imgui
assertion).
Fix: wrap the load_context_preset call in render_persona_selector_panel
with try/except KeyError, surface the error in app.ai_status, and
clear app.ui_active_context_preset to keep the GUI state consistent.
Tests: 2 new tests in tests/test_project_switch_persona_preset.py
- test_load_active_project_creates_persona_manager (regression guard)
- test_load_context_preset_missing_raises_keyerror (verifies the
contract that load_context_preset raises for missing names; the
GUI layer is now responsible for catching the error)
When switching projects, the previous project's project-specific persona and
presets remained selected in the AI Settings panel because:
1. self.personas was not reloaded after switching project root
2. self.ui_active_persona / tool_preset / bias_profile / project_preset_name
were not validated against the newly-loaded personas/presets
Fix:
- Reload self.personas from self.persona_manager in _refresh_from_project
- Validate each active selection and reset to None/empty if it does not
exist in the newly-loaded manager dictionaries
- Push the active tool preset and bias profile to ai_client after the swap
- Initialize self.ui_active_bias_profile in class attribute block (was only
set later in __init__, causing AttributeError on direct attribute access)
Tests: 4 new tests in tests/test_project_switch_persona_preset.py verify
the reset behavior for persona, preset, tool preset, and global preset
preservation.
Previously, context (files, screenshots) was always sent with every message,
even on subsequent messages where the AI provider already had the context
from the first message via its history mechanism.
This change:
- Detects if the discussion has any AI responses already
- Only sends md_content (stable_md) on the first message
- Subsequent messages pass empty string for md_content to avoid redundant sending
- Context now properly goes in md_content parameter, not crammed into user_message
The fix is in _api_generate() in src/app_controller.py
ROOT CAUSE: imgui_md (mekhontsev/imgui_md) BLOCK_P does NOT call ImGui::NewLine()
when m_list_stack is non-empty (verified in imgui_md.cpp). So a multi-paragraph
list item like:
- bullet text (long, wraps to 2 lines)
continuation paragraph
renders BOTH paragraphs at the same Y because the second BLOCK_P enters/exits
without advancing the cursor. The continuation crashes into the previous
paragraph's last wrapped line.
FIX: Add MarkdownRenderer._normalize_list_continuations preprocessor that
strips blank lines between a list item and its indented continuation. The
continuation then becomes a lazy continuation of the first paragraph (single
BLOCK_P in imgui_md, proper text wrapping, no overlap). Trade-off: users
cannot have separate paragraphs within a single list item. Acceptable.
Also: fixed a pre-existing bug in _normalize_nested_list_endings where a
duplicate conditional caused the function to return empty string (the
out.append(line) was inside the wrong scope). It was silently corrupting
all list content since fd5f4d0e.
TESTS: 23/23 markdown unit tests pass. 3 new tests for the new preprocessor
covering: blank-strip case, blank-preservation case, simple-list passthrough.
FIX 1 (src/markdown_table.py): Cells now use imgui_md.render(c) instead of
imgui.text_wrapped(c). imgui_md uses MD4C which strips backtick-delimited
inline code spans BEFORE rendering, so backticks no longer appear as
literal characters in cell content. Side benefit: inline emphasis
(*foo*, **bar**) now renders in cells too.
FIX 2 (src/markdown_helper.py): Added MarkdownRenderer._normalize_nested_list_endings.
Upstream imgui_md (mekhontsev/imgui_md) BLOCK_UL exit only calls
ImGui::NewLine() for top-level list endings. For nested list endings, no
NewLine is emitted, so the next text starts at the same Y as the last
list item, causing visual overlap. The preprocessor inserts a blank
line before any line that follows a list item with MORE indent than
itself, forcing a paragraph break. Cannot fix the C++ from Python.
Tests:
- test_markdown_table_wrapped.py: updated to assert imgui_md.render is
called for cell content (not imgui.text_wrapped).
- test_markdown_helper_bullets.py: added 4 tests for the new preprocessors
(nested-list blank insertion + bullet delimiter conversion + edge cases).
20/20 markdown unit tests pass. 1-space indentation throughout.
KNOWN LIMITATIONS (cannot fix without forking imgui_md C++):
- Inline code spans render as plain text (no monospace font in cells)
- The ' * ' bullet delimiter has a Y-overlap bug upstream
(workaround: pre-convert to '- ' via _normalize_bullet_delimiters)
- Nested list ending overlap (workaround: insert blank line via
_normalize_nested_list_endings)
Table fix (src/markdown_table.py):
- Add TableColumnFlags_.width_stretch to each table_setup_column call
(was missing — columns had no width to wrap against, so text_wrapped
couldn't grow row height → all rows squished together)
- Remove the explicit for-h-in-headers: table_next_column + text_wrapped(h)
loop. table_headers_row() already renders the header from the
table_setup_column() names; the explicit loop was drawing it AGAIN on
top → double-rendered header rows.
Bullet fix (src/markdown_helper.py):
- Revert _render_md_no_bullet_overlap → simple imgui_md.render(chunk);
imgui.spacing() (the original af0bbe97 approach). The complex
workaround was stripping '- ' and rendering stripped text to imgui_md,
which double-rendered '- 1. ...' content (imgui.bullet from my code +
numbered list marker from imgui_md).
- Add MarkdownRenderer._normalize_bullet_delimiters: regex-converts
'* ' markers to '- ' before passing to imgui_md. This works around
the upstream bug in mekhontsev/imgui_md BLOCK_LI where the '*' case
calls ImGui::Bullet() without ImGui::SameLine(), causing the bullet
to render on its own Y with the text on the next Y. The '-' case
uses Text+SameLine which is correct. Cannot fix from Python (we
can't subclass the C++ class) — pre-conversion is the cheapest fix.
Tests:
- test_markdown_table_wrapped.py: updated to assert new behavior
(text_wrapped count == cell count, not header+cell).
- test_markdown_table_columns.py: updated to assert exactly 6
table_next_column calls (cells only, not 9).
- test_markdown_helper_bullets.py: rewrote for new public-API behavior
(imgui_md.render called with the unstripped chunk).
16/16 markdown unit tests pass.
User reported that nested list items in the Discussion Hub's read_mode
entries were overlapping with adjacent text (e.g., '- gte_lw(...)'
overlapping with 'These are different things...' below it). This is
the imgui_md library's known issue with list item line height.
FIX: Add an imgui.spacing() call after each imgui_md.render() to force
a small vertical gap between chunks. This prevents adjacent list items
from rendering at overlapping Y positions.
Tests: 16/16 broad regression pass
ROOT CAUSE: src/markdown_table.py:render_table was missing
imgui.table_setup_column() calls. In ImGui, columns MUST be
configured via table_setup_column before table_headers_row is called.
Without it, the table has no defined columns, causing cells to
render at overlapping Y positions. This manifested as text overlap
in the Discussion Hub's read_mode entries (e.g., 'swc2 -> gte_sw'
overlapping the line above it).
FIX: Call imgui.table_setup_column(h, TableColumnFlags_.width_stretch)
for each header BEFORE table_headers_row(). Each column now has a
defined width (stretch = fills available space) and cells render
correctly without overlap.
Tests:
- New test_markdown_table_columns.py asserts setup_column is called
once per column and table_next_column is called for each cell.
- 16/16 broad regression pass (test_markdown_table,
test_markdown_table_render, test_markdown_render_robust,
test_gen_send_empty_context, test_gui_fast_render)
ed
r00tz
Conductor