From ccff6cd5e18263ce49b66b87505b290eb214558a Mon Sep 17 00:00:00 2001 From: Ed_ Date: Fri, 19 Jun 2026 01:09:30 -0400 Subject: [PATCH] conductor: register test_sandbox_hardening_20260619 in tracks.md Adds track 16 (priority A) to Active Tracks table: - 5-part fix for test data loss outside ./tests/ - 9-phase TDD plan with 30 tasks - Root cause: src/paths.py:get_config_path() silent fallback via SLOP_CONFIG env var - Per user directive: NO ENV VARS, --config CLI flag, config_overrides.toml naming - Baseline: 1288 + 4 + 0 (no regression allowed per VC8) Co-Authored-By: Claude --- conductor/tracks.md | 35 +++++++++++++++++++++++++++++++++++ 1 file changed, 35 insertions(+) diff --git a/conductor/tracks.md b/conductor/tracks.md index e2e1f3f1..71699403 100644 --- a/conductor/tracks.md +++ b/conductor/tracks.md @@ -34,6 +34,7 @@ Tracks that are unblocked and ready to start. Ordered by **dependency** (blocked | 7b | B | [Continued SQLite-Granularity Inline Docs for gui_2.py](#track-continued-sqlite-granularity-inline-docs-for-gui_2py) | spec ✓, plan ✓, complete | (none — independent) | | 7c | B | [SQLite-Granularity Inline Docs for ai_client.py](#track-sqlite-granularity-inline-docs-for-ai_clientpy) | spec ✓, plan ✓, ready to start | (none — independent) | | 7d | A | [Live GUI Test Infrastructure Fixes](#track-live-gui-test-infrastructure-fixes-new-2026-06-18) | spec ✓, plan ✓, metadata ✓, state ✓, **active**; addresses 2 issues reported for diff tracks by `result_migration_small_files_20260617` Phase 13: (1) `test_execution_sim_live` GUI subprocess (port 8999) crashes mid-test during script generation flow — same failure with both `gemini_cli` and `gemini`; NOT provider-specific; 90s timeout reached without AI text; (2) `test_live_gui_workspace_exists` xdist race — workspace cleanup timing under parallel xdist; passes in isolation. 4 phases: (1) Investigation + Issue 2 parent-commit verification; (2) Fix Issue 2 (TDD); (3) Fix Issue 1 (TDD + remove diagnostic logging); (4) Final verification (11/11 tiers PASS clean). | `result_migration_small_files_20260617` (shipped 2026-06-18 with the 2 issues reported for diff tracks) | (**NEW 2026-06-18**; test-infrastructure track; 2-3 files affected (test + src); TDD for each issue; 11-tier verification required; NO new `@pytest.mark.skip` markers per user directive; out of scope: the 4 Gemini 503 skip markers from sub-track 2 Phase 13 — deferred to a separate follow-up track that mocks the Gemini API in `summarize.summarise_file`) | +| 16 | A | [Test Sandbox Hardening](#track-test-sandbox-hardening-new-2026-06-19) | spec ✓, plan ✓, metadata ✓, state ✓, **ready to start**; 5-part fix for test data loss outside `./tests/`. Phase 1: investigation + baseline pass count + audit of `get_config_path()` callers. Phase 2: `scripts/audit_test_sandbox_violations.py` (FR4 static audit + `--strict` CI gate). Phase 3: `_enforce_test_sandbox` autouse fixture in conftest.py using `sys.addaudithook` (FR1 Python guard; hard fail on any write outside `./tests/`). Phase 4: root-cause fix — remove `SLOP_CONFIG` env-var fallback from `src/paths.py`; add `--config ` CLI flag to sloppy.py + conftest.py; `set_config_override(path)` module-level API (FR2). Phase 5: `isolate_workspace` migration off `tmp_path_factory.mktemp` to `tests/artifacts/_isolation_workspace_/`; pyproject.toml `--basetemp` addopts; `SLOP_CREDENTIALS`/`SLOP_MCP_ENV` env vars added to non-live_gui tests; tech-stack.md dated note (FR3). Phase 6: `scripts/run_tests_sandboxed.ps1` (FR5 Windows restricted-token wrapper, OPT-IN). Phase 7: `conductor/code_styleguides/test_sandbox.md` + updates to workspace_paths.md and guide_testing.md (FR7 docs). Phase 8: full 11-tier verification. Phase 9: end-of-track report. 13 regression tests in `tests/test_test_sandbox.py`. ~11 atomic commits. | (none — independent; **NEW 2026-06-19**; test-infrastructure + root-cause fix; primary motivation: user has lost important sample data multiple times over the past month because tests wrote to top-level TOML files; **NO ENV VARS for config path per user directive** — `--config` CLI flag is the only override mechanism; test workspace file naming: `config_overrides.toml`; hard fail on any sandbox violation; tests should never need AppData temp (`tempfile.mkdtemp/mkstemp` without `dir=` is flagged); baseline 1288 + 4 + 0; **out of scope**: converting the other 7 `SLOP_*` env vars (`SLOP_GLOBAL_PRESETS`, `SLOP_GLOBAL_TOOL_PRESETS`, `SLOP_GLOBAL_PERSONAS`, `SLOP_GLOBAL_WORKSPACE_PROFILES`, `SLOP_CREDENTIALS`, `SLOP_MCP_ENV`, `SLOP_LOGS_DIR`, `SLOP_SCRIPTS_DIR`) to CLI flags — user considers this a separate "mess" to address in follow-up tracks; deferred: macOS/Linux OS-level wrapper, per-fixture sandbox strictness tuning, read-side isolation) | | 8 | — | [Bootstrap gencpp Python Bindings](#track-bootstrap-gencpp-python-bindings) | spec TBD | (none — independent) | | 9 | — | [Tree-Sitter Lua MCP Tools](#track-tree-sitter-lua-mcp-tools) | spec TBD | (none — independent) | | 10 | — | [GDScript Language Support Tools](#track-gdscript-language-support-tools) | spec TBD | (none — independent) | @@ -806,6 +807,40 @@ Lightweight chronology; full spec/plan/state per track is in the linked folder. *Out of scope (deferred to follow-up track): the 4 `@pytest.mark.skip` markers for Gemini 503 pre-existing failures (`test_auto_aggregate_skip`, `test_view_mode_summary`, `test_view_mode_default_summary`, `test_view_mode_custom_empty_default_to_summary`). To remove them, mock the Gemini API in `summarize.summarise_file` for tests.* +#### Track: Test Sandbox Hardening (hard sandbox for tests; root-cause fix for test data loss) `[track-created: 2026-06-19]` +*Link: [./tracks/test_sandbox_hardening_20260619/](./tracks/test_sandbox_hardening_20260619/), Spec: [./tracks/test_sandbox_hardening_20260619/spec.md](./tracks/test_sandbox_hardening_20260619/spec.md), Plan: [./tracks/test_sandbox_hardening_20260619/plan.md](./tracks/test_sandbox_hardening_20260619/plan.md), Metadata: [./tracks/test_sandbox_hardening_20260619/metadata.json](./tracks/test_sandbox_hardening_20260619/metadata.json)* + +*Status: 2026-06-19 - SPEC + PLAN committed. Ready for Tier 2 implementation. 9 phases, 30 tasks, ~11 atomic commits.* + +*Goal: Make any `pytest` or `run_tests_batched.py` invocation provably incapable of writing files outside `./tests/`. Default-on Python guard + opt-in OS-level wrapper. Root-cause fix: eliminate the silent `SLOP_CONFIG` env-var fallback that lets tests accidentally touch the user's real `manual_slop.toml` and related top-level files.* + +*The 5 enforcement layers:* +1. **FR2 root-cause fix** — `src/paths.py:get_config_path()` no longer falls back to `/config.toml` via `SLOP_CONFIG`. New API: `paths.set_config_override(path)`. CLI flag `--config ` at the entry point (sloppy.py for production, conftest.py for tests). +2. **FR1 Python guard** — `sys.addaudithook` autouse fixture blocks writes outside `./tests/` with `RuntimeError("TEST_SANDBOX_VIOLATION: ...")`. Hard fail; reads unaffected. +3. **FR3 isolation migration** — `isolate_workspace` moved off `tmp_path_factory.mktemp` to `tests/artifacts/_isolation_workspace_/`. pyproject.toml adds `addopts = "--basetemp=tests/artifacts/_pytest_tmp"`. All test infra paths now under `./tests/`. +4. **FR4 static audit** — `scripts/audit_test_sandbox_violations.py` flags hardcoded paths to top-level TOMLs + `tempfile.mkdtemp/mkstemp` without `dir=`. CI gate (`--strict` exits 1). +5. **FR5 OS-level wrapper** — `scripts/run_tests_sandboxed.ps1` (Windows restricted-token + Job Object; OPT-IN). + +*User directives (locked 2026-06-19):* +- NO ENV VARS for config path. `--config` CLI flag is the only override mechanism. +- Test workspace file naming: `config_overrides.toml` (per user direction). +- Hard fail on any sandbox violation (no warnings, no soft fails). +- Tests should never need AppData temp. +- Out of scope (deferred to follow-up tracks): converting the other 7 `SLOP_*` env vars (`SLOP_GLOBAL_PRESETS`, `SLOP_GLOBAL_TOOL_PRESETS`, `SLOP_GLOBAL_PERSONAS`, `SLOP_GLOBAL_WORKSPACE_PROFILES`, `SLOP_CREDENTIALS`, `SLOP_MCP_ENV`, `SLOP_LOGS_DIR`, `SLOP_SCRIPTS_DIR`) — user considers this the "mess" to address separately. + +*Baseline (per `result_migration_small_files_20260617` shipped 2026-06-18): 1288 passed + 4 xdist-skipped. VC8 requires no regression vs. this baseline.* + +*Root causes of data loss (per Phase 1 audit):* +1. `src/paths.py:get_config_path()` at line 42 silently falls back to `/config.toml` when `SLOP_CONFIG` is unset (the default for tests). This is the silent default that bites. +2. `tests/conftest.py:isolate_workspace` at line 265 uses `tmp_path_factory.mktemp` which lives in `%TEMP%\pytest-of-\` on Windows — outside `./tests/`. +3. The Layer 1 Python guard is the runtime safety net; FR2 + FR3 are the proper fixes. + +*Deferred follow-up tracks (per metadata.json `deferred_to_followup_tracks`):* +- Convert the other 7 `SLOP_*` env vars to CLI flags (same pattern: `paths.set__override()` + entry-point flag). +- macOS/Linux OS-level sandbox wrapper (`run_tests_sandboxed.sh` using `bwrap`/`unshare`). +- Per-fixture sandbox strictness tuning (`@pytest.fixture(sandbox_strict=True)`). +- Read-side isolation (block reads of real config from tests). + ## Phase 9: Chore Tracks *Initialized: 2026-06-07*