ed/manual_slop
Private
Public Access
0
0
Fork 0
Code Activity

docs(sandbox): add test_sandbox.md styleguide + workspace_paths + guide_testing updates

Browse Source
This commit is contained in:
ed
2026-06-19 07:53:49 -04:00
parent 66c6421bbc
commit 5d29e40fe2
3 changed files with 199 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/guide_testing.md
+28 -5
View File
@@ -53,12 +53,16 @@ The `tests/conftest.py` file defines 7 fixtures. They are listed below in the or
**Purpose**: Give every test a fresh, isolated workspace so it cannot pollute the user's real `manual_slop.toml`, `presets.toml`, etc.
**Mechanism**:
1. Creates a temp directory via `tmp_path_factory.mktemp("isolated_workspace")`
2. Writes a fresh `config.toml` to the temp dir
3. Sets `SLOP_CONFIG`, `SLOP_GLOBAL_PRESETS`, `SLOP_GLOBAL_TOOL_PRESETS`, `SLOP_GLOBAL_PERSONAS`, `SLOP_GLOBAL_WORKSPACE_PROFILES` env vars to point at the temp dir
4. The app reads these env vars on startup; the test sees an isolated world
1. Uses the module-level `_ISOLATION_WORKSPACE = Path(f"tests/artifacts/_isolation_workspace_{_RUN_ID}")` (created at conftest import time)
2. Writes a fresh `config_overrides.toml` to the workspace if it doesn't exist
3. Touches placeholder `presets.toml`, `tool_presets.toml`, `personas.toml`, `workspace_profiles.toml`, `credentials.toml`, `mcp_env.toml`
4. Sets `SLOP_GLOBAL_PRESETS`, `SLOP_GLOBAL_TOOL_PRESETS`, `SLOP_GLOBAL_PERSONAS`, `SLOP_GLOBAL_WORKSPACE_PROFILES`, `SLOP_CREDENTIALS`, `SLOP_MCP_ENV` env vars to point at the workspace files
5. The actual `config.toml` path comes from conftest module body (`_parse_config_arg` parses `--config` from `sys.argv`; auto-defaults to `_ISOLATION_WORKSPACE / "config_overrides.toml"` and calls `paths.set_config_override(...)` BEFORE any `src/` import)
6. The app reads these paths on startup; the test sees an isolated world
**Verification**: `python scripts/check_test_toml_paths.py` exits 0 (no test references real TOMLs).
**Migration history**: As of `test_sandbox_hardening_20260619` (2026-06-19), this fixture no longer uses `tmp_path_factory.mktemp` (which lives in `%TEMP%`) and no longer sets `SLOP_CONFIG` (which is now an unsupported env var). See [Sandbox Hardening](#sandbox-hardening-added-2026-06-19) below.
**Verification**: `python scripts/check_test_toml_paths.py` and `python scripts/audit_test_sandbox_violations.py` both exit 0.
#### `reset_paths` (line 95)
@@ -161,6 +165,25 @@ def test_my_thing(live_gui):
---
## Sandbox Hardening (added 2026-06-19)
Added in `test_sandbox_hardening_20260619` track. The test suite runs under a 4-layer sandbox that prevents any pytest invocation from writing files outside `./tests/`. The user has lost "important sample data" multiple times because tests have silently written to top-level `manual_slop.toml`, `config.toml`, `presets.toml`, etc.
**The 4 layers:**
| Layer | Mechanism | Default-on? |
|---|---|---|
| 1. Python runtime guard | `sys.addaudithook` in `tests/conftest.py:_sandbox_audit_hook` raises `RuntimeError("TEST_SANDBOX_VIOLATION")` on writes outside `./tests/` | Yes |
| 2. Workspace migration | `pyproject.toml --basetemp=tests/artifacts/_pytest_tmp` + `isolate_workspace` uses `_ISOLATION_WORKSPACE` under `tests/artifacts/` (no more `tmp_path_factory.mktemp`) | Yes |
| 3. OS-level wrapper | `scripts/run_tests_sandboxed.ps1` (Windows restricted-token + Job Object) | **Opt-in** |
| 4. Static audit | `scripts/audit_test_sandbox_violations.py` flags hardcoded paths + `tempfile.mkdtemp()` without `dir=` | Yes (informational) / opt-in (`--strict`) |
**Root-cause fix**: `SLOP_CONFIG` env var is no longer consulted by `src/paths.py:get_config_path()`. The CLI flag `--config <path>` is the ONLY supported mechanism for overriding the default `<project_root>/config.toml`. `sloppy.py` accepts `--config`. `tests/conftest.py` parses sys.argv at module body (BEFORE any src/ import) and auto-defaults to `tests/artifacts/_isolation_workspace_<RUN_ID>/config_overrides.toml`.
**For full details see**: `conductor/code_styleguides/test_sandbox.md`. Regression tests in `tests/test_test_sandbox.py` cover all 4 layers.
---
## Per-test Subprocess Resilience (2026-06-09)
Added in `test_infrastructure_hardening_20260609` track. These three mechanisms address the "subprocess state pollution" and "controller state pollution" failure modes that caused batch regressions.