170 lines
8.3 KiB
Markdown
170 lines
8.3 KiB
Markdown
# Test Sandbox Hardening — Hard Rule
|
|
|
|
## TL;DR
|
|
|
|
The Manual Slop test suite runs under a 4-layer sandbox that prevents any pytest invocation from writing files outside `./tests/`. The root-cause fix removes the historical `SLOP_CONFIG` env-var fallback in favor of an explicit `--config` CLI flag. Any test that needs a config file must point at one inside `./tests/artifacts/`.
|
|
|
|
## The 4-Layer Model
|
|
|
|
| Layer | Mechanism | Where | Default-on? |
|
|
|---|---|---|---|
|
|
| Layer 1 | Python runtime file-I/O guard (`sys.addaudithook`) | `tests/conftest.py:_sandbox_audit_hook` | Yes |
|
|
| Layer 2 | `isolate_workspace` autouse + `pyproject.toml --basetemp` | `tests/conftest.py` + `pyproject.toml` | Yes |
|
|
| Layer 3 | OS-level restricted-token PowerShell wrapper | `scripts/run_tests_sandboxed.ps1` | **Opt-in** |
|
|
| Layer 4 | Static audit script (CI gate) | `scripts/audit_test_sandbox_violations.py` | Yes (informational) / opt-in (`--strict`) |
|
|
|
|
Layer 1 + Layer 2 + Layer 4 are file-presence-on = enabled (delete the relevant file to disable). Layer 3 requires explicit invocation.
|
|
|
|
## The `--config` CLI Flag (replaces `SLOP_CONFIG`)
|
|
|
|
The historical `SLOP_CONFIG` env var has been removed from `src/paths.py`. The CLI flag `--config <path>` is now the ONLY supported mechanism for overriding the default `<project_root>/config.toml` location.
|
|
|
|
### sloppy.py
|
|
|
|
```bash
|
|
# Use the default <project_root>/config.toml
|
|
uv run python sloppy.py
|
|
|
|
# Override
|
|
uv run python sloppy.py --config /path/to/your/config.toml
|
|
```
|
|
|
|
`sloppy.py` calls `paths.set_config_override(Path(args.config).resolve())` AFTER `parse_args()` and BEFORE any `from src.gui_2 import App` import. This is the only way to override the config path in production.
|
|
|
|
### tests/conftest.py
|
|
|
|
`tests/conftest.py` parses `sys.argv` for `--config` at MODULE BODY (BEFORE any `src/` import). If `--config` is not passed, conftest auto-defaults to `tests/artifacts/_isolation_workspace_<RUN_ID>/config_overrides.toml` (which lives inside `./tests/artifacts/`, so the Layer 1 guard allows writes to it).
|
|
|
|
```python
|
|
# Module body in tests/conftest.py (BEFORE any src/ import)
|
|
def _parse_config_arg(argv: list[str]) -> Path | None:
|
|
for i in range(1, len(argv)):
|
|
arg = argv[i]
|
|
if arg == "--config" and i + 1 < len(argv):
|
|
return Path(argv[i + 1]).resolve()
|
|
if arg.startswith("--config="):
|
|
return Path(arg.split("=", 1)[1]).resolve()
|
|
return None
|
|
|
|
_config_override_arg = _parse_config_arg(sys.argv)
|
|
if _config_override_arg is None:
|
|
_config_override_arg = _ISOLATION_WORKSPACE / "config_overrides.toml"
|
|
|
|
from src import paths as _paths # noqa: E402
|
|
_paths.set_config_override(_config_override_arg)
|
|
```
|
|
|
|
The fixture also auto-generates a placeholder `config_overrides.toml` (with `ai.provider`, `projects`, `gui.show_windows`) so src/ code that reads the config at startup does not crash.
|
|
|
|
## The `--basetemp` Rule
|
|
|
|
`pyproject.toml` sets `addopts = "--basetemp=tests/artifacts/_pytest_tmp"`. This redirects pytest's `tmp_path` and `tmp_path_factory` fixtures (which default to `%TEMP%\pytest-of-<user>\` on Windows) into `./tests/artifacts/`. This is what allows the Layer 1 allowlist to be a single rule: "anything under `./tests/` is allowed."
|
|
|
|
## Layer 1 Audit Hook Contract
|
|
|
|
`tests/conftest.py:_sandbox_audit_hook` is a `sys.addaudithook` callback. It fires on every `open()` call. Behavior:
|
|
|
|
- **Reads** (mode `r`, `rb`): pass through, no check
|
|
- **Writes** (mode contains `w`, `a`, `x`, `+`): check path
|
|
- **Allowed** if path resolves under:
|
|
- `<project_root>/tests/`
|
|
- Path contains `.pytest_cache`, `__pycache__`, `.coverage`, `.slop_cache`, or `.ruff_cache`
|
|
- Original path string starts with `\\.\` (Windows device namespace) or `/dev/` (Unix device namespace)
|
|
- **Blocked** otherwise: raises `RuntimeError("TEST_SANDBOX_VIOLATION: attempted to write to <path>...")`
|
|
|
|
**How to fix a violation:**
|
|
- Move the write under `<project_root>/tests/` (use `tmp_path`, `tests/artifacts/_<name>/`, etc.)
|
|
- For pytest internal files (cache, log): check if the path is in the allowlist; if not, open an issue to add it
|
|
|
|
## Layer 2 Workspace Convention (`config_overrides.toml`)
|
|
|
|
Tests that need a `config.toml` should use the auto-generated `tests/artifacts/_isolation_workspace_<RUN_ID>/config_overrides.toml`. The naming convention `config_overrides.toml` (instead of `config.toml`) signals that this file is an override for tests, not the production config.
|
|
|
|
Tests CAN pass `--config /some/other/path.toml` explicitly; conftest will honor it. But the default is fine for most cases.
|
|
|
|
## Layer 3 Opt-in OS-Level Wrapper
|
|
|
|
`scripts/run_tests_sandboxed.ps1` is the Windows-only restricted-token + Job Object wrapper for paranoid users. It mirrors `scripts/tier2/run_tier2_sandboxed.ps1`:
|
|
|
|
```bash
|
|
# Dry-run (no actual sandbox; just prints what would happen)
|
|
pwsh -File scripts/run_tests_sandboxed.ps1 -WhatIf
|
|
|
|
# Run the full suite in the sandbox
|
|
pwsh -File scripts/run_tests_sandboxed.ps1
|
|
|
|
# Run a specific test path
|
|
pwsh -File scripts/run_tests_sandboxed.ps1 -TestPath tests/test_paths.py
|
|
|
|
# Override config explicitly
|
|
pwsh -File scripts/run_tests_sandboxed.ps1 -ConfigPath /some/path/config.toml
|
|
```
|
|
|
|
The wrapper:
|
|
1. Acquires a restricted token via .NET DuplicateTokenEx
|
|
2. Sets cwd to `<project_root>`
|
|
3. Invokes `uv run python -m pytest $TestPath --basetemp=tests/artifacts/_pytest_tmp [--config=...]`
|
|
4. Forwards pytest exit code
|
|
|
|
## Layer 4 Static Audit
|
|
|
|
`scripts/audit_test_sandbox_violations.py` scans `tests/test_*.py` for hardcoded paths that would corrupt user files:
|
|
|
|
- `Path("manual_slop.toml")`, `Path("config.toml")`, `Path("credentials.toml")`, `Path("presets.toml")`, etc.
|
|
- `open("manual_slop.toml", "w")` and similar write-mode calls
|
|
- `Path("C:/projects/...")` and `Path("C:\\projects\\...")`
|
|
- `Path("tests/artifacts/...")` literal (violates workspace_paths.md; should use a fixture)
|
|
- `tempfile.mkdtemp()`, `tempfile.mkstemp()` (without `dir=`)
|
|
|
|
Default mode (informational) exits 0 and lists violations. `--strict` mode (CI gate) exits 1 on any violation.
|
|
|
|
```bash
|
|
# Informational
|
|
uv run python scripts/audit_test_sandbox_violations.py
|
|
|
|
# CI gate
|
|
uv run python scripts/audit_test_sandbox_violations.py --strict
|
|
```
|
|
|
|
## Why This Rule Exists
|
|
|
|
The user has lost "important sample data" multiple times over the past month because tests have written to `manual_slop.toml`, `manual_slop_history.toml`, `personas.toml`, `presets.toml`, `tool_presets.toml`, or `credentials.toml` at the top of the repo. The root cause was the silent `SLOP_CONFIG` env-var fallback in `src/paths.py` — any test could set the env var and have `paths.get_config_path()` return a project-root file.
|
|
|
|
This track fixes that and adds defense in depth.
|
|
|
|
## Forbidden Patterns (Hard Bans)
|
|
|
|
### 1. `SLOP_CONFIG` env var
|
|
|
|
Setting `SLOP_CONFIG` no longer affects `paths.get_config_path()`. Use `--config` instead.
|
|
|
|
### 2. `tempfile.mkdtemp()` / `tempfile.mkstemp()` without `dir=`
|
|
|
|
These default to `%TEMP%`, which the Layer 1 guard blocks. Use:
|
|
- `tempfile.mkdtemp(dir="tests/artifacts/")` (explicit under tests)
|
|
- `tmp_path` pytest fixture (resolves under `--basetemp`)
|
|
- `tmp_path_factory.mktemp("name")` (same)
|
|
|
|
### 3. Writing to `<project_root>/*.toml` or `<project_root>/*.ini`
|
|
|
|
The Layer 1 guard raises `TEST_SANDBOX_VIOLATION` on any write to a top-level TOML/INI file. Move the file under `tests/artifacts/`.
|
|
|
|
### 4. `Path(__file__).parent.parent / "config.toml"`
|
|
|
|
This pattern is a `..` traversal to the project root. Flagged by Layer 4 static audit.
|
|
|
|
## Audit Enforcement
|
|
|
|
- **Layer 4** runs as a pre-commit hook + CI gate (`--strict` mode)
|
|
- **Layer 1** fires at pytest runtime; cannot be bypassed without deleting `tests/conftest.py:_sandbox_audit_hook`
|
|
- **Layer 2** is enforced by `pyproject.toml` addopts; cannot be overridden per-invocation
|
|
|
|
## See Also
|
|
|
|
- `conductor/code_styleguides/workspace_paths.md` — the existing test-workspace rule (extended by this track)
|
|
- `conductor/code_styleguides/feature_flags.md` — file-presence = enabled convention
|
|
- `conductor/tech-stack.md` §"pyproject.toml pytest addopts" — dated note explaining `--basetemp`
|
|
- `scripts/audit_no_temp_writes.py` — pattern reference for Layer 4 audit
|
|
- `scripts/tier2/run_tier2_sandboxed.ps1` — pattern reference for Layer 3 wrapper
|
|
- `conductor/tracks/test_sandbox_hardening_20260619/` — this track's spec + plan + state
|
|
- `conductor/tracks/workspace_path_finalize_20260609/` — prior track that established `tests/artifacts/` workspace pattern |