docs: agent workflow docs + regular docs (v2.3 surfacing)
Per user request 'use your remaining context to update agent workflow
docs and then regular docs based on what was discussed in this report',
this commit creates/updates 15 files derived from the v2.3 nagent
review (the 12 new nagent additions + the 4 memory dimensions
reframing + the cache strategy + the RAG discipline + the knowledge
harvest pattern).
Agent workflow docs (4 files):
- AGENTS.md (UPDATE): add @import line to canonical DOD + 'Code
Styleguides' section pointing to the 6 new styleguides + new
'Human-Facing Documentation' section pointing to ./docs/AGENTS.md
- conductor/workflow.md (UPDATE): new section 'Additions (2026-06-12)
- the 12 patterns from the latest nagent corpus' with TDD
protocols for knowledge harvest, cache ordering, compaction, RAG
discipline
- conductor/product-guidelines.md (UPDATE): new sections 'Memory
Dimensions (added 2026-06-12)' + 'See Also - Updated' with the
6-styleguide catalog
- docs/AGENTS.md (NEW): the agent-facing mirror of docs/Readme.md
(per the nagent CLAUDE.md pattern). 10 sections + the per-tier
reading path + the 4 memory dimensions + the caching strategy +
the knowledge harvest + the RAG discipline + the feature flags
Regular docs (11 files):
- 6 new styleguides (the convention catalog):
* data_oriented_design.md: the canonical DOD reference (Tier
0/1/2; 3 defaults to reject; 8 core defaults; 7-question
simplification pass; 10-question self-check; 4 memory
dimensions in Manual Slop context)
* agent_memory_dimensions.md: the 4 memory dims (curation /
discussion / RAG / knowledge) + when to use each + the
boundaries
* rag_integration_discipline.md: the conservative-RAG rule
(opt-in, complement, provenance, no mutation, feature-gated,
graceful failure)
* cache_friendly_context.md: stable-to-volatile context
ordering + the cache TTL GUI contract + the byte-comparison
test
* knowledge_artifacts.md: the knowledge harvest pattern
(category files, provenance, sha256 ledger, digest
regeneration, 'delete to turn off')
* feature_flags.md: file presence vs config flags vs CLI flags
- 3 new project docs (the cross-cutting guides):
* guide_agent_memory_dimensions.md: the cross-cutting guide on
the 4 dims + the decision tree
* guide_caching_strategy.md: caching across providers +
stable-to-volatile ordering + cache TTL GUI + the byte-
comparison test + the 5th provider (claude-code)
* guide_knowledge_curation.md: the knowledge memory guide (4th
dim) + the 5 category files + per-file notes + the digest +
the ledger + the harvest workflow
- 2 existing doc updates:
* guide_mma.md: new sections 'Delegation as context management'
+ 'The 4 memory dimensions (the MMA scope)'
* guide_ai_client.md: new section 'Cache strategy and the 12-
layer model' + the 5th provider (claude-code)
All files use the same style as the v2.3 review (the user's preferred
format): 7-column tables, no JSON, SSDL shape tags, forth/array
notation, file:line citations, ASCII sketches where useful. The
human Readme files (Readme.md, docs/Readme.md) are NOT modified
(per repeated user instruction).
The 5th provider (claude-code) is documented in guide_ai_client.md
+ the data_oriented_design.md references the nagent pattern as the
source of the canonical rules.
The cross-references are bidirectional: the 6 styleguides reference
the 3 project docs; the 3 project docs reference the 6 styleguides;
the 2 doc updates reference both; AGENTS.md + ./docs/AGENTS.md
provide the entry points.
This commit is contained in:
@@ -0,0 +1,253 @@
|
||||
# Data-Oriented Design (the canonical rules)
|
||||
|
||||
**Status:** This is the canonical DOD reference for Manual Slop. Imported by `AGENTS.md` and injected into the Application's RAG / context assembly via `manual_slop.toml [agent].context_files`. One source of truth for both harnesses.
|
||||
**Source:** Adapted from Mike Acton's `context/data-oriented-design.md` (13,084 bytes, the nagent canonical reference).
|
||||
**Date:** 2026-06-12
|
||||
|
||||
> **What this is.** Operating rules, not philosophy: every rule here tells you what to *do*. Approach every problem — code, plan, pipeline, document — by understanding the real data first, then designing the simplest machine that transforms the input you actually have into the output you actually need, at a cost you can state. Decide from facts and measurement, not habit, analogy, or dogma.
|
||||
>
|
||||
> **Manual Slop context.** The project is an ImGui GUI orchestrator for LLM-driven coding sessions. The dominant data is *the conversation* — a typed message list with role + content + metadata + optional thinking segments. The data has to survive across workers (MMA Tier 3 subprocesses), across tools (the 45 MCP tools), across LLM providers (8 send paths), and across the user's editing session (per-entry edit, branch, undo). The data is the thing; the workers and processes are disposable.
|
||||
|
||||
---
|
||||
|
||||
## 0. Scope, tiers, and precedence
|
||||
|
||||
Scale the ceremony to the task. Decide the tier first; when unsure, pick the higher tier and say which you picked.
|
||||
|
||||
| Tier | When | What to do |
|
||||
|---|---|---|
|
||||
| **Tier 0** | Trivial: typo fixes, mechanical edits, one-line bugfixes, answering questions | Apply the defaults silently (naming, explicit error behavior, no speculative generality). No written plan or checklist |
|
||||
| **Tier 1** | Non-trivial change: new function or feature, behavior change, anything that touches a data layout, contract, or interface | Required: answer the framing + data questions in a short written plan *before* implementing, run the simplification pass, run the final self-check |
|
||||
| **Tier 2** | Subsystem-scale: new or substantially reworked subsystem, pipeline, or tool | Everything in tier 1 plus the enforceable deliverables (per §10) |
|
||||
|
||||
**Precedence when rules conflict:**
|
||||
|
||||
1. An explicit instruction from the user for the current task
|
||||
2. **This document** (`conductor/code_styleguides/data_oriented_design.md`)
|
||||
3. Existing codebase or workflow convention
|
||||
|
||||
When this document conflicts with existing convention and complying would mean a large refactor, **do not silently rewrite and do not silently conform**: state the conflict, estimate the cost of each option, and propose the smallest compliant change.
|
||||
|
||||
---
|
||||
|
||||
## 1. The 3 defaults to reject
|
||||
|
||||
These are the three default beliefs that produce bad solutions. Each comes with the replacement behavior — do the replacement, every time:
|
||||
|
||||
### 1.1 "The tools are the platform."
|
||||
|
||||
**Reality is the platform:** the actual hardware, organization, deadline, physics.
|
||||
|
||||
*Do instead:* before designing, name the real platform and the 2-3 of its fixed properties that constrain this solution, and design within them.
|
||||
|
||||
**For Manual Slop:** the platform is the user's machine (Windows; 1-8 cores; 16-128 GB RAM), the LLM provider API (rate limits, context window, cost), and the MCP tool surface (45 tools, 3-layer security). Not the ImGui API; not the Python version. The ImGui API is the *view*; the platform is the *view + the data + the user*.
|
||||
|
||||
### 1.2 "Design around a model of the world."
|
||||
|
||||
**World models** (objects, metaphors, idealized categories) hide the actual data and the actual cost.
|
||||
|
||||
*Do instead:* design around the data. Do not introduce an abstraction until you can describe, concretely, the data it organizes and the transform it serves — and what the abstraction costs.
|
||||
|
||||
**For Manual Slop:** the data is the `disc_entries` list, the `FileItem` schema, the `ContextPreset` schema, the `RAGEngine` index, the `comms.log` JSON-L. Not the *Discussion* or the *Persona* or the *Project* as objects. The objects are convenient summaries; the data is the ground truth.
|
||||
|
||||
### 1.3 "The solution matters more than the data."
|
||||
|
||||
**The only purpose of any solution is to transform data from one form to another.**
|
||||
|
||||
*Do instead:* start every task from the actual inputs and required outputs, never from the machinery you'd like to build.
|
||||
|
||||
**For Manual Slop:** before proposing a new class, module, or pipeline, write down (in a comment, in the plan, in the test) what the input is and what the output is. If you can't, that's the first task.
|
||||
|
||||
---
|
||||
|
||||
## 2. The 8 core defaults (any problem)
|
||||
|
||||
1. **The problem is the data.** Before proposing any solution, describe the input and output concretely. If you can't, getting that description *is* the first task.
|
||||
2. **State the cost.** Every design recommendation you make must state its cost (time, memory, complexity, maintenance) and on what platform that cost is paid. A recommendation without a cost is a guess.
|
||||
3. **Solve only the problem you have.** Different data is a different problem. Do not add parameters, options, abstraction layers, or extension points for hypothetical future needs. If you're tempted, write the one-line note of what you *didn't* build and why, and move on.
|
||||
4. **Where there is one, there are many.** Anything that happens once almost always happens many times — across space or across the time axis. Default every design to the batch; treat the single case as a batch of size one.
|
||||
5. **The common case dominates.** Identify the most common case explicitly and design the straight-line path for it. Handle rare and error cases, but outside that path — a "maybe" checked everywhere is an "always."
|
||||
6. **Exploit every constraint you have.** List the known constraints (ranges, volumes, rates, invariants) and use them to remove work. Do not discard a constraint to make the solution "more general" — that generality is a cost paid forever.
|
||||
7. **Simplicity is removing work.** Prefer fewer states, fewer steps, fewer special cases, fewer moving parts. Every added state or branch must be carried, tested, and explained — count them as cost.
|
||||
8. **"Can't be done" is a cost claim.** When something seems impossible, what is almost always true is that it costs more than it's worth. Say that, with the estimate, so the tradeoff can actually be decided.
|
||||
|
||||
---
|
||||
|
||||
## 3. Get the real data (required before designing)
|
||||
|
||||
You cannot observe data you were not given — so observe what you *can*, and label everything else:
|
||||
|
||||
- **Inspect before assuming.** Read representative input files, sample actual values, read the actual call sites, run the code on real input when a way to do so exists. Do not design from the type signatures or the docs alone.
|
||||
- **Label every assumption.** For each fact you need but cannot observe, write an explicit line — `ASSUMPTION: — affects ` — in your plan, and prefer designs that are cheap to revisit if the assumption is wrong. Ask the user only when the answer materially changes the design.
|
||||
- **Never fabricate.** Do not invent plausible-looking values, distributions, or measurements and treat them as real.
|
||||
|
||||
**Answer these about the data (in the tier 1+ plan):**
|
||||
|
||||
1. What does the input actually look like — shape, volume, source?
|
||||
2. What are the most common real values, and how are they distributed?
|
||||
3. What are the acceptable ranges, and what happens when out-of-range data arrives?
|
||||
4. What is the frequency of change — what is stable, what is volatile?
|
||||
5. What does the solution read and where does it come from? What does it write and where is it used? What does it touch that it doesn't need?
|
||||
|
||||
**For Manual Slop specifically:** the data is `disc_entries` (the conversation), `FileItem` (per-file curation), `ContextPreset` (per-preset curation), `RAGEngine` (semantic search), `comms.log` (audit), `Persona` (agent profile), `manual_slop.toml` (project config), `app_state` (live state). Read the actual files before designing.
|
||||
|
||||
---
|
||||
|
||||
## 4. Method (tier 1+)
|
||||
|
||||
Show this work as a short plan, a line or two per step:
|
||||
|
||||
1. **Frame it.** What is the problem, why is it worth solving, where is the limit beyond which it isn't, and what is plan B?
|
||||
2. **Get the data** (per §3).
|
||||
3. **State the cost** of the dominant transform on the real platform.
|
||||
4. **Design the transform:** a sequence or DAG of explicit transformations — what comes in, what goes out, what each step is responsible for, with explicit contracts (shape, meaning, ownership, lifetime, valid ranges) at each boundary.
|
||||
5. **Run the simplification pass** (per §5); say which questions applied and what work they removed.
|
||||
6. **Define done.** State the success criteria and what evidence would prove the approach wrong, before building.
|
||||
7. **Verify.** Check the result against the real data and the stated criteria, and report what was and wasn't verified.
|
||||
|
||||
---
|
||||
|
||||
## 5. The simplification pass (run recursively on every sub-problem)
|
||||
|
||||
The 7 questions, applied in order, to every sub-problem:
|
||||
|
||||
| # | Question | Reduces |
|
||||
|---|---|---|
|
||||
| 1 | Can we **not do this at all**? | Work that shouldn't exist |
|
||||
| 2 | Can we do this **only once** (precompute, cache, amortize)? | Repeated work |
|
||||
| 3 | Can we do this **fewer times**? | Frequency of work |
|
||||
| 4 | Can we **approximate** the result so that no one notices the difference? | Precision cost |
|
||||
| 5 | Can we use a **small lookup table**? | Branching cost |
|
||||
| 6 | Can we use a **large lookup table**? | Branching cost (alternative) |
|
||||
| 7 | Can we use a **small buffer/FIFO** to decouple producer from consumer? | Coupling cost |
|
||||
| 8 | Can we **constrain the problem further** so a simpler machine suffices? | Generality cost |
|
||||
|
||||
If any question applies, do the cheaper thing. If a question doesn't apply, say why and move on. The questions are not a checklist to score against; they're a habit.
|
||||
|
||||
---
|
||||
|
||||
## 6. Design rules
|
||||
|
||||
- **Minimize states and branches by design**, not by adding checks. Where the data genuinely varies, partition it by case and handle each partition straight-line, rather than re-deciding the case per element.
|
||||
- **Out-of-range and error behavior is always explicit** — clamp, reject, drop, or fail loudly; chosen deliberately and written down. Never leave undefined behavior as an implicit policy, in any tier.
|
||||
- **Complexity requires evidence.** Add complexity only against a real, observed need — never a hypothetical one.
|
||||
|
||||
---
|
||||
|
||||
## 7. Performance claims
|
||||
|
||||
- **Never assert an unmeasured performance result.** Not "this should be faster," not invented numbers.
|
||||
- If a way to measure exists (benchmark, profiler, test harness, counters), measure, and include before/after numbers with the change.
|
||||
- If no way to measure exists here, label the change **unverified**, state the expected effect as a hypothesis, and specify the exact measurement that would verify it.
|
||||
- If there is no measurable performance requirement, build the simplest correct design and skip speculative optimization entirely.
|
||||
|
||||
**For Manual Slop:** the existing audit scripts (`scripts/audit_main_thread_imports.py`, `scripts/audit_weak_types.py`, `scripts/check_test_toml_paths.py`) are the measurement infrastructure. Use them. Don't claim "faster" without a number from one of these.
|
||||
|
||||
---
|
||||
|
||||
## 8. Software specifics (systems, engine, embedded, game)
|
||||
|
||||
The rules above apply to any problem. These are their conclusions for software, where the hardware is unforgiving and the data volumes are real.
|
||||
|
||||
### 8.1 Batch-first transforms (plural by default)
|
||||
|
||||
- Write transforms to operate on **batches/arrays** by default, named in the **plural** (`update_things`, not `update_thing`).
|
||||
- A singular call is a degenerate batch: the same batch path with `count = 1`. Do not maintain separate singular logic without a proven, measured need.
|
||||
- Exception: true singletons (configuration state, a single shared resource). Taking the exception requires a written note: why the data is genuinely singular and batch semantics don't apply.
|
||||
|
||||
### 8.2 Memory, layout, and access
|
||||
|
||||
- **Indices over pointers/references/handles by default** (index into a contiguous array or table). Any pointer-heavy hot path must include a short written justification for why indices are insufficient.
|
||||
- Organize data by **access pattern, not conceptual ownership**. Split hot and cold fields when the cold fields aren't needed in the dominant loop.
|
||||
- For each hot path, write down the expected **access pattern** (linear / strided / random), expected **branch behavior** (predictable / unpredictable), and the hardware assumptions.
|
||||
- When branch entropy is high, prefer **partitioned passes** (bucket by state/tag, process each bucket straight-line) over per-element branching.
|
||||
- Keep the common-case path branch-minimal; rare and error handling lives outside the hot loop.
|
||||
|
||||
### 8.3 Data protocols between systems
|
||||
|
||||
Systems communicate through **explicit data protocols**, modeled after network protocols and file formats — explicit layout, versioning, documented meaning. The default is a **flat struct**: fixed layout, no hidden pointers, no OO-style interfaces. Use tagged unions or header-plus-payload when the flat struct genuinely can't express it. Do not model system boundaries as objects, virtual calls, or opaque handles.
|
||||
|
||||
**For Manual Slop:** the boundary between the AI client and the LLM provider is a *flat struct* (the `Message` dataclass: `role, content, tool_calls, tool_results`); the boundary between the MCP client and the tool implementer is a *flat struct* (the `tool_input` dict); the boundary between the LLM client and the GUI is the *comms.log* JSON-L. Not objects with virtual methods. Not opaque handles. Flat structs.
|
||||
|
||||
### 8.4 Hardware is the platform
|
||||
|
||||
Design with the actual hardware's properties — cache hierarchy, memory bandwidth, alignment, latency vs throughput — and to its strengths.
|
||||
|
||||
- **Latency and throughput are only the same thing in a sequential system.** For every performance requirement, identify which one it actually is before designing for it.
|
||||
- The compiler and language are tools, not magic: memory layout, access order, and the choice of what work to do at all are your job, not theirs — and they are roughly 90% of the problem. Know what the compiler can reasonably do with what you wrote, and don't delegate what it can't.
|
||||
|
||||
---
|
||||
|
||||
## 9. The 4 memory dimensions (the Manual Slop context)
|
||||
|
||||
The conversation data has 4 distinct memory dimensions. Each lives at a different layer; each serves a different purpose. The wrong shape for the wrong layer is a common mistake.
|
||||
|
||||
| # | Dim | Where it lives | What it stores | How it's edited | How it's queried | SSDL |
|
||||
|---|---|---|---|---|---|---|
|
||||
| 1 | **Curation** | `FileItem` + `ContextPreset` + Fuzzy Anchors | *How to render a file* in the AI's context window | Structural File Editor; project TOML | Implicit in `aggregate.py:run` at discussion start | `[Q]` |
|
||||
| 2 | **Discussion** | `app.disc_entries` + branching + UISnapshot | *What was said* in the conversation | GUI `[Edit]` mode; `[Branch]`; undo/redo | `build_markdown` renders as prior context | `o==>` |
|
||||
| 3 | **RAG** | `src/rag_engine.py` (ChromaDB) | *Semantic fingerprints* of indexed files | (opaque vector store) | `RAGEngine.search()` at LLM call time | `[Q]` |
|
||||
| 4 | **Knowledge** | `~/.manual_slop/knowledge/*.md` + per-file + digest + ledger | *Durable learnings* from past sessions | Plain markdown edit | Bounded digest as stable prefix | `o==>` |
|
||||
|
||||
**The shape rule:** curation is per-file structural; discussion is per-turn conversational; RAG is opt-in semantic; knowledge is per-project durable. A feature that wants one should use the matching dimension; mixing them is a maintenance liability.
|
||||
|
||||
See `conductor/code_styleguides/agent_memory_dimensions.md` for the full styleguide.
|
||||
|
||||
---
|
||||
|
||||
## 10. Enforceable deliverables (tier 2)
|
||||
|
||||
For each new or substantially reworked subsystem:
|
||||
|
||||
- One explicit **batch transform contract**: input layout, output layout, owner, lifetime, valid value ranges.
|
||||
- A **plural/batch path** for every transform; singular calls are thin wrappers over the batch implementation (`count = 1`) unless documented as a true singleton.
|
||||
- A written **justification for any pointer/reference/handle-heavy hot path** explaining why index-based access is insufficient.
|
||||
- Explicit **out-of-range behavior** (clamp/reject/drop/error) at every input boundary.
|
||||
- Unresolved design questions filed as **local issue files under `issues/`** — not GitHub issues, not inline TODOs.
|
||||
|
||||
**For Manual Slop specifically:** the equivalent of `issues/` is `docs/reports/` (where session retrospectives, audit reports, and design-issue docs live) or per-track `spec.md` §9 "Open Questions".
|
||||
|
||||
---
|
||||
|
||||
## 11. Final self-check (run before delivering tier 1+ work)
|
||||
|
||||
Verify, and fix or flag anything that fails:
|
||||
|
||||
- [ ] The plan answered the framing, data, and cost questions — or every gap is labeled `ASSUMPTION` with what it affects.
|
||||
- [ ] The most common case is identified and the design serves it straight-line; rare/error cases are out of the common path.
|
||||
- [ ] The simplification pass ran; the work it removed (or why nothing could be removed) is stated.
|
||||
- [ ] No speculative generality: no parameter, option, or abstraction exists for a need that isn't real yet.
|
||||
- [ ] Out-of-range and error behavior is explicit at every boundary.
|
||||
- [ ] Transforms are plural/batch, or the singleton exception is documented.
|
||||
- [ ] Pointer-heavy hot paths carry their written justification; everything else uses indices.
|
||||
- [ ] No unmeasured performance claim anywhere in code, comments, or summary; measurements included where possible, hypotheses labeled where not.
|
||||
- [ ] Done-criteria from the plan were checked, and the summary reports what was verified and what wasn't.
|
||||
- [ ] (Tier 2) Deliverables above are present; open questions are filed under `docs/reports/` or per-track `spec.md` §9.
|
||||
|
||||
---
|
||||
|
||||
## 12. Cross-references
|
||||
|
||||
- `AGENTS.md` — imports this file; the project-root agent-facing rules
|
||||
- `./docs/AGENTS.md` — the agent-facing mirror of `docs/Readme.md` (recommended first read for any agent scoping a feature)
|
||||
- `conductor/code_styleguides/agent_memory_dimensions.md` — the 4 memory dimensions
|
||||
- `conductor/code_styleguides/rag_integration_discipline.md` — the conservative-RAG rule
|
||||
- `conductor/code_styleguides/cache_friendly_context.md` — stable-to-volatile ordering + the cache TTL contract
|
||||
- `conductor/code_styleguides/knowledge_artifacts.md` — the knowledge harvest pattern
|
||||
- `conductor/code_styleguides/feature_flags.md` — "delete to turn off" + config flags
|
||||
- `conductor/product-guidelines.md` — the project's other product conventions
|
||||
- `conductor/tech-stack.md` — the tech stack constraints
|
||||
- `conductor/edit_workflow.md` — the edit-tool contract
|
||||
|
||||
---
|
||||
|
||||
## 13. External sources (the prior art this was adapted from)
|
||||
|
||||
- **Mike Acton, "Data-Oriented Design and C++"** (cppCon 2014) — the foundational DOD talk
|
||||
- **Casey Muratori, "The Big OOPs: Anatomy of a Thirty-Five-Year Mistake"** (BSC 2025) — the historical indictment of OOP
|
||||
- **Ryan Fleury, "A Taxonomy of Computation Shapes"** (Feb 2023) — the 6 computational shapes
|
||||
- **Ryan Fleury, "The Codepath Combinatoric Explosion"** (Apr 2023) — the nil-sentinel / immediate-mode defusing techniques
|
||||
- **Ryan Fleury, "Errors are just cases"** (the `Result[T, ErrorInfo]` pattern) — the data-oriented error handling
|
||||
- **Andrew Reece, "Assuming as Much as Possible"** (BSC 2025) — the Xar pattern; the engineering discipline for stripping layers
|
||||
- **John O'Donnell, "IMGUI / The Pitch / MVC"** — the immediate-mode + IEventTarget paradigm
|
||||
- **Mike Acton, `context/data-oriented-design.md`** (nagent canonical; 13,084 bytes) — the immediate source for the structure of this document
|
||||
Reference in New Issue
Block a user