ed
ee7b1e263e
Documents the MiniMax_understand_image workflow for converting screenshots to ASCII Layout Maps. Covers: when to use it, the 6-step workflow, the proportional-measurement prompt pattern, faithful rendering rules (width ratios, empty space, floating window position, color annotations, tab bars, table rows), multi-screenshot composition, and limitations.
343 lines
18 KiB
Markdown
343 lines
18 KiB
Markdown
# Guide to ASCII UI Layout Maps (DSL)
|
|
|
|
This document provides a comprehensive specification and reference manual for the **ASCII UI Layout Map DSL** used in the Manual Slop application. It establishes the conventions, visual grammar, widget vocabulary, and advanced techniques required to model dear imgui interfaces in structured text.
|
|
|
|
---
|
|
|
|
## 1. Core Paradigm: Text-Side UI Contracts
|
|
|
|
Manual Slop is built around a thread-safe, immediate-mode render pipeline in `src/gui_2.py`. In an environment where AI developer-agents operate with text-based tools and cannot directly view or interact with the runtime window, graphical user interfaces must be modeled as structured, human-auditable text contracts.
|
|
|
|
The ASCII UI Layout Map DSL solves this problem by defining a strict visual grammar. Because Dear ImGui layouts are rectilinear, immediate-mode, and deterministic, they map directly to rectangular text blocks. By pairing a visual **ASCII Layout Map** with an operational **SSDL (Spec/Sketch Description Language)** DAG, we can fully specify both the layout structure and the logic flow of a panel before writing any Python code.
|
|
|
|
---
|
|
|
|
## 2. Exhaustive Widget Vocabulary
|
|
|
|
To maintain visual clarity, the DSL maps ImGui widgets to specific text representations.
|
|
|
|
### 2.1 The Widget Reference Table
|
|
|
|
| ImGui Widget Type | Python Code Example | ASCII Layout Representation | Notes / Alignment Rules |
|
|
|---|---|---|---|
|
|
| **Window Frame** | `imgui.begin("Title")` | `+================= Title =================+` | Top/bottom borders use double horizontal bars or thick lines. |
|
|
| **Child Panel** | `imgui.begin_child("id")` | `+-----------------------------------------+` | Standard single-line border mapping child boundaries. |
|
|
| **Button** | `imgui.button("Save")` | `[Save]` | Square bracket boundaries. Fits text exactly. |
|
|
| **Checkbox (Off)** | `imgui.checkbox("A", False)` | `[ ] Option A` | Box containing a space, followed by the label. |
|
|
| **Checkbox (On)** | `imgui.checkbox("B", True)` | `[X] Option B` or `[✓] Option B` | Box containing `X` or checkmark `✓`. |
|
|
| **Radio Button (Off)**| `imgui.radio_button("X", False)`| `( ) Choice X` | Parentheses containing a space, followed by the label. |
|
|
| **Radio Button (On)** | `imgui.radio_button("Y", True)` | `(o) Choice Y` or `(•) Choice Y` | Parentheses containing `o` or bullet `•`. |
|
|
| **Dropdown / Combo** | `imgui.combo("##c", 0, opts)` | `[Selected Value v]` | Lowercase `v` represents the dropdown arrow. |
|
|
| **Named Dropdown** | `imgui.combo("Role", 1, roles)`| `[Role: Selected Value v]` | Combines parameter name with current selection. |
|
|
| **Text Input Box** | `imgui.input_text("##t", val)`| `|Current Value_______________________|` | Bounded by pipes (`\|`), underscores denote empty space. |
|
|
| **Slider / Drag** | `imgui.slider_float("##s", v)` | `[=======----------------] 0.70` | Shows progress track, slider handle, and numerical label. |
|
|
| **Closed Tree Node** | `imgui.tree_node("node")` | `> Node Label` | The `>` symbol denotes a collapsed section. |
|
|
| **Open Tree Node** | `imgui.tree_node("node")` | `v Node Label` | The lowercase `v` denotes an expanded section. |
|
|
| **Table Grid** | `imgui.table_next_column()` | `\| Cell 1 \| Cell 2 \|` | Cell bounds delimited by pipes (`\|`). |
|
|
| **Separator** | `imgui.separator()` | `---` | Standard horizontal dividing rule. |
|
|
|
|
---
|
|
|
|
## 3. Layout Mathematics & Spacing Alignment
|
|
|
|
ImGui's layout calculations (such as wrapping, horizontal nesting, and right-alignment) are captured in the ASCII DSL using specific alignment conventions.
|
|
|
|
### 3.1 Horizontal Nesting (`imgui.same_line()`)
|
|
In ImGui, widgets stack vertically by default. To place widgets on the same horizontal line, `imgui.same_line()` is invoked. In the ASCII DSL, this is represented by placing the brackets on the same text line separated by explicit spacing.
|
|
|
|
#### Python Code
|
|
```python
|
|
imgui.text("Model Parameter:")
|
|
imgui.same_line()
|
|
imgui.set_next_item_width(120)
|
|
imgui.combo("##model", active_idx, models)
|
|
imgui.same_line()
|
|
if imgui.button("Reset"):
|
|
reset_model()
|
|
```
|
|
|
|
#### ASCII Layout Representation
|
|
```
|
|
Model Parameter: [gemini-1.5-flash v] [Reset]
|
|
```
|
|
|
|
### 3.2 Right Alignment & Viewport Widths
|
|
To convey width ratios, the map uses proportional spacing. If a button is aligned to the right edge of a panel, it is placed at the far right of the ASCII boundary box.
|
|
|
|
#### Proportional Width Mappings
|
|
* **Fill Available:** `|Text Input Field_______________________________|` (width extends to boundary)
|
|
* **Fixed Width:** `[Close]` (width matches the label length)
|
|
* **Proportional Split:**
|
|
```
|
|
+-------------------------------------------------------------+
|
|
| Left Column (40%) | Right Column (60%) |
|
|
| |Input A________________| | |Input B_____________________| |
|
|
+-------------------------------------------------------------+
|
|
```
|
|
|
|
---
|
|
|
|
## 4. Advanced High-Resolution Techniques
|
|
|
|
When laying out complex, feature-dense screens, a standard single-pane ASCII wireframe will run out of resolution. The DSL utilizes three advanced techniques to handle high-density layouts.
|
|
|
|
### 4.1 Feature Zooming (Macro/Micro Decomposition)
|
|
|
|
Feature Zooming decomposes a master interface into a parent macro-layout and multiple zoomed micro-layouts. The parent layout references zoomed sections using a target badge `[Zoom: Unique Name]`.
|
|
|
|
#### Macro-Layout (The Orchestration Dashboard)
|
|
```
|
|
+============================== Conductor Dashboard =============================+
|
|
| [Conductor Setup] [Epic Planner] |
|
|
+------------------------------------+-------------------------------------------+
|
|
| | |
|
|
| [Zoom: Conductor Track Sidebar] | [Zoom: Task Execution Pane] |
|
|
| | |
|
|
+------------------------------------+-------------------------------------------+
|
|
| [Zoom: Token Budget Footer] |
|
|
+================================================================================+
|
|
```
|
|
|
|
#### Micro-Layout: Zoom 1 (Conductor Track Sidebar)
|
|
```
|
|
[Zoom: Conductor Track Sidebar]
|
|
+----------------------------------+
|
|
| Track Browser: |
|
|
| v ACTIVE TRACKS |
|
|
| (o) mcp_refactor [======] 40%|
|
|
| ( ) type_strengths [ ] 0%|
|
|
| > COMPLETED TRACKS |
|
|
| > ARCHIVED TRACKS |
|
|
+----------------------------------+
|
|
```
|
|
|
|
#### Micro-Layout: Zoom 2 (Task Execution Pane)
|
|
```
|
|
[Zoom: Task Execution Pane]
|
|
+-------------------------------------------------------------+
|
|
| Active: mcp_refactor |
|
|
| Target Tier: [Tier 2 Tech Lead v] |
|
|
| --- |
|
|
| Ticket Queue: |
|
|
| [✓] t2_1: Extract File IO (Ready) |
|
|
| [✓] t2_2: Extract Python (Blocked) |
|
|
| +---------------------------------------------------------+ |
|
|
| | [I:ticket_details] -> [B:approve_spawn] => [B:execute] | |
|
|
| +---------------------------------------------------------+ |
|
|
| [Execute Selected] [Skip Selected] |
|
|
+-------------------------------------------------------------+
|
|
```
|
|
|
|
---
|
|
|
|
### 4.2 Grid-Based Focus Layouts
|
|
|
|
For complex table configurations or form layouts, the DSL overlays a coordinate grid (Columns `A, B, C...` and Rows `1, 2, 3...`). This coordinates precise width constraints, alignment boundaries, and event triggers.
|
|
|
|
```
|
|
Grid Focus Map:
|
|
A (30%) B (50%) C (20%)
|
|
+------------------------+-------------------------+--------------------+
|
|
1 | Provider: [gemini v] | Model: [1.5-flash v] | Status: [Local] |
|
|
+------------------------+-------------------------+--------------------+
|
|
2 | Temp: [=====-----] 0.7 | Top-P: [===-------] 0.4 | [Reset Defaults] |
|
|
+------------------------+-------------------------+--------------------+
|
|
```
|
|
|
|
#### Grid Alignment Reference
|
|
|
|
* **Cell A1:** Combobox for provider selection. Spans exactly 30% of the table width.
|
|
* **Cell B1:** Combobox for model selection. Spans exactly 50% of the table width, aligned horizontally with A1.
|
|
* **Cell C1:** Status badge. Right-aligned inside its column with a green success color tint.
|
|
* **Cell A2/B2:** Numeric parameter sliders. Underneath Row 1, matching column boundaries.
|
|
* **Cell C2:** Action button. Right-aligned, triggers parameter resets.
|
|
|
|
---
|
|
|
|
### 4.3 State Multiplicity Annotation
|
|
|
|
GUI elements often mutate structure based on runtime state variables. Rather than cluttering a single ASCII sketch with multiple options, separate sketches are annotated with `[State: Condition]` expressions.
|
|
|
|
#### Read-Only State Layout
|
|
```
|
|
[State: app.show_editing_panel == False]
|
|
+-----------------------------------------------------------------+
|
|
| System Prompt: coding-assistant |
|
|
| "You are an AI coding assistant designed to solve tasks..." |
|
|
| [Edit Prompt] |
|
|
+-----------------------------------------------------------------+
|
|
```
|
|
|
|
#### Editing State Layout
|
|
```
|
|
[State: app.show_editing_panel == True]
|
|
+-----------------------------------------------------------------+
|
|
| System Prompt: coding-assistant |
|
|
| +-------------------------------------------------------------+ |
|
|
| | You are an AI coding assistant designed to solve tasks... | |
|
|
| +-------------------------------------------------------------+ |
|
|
| [Save Changes] [Cancel] |
|
|
+-----------------------------------------------------------------+
|
|
```
|
|
|
|
---
|
|
|
|
## 5. Integration with SSDL (Spec/Sketch Description Language)
|
|
|
|
SSDL defines the operational control flow and data mutation path through the immediate-mode graph. Every docstring in `gui_2.py` must align its visual **ASCII Layout Map** with its functional **SSDL** flow.
|
|
|
|
### 5.1 SSDL Primitive Dictionary
|
|
|
|
* `[I:name]` (Instruction): Represents a passive rendering unit (labels, static text, or headers).
|
|
* `->` (Sequential path): Represents linear vertical or horizontal flow.
|
|
* `=>` (Wide path): Denotes parallel branches, such as columns, tables, or tab item rendering.
|
|
* `[B:name]` (Branch): Points to conditional widgets (checkboxes, popups, or expandable trees).
|
|
* `[S:name]` (Stateful Mutation): Represents inputs, sliders, or buttons that modify data.
|
|
* `[Q:name]` (State Query): Reads parameters from the App config or state.
|
|
* `[N:name]` (Nil Sentinel): Represents safety check values to prevent rendering crashes.
|
|
|
|
### 5.2 Side-by-Side SSDL-to-ASCII Alignment
|
|
|
|
The SSDL sequence operates as a code-path map for the corresponding layout widgets:
|
|
|
|
```python
|
|
def render_example(app: App) -> None:
|
|
"""Renders the RAG status and rebuild control.
|
|
|
|
SSDL: `[Q:rag_enabled] -> [B:rag_checkbox] -> [I:status_label] => [S:rebuild_btn]`
|
|
|
|
ASCII Layout Map:
|
|
+---------------------------------------------------------+
|
|
| [X] Enable RAG Subsystem | <- [B:rag_checkbox] (reads [Q:rag_enabled])
|
|
| Status: READY | Chunk Size: 512 | <- [I:status_label]
|
|
| [Rebuild Vector Index] | <- [S:rebuild_btn]
|
|
+---------------------------------------------------------+
|
|
"""
|
|
```
|
|
|
|
---
|
|
|
|
## 6. Step-by-Step Collaborative Design Workflow
|
|
|
|
To align implementations with user intent, developer-agents and human partners follow a 5-step workflow:
|
|
|
|
```mermaid
|
|
graph TD
|
|
A[Step 1: Pick Target Panel] --> B[Step 2: Establish Boundaries]
|
|
B --> C[Step 3: Initial ASCII Draft]
|
|
C --> D{Step 4: Critique & Iterate}
|
|
D -- Refine Sketch --> C
|
|
D -- Approved --> E[Step 5: Lock Design Contract]
|
|
```
|
|
|
|
### Collaborative Dialogue Example
|
|
|
|
1. **Step 1 (Agent):** Proposes target: "Let's update the layout for the Workspace Profile Saver."
|
|
2. **Step 2 (Human):** Establishes boundaries: "Yes, it must fit inside a popup modal, require a profile name, and have checkable scopes for Window Layout and Project Settings."
|
|
3. **Step 3 (Agent):** Drafts initial ASCII sketch:
|
|
```
|
|
+---------------------------------------------------------+
|
|
| Save Workspace Profile [X] |
|
|
+---------------------------------------------------------+
|
|
| Name: |my-profile_____________________________________| |
|
|
| Scopes: |
|
|
| [X] Window Positions & Layouts |
|
|
| [X] Active Project Settings |
|
|
| [Save Profile] [Cancel] |
|
|
+---------------------------------------------------------+
|
|
```
|
|
4. **Step 4 (Human):** Critiques: "The Save and Cancel buttons should be aligned to the right, and let's add a scope for Provider configuration."
|
|
5. **Step 4 (Agent):** Redrafts the sketch:
|
|
```
|
|
+---------------------------------------------------------+
|
|
| Save Workspace Profile [X] |
|
|
+---------------------------------------------------------+
|
|
| Name: |my-profile_____________________________________| |
|
|
| Scopes: |
|
|
| [X] Window Positions & Layouts |
|
|
| [X] Active Project Settings |
|
|
| [X] Provider Configuration |
|
|
| [Save] [Cancel] |
|
|
+---------------------------------------------------------+
|
|
```
|
|
6. **Step 5 (Human):** Approves: "This is perfect. Lock the design."
|
|
|
|
---
|
|
|
|
## 7. Verification & Audit Protocols
|
|
|
|
Once a design contract is locked and implemented, it must pass a three-tiered verification gate:
|
|
|
|
1. **AST Integrity:** Every docstring modification must pass `py_check_syntax` to ensure it doesn't break python parsing.
|
|
2. **Regression Check:** The test runner (`pytest tests/`) must be run to verify zero side-effects. Docstring additions must never alter execution logic.
|
|
3. **Puppeteer Visual Audit:** In visual simulation tests, the captured Dear ImGui layout boundaries and widget visibility flags are compared against the rows, columns, and conditional states defined in the ASCII design contract.
|
|
|
|
---
|
|
|
|
## 8. Screenshot-to-ASCII Reverse Engineering (Opt-In Extension)
|
|
|
|
When a running GUI state needs to be captured as an ASCII Layout Map — for bug reports, regression documentation, or Tier 2 handoff — the `MiniMax_understand_image` MCP tool can reverse-engineer a screenshot into the DSL. This is an **opt-in** workflow; the standard DSL (§1-§7) remains the forward-design path (text-first, code-second). This section covers the reverse path (screenshot-first, text-second).
|
|
|
|
### 8.1 When to Use This Extension
|
|
|
|
- **Bug reports**: the user sees a broken layout and screenshots it; the agent converts to ASCII for the report
|
|
- **Regression documentation**: before/after screenshots converted to ASCII pairs to document what changed
|
|
- **Tier 2 handoff**: the user provides a screenshot of the current working state; Tier 1 converts to ASCII so Tier 2 can see the target layout without running the GUI
|
|
- **Layout audit**: the user provides a screenshot of a misbehaving panel; the agent converts to ASCII to reason about the structure
|
|
|
|
### 8.2 The Workflow
|
|
|
|
```
|
|
Step 1: User provides screenshot file path(s)
|
|
Step 2: Agent calls MiniMax_understand_image with a proportional-measurement prompt
|
|
Step 3: Agent converts the structured description into an ASCII Layout Map
|
|
Step 4: User reviews + corrects proportions ("the left panel is wider", "the Debug window is top-right not center")
|
|
Step 5: Agent revises until the ASCII faithfully represents the screenshot
|
|
Step 6: The final ASCII map is committed to docs or a track spec
|
|
```
|
|
|
|
### 8.3 The Proportional-Measurement Prompt
|
|
|
|
The first `MiniMax_understand_image` call must ask for **precise proportional measurements**, not just a list of elements. The prompt should request:
|
|
|
|
1. Panel width percentages (left panel X%, right panel Y%)
|
|
2. Vertical order and height proportions of each section within each panel
|
|
3. Exact position of floating/overlay windows (which panel, which corner, relative size)
|
|
4. Exact text labels, button labels, tab names, checkbox states
|
|
5. Color annotations for status text (red for errors, green for success, blue for info)
|
|
6. Empty space proportions (how much of each panel is blank)
|
|
|
|
Without proportional measurements, the resulting ASCII will be "scrunched" — elements compressed into too-small areas, losing the visual hierarchy that makes the layout map useful.
|
|
|
|
### 8.4 Faithful Rendering Rules
|
|
|
|
When converting the structured description to ASCII:
|
|
|
|
- **Width ratios must be preserved.** If the left panel is 25% and the right is 75%, the ASCII must show the left panel as roughly 1/4 the total width and the right as 3/4. Do not make them 50/50.
|
|
- **Empty space must be represented.** If 80% of a panel is blank, the ASCII must show that blank space as empty lines within the panel border. Do not compress it away.
|
|
- **Floating windows must be positioned correctly.** If the Debug window is top-right of the Discussion Hub, it must appear in the top-right area of the right panel in the ASCII, not centered or bottom.
|
|
- **Color annotations use inline markers.** Red text: `1 failed` with a note `^^^ in red`. Green text: `OUT request` with a note. Blue text: `tool_call` with a note.
|
|
- **Tab bars list all tabs.** Even inactive tabs must appear so the reader can see the full navigation surface.
|
|
- **Tables show all visible rows.** The telemetry table with 4 data rows must show all 4 rows, not just 1-2.
|
|
|
|
### 8.5 Multi-Screenshot Composition
|
|
|
|
When the user provides multiple screenshots (e.g., different panel configurations, before/after states), each gets its own ASCII Layout Map. The maps are presented sequentially with a header line identifying the screenshot source:
|
|
|
|
```
|
|
**Screenshot 1** (timestamp) — Panel A + Panel B:
|
|
<ASCII map>
|
|
|
|
**Screenshot 2** (timestamp) — Panel A + Panel C + Debug overlay:
|
|
<ASCII map>
|
|
```
|
|
|
|
Do not attempt to merge multiple screenshots into a single composite ASCII. Each screenshot is its own layout state.
|
|
|
|
### 8.6 Limitations
|
|
|
|
- The `MiniMax_understand_image` tool cannot read images from the clipboard directly; the user must provide a file path (e.g., a ShareX screenshot path).
|
|
- The proportional measurements are estimates, not pixel-perfect. The user must review and correct.
|
|
- Complex layouts with many small elements may lose resolution in the ASCII. Use the Feature Zooming technique (§4.1) to decompose dense areas into zoomed micro-layouts.
|
|
- Color information is lost in ASCII. Use inline text annotations (`^^^ in red`) to preserve critical color signals.
|