# Track Specification: True Parallel Worker Execution (true_parallel_worker_execution_20260306)

## Overview
Add worker pool management and configurable concurrency limits to the DAG engine. Currently workers execute in parallel per tick but with no limits or tracking; this track adds max_workers configuration, worker tracking, and proper pool management.

## Current State Audit

### Already Implemented (DO NOT re-implement)

#### Parallel Execution EXISTS (multi_agent_conductor.py)
- **`ConductorEngine.run()` already spawns parallel workers**:
  ```python
  threads = []
  for ticket in to_run:
      t = threading.Thread(
          target=run_worker_lifecycle,
          args=(ticket, context, context_files, self.event_queue, self, md_content),
          daemon=True
      )
      threads.append(t)
      t.start()
  
  for t in threads:
      t.join()
  ```
- **Current behavior**: ALL ready tickets spawn immediately, no limit
- **Limitation**: All threads join before next tick - blocks until all complete

#### Thread Safety (existing)
- **`_send_lock`** in ai_client.py serializes API calls
- **`_pending_gui_tasks_lock`** protects GUI state updates
- **Ticket status updates** via `engine.update_task_status()`

### Gaps to Fill (This Track's Scope)
- No max_workers limit - spawns unlimited threads
- No worker tracking - thread references discarded after join
- No configurable pool size
- No graceful degradation under load

## Architectural Constraints

### Thread Safety (CRITICAL)
- **`_send_lock` already exists**: All AI calls serialized automatically
- **Ticket status updates MUST be atomic**: Use lock on status changes
- **DAG state MUST be protected**: `get_ready_tasks()` returns snapshot

### Worker Pool Pattern
- Use `threading.Semaphore` to limit concurrent workers
- Track active threads in `_active_workers: dict[str, Thread]`
- Non-blocking: don't wait for all to complete before next tick

## Architecture Reference

### Key Integration Points

| File | Lines | Purpose |
|------|-------|---------|
| `src/multi_agent_conductor.py` | 100-150 | `ConductorEngine.run()` - add pool logic |
| `src/multi_agent_conductor.py` | 50-60 | `__init__` - add `_worker_semaphore`, `_active_workers` |
| `src/dag_engine.py` | 50-100 | `ExecutionEngine` - ready task query |
| `config.toml` | N/A | Add `[mma] max_workers = 4` |

### Proposed Pool Pattern
```python
# In ConductorEngine.__init__:
self._max_workers: int = 4  # Configurable
self._worker_semaphore = threading.Semaphore(self._max_workers)
self._active_workers: dict[str, threading.Thread] = {}
self._workers_lock = threading.Lock()

# In run():
ready_tasks = self.engine.tick()
# Limit to available semaphore slots
available = min(len(ready_tasks), self._max_workers - len(self._active_workers))
to_spawn = ready_tasks[:available]

for ticket in to_spawn:
    def worker_wrapper(ticket):
        with self._worker_semaphore:
            run_worker_lifecycle(ticket, ...)
        with self._workers_lock:
            self._active_workers.pop(ticket.id, None)
    
    t = threading.Thread(target=worker_wrapper, args=(ticket,), daemon=True)
    with self._workers_lock:
        self._active_workers[ticket.id] = t
    t.start()

# Don't join - let them complete independently
time.sleep(0.5)  # Yield before next tick
```

## Functional Requirements

### FR1: Configurable Max Workers
- Default: 4 concurrent workers
- Read from config.toml: `[mma] max_workers = 4`
- Clamp to reasonable range (1-16)

### FR2: Worker Pool with Semaphore
- Use `threading.Semaphore(max_workers)` to limit concurrency
- Workers acquire semaphore on start, release on completion
- No spawning if semaphore exhausted

### FR3: Worker Tracking
- Store thread references in `_active_workers[ticket_id]`
- Remove on completion
- Enable kill functionality (coordinates with kill_abort_workers track)

### FR4: Non-Blocking Execution
- Don't join all threads before next tick
- Check completed workers, spawn new ready tasks
- Continue until all complete or blocked

## Non-Functional Requirements

| Requirement | Constraint |
|-------------|------------|
| Throughput | Configurable via max_workers |
| Memory | Per-worker stack space (~8MB each) |
| API Rate | Respect provider rate limits |

## Testing Requirements

### Unit Tests
- Test semaphore limits workers correctly
- Test worker tracking dict updates
- Test config loading for max_workers

### Integration Tests (via `live_gui` fixture)
- Create 10 independent tickets, set max_workers=4
- Verify only 4 run at a time
- Verify no race conditions on status updates

### Stress Tests
- 20+ workers with pool of 4
- Verify no deadlock
- Verify no memory leak

## Dependencies
- **Coordinates with**: `kill_abort_workers_20260306` (uses worker tracking)
- **Coordinates with**: `mma_multiworker_viz_20260306` (displays worker status)

## Out of Scope
- GPU parallelism (not applicable)
- Distributed execution (single machine only)
- Priority-based scheduling (separate track)

## Acceptance Criteria
- [ ] max_workers configurable via config.toml
- [ ] Semaphore limits concurrent workers
- [ ] Worker tracking dict maintained
- [ ] No race conditions on ticket status
- [ ] Non-blocking execution (no join-all)
- [ ] >80% test coverage for pool code
- [ ] 1-space indentation maintained
