manual_slop/conductor/tracks/tool_bias_tuning_20260308/spec.md

Specification: Agent Tool Preference & Bias Tuning

Overview

This track introduces a mechanism to influence AI agent tool selection by implementing a weighting and scoring system at the orchestration layer. Since model APIs do not natively support tool priority, this feature uses semantic nudging (tags in tool descriptions) and explicit system instructions to "bias" the agent toward preferred tools and parameters.

Dependencies

• This track is strictly dependent on the completion of the Saved Tool Presets track, as it extends the tool preset data model.

Functional Requirements

• Weighting Mechanism (Hybrid):
  • Description Nudging: Automatically prefix or suffix tool descriptions with priority indicators (e.g., [HIGH PRIORITY], [PREFERRED]) based on their assigned weight.
  • Strategy Injection: Dynamically generate a "Tooling Strategy" section in the System Prompt that lists preferred tools and usage guidelines based on the active preset and global bias.
• Priority Levels:
  • Support a 5-level priority scale (1: Lowest, 5: Highest).
  • Default all tools to Level 3 (Neutral).
• Parameter-Level Bias:
  • Allow users to assign "Preferred" or "Discouraged" flags to specific tool parameters (e.g., biasing search_files to always use pattern instead of just path).
• Configuration & Storage:
  • Preset-Based: Store tool and parameter weights within the tool_presets.toml file.
  • Global Bias: Implement a global "Bias Profile" (e.g., Balanced, Discovery-Heavy, Execution-Focused) that applies multipliers to tool categories.
• GUI Integration:
  • Priority Badges: Display color-coded badges (e.g., Red for High, Gray for Low) in tool lists and the Operations panel.
  • Weight Sliders: Add sliders to the Tool Preset manager to allow fine-grained adjustment of tool weights.
  • Active Bias Control: Include a "Bias Override" dropdown in the agent focus modal to allow temporary adjustments before spawning a worker.

Non-Functional Requirements

• Low Latency: The dynamic generation of nudged descriptions and system instructions must not noticeably delay agent initialization.
• Provider Consistency: The biasing strategy must be effective across Gemini, Anthropic, and OpenAI models.
• Scalability: The system should handle future additions of new tools and parameters without requiring core logic changes.

Acceptance Criteria

• Users can adjust tool weights in the Tool Preset manager and see the changes reflected in color-coded badges.
• Tool descriptions sent to the AI include semantic priority tags based on the assigned weights.
• The System Prompt includes a dynamically generated "Tooling Strategy" section.
• Agents show a statistically significant preference for high-weight tools in controlled tests.
• Parameter-level bias correctly influences how agents formulate tool calls.

Out of Scope

• Implementing reinforcement learning to "learn" tool weights automatically.
• Hardcoding weights into the AI client (all weights must be user-configurable via presets).