Multi-Model Agent System

How to use different LLM models for different agents in Present-Agent2.

🎯 Strategy: Model Selection by Agent Role

Recommended Model Assignment

Cost vs Quality Trade-offs

High Stakes (Use Premium Models):

• Architecture decisions → Claude Sonnet 3.5
• Complex algorithms → Claude Sonnet 3.5 or GPT-4
• Product strategy → Claude Sonnet 3.5

Medium Stakes (Use Mid-tier):

• Code implementation → GPT-4 Turbo
• Task organization → GPT-4 Turbo
• Technical reviews → Claude Sonnet

Low Stakes (Use Fast/Cheap):

• Test execution → GPT-3.5 Turbo
• Bug reporting → GPT-3.5 Turbo
• User simulation → Claude Haiku or GPT-3.5

🛠️ Implementation Options

Option 1: External Agent Scripts (Recommended)

Create standalone scripts that use OpenAI/Anthropic APIs directly:

// scripts/agents/product-manager-agent.ts
import Anthropic from '@anthropic-ai/sdk';
import { readIssue, commentOnIssue } from './github-utils';

const anthropic = new Anthropic({
  apiKey: process.env.ANTHROPIC_API_KEY
});

async function productManagerResearch(issueNumber: number) {
  const issue = await readIssue(issueNumber);

  const message = await anthropic.messages.create({
    model: "claude-3-5-sonnet-20241022",
    max_tokens: 8192,
    messages: [{
      role: "user",
      content: `You are the Product Manager agent. Research: ${issue.title}\n\n${issue.body}`
    }]
  });

  await commentOnIssue(issueNumber, message.content[0].text);
}

// scripts/agents/coding-agent.ts
import OpenAI from 'openai';
import { readIssue, commentOnIssue } from './github-utils';

const openai = new OpenAI({
  apiKey: process.env.OPENAI_API_KEY
});

async function codingAgentImplement(issueNumber: number) {
  const issue = await readIssue(issueNumber);

  const response = await openai.chat.completions.create({
    model: "gpt-4-turbo-preview",
    messages: [{
      role: "system",
      content: "You are the Coding Agent. Follow TypeScript strict mode..."
    }, {
      role: "user",
      content: `Implement: ${issue.title}\n\n${issue.body}`
    }]
  });

  await commentOnIssue(issueNumber, response.choices[0].message.content);
}

Option 2: Model Router Function

Create a utility that routes to appropriate model:

// src/lib/model-router.ts
import Anthropic from '@anthropic-ai/sdk';
import OpenAI from 'openai';

type AgentRole = 'product-manager' | 'tickets-manager' | 'engineering-manager'
  | 'coding-agent' | 'testing-agent' | 'user-simulator';

const MODEL_CONFIG: Record<AgentRole, { provider: 'anthropic' | 'openai', model: string }> = {
  'product-manager': { provider: 'anthropic', model: 'claude-3-5-sonnet-20241022' },
  'tickets-manager': { provider: 'openai', model: 'gpt-4-turbo-preview' },
  'engineering-manager': { provider: 'anthropic', model: 'claude-3-5-sonnet-20241022' },
  'coding-agent': { provider: 'openai', model: 'gpt-4-turbo-preview' },
  'testing-agent': { provider: 'openai', model: 'gpt-3.5-turbo' },
  'user-simulator': { provider: 'anthropic', model: 'claude-3-haiku-20240307' }
};

export async function callAgent(
  role: AgentRole,
  systemPrompt: string,
  userPrompt: string
): Promise<string> {
  const config = MODEL_CONFIG[role];

  if (config.provider === 'anthropic') {
    const anthropic = new Anthropic({ apiKey: process.env.ANTHROPIC_API_KEY });
    const message = await anthropic.messages.create({
      model: config.model,
      max_tokens: 8192,
      system: systemPrompt,
      messages: [{ role: 'user', content: userPrompt }]
    });
    return message.content[0].text;
  } else {
    const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });
    const response = await openai.chat.completions.create({
      model: config.model,
      messages: [
        { role: 'system', content: systemPrompt },
        { role: 'user', content: userPrompt }
      ]
    });
    return response.choices[0].message.content!;
  }
}

Option 3: Agent Workflow Orchestrator

Full automation script that runs the entire workflow:

// scripts/run-agent-workflow.ts
import { callAgent } from '../src/lib/model-router';
import { readIssue, commentOnIssue, readFile } from './utils';

async function runFullWorkflow(featureRequest: string) {
  const issueNumber = await createIssue(`[FEATURE] ${featureRequest}`);

  // 1. Product Manager (Claude) - Research
  const pmSystemPrompt = await readFile('.claude/agents/product-manager.md');
  const research = await callAgent(
    'product-manager',
    pmSystemPrompt,
    `Research and create spec for: ${featureRequest}`
  );
  await commentOnIssue(issueNumber, `## Product Manager - Research\n\n${research}`);

  // 2. Tickets Manager (GPT-4) - Review
  const tmSystemPrompt = await readFile('.claude/agents/tickets-manager.md');
  const review = await callAgent(
    'tickets-manager',
    tmSystemPrompt,
    `Review this spec:\n\n${research}`
  );
  await commentOnIssue(issueNumber, `## Tickets Manager - Review\n\n${review}`);

  // 3. Engineering Manager (Claude) - Technical Review
  const emSystemPrompt = await readFile('.claude/agents/engineering-manager.md');
  const techReview = await callAgent(
    'engineering-manager',
    emSystemPrompt,
    `Technical review:\n\n${research}\n\n${review}`
  );
  await commentOnIssue(issueNumber, `## Engineering Manager - Review\n\n${techReview}`);

  // 4. Coding Agent (GPT-4) - Implement
  const caSystemPrompt = await readFile('.claude/agents/architect.md');
  const implementation = await callAgent(
    'coding-agent',
    caSystemPrompt,
    `Implement:\n\n${techReview}`
  );
  await commentOnIssue(issueNumber, `## Coding Agent - Implementation\n\n${implementation}`);

  // 5. Testing Agent (GPT-3.5) - Test
  const taSystemPrompt = await readFile('.claude/agents/testing-agent.md');
  const testResults = await callAgent(
    'testing-agent',
    taSystemPrompt,
    `Test this implementation:\n\n${implementation}`
  );
  await commentOnIssue(issueNumber, `## Testing Agent - Results\n\n${testResults}`);

  // 6. User Simulator (Claude Haiku) - Validate
  const usSystemPrompt = await readFile('.claude/agents/user-simulator.md');
  const uxValidation = await callAgent(
    'user-simulator',
    usSystemPrompt,
    `Validate UX:\n\n${implementation}`
  );
  await commentOnIssue(issueNumber, `## User Simulation - Validation\n\n${uxValidation}`);

  console.log(`✅ Workflow complete! Issue #${issueNumber}`);
}

// Usage
runFullWorkflow("Core Recommendation Engine");

💰 Cost Optimization

Estimated Costs Per Feature (Full Workflow)

Total per feature: ~$0.32

For 10 features: ~$3.20

Cost Reduction Strategies

1. Use cheaper models for simple tasks

   • User simulation → Claude Haiku ($0.25/1M input)
   • Testing → GPT-3.5 Turbo ($0.50/1M input)

2. Cache prompts (Claude supports prompt caching)

   const message = await anthropic.messages.create({
     model: "claude-3-5-sonnet-20241022",
     system: [
       {
         type: "text",
         text: systemPrompt,
         cache_control: { type: "ephemeral" }  // Cache this!
       }
     ],
     messages: [...]
   });
   

3. Batch operations when possible

🚀 Quick Start

1. Install Dependencies

npm install @anthropic-ai/sdk openai @octokit/rest

2. Configure Environment

# .env
ANTHROPIC_API_KEY=sk-ant-...
OPENAI_API_KEY=sk-...
GITHUB_TOKEN=ghp_...

3. Create Agent Runner

# Run full workflow
npm run agent-workflow "Build recommendation engine"

# Run specific agent
npm run agent product-manager issue-8
npm run agent coding-agent issue-8

4. Add to package.json

{
  "scripts": {
    "agent-workflow": "tsx scripts/run-agent-workflow.ts",
    "agent": "tsx scripts/run-single-agent.ts"
  }
}

🎯 Hybrid Approach: Claude Code + External Scripts

Best of both worlds:

1. Use Claude Code (current setup) for interactive development
2. Use external scripts for automated workflows

# Interactive (Claude Code - you control)
"Use the product-manager agent to research X"

# Automated (External script - fully automated)
npm run agent-workflow "Build feature X"

📊 Model Performance by Task

Based on testing:

🔄 Integration with Current Setup

Your current Claude Code agents can call these external scripts:

# In .claude/agents/product-manager.md

When running automated workflows, you can execute:

\`\`\`bash
tsx scripts/agents/product-manager.ts issue-8
\`\`\`

This will use Claude Sonnet 3.5 directly for research.

⚠️ Important Considerations

1. Context Management: Different models have different context windows

   • Claude Sonnet: 200K tokens
   • GPT-4 Turbo: 128K tokens
   • GPT-3.5: 16K tokens

2. Prompt Compatibility: Test prompts across models (they respond differently)

3. API Rate Limits: Monitor usage across providers

4. Consistency: Document which model is used for each agent role

🎉 Recommended Setup

For your use case (Present-Agent2):

// Recommended model assignments
const AGENTS = {
  'product-manager': 'claude-3-5-sonnet-20241022',     // Strategic
  'tickets-manager': 'gpt-4-turbo-preview',            // Structured
  'engineering-manager': 'claude-3-5-sonnet-20241022', // Architecture
  'coding-agent': 'gpt-4-turbo-preview',               // Implementation
  'testing-agent': 'gpt-3.5-turbo',                    // Cost-effective
  'user-simulator': 'claude-3-haiku-20240307'          // Fast/cheap
};

Want me to implement the multi-model agent system for you?