/trainCustomRouter

Train a custom router on your evaluation data to optimize routing for your specific use case.

This endpoint allows you to train a domain-specific router that learns which models perform best for different types of queries in your application. The router analyzes your evaluation dataset, clusters similar queries, and learns model performance patterns.

Training Process:

1. Upload a CSV file with your evaluation data
2. Specify which models to route between
3. Define the evaluation metric (score column)
4. The system trains asynchronously and returns a preference_id
5. Use the preference_id in model_select() calls once training completes

Dataset Requirements:

• Format: CSV file
• Minimum samples: 25 (more is better for accuracy)
• Required columns:
  • Prompt column (specified in prompt_column parameter)
  • For each model: {provider}/{model}/score and {provider}/{model}/response

Example CSV structure:

prompt,openai/gpt-4o/score,openai/gpt-4o/response,anthropic/claude-sonnet-4-5-20250929/score,anthropic/claude-sonnet-4-5-20250929/response
"Explain quantum computing",0.95,"Quantum computing uses...",0.87,"Quantum computers leverage..."
"Write a Python function",0.82,"def my_function()...",0.91,"Here's a Python function..."

Model Selection:

• Specify standard models: {"provider": "openai", "model": "gpt-4o"}
• Or custom models with pricing: {"provider": "custom", "model": "my-model", "is_custom": true, "input_price": 10.0, "output_price": 30.0, "context_length": 8192, "latency": 1.5}

Training Time:

• Training is asynchronous and typically takes 5-15 minutes
• Larger datasets or more models take longer
• You'll receive a preference_id immediately
• Check training status by attempting to use the preference_id in model_select()

Best Practices:

1. Use diverse, representative examples from your production workload
2. Include at least 50-100 samples for best results
3. Ensure consistent evaluation metrics across all models
4. Use the same models you plan to route between in production

Related Documentation: See https://docs.notdiamond.ai/docs/adapting-prompts-to-new-models for detailed guide.