Model evaluation

Pick the right model for a task by running it against the right benchmark.

Picking a model is the most expensive decision you'll make as an AI team. The wrong choice locks in months of wasted iteration on a foundation that was never going to work. Stratix's model-evaluation use case is the disciplined version of that decision.

The shape of the work

1. Frame the question. What does your feature actually need? Reasoning, structured output, multi-turn dialog, code, multilingual, multimodal? Don't pick a model on vibes.
2. Find the right benchmarks. Browse the benchmarks catalog. Pick 1-3 that map to your task. (Most teams pick too many; resist.)
3. Browse the public leaderboard. Compare models head-to-head. Get to a shortlist of 3-5.
4. Run a private evaluation on your data. Public scores tell you what's strong in general. Your data tells you what's strong for you. Stratix Premium runs a private evaluation against your dataset.
5. Read the verdict, not the headline. A model that scores 2% higher overall but 15% lower on your one critical dimension is the wrong pick.

Why it works on Stratix

• Public catalog gives you the shortlist for free. No multi-week vendor proof-of-concepts to find candidate models.
• Private evaluations on your data give you the verdict. Public leaderboards don't see your prompts or your distribution.
• Compare models is built in. Side-by-side score tables with confidence intervals.
• Costs are visible up-front. ECU consumption is shown before you run.

Tools you'll use

• Stratix Public — Models catalog
• Stratix Public — Compare models
• Stratix Premium — Evaluations
• Stratix Premium — Benchmarks
• SDK: client.evaluations.create()

Outcomes you should see

You'll know this is working when:

• Model selection time drops from weeks to hours. Public catalog narrows; private eval decides.
• Every model decision has a citable evaluation ID. No "we picked X because it felt better."
• Re-evaluating a new candidate model takes under 1 hour. Including standing up the eval against your dataset.
• Cost and latency are part of the choice, not afterthoughts. You can tell the team "we picked the cheaper model because at 95% of the quality it's 3× cheaper."

Anti-patterns

• Benchmark cargo-culting. "Everyone uses MMLU" is not why MMLU is right for your task.
• One-shot picking. The model frontier moves quarterly. Re-evaluate the candidates as new models drop.
• Ignoring cost and latency. A 2% accuracy gain at 4× cost is rarely worth shipping.

Where to next

• Concept: Models and benchmarks
• Tutorial: First evaluation in 10 minutes
• Workflow: Compare
• Cookbook: model-selection recipes