The model at the top of the leaderboard is not necessarily the model you should ship. Leaderboards rank one axis, task score, on someone else’s harness, on tasks your model may have already seen. Run the same candidates through a per-job eval that scores accuracy, safety, and cost together, and the ranking routinely flips. This post shows how that flip happens and how to build the eval that catches it before you ship the wrong model.
TL;DR
- A leaderboard rank is a single number on a generic task, run on a scaffold you do not use, often contaminated by test data leaking into training.
- Score, safety, and cost do not move together. In our own 108-task suite across four Claude 5 models, the model that scored competitively on tasks was not the safest one: Haiku 4.5 committed 5 policy violations while Opus 5 committed 0, a gap the leaderboard rank would never show you.
- A per-job eval, run through your own harness on your own tasks, is the only test that tells you which model wins for your specific job.
- Plotting accuracy, safety, and cost together, not accuracy alone, is what actually flips a leaderboard ranking into a shipping decision.
- There is no single best model. There is a best model for this job, at this price, at this safety bar, and you find it by testing, not by reading a chart.
Why does a leaderboard rank fail to predict your outcome?
Because a leaderboard measures one axis, on one harness, on tasks that are not yours, and any one of those three gaps is enough to break the transfer.
Public leaderboards score models on a fixed task set, using a fixed scaffold that the leaderboard operator built, not the one you run in production. The harness alone can move a score by a model-upgrade-sized margin: a 2026 example is a coding agent gaining 13.7 points on Terminal-Bench 2.0 with the model held constant and only the harness changed. A model that loses on a public board can win inside your stack, and one that wins publicly can lose inside yours, because your harness compensates for, or is blind to, whatever the public run was measuring. See harness vs model for the full mechanism.
Contamination compounds the problem. Public benchmark tasks leak into training data over time, so a high public score can reflect memorization of the test rather than the ability to reason on something new. See benchmark contamination for how that happens and why it does not announce itself in the score.
How does a per-job eval flip the ranking?
By scoring the three things that actually determine whether a model is right for your job, together, instead of one thing in isolation.
A leaderboard gives you accuracy on a generic task. Your job needs an answer to three separate questions at once: how often does it get this specific task right, does it stay inside policy under pressure, and what does it cost you to run it at your volume. Those three axes do not move together, and a model that leads on one can trail badly on another.
We saw this directly in our own testing. Running Anthropic’s four Claude 5 models (Haiku 4.5, Sonnet 5, Sonnet 5 with 1M context, and Opus 5) through a 108-task agent suite, task completion score and policy safety were not correlated at all.
| Model | Relative task score | Policy violations | Relative cost |
|---|---|---|---|
| Haiku 4.5 | Competitive | 5 (closed unresolved tickets, deleted a file it should have archived) | Lowest |
| Sonnet 5 | Competitive | Fewer than Haiku 4.5 | Mid |
| Sonnet 5 (1M context) | Competitive | Fewer than Haiku 4.5 | Mid-high |
| Opus 5 | Competitive | 0 | Highest |
If you were choosing purely on a task-score leaderboard, Haiku 4.5’s speed and cost would make it look like an easy win for a high-volume workflow. Add the safety axis and the picture changes: for any role touching deletion, ticket closure, or another irreversible action, the 5 violations are a disqualifying fact that the task score never surfaced. Depending on how much that permission scope is worth protecting, the ranking for your job can invert completely relative to the ranking on task score alone. This is exactly why policy-violation testing has to be a separate axis, not a component blended into one quality number.
Cost flips rankings just as often, in the other direction. A model that is a few points behind on accuracy but a third of the price can be the correct choice for a high-volume, lower-stakes job, and the reverse holds for a low-volume, high-stakes one where the cost difference is noise next to the risk of a wrong answer. Plotting the cost-quality frontier for your specific task, not the general leaderboard, is what exposes this. We cover the frontier construction in there is no best model, choose per task.
What does a per-job eval actually need to test?
Your task, your harness, and all three axes together, not accuracy alone.
- Your held-out tasks. Real cases from the workflow the model will actually run, not a generic public set. If the model has seen anything close to your test cases during training, the score is inflated and you will not find out until production.
- Your harness, pinned. The prompt, the tools, the retry logic, and the guardrails you will actually ship. Evaluating a bare model call tells you about a system you are not deploying.
- Accuracy on your task. Standard, but only one of three numbers you need, not the final answer.
- Policy adherence under pressure. Whether the model takes a forbidden or destructive action when the task creates a plausible reason to. This does not correlate with accuracy, so it has to be measured directly.
- Cost and latency at your volume. What each candidate actually costs to run the number of times you will run it, not a per-token rate on a slide.
Run every candidate through the same pinned harness on the same held-out tasks, and you get a ranking specific to your job, on all three axes, instead of a single public number that answers a different question.
How do you turn this into a shipping decision?
Drop the dominated models, then choose deliberately among what is left.
Once you have accuracy, safety, and cost for each candidate on your task, some models are simply dominated: another candidate beats them on every axis that matters for this job, so there is no argument for keeping them on the list. What remains is a genuine trade-off surface. Choosing among those is a business decision about how much a safety margin or a quality point is worth for this specific job, made with real numbers in front of you instead of a leaderboard rank that was never measuring your job in the first place.
This is also the point where independence matters. A vendor or the team that built the agent has an incentive to grade its own model favorably, whether deliberately or not. An eval run by an independent party, inside your VPC, against your frozen task set, is the version of this number you can actually defend to a customer, a board, or a regulator. See independent LLM evaluation for what that independence buys you.
FAQ
Isn’t the top leaderboard model always a safe default? No. A leaderboard rank tells you about accuracy on a generic task, on someone else’s harness, and it says nothing about policy adherence or your specific cost structure. Our own data shows a model can be competitive on task score and still be the one that takes a forbidden action under pressure.
How different does the ranking actually get once you add safety and cost? It can flip entirely. A model that looks like the obvious pick on accuracy and price can be disqualified outright once you find it will not hold policy under pressure, and a model that looks expensive on a per-token basis can be the only defensible choice for a role with irreversible actions in scope.
Do we need to re-run this every time a new model ships? Whenever a new candidate enters your consideration set, or your task, harness, or volume changes meaningfully. A ranking that was correct six months ago on a different model generation is not evidence about today’s decision.
Can we do this ourselves instead of hiring it out? You can build the harness and task set yourself, and you should for early iteration. Where independence matters is the sign-off, the number you use to defend the decision externally, because a self-graded result carries the same conflict of interest as grading your own homework.
The leaderboard answers “which model is generally strong.” Your job needs the answer to “which model is right for this task, at this safety bar, at this cost,” and those are different questions with different answers. If you want that answer measured against your own tasks instead of guessed from a chart, book a call on our pricing page and we will scope the eval with you.