Use a public benchmark to survey the field and shortlist models. Use a private eval to decide what you ship. A benchmark is a shared public corpus that everyone runs. An eval is private, built from your own data, and specific to your job. They answer different questions, and using one where you need the other is how teams pick the wrong model with high confidence.
TL;DR
- A benchmark is a public shared corpus like MMLU, SWE-bench, or Terminal-Bench. An eval is a private test built from your own data for your own task.
- Public benchmarks are good for research progress and coarse screening. They are cheap, comparable across models, and fine for excluding weak candidates.
- They fail for production decisions because they are contamination-prone, harness-blind, and not measuring your task.
- A private eval built from your post-cutoff traffic is fresh, job-specific, and hard to game. It is the number that predicts what your users will see.
- Frontier teams already run private evals. Razorpay, Stripe, and Anthropic build their own rather than trust a leaderboard.
What is the difference between a benchmark and an eval?
A benchmark is public and shared. An eval is private and specific to you.
A public benchmark is a fixed corpus of tasks that the whole field runs against every model, so scores are comparable. MMLU tests broad knowledge, SWE-bench tests bug fixing on open-source repos, Terminal-Bench tests command-line agents. It gives everyone a common yardstick.
A private eval is a test set you assemble from your own inputs, graded against what correct means for your product. Nobody else runs it. It is not on the internet. It reflects your traffic, your edge cases, and your definition of good, so it predicts behavior on the work you actually do.
Definition. Benchmark: a public shared corpus used to compare models across the field. Eval: a private corpus, specific to one organization, used to decide what that organization ships.
What are public benchmarks good for?
Public benchmarks are good for tracking research progress and for coarse model screening. Inside that lane they are the right tool.
Research needs a common yardstick. When a lab reports a jump on SWE-bench, the field can compare it to every prior model because the corpus is fixed and shared. That comparability is exactly what drives progress and lets you follow it without running anything yourself.
Screening is the second job. When a new model ships, a benchmark tells you fast whether it is even in the conversation. A model far below the pack on coding is not a candidate for your coding workload, and you learned that for the price of reading a table. Benchmarks are a cheap, defensible filter for excluding clearly weak models. Keep using them for what they do well.
Why do public benchmarks fail for production decisions?
They fail for three structural reasons: contamination, harness-blindness, and the fact that they are not your task. None of these is fixed by picking a better benchmark.
Contamination. Public benchmarks leak into training data. Questions and answers get scraped, quoted, and paraphrased across the web, then swept into the next pretraining run. The model recalls items instead of solving them, so the score measures memorization. Detection is unreliable, so you cannot clean your way out. We cover the mechanism in benchmark contamination.
Harness-blindness. A leaderboard number is the model plus a specific scaffold: prompt template, parsing, retries, tool wiring. Two harnesses running the same model on the same benchmark can report very different scores. The published number tells you what someone else’s harness achieved, not what yours will. Your production system is a different harness, so the number does not transfer.
Not your task. MMLU is not your support inbox. SWE-bench is not your codebase. A benchmark measures a generic task on generic inputs, while your users bring your distribution, your formats, and your failure modes. A model can top a benchmark and still miss the cases that matter to you, because those cases were never in the test.
Together, these mean a public benchmark can look decisive and still send you to the wrong model. It is the right tool for the wrong decision.
What does a private eval give you?
A private eval gives you a fresh, job-specific, hard-to-game measurement of the decision in front of you.
It is fresh because you build it from data created after the model’s training cutoff, ideally your own recent traffic. Post-cutoff data could not have been in training, so the set is contamination-free by construction rather than by a detector’s guess. See contamination-free evaluation for how to build that corpus.
It is job-specific because the items are your inputs and the rubric is your definition of correct. You score the model on the work it will actually do, through a harness that resembles the one you will ship, so the result predicts production instead of approximating it.
It is hard to game because the set is private. No vendor can train against a corpus they have never seen, and no score can be inflated by memorization. That is why frontier teams build their own. Razorpay stood up private evals in August 2026, and Stripe and Anthropic run internal evals rather than trust a leaderboard for decisions that carry real risk.
Private eval vs public benchmark: how do they compare?
| Dimension | Public benchmark | Private eval |
|---|---|---|
| Ownership | Shared, public corpus | Private, yours alone |
| Data source | Fixed dataset (MMLU, SWE-bench) | Your post-cutoff traffic |
| Contamination | Prone, hard to detect | Fresh by construction |
| Harness | Someone else’s scaffold | Your production harness |
| Task fit | Generic | Your exact job |
| Gameability | High (train to the test) | Low (set is unseen) |
| Comparability | High across all models | Specific to you |
| Best for | Research, coarse screening | Production go/no-go |
Neither column is better. They answer different questions. The mistake is reading a benchmark row as if it were the eval row.
How do you decide which one to use?
Match the tool to the stakes. The decision is not benchmark or eval, it is benchmark then eval.
Use a public benchmark when:
- You are following research or surveying what a new model class can do.
- You are shortlisting, and want to exclude models that are clearly out of their depth.
- The decision is cheap to reverse and nobody outside your team needs to trust it.
Use a private eval when:
- The result gates a launch that touches money, safety, or customers at scale.
- You are separating the top two or three candidates, where benchmark noise is larger than the gap.
- Someone outside your team must trust the number: a regulator, a board, a procurement reviewer.
- The task is yours in a way no public set captures, such as your agent trajectories or tool selection.
In practice you run both, in order. Benchmarks narrow a dozen models to three. A private eval, built from your data and run through your harness, picks the one you ship. Our method is built around producing that second number.
FAQ
Is a benchmark the same as an eval? No. A benchmark is a public shared corpus everyone runs, so scores are comparable across the field. An eval is a private test built from your own data for your own task. Benchmarks compare models in general; evals decide what you ship.
Can I use public benchmarks if I have no training data of my own? For screening, yes. For a production go/no-go, no. Build a small private eval from recent traffic, support logs, or a labeled sample of real inputs. A few hundred well-chosen, post-cutoff items beat any leaderboard for your specific launch.
Why not trust a leaderboard when models are within a point or two? At that margin, contamination and harness differences are larger than the gap, so the ranking can reorder run to run. When candidates are close, the only tiebreaker that predicts production is a private eval on your data.
Do private evals replace public benchmarks entirely? No. Keep benchmarks for research and coarse screening. They are cheap and comparable. Add a private eval on top for the decisions that carry real risk. The two are complementary, not competing.
Public benchmarks tell you who is in the race. A private eval tells you who wins yours. If you have a model choice or a launch coming and want a number you can stand behind, book a free eval diagnostic and we will scope a contamination-free corpus from your own traffic.