Everything you need to argue about evals, from first principles.

Strip away the vocabulary and an evaluation has exactly three parts: a use case (one job, narrowly defined, not “coding” but “reviewing a pull request in our payments service”), a corpus (real examples of that job, each one an item), and a scorer (a repeatable way to turn an output into a number).

A benchmark is a public, shared corpus, MMLU, SWE-bench, Terminal-Bench. An eval in the sense that matters commercially is private and specific to one organisation. That distinction is the whole game, for two reasons that each get a section below.

A public benchmark gets published, discussed, copied into blogs, mirrored on GitHub. The next generation of models trains on a crawl of the web that now contains the benchmark, and often its answers. The score goes up; the ability does not.

This isn't fringe. GPT-3's own paper flagged overlap on most of its benchmarks; Llama 2 reported meaningful MMLU contamination; recent audits find roughly a quarter to a third of MMLU items contaminated. Detection methods exist, Min-K% Prob, membership inference, n-gram overlap, but multiple 2024–2026 surveys find they work poorly and inconsistently.

So the honest claim is never “we detect contamination.” It's structural: a corpus built from your private artefacts, after a model's training cut-off, cannot have leaked into that model. Freshness beats forensics.

A benchmark scores a model in isolation. In production the model sits inside a harness: a prompt, a scaffold that loops it, tools it can call, a retrieval layer, a policy engine, a retry strategy, a merge gate. Every layer is a knob, and every knob interacts with the model.

The evidence is stark. In 2026 LangChain improved a coding agent's Terminal-Bench 2.0 score by 13.7 points, from roughly thirtieth place to the top five, with the model held constant. Only the harness changed. That's a bigger move than most model upgrades.

Which means a leaderboard ranking tells you almost nothing about what will happen inside your stack. A model that loses publicly can win privately, and the only way to know is to measure your model, on your harness, doing your job.

Always reach for the cheapest trustworthy scorer first. If an item has a known-correct output, an extracted total, a resolved category, a passing test, the right tool call, score it deterministically. Free, instant, reproducible, immune to every judge bias. Most “we need an LLM to judge this” problems are golden-answer problems in disguise.

For open-ended output you use another model as the grader, and it has three well-documented lies: position bias (favours the first answer, fix by running both orders), verbosity bias (favours the longer answer, fix by length-controlling the rubric), and self-preference (favours its own output, fix by never letting a model grade itself).

The structural fix is a jury: several smaller, diverse judges outperform one large judge, cost less, and let no single model's taste dominate. And before you trust any judge, you align it against human labels and measure agreement with a chance-corrected statistic, Cohen's kappa, weighted kappa, Krippendorff's alpha.

Error bars. A score is an estimate from a sample. 71% versus 68% on 200 items may be indistinguishable noise. Reporting a bare number without an interval is the most common error in the field.

Clustering. Ten questions from one document aren't ten independent observations, they share whatever makes that document easy or hard. Treating them as independent shrinks your error bars to a fiction. Compute variance at the level of the cluster.

Pairing. To compare two models, run both on the same items and analyse the per-item difference. Pairing removes item difficulty from the variance and can cut the sample size you need by a large factor, which is why seeded, deterministic item selection isn't fussiness.

pass@k is the probability of at least one success in k attempts, is this capability in there somewhere? Fine for research, and for workflows where a human picks the best of several drafts.

pass^k is the probability of succeeding on all k attempts, can I leave this unattended? For anything agentic and autonomous, this is the honest metric, and it is brutal.

Scoring only the final answer is adequate for a chatbot and dangerously insufficient for an agent. An agent that reaches the right answer by calling the wrong tool, retrying six times, leaking a record into a log, and burning forty cents has failed, and final-answer scoring records it as a pass.

Trajectory evaluation scores the path: tool-selection accuracy, parameter accuracy, trajectory efficiency, policy adherence, and cost and latency per completed task. This is exactly where agents fail, and where public MCP benchmarks, built on fixed public toolsets, can't help you, because they can't see your tools under your policy.

Scores near 100%. Your eval is too easy and has stopped carrying information. Retire it and build a harder one.

Scores improving while complaints don't. You're optimising the measurement, not the product. Classic Goodhart. Re-derive the rubric from fresh production failures.

Criteria drift. As people look at outputs, their idea of “good” quietly changes. The remedy is to re-align the judge against human labels on a schedule, not once at the start.