A demo tells you an agent can do a task once. Production asks whether it can do the task a thousand times unattended without a costly mistake. Those are different questions, and only the second one matters when you ship. Here is what to measure to answer it.
TL;DR
- A single successful run says almost nothing about reliability. Agents run repeatedly, and small per-step failure rates compound fast.
- Measure
pass^k, notpass@k. A step that is 95% reliable across a 20-step task lands near0.95^20 ~= 36%end-to-end success. - Score the trajectory, not just the final answer: tool-selection accuracy, parameter accuracy, trajectory efficiency, policy adherence, and cost and latency per completed task.
- Final-answer scoring hides wrong-path successes. A right answer reached the wrong way is a latent failure.
- Measure on a private, contamination-free corpus built from your own production traces, and report error bars with clustered confidence intervals.
- Set a reliability bar before you ship. See our method or book a free eval diagnostic on our pricing page.
Why do agent demos lie about reliability?
Because a demo is a single sample, and reliability is a distribution. When you watch an agent complete a task once, you are seeing one draw from a process that also produces failures you did not see.
The question is not “can it do this?” It is “how often does it do this correctly, on my inputs, without me watching?”
Two things inflate demo performance. First, demo tasks tend to resemble the examples the model saw during training, so success there tells you little about your actual workload. Second, a demo has a human in the loop who quietly retries, rewords, or picks the good run to show. Unattended production has none of that. You answer the real question by running the task many times and measuring the spread.
What does pass^k tell you that pass@k does not?
pass@k asks whether the agent succeeds at least once in k attempts. That metric rewards luck. It is the right question for a research benchmark where you can sample many times and keep the best, and the wrong question for production, where each run stands alone and a single bad run has real cost.
pass^k asks whether the agent succeeds on all k independent attempts. That is the reliability question. If you run a task unattended twenty times a day, you care that all twenty land, not that one of twenty did.
The math is unforgiving. A per-run success rate of 95% feels shippable until you compound it. Across 20 runs, 0.95^20 is roughly 36%. So an agent that looks “95% reliable” fails at least once on a typical day more often than not. That is why a small reliability gap turns into a large operational one. We go deeper in pass@k vs pass^k.
Which agent reliability metrics actually matter?
Final-answer correctness is necessary but not sufficient. An agent can return the right answer while calling the wrong tool, retrying five times, and reading a record it was never allowed to touch. Final-answer scoring passes all three, then those failures page you later.
Score the trajectory instead: the ordered sequence of decisions the agent makes on the way to an outcome. These are the metrics that predict unattended behavior.
| Metric | What it measures | Why it matters unattended |
|---|---|---|
| Final-answer correctness | Is the last output right | Necessary, but hides how it got there |
| Tool-selection accuracy | Did it pick the right tool for the state | Wrong tool, right answer is a latent bug |
| Parameter accuracy | Were arguments valid and correctly typed | Bad params drive silent retries and errors |
| Trajectory efficiency | Steps and retries versus the ideal path | Thrashing burns cost and latency budget |
| Policy adherence | Did it stay inside allowed actions and data | The failure that becomes a compliance incident |
| Cost per completed task | Spend across all attempts, not per call | The number your finance team will ask about |
| Latency per completed task | Wall-clock to a correct result | What a downstream system or user waits on |
The last two are worth stressing. Report cost and latency per completed task, not per call. See agent trajectory evaluation for how these sub-scores combine.
Why measure cost and latency per completed task?
Because retries are invisible in per-call accounting. An agent that succeeds only after four attempts costs you five API calls per result, not one, and the extra four do not show up if you average across calls instead of tasks.
Per-completed-task numbers also make trade-offs legible. A model that is more accurate but doubles cost per completed task might still be right for a high-stakes workflow and wrong for a high-volume one. You cannot make that call unless the denominator is “tasks that actually finished correctly.” Efficiency and cost move together: the agent that thrashes through extra steps is the same one inflating your bill and your tail latency.
Why measure on a production-derived corpus?
Because public benchmarks tell you how the agent does on someone else’s work, and there is no way to know how much of that benchmark leaked into training. A high public score can reflect memorization rather than capability. That is benchmark contamination, and it makes public numbers unsafe as a shipping signal.
Build your eval corpus from your own production traces instead. Real user inputs, real edge cases, real distribution. A private corpus is contamination-free by construction, because the model could not have trained on data that did not exist yet, and it measures the workload you actually run rather than a proxy for it. This is the core of our method.
Then report uncertainty. A single accuracy number with no error bars is not a measurement, it is an anecdote. Because tasks from the same user or session are correlated, use clustered confidence intervals so your error bars reflect real variance rather than an inflated sample size. That is what lets you say one agent is genuinely better than another instead of noise apart.
How do you set a reliability bar before shipping?
Decide the bar before you see the score, so the number cannot talk you into shipping. Work backwards from the cost of a failure.
- Define the task and the unattended run volume. How many times per day, with no human checking each one?
- Set the
pass^ktarget for that volume. If a single failure is expensive,kis large and the bar is high. - Set floors on the trajectory metrics that matter for your domain. For regulated work, policy adherence is a hard gate.
- Set a ceiling on cost and latency per completed task.
- Measure on your production-derived corpus with clustered CIs. Ship only if the lower bound clears the bar.
The discipline is committing to the threshold first. An agent that clears a bar you set afterward has told you nothing.
FAQ
What is a good reliability number for a production agent?
There is no universal number. It depends on run volume and the cost of a single failure. Derive it: pick your pass^k target from how many times the agent runs unattended and how expensive one bad run is. High-volume or high-stakes work needs a high per-run rate because failures compound.
Is final-answer accuracy enough to trust an agent? No. Final-answer scoring passes agents that used the wrong tool, retried repeatedly, or touched data they should not have. You need trajectory metrics to catch wrong-path successes before they become incidents.
Why not just use published benchmark scores? Because you cannot verify how much of a public benchmark leaked into training, so a high score may reflect memorization. Measure on a private corpus built from your own traces to get a contamination-free signal.
Can we run these evals ourselves? You can, and many teams do. The trade-offs of building versus buying are covered in eval-as-a-service vs in-house. The hard parts are an independent corpus and honest statistics, which are also the parts easiest to get wrong when you grade your own work.
Reliability is not something you observe in a demo, it is something you measure against a bar you set in advance. Bring your production traces and a threshold, and we will tell you whether your agent clears it. Book a free eval diagnostic on our pricing page.