pass@k is the probability that an agent succeeds on at least one of k tries. pass^k is the probability that it succeeds on all k tries. That one operator flip is the difference between a metric that flatters your demo and a metric that tells you whether you can leave the agent unattended. If you are shipping autonomous agents and reporting pass@k, you are measuring the wrong thing.
TL;DR
- pass@k measures capability: can the agent get it right at least once in k attempts?
- pass^k measures reliability: can the agent get it right every single time across k attempts?
- For autonomous agents, pass^k is the honest metric. Nobody reruns a production agent 10 times and keeps the one good answer.
- Reliability compounds badly. A step that works 95% of the time succeeds across 20 sequential steps only about 36% of the time (0.95^20 is about 0.358).
- That compounding is why agents that demo cleanly still fail in production.
- Measure per-step reliability, then multiply. Book a free eval diagnostic on our pricing page.
What is the difference between pass@k and pass^k?
Both metrics run k independent attempts at a task. They differ in how they count success.
pass@k counts a win if any one of the k attempts succeeds. It answers a capability question: is the ability there at all? This is the right metric when a human is in the loop, sampling, retrying, and picking the best output. It is also the metric most model reports lead with, because sampling more times only ever makes the number go up.
pass^k counts a win only if every one of the k attempts succeeds. It answers a reliability question: can the agent do this consistently, without supervision? This is the metric that matters when the agent runs on its own and each run has to stand on its own.
The gap between them is not academic. On the same underlying model, pass@k rises toward 100% as you add attempts, while pass^k falls toward 0%. Same system, same task, opposite trajectories. Which curve you report decides whether your eval tells the truth about autonomy.
Why does pass@k flatter and pass^k tell the truth?
pass@k is built for a world with a safety net. A developer generates five candidate functions and keeps the one that passes tests. A researcher samples ten solutions and grades the best. In that world, one success out of k is a genuine win, because a human harvests it.
Autonomous agents do not live in that world. When an agent books the travel, files the ticket, or moves the money, there is no human quietly discarding the four bad runs. The one run that happens is the one that ships. So the honest question is not “can it succeed once in k tries?” It is “what fraction of the time does it succeed when it only gets one try, repeated?” That is pass^k.
This is why a system can look production-ready under pass@k and fall apart in the field. The demo you saw was, in effect, a pass@k sample: the best run someone chose to show you. Production is pass^1 repeated thousands of times, which is exactly what pass^k models.
pass@k vs pass^k at a glance
| Dimension | pass@k | pass^k |
|---|---|---|
| Success rule | At least one of k attempts succeeds | All k attempts succeed |
| Question answered | Is the capability there? | Can it run unattended? |
| As k grows | Rises toward 100% | Falls toward 0% |
| Right when | A human samples and picks the best | The agent acts autonomously |
| Failure it hides | Inconsistency between runs | Nothing; it exposes inconsistency |
| Best used for | Model capability research, code assist with review | Agent deployment readiness, SLAs |
Why does agent reliability compound?
An autonomous agent almost never does one thing. It chains steps: read state, pick a tool, pass parameters, act, check the result, act again. Each step has its own success rate, and the steps are sequential. To finish the task, every step has to land.
When independent probabilities have to all succeed, you multiply them. A single 95% step feels safe. Twenty of them in a row do not, because you are computing 0.95 to the 20th power, not 0.95. That is the compounding trap, and it is the mechanism behind pass^k. A long agent trajectory is a pass^k measurement whether you name it or not.
A worked reliability example
Take an agent whose every step succeeds 95% of the time. That is a strong per-step number. Watch what happens as the task gets longer, assuming steps are independent.
| Per-step reliability | Steps | End-to-end success (reliability^steps) |
|---|---|---|
| 95% | 1 | 95% |
| 95% | 5 | about 77% |
| 95% | 10 | about 60% |
| 95% | 20 | about 36% |
| 99% | 20 | about 82% |
| 99.9% | 20 | about 98% |
The headline row is 95% per step across 20 steps landing at roughly 36% (0.95^20 is about 0.358). A demo that shows off one clean 20-step run is showing you a 36% event and calling it the product. Run it three times unattended and, on average, two of those runs break somewhere.
The bottom two rows show the only way out. You do not fix a long-horizon agent by adding retries around a shaky 95% step. You fix it by driving per-step reliability from 95% toward 99% and beyond, because the exponent is unforgiving. Cutting the per-step failure rate by 5x (95% to 99%) roughly doubles end-to-end success over 20 steps.
How do you measure pass^k for an agent?
Reporting an honest reliability number takes a few disciplines.
- Fix k to how the agent actually runs. If a task runs once with no human review, your headline reliability is pass^1 measured over many independent tasks. If an orchestrator legitimately retries three times before giving up, pass^3 on that sub-task is fair. Match k to the real retry policy, not to whatever makes the chart look good.
- Decompose the trajectory and measure each step. End-to-end pass^k tells you there is a problem; per-step reliability tells you where. Score tool selection, parameter accuracy, and result checks step by step. This is the heart of agent trajectory evaluation, and it is what lets you find the one weak step dragging the product down.
- Use enough independent samples. A reliability estimate from five runs is noise. You need enough repeated, independent trials per task to separate a real 95% step from a lucky 100%-in-five-runs step.
- Keep the eval set clean. A reliability number is worthless if the tasks leaked into training, because the agent memorized rather than reasoned. Independent, contamination-free measurement is the point of independent LLM evaluation, and it is how we run every diagnostic. See our method for how we hold that line.
What does this mean for agent deployment?
Set reliability targets end to end, not per step, and back them out through the exponent. If your product needs 90% task success across a 20-step workflow, you need roughly 99.5% per step, not 95%. Say that out loud in planning and the scope of the reliability work becomes obvious.
Then shorten trajectories where you can, because every step you remove is one fewer factor in the product. And gate deployment on pass^k, not pass@k. A model that tops the pass@k leaderboards can still be the wrong choice for autonomy if its runs are inconsistent.
FAQ
Is pass@k ever the right metric? Yes, when a human is in the loop. Code assistants where a developer reviews suggestions, or research pipelines that sample and grade, are genuine pass@k settings. The mistake is using it for agents that act without review.
What value of k should I use for pass^k? Match it to how the agent runs in production. Single-shot autonomous tasks are pass^1 measured across many tasks. If your system genuinely retries n times, pass^n on that step is fair. Do not inflate k to soften the number.
My per-step reliability is 95%. Is that good? For one step, fine. For a 20-step chain, it means roughly 36% end-to-end success. Whether 95% is good depends entirely on how many times you multiply it.
Does adding retries fix a low pass^k? Only if failures are independent and cheap to retry, and only for steps that can safely repeat. Retries do nothing for steps with side effects, like a payment or an email, where a wrong action cannot be taken back.
Capability is table stakes; reliability is the product. Measure pass^k, drive per-step reliability up, and keep your eval independent so the number means something. Book a free eval diagnostic on our pricing page.