pass@k is the probability that an agent succeeds on at least one of k tries. pass^k is the probability that it succeeds on all k tries. That…
A demo tells you an agent can do a task once. Production asks whether it can do the task a thousand times unattended without a costly mistak…