An agent that scores well on your task benchmark can still delete the wrong file, close a ticket it should have escalated, or push a change…
Most teams ship an MCP agent to production the same way they ship a feature: it passed the demo, the tests are green, and someone says "look…
The model at the top of the leaderboard is not necessarily the model you should ship. Leaderboards rank one axis, task score, on someone els…
OpenRouter's Ori Eval is the most credible self-serve agent eval we have seen ship this year, and it valid…
Public MCP benchmarks cannot tell you whether your agent picks the right tool from your MCP servers. They test generic, fixed toolsets under…
Scoring only the final answer of an agent is dangerous. An agent can return the correct result while calling the wrong tool, retrying five t…
Yes, the harness affects LLM performance, and it affects it a lot. You do not ship a model. You ship a system: a prompt, an agent loop, tool…
pass@k is the probability that an agent succeeds on at least one of k tries. pass^k is the probability that it succeeds on all k tries. That…
A demo tells you an agent can do a task once. Production asks whether it can do the task a thousand times unattended without a costly mistak…