An agent that scores well on your task benchmark can still delete the wrong file, close a ticket it should have escalated, or push a change…
The model at the top of the leaderboard is not necessarily the model you should ship. Leaderboards rank one axis, task score, on someone els…