Public MCP benchmarks cannot tell you whether your agent picks the right tool from your MCP servers. They test generic, fixed toolsets under generic policy. To know if your agent selects, parameterizes, and sequences your tools correctly, you have to build a private tool-selection eval against your own servers and your own rules. This post shows how.
TL;DR
- MCP tool selection evaluation measures whether an agent chooses the correct tool, with correct parameters, in the correct order, from the tools you expose.
- Final-answer scoring hides wrong-tool failures. An agent can reach a plausible answer while calling the wrong tool or skipping a required approval step.
- Score five sub-metrics: tool-selection accuracy, parameter accuracy, trajectory efficiency, policy adherence, and cost/latency per completed task.
- Public MCP benchmarks (MCP-Bench, MCPEval, MCP-Universe) use fixed public toolsets. None evaluate your private tools under your policy, and public tasks are contamination-prone.
- Build a private eval: freeze your tool schemas, write graded scenarios with known-correct tool calls, and score the trajectory, not just the output.
What is MCP tool selection?
Tool selection is the decision your agent makes when, given a task and a set of MCP tools, it picks which tool to call and how to call it.
Model Context Protocol servers expose tools with names, descriptions, and input schemas. When you connect three or four servers, the model may see dozens of tools with overlapping purposes. search_orders and lookup_invoice might both plausibly answer “find the customer’s last payment.” One is correct under your data model. The other returns something close enough to fool a shallow check.
Tool selection is the first link in the agent trajectory. If it breaks, everything downstream inherits the error.
Why does final-answer scoring hide wrong-tool failures?
Because a wrong tool can still produce a right-looking answer.
Most teams grade agents on the final response. That works for a chatbot. It fails for a tool-using agent, because the answer and the actions are different things. Consider a support agent asked to issue a refund:
- It calls
get_customercorrectly. - It skips
check_refund_eligibility, which your policy requires. - It calls
issue_refundand returns “Your refund is processed.”
The final answer is correct. The trajectory violated policy. Final-answer scoring gives this a pass. In production it is a compliance incident.
You cannot see this failure without inspecting the sequence of tool calls. That is why tool-selection evaluation is a subset of agent trajectory evaluation, not of output evaluation.
What sub-scores should an MCP tool-selection eval produce?
Score five dimensions per task. A single pass/fail number tells you nothing actionable.
| Sub-score | What it measures | Failure it catches |
|---|---|---|
| Tool-selection accuracy | Did the agent call the correct tool(s)? | Picking lookup_invoice when search_orders was needed |
| Parameter accuracy | Were arguments correct and well-typed? | Right tool, wrong customer ID or malformed date |
| Trajectory efficiency | Did it reach the goal without redundant calls? | Calling the same tool five times, looping |
| Policy adherence | Did it follow required steps and guardrails? | Skipping an eligibility or approval check |
| Cost / latency per completed task | Tokens, tool calls, and time to done | Correct but too slow or too expensive to ship |
Reporting these separately lets you fix the right layer. Low parameter accuracy is a schema or prompting problem. Low policy adherence is a guardrail problem. They need different fixes.
Why do public MCP benchmarks fall short?
They answer a different question than the one you have.
Public MCP benchmarks such as MCP-Bench, MCPEval, and MCP-Universe are useful for comparing models on generic tool use. But they share three limits that make them the wrong tool for validating your agent:
- Generic tasks. They test broad, public scenarios, not your workflows, your data model, or your edge cases.
- Fixed public toolsets. They evaluate against a predefined set of public tools. Your agent talks to your MCP servers, with your tool names, descriptions, and schemas. None of these benchmarks touch your tools.
- Contamination-prone. Public tasks and toolsets can leak into training data, so a high score may reflect memorization rather than reasoning on unseen problems.
A public MCP benchmark tells you a model is generally competent at calling tools. It cannot tell you the model calls your tools correctly under your policy. For that you need a private eval, ideally run by someone independent. See private vs public benchmarks for the full argument, and our method for how we keep evals contamination-free.
How do you build a private tool-selection eval?
Freeze your tools, write graded scenarios, and score the trajectory. Here is the sequence.
1. Snapshot your tool schemas. Export the exact names, descriptions, and input schemas your MCP servers expose. This is the surface the model reasons over. When you change a description, the eval must rerun, because tool selection is sensitive to wording.
2. Write scenarios with known-correct trajectories. For each task, define the correct tool(s), the correct parameters, the required policy steps, and any tool that must not be called. Cover the ambiguous cases where two tools overlap, since that is where selection breaks.
3. Run the agent and capture the full trace. Record every tool call, argument, and result in order, not just the final answer.
4. Score against the trajectory. Grade the five sub-scores above. Use deterministic checks where you can (was issue_refund preceded by check_refund_eligibility?) and reserve model-graded judgment for the fuzzy parts.
5. Fix the harness, not just the model. Tool selection is highly sensitive to scaffolding. In model-fixed testing, harness choice alone can swing agent scores materially: one reported example is a 13.7-point Terminal-Bench 2.0 gap from the agent framework with the model held constant. Hold your harness steady across runs, or you will misattribute a scaffolding change to the model.
Parameter accuracy and policy adherence
These two sub-scores are where most private tool-selection evals earn their keep.
Parameter accuracy is checkable against your schemas. Validate types, enums, required fields, and referential correctness (does the customer ID exist and belong to this session?). A right tool with wrong arguments is still a failure.
Policy adherence encodes your rules as trajectory assertions: certain tools require a prior check, some tools are mutually exclusive, some are forbidden without an approval token. This is also where compliance mapping lives, so an auditor can trace a control to a passing test.
How does this connect to production reliability?
Per-step accuracy compounds, so small tool-selection error rates become large task-failure rates.
If each of 20 steps in a trajectory is 95 percent reliable, the whole task succeeds about 0.95^20, roughly 36 percent of the time. Tool selection is one of those steps, and it gates the rest. This is also why you measure pass^k (does it succeed k times in a row) rather than only pass@k (does it succeed in at least one of k tries) for anything that runs unattended. See measuring agent reliability in production for how these numbers behave at scale, and our field notes for where we see them bite.
FAQ
Can I use MCP-Bench or MCPEval to validate my production agent? Use them to compare models in general. Do not use them to sign off your agent. They run fixed public toolsets and generic tasks, so they never touch your tools or your policy.
Do I need the actual MCP servers, or can I mock them? Mock the tool responses for deterministic scoring, but use the real tool schemas. Selection depends on the names and descriptions your servers advertise, so those must be exact.
How is tool-selection evaluation different from trajectory evaluation? Tool selection is one dimension of the trajectory. Trajectory evaluation also covers efficiency, ordering, and end-to-end cost. Selection is the first and most consequential step.
Why have an independent party run it? The team that built the agent tends to write tests it already passes, and public scores can be contaminated. An independent eval, run in your VPC against your frozen tools, gives you a number you can defend.
Public MCP benchmarks measure general competence, not whether your agent picks the right tool from your servers under your policy. A private, contamination-free tool-selection eval is the only thing that answers that. Book a free eval diagnostic on our pricing page and we will scope one against your MCP servers.