Scoring only the final answer of an agent is dangerous. An agent can return the correct result while calling the wrong tool, retrying five times, and reading a record it was never allowed to touch. Final-answer scoring passes all three. Trajectory evaluation is how you catch them.
TL;DR
- Trajectory evaluation scores the path an agent takes, not just whether the final answer is right.
- The core sub-scores are tool-selection accuracy, parameter accuracy, trajectory efficiency, policy adherence, and cost and latency per completed task.
- A right answer reached the wrong way is a latent failure. It looks green today and pages you in production.
- Policy adherence is a first-class metric, treated that way by benchmarks like tau-bench from Sierra. For regulated work it is not optional.
- Trajectory eval pairs directly with MCP tool-selection evaluation: the path is only as good as the tool choices inside it.
- Book a free eval diagnostic on our pricing page.
What is agent trajectory evaluation?
Agent trajectory evaluation scores every step an agent takes to reach an outcome, not only the outcome itself. The trajectory is the ordered sequence of decisions: which tool the agent picked, what parameters it passed, how many times it retried, what it read, and what it wrote.
A final-answer eval asks one question: is the last output correct? A trajectory eval asks a series of them at each step. Did the agent choose the right tool for this state? Were the arguments valid and minimal? Did it reach the answer efficiently, or thrash? Did it stay inside policy the entire time?
The distinction matters because agents are stateful. A single task is a chain of dependent decisions, and a wrong decision early can be masked by a lucky recovery later. Final-answer scoring only sees the recovery.
Why is scoring only the final answer dangerous?
Because agents fail silently. Here is a concrete case we see repeatedly.
A support agent is asked to refund an order. The correct refund amount lands in the final response, so a final-answer eval marks it green. Look at the trajectory and three problems appear:
- The agent selected
search_all_customersinstead oflookup_order_by_id, then filtered client-side. Wrong tool, right answer. - It retried the payment API four times because it passed the amount in dollars where the tool expected cents, correcting only on the fifth attempt.
- During the search it read a second customer’s full billing record that had nothing to do with the task. A leaked record.
Every one of these is invisible to final-answer scoring. In production, the wrong tool doubles your API cost, the retries blow your latency budget, and the leaked record is a compliance incident. The eval said pass. The system is broken.
This is the same reasoning behind pass@k versus pass^k. Passing once is not the same as passing reliably. An agent that gets the answer through a fragile, wrong path will not hold up across runs.
What are the sub-scores in a trajectory eval?
Break the path into measurable components. Each one isolates a different failure mode.
| Sub-score | Question it answers | Failure it catches |
|---|---|---|
| Tool-selection accuracy | Did the agent pick the right tool for each state? | Right answer via wrong tool |
| Parameter accuracy | Were the arguments correct, valid, and minimal? | Malformed calls, retries, over-fetching |
| Trajectory efficiency | Did it reach the goal without wasted steps? | Loops, thrashing, redundant calls |
| Policy adherence | Did it stay inside allowed actions and data? | Leaked records, out-of-scope writes |
| Cost and latency per completed task | What did the successful path actually cost? | Silent budget and SLA blowouts |
Notice the unit on the last row: per completed task, not per call. A cheap model that takes twelve steps can cost more than an expensive one that takes three. Trajectory efficiency and cost move together, and you only understand either by scoring the whole path.
Why is policy adherence a first-class metric?
Because in regulated work, how the agent got the answer is often the thing under audit. A refund that reads another customer’s data is a breach even when the number is right. An agent that writes to a system it had read-only scope for is a control failure even when the task succeeds.
Policy adherence checks that the agent stayed inside allowed actions, allowed data, and allowed order of operations across the entire trajectory. It is not a post-hoc filter on the final output. It is a per-step constraint. Benchmarks like tau-bench from Sierra treat policy adherence as a first-class score for exactly this reason: agents operating against real tools and rules must be graded on the rules, not only the result.
For teams under frameworks like the EU AI Act or bank model-risk regimes, this is where trajectory eval stops being a quality-of-life feature and becomes evidence. The path is the audit trail. If you cannot score it, you cannot attest to it.
How does trajectory eval catch what final-answer scoring misses?
It changes the question from “was the output right?” to “was every decision right?” Those are different tests, and the gap between them is where production incidents live.
Consider reliability across a multi-step task. If each of twenty steps is 95 percent correct in isolation, the chance of a clean end-to-end run is 0.95^20, which is roughly 36 percent. Final-answer scoring on a small sample can easily land inside the lucky 36 percent and report a healthy pass rate. Trajectory scoring inspects each step, so it surfaces the weak links that compound into that 64 percent failure tail.
That is the whole argument. Answer-level metrics measure luck at the end. Trajectory metrics measure competence along the way, which is what actually determines whether the agent holds up at scale.
How does trajectory eval connect to MCP tool selection?
A trajectory is a sequence of tool calls, so the quality of the path is bounded by the quality of the tool choices inside it. If your agent picks the wrong MCP tool, no amount of downstream cleverness makes the trajectory good.
That is why tool-selection accuracy is the first sub-score, and why trajectory eval and MCP tool-selection evaluation are two views of the same problem. Tool-selection eval zooms in on the single decision: given this state and this tool catalog, is the chosen tool correct? Trajectory eval zooms out to the sequence. You want both. Fix tool selection first, because a broken tool choice poisons every step that follows it.
For how we run these as an independent, contamination-free service inside your own VPC, see our method.
FAQ
Do I still need final-answer scoring? Yes. It is necessary but not sufficient. Keep it as the outcome gate, and add trajectory scoring underneath it so a correct answer cannot hide a broken path.
Can an LLM judge score trajectories? Yes, for the softer sub-scores like efficiency and reasonable tool choice. Pin the judge and validate it against human review. For hard constraints like policy adherence, prefer deterministic checks over judge calls where you can express the rule.
Is trajectory eval only for complex agents? Any agent that takes more than one action benefits. The more steps, the more room for a right answer to arrive through a wrong path, and the more a final-answer check misleads you.
How is this different from unit tests on tools? Unit tests check a tool in isolation. Trajectory eval checks the agent’s decisions about which tools to use, in what order, with what data, across a real task. Both matter, and they catch different failures.
Final-answer scoring tells you the agent got lucky once. Trajectory evaluation tells you whether it will hold up under load, under audit, and under scale. Book a free eval diagnostic on our pricing page and we will run your agent’s real trajectories, not just its answers.