Most teams ship an MCP agent to production the same way they ship a feature: it passed the demo, the tests are green, and someone says “looks good.” That is not a sign-off, it is a hope. A pre-deployment checklist for an agent has to answer six specific questions before it touches real tools and real data, and this post is that checklist for a technical leader who has to put their name on the decision.
TL;DR
- An MCP agent going to production needs six checks: tool selection under your policy, parameter and schema correctness, policy adherence on forbidden actions, reliability across repeated runs (pass^k, not pass@k), contamination-free test data, and an independent sign-off.
- A demo that works once tells you almost nothing about production behavior. Reliability compounds across steps, so a 95 percent per-step model can fail the full task most of the time.
- Score does not predict safety. A model can complete tasks well and still take a forbidden action under pressure, so policy adherence has to be its own gate, not a blended number.
- Public and vendor benchmarks cannot sign off your agent, because they run generic tools on generic tasks, not your MCP servers under your rules.
- Use this as a literal checklist before go-live, and keep the evidence, because “we checked it” is not an answer an auditor or a customer will accept without a record.
Why isn’t a passing demo enough to ship?
Because a demo is one run, on one path, usually the happy path someone rehearsed.
An MCP agent in production faces ambiguous tickets, malformed inputs, tool responses it has not seen, and users who phrase things differently than your test script. A demo tells you the agent can succeed. It tells you nothing about how often it succeeds, what it does when a step is ambiguous, or whether it will take an action you did not intend to authorize. Sign-off requires evidence across all three, not a recording of the best run you got.
The six-item checklist
Each item below is a distinct question with its own failure mode. Passing one does not imply passing another, so grade them separately.
| # | Check | What it answers | Where a failure hides |
|---|---|---|---|
| 1 | Tool selection | Does the agent pick the right tool from your MCP servers, not a generic one | Final-answer scoring, which hides a wrong tool behind a right-looking answer |
| 2 | Parameters and schema | Are arguments correctly typed, referentially valid, and complete | Right tool, wrong customer ID, malformed date, missing required field |
| 3 | Policy adherence | Does it follow required steps and refuse forbidden or destructive actions | A task that scores well while skipping an approval or taking a forbidden action |
| 4 | Reliability (pass^k) | Does it succeed k times in a row, not just once in k tries | Unattended agents that only need to fail once to cause an incident |
| 5 | Contamination control | Is the test data unseen by the model, so the score reflects capability | Public or leaked eval data inflating a score through memorization |
| 6 | Independent sign-off | Was this graded by a party other than the team that built the agent | Self-graded evals that pass because they were written to pass |
1. Tool selection
Does the agent choose the correct tool, from the tools you actually expose, not a generic public toolset. When you connect several MCP servers the model may see dozens of overlapping tools, and the wrong one can produce an answer that still looks right. This is the first link in the trajectory, and if it breaks, everything downstream inherits the error. See MCP tool selection evaluation for how to build this test against your own servers.
2. Parameters and schema
A right tool called with a wrong or malformed argument is still a failure, and it is a common one. Validate types, enums, required fields, and referential correctness, such as whether the customer ID in the call actually belongs to this session. This is checkable deterministically against your schemas, so there is no excuse for skipping it before go-live.
3. Policy adherence
Does the agent follow required approval steps, and does it refuse forbidden or destructive actions when a task pressures it toward one. This is the check most teams skip because it does not show up in a normal task-completion score. It should not be folded into a blended quality number, because our own testing shows score and safety are not correlated: across four Claude 5 models on a 108-task suite, Haiku 4.5 committed 5 policy violations, including closing unresolved tickets and deleting a file it was told to archive, while Opus 5 committed zero, and this had nothing to do with which model scored higher on the task itself. See policy-violation testing for how to build this gate.
4. Reliability across repeated runs
An agent that succeeds once in three tries is not the same as one that succeeds three times in a row, and only the second is safe to run unattended. Pass@k (did it succeed in at least one of k attempts) is the wrong metric for anything that runs without a human checking every output. Pass^k (does it succeed k times in a row) is the one that maps to unattended reliability, and it degrades fast: a step that is individually 95 percent reliable, repeated across a 20-step trajectory, succeeds roughly 36 percent of the time end to end. See pass@k vs pass^k and measuring agent reliability in production for the math and what it means at scale.
5. Contamination control
If your test scenarios, or anything close to them, are public, you cannot tell whether a high score reflects genuine capability or memorization. This matters more, not less, for an agent that will touch real systems, because the failure mode you are trying to catch, the same edge case you tested for, is exactly the one contamination can mask. Use a private, frozen corpus your candidate model has not seen. See benchmark contamination and private vs public benchmarks.
6. Independent sign-off
The team that built the agent tends to write tests it already passes, and that is not a criticism, it is a structural conflict of interest. A regulator, a board, or a large customer asking “who checked this” does not accept “we checked it ourselves” as an answer. Sign-off needs a third party, ideally one running the eval inside your VPC so the data never leaves your boundary. See independent LLM evaluation and in-VPC LLM evaluation.
What does a completed checklist look like as evidence?
A durable, timestamped record per check, not a verbal confirmation.
For each of the six items: the scenario set used, the pass rate or violation count, the model and harness pinned for the run, and who ran it. That is what turns “we evaluated this” into something an auditor, a customer security review, or your own board can trace later. A sample report shows the shape of that evidence, and our agent safety checklist is the same six-item structure above as a working document you can run against your own agent.
What happens if you skip one of these?
The failure shows up in production instead of in testing, at a worse time and a higher cost.
Skip tool selection testing and you find out your agent silently calls the wrong endpoint under load. Skip policy adherence and you find out the hard way which model closes tickets it should not or deletes what it should archive. Skip pass^k testing and an agent that looked fine in a demo fails on the third unattended run of the week. None of these are hypothetical, they are the specific failure modes each check exists to catch, which is why treating this as a checklist rather than a vibe is the entire point.
FAQ
Do all six checks need to pass before go-live? Tool selection, parameters, and policy adherence should be hard gates, meaning a failure blocks deployment for that permission scope. Reliability and contamination control set your confidence level in the result. Independent sign-off is what makes the other five defensible to someone outside your team.
Can we run this checklist ourselves instead of independently? You can for early iteration, and you should, the same way you would run your own unit tests. But self-graded results are the wrong evidence for a sign-off decision that a regulator, a customer, or your own leadership will hold you to later. Use your own runs to fix problems, then get an independent run to certify the result.
How often do we need to re-run this after go-live? Any time the model, the harness, the tool schemas, or your policy changes. Each of those can move the result by as much as a full model swap, so a checklist that is a year old and untouched since is not evidence of anything current.
What is the fastest way to start? Scope a single high-risk workflow, the one with delete, payment, or irreversible actions, and run the six checks against just that workflow first. That gives you a real, defensible answer on your highest-stakes surface before you try to cover everything at once.
A demo tells you an agent can work. This checklist tells you whether it is safe to let it work unattended, in production, against your real tools and your real policy. Book a free eval diagnostic on our pricing page and we will run the six checks against your MCP agent before it ships.