Run every vendor through one identical exam built from your own data, score it blind, and report results with error bars. That is the whole trick to a bake-off that holds up when procurement, legal, and your board start asking questions.
If you are choosing between LLM vendors, the demos will not decide it for you. They are designed to win. Here is how to build a comparison you can actually defend.
TL;DR
- Vendor-run demos are sales tools. They pick the prompts, the data, and the framing, so they cannot tell you who wins on your work.
- A fair bake-off uses a private, contamination-free corpus drawn from your own data, so no vendor has seen the questions.
- Use seeded deterministic item selection so every vendor sits the identical exam. Same items, same order, same rubric.
- Score blind. Graders should not know which vendor produced which answer.
- Report paired comparisons with clustered confidence intervals, not raw win counts. A 2 point gap inside the error bars is a tie.
- The right party to run it is an independent third party. A vendor cannot grade itself, and neither can the team that already has a favorite.
Why do vendor demos mislead you?
Because the vendor controls every variable that matters.
They choose the prompts that flatter their model. They use example inputs that sit inside their training distribution. They tune the system prompt and retrieval in ways you will not replicate. And they show you the good runs, not the median run. None of this is fraud. It is just what a demo is for.
The result is a comparison where each vendor looks best on their own turf. You learn who demos well. You do not learn who performs on your actual traffic, which is the only question procurement cares about.
What makes a bake-off fair?
Three things: a shared exam, a private corpus, and blind scoring.
One shared exam. Every vendor answers the exact same items, in the same order, under the same rubric. If Vendor A gets 200 support tickets and Vendor B gets a different 200, you have measured two different tests. Use seeded deterministic item selection so the sample is reproducible and identical for everyone. Anyone can rerun the seed and get the same exam.
A private, contamination-free corpus. Build the test set from your own data, not public benchmarks. Public benchmarks leak into training sets, so a high score can mean the model memorized the answer rather than reasoned to it. A corpus assembled from your tickets, documents, or transactions cannot have been trained on, which is the only way to know you are measuring capability instead of recall. This is the part most in-house attempts skip, and it is the part that makes the number trustworthy. More on that in contamination-free evaluation.
Blind scoring. Graders, whether human or an LLM judge, should not see which vendor produced an answer. Anonymize and shuffle before scoring. Knowing the brand changes how a grader reads the same sentence.
What should you measure, and how do you score it?
Pick axes that map to your use case, then weight them by what the deployment actually needs.
For most enterprise deployments the axes are some mix of task accuracy, faithfulness to source, instruction following, tool or function selection, latency, cost per task, and refusal or safety behavior. Do not average a single blended score and call it a winner. A model that is cheapest but wrong is not a bargain, and a model that is most accurate but too slow for your SLA is not deployable.
Score each axis separately, then apply weights that reflect the job. This is also why there is rarely one best model across everything. Different tasks reward different axes, so the honest output is a per-task recommendation, not a crown.
Here is the shape of a procurement scorecard. Numbers are illustrative.
| Axis | Weight | Vendor A | Vendor B | Vendor C |
|---|---|---|---|---|
| Task accuracy | 30% | 0.81 (+/-0.04) | 0.79 (+/-0.04) | 0.74 (+/-0.05) |
| Faithfulness to source | 25% | 0.88 (+/-0.03) | 0.90 (+/-0.03) | 0.83 (+/-0.04) |
| Instruction following | 15% | 0.92 (+/-0.02) | 0.85 (+/-0.03) | 0.87 (+/-0.03) |
| Tool selection | 10% | 0.76 (+/-0.05) | 0.71 (+/-0.05) | 0.79 (+/-0.05) |
| p95 latency | 10% | 2.1s | 3.4s | 1.8s |
| Cost per 1k tasks | 10% | $ | $$ | $ |
| Weighted verdict | Lead on accuracy | Lead on faithfulness | Fastest, cheapest |
The verdict row matters more than any single cell. It tells procurement what each vendor is for, which is what a sourcing decision actually needs.
How do you keep the statistics honest?
Report error bars and compare in pairs. Otherwise a coin flip looks like a winner.
Two failures show up constantly in vendor decks. The first is reporting a single point score with no interval, so you cannot tell a real gap from noise. The second is comparing raw win counts across different samples, which double counts easy items and hides variance.
Do it this way instead. Run a paired comparison, where every vendor is scored on the identical item, so you are measuring the difference item by item rather than aggregate to aggregate. Then compute clustered confidence intervals, because items from the same document or the same customer are correlated and naive intervals will be too tight. If the gap between two vendors sits inside the overlapping intervals, call it a tie and let cost or latency break it. A defensible bake-off says “A beats C by a real margin, A and B are within noise” and shows the math.
Who should run the bake-off?
An independent third party, for the same reason you do not let a student grade their own exam.
A vendor cannot grade itself without a conflict. And your internal team is not neutral either once someone has championed a choice, or once the eval is built by the same people who will operate the system. Independence is structural, not a question of good intentions. The party that designs the corpus, seeds the selection, and scores the answers should have no stake in which vendor wins.
This is the core argument for buying evaluation rather than building it for procurement decisions. Your in-house harness is great for the development loop. It is not the right instrument for a sourcing decision that legal and finance will sign. We walk through that split in eval as a service vs in-house and the broader case in independent LLM evaluation. Our full approach lives on the method page.
FAQ
How big should the test corpus be? Big enough that your confidence intervals separate the vendors you care about. That depends on how close they are and how noisy the task is. Start with a few hundred items per task type, look at the interval widths, and expand where the gaps are still ambiguous.
Can we just use public benchmarks to compare vendors? No, not on their own. Public benchmarks are useful context but they leak into training data, so scores can reflect memorization. For a procurement decision you need a private corpus built from your data that no vendor has seen.
Should we score with humans or an LLM judge? Both have a place. An LLM judge is fast and consistent for scale, humans anchor the rubric and audit edge cases. Whichever you use, keep it blind to vendor identity and validate the judge against human labels on a sample.
What is the deliverable we actually get? A procurement scorecard: per-axis scores with confidence intervals, paired verdicts, and a per-task recommendation, plus the seeds and corpus definition so anyone can reproduce the run.
A bake-off is only worth running if the result survives the people who did not want to hear it. Build it on your own data, run it blind, report it with error bars, and have someone independent hold the pen.
Want a scorecard your procurement team will trust? Book a free eval diagnostic and we will scope the bake-off with you.