The method
Most of the engineering here goes into trust, not into calling models. This is the pipeline we build, the rules that make it defensible, and the statistics that separate a real practice from a weekend script.
01 The anatomy of an eval run
Your real items, PRs, tickets, transcripts, tool-call logs, sampled from production, stratified, PII-handled, frozen and versioned.
Deterministic. The same corpus and seed always picks the same items in the same order, so every model sits an identical exam.
Not the model in a vacuum. Your prompt, your agent loop, your tools, your retries, the assembly that actually runs in production.
Golden-answer deterministic checks wherever a right answer exists; a panel of diverse judges, aligned to your labels, for the open-ended rest.
We persist every judge vote, never a verdict. Change your decision rule six months later and re-score the entire history without spending a token.
Model × axis, each cell an estimate with a clustered confidence interval, the thing you actually buy.
Two properties make this sellable rather than scriptable: seeded selection, so any two runs compare item by item, and the raw-vote store, so a decision rule can change without re-spending a token.
02 The architecture
The framework knows nothing about any domain, it owns model access, orchestration, judging, corpus handling, and statistics. A plugin knows only its domain: what an item is, how the candidate is exercised, what the rubric says. Nothing below the boundary reaches up.
That constraint is why a new engagement is a configuration exercise, not a rebuild, and why roughly seven in ten evaluations we author need no custom code at all.
Plugins, mostly config
Framework, your IP, built once
03 How we score
When there's a right answer
An extracted total, a resolved category, a test suite that passes, a tool call that was or wasn't correct, score it with string match, numeric tolerance, or exact set comparison. Free, instant, perfectly reproducible, immune to every judge bias. Most 'we need an LLM to judge this' problems are golden-answer problems in disguise.
When it's open-ended
For 'was this review helpful' questions, we use a panel of diverse models against an atomic rubric, a jury beats a single large judge and no one model's taste dominates. Position bias is controlled by running both orders; verbosity by length-controlling the rubric; self-preference by never letting a model grade itself.
Before we trust any judge, we align it: a human labels a sample, the judge labels the same sample, and we measure agreement with Cohen's kappa or Krippendorff's alpha. If the judge doesn't agree with your expert, it isn't measuring quality, it's measuring the judge.
04 The statistics that make it credible
An eval score is an estimate from a sample. 71% vs 68% on 200 items may be indistinguishable noise. We treat questions as drawn from a super-population and report the interval, the single most-skipped step in the field.
Ten questions from one document aren't ten independent observations. Treating them as such shrinks your error bars to a fiction. We compute variance at the level of the cluster, the document, the repo, the customer.
To compare two models we analyse the per-item difference on the same items, not two independent averages. Pairing removes item difficulty from the variance and can cut the sample size you need by a large factor.
05 The trust machinery
Same corpus, same seed, same items, same order, always. Every model sits the identical exam, which is what makes paired comparison possible.
A misconfigured eval fails before it costs a single token. Clients notice this immediately.
If a judge call flakes, we mark the item unadjudicated. We never guess a score.
If a decision rule is unknown, we raise an error rather than silently pick one. No hidden defaults.
Change the decision rule and re-score all history with zero new spend. Your run history compounds inside the system.
Re-running a finished eval does nothing. Adding one model runs only that model. A monthly refresh is nearly free to deliver.
Common questions
No. The evaluation runs inside your VPC (your own cloud), with judge models called using your keys. Your production data and weights never leave your perimeter.
A diagnostic of your existing eval is one week. A full corpus engagement, where we build the corpus, align the judge, and run your slate, is 4 to 6 weeks.
Yes. We complement tools like promptfoo or DeepEval; what we add is independence, a contamination-free corpus from your own data, aligned judges, and clustered statistics your own setup usually lacks.
Independence from the team that built the model, a corpus the model has never seen, error bars instead of bare scores, and an immutable record of every run and decision.
Free · 30 minutes
Bring the spreadsheet of prompts someone eyeballed once. We'll tell you which of the six rules it breaks, and what it would take to make it trustworthy.