How rigorous is your eval, really?

Our eval set is built from real production data, not hand-written prompts.

The eval data was created after the model's training cutoff (contamination-free).

We stratify and tag items by difficulty, source, and segment.

We hold out a private slice that nobody tunes against.

We score deterministically (golden answers) wherever a correct answer exists.

For open-ended output we score against a written rubric, not a gut read.

We align our LLM judge to human labels before trusting it.

We use a panel of judges rather than one model grading its own kind.

We report confidence intervals, not bare average scores.

We cluster standard errors when items share a source.

We compare models with paired, per-item analysis.

We confirm a score difference is real before acting on it.

We score the agent's trajectory (tool calls, steps), not just the final answer.

We measure whether the agent picks the right tool from our own tools.

We track reliability with pass^k, not just pass@k.

We measure cost and latency per completed task, not per token.

Someone other than the team that built the system signs off the evaluation.

Our eval evidence maps to the controls our auditors or regulators care about.

We keep an immutable audit trail of eval runs and decisions.

We could hand our eval report to a regulator or customer and defend it.

Our eval runs inside our own environment; data does not leave our perimeter.

We redact or tokenize PII before anything is scored.

We version and freeze our eval set like code.

We re-run the eval on a schedule as models and prompts change.