Field Notes

Field notes on evaluating LLMs and agents.

How to measure whether your models and agents are good enough to ship, and prove it: independence, contamination, agents, and compliance.

Agents (9) Evaluation (28) Llmops (15) Ai (21) Engineering (2) Quality (1) Compliance (12) Cost optimization (1) Safety (1)

Evaluation · 12 Aug, 2026 · 06 Mins read

Policy-Violation Testing: Will Your Agent Refuse a Bad Instruction?

An agent that scores well on your task benchmark can still delete the wrong file, close a ticket it should have escalated, or push a change it was told not to push, the moment the task gets ambiguous or someone leans on…

Read note →

Evaluation · 12 Aug, 2026 · 06 Mins read

Signing Off an MCP Agent Before Production: The Pre-Deployment Checklist

Most teams ship an MCP agent to production the same way they ship a feature: it passed the demo, the tests are green, and someone says "look…

Read note →

Evaluation · 12 Aug, 2026 · 06 Mins read

Choosing a Model for Your Agent With Evidence, Not a Leaderboard

The model at the top of the leaderboard is not necessarily the model you should ship. Leaderboards rank one axis, task score, on someone els…

Read note →

Evaluation · 12 Aug, 2026 · 05 Mins read

Ori Eval vs Independent Evaluation: When Self-Serve Is Not Enough

OpenRouter's Ori Eval is the most credible self-serve agent eval we have seen ship this year, and it valid…

Read note →

Evaluation · 11 Aug, 2026 · 05 Mins read

MCP Tool Selection Evaluation: Test Your Own Tools

Public MCP benchmarks cannot tell you whether your agent picks the right tool from your MCP servers. They test generic, fixed toolsets under…

Read note →

Agents · 10 Aug, 2026 · 05 Mins read

Agent Trajectory Evaluation: Score the Path, Not Just the Answer

Scoring only the final answer of an agent is dangerous. An agent can return the correct result while calling the wrong tool, retrying five t…

Read note →

AI · 09 Aug, 2026 · 05 Mins read

Independent LLM Evaluation: Why a Third Party Beats Self-Scoring

Independent LLM evaluation is when a separate party, with no stake in the outcome and no exposure to your test data during training, grades…

Read note →

AI · 08 Aug, 2026 · 05 Mins read

Benchmark Contamination: Why LLM Leaderboards Lie

Benchmark contamination is when benchmark data leaks into a model's training data, so the model has effectively seen the test before taking…

Read note →

AI · 07 Aug, 2026 · 05 Mins read

There Is No Best LLM: How to Choose the Right Model per Task

There is no best LLM. There is only the best model for a specific job, measured on your task, through your harness, against your budget. So…

Read note →

AI · 06 Aug, 2026 · 05 Mins read

Agent Harness vs Model: Why the Harness Moves Your Scores

Yes, the harness affects LLM performance, and it affects it a lot. You do not ship a model. You ship a system: a prompt, an agent loop, tool…

Read note →

AI · 05 Aug, 2026 · 05 Mins read

Is LLM-as-a-Judge Reliable? Biases and How to Fix Them

LLM-as-a-judge is reliable, but only when you control for its known failure modes. Used as a raw scorer it inherits three documented biases…

Read note →

Agents · 04 Aug, 2026 · 06 Mins read

pass@k vs pass^k: The Honest Agent Reliability Metric

pass@k is the probability that an agent succeeds on at least one of k tries. pass^k is the probability that it succeeds on all k tries. That…

Read note →