Why Evaluation Is the Work
By the end of this module
- Explain why an ordinary test cannot protect a model-backed system
- Read a scorecard and spot a regression a headline number hides
- Say what a suite is actually measuring, and what it is not
Definition
Unit one left a loose end. If the same input can produce different output, then a test that asserts an exact answer passes on the day it is written and fails on a rerun. The third response on that list was to accept the variation and measure the distribution instead. This unit is that.
An eval is a set of cases with a way of scoring the result, run as a batch, reported as numbers. It replaces the assertion that one answer is right with a measurement of how often answers are good enough, which is the only claim available once the system stopped being deterministic.
Why Tests Stop Working
| A unit test | An eval suite | |
|---|---|---|
| What it asserts | This input produces this output. | Across these cases, the output is good enough this often. |
| What a failure means | Something is broken. | Something may be worse than it was. The number moved. |
| How it is scored | Equality. | A rule, a rubric, a model acting as judge, or a person. |
| How many cases | One per behaviour. | Enough that a few results do not move the number. |
| When it runs | Every commit, in seconds. | On changes that could affect behaviour, in minutes, and it costs money. |
This is why teams who skip evals end up with a suite of tests marked flaky and nothing actually guarding the behaviour. The tests were not flaky. They were asserting a property the system had stopped having.
A Run, Before and After
Below is a retrieval eval over 60 queries, split into six kinds. Someone has changed how results are ranked. Look at the headline numbers first, then switch.
my agent keeps calling the same tool over and over
want /course/typed-exits
got /ai/agentic-errors | /ai/guardrails
The baseline. One miss, on a query that describes a symptom rather than naming a concept.
The change improved the number most people would have quoted. Recall at one went up five points, which is a real improvement: the top result is right more often. Recall at five went down five, which means four queries that previously had a correct answer somewhere on the page now have none at all.
Every point of that loss is in one kind. Queries that describe a problem in the user's own words, rather than naming the concept the way the pages do, got worse. A suite reporting one aggregate would have shown a small wobble and this would have shipped. The split by kind is what makes the regression visible, and the kinds are chosen to be the shapes the system is expected to find hard.
What a Suite Measures
A scorecard is easy to over-read. Three things worth being precise about.
| It does measure | It does not measure |
|---|---|
| Whether a change made things worse on the cases you wrote. | Whether it made things worse on the cases you did not think of. |
| How often the system is good enough, by the rule you encoded. | Whether that rule is the right definition of good enough. |
| A comparison between two versions, on one day. | How the system behaves on real traffic, which is distributed differently. |
The honest framing is that an eval is a regression detector, not a quality certificate. It tells you whether today's change made the thing worse on cases you thought to write down. That is far more than nothing, and it is less than it sounds like.
A change raises overall eval score from 91% to 93%. What should you look at before shipping it?
Discussion
For something your team ships that uses a model, ask what would have to break before anyone found out. If the answer is a customer complaint, the next module is about where the cases to prevent that come from.