Building an Eval Set
By the end of this module
- Name the sources cases come from, and the blind spot each one leaves
- Choose a way to score a case that matches what you actually care about
- Read recall at k, and say what choosing k commits you to
Definition
The previous module showed a suite catching a regression that a headline number hid. That only worked because the suite contained the right cases. This module is about where those come from, which is the part teams find hardest and usually do last.
Where Cases Come From
Four sources, each with a different blind spot. Start with the default selection, which is how most suites begin, then add the others.
20 cases. 3 kinds with nothing in them.
Nothing covers natural, prefix, typo. Imagination produces the vocabulary the team already uses, which is the shape the system handles best. The queries users actually type are the ones missing.
Imagination is the natural starting point and the weakest source on its own. People write the queries they would type, in the vocabulary they already use, which is the shape the system is best at. The cases that would have caught something are the ones phrased the way a user who has never read your documentation would phrase them.
Production incidents are the opposite problem. Every case is real, and the set only ever contains failures somebody already suffered. A suite made entirely of past incidents is a guarantee you will not repeat yourself, and says nothing about what breaks next.
Logs are the highest-value source, because they are the actual distribution. The deliberately hard cases are what stop the suite drifting towards the easy middle of that distribution as the system improves.
Scoring a Case
A case needs a way of deciding whether the result was acceptable. The options run from cheap and narrow to expensive and general, and the right answer is usually the cheapest one that captures what you care about.
| Scorer | Good for | Where it goes wrong |
|---|---|---|
| Exact match | Classifications, extractions, anything with one right answer. | Useless the moment the answer is a sentence. |
| Containment | Did the right page, fact or identifier appear at all. | Says nothing about whether the rest of the answer was sound. |
| A rule or a schema | Format, length, required fields, forbidden content. | Checks the shape of an answer and not its truth. |
| A model as judge | Open-ended answers where a rule cannot express the standard. | Costs money, varies, and inherits the biases of the thing it grades. Covered in the next module. |
| A person | The standard that matters most, and the one everything else approximates. | Does not scale, so it is used to calibrate the cheaper scorers rather than to run the suite. |
Note what the retrieval suite in the previous module did. It did not score the wording of anything. It asked whether the right page appeared in the results, which is a containment check and the cheapest scorer on the list. Finding a question whose answer is checkable like that is worth more than building a better judge.
Recall at k
Recall at k asks whether a correct answer appears anywhere in the top k results. Recall at 1 is strict: the very first result has to be right. Recall at 5 is lenient: a right answer somewhere on the page counts.
Reporting several values of k at once is what made the previous module's regression legible. The change improved the strict measure and damaged the lenient one, which is a specific and recognisable failure: results became more confident and less complete. One number could not have shown that, whichever k it had been.
Choosing which k matters is a product decision rather than a technical one. If a person scans a list of results, recall at 5 is close to what they experience. If something downstream consumes the first result without a person seeing it, recall at 1 is the only honest measure, and the other numbers are flattering.
A team builds its first eval set from the last year of bug reports. What is the main weakness?
Discussion
Pick one thing your team ships and write down five eval cases on the spot. Then ask which of the four sources each one came from. Most rooms produce five cases from imagination and notice it as they say them out loud.