Building an eval set

Where the cases come from

From production, as far as this is genuinely possible. Real inputs carry the typos, the ambiguity, the unexpected language mixing, and the adversarial edge cases that actually make a system fail in practice; invented inputs carry only the assumptions of whoever invented them about what failure looks like, which is a strictly smaller and more comfortable set than reality provides. If there is no production traffic yet, the next best source is anything real users actually wrote elsewhere — support tickets, search logs, internal query logs from an earlier manual process — rather than examples a developer writes from imagination. Deliberately keep the awkward cases. A set made only of clean, well-formed inputs certifies that the system handles the easy half of its job and says nothing about the half that actually causes incidents.

Synthetic data, and its real pitfalls

When genuinely no real usage exists yet, generating synthetic test cases with a model is a reasonable stopgap, but it carries a specific risk worth naming: a model generating test inputs tends to generate inputs that are themselves fairly easy for a model to handle, simply because both the generator and the system under test share similar blind spots and similar notions of what a plausible question looks like. Synthetic data is a placeholder to get started, not a long-term substitute for real usage, and it should be replaced with production-derived cases as soon as any real traffic exists at all.

Stratified sampling, so coverage is not accidental

Pulling cases at random from production traffic will, by construction, mostly return whatever is most common — the easy, high-volume cases most users actually send — and under-represent the rare but important ones. Stratified sampling deliberately pulls a meaningful number of cases from each known category of input — by language, by product area, by user type, by length — rather than letting the most frequent category dominate the eval set purely because it is the most frequent category in traffic. This is how a set stays useful for catching a regression that only shows up on, say, non-English input or on an unusually long request, even though such inputs are a small fraction of overall volume.

Labelling, and knowing whether your labels can be trusted

Where human judgement is involved, inter-rater reliability — whether two different people labelling the same case independently tend to agree — is worth actually checking rather than assuming. If two reasonable people disagree often on what counts as a good answer for a given case, the rubric being used to label it is ambiguous, and that ambiguity will show up later as noisy, hard-to-interpret eval scores, regardless of how sophisticated the rest of the pipeline is. Tightening the rubric, typically by adding worked examples of what each grade level actually looks like, usually resolves disagreement faster than simply asking labellers to try harder.

Size, and the trigger for growing it

Start smaller than feels respectable. Fifty to a hundred well-chosen cases spanning the real, known failure modes beats a thousand near-duplicate easy cases, if only because a hundred cases can actually be read in full by a person, while a thousand cannot. Grow the set when a genuinely new failure mode appears in production that nothing currently in the set would have caught — that specific event is the trigger, not a calendar schedule or a round-number target.

Splitting, so you do not just fit the examples

Keep two distinct sets. A development set, iterated against freely and often while making changes. And a held-out set, run only occasionally and never directly tuned against, which exists purely to answer whether an improvement measured on the development set actually generalises or whether it was fitted specifically to the handful of cases being stared at during iteration. Without a genuinely held-out set, the easy failure mode is optimising until the development score looks good and then discovering in production that the improvement does not transfer, because the examples were fitted rather than the underlying problem actually solved.

Staleness, and keeping the set honest over time

An eval set is a snapshot describing the system as it was at the moment the set was built. Prompts change, the model behind the endpoint changes, and the population of real users and their requests changes alongside both. Re-read a sample of the eval set on a regular cadence, retire cases that no longer represent anything meaningful about current usage, and add whatever new failure appeared since the last review. A set nobody revisits quietly becomes a green light that has stopped meaning anything about current quality.

Assert before you judge

Every check expressible in code is a check nobody ever has to argue about later. Did it return syntactically valid output in the required format. Does it cite at least one source when a citation is required. Does it avoid a specific phrase a stakeholder has flagged as unacceptable. Is it under a stated length limit. Did it correctly abstain on the specific case with no real answer. These cost almost nothing to run, return the same result every single time, and in practice catch a surprising number of real regressions — often more than the more sophisticated, judgement-based grading that everyone reaches for first. Model-based judgement, covered next, is for whatever genuinely cannot be reduced to a deterministic check.