LLM as judge

A measuring instrument needs calibrating

Some qualities cannot be checked by code no matter how cleverly the assertion is written. Whether an answer is actually grounded in its cited sources, whether a tone matches what a brand guideline asks for, whether a summary preserved the genuinely important parts of a longer document — these need real judgement, and at any meaningful volume that judgement has to be automated to be practical at all. The mistake to avoid is treating a judge’s output as a trustworthy measurement simply because it arrives as a tidy number. A judge is itself another LLM-based system, with its own prompt and its own failure modes, and it inherits essentially every weakness covered elsewhere in this roadmap. An unvalidated judge produces a confident-looking number with an unknown, unmeasured relationship to actual quality — which is more dangerous than having no number at all, precisely because teams will act on a number that looks authoritative.

Reference-based versus reference-free grading

A reference-based judge compares a candidate answer against a known-good reference answer and scores how closely they match in meaning — useful when a genuinely correct answer can be written down in advance for each test case. A reference-free judge instead grades a candidate answer purely against a stated rubric, with no single correct answer to compare it to — necessary for open-ended tasks like “write a helpful response to this support ticket,” where many different phrasings could all be equally acceptable and no single reference text could capture that. Reference-free grading is more broadly applicable but correspondingly harder to calibrate, since there is no ground truth text to check the judge’s reasoning against directly.

How to align a judge, as a concrete procedure

Label a representative sample yourself — on the order of a hundred outputs, graded by someone who genuinely understands what a good answer looks like for this specific task. Run the candidate judge over that exact same hundred outputs. Compare the two sets of labels directly. Wherever they disagree, read those specific cases closely: the disagreement is usually a sign that the rubric itself is ambiguous on that particular case, more often than it is a sign the judge reasoned incorrectly. Tightening the rubric, typically by adding a worked example of each grade level directly into the judge’s own prompt — the judge is, after all, just another model responding to a prompt, and benefits from the same few-shot technique covered in few-shot prompting — usually resolves most of the disagreement. Once agreement between the judge and the human labels is high enough to trust, keep those original human labels as the judge’s own permanent regression set: any time the judge’s prompt or underlying model changes, re-running it against this same labelled set tells you immediately whether the judge still agrees with the humans or has quietly drifted.

The specific biases to design around

Position bias: in a pairwise comparison between two candidate answers, the judge measurably favours whichever position — first or second — it was shown, independent of actual content. The standard mitigation is running the comparison in both orders and either averaging the result or treating a disagreement between the two orders as effectively a tie rather than a clear verdict.

Verbosity bias: a longer answer tends to score higher from a judge regardless of whether the additional length added any real value. This means a prompt change that only makes output more verbose, with no actual improvement in correctness, can show up as a rising judge score — a trap worth watching for specifically whenever a score improves alongside a noticeable increase in output length.

Self-preference bias: a judge measurably tends to prefer text generated by its own model family over text from a different one, even when quality is genuinely comparable. Using a different model as the judge than the one being evaluated is cheap, easy insurance against this specific failure.

Scale compression: asking a judge for a numeric rating on a wide scale, such as one to ten, tends to produce a narrow cluster of sevens and eights regardless of true quality spread, because the judge has little basis for fine discrimination along such a wide, under-specified scale. Narrow, precisely defined scales with a description of what each point actually means — or better, binary pass-or-fail questions against one specific stated criterion — carry far more usable signal than an open-ended numeric rating.

Ensembling, and what it buys

Running several independent judge calls — different prompts, different models, or the same judge run multiple times at non-zero temperature — and combining their verdicts by majority vote or averaging reduces the influence of any single judge’s idiosyncratic bias on a given case, at the direct cost of multiplying the number of calls needed per evaluated output. This is usually reserved for the highest-stakes evaluations, such as a release gate that blocks a deployment, rather than routine day-to-day development iteration, where a single well-calibrated judge is normally fast and cheap enough to use constantly.

Prefer narrow, binary questions over holistic ratings

“Does every factual claim in this answer appear somewhere in the provided source material?” produces a genuinely usable, checkable judgement. “Rate the overall quality of this answer from one to ten” produces, more often than not, a seven — and tells you almost nothing actionable about what specifically to fix.