LLM-as-Judge: Can You Trust an AI to Grade Your Tests?

The team built an LLM judge to evaluate their customer support chatbot. They gave the judge a rubric, ran it against 500 responses, and got a score of 91%. Leadership was happy. The QA lead was not, because she had read 20 of the responses the judge scored as high-quality and counted 6 that she would have flagged. A human scoring the same 500 responses would have come back with something closer to 72%.

The judge was not broken. It was measuring something different from what they thought they were measuring. It scored fluency well. It scored relevance to the question reasonably well. It did not catch the cases where the chatbot gave a technically accurate answer that completely missed the customer’s emotional state, gave directions for a feature that no longer existed, or answered a question the customer did not actually ask by interpreting the phrasing too literally.

LLM-as-judge is a legitimate and widely-used technique for evaluating language model outputs at scale. It is also a technique with specific failure modes that, if you do not understand them, will give you a confidence score that is not justified by the underlying quality. This post covers when LLM-as-judge works, when it does not, how to calibrate a judge so its scores mean something, and what still requires a human.

Why the Technique Exists

Before LLM-as-judge, teams had two options for evaluating open-ended language model outputs. They could write exact-match assertions, which only work for outputs with a single correct answer, or they could pay humans to rate every output, which costs money and does not scale.

LLM-as-judge fills the gap between those two options. You describe the quality criteria in a prompt, send the input and the model’s output to a judge model, and get a structured score back. You can do this for thousands of examples overnight, at a cost of fractions of a cent per evaluation, without any human involvement.

The technique works because current language models are surprisingly good at applying rubrics. If you tell a judge model “rate this summary on a scale of 1 to 5 for factual accuracy relative to the source document, where 5 means all claims are directly supported and 1 means most claims are unsupported,” the judge will apply that rubric consistently and its scores will correlate meaningfully with human judgment on many tasks. The correlation is not perfect, but for many evaluation tasks, it is good enough to catch regressions between model versions and flag egregious quality failures.

The technique has a ceiling, and the ceiling is defined by the judge model’s own limitations. The judge cannot catch failure modes it cannot recognize. It cannot apply criteria that require domain expertise it does not have. And it will systematically mis-score outputs that match surface patterns of quality without having the underlying quality.

The Failure Modes

Understanding the failure modes is what separates a calibrated LLM-judge setup from a false confidence generator.

Positional bias: many judge models show a preference for one position in a comparison task. If you ask a judge to compare two responses and pick the better one, it will often favor the response presented first, or in some models, the response presented last. If you use pairwise comparison without controlling for position (by running each comparison both ways and taking the majority vote), your judge scores are contaminated by this artifact.

Verbosity bias: longer responses tend to score higher with judge models, independent of their accuracy or relevance. A response that uses more words to say less will often outperform a concise accurate response. If your use case values conciseness, you need to explicitly counter this in your rubric or your judge will systematically reward the wrong thing.

Self-preference: a judge model tends to score responses generated by models of the same family higher than responses from other families, even when human raters prefer the latter. If your production model and your judge model are both from the same provider, be aware that this bias may inflate your scores. Cross-provider judging reduces this effect.

Sycophancy toward confident-sounding outputs: judge models, like the models they evaluate, were trained in part to favor responses that sound confident and authoritative. A response that states something wrong with confidence will often outscore a response that correctly expresses uncertainty. For tasks where calibrated uncertainty is important, this is a significant blind spot.

Domain expertise gaps: a judge model evaluating medical, legal, financial, or highly technical outputs may not have the expertise to recognize subtle errors that a domain expert would catch immediately. If your use case is domain-specialized, your judge needs domain-specific rubrics and your calibration set needs to include the kinds of errors that require domain knowledge to identify.

Calibrating the Judge

Calibration is what separates an LLM judge you can trust from one that gives you numbers that feel meaningful but are not.

The calibration process requires human judgments on a subset of examples. Take 50 to 100 outputs from your production distribution, have subject-matter experts rate them against the same rubric you give the judge, and compare the judge’s scores to the human scores. This comparison tells you three things: the judge’s overall correlation with human judgment, the systematic biases (does it consistently rate certain categories too high or too low?), and the failure cases where the judge diverges significantly from human raters.

Correlation above 0.7 with human judgment is a commonly cited threshold for a judge you can use with some confidence. Below 0.5, the judge is measuring something that correlates loosely with quality but is not reliable enough to use as a quality gate. Between 0.5 and 0.7, the judge can be useful for catching large regressions but should not be trusted for fine-grained quality distinctions.

The calibration comparison also tells you whether the judge is biased up or down relative to human raters. If human raters average 3.6 out of 5 and the judge averages 4.2, you have a systematic inflation of 0.6 points. You can correct for this by adjusting your thresholds accordingly: what the judge calls a 4.2 is what humans would call a 3.6. The bias does not invalidate the judge — it just means you need to interpret scores in the context of the calibration data.

Repeat the calibration on a fresh sample every quarter, or whenever the production model changes significantly. Judge calibration is not a one-time measurement. The judge’s behavior relative to your rubric can drift as the judge model itself is updated, and the production distribution of outputs changes over time in ways that can move the judge outside its calibrated range.

The Rubric Is the Most Important Variable

Teams new to LLM judging typically under-invest in the rubric and over-invest in trying different judge models. The judge model matters. The rubric matters more.

A rubric is specific about what each score level means. A rubric that says “1 = bad, 3 = okay, 5 = great” will produce scores that are loosely correlated with quality but not comparable across different examples or different runs. The judge is applying its own interpretation of “bad” and “great,” which may not match yours.

A useful rubric defines each score level concretely, with reference to the specific failure modes you care about. For a customer support response evaluator: “1 = the response addresses a different question than the one asked OR contains factual errors about the product; 2 = the response addresses the question but omits critical information; 3 = the response addresses the question accurately but is unclear or unnecessarily long; 4 = the response accurately addresses the question clearly and appropriately; 5 = the response addresses the question, is concise, and specifically acknowledges the customer’s context or concern.”

Each level should be distinguishable from its neighbors by reference to observable properties of the response. A judge reading this rubric can apply it consistently; a judge reading a vague rubric will apply its own interpretation, which will vary across calls.

Include few-shot examples in the judge prompt. Two examples per score level, with brief explanations of why each example received its score, significantly improve the judge’s consistency. The examples serve as anchors that keep the judge’s interpretation of the rubric stable across diverse inputs.

Strategies for Reducing Variance

A single judge call on a single example has meaningful variance. The same judge model on the same input will sometimes return different scores on different calls, because the model is probabilistic and rubric interpretation has some inherent ambiguity.

Three strategies reduce variance to usable levels.

Majority voting: run three independent judge calls per example and take the majority verdict. If two out of three calls return a score of 4, the verdict is 4. This costs three times the API budget but cuts the score variance significantly and is standard practice for production eval systems.

Lower temperature: judge calls should use lower temperature settings (0.0 to 0.3) to reduce the randomness in the model’s output. You want the judge to apply the rubric consistently, not to be creative about it. Lower temperature does not eliminate variance entirely, but it reduces the range.

Structured output: require the judge to return a structured response (score as a number, brief justification as a string) rather than free text. Parse the score programmatically rather than having the judge return prose that you then try to extract a score from. Ambiguous output is a source of variance that structured output eliminates.

For binary judgments (pass/fail, acceptable/unacceptable), a threshold on a 5-point scale is cleaner than asking the judge to make the binary determination directly. “Score the response 1-5. Scores of 4 or 5 pass the quality gate.” This separates the measurement from the threshold decision and makes it easier to adjust the threshold based on calibration data.

When Human Review Still Wins

LLM-as-judge is a tool for scaling judgment, not a replacement for judgment in situations where the stakes are high enough that errors have real consequences.

Any evaluation where errors in one direction are much more costly than errors in the other direction needs human review on the tail cases. A safety classifier that has to decide whether a response is harmful should use an LLM judge to triage the easy cases and route borderline cases to human reviewers, not route everything to the judge and ship whatever it approves.

Any domain where the judge lacks expertise needs domain-expert review on a sample. If you are evaluating medical advice, legal guidance, or financial recommendations, an LLM judge without domain expertise will miss the nuanced errors that a domain expert would catch. Use the judge to flag the obvious failures and route a representative sample to experts.

Any evaluation where the quality criteria are still being defined needs human review before the judge. You cannot automate measurement of a rubric you have not finished writing. Early in a product’s development, when you are still discovering what “good” means for your use case, human review of a small sample is more valuable than a judge scoring thousands of outputs against a half-formed rubric.

Building a Judge You Can Audit

The judge calls that produce your quality scores should be logged, not discarded. Every judge call includes the input, the output being judged, the rubric, the score returned, and the justification the judge provided. This log is your audit trail.

When a quality regression surfaces, the audit trail lets you trace it: which examples drove the score down, what did the judge say about them, and does the judge’s justification match what a human would say? If the judge is penalizing examples that a human would score highly, that is a rubric problem. If the judge is scoring examples highly that a human would penalize, that is a failure-mode gap in the rubric.

The audit trail also lets you identify calibration drift. If you run a weekly comparison of judge scores against a fixed set of human-rated examples and the correlation starts dropping, you know the judge’s behavior has shifted relative to your rubric before that shift affects your production quality signal.

LLM-as-judge is a powerful technique used correctly and a false confidence generator used carelessly. The difference is calibration, specific rubrics, variance reduction, and knowing where the technique’s ceiling is. Build the infrastructure to audit the judge alongside the infrastructure to run it, and you have a scaling tool. Skip the auditing and you have a number that will eventually diverge from reality in a way you will not see coming until a customer or a stakeholder shows you the evidence.