An honest look at LLM-as-a-judge

Asked to score the same response five times, a local LLM judge gave the identical answer 36% of the time. Asked to match a human rater's score, it matched only 36% of the time too. The judge is precise. Precision is not accuracy.

Key takeaways

• A registered gate required Spearman correlation of at least 0.5 between the judge's scores and human helpfulness ratings. The actual figure was 0.395, significant but weak - the gate fired, and the claim that this judge tracks human judgement well enough to trust is rejected.
• A second gate required the judge's own repeat-to-repeat variation to stay below 0.75 points on a 0-4 scale. The actual figure was 0.386 - the gate passed, with margin. The judge is internally consistent.
• 36% of items got the exact same score across five independent runs. Separately, only 36% of items had the judge's score match the human rater's at all. The two numbers measure different things and happen to coincide.
• One response, a well-formatted table of ten Bruce Lee video games, got a human rating of 4 (extremely helpful) and a judge rating of 0, not helpful at all, across all five repeats. No internal disagreement flagged the error.

What does a kill-gate for a judge actually test?

HelpSteer2 (Wang et al., 2024) is a public dataset of prompts and responses, each rated by human annotators on five dimensions, helpfulness among them, on a 0-4 scale - a ready-made ground truth. A local model, gpt-oss run through Ollama on an RTX 5070 Ti, rated the same helpfulness dimension on 150 randomly sampled validation items, using the same 0-4 definition the annotators used. Each item was rated five separate times at a non-zero temperature, not once, because a single score cannot distinguish a stable judgement from a lucky one.

Two thresholds were registered before any score was collected. Agreement with humans, as Spearman correlation across the 150 items, had to reach 0.5 to call the judge a usable proxy. The judge's own spread across its five repeats, averaged across items, had to stay under 0.75 points to call its single-shot output trustworthy. Sheng et al. (2025) make the case for treating judge uncertainty as something to measure, not assume away - the same instinct behind every kill-gate on this site: a number fixed before the run is a commitment, not a suggestion to renegotiate once the data arrives.

A third check was purely diagnostic, no threshold attached: whether a judge's own disagreement with itself predicts how far it lands from the human score. That one decides nothing alone, but answers a question the two pass/fail gates cannot: does instability at least work as a warning sign.

How did the agreement gate fire?

Spearman rho came in at 0.395, comfortably below the 0.5 bar, with a p-value under 0.0001 - the weakness is not noise, it is a real, measurable gap. Human raters were generous: 119 of 150 items received a 3 or 4. The judge was not: only 30 items got a 4, and a disproportionate 42 landed on 2, a score humans gave only 9 times. Mean absolute error across all 150 items was 0.97, nearly a full point on a five-point scale. None of this is a close call that rounding would fix.

The shape of the disagreement matters as much as its size. A judge that was simply harsher across the board, shifting every score down by roughly the same amount, would still correlate well with human ratings even while disagreeing on the absolute number. That is not what happened here: the judge's distribution bulges in the middle of the scale where the human distribution does not, which is a difference in shape, not just in strictness, and a weak correlation coefficient is exactly the number that shape produces.

Why does low variance not mean the judge is right?

The clean separation in this experiment is between how consistent the judge is with itself and how correct it is against an outside standard. 36% of items got an identical score across all five runs - zero internal disagreement. Averaged over every item, the standard deviation across repeats was 0.386, which is why the stability gate passed easily. None of that stability says anything about correctness.

The table of Bruce Lee video games makes the gap concrete. It is well formatted, directly answers the prompt, and a human rater called it extremely helpful. The judge called it not helpful at all, and did so identically five times out of five. A registered diagnostic gate checked whether instability at least flags likely errors: the correlation between a judge's spread and its error was 0.231, real but weak. A confident, repeatable answer carried no signal that it might be wrong.

What does this mean for using a local judge in practice?

The practical reading is not that a local LLM judge is worthless, but that its self-reported consistency is the wrong thing to check for trust. A judge that always gives the same answer has removed one failure mode, random noise, while leaving the larger one, systematic disagreement with the standard that actually matters, completely open. Trusting a judge because it is stable is the same mistake as trusting a model because it sounds confident: a property of the output being checked, not the property that was actually asked about.

The registered-gate discipline earns its keep here. Without a pre-set bar, a correlation of 0.395 is easy to call "reasonably correlated" and move on. With the bar fixed at 0.5 in advance, the number is unambiguous: below the line, claim rejected, no rounding up afterward. The stability gate passing easily could have been reported alone, as the one number that looked good. Registering both before running either is what keeps the uncomfortable number in the final report too.

FAQ

Does this mean LLM-as-a-judge is useless for this kind of task?

Not useless, but not a drop-in replacement for human rating, at least for this model and dimension. A rho of 0.395 is a real, positive signal, just too weak to clear the bar set for trusting a single automated score in place of a human one.

Would a larger model do better?

Plausibly, but this experiment did not test that. Sheng et al. (2025) and related work suggest judge quality varies substantially by model and task, which is exactly why the registered scope here is one model, one dimension, not a general claim about LLM judges.

Why measure stability at all if it does not predict accuracy well?

Because the alternative, assuming a model is reliable because it answers the same way twice, is a common and untested shortcut. Measuring it and finding it insufficient is itself the finding: consistency was the easier property to check, and checking it first revealed that it was not the property that mattered.

What would make this result more convincing?

Testing more than one judge model, more than one rated dimension from HelpSteer2, and a larger sample than 150 items. The current scope was chosen to fit a single local GPU in reasonable time, not because it is the ceiling of what is worth checking.

References

• Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy J. Zhang, Makesh Narsimhan Sreedhar, Oleksii Kuchaiev. HelpSteer2: Open-source dataset for training top-performing reward models, 2024. arxiv.org/abs/2406.08673
• Huanxin Sheng, Xinyi Liu, Hangfeng He, Jieyu Zhao, Jian Kang. Analyzing Uncertainty of LLM-as-a-Judge: Interval Evaluations with Conformal Prediction, 2025. arxiv.org/abs/2509.18658