Fluency is not understanding

A model that sounds equally confident whether it is right or wrong is not a rare failure mode. It is the default: a ResNet-34 I tested claimed 99% certainty on predictions that were only 94% accurate, and nothing in its tone gave that away.

Key takeaways

• Fluency and correctness are produced by different parts of a model's behaviour, and nothing forces them to move together. A confident-sounding answer is evidence of confident-sounding training, not of accuracy.
• In a registered experiment on a public CIFAR-100 checkpoint, the model's highest-confidence bin held 70% of all predictions, claimed 99.11% certainty, and delivered 94.27% accuracy. The gap did not announce itself.
• Guo et al. (2017) found this overconfidence pattern across multiple architectures, not one unlucky model - closer to a property of how modern networks train than a bug in one of them.
• Large language models make this harder to catch: there is no clean softmax over a fixed label set to audit, only fluent text that can sound equally sure when right and when fabricating.

Why does a model's tone not track its accuracy?

A classifier's stated confidence and its actual accuracy are two different outputs of the same training process, and nothing in standard training forces them to agree. A network is optimised to get the label right, not to know how sure it should sound doing it. Guo et al. (2017) showed that as networks grew deeper and more capable through the 2010s, their confidence tended to outpace their accuracy. The pattern showed up across architectures and datasets, which is the detail that matters: not one poorly trained model having a bad day, but a systematic side effect of how the field made models more accurate.

Fluency, in the language model sense, is the same phenomenon wearing different clothes. A model trained to produce coherent, well-formed text learns to sound coherent and well-formed regardless of whether the content is correct. Sounding sure and being right are correlated often enough that it feels safe to conflate them, and decoupled often enough that the conflation eventually costs something.

What does this look like in a real measurement?

Abstract claims about overconfidence are easy to nod along to and easy to forget. A number is harder to wave away. In a registered calibration experiment on a public ResNet-34 checkpoint, I measured Expected Calibration Error (ECE) on half of the official CIFAR-100 test set before touching anything: 7.87%, more than double the 3% threshold that had been set in advance to kill the claim that the model was overconfident.

The gap concentrated, not scattered: 70.1% of all predictions landed in the single highest-confidence bin, claiming 99.11% average certainty and delivering 94.27% actual accuracy. That 4.84 point gap sits inside the bin where most of the model's output lives, not in some rare tail case a user would already distrust. Temperature scaling, one learned number, closed most of it without changing a single prediction, which says the overconfidence was a correctable distortion, not evidence the model secretly knew less than it claimed.

Why does this matter more for language models than classifiers?

A classifier's confidence is at least auditable in principle: a number between 0 and 1, attached to a fixed set of labels, sitting there waiting for someone to compute an ECE against it. A language model's fluency offers no equivalent handle. There is no small label set to bin against, no single softmax value standing in for "how sure was I," just text that can hedge, assert, or fabricate in the same even, competent register.

Liu et al. (2025) survey a large, growing body of work trying to build something like a confidence signal for language model output, precisely because none exists by default. Token-level probabilities measure how predictable the next token was, not how well-founded the claim in it is; a model can be entirely unsurprised by its own fabrication. The classifier case here is the easy version of the problem, with a clean number to measure and a known fix. The language model version has neither yet.

What should change in how we read model output?

The practical shift is not learning to detect uncertainty by ear. That skill does not exist, because the whole finding is that tone does not carry the information. What holds up instead is refusing to treat fluency as evidence and asking, for any claim that matters, what would actually confirm or kill it: a benchmark with a pre-registered threshold, a citation checked rather than trusted, a kill-gate written down before the number exists. None of that is available mid-conversation, which is why it has to be built into the process around the model, not read off its tone in the moment.

This is a workflow change, not a model change. It moves the question from "does this answer sound right" to "what would have to be true for it to be right, and has anyone checked." Fluency answers the first question for free, every time, whether it deserves to be trusted or not. The second question needs a step outside the conversation: a test run before the claim is trusted, a source opened and read, a threshold set before the output existed. Neither classifiers nor language models announce which outputs need that step, so the safer default treats any output a real decision rests on as one that does.

FAQ

Isn't a well-written, coherent answer at least some evidence the model understood the question?

Some, but weaker than it feels. Coherence shows the model produced text consistent with learned patterns, which correlates with correctness often enough to be useful and fails often enough to be dangerous. It is a prior, not a verification, and the two get conflated exactly when it matters most.

Can people just get better at detecting an uncertain model by how it phrases things?

Not reliably. The CIFAR-100 experiment measured a classifier with no phrasing at all, just a number, and the overconfidence was still large. Hedging language in a text model is a stylistic choice the model can produce or omit independent of whether the underlying claim is true.

Does temperature scaling or similar fixes solve this for language models too?

Not directly. Temperature scaling works cleanly on a softmax over fixed labels, which is what a classifier has and a language model's free-form output does not. Liu et al. (2025) cover the wider set of methods attempting something equivalent for generative models, none of them as settled as temperature scaling is for classifiers.

Is overconfidence only a problem for smaller or older models?

No. Guo et al. (2017) found the pattern in the architectures that were state of the art at the time specifically because they were more capable, not despite it. There is no basis in that finding for assuming a newer or larger model is exempt by default.

References

• Chuan Guo, Geoff Pleiss, Yu Sun, Kilian Q. Weinberger. On Calibration of Modern Neural Networks, 2017. arxiv.org/abs/1706.04599
• Xiaoou Liu, Tiejin Chen, Longchao Da, Chacha Chen, Zhen Lin, Hua Wei. Uncertainty Quantification and Confidence Calibration in Large Language Models: A Survey, 2025. arxiv.org/abs/2503.15850