{"id":36615,"date":"2025-09-07T06:27:29","date_gmt":"2025-09-07T06:27:29","guid":{"rendered":"https:\/\/youzum.net\/from-pretraining-to-post-training-why-language-models-hallucinate-and-how-evaluation-methods-reinforce-the-problem\/"},"modified":"2025-09-07T06:27:29","modified_gmt":"2025-09-07T06:27:29","slug":"from-pretraining-to-post-training-why-language-models-hallucinate-and-how-evaluation-methods-reinforce-the-problem","status":"publish","type":"post","link":"https:\/\/youzum.net\/ja\/from-pretraining-to-post-training-why-language-models-hallucinate-and-how-evaluation-methods-reinforce-the-problem\/","title":{"rendered":"From Pretraining to Post-Training: Why Language Models Hallucinate and How Evaluation Methods Reinforce the Problem"},"content":{"rendered":"

Large language models (LLMs) very often generate \u201challucinations\u201d\u2014confident yet incorrect outputs that appear plausible. Despite improvements in training methods and architectures, hallucinations persist. A new research from OpenAI<\/strong> provides a rigorous explanation: hallucinations stem from statistical properties of supervised versus self-supervised learning, and their persistence is reinforced by misaligned evaluation benchmarks.<\/p>\n

What Makes Hallucinations Statistically Inevitable?<\/strong><\/h2>\n

The research team explains hallucinations as errors inherent to generative modeling. Even with perfectly clean training data, the cross-entropy objective used in pretraining introduces statistical pressures that produce errors.<\/p>\n

The research team reduce the problem to a supervised binary classification task called Is-It-Valid (IIV)<\/em>: determining whether a model\u2019s output is valid or erroneous. They prove that the generative error rate of an LLM is at least twice its IIV misclassification rate. In other words, hallucinations occur for the same reasons misclassifications appear in supervised learning: epistemic uncertainty, poor models, distribution shift, or noisy data.<\/p>\n

Why Do Rare Facts Trigger More Hallucinations?<\/strong><\/h2>\n

One major driver is the singleton rate<\/strong>\u2014the fraction of facts that appear only once in training data. By analogy to Good\u2013Turing missing-mass estimation, if 20% of facts are singletons, at least 20% of them will be hallucinated. This explains why LLMs answer reliably about widely repeated facts (e.g., Einstein\u2019s birthday) but fail on obscure or rarely mentioned ones.<\/p>\n

Can Poor Model Families Lead to Hallucinations?<\/strong><\/h2>\n

Yes. Hallucinations also emerge when the model class cannot adequately represent a pattern. Classic examples include n-gram models generating ungrammatical sentences, or modern tokenized models miscounting letters because characters are hidden inside subword tokens. These representational limits cause systematic errors even when the data itself is sufficient.<\/p>\n

\n
\"\"<\/figure>\n<\/div>\n

Why Doesn\u2019t Post-Training Eliminate Hallucinations?<\/strong><\/h2>\n

Post-training methods such as RLHF (reinforcement learning from human feedback), DPO, and RLAIF reduce some errors, especially harmful or conspiratorial outputs. But overconfident hallucinations remain because evaluation incentives are misaligned.<\/p>\n

Like students guessing on multiple-choice exams, LLMs are rewarded for bluffing when unsure. Most benchmarks\u2014such as MMLU, GPQA, and SWE-bench\u2014apply binary scoring: correct answers get credit, abstentions (\u201cI don\u2019t know\u201d) get none, and incorrect answers are penalized no more harshly than abstentions. Under this scheme, guessing maximizes benchmark scores, even if it fosters hallucinations.<\/p>\n

How Do Leaderboards Reinforce Hallucinations?<\/strong><\/h2>\n

A review of popular benchmarks shows that nearly all use binary grading with no partial credit for uncertainty. As a result, models that truthfully express uncertainty perform worse than those that always guess. This creates systemic pressure for developers to optimize models for confident answers rather than calibrated ones.<\/p>\n

What Changes Could Reduce Hallucinations?<\/strong><\/h2>\n

The research team argue that fixing hallucinations requires socio-technical change, not just new evaluation suites. They propose explicit confidence targets<\/strong>: benchmarks should clearly specify penalties for wrong answers and partial credit for abstentions.<\/p>\n

For example: \u201cAnswer only if you are >75% confident. Mistakes lose 2 points; correct answers earn 1; \u2018I don\u2019t know\u2019 earns 0.\u201d<\/em><\/p>\n

This design mirrors real-world exams like earlier SAT and GRE formats, where guessing carried penalties. It encourages behavioral calibration<\/strong>\u2014models abstain when their confidence is below the threshold, producing fewer overconfident hallucinations while still optimizing for benchmark performance.<\/p>\n

What Are the Broader Implications?<\/strong><\/h2>\n

This work reframes hallucinations as predictable outcomes of training objectives and evaluation misalignment rather than inexplicable quirks. The findings highlight:<\/p>\n