Hallucination

Definitions worth keeping separate#

• Faithfulness failure: the output contradicts a provided source. Detectable by entailment checking against that source.
• Factuality failure: the output contradicts the world. Requires an external oracle, and is therefore much harder to evaluate at scale.
• Intrinsic: contradicts the input context. Extrinsic: unverifiable from the context, true or false.

Ji et al. (2023) established this taxonomy for generation broadly; Huang et al. (2023) survey the LLM-specific literature. Conflating faithfulness and factuality is the most common source of confused benchmark results, because a system can be perfectly faithful to a wrong retrieved document.

Why the objective produces it#

Pretraining maximises Σ log p_θ(x_t | x_{<t}) over a text corpus. This is density estimation over strings. Nothing in the objective references a truth predicate, and no term distinguishes a true string from a fluent false one of equal corpus likelihood.

Two consequences follow directly, and both are formalised in recent work.

A calibration lower bound. Kalai and Vempala (2024), Calibrated Language Models Must Hallucinate, prove that a model calibrated to the training distribution must hallucinate at a rate bounded below by roughly the fraction of facts appearing exactly once in the corpus — the monofact rate, estimated by Good–Turing missing-mass arguments. For arbitrary facts with no learnable structure, the bound is not small. Crucially the result is architecture-independent: it applies to transformers, to n-gram models, and to anything else fitted to the same distribution. Reducing hallucination below this bound requires breaking calibration, which is what post-training and retrieval both do.

Evaluation rewards guessing. Kalai, Nachum, Vempala and Zhang (2025), Why Language Models Hallucinate, argue that persistence after pretraining is a scoring problem. Under binary-graded benchmarks, abstention scores zero and a guess has positive expected value, so a model optimised against those leaderboards learns to guess. Their prescription is a scoring change — explicit confidence targets with penalties for confident errors — rather than a new decoding trick. This reframes hallucination as partly a measurement artefact, which is a genuinely useful shift.

Aggravating factors#

Snowballing. Zhang et al. (2023) show that once a model has committed to an incorrect claim, it produces further incorrect claims to remain consistent with it, including on questions it answers correctly in isolation. Verification must therefore be interleaved with generation, not deferred to the end.

Fine-tuning on new knowledge. Gekhman et al. (2024) find that SFT examples containing facts absent from pretraining are fitted slowly, and that as the model fits them its hallucination rate on unrelated questions rises. Teaching new facts through SFT trains the behaviour of asserting unsupported claims. This is the strongest available argument for retrieval over fine-tuning as a knowledge-injection mechanism.

RLHF degrades calibration. The GPT-4 system card reports a pretrained model well calibrated on MMLU, and the same model substantially miscalibrated after RLHF. Preference training optimises for answers humans like, and humans like confident answers.

Detection#

Sampling-based consistency. SelfCheckGPT (Manakul et al., 2023) samples multiple responses and measures agreement, requiring no logits and no external resource. Effective and expensive.

Semantic entropy. Farquhar et al. (2024), in Nature, cluster sampled generations by bidirectional entailment before computing entropy, so paraphrases of one answer count once. This isolates uncertainty about meaning from uncertainty about wording, and outperforms naive token-entropy baselines across models and datasets. It is the strongest general-purpose detector currently published, and it detects confabulation — arbitrary, prompt-sensitive claims — rather than consistently held false beliefs.

Atomic fact scoring. FActScore (Min et al., 2023) decomposes a generation into atomic claims and verifies each against a knowledge source, giving a percentage rather than a binary judgement. The right granularity for long-form output, where a whole-answer label is nearly meaningless.

Internal-state probes. Linear probes on hidden activations detect statements the model itself represents as false at above-chance rates. Azaria and Mitchell (2023) and related work show the signal exists; generalisation across domains remains weak, and a probe that transfers reliably has not been demonstrated.

Mitigation, with honest effect sizes#

• Retrieval with attribution. The largest single reduction available, and the one with clear failure modes: retrieval miss, wrong passage, context ignored. See RAG.
• Abstention training. R-Tuning (Zhang et al., 2024) fine-tunes on refusals for questions outside the model's parametric knowledge, improving willingness to decline without collapsing helpfulness.
• Contrastive decoding. DoLa (Chuang et al., 2023) contrasts logits from later and earlier layers, exploiting the observation that factual content is refined in the upper layers. Cheap, no training, modest gains.
• Activation steering. Inference-Time Intervention (Li et al., 2023) shifts activations along a truth-correlated direction found on TruthfulQA, improving that benchmark substantially. Transfer beyond the benchmark it was fitted on is the open question.
• Constrained decoding and tool use. For anything with a checkable grammar — schemas, identifiers, arithmetic, dates — remove the freedom rather than correcting the output. This eliminates a whole class of failures instead of reducing it.

Benchmarks and their limits#

TruthfulQA (Lin et al., 2022) targets imitative falsehoods, statements that are common in text and false. It is adversarially constructed, so scores do not estimate a real-world hallucination rate. HaluEval and FActScore cover recognition and long-form factuality respectively.

All published benchmarks now face contamination: they are on the public internet, and therefore in training corpora. Held-out evaluation on private data, refreshed over time, is the only defensible measurement. Report abstention rate alongside accuracy, or you are rewarding exactly the guessing behaviour Kalai et al. identify.

Papers#

• Ji et al., Survey of Hallucination in Natural Language Generation, ACM Computing Surveys, 2023 — arxiv.org/abs/2202.03629
• Lin, Hilton and Evans, TruthfulQA, 2021 — arxiv.org/abs/2109.07958
• Manakul, Liusie and Gales, SelfCheckGPT, 2023 — arxiv.org/abs/2303.08896
• Min et al., FActScore, 2023 — arxiv.org/abs/2305.14251
• Zhang et al., How Language Model Hallucinations Can Snowball, 2023 — arxiv.org/abs/2305.13534
• Kalai and Vempala, Calibrated Language Models Must Hallucinate, 2024 — arxiv.org/abs/2311.14648
• Farquhar et al., Detecting Hallucinations in Large Language Models Using Semantic Entropy, Nature 630, 2024.
• Gekhman et al., Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?, 2024 — arxiv.org/abs/2405.05904
• Kalai, Nachum, Vempala and Zhang, Why Language Models Hallucinate, 2025 — arxiv.org/abs/2509.04664

What to learn next#

• What is RAG? — grounding an answer in a document you control.
• Prompt engineering — writing the instruction that leaves less room to invent.
• Fine-tuning — why training new facts in tends to make this worse, not better.