II · THE IDEA · ARTIFICIAL INTELLIGENCE
Hallucination
▶ Listen · narrated
A model will tell you that the Golden Gate Bridge was transported from London in 1937 with the same calm certainty it reserves for true facts.
At a glance
- What it is
- Generating content that is fluent but factually incorrect, unfaithful to source material, or internally inconsistent
- Root cause
- Training objective optimises for plausible next tokens, not true ones
- Human baseline
- TruthfulQA: humans score 94% truthful, best models (at publication) 58%
- What helps
- Retrieval augmentation, reinforcement from human feedback, explicit uncertainty prompts
Imagine you are learning a language by reading thousands of books, but nobody ever tells you which sentences are true and which are made up. You learn that "The moon is made of rock" and "The moon is made of cheese" are both grammatically correct sentences that appear in English text. When someone asks you about the moon, you produce whichever sentence seems more typical given the conversation so far. You are fluent. You are also sometimes wrong, and you cannot tell the difference. A language model is in this position permanently. It was trained to predict the next word in a sentence, and it saw billions of sentences—true ones, false ones, jokes, guesses, errors. It learned the patterns of English text, not the patterns of reality. This is why it can state a false fact with the same confidence it uses for a true one. The confidence reflects how typical the sentence is, not how accurate.
Hallucination arises because the pre-training objective—minimising cross-entropy loss on next-token prediction—does not distinguish between true and false continuations. The model learns P(token | context) over the entire training corpus, which contains factual statements, counterfactuals, fiction, errors and lies in unmarked mixture. At inference time, the model samples from this learned distribution. High probability means the token sequence is typical of the training data, not that it corresponds to reality. Retrieval-augmented generation (RAG) mitigates this by injecting retrieved documents into the context window, shifting the distribution toward content more likely to be accurate. But RAG does not eliminate hallucination: the model may ignore the retrieved passage, misinterpret it, or retrieve nothing relevant and fall back on generative behaviour. RLHF addresses the problem from a different angle, introducing a learned reward model trained on human preferences. The policy is fine-tuned to maximise expected reward, which correlates imperfectly with truthfulness but does provide a signal absent from pre-training. Even after RLHF, models hallucinate, because the reward model itself is imperfect and because the base model's tendency to produce plausible-but-false text is still present in the policy. Mechanistically, there is no internal fact-checking step. The model does not retrieve a stored fact and compare it to the generated token. It computes a probability distribution over the vocabulary and samples. If the training data made false token sequences probable in certain contexts, the model will generate them.
Look closer
The model was never shown the difference
During training, the model sees a mixture: true statements, false statements, speculation, fiction, sarcasm, outdated information. The objective function assigns equal weight to predicting the next token in a conspiracy theory and the next token in a peer-reviewed summary. Both reduce the loss. The model learns the statistical patterns of text that humans write, and humans write false things in grammatically correct sentences. There is no signal in the training data that says "this token sequence is true" versus "this token sequence is false".
Confidence and accuracy do not correlate
A model can assign high probability to a completely invented fact. The softmax distribution over the vocabulary reflects how typical the next token is given the preceding context, not how true the resulting statement is. A common surname after "Dr" gets high probability regardless of whether that person exists. A plausible-sounding year after "invented in" gets high probability even if the invention never happened. The model has learned that certain syntactic shapes are common, and it continues them confidently.
Retrieval changes the game but does not solve it
Augmenting generation with retrieved documents gives the model grounding it otherwise lacks. The retrieved passages appear in the context window, and the model can condition its output on them. This measurably reduces hallucination in domains where reliable retrieval is possible. But it introduces new failure modes: the model may ignore the retrieved document, misread it, or blend it with memorised patterns that contradict it. And retrieval itself can fail—return irrelevant passages, or nothing at all—leaving the model to fall back on the same generative behaviour that produced hallucinations in the first place.
The story
The term hallucination, borrowed from psychology, names the phenomenon where a language model generates content that is fluent, coherent and completely wrong. It might invent a legal precedent, cite a nonexistent paper, or state a false biographical fact with no hesitation. The behaviour is not a defect bolted onto an otherwise truthful system. It is a direct consequence of the training objective.
Language models are trained to predict the next token given all preceding tokens. The loss function penalises the model when it assigns low probability to the token that actually appeared in the training data, and rewards it when it assigns high probability. There is no separate signal for truth. A sentence that says "The Eiffel Tower is in Berlin" appears in the training corpus—perhaps as part of a forum post, a creative writing exercise, or a list of common misconceptions—and the model learns to predict those tokens in that order when the context makes them statistically likely. The model is not modelling the world. It is modelling the text humans produce about the world, and that text is a lossy, biased, sometimes deliberately false reflection.
This explains why scaling alone does not eliminate hallucination. A larger model, trained on more data, learns richer statistical patterns. It becomes better at predicting what token comes next in a wide variety of contexts. But if the training data contains false statements presented in the same grammatical register as true ones, the model learns to produce both with equal fluency. The TruthfulQA benchmark, published in 2021, tested this directly. It comprised 817 questions spanning health, law, science and common misconceptions—questions designed to elicit answers humans often get wrong. The best models at the time of publication scored 58% on truthfulness, compared to 94% for human performance. The models had learned to mimic not just human knowledge but human falsehoods.
What actually reduces hallucination is changing what the model optimises for after the initial pre-training. Reinforcement learning from human feedback introduces a second objective: human raters score outputs for helpfulness, harmlessness and honesty, and the model is fine-tuned to increase the probability of highly rated responses. This does not give the model access to ground truth, but it does give it a signal that some outputs are preferable to others along dimensions that correlate loosely with accuracy. Retrieval-augmented generation takes a different approach: instead of relying solely on the model's parameters, the system retrieves relevant documents at inference time and includes them in the context window. The model can then condition its generation on text that is more likely to be accurate than its own memorised associations. Both techniques measurably reduce hallucination rates, but neither eliminates the behaviour.
There is also a less technical intervention that helps: prompting the model to express uncertainty. Asking "Are you sure?" or "What is your confidence level?" can sometimes cause the model to hedge or retract a false statement. This works not because the model has a reliable internal confidence measure, but because the training data contains examples of humans expressing doubt, and the prompt shifts the statistical context toward that pattern. It is a reminder that the model's behaviour is always a function of the text it has seen and the text it is now generating, token by token, with no global plan and no fact-checking step.
The architectural choice to train on next-token prediction over web-scale text gave us models with extraordinary fluency and breadth. It also gave us hallucination, not as an accident, but as an inevitable consequence. The model learned exactly what it was taught: how to continue text in ways that humans find plausible.
Why it mattered then
When the first large language models began producing fluent multi-paragraph text, the immediate reaction was wonder at the coherence. The models could draft essays, summarise arguments, translate between languages. But as people began using them for knowledge work—legal research, medical questions, historical claims—the hallucinations became impossible to ignore. A model would cite a court case that did not exist. It would invent a plausible-sounding chemical formula. It would state a false date with no hesitation. The term hallucination, already used informally in machine learning circles, gained currency as a way to name the problem without anthropomorphising it too much. The behaviour mattered urgently because the same fluency that made the models useful also made their errors dangerous. A user could not reliably distinguish a true answer from a false one by tone or confidence. Both arrived in the same calm, grammatical sentences.
Why it matters now
Hallucination remains the central limitation when deploying language models in high-stakes domains. Legal firms experimenting with AI-assisted research have discovered invented case citations in generated briefs. Medical chatbots have offered advice contradicted by clinical guidelines. Customer service systems have stated policies their companies do not have. The problem is not that the models are unreliable in some vague sense—it is that their unreliability is unpredictable and often invisible until someone checks. Retrieval-augmented systems and reinforcement learning from human feedback have reduced the rate of hallucination, but the behaviour persists because the underlying architecture has not changed. The model is still predicting tokens, still trained on a mixture of true and false text, still without a built-in mechanism to distinguish the two. Understanding this is not academic. It determines where models can be deployed safely, what kind of human oversight is necessary, and how to design interfaces that do not mislead users into trusting output they should verify.
The surprising detail
The TruthfulQA benchmark revealed something unexpected: larger models did not automatically become more truthful. In some categories, they became less truthful, because they had learned more of the false patterns present in human text. A small model might fail to answer a question at all. A large model would answer confidently and wrongly, having memorised a common misconception. This inverts the usual assumption that scale equals improvement. It suggests that truthfulness is not a capability that emerges from more parameters and more data—it is a separate objective that must be trained for explicitly, often in tension with the fluency and breadth that pre-training provides.
Remember this
Hallucination is not a bug. It is what you get when you train a model to predict text without teaching it to care whether that text is true.
Test yourself
A company builds a retrieval-augmented system that fetches relevant documents and includes them in the prompt before the model generates an answer. Hallucination rates drop significantly. Then a user asks a question for which no relevant documents exist in the retrieval corpus. What happens to the hallucination risk for that query, and why?
The hallucination risk returns to baseline—or higher. When retrieval fails, the model falls back on the same generative behaviour it always had: predicting tokens based on patterns learned during pre-training. The model has not learned to say "I don't know" unless that phrase appeared often in its training data in similar contexts. Instead, it continues the prompt in whatever way seems statistically plausible, which may mean inventing an answer. The risk may actually be higher than baseline if the system design leads users to trust the output more because they know retrieval is usually happening. The user does not see that retrieval returned nothing, only that the model answered confidently.
Go deeper
- A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions · arXiv · Lei Huang et al. · 2023-11-09
- TruthfulQA: Measuring How Models Mimic Human Falsehoods · arXiv · Stephanie Lin et al. · 2021-09-08
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.