Benchmarking Hallucination in Language Models

Hallucination, in the sense used across natural language generation research, is not mere nonsense. The awkward fact is that the text often reads well. Sentences parse, entities look familiar, and the tone matches the prompt — yet the content contradicts a source document, invents a citation, or states a confident falsehood. Benchmarking that failure means defining what counts as unfounded, fixing a reference against which to judge, and reporting rates in a way other groups can repeat.

A 2022 survey of hallucination in NLG organises the problem across tasks such as summarisation, dialogue, and data-to-text. It stresses that evaluation inherits the shape of the task. Where a source document or table exists, the natural question is faithfulness: does the output stay within what the source supports? Where the model is asked to speak about the world at large, the question becomes factuality against external knowledge. The survey’s intrinsic and extrinsic distinction is practical here. Intrinsic errors clash with the given input. Extrinsic errors add material the input does not license. Both can look fluent; only the first is a direct contradiction you can often spot with overlap, entailment, or question-answering checks against the source.

Benchmarks therefore tend to bundle three ingredients. First, a corpus of inputs with known constraints — news articles for summarisation, records for data-to-text, multi-turn contexts for dialogue. Second, a generation setup that is held fixed so systems are comparable. Third, a scoring layer that may mix automatic metrics with human annotation. Automatic scores are attractive because they scale, but the survey is frank that they only partly track human judgements of hallucination. Lexical overlap can punish valid paraphrase and miss fluent invention. Learned metrics and natural-language inference models help on intrinsic cases and still struggle when the error is an unsupported but plausible detail.

Open-ended generation exposes a further gap: there may be no single source document. Factuality checks then lean on knowledge bases, retrieved evidence, or human fact-checkers. That shifts cost and introduces its own disagreements about what is current, contested, or underspecified. Benchmarks in this regime often report both error rates and inter-annotator agreement, because the label itself is partly a matter of judgement.

A complementary line of work asks whether the model can contribute a signal about its own reliability. In Language Models (Mostly) Know What They Know, models are prompted to answer and then to assess whether those answers are correct. Self-evaluation — including probability mass on a true/false follow-up — often correlates with actual accuracy, and calibration tends to improve with scale. For benchmarking, that matters because a system that hallucinates less is only one goal; a system that flags its own shaky claims is another. Self-knowledge scores do not replace external faithfulness tests, but they add a measurable axis: not only how often the model is wrong, but how often it can tell.

Taken together, the practice of benchmarking hallucination is less a single leaderboard than a set of disciplined comparisons. Define the reference. Separate contradiction from unsupported addition where the task allows. Report automatic scores with their known blind spots. Keep humans in the loop for the cases metrics blur. And, where self-evaluation is available, record whether the model’s confidence tracks the truth of what it just said. Progress claims become comparable only when those choices are explicit.