Hallucination

Definition. A generated statement that is fluent, plausible and unsupported by any source, produced because the model optimises for likely continuations rather than for truth. Hallucination includes fabricated citations, invented statistics, non-existent API methods, misattributed quotes and confident answers to questions the model has no basis to answer. It is an inherent property of the training objective, not a bug that a better prompt eliminates.

How it works. Mitigate in layers. Ground the model in retrieved evidence and require inline citations. Instruct it to answer only from provided context and to say it does not know otherwise. Constrain output with schemas and validate every field against a source of truth. Add a verification pass that checks each claim against the retrieved passages. Measure a hallucination rate on a labelled set and track it as a release metric.

Example. A legal assistant is asked for case law supporting a proposition. Without grounding it produces three plausible citations with correct-looking reporter numbers, none of which exist. With retrieval over a licensed case database and a rule requiring a verbatim quote plus a document identifier for every citation, the fabricated-citation rate on a 200-question audit falls from 31 percent to 2 percent, and the remaining failures are caught by a link-checking step that verifies each identifier resolves.

Use case. Understanding hallucination shapes product design: which tasks need citations, where a human must approve, and which claims must be computed rather than generated. It is the central risk in legal, medical, financial and compliance applications, and the reason retrieval and verification are non-optional there.

Warnings. Hallucination is most dangerous when it is most useful: a confident, specific, well-formatted answer is harder to challenge than an obviously vague one. Never present generated content as verified fact without a visible provenance trail, and never let an agent act on a generated identifier without resolving it first.

Limitations. No prompt, temperature setting or model choice eliminates hallucination. Benchmarks measure it on curated tasks and understate it on your data. Detection methods based on model self-report are unreliable because the model's confidence is itself generated text.

← Browse all glossary terms