Hallucinations explained: why LLMs make things up (and how to stop them)

The single most common worry about deploying an LLM to real users is: “what if it just makes something up?”

It’s a fair worry. LLMs do make things up. They do it often. And they do it with the same confident tone as when they’re correct, which is what makes hallucinations dangerous in production.

The good news is that hallucinations aren’t mysterious — and they aren’t a fixed property of the model. They’re a system-level problem with system-level solutions.

What is a hallucination, actually?

A useful definition: a hallucination is any output that asserts something as true when the model has no real basis for it.

That includes:

• Inventing a fact (a person, a citation, a number).
• Citing a source that doesn’t exist.
• Stating a relationship or cause that the data doesn’t support.
• Confidently answering a question when the right answer was “I don’t know.”

It does not include:

• Reasoning errors. The model attempted to think through something and got it wrong — different problem.
• Refusing to answer when it shouldn’t. Different problem.
• Style or format misses. Annoying, not a hallucination.

Sorting these matters because they have different fixes.

Why hallucinations happen

LLMs are next-token predictors. They generate text that looks like a plausible continuation of what came before. They don’t have a separate “truthfulness” check; they have a probability distribution over likely next tokens.

If your question pattern-matches strongly to something in the training data, the model usually gives you that. If it doesn’t — if the question is ambiguous, or your domain isn’t well-represented — the model still has to produce something, and “something” looks like a confident answer because that’s how text in the training data is shaped.

Two specific failure modes:

• Unsupported confidence. The model doesn’t know, but is structurally incapable of generating uncertainty unless explicitly designed to.
• Plausible patterns. The model knows what a citation looks like, what a phone number looks like, what a court case name looks like — even when the specific citation, number, or case is invented.

What actually reduces hallucinations

There’s no single fix. Production-ready systems combine several:

1. Ground the model in real data (RAG)

The most important single change. When the model is asked to answer from a set of retrieved documents instead of from its training, it has something concrete to reference. Hallucinations drop significantly — but only if retrieval is good. See: what is RAG.

2. Tell the model how to say “I don’t know”

Default prompts implicitly demand an answer. Better prompts explicitly allow — and reward — refusal:

“If the provided context doesn’t contain enough information to answer the question, say so. Do not guess.”

This single instruction, properly written, cuts a meaningful percentage of hallucinations.

3. Constrain the format

Free-form prose gives the model the most room to drift. Structured output — JSON with required fields, a known schema, citations as their own field — narrows the space in which hallucinations can happen.

4. Check the answer against the source

After generation, you can run a second step: “does this answer follow from the provided documents?” This faithfulness check is itself an LLM call, and catches a meaningful portion of cases where the main answer drifted from the source.

5. Cite explicitly

Force the model to attach each claim to a specific source passage. Hallucinated claims become much harder to disguise: there’s nothing to cite.

6. Choose the right model for the task

Some models are noticeably more or less prone to confident hallucination. Worth testing your top 2-3 candidate models on the exact questions where hallucinations would hurt you, before committing.

7. Measure your hallucination rate

The most under-used technique: build a labelled set of test questions, including ones where the right answer is “I don’t know.” Run it before any change ships, run it on a sample of production traffic. If you don’t measure, the rate will silently change every time you update a prompt or a model.

What doesn’t work

A short list of things that sound like they should help but mostly don’t:

• “Just tell the model to be accurate.” Adding “be accurate” or “do not hallucinate” to a prompt is roughly placebo. The model already believes its answers are accurate.
• Bigger model. Larger models hallucinate less in many cases, but not all — and they don’t stop hallucinating. Don’t bet your product on the next model release fixing it.
• Temperature = 0. Lower randomness reduces some variance but doesn’t address the underlying issue. The most likely answer is sometimes a confidently wrong one.

The hard truth

There is no production LLM system that doesn’t hallucinate sometimes. The question is how often, in what failure modes, and whether the system catches them before users see them.

A serious deployment is one where:

• You know your hallucination rate.
• You know what kind of hallucinations dominate.
• You have a refusal path that’s actually used.
• You have monitoring that flags suspicious outputs in production.

That’s a problem you can solve. “Make hallucinations impossible” isn’t.