Preventing AI hallucinations: why they happen and how to keep them out of your processes

Someone asks AI for a few market figures for a presentation. They look good, there's a source attached, and they go to the customer. Later it turns out the source doesn't exist. That's a hallucination: output that sounds convincing but is wrong.

It isn't a bug that an update will fix. It follows from how language models work. You can't switch it off. You can set things up so a hallucination never reaches a customer unchecked.

Why does AI make things up?

A language model calculates, word by word, what is likely to come next. Nothing is looked up and nothing is checked, unless the model is allowed to use a source or search function.

Ask about a legal provision the model doesn't know, and "article 14, paragraph 3" is a perfectly good continuation. It looks like a legal provision and sounds like one. That it doesn't exist plays no part for the model.

That leads to the pattern that does most damage: persuasiveness says nothing about accuracy. A vague answer is rarely made up. A precise, formal answer with a citation sometimes is, precisely because the model can imitate that format perfectly.

Where does it go wrong in businesses?

Figures in customer material

Market figures for a presentation or quote that nobody checks. This happens most often and hurts most, because it goes outside the company.

Legal and tax texts

A smoothly written clause with a reference to legislation that's wrong. Without a check by someone who knows the subject, it goes out the door.

Product specifications

A model that doesn't know your products fills in plausible specifications. In a quote, that's a commitment. See creating quotes with AI.

Agents that take several steps

This case is underestimated. A mistake in step two becomes the assumption in step three and the basis for step six. In a chain, an error doesn't stand out, it compounds. See AI agent use cases for SMEs.

Six measures that work

1. Provide sources instead of asking

By far the most important. A model that has the contract, the manual or the price list in front of it has nothing to make up. "What's the notice period for this kind of contract?" is guessing. "Here's the contract, what does it say about the notice period?" is reading.

2. Ask for the exact passage

"Quote the sentence you're basing this on." If that sentence isn't in the document, you see it at once.

3. Make checking a rule

Set out in your AI policy which types of information are always checked before they go out: figures, dates, legal provisions, quotes, names and commitments. A rule gets followed, a good habit doesn't. See drawing up an AI policy.

4. Look at where the output ends up

A mistake in an internal note is an inconvenience. The same mistake in a quote, annual report or legal document is something else. Match the check to the destination.

5. Break tasks up

One request that fetches data, analyses it, draws conclusions and writes a text all at once can't be checked. Four separate steps with a look in between can.

6. Teach people how it works

Anyone who understands that a model predicts text rather than looking up facts will start checking by themselves. That's the part of AI literacy that changes behaviour most. See AI literacy is mandatory.

What doesn't work

• Asking whether it's right. The model generates likely text again and usually confirms its own invention. It feels like a check, but it isn't.
• Putting "don't make anything up" in the instructions. It helps a little and mostly gives a false sense of security. The model can't tell that it's making something up.
• Having a second model check. Two models without a source can both go plausibly wrong and then confirm each other.
• Waiting for better models. They've become better at signalling doubt and at using sources. The mechanism stays the same.

How much checking is enough?

• Internal brainstorm or draft text: low risk. Common sense will do.
• Summary of a supplied document: low risk. Spot-check the key points.
• Customer email or quote text: medium. Read in full before sending, check amounts and commitments separately.
• Figures and market data in external material: high. Trace every claim to a source.
• Legal, tax or medical content: high. A check by a specialist, without exception.
• Agent acting on its own: high. A person at every action that can't be undone.

The rule behind this: match the check to the consequences of a mistake, not to how reliable the answer feels.

How we do it ourselves

When we build an agent, it gets the sources it has to work with. An answer without a source excerpt doesn't go through. Amounts come from the system, not the model. And for any action that can't be undone, there's a person in between. What that looks like in your own systems is in connecting an AI agent to your existing systems.

Hallucinations aren't a reason not to use AI. They're a reason to set it up differently. Want to know where AI can safely add value in your processes? We map that out in the AI Opportunity Audit.