What is AI hallucination? Why does AI make things up, and how do we fix it?

You ask the AI agent, "How many days is your return window?"

It answers, "30 days."

You check — it's actually 14 days.

AI doesn't not know. It doesn't even know that it doesn't know. It just confidently delivers an answer that sounds reasonable.

This is hallucination. And it's the issue that scares brand owners most about AI customer service.

TL;DR

• Hallucination is when AI confidently states incorrect information
• A more accurate way to put it Confabulation: it's not "making things up," it's "constructing a story that sounds plausible"
• This isn't AI breaking — it's a structural issue with language models
• General-purpose AI (using ChatGPT / Claude directly) for customer service carries the highest hallucination risk
• RAG plus rigorous prompt designcan reduce it significantly, but not to zero
• The most dangerous hallucination types in Taiwan customer service: prices, dates, and policy details

First, an example I witnessed myself

I noticed something while testing Claude.

On writing tasks, Claude just gets on with it. But for code, it usually won't act directly — it tells me to do it myself.

I asked Claude why.

It answered: "Because Anthropic provides additional guidance for coding contexts."

Sounded reasonable. I pressed further: "When did Anthropic say that? Can you find it?"

Then Claude paused.

It said: it shouldn't have spoken with that kind of certainty, because it actually didn't know. It could observe that its behavior differed between the two contexts, but it didn't know why. What it had just said was "reverse-engineering a plausible reason from its own behavior."

This act is called confabulation.

Hallucination vs. confabulation: what's the difference?

The word "hallucination" suggests AI is dreaming, seeing things that don't exist.

But a more accurate term is confabulation — in neuroscience, this describes brain-injured patients who fill memory gaps with "fabricated but subjectively real memories." The patient isn't lying. They genuinely believe what they're saying.

What language models do is very similar.

Their task is to "produce the next most likely token."

Not to "find the correct answer."

It's to "produce the continuation that best fits the language pattern, given the context."

So when it doesn't know the answer, it doesn't say "I don't know." It produces "the most plausible answer for this context."

Sometimes this answer is right. Sometimes it's wrong. The AI itself can't tell.

Why is general-purpose AI especially dangerous for customer service?

ChatGPT, Claude, Gemini — these general-purpose large language models are trained on the entire internet.

They know a lot. But what they "know" and your brand's actual policies are two different things.

Plug ChatGPT directly into customer service without giving it your knowledge base, and it'll answer your customers using "the most common return policy in the world."

Your return window is 14 days; it says 30. Because most e-commerce sites are 30.

This isn't AI deliberately deceiving people. It's the most reasonable guess it can make based on language statistics.

Which questions trigger hallucination most easily?

From Satsuma's hands-on experience, here are the most dangerous categories in Taiwan customer service:

Numbers: prices, shipping fees, discounts, deadlines. AI has no particular reverence for "numerical correctness" — as long as a number fits the context, it'll say it.

Policy details: return conditions, warranty scope, exceptions. These vary by brand, and without a reliable source, AI applies what it considers the "typical" answer.

Latest information: AI training data has a cutoff date. Ask about the latest campaign or newest product specs and it'll either say it doesn't know, or give you an outdated answer — possibly with great confidence.

Gap-filling under uncertainty: this is the most dangerous kind. When a question sits at the boundary of what it knows, it won't say "I'm not sure." It will stitch the known parts into a complete answer, including the parts it doesn't actually know.

How do we fix it?

There's no complete fix. But there are ways to push the risk down to an acceptable level.

First approach: RAG

Constrain AI's answers to your knowledge base. It can only respond based on what you've provided — if there's no data, it says so.

This is the single most effective measure for reducing hallucination.

But RAG isn't a silver bullet. If the knowledge base itself is wrong, or the embedding retrieves inaccurate data, AI will still get it wrong.

Second approach: rigorous prompt design

Tell the AI explicitly: "If it's not in the knowledge base, say you don't know. Don't guess."

Sounds simple. But many people don't do this, or do it without covering all scenarios.

Watch out especially for "half-knowing" situations — when AI has part of the answer but is uncertain about the rest. In these moments, it easily mixes the certain and uncertain parts together.

Third approach: human review for high-risk answers

Don't let AI directly reply on prices, refunds, or legal terms. Have the AI say, "Let me have our team confirm and get back to you," then hand off to a human.

This isn't laziness — it's the correct architectural call.

Fourth approach: continuous monitoring

After launch, review AI's response logs regularly.

Pay special attention to conversations where customers pushed back. "AI told me yes, but you say no" — these conversations are often the result of hallucination.

A test you can run right now

Find your AI customer service and ask a few questions where you know the answer but the knowledge base doesn't explicitly cover: - "What does your warranty cover?" - "If I've used half of it, can I still return it?" - "Are prices the same in Taiwan and Hong Kong?"

Watch how the AI answers.

If it gives a plausible-sounding answer you never authorized — you've just seen hallucination.

Summary

Hallucination isn't a bug — it's a design feature of language models.

Their task is to "produce plausible language," not to "verify factual correctness."

This doesn't mean AI customer service can't be used. It means you need to build it the right way:

Hallucination in AI customer service can't be reduced to zero. But it can be designed so that "when it happens, you know" and "when you know, you can fix it."

That's what mature AI customer service architecture looks like.

This article is part of the "AI customer service tech explainer" series: - What is RAG? - What is embedding? - What is AI memory? - What is AI hallucination? ← You are here

FAQ

Q: Is hallucination the same as AI lying?

No. Lying requires knowing the truth and deliberately stating something false. AI has no "knowing the truth" premise. Every word it produces is what it genuinely considers "the most plausible continuation in this context." That actually makes it worse — because it doesn't know it's wrong.

Q: GPT-4o is newer — does it hallucinate less?

Less than older versions, but not gone. Newer models improve on "factual questions," but not on "information specific to your brand." Your return policy isn't in its training data. This is an architectural issue, not a question of model strength.

Q: My AI customer service is live — how do I know if it's hallucinating?

The most direct way: review 10 to 20 customer conversations each week, especially ones where customers responded or pushed back. You can also design "honeypot tests" — ask questions where you know the answer and check if AI gets it right. Test regularly, otherwise you'll forget once it's live.