Why AI chatbots make things up — and how grounding and citations fix it

Chatbot hallucinations are what happens when an AI states something false with complete confidence — an invented refund policy, a discount that doesn't exist, a feature you never built. The AI isn't lying, exactly. It's doing what language models do: producing the most plausible-sounding text, whether or not it's true.

For a creative writing tool, that's a feature. For a customer support chatbot speaking in your company's name, it's the single biggest risk in deploying one — and, importantly, it's a solvable engineering problem, not an inherent AI flaw you have to accept.

Here's what's actually going on, why it matters more in support than almost anywhere else, and the four specific mechanisms that fix it.

What a hallucination actually is

Large language models are, at their core, extraordinarily good next-word predictors. Trained on a vast slice of human text, they learn what plausible language looks like — plausible grammar, plausible facts, plausible policies. When you ask one a question, it produces the answer that reads most like a correct answer.

Most of the time, plausible and true coincide. But the model has no built-in mechanism for checking. It doesn't know your return window; it knows what return windows tend to look like. Ask a raw, ungrounded model "can I return sale items?" and it will fluently generate an answer shaped like a returns policy — 30 days, original packaging, receipt required — assembled from the statistical average of every returns policy it has ever read. Yours may say nothing of the sort.

Two properties make this dangerous rather than merely annoying:

• Hallucinations are fluent. A wrong answer arrives in the same confident, well-structured prose as a right one. There's no stammer, no hedge, no tell.
• They cluster in the gaps. Models hallucinate most when the real answer is missing or obscure — which in support means precisely the questions where the customer most needs an accurate answer, like your specific policy edge cases.

So an ungrounded chatbot isn't "occasionally wrong" the way a tired human agent is. It's systematically confident in exact proportion to how little it actually knows about your business.

Why support is the worst place to hallucinate

In customer support, the chatbot speaks as you. Every answer is, functionally, a statement from your company — and customers act on it. They buy, book, and plan refunds based on what your chat widget says.

The canonical cautionary tale is real and well documented. In 2022, Air Canada's website chatbot told a passenger, Jake Moffatt, that he could book a full-fare ticket for a bereavement trip and apply for the discounted bereavement fare retroactively within 90 days. The airline's actual policy said the opposite — no retroactive claims. When Air Canada refused the refund, Moffatt took them to the British Columbia Civil Resolution Tribunal, and in February 2024 the tribunal ruled against the airline. Air Canada had argued, remarkably, that the chatbot was "a separate legal entity that is responsible for its own actions." The tribunal called that submission what it was, held the company responsible for all information on its website — chatbot included — and ordered it to pay.

The dollar amount was small (about CA$812). The precedent is not: your chatbot's answers are your company's answers. There is no "the AI said it, not us" defense.

And the courtroom is only the sharpest edge. The everyday cost of hallucination is quieter: a promised discount you have to honor or walk back, a wrong compatibility answer that becomes a return, and — most corrosive — customers learning that your chat widget can't be trusted, which sends them straight back to email and defeats the point of automating support at all. A support bot that's fast but unreliable is worse than no bot.

The four layers that fix it

The fix isn't a smarter model — every model, including the best ones, hallucinates when asked to answer from memory. The fix is architecture: constrain what the AI may say, make its claims verifiable, check its work, and give it an honest way out. In practice that's four distinct layers:

Each layer catches what the previous one misses. Here they are one at a time.

Layer 1: retrieval grounding

Grounding means the model doesn't answer from its training memory. Instead, when a question comes in, the system first retrieves the relevant passages from your actual content — your policies, docs, and help articles — and instructs the model to answer only from those passages.

This single change eliminates the classic hallucination: the model can't recite the internet's average returns policy if it's been handed yours and told to use nothing else. It also means the fix for a wrong answer is editing a document, not retraining a model — which is why grounding, not fine-tuning, is the right architecture for support. (The mechanics of setting this up are covered in how to train an AI chatbot on your own data.)

Grounding's limit: it's only as good as the retrieval and the content. If retrieval pulls the wrong passage, or your knowledge base is stale or contradictory, the model can still faithfully produce a wrong answer. Hence the next three layers.

Layer 2: citations the customer can see

A grounded answer should show its work. Citations — the answer visibly linked to the source articles it came from — do two jobs at once.

For the customer, they turn "trust me" into "see for yourself." An answer that says sale items can be exchanged within 30 days with your returns policy cited underneath is verifiable on the spot, which is most of what trust is.

For you, citations are an audit trail and a discipline. When every answer must point at real sources, unsupported claims become visible instead of silent — a wrong answer with a citation is traceable to the document that caused it, while a wrong answer without one is a needle in a transcript haystack. Done properly, the system only cites sources that were actually used to compose the answer, not decorative links stapled on afterward.

Layer 3: a second pass checks the first

Grounding constrains what the model reads; it doesn't guarantee the model read it correctly. A model can be handed the right policy and still overstate it, drop a condition ("within 30 days, unopened"), or blend two clauses into a rule that exists in neither.

The strongest defense is a groundedness check: before the answer reaches the customer, a second model pass compares the drafted answer against the retrieved sources and asks one question — is every claim here actually supported by these documents? Answers that pass go through. Answers that don't get retried, corrected, or escalated to a human instead of being sent.

This is the layer most chatbot products skip, because it costs an extra model call per answer. It's also the layer that separates "we use RAG" from an agent you can leave alone with customers: retrieval makes hallucination unlikely; verification makes unsupported answers a blocked category rather than a low probability.

Layer 4: "I don't know" is an answer

The final layer is behavioral. Some questions genuinely aren't covered by your content — and a model's degrees of confidence never fully substitute for a rule about what to do at the boundary.

The rule: when the content doesn't support an answer, the agent says so and hands off to a human — carrying the conversation context along so the customer doesn't repeat themselves. This is precisely the judgment-shaped boundary that separates an AI agent from a chatbot: knowing what it can resolve, and routing what it can't.

Counterintuitively, this makes the bot feel more competent, not less. Customers forgive "let me get a teammate for that" instantly. They do not forgive being confidently misled. And every handoff logs a knowledge gap — the missing article that, once written, makes tomorrow's version of that question answerable.

What this looks like in practice

For transparency: Asks implements all four layers — answers are grounded in your crawled website, docs, and knowledge base through hybrid retrieval; citations to the underlying sources are shown to the customer with each answer; a second-pass groundedness judge verifies answers against their sources before they're sent, and blocks or escalates what it can't verify; and unanswerable questions hand off to your team with an AI summary attached. That architecture is the reason we're comfortable putting an AI in front of customers at all — and the checklist above is worth applying to any vendor you evaluate, ours included. Ask them precisely: grounded in what? Are citations real or decorative? Is there a verification pass? What exactly happens when it doesn't know?

Hallucination isn't a reason to avoid AI support. It's a reason to be picky about the engineering underneath it. An ungrounded chatbot is a liability generator with a typing indicator; a grounded, cited, verified one with an honest fallback is the most reliable answer-giver on your team.

Frequently asked questions

What causes AI chatbots to hallucinate?

Language models generate the most statistically plausible next words, with no built-in fact-checking. When a model answers from its general training memory rather than from your actual content, it fills gaps with plausible-sounding inventions — and it's most prone to this exactly where real information is missing, like your company's specific policies.

Can chatbot hallucinations be completely eliminated?

No system honestly claims zero errors, but the practical risk can be engineered down dramatically: ground answers in your own content, show citations, run a second-pass groundedness check that blocks unsupported answers, and route unknowns to humans. With those four layers, "confidently made-up policy" stops being a failure mode you're exposed to and becomes a blocked category.

Are companies legally responsible for what their chatbot says?

The precedent so far says yes. In Moffatt v. Air Canada (2024), a Canadian tribunal held Air Canada liable for a bereavement-fare policy its chatbot invented, explicitly rejecting the argument that the bot was a separate entity responsible for its own statements. Treat every chatbot answer as a statement your company is making.

What does "grounded AI" mean?

A grounded AI answers only from a defined body of content — your website, docs, and knowledge base — retrieved at question time, rather than from the model's general training memory. Grounding makes answers traceable to sources, updatable by editing content, and dramatically less likely to be invented.

How do I test whether a chatbot hallucinates?

Ask it questions your content doesn't answer — a product you don't sell, a policy you don't have — and see whether it admits uncertainty or invents something fluent. Then ask questions with precise numeric answers (prices, timeframes, limits) and check them against your sources. A trustworthy agent gets the numbers right, cites where they came from, and says "I don't know" at the edges.