---
title: "Hallucination"
url: "https://mahouse.world/glossary/hallucination"
type: "glossary term"
topic: "AI risks"
author: "Ihor Vlasov"
reviewed: "2026-09-20"
published: "2026-08-10"
---

# Hallucination

A hallucination is an output from a language model that is false, invented or unsupported — a citation that does not exist, a fee the product does not charge, a regulation misquoted — delivered in the same confident register as a correct answer. It is a property of how the models work, reduced by grounding in retrieved documents and constraining what the model may assert; for a fintech it is why a customer-facing model needs a source and a human.

**Also known as:** AI hallucination, Confabulation, Fabricated output

## Key facts

- **Used at:** every deployment of a language model — customer support, document summarisation, research assistants, code generation, compliance drafting — and the first thing a fintech's risk committee asks about
- **Binding:** no law names it, but the outcomes are regulated — a chatbot that misstates fees or terms is a misleading statement; a compliance assistant that invents a rule is a control failure; Air Canada was held liable in 2024 for a refund policy its chatbot invented
- **Typical causes:** the model predicts plausible text rather than retrieving facts; gaps in training data filled by pattern; questions outside the model's knowledge answered anyway; prompts that reward confident completion; long contexts in which the model loses track
- **Typical mitigations:** retrieval-augmented generation (the model answers from documents it is given and cites them), constrained outputs (the model fills a template rather than composing freely), tool use (the model calls the balance API instead of guessing), verification passes, confidence thresholds with fallback to a human, and evaluation suites that measure hallucination rates by task
- **Parties:** the model developers (who reduce it at training), the deploying business (which reduces it at deployment and answers for what remains), the customers, and the regulators reading the transcripts

## How hallucination works in practice

A customer asks a fintech's support bot whether there is a fee for a SEPA transfer over €10,000. The bot, built on a general model with no access to the fee schedule, answers "yes, 0.5% above €10,000" — fluently, and falsely; the product charges nothing. The customer screenshots it and complains; the regulator, reading the complaint, asks how a bot was allowed to state fees. The same bot, built with retrieval — the fee schedule is fetched and placed in the model's context, and the model is instructed to answer only from it and to say when it cannot — answers "SEPA transfers are free under the standard plan (fee schedule, section 3)", or "I can't find that in the fee schedule; let me connect you to an agent." The model did not become more truthful; it was given the truth and told to use it.

The engineering is layered. Retrieval grounds answers in the company's documents; tools replace guessing with lookups (balances, transaction status, eligibility); templates constrain the shape of the answer; a second model or a rule set checks the first's claims against the sources; a confidence gate routes uncertain cases to a person; and an evaluation suite measures, before release and continuously, how often the system says something the sources do not support. The residual rate is never zero, which is why a fintech decides what the model may assert unaided (nothing about money) and what it may only relay.

For a buyer of an AI vendor or an AI-heavy fintech, the hallucination rate on the deployed tasks — measured, not claimed — and the controls around it are the technology diligence.

## Example

A wealth platform launches a research assistant that answers customers' questions about funds. A pre-launch evaluation on 500 questions finds 9% of answers contain a claim unsupported by the fund documents — performance figures rounded wrong, a fee quoted from a different share class, a manager named who left. The team adds retrieval over the fund documents with citations, a rule that every numeric claim must appear verbatim in a cited source, a verification pass, and a fallback ("I don't have a source for that") when no citation is found. The rate falls to 1.2%, and every remaining case is a wrong citation rather than an invented fact — visible to the customer, who can click through. The compliance team signs off with a monthly sample review of 200 answers; the regulator, shown the evaluation, asks for the number in the platform's next report.

## Common mistakes

- Deploying a general model to answer questions about money without retrieval or tools.
- Measuring "accuracy" once on a demo and never on the deployed task.
- Prompts that forbid the model to say "I don't know".
- Treating a citation as proof; the model can cite a document that does not say what it claims.
- Assuming a lower hallucination rate in a vendor's benchmark applies to the fintech's own documents and questions.

## Related terms

- [RAG](https://mahouse.world/glossary/rag.md)
- [Guardrails](https://mahouse.world/glossary/guardrails.md)
- [Large Language Model](https://mahouse.world/glossary/large-language-model.md)
- [Prompt Injection Attack](https://mahouse.world/glossary/prompt-injection-attack.md)
- [Explainable AI](https://mahouse.world/glossary/explainable-ai.md)
- [Audit Trail](https://mahouse.world/glossary/audit-trail.md)

— Ihor Vlasov, M&A FinTech Expert. Last reviewed 2026-09-20. https://mahouse.world/authors/ihor-vlasov