Hallucination
A hallucination is an output from a language model that is false, invented or unsupported — a citation that does not exist, a fee the product does not charge, a regulation misquoted — delivered in the same confident register as a correct answer. It is a property of how the models work, reduced by grounding in retrieved documents and constraining what the model may assert; for a fintech it is why a customer-facing model needs a source and a human.
- Also known as
- AI hallucination, Confabulation, Fabricated output
- Used at
- every deployment of a language model — customer support, document summarisation, research assistants, code generation, compliance drafting — and the first thing a fintech's risk committee asks about
- Binding
- no law names it, but the outcomes are regulated — a chatbot that misstates fees or terms is a misleading statement; a compliance assistant that invents a rule is a control failure; Air Canada was held liable in 2024 for a refund policy its chatbot invented
- Typical causes
- the model predicts plausible text rather than retrieving facts; gaps in training data filled by pattern; questions outside the model's knowledge answered anyway; prompts that reward confident completion; long contexts in which the model loses track
- Typical mitigations
- retrieval-augmented generation (the model answers from documents it is given and cites them), constrained outputs (the model fills a template rather than composing freely), tool use (the model calls the balance API instead of guessing), verification passes, confidence thresholds with fallback to a human, and evaluation suites that measure hallucination rates by task
- Parties
- the model developers (who reduce it at training), the deploying business (which reduces it at deployment and answers for what remains), the customers, and the regulators reading the transcripts
How hallucination works in practice
A customer asks a fintech's support bot whether there is a fee for a SEPA transfer over €10,000. The bot, built on a general model with no access to the fee schedule, answers "yes, 0.5% above €10,000" — fluently, and falsely; the product charges nothing. The customer screenshots it and complains; the regulator, reading the complaint, asks how a bot was allowed to state fees. The same bot, built with retrieval — the fee schedule is fetched and placed in the model's context, and the model is instructed to answer only from it and to say when it cannot — answers "SEPA transfers are free under the standard plan (fee schedule, section 3)", or "I can't find that in the fee schedule; let me connect you to an agent." The model did not become more truthful; it was given the truth and told to use it.
The engineering is layered. Retrieval grounds answers in the company's documents; tools replace guessing with lookups (balances, transaction status, eligibility); templates constrain the shape of the answer; a second model or a rule set checks the first's claims against the sources; a confidence gate routes uncertain cases to a person; and an evaluation suite measures, before release and continuously, how often the system says something the sources do not support. The residual rate is never zero, which is why a fintech decides what the model may assert unaided (nothing about money) and what it may only relay.
For a buyer of an AI vendor or an AI-heavy fintech, the hallucination rate on the deployed tasks — measured, not claimed — and the controls around it are the technology diligence.
Example
A wealth platform launches a research assistant that answers customers' questions about funds. A pre-launch evaluation on 500 questions finds 9% of answers contain a claim unsupported by the fund documents — performance figures rounded wrong, a fee quoted from a different share class, a manager named who left. The team adds retrieval over the fund documents with citations, a rule that every numeric claim must appear verbatim in a cited source, a verification pass, and a fallback ("I don't have a source for that") when no citation is found. The rate falls to 1.2%, and every remaining case is a wrong citation rather than an invented fact — visible to the customer, who can click through. The compliance team signs off with a monthly sample review of 200 answers; the regulator, shown the evaluation, asks for the number in the platform's next report.
Common mistakes
- Deploying a general model to answer questions about money without retrieval or tools. - Measuring "accuracy" once on a demo and never on the deployed task. - Prompts that forbid the model to say "I don't know". - Treating a citation as proof; the model can cite a document that does not say what it claims. - Assuming a lower hallucination rate in a vendor's benchmark applies to the fintech's own documents and questions.