RAG
Retrieval-augmented generation (RAG) is the design in which a system first retrieves the documents relevant to a question — from a knowledge base, a customer's records, a policy library — and then gives them to a language model as context, so that the answer is grounded in specific, current, citable sources rather than in what the model remembers from training. It is the standard architecture for enterprise assistants and the main defence against hallucination.
- Also known as
- Retrieval-augmented generation, Retrieval augmented generation, Grounded generation, RAG pipeline
- Used at
- customer and employee assistants over policies and knowledge bases, document question-answering, compliance and legal research, analyst tools over filings and reports, support with the customer's own records as context, and any application where the answer must come from the company's sources
- Binding
- data protection on what is retrieved and sent to the model (access controls must carry through — a user must not retrieve what they could not read); vendor terms on the context sent; sector rules on what an assistant may tell a customer; the EU AI Act's transparency duties for AI interactions
- Typical pipeline
- documents are split into chunks and embedded into vectors stored in a vector index (with metadata for filtering); a question is embedded and the nearest chunks retrieved, often combined with keyword search and re-ranked; the top chunks go into the model's prompt with instructions to answer from them and cite; the answer is checked for grounding; the whole is evaluated on a set of questions with known answers
- Typical failure modes
- the right document not retrieved (chunking, embedding or ranking failure), the wrong document retrieved (ambiguity, stale content), the model answering from memory despite the context, citations that do not support the claim, access controls not enforced at retrieval, and an index that lags the source
- Parties
- the engineers building the pipeline, the content owners whose documents are the knowledge, the vector-database and embedding vendors, the model provider, the security team for access control, and the users who trust the citations
How RAG works in practice
A merchant asks the assistant "can I get my reserve released early?". The system embeds the question, retrieves the reserve policy, the merchant's account record showing a reserve applied after a chargeback spike, and the appeal procedure; re-ranks them; and sends the top five chunks to the model with the instruction to answer only from them and cite each claim. The model answers: reserves are reviewed after 90 days or on appeal with evidence; this merchant's was applied on 3 March; here is the appeal form — with citations to the policy paragraph and the account record. A grounding check confirms each sentence is supported by a cited chunk. The merchant gets a correct, current, specific answer; the model never had to know the policy, and when the policy changes the index updates and the answer changes with it.
The engineering is mostly in retrieval. Chunk documents badly and the relevant paragraph is split from its heading; embed with a weak model and synonyms miss; skip keyword search and exact identifiers (a reason code, a product name) fail; skip re-ranking and the fifth-best chunk is sent instead of the first. Access control is the security question: the retrieval must apply the user's permissions, or the assistant becomes a way to read documents the user could not open. Evaluation is the discipline: a set of questions with known correct answers and sources, run on every change, measuring retrieval hit rate and answer correctness separately, because a wrong answer with a good retrieval is a prompt problem and one with a bad retrieval is an index problem. For a fintech, RAG is how a rented model becomes a company assistant that cites its sources; for a buyer, the target's evaluation results and access-control design say whether the assistant is safe.
For a fintech founder, RAG is the architecture that makes a language model useful and defensible in a regulated business; the retrieval is the product, the model is a component.
Example
A payments company launches an internal assistant over 4,000 policy and procedure documents for its 300 support agents. Version one: naive chunking, embedding search only, top three chunks, no evaluation. Agents report wrong answers; a spot check finds 30% of answers unsupported by the cited sources, mostly because the relevant paragraph was not retrieved. Version two: chunking by document section with headings attached, hybrid search (embeddings plus keywords), re-ranking, top eight chunks, a grounding check that flags unsupported sentences, access control from the document system, and an evaluation set of 400 questions written by the compliance team with reference answers. Retrieval hit rate rises from 64% to 93%; answer correctness from 70% to 94%; unsupported claims fall to 2% and are flagged. Average handling time falls by 25%, and the compliance team reviews the evaluation results monthly.
Common mistakes
- Blaming the model for a retrieval failure; measure them separately. - Chunking that separates paragraphs from their context. - Embedding search alone; exact terms need keyword search too. - Retrieval that ignores the user's permissions. - No evaluation set, so quality is an impression.