Guardrails
Guardrails are the controls placed around an AI system — rather than inside the model — that constrain what it is permitted to do or say: filters on inputs, checks on outputs (grounding, format, policy, figures verified against a source), limits on actions (which tools the system may call, under whose approval), and the monitoring and kill switches around all of it. They exist because a model's training cannot be relied on for a company's rules.
- Also known as
- AI guardrails, LLM guardrails, Output filters, Safety controls
- Used at
- every production language-model application — customer assistants, document pipelines, agents with tools, internal copilots — and, in a broader sense, around any model whose outputs reach customers or systems
- Binding
- the EU AI Act's risk-management and human-oversight obligations for high-risk systems, the transparency duties for AI interactions, and the general-purpose model providers' safety obligations; sector rules on what may be said to a customer (financial promotions, advice, complaints handling); data protection on what enters and leaves the model; and model-risk expectations that controls are documented and tested
- Typical layers
- input guardrails (injection detection, PII redaction, topic and intent classification, rate limits); model-level controls (system prompt rules, structured output schemas, temperature); output guardrails (grounding checks against sources, policy classifiers, format validation, figure verification, refusal of advice); action guardrails (allow-lists of tools, parameter validation, transaction limits, human approval for consequential actions, idempotency); and operational guardrails (logging, monitoring, evaluation, kill switch, incident process)
- Typical principle
- never rely on the prompt for safety — a model can be instructed around any instruction — so the controls that matter are architectural: the assistant that cannot send money cannot be tricked into sending money
- Parties
- the engineers who build them, the compliance and risk teams who specify them, the red teams that test them, the vendors of guardrail tooling and classifiers, and the regulators who ask to see them
How guardrails work in practice
A payments company's merchant assistant is built in layers. Input: every message passes an injection classifier and a PII scrubber; requests for account changes, refunds or disputes are recognised by intent and routed to a human path with the context prepared. Model: a system prompt sets role, tone, rules and the requirement to cite; outputs are constrained to a schema with fields for the answer, the citations and a confidence flag. Output: a grounding check verifies every factual sentence against a retrieved source and blocks unsupported ones; a policy classifier rejects anything that reads as financial advice or a promise; figures are recomputed from the system of record. Action: the assistant has read access to the merchant's record and no write access to anything — it can explain a payout delay and cannot change a bank account, whatever it is told. Operations: every conversation is logged, 5% are reviewed weekly, an evaluation set of 1,000 conversations and 300 attack prompts runs on every change, and a switch turns the assistant off in one minute.
The design discipline is to assume the model will be manipulated and to make the manipulation harmless. Prompt injection will get through some fraction of classifiers; the question is what the attacker can then do, and the answer must be "nothing consequential". Hallucination will occur; the grounding check and the human path are what happen next. The controls are tested by red-teaming — attack prompts, injected documents, adversarial customers — and the results are the evidence a regulator reads. Agents raise the stakes: an agent with tools that act — issue a refund, update a record, send an email — needs allow-lists, parameter validation, transaction limits, approval steps and audit logs, because the model chooses the actions and the guardrails decide which are allowed. For a fintech, guardrails are the model-risk framework applied to a system that talks; for a buyer, the target's guardrail design, red-team results and incident log are the diligence.
For a fintech founder, guardrails are the reason the assistant can be deployed at all: the model is rented, the controls are the company's, and the controls are what the regulator will examine.
Example
A lender's customer assistant, in its first month, is manipulated by a customer into stating that a payment holiday is "guaranteed on request" — a hallucinated policy the model produced under pressure — and the complaint reaches the regulator. The rebuild: a grounding guardrail that blocks any policy statement not supported by a cited policy document; a classifier that routes anything resembling a commitment, advice or a complaint to a human; a red-team suite of 400 prompts, including 150 manipulation attempts, run weekly; and a rule that the assistant never states eligibility for any product, only the criteria and the path to apply. In the following six months: zero unsupported policy statements in the reviewed sample, 3% of conversations routed to humans by the classifier (up from 1%), and CSAT unchanged. The regulator's follow-up review reads the red-team results and the routing logs and closes the matter.
Common mistakes
- Relying on the system prompt for safety. - An assistant with write access to anything it does not need. - No grounding check, so hallucinated policy reaches customers. - Guardrails never red-teamed, so nobody knows what gets through. - Agents with tools and no limits, approvals or audit trail.