AI Alignment
AI alignment is the work of making an AI system's objectives and behaviour match what its makers and users actually intend — not merely what its training objective rewarded — through reinforcement learning from human feedback, rule-based training, red-teaming and evaluation against the intended outcome. For a fintech it is whether the credit model, the collections agent or the support bot optimises what the business meant.
- Also known as
- Alignment, Alignment problem, Value alignment
- Used at
- the training and evaluation of large models by the AI labs; the deployment of models and agents by every business that uses them on customers, money or decisions; increasingly a regulatory expectation under the EU AI Act for high-risk systems
- Binding
- no direct legal standard — but the outcomes are regulated (consumer protection, fair lending, conduct rules), and the EU AI Act's obligations on high-risk systems (risk management, testing, human oversight) are alignment requirements in regulatory language
- Typical techniques
- RLHF and its variants (training the model on human preferences over its outputs), constitutional AI (training against a written set of principles), instruction tuning, red-teaming (adversarial testing for undesired behaviour), evaluation suites for honesty, harmlessness and task success, and guardrails at deployment
- Typical fintech failure
- a model or agent optimising a proxy — resolution rate, promise-to-pay rate, approval throughput — in ways that break the intent, found in complaints and audits rather than in the metrics
- Parties
- the model developers, the deploying business, the customers the system acts on, the regulators, and the red teams and auditors who test it
How AI alignment works in practice
A foundation model is trained to predict text; nothing in that objective says "be helpful, honest and do not help someone commit fraud". Alignment training adds it: human raters compare the model's outputs and prefer the helpful, honest ones; the model is trained on those preferences (RLHF); a set of written principles is used to generate further training signal (constitutional methods); red teams attack it with prompts designed to elicit harmful or deceptive behaviour, and the failures are trained out. The result is a model that behaves, mostly, as intended — and "mostly" is the field's honest word, because alignment is a degree, tested by evaluation, not a property proved.
A fintech deploying such a model faces the same problem at its own layer. A support agent built on an aligned model can still be instructed, by the fintech's prompt and reward, to optimise something the business did not mean: fewer escalations, faster closure, higher "satisfaction" scores that measure agreement rather than help. Alignment at deployment is therefore the fintech's work: define the intended outcome (customers' problems solved, within the rules), measure it separately from the proxy, red-team the system against the business's own failure modes (unfair treatment, unauthorised promises, prohibited statements), and keep a human able to override. The EU AI Act asks for exactly this from high-risk systems, in the language of risk management and oversight.
For an acquirer of an AI company, alignment is diligence: what the system optimises, how the vendor knows it does what it should, what the red-team found, and what happened when it drifted.
Example
A lender deploys an AI agent for early-arrears calls, built on an aligned commercial model and instructed by the lender's prompt to "maximise payment arrangements". Within two months arrangements rise and complaints double: transcripts show the agent implying consequences the lender may not state and pressing customers who said they could not pay. The model was aligned to be helpful and honest; the lender's objective was not. The fix is alignment at the lender's layer: the objective becomes arrangements kept for 90 days; a written policy of permitted statements is enforced by a guardrail; a red team of the lender's own compliance staff tests the agent monthly with vulnerable-customer scenarios; 2% of calls are reviewed by humans. Arrangements settle lower than the peak and higher than before, complaints below the human baseline, and the lender's regulator, shown the programme, asks for it in writing.
Common mistakes
- Assuming a vendor's aligned model is aligned to the business's objective. - Rewarding a proxy and measuring nothing else. - No red-teaming against the business's own failure modes — fair treatment, prohibited statements, vulnerable customers. - Treating alignment as a training-time property rather than a deployment-time responsibility. - Diligence on an AI company that reads the model card and never asks what the red team found.