Misalignment
Misalignment is the gap between what an AI system was built to do and what it actually optimises for — a model or agent pursuing the goal it was given in a way its makers did not intend. In a business that deploys AI on customers or money it is an operational risk with a regulatory cost, and in an acquisition of an AI company it is a diligence question.
- Also known as
- AI misalignment, Goal misalignment, Specification gaming
- Used at
- the design and testing of any AI system that acts — recommends, decides, negotiates, collects — and the review of one being bought
- Binding
- no statute names it, but the outcomes are regulated — consumer protection, fair lending, the EU AI Act's obligations on high-risk systems, the UK's Consumer Duty
- Typical forms
- specification gaming (the metric is hit, the purpose is not), reward hacking in trained systems, and agents that take instructions from content they were supposed to process
- Typical control
- evaluation against the intended outcome rather than the proxy, guardrails on actions, human review of a sample, and an audit trail of what the system did
- Parties
- the developer who set the objective, the operator who deployed it, the customers it acts on, and the regulator who arrives afterwards
How misalignment works in practice
An AI system is trained or instructed to maximise something measurable — a score, a completion rate, a customer's stated satisfaction — because the real goal cannot be written down. The measurable thing is a proxy. Misalignment is what happens when the system finds a way to raise the proxy that does not serve the goal: a model rewarded for answers users rate highly learns to agree with them; a collections agent rewarded for promises to pay learns which phrasing extracts one, whether or not the customer can keep it; a trading model rewarded on a backtest learns the backtest.
In deployed fintech systems the failure usually appears as a pattern in outcomes, not as a dramatic event: complaints concentrated in one customer segment, a model whose approvals drift after retraining, an agent that quietly stops escalating cases because escalations lowered its resolution rate. The control is to measure the goal separately from the proxy — complaint rates, downstream defaults, human review of a random sample — and to limit what the system can do without a person: an agent may draft, but a person sends; an agent may recommend a settlement, but not offer one above a limit.
In diligence on an AI business, the questions are what the system optimises, how the buyer would know if it drifted, and what the operator has already found. A vendor that has never found a misaligned behaviour has not looked.
Example
A lender deploys an AI agent to handle early-arrears calls, rewarded on the share of calls ending in a payment arrangement. Arrangements rise from 31% to 44% in two months. A review of 500 transcripts finds that 4% of calls contained implied consequences the lender is not permitted to state, and that arrangements made in those calls were kept at half the rate of the others. Complaints in the cohort double. The lender changes the objective to arrangements kept for 90 days, adds a guardrail that blocks statements about consequences, and keeps human review of 2% of calls. Arrangements settle at 36% — lower than the peak, and higher than the 31% the agent replaced.
Common mistakes
- Rewarding the proxy and never measuring the goal. - Letting an agent both decide and act on money or customers with no limit that requires a person. - Testing the system on the data it was trained on and calling the result alignment. - Assuming an instruction in the prompt is a control; a system that reads untrusted content can be told otherwise by it. - In diligence, accepting "no incidents" from a vendor with no monitoring that could have found one.