AI Bias
AI bias is the tendency of a model to produce systematically different outcomes for different groups — by sex, race, age, postcode, nationality — not because the group matters to the task but because the training data, the features or the objective encoded the difference. In fintech it is a fair-lending and consumer-protection problem with law behind it: a credit model that declines women more, an identity check that fails darker faces more.
- Also known as
- Algorithmic bias, Model bias, Disparate impact
- Used at
- credit scoring and pricing, fraud and AML models, identity verification, insurance pricing, hiring, customer-service routing — anywhere a model decides about people
- Binding
- anti-discrimination and fair-lending law (the US Equal Credit Opportunity Act and disparate-impact doctrine, the EU's equality directives and the AI Act's obligations on high-risk systems, the UK's Equality Act and Consumer Duty); regulators expect testing and mitigation, and enforcement has followed
- Typical sources
- training data that reflects past discrimination; proxies for protected characteristics (postcode for race, first name for sex, shopping patterns for age); objectives that reward outcomes correlated with a group; and measurement — a model tested on a population unlike the one it serves
- Typical measures
- outcome rates by group (approval, pricing, false-positive rates), disparate-impact ratios (the four-fifths rule in US practice), equalised odds and calibration across groups; mitigation by removing proxies, reweighting, constrained training, and human review of edge cases
- Parties
- the model's developers, the business deploying it, the customers affected, the regulators, and the auditors and fairness researchers who test it
How AI bias works in practice
A lender trains a credit model on ten years of its own lending. The data contains the decisions of human underwriters, who approved fewer applicants from certain postcodes; the model learns that postcode predicts default — partly because it does, and partly because those applicants were given worse terms and defaulted more. The model never sees race; it sees postcode, and postcode is race in much of the country. Deployed, it declines applicants in those postcodes at twice the rate, with a default prediction that is partly true and partly the history it inherited. The lender's fair-lending test — approval rates by protected group, with a disparate-impact ratio below 0.8 — finds it; the regulator, if the lender does not, finds it later.
Mitigation is engineering and judgement. Remove the proxies, and the model finds others; constrain the training so that outcome rates across groups fall within tolerance, and predictive power drops a little; test on the population the model will serve, not the one it was trained on; keep the features the model may use to those with a business justification the lender can defend; and document all of it, because the regulator's question is not "is the model fair" but "how do you know". Identity-verification vendors learned the same lesson from face-matching models trained on lighter faces, and fraud teams from models that flagged foreign names.
For an acquirer of a fintech, the model's fairness testing is diligence with a legal consequence: a biased model is a redress liability and a regulatory finding the buyer inherits.
Example
A consumer lender's new model lifts approval by 8% at the same loss rate. Its fairness review before deployment: approval rates by sex, age band, and (inferred, for testing only) ethnicity; the model approves applicants under 25 at 62% of the rate of those over 40 at the same score, driven by a "years at current address" feature. Age is a protected characteristic; the feature is a proxy. The team removes it and two related features, retrains, and loses 1.5 points of the lift; the under-25 ratio moves to 0.91. The documentation — the tests, the features removed, the trade-off accepted — goes to the model-risk committee and, a year later, to the regulator's thematic review, which cites the lender as an example of what it expected and fines two peers that had not tested.
Common mistakes
- Removing the protected characteristic and believing the model cannot discriminate. - Testing fairness once, at build, and never after retraining on new data. - Fairness measured on the training population rather than the deployed one. - Vendors' models deployed with no fairness testing because "the vendor did it". - Treating a fairness finding as a PR problem when it is a legal one.