Explainable AI
Explainable AI is the set of methods that make a model's outputs understandable to a person — inherently interpretable models (scorecards, decision trees), post-hoc explanations of complex ones (feature attributions such as SHAP, counterfactuals: "you would have been approved with €300 more income"), and the documentation around them. For a lender it is the law: a declined applicant must be told the principal reasons.
- Also known as
- Explainability, XAI, Model explainability, Interpretable machine learning
- Used at
- credit decisions (adverse-action reasons in the US, the EU's right to an explanation of automated decisions under GDPR and the AI Act's transparency duties), fraud and AML investigations (an analyst must be able to justify closing or escalating an alert), model risk management, and the diligence of any AI company
- Binding
- yes, for high-stakes decisions — ECOA and Regulation B in the US require specific reasons for a decline; GDPR Article 22 and the AI Act in the EU; the UK's guidance on explaining AI decisions; model-risk rules (SR 11-7 and its equivalents) require models to be understood by those who validate them
- Typical methods
- interpretable-by-design models where the stakes justify the accuracy cost; feature-attribution methods (SHAP values decompose a prediction into each feature's contribution; LIME fits a local linear model); counterfactual explanations; global explanations of what the model learned; for language models, much weaker tools — citations, chain-of-thought, and honesty about the limits
- Typical tension
- the most accurate models (gradient-boosted trees, deep networks, large language models) are the least interpretable; post-hoc explanations describe the model's behaviour approximately and can be gamed or misread; regulators increasingly accept them for credit with governance, and do not accept "the model said so"
- Parties
- the model's developers and validators, the business using it, the customers entitled to reasons, the regulators, and the vendors of explanation tooling
How explainable AI works in practice
A lender's gradient-boosted credit model declines an applicant. The regulation requires the principal reasons. The lender's explanation layer computes SHAP values for the decision — how much each feature pushed the score toward decline — and maps the top contributors to reason codes in plain language: "high utilisation of existing credit", "short credit history", "recent missed payment". The applicant receives them; a fair-lending reviewer can check that the reasons are legitimate and not proxies; the model-risk team can see, across all declines, which features drive decisions and whether that matches the model's documentation. The model is still a thousand trees nobody can read; the explanation is a faithful-enough summary that the law and the business can use.
The choice of method is a choice about stakes. A fraud model that blocks a transaction can be a black box with a review path, because a wrong decision is reversible in minutes; a credit model that sets someone's mortgage rate should be interpretable enough that its decisions can be defended one by one, and lenders often accept a little accuracy for a scorecard they can explain. Language models are the hard case: their "explanations" are more text from the same model, and the honest practice is to constrain what they decide, cite sources, and keep a human accountable for outcomes.
For a buyer of an AI-driven lender or a fintech vendor, explainability is diligence: can the company produce the reasons for a decision, do the reasons stand up, and has a regulator asked.
Example
A BNPL provider's decision model is a deep network with an approval rate 6% above its old scorecard at the same loss. Its regulator's review asks how declined customers are given reasons. The provider's answer — a generic letter — fails. It builds an explanation layer: SHAP attributions per decision mapped to twelve reason codes, validated by checking that the codes cited match what a change in the feature would do (counterfactual tests), and a quarterly report of reason-code frequencies by group. The layer costs four months and a data scientist; the regulator accepts it; the fair-lending review it enables finds that one reason code ("thin file") is cited for young applicants at three times the rate, which leads to a feature review. The model keeps its 6%; the provider keeps its licence.
Common mistakes
- Deploying an unexplainable model for a decision the law requires reasons for. - Explanations generated by the model itself, and trusted. - SHAP values mapped to reason codes nobody validated, so the reasons are plausible and wrong. - Reading explainability as a one-off deliverable rather than a control that runs on every decision. - Buyers accepting "the model is explainable" without asking to see a decision explained.