Overfitting
Overfitting is a model learning its training data so closely — including its noise, its coincidences and its particular quirks — that it performs well on the examples it saw and worse on anything new; the model has memorised rather than generalised. It is detected by the gap between training and validation performance, prevented by more data, simpler models, regularisation and early stopping, and it is the most common reason a model that looked excellent in development disappoints in production.
- Also known as
- Overfit, Overtraining, Memorisation
- Used at
- every trained model; the training-versus-validation gap is the first number a reviewer reads, and the reason the validation and test sets exist
- Binding
- none; but a model that overfitted its development data and underperforms in production is a model-risk finding, and an overfitted credit model misstates expected loss
- Typical signs
- training accuracy far above validation accuracy; validation loss rising while training loss falls; performance that degrades sharply on data from a later period or a different segment; a model that has "learned" features that cannot matter (an identifier, a timestamp)
- Typical causes
- too many parameters for the data, too many training epochs, too little data, leakage of the target into features, and tuning on the test set (which overfits the selection rather than the model)
- Typical remedies
- more data (real or augmented), a smaller or more regularised model, early stopping at the validation minimum, dropout, weight decay, cross-validation, feature reduction, and out-of-time validation
- Parties
- data scientists, whose daily work is the training–validation gap, and model-risk reviewers, whose first question is how the model performs on data it did not see
How overfitting works in practice
A fraud model with 800 features is trained on six months of data and reaches 99.2% training accuracy and an AUC of 0.99 on training; on the following month it manages 0.87. The model has learned patterns that were true in the sample — that transactions from a particular merchant ID on Tuesdays in March were fraud, because a fraud ring was active then — which are not patterns of fraud but of that sample. Each extra feature and each extra epoch gave it more capacity to memorise; the noise in six months of data is large; and no one checked against the seventh month until deployment. The fix is procedural before it is technical: a validation set from a later period, training stopped when validation performance peaks, features that cannot generalise removed, and the test set touched once.
The regularisation family — penalties on large weights, dropout that randomly disables units, pruning that limits tree depth — makes overfitting harder by constraining what the model can memorise; more data makes it harder by making the noise smaller relative to the signal; a simpler model has less capacity to memorise. Double descent complicates the classical story (very large models can generalise well) but not the diagnostic: the validation gap is still the measure. Leakage is the special case worth naming: if a feature encodes the answer (a "chargeback flag" that is set after the fraud), the model will fit the training data perfectly and be useless — and it will look like extreme overfitting until someone reads the features.
For a fintech, overfitting is the difference between the model that was approved and the model that runs, and the out-of-time validation is the control that catches it before the portfolio does.
Example
A lender's credit model reports a Gini of 0.71 on its development sample, well above the incumbent's 0.58. Model risk asks for an out-of-time test: on applications from the six months after the development window, the new model scores 0.55 — below the incumbent. Investigation finds 40 features derived from a bureau file that was refreshed after the outcome date, so the features partly encoded the outcome; and a tree depth of 14 that let the model memorise segment-level noise. Rebuilt with point-in-time bureau data, depth 6, and early stopping on a later validation period, the model scores 0.63 in development and 0.62 out of time — a real improvement of four points over the incumbent, instead of an illusory thirteen.
Common mistakes
- Reporting training performance, or validation performance after tuning on it, as the expected result. - No out-of-time test; random splits hide drift and leakage. - Leaked features that encode the outcome and produce impossible accuracy. - Training to a fixed epoch count rather than stopping at the validation minimum. - Adding features and depth in pursuit of a development-sample score.