Regularisation
Regularisation is any addition to training that discourages a model from fitting its training data too closely — a penalty on large weights (L2, weight decay), a penalty that drives weights to zero (L1), random disabling of units during training (dropout), limits on tree depth, early stopping, and data augmentation — so that the model captures the pattern rather than the noise. It is the standard defence against overfitting; its strength is set on validation data; too much underfits.
- Also known as
- Regularization, Weight decay, L1 and L2 regularisation, Dropout
- Used at
- nearly every trained model — gradient-boosted trees (depth, leaf size, shrinkage, subsampling), neural networks (weight decay, dropout, normalisation, early stopping), linear models (ridge and lasso), and language-model fine-tuning (low learning rates, few epochs, LoRA's small rank are all regularising)
- Binding
- none
- Typical mechanics
- L2 adds the sum of squared weights, times a strength λ, to the loss, so gradient descent shrinks weights unless the data justifies them; L1 adds the sum of absolute values and produces sparse models with many exact zeros (a form of feature selection); dropout disables a random fraction of units on each training step so no unit can rely on another; early stopping halts training at the validation minimum; augmentation regularises by enlarging the data
- Typical strengths
- chosen by validation — a sweep of λ or dropout rate, watched on the validation metric; too little and the training–validation gap opens; too much and both fall
- Typical interpretation
- regularisation encodes a prior that simpler models are more likely to be right; L1 says few features matter, L2 says none should dominate, dropout says the representation should be redundant
- Parties
- data scientists, for whom it is a daily setting, and reviewers, for whom the chosen strength and its validation are part of the model record
How regularisation works in practice
A credit model with 400 features and gradient-boosted trees of depth 12 reaches a Gini of 0.70 on training and 0.56 on validation — a 14-point gap. Regularise: depth 5, minimum 200 applications per leaf, learning rate 0.03 with 800 trees, 70% feature sampling per tree, and an L2 term on leaf values. Training Gini falls to 0.64; validation rises to 0.62. The gap is now two points, the model is simpler, and its out-of-time performance holds. Each setting constrained what the trees could memorise; the data still determined what they learned. A sweep of the regularisation settings on validation found the combination; stronger settings started to lose validation performance, which is the underfitting edge.
In neural networks the same logic uses different tools. Weight decay keeps weights small so the function the network computes stays smooth; dropout at 0.1–0.5 forces redundancy; batch and layer normalisation stabilise training and regularise as a side effect; early stopping is the simplest and often the most effective. For language-model fine-tuning, the regularising choices are mostly implicit — a learning rate of 10⁻⁵, one to three epochs, a LoRA rank of 8 to 64 — and the failure of too little is a model that memorises the fine-tuning set and forgets its general ability. In all cases the method is the same: sweep the strength, watch validation, choose the peak.
For a fintech, regularisation is why a model with hundreds of features and thousands of trees can still generalise, and the record of its settings and their validation is what a reviewer reads to believe it.
Example
A payments company's chargeback-prediction network overfits: 0.93 AUC on training, 0.81 on validation, 0.78 on the next quarter. The team sweeps regularisation: weight decay from 10⁻⁶ to 10⁻², dropout from 0 to 0.5, early stopping on validation. The best combination (weight decay 10⁻⁴, dropout 0.3, stopped at epoch 11 of 40) gives 0.86 training, 0.84 validation, 0.83 out of time — a five-point gain in production from settings alone, with no new data or features. A stronger setting (dropout 0.5, decay 10⁻²) drops validation to 0.80: underfitting. The sweep's 36 runs, the chosen values and the out-of-time result go into the model card, and the monitoring rule watches the production AUC against the 0.83 baseline.
Common mistakes
- No regularisation on a flexible model, then attributing the validation gap to the data. - Strength chosen by habit rather than by a validation sweep. - Regularising until the model underfits and reading the flat result as robustness. - Ignoring early stopping, the cheapest regulariser there is. - Fine-tuning a language model for many epochs at a high rate and losing its general ability.