Gradient Descent
Gradient descent is the optimisation method by which nearly every machine-learning model is trained: compute the gradient of the loss function — the direction in which error increases fastest — and move the parameters a small step the other way; repeat until the loss stops falling. Stochastic gradient descent estimates the gradient from a small batch rather than the whole dataset, which makes training on billions of examples possible. The step size, the learning rate, matters most.
- Also known as
- Stochastic gradient descent, SGD, Optimiser, Adam
- Used at
- training neural networks, logistic regression, gradient-boosted trees (in a functional form), and fine-tuning; every model a fintech trains or rents was made with it
- Binding
- none
- Typical mechanics
- a batch of examples (32 to several thousand) goes through a forward pass; the loss is computed; backpropagation gives the gradient; the optimiser updates each parameter by the learning rate times the gradient (SGD), often with momentum, or with per-parameter scaling (Adam, AdamW); one pass over the data is an epoch; learning-rate schedules warm up, hold and decay
- Typical settings
- learning rates from 10⁻⁵ (fine-tuning a language model) to 10⁻¹ (a small model from scratch); batch sizes from 32 to millions of tokens; Adam with weight decay is the default for deep learning
- Typical failures
- too high a learning rate and the loss diverges; too low and training crawls or stalls in a poor region; a loss that plateaus, oscillates or spikes is diagnosed from the curve
- Parties
- the researchers who developed the variants (momentum, Adam, 2014), the frameworks that implement them, and every practitioner who has watched a loss curve
How gradient descent works in practice
Picture the loss as a landscape over the model's parameters — a surface with 110,000 dimensions for a fraud model, billions for a language model — where height is error. Training starts at a random point. The gradient says which way is uphill; the optimiser steps downhill; a thousand steps later the model is in a valley where predictions are good. The stochastic part is that computing the exact gradient over 40 million transactions per step would be too slow, so each step uses a batch of 1,024 and accepts a noisy estimate; the noise turns out to help, steering away from sharp minima that generalise poorly. Adam scales each parameter's step by the history of its gradients, so rarely-updated weights take larger steps and frequently-updated ones smaller.
The practitioner's contact with all this is the learning rate and the curve. A learning rate of 0.001 trains a model in an hour; 0.01 diverges in the first minute; 0.0001 takes a day to reach a worse result. Warm-up (small steps at first, when the random model's gradients are unreliable) and decay (smaller steps later, to settle) are standard. The training curve — loss against steps — is the diagnostic: falling smoothly is right; flat means the rate is too low or the model too small; spiking means too high or a data problem; falling on training data while rising on validation means overfitting, and the stopping point is the validation minimum.
For a fintech training its own models, gradient descent is the routine behind "training took four hours" and the reason a model-risk file contains a curve; for one renting models, it is the process that cost the vendor hundreds of millions.
Example
A fintech fine-tunes a language model for document extraction. The first run uses a learning rate of 10⁻³, copied from a tutorial for small models: the loss spikes at step 40 and the model outputs gibberish. The second uses 10⁻⁵ with 100 warm-up steps and cosine decay over 2,000 steps, batch size 16: the loss falls from 2.1 to 0.4 by step 800 and flattens; validation accuracy peaks at step 1,100 and declines slightly after — the checkpoint at 1,100 is kept. A third run with 2×10⁻⁵ reaches the same validation accuracy in 700 steps. Total cost of the three runs: $31 in GPU time. The curves, settings and checkpoint hash go into the model card.
Common mistakes
- Copying a learning rate from a different model size or task. - No warm-up or decay on a large model; the first and last steps matter most. - Reading the training loss alone; the validation curve is the one that decides. - Stopping at a fixed epoch count rather than the validation minimum. - No record of settings and curves, so the run cannot be reproduced or reviewed.