---
title: "Gradient Descent"
url: "https://mahouse.world/glossary/gradient-descent"
type: "glossary term"
topic: "Training"
author: "Ihor Vlasov"
reviewed: "2026-09-20"
published: "2026-08-10"
---

# Gradient Descent

Gradient descent is the optimisation method by which nearly every machine-learning model is trained: compute the gradient of the loss function — the direction in which error increases fastest — and move the parameters a small step the other way; repeat until the loss stops falling. Stochastic gradient descent estimates the gradient from a small batch rather than the whole dataset, which makes training on billions of examples possible. The step size, the learning rate, matters most.

**Also known as:** Stochastic gradient descent, SGD, Optimiser, Adam

## Key facts

- **Used at:** training neural networks, logistic regression, gradient-boosted trees (in a functional form), and fine-tuning; every model a fintech trains or rents was made with it
- **Binding:** none
- **Typical mechanics:** a batch of examples (32 to several thousand) goes through a forward pass; the loss is computed; backpropagation gives the gradient; the optimiser updates each parameter by the learning rate times the gradient (SGD), often with momentum, or with per-parameter scaling (Adam, AdamW); one pass over the data is an epoch; learning-rate schedules warm up, hold and decay
- **Typical settings:** learning rates from 10⁻⁵ (fine-tuning a language model) to 10⁻¹ (a small model from scratch); batch sizes from 32 to millions of tokens; Adam with weight decay is the default for deep learning
- **Typical failures:** too high a learning rate and the loss diverges; too low and training crawls or stalls in a poor region; a loss that plateaus, oscillates or spikes is diagnosed from the curve
- **Parties:** the researchers who developed the variants (momentum, Adam, 2014), the frameworks that implement them, and every practitioner who has watched a loss curve

## How gradient descent works in practice

Picture the loss as a landscape over the model's parameters — a surface with 110,000 dimensions for a fraud model, billions for a language model — where height is error. Training starts at a random point. The gradient says which way is uphill; the optimiser steps downhill; a thousand steps later the model is in a valley where predictions are good. The stochastic part is that computing the exact gradient over 40 million transactions per step would be too slow, so each step uses a batch of 1,024 and accepts a noisy estimate; the noise turns out to help, steering away from sharp minima that generalise poorly. Adam scales each parameter's step by the history of its gradients, so rarely-updated weights take larger steps and frequently-updated ones smaller.

The practitioner's contact with all this is the learning rate and the curve. A learning rate of 0.001 trains a model in an hour; 0.01 diverges in the first minute; 0.0001 takes a day to reach a worse result. Warm-up (small steps at first, when the random model's gradients are unreliable) and decay (smaller steps later, to settle) are standard. The training curve — loss against steps — is the diagnostic: falling smoothly is right; flat means the rate is too low or the model too small; spiking means too high or a data problem; falling on training data while rising on validation means overfitting, and the stopping point is the validation minimum.

For a fintech training its own models, gradient descent is the routine behind "training took four hours" and the reason a model-risk file contains a curve; for one renting models, it is the process that cost the vendor hundreds of millions.

## Example

A fintech fine-tunes a language model for document extraction. The first run uses a learning rate of 10⁻³, copied from a tutorial for small models: the loss spikes at step 40 and the model outputs gibberish. The second uses 10⁻⁵ with 100 warm-up steps and cosine decay over 2,000 steps, batch size 16: the loss falls from 2.1 to 0.4 by step 800 and flattens; validation accuracy peaks at step 1,100 and declines slightly after — the checkpoint at 1,100 is kept. A third run with 2×10⁻⁵ reaches the same validation accuracy in 700 steps. Total cost of the three runs: $31 in GPU time. The curves, settings and checkpoint hash go into the model card.

## Common mistakes

- Copying a learning rate from a different model size or task.
- No warm-up or decay on a large model; the first and last steps matter most.
- Reading the training loss alone; the validation curve is the one that decides.
- Stopping at a fixed epoch count rather than the validation minimum.
- No record of settings and curves, so the run cannot be reproduced or reviewed.

## Related terms

- [Backpropagation](https://mahouse.world/glossary/backpropagation.md)
- [Loss Function](https://mahouse.world/glossary/loss-function.md)
- [Hyperparameter Tuning](https://mahouse.world/glossary/hyperparameter-tuning.md)
- [Model Parameters](https://mahouse.world/glossary/model-parameters.md)
- [Overfitting](https://mahouse.world/glossary/overfitting.md)
- [Deep Learning](https://mahouse.world/glossary/deep-learning.md)

— Ihor Vlasov, M&A FinTech Expert. Last reviewed 2026-09-20. https://mahouse.world/authors/ihor-vlasov