---
title: "Backpropagation"
url: "https://mahouse.world/glossary/backpropagation"
type: "glossary term"
topic: "Training"
author: "Ihor Vlasov"
reviewed: "2026-09-20"
published: "2026-08-10"
---

# Backpropagation

Backpropagation is the algorithm that trains a neural network: after a forward pass produces a prediction and a loss function measures its error, backpropagation applies the chain rule backward through the layers to compute how much each parameter contributed to the error — the gradient — and gradient descent adjusts every parameter a small step to reduce it. Popularised in 1986 and made practical by GPUs, it is the engine of all deep learning.

**Also known as:** Backprop, Reverse-mode differentiation, Gradient backpropagation

## Key facts

- **Used at:** the training of every neural network, from a fraud model to a frontier language model; also fine-tuning, where the same algorithm adjusts a pre-trained model's weights on new data
- **Binding:** none
- **Typical mechanics:** forward pass computes the prediction and loss; the backward pass computes the loss's derivative with respect to the output, then, layer by layer in reverse, with respect to each layer's parameters and inputs, reusing each layer's stored activations; the optimiser (SGD, Adam) updates the parameters; repeat over batches of data for many epochs. Automatic-differentiation frameworks (PyTorch, JAX) do the calculus
- **Typical cost:** about two-thirds of training compute is the backward pass; training a model is on the order of six operations per parameter per training token, versus two for inference; memory must hold activations from the forward pass for the backward one, which limits batch size
- **Typical difficulties:** vanishing and exploding gradients in deep networks (addressed by normalisation, skip connections, careful initialisation), instability at scale (learning-rate schedules, gradient clipping), and cost
- **Parties:** the researchers (Rumelhart, Hinton and Williams's 1986 paper; the earlier and independent derivations), the framework developers, and everyone who trains or fine-tunes a model

## How backpropagation works in practice

A fraud model predicts 0.2 probability of fraud for a transaction that was fraud; the loss (cross-entropy) is large. Backpropagation asks, for each of the model's 110,000 parameters, "if this weight were slightly larger, would the loss go down or up, and how much?" — and answers all 110,000 questions in one backward pass, because the chain rule lets each layer's gradient be computed from the next layer's, using the activations saved during the forward pass. The optimiser then nudges each weight against its gradient. Over millions of transactions and dozens of passes through the data, the weights settle where the loss is low: the model has learned.

At the scale of a language model the same algorithm runs across thousands of GPUs for months, and the engineering is about keeping it stable and fed: gradients are computed on each GPU's slice of data and averaged; activations are recomputed rather than stored to save memory; learning rates warm up and decay; a spike in the loss curve means an intervention. Fine-tuning is backpropagation on a pre-trained model with a small learning rate and the company's own data, and parameter-efficient methods (LoRA) backpropagate into a few million added weights rather than the billions of original ones, which makes it affordable on a single GPU.

For a fintech that trains its own models, backpropagation is a cost line (the GPU hours) and a discipline (the training runs, curves and checkpoints that model-risk reviews); for one that only rents models, it is why fine-tuning is cheap and pre-training is not.

## Example

A payments company fine-tunes an open-weight 8-billion-parameter language model to classify merchant-support tickets into 40 categories with company-specific meanings. Full fine-tuning would backpropagate into all 8 billion parameters and need eight 80 GB GPUs for the activations and optimiser state. With LoRA, the team adds 12 million trainable parameters, backpropagates into those alone, and trains on one GPU in three hours over 6,000 labelled tickets at a cloud cost of $14. Accuracy on a held-out set: 93%, against 86% for the base model with a few-shot prompt. The training curve, the held-out score and the checkpoint go into the model-risk file, and the run is reproducible from the seed and the data snapshot.

## Common mistakes

- Full fine-tuning when a parameter-efficient method gives the same result at a hundredth of the cost.
- Training without a held-out set; the loss going down on training data proves nothing.
- Ignoring instability; a loss spike is a signal, not noise.
- Assuming training cost scales like inference; it is about three times more per token, over far more tokens.
- No record of the run — seed, data, curve, checkpoint — when model risk asks how the model was made.

## Related terms

- [Gradient Descent](https://mahouse.world/glossary/gradient-descent.md)
- [Forward Propagation](https://mahouse.world/glossary/forward-propagation.md)
- [Loss Function](https://mahouse.world/glossary/loss-function.md)
- [Deep Learning](https://mahouse.world/glossary/deep-learning.md)
- [Model Parameters](https://mahouse.world/glossary/model-parameters.md)

— Ihor Vlasov, M&A FinTech Expert. Last reviewed 2026-09-20. https://mahouse.world/authors/ihor-vlasov