Loss Function
A loss function is the measure of how wrong a model's predictions are on its training examples — mean squared error for a number, cross-entropy for a probability, and many task-specific variants — and the quantity training exists to minimise: gradient descent moves the parameters to reduce it, and the resulting model is the one that scores lowest on it. The loss decides what "better" means to training; one that ignores the business cost of errors trains the wrong model.
- Also known as
- Loss, Cost function, Training loss, Cross-entropy loss
- Used at
- every trained model; the training curve a data scientist watches is the loss against steps, and the validation loss is the first signal of overfitting
- Binding
- none; but a loss that treats a false decline and a missed fraud as equal errors produces a model whose costs the business did not choose
- Typical choices
- mean squared error for regression; cross-entropy (log loss) for classification and for language models (predicting the next token); hinge loss for margin classifiers; contrastive losses for embeddings; weighted or focal losses for imbalanced classes; custom losses that encode asymmetric costs
- Typical relations
- the loss is the training-time proxy for the metric the business cares about (AUC, precision at a threshold, revenue), which is usually not differentiable and cannot be minimised directly; the gap between the two is a design decision
- Typical diagnostics
- training loss falling and validation loss rising means overfitting; both flat means the model or rate is wrong; a loss that drops suddenly may mean a leak
- Parties
- the data scientists who choose and sometimes design it, and the business owners whose cost of errors it should encode
How a loss function works in practice
A fraud model outputs a probability; the true label is fraud or not; cross-entropy penalises the model heavily for a confident wrong answer (0.01 for a fraud that happened) and lightly for a confident right one. Summed over a batch, that is the number backpropagation differentiates and gradient descent reduces. After training, the model's cross-entropy is low — but the business does not care about cross-entropy; it cares about fraud caught per false decline at the threshold it operates. The loss is a proxy chosen because it is smooth and correlates with the metric; when the correlation is weak, the model is good at the proxy and mediocre at the job.
The lever is to move the loss toward the cost. Weighting the fraud class ten times makes a missed fraud ten times as painful in training as a false alarm, and the model shifts accordingly. A custom loss can encode the actual cost — a missed €500 fraud versus a declined €50 legitimate purchase — so the model's notion of error is the finance team's. For language models the loss is next-token cross-entropy over trillions of tokens, and the fact that this proxy produces models that answer questions is one of the field's remarkable findings; the post-training stages then optimise other objectives (preferences, helpfulness) that the pre-training loss does not capture.
For a fintech, the loss function is where the business's cost of errors enters the model, or fails to; a model-risk review that asks "what was minimised, and is that what we care about?" is asking the right question.
Example
A payments company's decline model is trained with unweighted cross-entropy on 0.3% fraud. It reaches a low validation loss and, at the operating threshold, catches 68% of fraud with a 1.2% false-decline rate. The finance team's cost model says a false decline costs €14 in lost margin and a missed fraud €95 on average. The data team retrains with a class weight reflecting that ratio; validation cross-entropy rises (the model is now "worse" on the old measure), but at the operating threshold it catches 76% of fraud at 1.4% false declines, and the total cost of errors falls by €410,000 a year. The model card now states the weighted loss and its business derivation, and the review adds a rule that the weights are revisited when the cost model changes.
Common mistakes
- Minimising a loss that ignores the asymmetry between the two kinds of error. - Reporting the loss as if it were the business metric. - Reading the training loss alone; the validation loss is the one that warns. - Changing the loss without re-tuning the threshold and re-checking calibration. - A loss that drops suddenly and is celebrated; it is usually a leak.