---
title: "Validation Data"
url: "https://mahouse.world/glossary/validation-data"
type: "glossary term"
topic: "Training"
author: "Ihor Vlasov"
reviewed: "2026-09-20"
published: "2026-08-10"
---

# Validation Data

Validation data is the slice of data held out from training and used during development to make decisions — which hyperparameters, which features, when to stop training, which candidate model — so that those decisions rest on performance on examples the model did not learn from; it is kept separate from the test set, which is touched once at the end to report the result, because data used to choose a model no longer measures it honestly.

**Also known as:** Validation set, Hold-out set, Development set, Train–validation–test split

## Key facts

- **Used at:** every model's development; the validation curve is where overfitting is first seen and where early stopping fires, and the validation score is what tuning optimises
- **Binding:** none; but a model-risk reviewer will ask how the sets were split and whether the test set stayed untouched, and an answer of "we tuned on test" invalidates the reported number
- **Typical splits:** 70/15/15 or 80/10/10 by row for large datasets; k-fold cross-validation (train on k−1 folds, validate on the k-th, rotate) when data is scarce; out-of-time splits (train on earlier periods, validate and test on later) for anything where the world changes, which in finance is everything; group splits (all of a customer's rows in one set) to prevent leakage across rows
- **Typical errors:** random splits that put a customer's transactions on both sides (the model learns the customer, not the pattern); validation from the same period as training when production will be a later one; repeated tuning on the test set; validation sets too small to distinguish candidates; label leakage across the split through features computed on the whole dataset
- **Typical reading:** training score high, validation low — overfitting; both low — underfitting; validation good, test much worse — the validation set was tuned on too hard or does not resemble the test; test good, production worse — drift, or the test set did not resemble production
- **Parties:** data scientists, who own the split, and reviewers, who audit it

## How validation data works in practice

A credit model is developed on applications from 2023–2024. The split: training on January 2023–June 2024, validation on July–September 2024, test on October–December 2024, and every applicant's rows in one set only. The team tunes 140 configurations on validation and picks the best; early stopping uses the validation loss; the feature set is chosen by validation gain. The test set is then scored once: a Gini of 0.61 against validation's 0.63 — the two-point difference is the optimism of selection, and 0.61 is the number reported. Had the team split randomly, the validation and test scores would have been higher and less true, because the model would have learned the 2023 economy and been tested on it, when production is 2025.

Cross-validation handles small data: with 6,000 examples, a single 15% validation set is 900 rows, too few to rank candidates reliably; five-fold cross-validation uses all 6,000 for validation across folds and averages. The test set is still held apart. The mechanics are simple; the discipline is what fails. A team that looks at the test score, adjusts, and looks again has turned the test set into a validation set, and the final number is optimistic by an unknown amount. The rule that survives contact with deadlines is procedural: the test set is locked, its score is computed by a script that runs once, and the model card records the date.

For a fintech, validation data is the difference between a model that was measured and one that was flattered, and the split's design — by time, by customer — is the first thing a competent reviewer checks.

## Example

A payments company's chargeback model reports 0.89 AUC. The model-risk reviewer asks for the split: random, 80/20, from the same 12 months, and the "test" set was used across 30 tuning iterations. The reviewer requests a re-evaluation with an out-of-time test set from the two months after the development window, locked and scored once: 0.82. The seven-point gap decomposes into three points of selection optimism and four of drift the random split had hidden. The team rebuilds with an out-of-time validation period, tunes on it, and scores the locked test once: 0.84 validation, 0.83 test. The model goes live at 0.83, the monitoring baseline is 0.83, and the production AUC in the first quarter is 0.82 — as forecast, which is the point.

## Common mistakes

- Tuning on the test set and reporting the best score.
- Random splits when production will be a later period.
- A customer's rows on both sides of the split.
- Features computed on the whole dataset before splitting, leaking the future into training.
- A validation set too small to distinguish candidates, then trusting the ranking.

## Related terms

- [Training Data](https://mahouse.world/glossary/training-data.md)
- [Overfitting](https://mahouse.world/glossary/overfitting.md)
- [Hyperparameter Tuning](https://mahouse.world/glossary/hyperparameter-tuning.md)
- [Supervised Learning](https://mahouse.world/glossary/supervised-learning.md)
- [Underfitting](https://mahouse.world/glossary/underfitting.md)
- [MLOps](https://mahouse.world/glossary/mlops.md)

— Ihor Vlasov, M&A FinTech Expert. Last reviewed 2026-09-20. https://mahouse.world/authors/ihor-vlasov