---
title: "Hyperparameter Tuning"
url: "https://mahouse.world/glossary/hyperparameter-tuning"
type: "glossary term"
topic: "Training"
author: "Ihor Vlasov"
reviewed: "2026-09-20"
published: "2026-08-10"
---

# Hyperparameter Tuning

Hyperparameter tuning is the search for the settings that govern how a model is trained or shaped but are not learned from data — learning rate, batch size, layers, regularisation strength, tree depth — by training candidates with different settings and comparing them on validation data. Grid, random and Bayesian search are the methods; the validation set is the judge; and the common failure is tuning against the data then used to report the result, which flatters the model.

**Also known as:** Hyperparameter optimisation, Hyperparameter search, HPO, Tuning

## Key facts

- **Used at:** every trained model beyond a default configuration — fraud and credit models (tree depth, learning rate, feature sampling), neural networks (architecture, optimiser settings, dropout), and fine-tuning (rate, epochs, LoRA rank)
- **Binding:** none; but model-risk reviews ask how settings were chosen and whether the test set stayed untouched
- **Typical methods:** grid search (every combination of a few values each; exhaustive and expensive), random search (random combinations; usually better per trial), Bayesian optimisation (a model of the search space proposes the next trial; best for expensive training), and successive halving or Hyperband (start many, stop the poor ones early)
- **Typical scale:** tens to hundreds of trials for a gradient-boosted model, each minutes; a handful for a large neural network, each hours or days; the labs run small-scale sweeps and extrapolate with scaling laws
- **Typical protocol:** split data into train, validation and test; tune on validation; touch the test set once, at the end; with small datasets, cross-validate the tuning; record every trial
- **Parties:** the data scientists who tune, the platform teams whose tooling runs trials (Optuna, Ray Tune, cloud services), and the reviewers who check the test set was not tuned on

## How hyperparameter tuning works in practice

A gradient-boosted fraud model has a dozen settings that matter: learning rate, number of trees, depth, minimum leaf size, feature and row sampling, regularisation terms. Defaults give an AUC of 0.905 on validation. A random search of 120 combinations over a night finds a configuration at 0.921 — shallower trees than the default, a lower rate, more trees, stronger regularisation. Bayesian optimisation gets to 0.923 in 60 trials. The gain of two points is real money in a fraud model, and it came from settings, not data or features. The team then evaluates the chosen configuration once on the test set: 0.919 — slightly lower, as expected, because the validation number carries the optimism of having been selected.

The discipline is in what the test set is for. If the team had tuned on the test set — run the 120 trials and reported the best test score — the 0.923 would be partly luck, the best of 120 draws, and the production model would disappoint. The three-way split, or cross-validated tuning when data is scarce, is how the reported number stays honest. The other discipline is scope: tuning a model to the third decimal place is worth less than a better feature or more data, and a tuning run that finds a configuration much better than the rest may have found a leak. For fine-tuning language models, the space is smaller (learning rate, epochs, LoRA rank, a few others) and a handful of trials suffice.

For a fintech, hyperparameter tuning is the routine that turns a default model into a good one and a governance point: the trials are recorded, the test set is sacred, and the chosen settings are in the model card.

## Example

A credit-scoring team tunes a model on 120,000 applications split 70/15/15. The first pass tunes on the test set by mistake — the pipeline's split names were swapped — and reports a Gini of 0.64. Model risk asks for the validation curve and notices the test set appears in the tuning log. The team reruns correctly: tuning on validation lands at 0.63, and the test set, touched once, gives 0.61. The 0.03 gap between the first report and the honest one is the optimism of selection; on a portfolio of €200 million, it is the difference between the expected-loss forecast holding and missing. The corrected model card records 140 trials, the chosen settings, the validation curve, and the single test evaluation.

## Common mistakes

- Tuning on the test set and reporting the best trial as the expected performance.
- Grid search over many settings when random or Bayesian search finds better in fewer trials.
- Tuning to the third decimal while features and data have first-decimal problems.
- Not recording trials, so the chosen configuration cannot be justified or reproduced.
- Reusing settings from a different dataset size or class balance without re-tuning.

## Related terms

- [Gradient Descent](https://mahouse.world/glossary/gradient-descent.md)
- [Validation Data](https://mahouse.world/glossary/validation-data.md)
- [Overfitting](https://mahouse.world/glossary/overfitting.md)
- [Regularisation](https://mahouse.world/glossary/regularisation.md)
- [Model Parameters](https://mahouse.world/glossary/model-parameters.md)
- [MLOps](https://mahouse.world/glossary/mlops.md)

— Ihor Vlasov, M&A FinTech Expert. Last reviewed 2026-09-20. https://mahouse.world/authors/ihor-vlasov