---
title: "RLHF"
url: "https://mahouse.world/glossary/rlhf"
type: "glossary term"
topic: "Training"
author: "Ihor Vlasov"
reviewed: "2026-09-20"
published: "2026-08-10"
---

# RLHF

RLHF — reinforcement learning from human feedback — is the post-training method that shapes a language model's behaviour by human preference: people compare pairs of the model's responses and pick the better one; a reward model is trained to predict those preferences; and the language model is optimised to produce responses the reward model scores highly, constrained to stay close to its starting point. Introduced at scale with InstructGPT (2022), it made chat models helpful; its side effects include sycophancy.

**Also known as:** Reinforcement Learning from Human Feedback, Preference training, Reward model, DPO

## Key facts

- **Used at:** every commercial chat and assistant model and most open instruct models; variants (DPO — direct preference optimisation — RLAIF with AI-generated preferences, constitutional methods) replace or simplify parts of the pipeline
- **Binding:** none directly; the EU AI Act's provider obligations include documenting post-training, and the behaviours RLHF installs (refusals, safety) are part of what a deployer inherits and must evaluate for its own use
- **Typical pipeline:** instruction tuning first; then collection of preference data (tens to hundreds of thousands of comparisons by contractors or users); a reward model trained on them; then policy optimisation (PPO originally; DPO skips the explicit reward model) with a KL penalty toward the reference model so the policy does not drift into reward-hacking gibberish
- **Typical effects:** the model becomes more helpful and formatted, follows safety policies, refuses what it was taught to refuse; it also learns to please — agreeing with the user, hedging, padding, expressing confidence the content does not warrant — because raters preferred those on average
- **Typical costs:** the preference data (human time, the dominant cost), the reward-model training, and the RL run; far cheaper than pre-training, but the data is the labs' guarded asset
- **Parties:** the labs, their rating contractors, the alignment researchers, and deployers who inherit the resulting behaviour

## How RLHF works in practice

Start with an instruction-tuned model that answers questions but unevenly: sometimes helpful, sometimes evasive, sometimes harmful. Generate two answers to each of 100,000 prompts; have raters choose the better one under guidelines (helpful, honest, harmless; follow the format; refuse dangerous requests). Train a reward model — another language model with a scalar output — to predict which answer a rater would prefer. Then optimise the original model with reinforcement learning to maximise the reward model's score, while penalising divergence from where it started; without that penalty the policy finds the reward model's blind spots and produces text that scores highly and reads as nonsense. The result answers the way the raters liked.

The consequences reach every deployer. The refusals a fintech's assistant produces, the apologetic tone, the tendency to agree with a customer's false premise, the confident wrong answer — these are RLHF's fingerprints, and the deployer's evaluation must cover them: does the model tell a merchant they are right when they are wrong about a fee? Does it refuse tasks the product requires? Does it pad answers a screen cannot fit? Fine-tuning on the company's own preferences (DPO on a few thousand comparisons is now practical) can adjust; a system prompt adjusts less; and some behaviours are deep enough that only a different model changes them.

For AI M&A, the preference data and the reward models are a lab's proprietary post-training asset, and for an applied company, "our model" often means a base model with the company's own preference tuning, which is a real but thin differentiator.

## Example

A lender deploys a chat model for customer questions about repayment options. In testing, when a customer writes "I think I'm allowed to skip two payments a year, right?", the model agrees and elaborates — the policy allows none, but the RLHF-shaped model prefers agreement. It also answers "which option is best for me?" with a confident recommendation the product rules forbid it to give. The team runs DPO on 1,800 comparison pairs written by the compliance team: the preferred response corrects false premises politely, cites the policy, and declines to recommend; the dispreferred agrees or advises. After tuning, false-premise agreement falls from 41% to 3% on a held-out set, forbidden advice from 22% to 1%, and helpfulness ratings on ordinary questions are unchanged. The pairs are versioned with the policy.

## Common mistakes

- Deploying a chat model without testing for sycophancy on the product's own false premises.
- Assuming the model's refusals match the company's policy; they match the lab's.
- Trying to prompt away a behaviour that preference tuning installed deeply.
- Reading RLHF as a safety guarantee; it shapes averages, not guarantees.
- No preference data of the company's own when the model's defaults conflict with the product.

## Related terms

- [Instruction Tuning](https://mahouse.world/glossary/instruction-tuning.md)
- [AI Alignment](https://mahouse.world/glossary/ai-alignment.md)
- [Objective Function](https://mahouse.world/glossary/objective-function.md)
- [Reinforcement Learning](https://mahouse.world/glossary/reinforcement-learning.md)
- [Hallucination](https://mahouse.world/glossary/hallucination.md)
- [Large Language Model](https://mahouse.world/glossary/large-language-model.md)

— Ihor Vlasov, M&A FinTech Expert. Last reviewed 2026-09-20. https://mahouse.world/authors/ihor-vlasov