RLHF (Reinforcement Learning from Human Feedback)

Definition. An alignment technique that trains a reward model on human preference comparisons between candidate outputs, then optimises the language model against that reward using reinforcement learning, classically PPO. It is the step that turns a raw next-token predictor into an assistant that follows instructions, refuses harmful requests and prefers helpful answers. Modern variants replace PPO with simpler direct preference optimisation.

How it works. As an application builder you almost never run RLHF yourself; you consume its result. What matters operationally is that you can replicate the effect cheaply with preference data: collect pairwise judgements from your own users, train a small reward model or use DPO on a small open model, and evaluate against a held-out preference set. For most business use cases, prompt engineering plus a few hundred curated examples gets most of the benefit at a fraction of the cost.

Example. A writing assistant collects 8,000 pairwise comparisons from editors choosing between two drafts. A DPO run on a 7-billion-parameter open model using those pairs raises the win rate against the base model from 50 to 68 percent on a held-out set of 500 new prompts, at a training cost under $400. The team ships it as a specialised drafting model while keeping a larger general model for complex requests.

Use case. Preference tuning is appropriate when you can define quality only by comparison, such as tone, helpfulness, concision or stylistic fit, and when you have a pool of expert raters. It is the right lever for products where output quality is judged subjectively and consistently by humans.

Warnings. Reward models are proxies and will be gamed. Optimising too hard against a learned reward produces reward hacking: outputs that score well but are worse to humans, often longer, more confident and more sycophantic. Always keep a human evaluation gate.

Limitations. RLHF is expensive, slow, and inherits the biases and preferences of its raters. It can reduce output diversity and increase sycophancy. It also provides no guarantee of factual accuracy, because raters judge plausibility, not truth.

Pros and cons
ProsCons
Trains toward preferences that are hard to express as rules.Requires expensive ratings and can inherit rater biases.
← Browse all glossary terms