DPO
Direct Preference Optimization — a simpler alternative to RLHF that trains models directly on preference data without needing a separate reward model or reinforcement learning loop.
DPO is the 2023–2024 simplification of RLHF. Where RLHF requires training a reward model and then running PPO (a complex RL algorithm), DPO reformulates preference learning as a straightforward classification problem. Same input — pairs of "preferred" and "rejected" responses — but the training is dramatically simpler.
For practical impact: DPO requires roughly 10× less compute than RLHF for similar quality. It is also more stable — RLHF training notoriously diverges or collapses; DPO trains predictably. Most open-weight model labs in 2026 use DPO or close variants instead of full RLHF.
Newer variants like KTO, ORPO, and SimPO build on DPO with further simplifications. The field is evolving fast; DPO is the current baseline alternative to RLHF.
Frequently asked
Is DPO better than RLHF?+
Simpler, cheaper, more stable. Quality is comparable for most tasks. RLHF still has some niche advantages on the hardest reward modeling problems but DPO is the default for new alignment work.
Can I do DPO on my own model?+
Yes, more practically than RLHF. With LoRA + DPO you can run preference alignment on a single GPU. The hard part remains collecting preference data — usually 5K–50K labeled pairs.