LLM Alignment & Preference Optimization
Preference optimization in practice — DPO, SimPO, ORPO, CPO, KTO, reward modeling, and RLHF. What each method does, when it helps, and how to choose between them.
DPO: Direct Preference Optimization for LLM Alignment
DPO aligns an LLM to human preferences with no reward model and no RL loop — the objective, the pair format, and when pairs beat labels.
SimPO: Simple Preference Optimization Without Reference Models
SimPO aligns an LLM with no reference model in memory — the length-normalized objective, the data it needs, and when it beats DPO.
ORPO: Odds Ratio Preference Optimization
ORPO folds SFT and preference alignment into a single stage — the odds-ratio objective, the dataset it needs, and when one stage wins.
CPO: Contrastive Preference Optimization for LLM Alignment
CPO pairs a contrastive objective with behavior cloning — chosen responses stay high-probability, rejected ones suppressed — and when that matters.
KTO: Kahneman-Tversky Optimization for AI Alignment
KTO aligns an LLM on binary thumbs-up/down feedback through prospect theory — losses weighted heavier than gains — no pairing required.
Reward Modeling for RLHF: Training Custom Reward Functions
Training a reward model for RLHF — the pipeline it anchors, the preference data it eats, and how it is evaluated before any policy trusts it.
RLHF Methods: Offline Alignment vs Online Reinforcement Learning
Compare offline alignment methods (DPO, SimPO, ORPO, CPO, KTO) with online RL (PPO, GRPO, GKD). Understand when to use each and what is coming to Vidman AI.
Red-Teaming Your LLM Before Users Do
Red-teaming is the review where you attack your own model before users do. The attack catalog, who should run it, and how findings become fixes.
RLVR: Learning With Verifiable Rewards
RLVR trains models on rewards that can be checked, not judged. Where verifiable rewards shine, the reward-hacking traps, and when preferences still win.
Browse other topics