RLHF Methods: Offline Alignment vs Online Reinforcement Learning

Reinforcement Learning from Human Feedback (RLHF) is the umbrella term for techniques that align language model behaviour with human preferences. The original pipeline ran three steps: supervised fine-tuning (SFT), reward model training, and reinforcement learning (typically PPO). Since then the field has split into offline methods that skip the RL step and online methods that keep it.

Both share one goal: outputs humans prefer. They differ in how the preference signal is encoded and optimised — with real implications for training complexity, compute cost, and alignment quality.

Offline methods optimise directly on a fixed dataset of preference pairs, generating nothing during training. DPO (Direct Preference Optimization) reformulates the RL objective as binary classification over chosen versus rejected pairs. SimPO removes the reference-model requirement. ORPO merges SFT and alignment into one stage. CPO adds a behaviour-cloning regulariser. KTO works with non-paired binary feedback.

All six offline methods are available on the platform today. Their main advantage is simplicity: no reward model needed (except for reward modeling itself), no generation during training, straightforward hyperparameter tuning — stable and predictable.

The wizard offers any of the six, configuring method-specific parameters automatically:

Online RL methods generate fresh responses during training, score them with a reward model, and use the reward signal to update the policy. PPO (Proximal Policy Optimization) is the classic, using a clipped surrogate objective for stable policy updates. GRPO (Group Relative Policy Optimization) computes advantages relative to a group of sampled responses instead of a value-function baseline. GKD (Generalized Knowledge Distillation) combines reward optimisation with distillation from a teacher model.

These methods are coming soon to the platform. They require a trained reward model — already buildable today — and are more compute-intensive than offline methods, but they can discover and reinforce high-quality behaviours absent from the static preference dataset.

The two families part ways on three axes:

• Compute cost: offline methods process a fixed dataset and finish in predictable time; online methods generate and score responses during training, multiplying compute by 3-5x.
• Stability: offline methods have well-understood training dynamics; online RL can be unstable, demanding careful reward-model calibration and PPO hyperparameter tuning.
• Alignment quality: online methods explore the response space and find high-quality outputs the preference dataset does not contain; offline methods are bounded by the quality distribution of the training data.

For most production use cases, offline methods deliver sufficient alignment quality. Online RL matters most when alignment quality must be pushed beyond what the preference data captures.

Start with DPO on paired preference data — the most studied offline method, with reliable results. SimPO when DPO’s VRAM requirements bite. ORPO to skip the SFT stage. KTO with binary-only feedback. Train a reward model when online RL is on the roadmap.

For most teams, the offline methods available today deliver excellent alignment quality; online RL will add a tool for teams pushing the frontier of model quality, willing to invest the extra compute and tuning.

Results from different alignment methods compare side by side in the training detail view:

When instruction tuning has not happened yet. Every method in this family aligns behaviour that already exists; applied to a raw base model, alignment polishes noise. SFT first, evaluate, then align.

When the problem is knowledge, not preference. RLHF changes which answers the model prefers among ones it can already produce; it cannot teach facts. “Does not know” failures belong to retrieval or continued pre-training — preference training only makes the wrong answers more confident.

And when the preference data does not exist yet. Teams regularly schedule alignment runs before a single pair is collected. The dataset is the method — build a small, clean set by hand first, run DPO on it, and let the result say whether the frontier methods are worth the pipeline.

Whatever the method, a few practices apply universally. First, always start from an SFT-trained model: alignment refines a capable model, it does not teach basic instruction following. A base model that cannot produce coherent domain responses will not be fixed by alignment — SFT first, validate, then align.

Second, invest in preference data quality. Nothing shapes alignment results more than annotation quality, whichever method is picked. Clear, consistent, informative labels produce good alignment; noisy, contradictory, or trivially obvious labels produce weak alignment. For human annotation: clear guidelines, trained annotators, measured inter-annotator agreement.

Third, evaluate on the dimensions that matter, not just general benchmarks. Aligning for safety means safety-relevant test cases; for tone and style, style-sensitive prompts. General benchmarks like MT-Bench measure overall quality but may miss the specific improvements being targeted.

Fourth, do not over-align. Too many epochs on preference data can make the model overly cautious, repetitive, or narrow. Watch reward accuracy during training: past 90 to 95 percent, further training is likely overfitting to preference patterns rather than improving alignment. Stop at the checkpoint with the best validation metrics, not at the end of the scheduled epochs.

Alignment coverage here is the deepest of any comparable platform: all six offline methods (DPO, SimPO, ORPO, CPO, KTO, and reward modeling) in one interface. The path: an SFT-trained base, a preference dataset in the right shape, one method selected, and the run launched with the defaults.

The wizard walks the method choice and its parameters; the training list tracks every run side by side — loss curves, reward accuracy where relevant, checkpoints — so comparing methods on one dataset is a workflow rather than a project.

When online RL lands, the reward models trained today plug straight in: PPO, GRPO, and GKD will consume the same scorers. Building the reward model now is the cheapest preparation for the frontier later.