SimPO: Simple Preference Optimization Without Reference Models

SimPO — Simple Preference Optimization — trains a language model to reach for human-chosen responses and shy from rejected ones, much as DPO does, but with no reference model held in memory during training. That removes the frozen copy of the base model from VRAM and roughly halves the memory bill against DPO.

It gets there by using the average log probability of the generated sequence as an implicit reward, instead of measuring KL divergence from a reference policy. Because the reward is normalised by length, the model cannot game the objective by drifting toward shorter answers.

DPO keeps two models resident — the training model and a frozen reference. For a 7B model that is roughly 28GB before optimizer states even enter the picture; SimPO carries a single copy and cuts the figure to 14GB.

The simplified objective gives up nothing measurable: SimPO matches or beats DPO on the standard alignment benchmarks, MT-Bench and AlpacaEval among them. Its length normalization also keeps response lengths steadier, avoiding the verbosity DPO-tuned models sometimes slide into.

The data looks exactly as it does for DPO: each example carries a prompt, a chosen response, and a rejected response. Pair quality carries the same weight — chosen should be clearly better, not marginally different.

A workable alignment dataset runs 5,000 to 50,000 preference pairs. Below that range overfitting looms; far above it, quality stops scaling proportionally. Vidman AI validates the format during upload and previews the pairs before training begins.

Reach for SimPO when VRAM is the constraint and the goal is aligning larger models on smaller GPUs. It shines where the preference data is clean and the quality gap between chosen and rejected is unambiguous.

It is also a sound default when no alignment method has an obvious claim: fewer hyperparameters to tune, and reference-free training runs faster per step. On Vidman AI, SimPO runs typically finish 15-25% faster than their DPO equivalents.

Pick SimPO as the training method in the configurator. The hyperparameters that matter: beta (preference strength, default 2.0), gamma (length-bonus coefficient, default 0.5), and the usual training settings — learning rate, batch size.

The wizard surfaces the SimPO-specific options the moment the method is selected:

Vidman AI computes the reward internally, applying the length-normalized log-probability objective. Reward margins, accuracy over the preference pairs, and the training loss all stream through the live metrics dashboard.

The training detail view carries the SimPO metrics in real time, so alignment progress is visible as it happens:

When memory is abundant. SimPO's headline win is dropping the reference model, which matters most when VRAM is tight; with headroom to spare, DPO's reference anchor is a stabiliser you can simply afford.

When your outputs are length-sensitive. SimPO normalises reward by response length, which shifts how the model treats verbosity; where the workload punishes any drift in length — strict templates, fixed-field extraction — validate that behaviour on your own eval before committing.

And when the preference data is thin. Every offline method is bounded by its dataset, and a newer method with less community debugging behind it is a harder place to be stuck. With few and precious pairs, open with the method whose failure modes are best documented, then compare.

Much of DPO's discipline carries over, with a few SimPO-specific twists. Without a reference model, beta plays a different role than in DPO: it sets the strength of preference optimisation directly, rather than regulating divergence from a reference. Open at the default 2.0 and let results arbitrate — alignment too weak means raising beta; a model turning constrained or repetitive means lowering it.

Gamma, the length bonus, belongs to SimPO alone. Positive gamma rewards longer responses; lower gamma encourages conciseness. Where answers should run deep — detailed explanations, thorough analyses — gamma of 0.5 to 1.0 serves well. Where brevity is the virtue, drop it to 0.1 or 0.0.

Because rewards are length-normalised, the dataset ideally keeps chosen and rejected at similar lengths. If chosen is systematically much longer or shorter than rejected, the normalisation can tangle with the preference signal — review pairs to be sure the quality difference is not confounded with a length difference.

And since SimPO skips the reference-model log-probability pass, each step is cheaper than DPO's. Spend the saving on more experiments across hyperparameter settings.

The inputs match DPO's exactly: a preference dataset in JSONL with prompt, chosen, and rejected. Upload through the Datasets page, verify with the preview, then create a new training run, select SimPO, and point it at the SFT-trained base model.

Open with the defaults — beta 2.0, gamma 0.5, learning rate 5e-7 — and train 1 to 2 epochs. Watch reward margins and preference accuracy in the dashboard; SimPO usually converges faster than DPO, and a single epoch is often enough. Evaluate on the test set against the SFT baseline to measure the gain.

Coming from a DPO run and curious? Every training run stays in the training list, so both methods' metrics sit side by side. A comparison experiment costs almost nothing — you pay only for the seconds of compute it burns.