ORPO: Odds Ratio Preference Optimization

ORPO — Odds Ratio Preference Optimization — is a single-stage alignment method that folds supervised fine-tuning and preference optimization into one training objective. Rather than SFT first and DPO after, ORPO runs both at once by bolting an odds-ratio penalty onto the standard language modeling loss.

The odds-ratio component fines the model whenever it assigns higher probability to a rejected response than to its chosen counterpart. It is computed from the ratio of output probabilities and weighted by a lambda hyperparameter that sets how hard the preference signal pulls.

The conventional alignment pipeline demands two training stages — SFT to teach instruction-following, then DPO or a sibling to teach preferences — each with its own hyperparameters, monitoring, and compute budget.

ORPO collapses them, roughly halving total training time. It also sidesteps the alignment tax, where the DPO stage erodes capabilities the SFT stage had installed. Learning both objectives jointly, the model weighs instruction-following and preference alignment together across the whole run.

ORPO needs data carrying both signals at once: instruction-following and preference pairs. Each example holds a prompt, a chosen response, and a rejected response — and the chosen response doubles as the SFT target, so it must be a genuinely high-quality instruction-following answer.

That raises the curation bar above a standard DPO dataset: chosen responses have to work as SFT training examples too. Vidman AI checks that every required field is present and previews the prompt, chosen, and rejected columns before training begins.

ORPO earns its keep when the starting point is a base model — no SFT checkpoint yet — and the goal is an aligned model in one run. It suits tight compute budgets equally well, where the two-stage SFT-then-DPO pipeline is simply unaffordable.

It performs best on moderate datasets (10K-50K examples) with clearly differentiated preference pairs. For very large datasets, or where the SFT and alignment stages need separate, precise control, the traditional two-stage route remains the better instrument.

Pick ORPO as the method in the configurator. The headline hyperparameter is lambda — the weight of the preference objective against the SFT loss. The default 0.1 serves most tasks; higher values lean harder into preference alignment.

The wizard surfaces the ORPO-specific parameters, lambda included:

Watch both the SFT loss and the odds-ratio loss in the dashboard. A healthy run shows both descending, with the SFT loss usually settling first.

When a validated SFT checkpoint must stay untouched. The single stage is also the constraint: instruction-tuning and preference alignment merge into one run. Where a validated SFT model exists and touching it triggers a re-validation cycle, a two-stage method like DPO layers alignment on top without disturbing what is already approved.

When the stages need separate iteration. Merged stages mean merged failure modes: a quality problem in an ORPO run might live in the SFT data or the preference data, and untangling them after the fact costs more than training them apart in the first place.

And when the preference set is far smaller than the SFT set. The joint objective balances the two signals, and heavily mismatched dataset sizes tip the run toward whichever dominates — an argument for staging, not for a bigger optimizer.

Because ORPO runs SFT and alignment together, the dataset pulls double duty. Chosen responses are simultaneously the SFT targets and the positive preference examples — they must be high-quality instruction-following answers that also model the preferred style. Rejected responses should be clearly inferior yet still coherent; bottom-scraping rejections carry a weak signal.

Lambda is the lever that matters most. 0.1 gives a moderate preference signal that suits most tasks. Instructions followed well but preferences landing soft? Raise lambda toward 0.2 or 0.3. The model struggling with basic instructions? The preference signal is likely too strong — drop lambda to 0.05.

ORPO also wants somewhat more data than a standalone SFT stage, since instruction-following and preference alignment must both be learned from the same examples. Aim for at least 10,000 high-quality examples; below that, the preference contrast may be too thin to teach meaningful alignment even where the SFT signal suffices.

Track both loss components in the dashboard. The SFT loss should fall steadily throughout; the odds-ratio loss may swing early but should trend clearly down. An odds-ratio loss that plateaus while SFT keeps improving means the model is learning instructions but not preferences — raise lambda, or take a hard look at the pairs.

ORPO appeals most to teams that want aligned models with minimal pipeline machinery: one job, run straight from a base model, produces the aligned result. Total training time shrinks, experiment tracking simplifies, and the alignment tax — preference optimization eroding SFT capabilities — never gets the chance.

Prepare a dataset in which every example carries a prompt, a high-quality chosen response, and a clearly inferior rejected response. Chosen is the quality bar the model should aim for — instruction-following target and positive preference example in one. Upload to Vidman AI and preview the pairs before committing.

Create a new run with ORPO as the method and select a base model — not an SFT checkpoint, since ORPO handles that stage internally. Open with lambda 0.1 and learning rate 5e-6, and train 2 to 3 epochs. Both losses stream live. When training completes, evaluate: the model should follow instructions correctly and produce outputs in the preferred style.