CPO: Contrastive Preference Optimization for LLM Alignment

CPO — Contrastive Preference Optimization — aligns language models to human preferences through a contrastive objective. Where DPO looks only at the relative probability between chosen and rejected, CPO pushes the absolute probability of chosen responses up while driving rejected ones down.

The contrastive framing closes a known DPO failure mode: a model can satisfy a relative objective by sinking both probabilities, so long as chosen stays above rejected. CPO keeps generation quality high for the preferred response while steering well clear of the undesirable ones.

The CPO loss blends two terms: a behavior-cloning loss over the chosen response — pulling the model toward reproducing preferred outputs — and a contrastive margin loss between chosen and rejected, carrying the preference signal. The cloning term doubles as a regulariser, keeping the model from drifting away from high-quality generation.

A hyperparameter weighs the two objectives against each other: more contrastive weight yields stronger alignment, more cloning weight preserves general capability. And like SimPO, CPO runs without a reference model, trimming memory requirements below DPO's.

The data is the standard paired format — prompt, chosen response, rejected response — with quality expectations matching DPO's. Clear preference signals between chosen and rejected produce the best training outcomes.

CPO tends to tolerate noisy preference labels better than DPO: the behavior-cloning component over chosen responses supplies a stable learning signal even where some pairs are ambiguous or mislabelled.

CPO is a strong pick when alignment must not cost generation quality. The cloning term keeps the model producing genuinely high-quality output in the preferred style — not merely output that outscores the rejected alternative.

It performs especially well on tasks where quality matters as much as preference: creative writing, code generation, technical documentation. Those demand both — following preferences for tone, format, accuracy while holding generative capability high.

Pick CPO as the training method in the configurator. The main hyperparameter is beta, setting the contrastive loss strength; Vidman AI ships defaults tuned for common use cases. Upload the preference dataset and the chosen/rejected format is validated automatically.

The wizard surfaces the CPO-specific parameters:

Track the chosen loss and the contrastive loss separately in the dashboard. A healthy run shows the chosen loss descending — the model learning to reproduce preferred outputs — while the margin between chosen and rejected probabilities widens.

When the pairs carry noise. Every contrastive method amplifies the difference between chosen and rejected; if that difference is inconsistent — different annotators, different standards — the method learns the noise with conviction. Audit the pairs before the hyperparameters.

When the model has not been instruction-tuned. CPO, like the rest of the preference family, aligns a model that already behaves; it does not create the behaviour. SFT first, always.

And when the gap you need is capability, not alignment. Preference methods adjust which of several plausible behaviours the model favours; they do not teach it to solve problems it currently fails. A "cannot" calls for capability training; a "could, but chooses poorly" puts you in the right post.

CPO leans hard on the quality of chosen responses, because the cloning component trains the model to reproduce them directly. Unlike DPO, where only the relative preference is learned, CPO explicitly teaches the model to generate chosen-quality output — so chosen responses should be exemplary. They set the ceiling for what the model can produce after alignment.

Beta weighs the cloning and contrastive objectives. Open at the Vidman AI default and let evaluation arbitrate: high-quality output that is insufficiently aligned means raising beta; generation quality slipping after alignment means lowering it to favour cloning.

CPO suits domains where output quality is non-negotiable — code generation, technical writing, customer-facing content. There, preferring the right style is not enough; generative capability must hold. The cloning regulariser makes that guarantee explicit in a way DPO's does not, which earns CPO default status for quality-sensitive applications.

When building the dataset, spend extra on chosen responses. They double as cloning targets, so each should be the best possible output for its prompt — senior annotators or domain experts writing chosen, rather than model-generated examples.

CPO starts with the dataset: prompt, chosen, rejected — chosen being high-quality exemplars of the output style you want. Upload it, verify the pairs in the preview, then create a new training run, select CPO, and set beta.

Open at learning rate 5e-7 and train 1 to 2 epochs. Watch the chosen loss and the contrastive loss separately in the dashboard; in a healthy run both descend. The chosen loss tracks how well the model reproduces preferred output, the contrastive loss how well it tells preferred from dispreferred.

After training, compare the CPO-aligned model against the SFT baseline and any other alignment methods tried — DPO, SimPO. The training list keeps every experiment side by side. For quality-sensitive tasks, where generation capability must survive alongside preference alignment, CPO often comes out ahead.