KTO: Kahneman-Tversky Optimization for AI Alignment

KTO — Kahneman-Tversky Optimization — is an alignment method with its roots in prospect theory, from behavioural economics. Where DPO and SimPO demand paired preferences (chosen versus rejected for the same prompt), KTO trains on single-response binary feedback: every example is labelled good (thumbs up) or bad (thumbs down), independently.

That matters because paired preference data is expensive to gather. Most real-world feedback systems emit binary signals — users click thumbs up or down on individual responses; they do not compare two alternatives side by side. KTO consumes that naturally occurring format directly.

Prospect theory, the work of Daniel Kahneman and Amos Tversky, describes how people weigh gains and losses asymmetrically: losses bite harder than equivalent gains. Losing $100 hurts more than gaining $100 pleases. KTO carries that principle into training.

The KTO loss weights negative examples (bad responses) more heavily than positive ones, matching the psychological reality that users care more about dodging bad output than about maximising good output. That asymmetry makes KTO especially effective at suppressing harmful or low-quality responses.

A KTO dataset carries three fields — prompt, response, and a binary label (desirable or undesirable). No pairing; every example stands alone. Collection is therefore simple: log model outputs, have annotators or automated systems mark each one good or bad, and train.

The desirable-to-undesirable ratio shapes the run. KTO behaves well between 1:1 and 3:1, and the algorithm adjusts for class imbalance internally — but extreme skew (10:1 or worse) may call for tuning the beta hyperparameter.

KTO is the natural fit when feedback is binary rather than paired. User thumbs-up/down logs, helpfulness ratings collapsed to binary, automated quality scores thresholded into good and bad — KTO takes all of it directly, with no artificial step to fabricate preference pairs.

It also earns its place on safety work. Because the loss weights bad examples harder, KTO is effective at teaching a model to avoid specific failure modes: gather examples of undesirable behaviour, label them bad, set them against good examples, and train. The model learns to steer clear of the negative patterns.

Pick KTO as the method in the configurator and upload the dataset with prompt, response, and label columns. Format is validated, and the distribution of desirable versus undesirable examples is displayed.

The wizard carries the KTO-specific parameters — the beta weight and the loss asymmetry:

The headline hyperparameter is beta, setting the strength of the KTO loss (default 0.1). Higher beta aligns harder but can erode general capability. Track the desirable and undesirable loss components separately in the dashboard; both should descend.

When clean paired preferences already exist. If annotators compared responses head to head, that structure is information — and the DPO family is built to exploit exactly it. Flattening pairs into binary labels discards the comparison you paid for.

When the binary labels are unbalanced to a point. KTO tolerates imbalance well, but a handful of "undesirable" examples among thousands of "desirable" ones leaves the loss little to push against. Curate both sides, even where the negative side is smaller.

And when the feedback is ambiguous about what "good" means. A thumbs-down can signal wrong facts, wrong tone, or wrong format; an undifferentiated label teaches the model an average aversion. Clean the criteria before training on them.

A few KTO-specific practices matter beyond the paired-method basics. The first is ratio discipline: desirable to undesirable sits best between 1:1 and 3:1. Extreme skew — 10:1 desirable — can leave the negative signal underweighted and the alignment against bad behaviour flimsy.

The loss asymmetry means the model learns hardest from undesirable examples — prospect theory's losses-looming-larger-than-gains, operationalised. That makes KTO potent at teaching what not to do. Build the undesirable side deliberately: the specific failure modes you want gone, whether hallucinations, harmful content, or off-topic drift.

On beta: open at the 0.1 default and let evaluation arbitrate. Undesirable output surviving training means raising beta; a model turning overly cautious or refusing appropriate requests means it is too high — lower it for more freedom. KTO is generally less hyperparameter-sensitive than DPO, the loss having fewer moving parts.

Its unique strength is safety alignment, the asymmetry naturally emphasising avoidance. Where the primary goal is fewer harmful, incorrect, or policy-violating responses, KTO with a dataset centred on undesirable behaviour can outperform DPO for that specific objective.

KTO is the easiest alignment method to start with, because the dataset format is the simplest — no paired comparisons to construct. Collect model outputs and label each desirable or undesirable; the labels can come from human annotators, automated quality metrics, or production signals like thumbs-up and thumbs-down buttons.

Shape the dataset as JSONL with prompt, response, and label fields. Upload to Vidman AI and check format and distribution in the preview — the desirable-to-undesirable ratio is shown, confirming the balance is sane.

Create a new run, select KTO, and open with beta 0.1 and learning rate 5e-7. Train 1 to 2 epochs, watching the separate loss components: both should descend, with the undesirable loss typically falling steeper thanks to the asymmetry.

Afterwards, evaluate by generating outputs for your test prompts and checking whether the specific failure modes in your undesirable set have actually receded. KTO proves itself against the behaviours it was trained to avoid — not on general-purpose benchmarks.