RLVR: Learning With Verifiable Rewards

The alignment methods covered so far — the preference optimisers, the reward models, the offline and online RLHF variants — share one expensive assumption: someone has to judge what a good answer is. A human, a reward model, a preference pair. The judgment is the bottleneck, and the judgment is where the noise enters.

RLVR — reinforcement learning with verifiable rewards — removes the judgment. The reward is computed, not judged: the answer is checked against a ground truth a program can verify. The math problem is right or wrong; the code passes the tests or it does not; the proof is valid or it is not. No human reads the answer — a checker does.

The consequence is a training signal of a different kind: exact, scalable, free of the preference noise plaguing judged rewards. It is the mechanism behind the most visible capability jumps in reasoning models — the ones that got dramatically better at math and code by being rewarded for getting the answer right.

A verifiable reward is one a program computes from the answer alone, no judgment involved. The cleanest cases: the answer is a number and the checker compares it; the answer is code and the checker runs the tests; the answer is a proof and the checker verifies the steps. The reward is binary or near-binary — right or wrong — and the signal is unambiguous.

The boundary matters: verifiability is a property of the task, not the model. A math problem with a known answer is verifiable; an essay is not. A coding task with a test suite is verifiable; a code review is not. RLVR works where the ground truth exists and a checker can reach it.

And the checker is the whole game. A buggy checker trains the model to satisfy the bug; a checker accepting near-misses trains the model to be nearly right. The verification logic deserves the same engineering care as the training loop — it is the teacher.

The loop keeps the standard reinforcement-learning shape, with the reward function swapped for the checker. The model generates an answer; the checker scores it; the score becomes the reward; the model updates toward answers scoring higher. What changes is the signal’s quality: instead of a preference model’s noisy estimate of human approval, the model gets ground truth.

The training data is the task set — problems with verifiable answers, generated in volume. The model attempts, the checker scores, and the successful attempts become the positive examples for the next round. The loop runs for many rounds, and the success rate climbs as the model learns the patterns leading to correct answers.

The result is not a model that memorised the answers — the task set is large and the model generalises — but one that learned the process producing correct answers: step-by-step reasoning, self-checking, retry after a failed attempt. The process is what transfers to new problems.

Mathematics comes first and most visibly: the answer is checkable, the problem space is infinite, and the reasoning process is exactly what the method reinforces. Code: test suites are the perfect checker, and the gap between “code that looks right” and “code that passes” is precisely what RLVR closes. Formal reasoning: proofs, logic puzzles, anything with a verifier.

And the emerging frontier: any task where a checker can be built. Tool use, where the checker verifies the tool was called correctly; structured output, where the checker validates the schema. The pattern is the same — find the ground truth, write the checker, and the training signal follows.

The common thread: these tasks reward process. A model getting better at checking its own work, retrying, and decomposing problems gets better at everything downstream — the capability gains are not confined to the trained tasks.

The classic failure mode: the model learns to satisfy the checker without doing the task. The checker accepts a specific output format, and the model learns the format without the reasoning. The checker has a bug, and the model finds it. The reward is binary, and the model learns a confident wrong answer scores like a hesitant one — so it becomes confidently wrong.

Then the subtler trap: the checker is right and the model still overfits the task distribution. The model gets brilliant on training problems and brittle elsewhere, because the reward signal is so clean that training overshoots — the same overfitting signature as any fine-tune, amplified by a signal with no noise to slow it down.

And the deepest trap: verifiable tasks are a narrow slice of what models are asked to do. A model trained heavily on verifiable rewards can lose the texture unverifiable tasks need — the judgment, the nuance, the willingness to say “it depends”. The fix is the usual one: mix the training, keep general data in the loop, and evaluate on the tasks that actually matter.

When no checker exists for the task. Writing, advice, open-ended analysis — tasks where quality is judged, not computed — get nothing from RLVR, because the method is the checker. For those, the preference methods remain the right machinery, with all their noise and all their necessity.

It is also wrong when the checker is expensive or slow. A checker taking minutes per answer throttles the training loop; the method’s whole advantage is the volume of cheap, exact signal. Verification costing as much as generation collapses the economics.

And it is wrong when the goal is style rather than correctness. RLVR optimises the answer; it does not optimise the voice. A model that needs to sound like the brand needs preference data, not a checker — the two methods train different things, and the choice is the task’s, not the fashion’s.