Reward Modeling for RLHF: Training Custom Reward Functions

A reward model is a network taught to predict human preferences. Given a prompt and a response, it emits a scalar score for how good the response is by human judgment — a learned reward function that can then drive reinforcement learning of a language model.

Reward modeling is the opening step of the classical RLHF pipeline: gather preference data, train a reward model, then let that model guide RL training of the language model. Offline methods like DPO skip the reward model entirely, but explicit reward models still earn their place — as the judge for online RL, and as an instrument for evaluating model quality.

In the full pipeline, the reward model stands in for human judgment. Once trained, it supplies the reward signal for reinforcement learning algorithms — PPO (Proximal Policy Optimization), GRPO (Group Relative Policy Optimization), or GKD (Generalized Knowledge Distillation).

The reward model's quality caps the RL-trained model's quality. A judge that scores verbose-but-unhelpful responses highly produces a verbose policy. Hence the stress on reward model evaluation: verify that its rankings correlate with real human preferences before any policy is trained against it.

Reward model datasets use the paired format DPO uses — prompt, chosen response, rejected response — and the model learns to score chosen above rejected. The Bradley-Terry model typically converts those pairwise preferences into a pointwise reward.

For reward models, quality beats quantity outright. Inconsistent labels — annotators disagreeing on which response is better — inject noise that degrades scoring accuracy. Hold inter-annotator agreement above 70 percent, and consider filtering out examples where annotators strongly disagree.

Pick Reward Modeling as the method in the configurator and choose a base model — typically an SFT model of the same architecture as the target policy. Upload the preference dataset and set the training hyperparameters.

The wizard walks the reward model configuration:

Vidman AI trains it by swapping the language model head for a scalar output head, maximising the margin between chosen and rejected scores. The platform reports accuracy (share of pairs ranked correctly), mean reward for chosen versus rejected, and the reward margin distribution.

Training progress and reward accuracy stream live in the dashboard:

Vidman AI is bringing online reinforcement learning methods — PPO, GRPO, and GKD among them. These will use the trained reward model to optimise the policy in real time: generate responses, score them with the reward model, update the policy toward higher reward, repeat.

Online RL can align more strongly than offline methods, because it explores the response space during training and discovers behaviours a static preference dataset may never cover. Availability will be announced here on the blog.

When offline methods already clear the bar. A reward model earns its keep as the judge for online RL — PPO-class training that generates and scores fresh responses. If DPO or a sibling reaches the quality target, the reward model is infrastructure for a pipeline not being run.

When the preference data is thin. A reward model is itself a model trained on your preferences; a sparse preference set yields a judge with confident blind spots, and optimising against a flawed judge is precisely how reward hacking happens — outputs that score well and read badly.

And when the desired behaviour can be written down directly. If "good" reduces to a rubric or a validator, use it — a programmatic check never drifts, never gets charmed by fluent wrongness, and costs nothing per call. Train a judge only for the judgments that cannot be written.

A reliable reward model rests on data quality and evaluation discipline. The critical requirement is annotation consistency: inter-annotator agreement above 70 percent. Frequent disagreement teaches the model a noisy signal and unreliable scores; multiple annotators per example with majority voting lift label quality.

Size matters, within reason. The reward model must be large enough to capture the nuance of the preference criteria, but it need not match the policy it scores — a 7B judge can steer a 7B or even 13B policy effectively. Going much smaller than the policy invites reward hacking: the policy finds responses that score well on the simpler judge without being good.

Evaluate thoroughly before any RL trusts it. Split the preference data into training and test sets; report test accuracy — how often chosen ranks above rejected. Inspect the score distributions too: a healthy judge separates chosen and rejected cleanly, while overlapping distributions say it cannot tell quality levels apart.

Watch for overconfidence. Top marks for chosen and bottom marks for rejected on the training set, with poor test performance, means overfitting — cut epochs or add regularization.

Two ingredients feed a first reward model: a preference dataset and an SFT-trained base model. The dataset should carry at least 5,000 pairs — consistent annotations, clear quality gaps between chosen and rejected. Upload through the Datasets page and check the format in the preview.

Create a new run, select Reward Modeling, and point it at the SFT-trained base. The language model head is swapped for a scalar output head automatically. Set learning rate (typically 1e-5 to 5e-5), batch size, and epochs — 2 to 5 is the usual range for reward models.

Launch and watch the accuracy metric: the share of pairs ranked correctly. A well-trained judge reaches 75 percent or better on the training set; accuracy plateauing under 70 percent points to noisy preference data or a model too small for the criteria's complexity.

Once trained and evaluated, the reward model doubles as a quality evaluator for other fine-tuned models — score responses from different checkpoints to find which produces the best output by your learned criteria. And when online RL lands on Vidman AI, the trained judge is ready to drive PPO, GRPO, or GKD directly.