Skip to content
All apps

RLHF trainer — you are the human

About this app

You compare answers: a reward model learns from them (σ(r_A − r_B)), the policy shifts under a KL leash; with a loose leash the flattering answer wins (reward hacking); plus the DPO route.

What RLHF is

Reinforcement learning from human feedback (RLHF) aligns a pretrained, SFT-tuned language model with human preferences. People compare two answers at a time and pick the better one. A reward model learns scores r from this using the Bradley–Terry approach: P(A ≻ B) = σ(r_A − r_B). The policy is then optimised for high reward, usually with PPO, while a KL penalty with weight β keeps it close to the SFT model. The optimum is π(y) ∝ π_ref(y)·e^(r(y)/β). DPO (direct preference optimisation) reaches the same target without a separate reward model, directly from the pairs.

What the app shows

You are the human in the loop: across 15 comparisons you choose between two answers to the same prompt. The answer types range from helpful, curt, wrong and rambling to flattering and long. The reward model only sees text features such as “sounds competent”, “friendly” and “long”, and it shows its learned weights and its test loss on 40 held-out pairs. A bar chart compares the policy’s probabilities with the SFT model, and a curve plots reward against a gold rating by careful humans over the KL divergence.

Things to try

Label some pairs yourself or let five pairs be labelled honestly, then optimise the policy. Adjust the KL leash β: with a loose leash such as β = 0.03, the flattering, long answer moves to the top, even though it never appeared in your pairs. Reward keeps rising while the gold rating falls, which is reward hacking. Switch to DPO and watch each pair shift the policy directly. The before/after view shows the same model before and after optimisation.

Frequently asked questions

What is RLHF in simple terms?
RLHF stands for reinforcement learning from human feedback. People rate a language model’s answers, usually by comparing two of them. A reward model learns from these ratings, and the language model is then trained to make highly rewarded answers more likely.
What is reward hacking?
Reward hacking means a model finds answers that the reward model scores highly even though they are not actually better. The reward model is only an approximation of human judgement and may, for example, overrate length or friendliness. A KL penalty that keeps the model close to its starting point limits this effect.
What is the difference between RLHF and DPO?
RLHF first trains a separate reward model and then optimises the policy with reinforcement learning, for example PPO. DPO plugs the implicit reward r = β·log(π/π_ref) directly into the comparison loss and trains the policy on the preference pairs. Both aim at the same target, but DPO needs no reward model and no sampling loop.
What can I try in this app?
You compare answer pairs yourself, use them to train a reward model and optimise the policy. The KL leash slider β lets you trigger reward hacking and bring the helpful answer back, and you can switch between RLHF and DPO.

Subject: Neural networks

More from Neural networks

A VisuApp by heyprof: interactive, free in your browser, no sign-up. What is a VisuApp? · All apps