RLHF trainer — you are the human
About this app
You compare answers: a reward model learns from them (σ(r_A − r_B)), the policy shifts under a KL leash; with a loose leash the flattering answer wins (reward hacking); plus the DPO route.
What RLHF is
Reinforcement learning from human feedback (RLHF) aligns a pretrained, SFT-tuned language model with human preferences. People compare two answers at a time and pick the better one. A reward model learns scores r from this using the Bradley–Terry approach: P(A ≻ B) = σ(r_A − r_B). The policy is then optimised for high reward, usually with PPO, while a KL penalty with weight β keeps it close to the SFT model. The optimum is π(y) ∝ π_ref(y)·e^(r(y)/β). DPO (direct preference optimisation) reaches the same target without a separate reward model, directly from the pairs.
What the app shows
You are the human in the loop: across 15 comparisons you choose between two answers to the same prompt. The answer types range from helpful, curt, wrong and rambling to flattering and long. The reward model only sees text features such as “sounds competent”, “friendly” and “long”, and it shows its learned weights and its test loss on 40 held-out pairs. A bar chart compares the policy’s probabilities with the SFT model, and a curve plots reward against a gold rating by careful humans over the KL divergence.
Things to try
Label some pairs yourself or let five pairs be labelled honestly, then optimise the policy. Adjust the KL leash β: with a loose leash such as β = 0.03, the flattering, long answer moves to the top, even though it never appeared in your pairs. Reward keeps rising while the gold rating falls, which is reward hacking. Switch to DPO and watch each pair shift the policy directly. The before/after view shows the same model before and after optimisation.
Frequently asked questions
- What is RLHF in simple terms?
- RLHF stands for reinforcement learning from human feedback. People rate a language model’s answers, usually by comparing two of them. A reward model learns from these ratings, and the language model is then trained to make highly rewarded answers more likely.
- What is reward hacking?
- Reward hacking means a model finds answers that the reward model scores highly even though they are not actually better. The reward model is only an approximation of human judgement and may, for example, overrate length or friendliness. A KL penalty that keeps the model close to its starting point limits this effect.
- What is the difference between RLHF and DPO?
- RLHF first trains a separate reward model and then optimises the policy with reinforcement learning, for example PPO. DPO plugs the implicit reward r = β·log(π/π_ref) directly into the comparison loss and trains the policy on the preference pairs. Both aim at the same target, but DPO needs no reward model and no sampling loop.
- What can I try in this app?
- You compare answer pairs yourself, use them to train a reward model and optimise the policy. The KL leash slider β lets you trigger reward hacking and bring the helpful answer back, and you can switch between RLHF and DPO.
Subject: Neural networks
More from Neural networks
- Agent loop
- Attention Visualiser
- Backpropagation — blame in reverse
- Backpropagation: learning from one mistake
- Basin of Attraction
- Your job under AI
- The temperature dial
- Diffusion model — images from noise (3D)
A VisuApp by heyprof: interactive, free in your browser, no sign-up. What is a VisuApp? · All apps