Play the role of a human rater and watch the reward model learn your preferences. See how RLHF turns human judgments into AI behaviour.
0% complete · 0/5 ratings · 0/3 phases explored
Sign in to save progress
Expand all three phases to complete this section.
For each question, pick the better AI response. Your choices train the reward model.
Prompt
Explain photosynthesis
0/5 ratings collected
Illustrative curve: reward model accuracy (agreement with held-out human preference labels) vs. number of human comparisons. Shape and endpoints anchored to published InstructGPT-era results.
At 100 comparisons the reward model is barely better than chance (50% on pairwise picks). With large comparison datasets, InstructGPT-scale reward models reached roughly 69-73% agreement with held-out labelers (Ouyang et al. 2022), close to the ~73% rate at which labelers agree with each other. That ceiling is real: the reward model stays imperfect, which is why reward hacking remains a problem.
DPO (Direct Preference Optimisation) skips the separate reward model entirely: it re-frames preference learning as a classification problem directly on the policy, eliminating the PPO loop. RLAIF (RL from AI Feedback) replaces human raters with a strong AI model like Claude or GPT-4, dramatically cutting annotation cost.
Reward Hacking
The policy discovers inputs that score highly on the reward model but are not actually preferred by humans, exploiting its blind spots.
Over-Optimisation
Without a KL penalty, the model drifts far from the original SFT policy, producing degenerate outputs that fool the reward model.
Cost
Collecting high-quality human preference data is slow and expensive. OpenAI spent millions on annotators for InstructGPT.
Key Insight
InstructGPT (2022) showed that a 1.3B RLHF model was preferred over a 175B GPT-3 by human evaluators. Scale doesn't matter as much as alignment. This was the breakthrough that led directly to ChatGPT.