Part 3 of 4 · 2 min read
Exercises
Easy
Easy 1: Run it (30 minutes)
Run python -m pytest -q and python run_all.py in code/mini-preference (the second takes about two minutes).
Done when: you can explain each table in the results.
Easy 2: Closed form by hand (30 minutes)
For one prompt with three responses, reference probabilities 0.5, 0.3 and 0.2 and rewards 1, 0, 2, compute pi* at beta 1.
Done when: your numbers match optimal_policy.
Hint
multiply each reference probability by exp(reward over beta), then normalise.
Easy 3: Why the normaliser cancels (30 minutes)
Derive the DPO gradient for a softmax policy.
Done when: you show the pair update is "raise the winner's logit, lower the loser's".
Hint
the log-normaliser appears in both the winner and loser terms, so it cancels.
Medium
Medium 1: Calibrate the reward model (3 hours)
Add temperature scaling to the reward model so its reward scale is chosen on held-out pairs.
Done when: RLHF is robust to the L2 setting.
Hint
fit one scalar multiplier on held-out pairs.
Medium 2: Reward ensembles (3 hours)
Train five reward models on bootstrap samples and optimise against their mean minus a multiple of their spread.
Done when: the share of runs worse than the reference falls.
Hint
use the lower confidence bound as the reward.
Medium 3: Sampled RLHF (3 hours)
Replace the exact gradient with REINFORCE using sampled responses and a baseline.
Done when: you report the extra variance against the exact gradient.
Hard
Hard 1: DPO on a small model (8 hours)
Train DPO on a small open model with a public preference set and compare with SFT using a held-out judge. (I did not run this, so your numbers are the ground truth.)
Done when: you report win rate, length drift and any safety regressions.
Hint
hold out prompts, not just responses.
Hard 2: Reward model and best-of-N (8 hours)
Train a reward model on your own comparisons and measure best-of-N against a human check. (I did not run this.)
Done when: you find the N after which human-judged quality stops improving.
Hard 3: Over-optimisation curve (6 hours)
Plot the proxy and an independent score against KL during RLHF on a real model. (I did not run this.)
Done when: you mark where they diverge.