CSE Roadmap

Common tier · about 4 hours · 4 parts

Part 2 of 4 · 14 min read

Definitions

Covers: Definitions; The two routes; Results; Design rules; Interview questions, with the follow-ups; Common mistakes; What the example does not do.

Definitions

Preference data

  • Preference data: (prompt, winning response, losing response) triples from humans, AI judges or verifiers.

Reward model

  • Reward model: a model that maps (prompt, response) to a scalar score, trained so winners score higher than losers.

Bradley-Terry model

  • Bradley-Terry: P(a beats b) = sigmoid(r(a) - r(b)); the basis of the reward-model loss.

KL divergence and the reference model

  • Reference model: the starting (usually SFT) policy. KL divergence: how far the new policy's distribution is from the reference. RLHF and DPO penalise it with weight beta.

RLHF, PPO, DPO, RLAIF

  • RLHF: reinforcement learning from human feedback via a reward model. PPO: a reinforcement learning algorithm used for the optimisation step. DPO: direct preference optimisation, which trains on the pairs without a reward model. RLAIF: preferences produced by an AI judge.

Reward over-optimisation

  • Over-optimisation (Goodhart): pushing a proxy reward until the true objective gets worse.

Best-of-N

  • Best-of-N: sample N outputs and keep the one the reward model scores highest.

The two routes

AAAWgmp1bWIAAAAeanVtZGMycGEAEQAQgAAAqgA4m3EDYzJwYQAAABZcanVtYgAAAEdqdW1kYzJtYQARABCAAACqADibcQN1cm46YzJwYTpiMDg3OWEzNi0yMGU2LTQyMDQtYjc4OC04NjRmZTMwNDRiM2QAAAADl2p1bWIAAAApanVtZGMyYXMAEQAQgAAAqgA4m3EDYzJwYS5hc3NlcnRpb25zAAAAALxqdW1iAAAARGp1bWRjYm9yABEAEIAAAKoAOJtxE2MycGEuaW5ncmVkaWVudC52MwAAAAAYYzJzaKF5Ip6CCm3GNkGrntYLpMkAAABwY2JvcqNpZGM6Zm9ybWF0bWltYWdlL3N2Zyt4bWxqaW5zdGFuY2VJRHgseG1wOmlpZDo3OWM1NzJhMy02NzQyLTRiNTMtOWY4Ni0wZTcwYmQ5YTJiYjZscmVsYXRpb25zaGlwaHBhcmVudE9mAAAB4mp1bWIAAABBanVtZGNib3IAEQAQgAAAqgA4m3ETYzJwYS5hY3Rpb25zLnYyAAAAABhjMnNo/EKPVuSlEe349pnElE7b1gAAAZljYm9yomdhY3Rpb25zgqJmYWN0aW9ua2MycGEub3BlbmVkanBhcmFtZXRlcnOha2luZ3JlZGllbnRzgaJjdXJseC1zZWxmI2p1bWJmPWMycGEuYXNzZXJ0aW9ucy9jMnBhLmluZ3JlZGllbnQudjNkaGFzaFggbuT0N2eD/Ml6QxHWH2m0qwz0lHGhH3f+W67nahiiUOekZmFjdGlvbngdY29tLmFudGhyb3BpYy5jbGF1ZGUucHJvdmlkZWRqcGFyYW1ldGVyc6F4H2NvbS5hbnRocm9waWMub3JpZ2luLWNvbmZpZGVuY2VndW5rbm93bmtkZXNjcmlwdGlvbnhmQ2xhdWRlIHByb3ZpZGVkIHRoaXMgZmlsZSBhdCB0aGUgcmVxdWVzdCBvZiBhIHVzZXIgYW5kIG1heSBoYXZlIGNyZWF0ZWQgb3IgbW9kaWZpZWQgdGhlIGZpbGUgY29udGVudHMubXNvZnR3YXJlQWdlbnShZG5hbWVmQ2xhdWRlcmFsbEFjdGlvbnNJbmNsdWRlZPUAAADIanVtYgAAAEBqdW1kY2JvcgARABCAAACqADibcRNjMnBhLmhhc2guZGF0YQAAAAAYYzJzaMqDOKuZ0Y9vX3ZMu6GT+oQAAACAY2JvcqVjYWxnZnNoYTI1NmNwYWRNAAAAAAAAAAAAAAAAAGRoYXNoWCBDnQxTyetDo51aAoVw+BG7Q/QeJdsmxJTRune4g4Nuf2RuYW1lbmp1bWJmIG1hbmlmZXN0amV4Y2x1c2lvbnOBomVzdGFydBifZmxlbmd0aBkeBAAAAj5qdW1iAAAAJ2p1bWRjMmNsABEAEIAAAKoAOJtxA2MycGEuY2xhaW0udjIAAAACD2Nib3KlY2FsZ2ZzaGEyNTZpc2lnbmF0dXJleE1zZWxmI2p1bWJmPS9jMnBhL3VybjpjMnBhOmIwODc5YTM2LTIwZTYtNDIwNC1iNzg4LTg2NGZlMzA0NGIzZC9jMnBhLnNpZ25hdHVyZWppbnN0YW5jZUlEeCx4bXA6aWlkOmM3NjU4YmM0LTcxZmUtNDIzNC1hMTk3LWJmMzQ0ZTI0OTkxOXJjcmVhdGVkX2Fzc2VydGlvbnODomN1cmx4LXNlbGYjanVtYmY9YzJwYS5hc3NlcnRpb25zL2MycGEuaW5ncmVkaWVudC52M2RoYXNoWCBu5PQ3Z4P8yXpDEdYfabSrDPSUcaEfd/5brudqGKJQ56JjdXJseCpzZWxmI2p1bWJmPWMycGEuYXNzZXJ0aW9ucy9jMnBhLmFjdGlvbnMudjJkaGFzaFggyVSUF/yzMTSINibSuhTtjRwNhgBAKNJGZpcGyqwiIYGiY3VybHgpc2VsZiNqdW1iZj1jMnBhLmFzc2VydGlvbnMvYzJwYS5oYXNoLmRhdGFkaGFzaFgg4Xng7L6Nj25TwvltD9GzQOtoydswfkK2BABQSZJqmER0Y2xhaW1fZ2VuZXJhdG9yX2luZm+jZG5hbWVvQW50aHJvcGljIEZpbGVzZ3ZlcnNpb25lMS4wLjBrc3BlY1ZlcnNpb25lMi40LjAAABA4anVtYgAAAChqdW1kYzJjcwARABCAAACqADibcQNjMnBhLnNpZ25hdHVyZQAAABAIY2JvctKEWQISogEmGCFZAgowggIGMIIBjaADAgECAhRA5aAK7sI50L64g/oGQgU9Z1UTADAKBggqhkjOPQQDAzBJMRcwFQYDVQQKEw5BbnRocm9waWMsIFBCQzEuMCwGA1UEAxMlQW50aHJvcGljIENvbnRlbnQgQ3JlZGVudGlhbHMgUm9vdCBDQTAeFw0yNjA4MDcxODQzNTZaFw0yODA4MDYxOTQzNTZaMEQxFzAVBgNVBAoTDkFudGhyb3BpYywgUEJDMSkwJwYDVQQDEyBBbnRocm9waWMgQ2xhdWRlIENvbnRlbnQgU2lnbmluZzBZMBMGByqGSM49AgEGCCqGSM49AwEHA0IABJh6CmvLUBgFFNU0vUKlOVtE6djd17L5SuwX0LemFisBM3dkd/3cyjxFA3Qo5S46fX0/ihY0VZ7mfb9KF703t5OjWDBWMA4GA1UdDwEB/wQEAwIHgDAVBgNVHSUEDjAMBgorBgEEAYPoXgIBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUzlHiBIFOZFsj+OPEz5o+nMHXXMIwCgYIKoZIzj0EAwMDZwAwZAIwMXMdFJ4BetLLVY7ORuE9noqbbAZOZn/aArXyTwFAZfKrPzxF2vPoJNf1+UCdg1XGAjBwX1zd9WGqYkqmL5SFqw1QySjr1zJfpJM9+1rdDwSPLMOPOjKuiXjoU/pUUeG9RwmhY3BhZFkNngAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAPZYQD3eU+/o5ncuc8TE4aO3MV3D513KzCoSpVNrYdE+Qrmuc5Z5inBtAngrlS7bGyFePovKVcZmy8mw2zrZGM7fUKo=Two routes from preference data to a tuned policy, and what they shareRLHF (two stages)1. Collect comparisons (prompt, winner, loser)2. Fit a reward model: minimise -log sigmoid(r(w) - r(l))3. Optimise the policy to raise reward with a KL leash to the reference model (PPO in practice; exact gradient in my toy)Pros: reusable reward model, best-of-N, more control. Cons: more moving parts.DPO (one stage)1. Collect comparisons (same data)2. Minimise -log sigmoid(beta[(log pi(w) - log ref(w)) - (log pi(l) - log ref(l))]) No separate reward model, no sampling loop.Meta moved from RLHF (Llama 2) to DPO (Llama 3) to cut complexity.What they share: both aim at pi*(a|c) proportional to pi_ref(a|c) * exp(r(c,a) / beta)beta is the KL leash: large beta stays near the reference, small beta chases the reward (and its errors).DPO's implicit reward is beta * log(pi / pi_ref), known only up to a per-prompt constant.Check (my toy, true reward): RLHF's exact-gradient result vs the closed form, largest probability difference:beta 0.1: 8.3e-03 beta 0.3: 2.0e-03 beta 1.0: 1.0e-04 beta 3.0: 6.5e-08Both use a softmax policy over 10 responses for each of 6 prompts; the rewards are random draws. Not an LLM.
RLHF and DPO pipelines and the optimum they share.

Results

Beta (3,000 clean pairs, 10 worlds) RLHF DPO Perfect KL-regularised policy KL of DPO from reference
0.05 2.14 2.13 2.16 1.46
0.1 2.13 2.13 2.16 1.41
0.3 2.06 2.07 2.08 1.08
1 1.65 1.68 1.69 0.37
3 1.18 1.21 1.21 0.07
Pairs of data (beta 0.3) RLHF DPO Perfect
30 1.36 1.36 2.08
100 1.65 1.65 2.08
300 1.78 1.82 2.08
1,000 2.03 2.04 2.08
3,000 2.06 2.07 2.08
Labels flipped (1,000 pairs, beta 0.3) RLHF DPO
0% 2.03 2.04
10% 1.92 1.93
20% 1.68 1.69
30% 1.25 1.28
40% 0.78 0.83
Reward-model L2 (1,000 pairs, RLHF at beta 0.3) True reward KL Spread of fitted reward
0.1 1.01 0.02 0.05
0.01 1.65 0.41 0.26
0.001 2.00 0.96 0.75
0.0001 2.03 1.13 1.26
Best of N (reward model on 300 pairs) N=1 2 4 8 16 32
True reward 0.80 1.28 1.60 1.76 1.83 1.81
AAAWgmp1bWIAAAAeanVtZGMycGEAEQAQgAAAqgA4m3EDYzJwYQAAABZcanVtYgAAAEdqdW1kYzJtYQARABCAAACqADibcQN1cm46YzJwYTpiNWI4MjU3Yi02YWYyLTRiN2QtOTUyZi0wZTcwMzljNzVhNjcAAAADl2p1bWIAAAApanVtZGMyYXMAEQAQgAAAqgA4m3EDYzJwYS5hc3NlcnRpb25zAAAAALxqdW1iAAAARGp1bWRjYm9yABEAEIAAAKoAOJtxE2MycGEuaW5ncmVkaWVudC52MwAAAAAYYzJzaCj47ZLVpa3y2yxaENV03HMAAABwY2JvcqNpZGM6Zm9ybWF0bWltYWdlL3N2Zyt4bWxqaW5zdGFuY2VJRHgseG1wOmlpZDo1ZjNiYTJmOS1mNjMwLTRkMTUtYjFhMi1hMWFiZmIyNTE3OTlscmVsYXRpb25zaGlwaHBhcmVudE9mAAAB4mp1bWIAAABBanVtZGNib3IAEQAQgAAAqgA4m3ETYzJwYS5hY3Rpb25zLnYyAAAAABhjMnNoChQzm7rHSS6D5uMyC/lVnQAAAZljYm9yomdhY3Rpb25zgqJmYWN0aW9ua2MycGEub3BlbmVkanBhcmFtZXRlcnOha2luZ3JlZGllbnRzgaJjdXJseC1zZWxmI2p1bWJmPWMycGEuYXNzZXJ0aW9ucy9jMnBhLmluZ3JlZGllbnQudjNkaGFzaFgg2ZOGGJWziSIDbGyfkbTbD9qHwqC6fpm66JZ+7FcnkY+kZmFjdGlvbngdY29tLmFudGhyb3BpYy5jbGF1ZGUucHJvdmlkZWRqcGFyYW1ldGVyc6F4H2NvbS5hbnRocm9waWMub3JpZ2luLWNvbmZpZGVuY2VndW5rbm93bmtkZXNjcmlwdGlvbnhmQ2xhdWRlIHByb3ZpZGVkIHRoaXMgZmlsZSBhdCB0aGUgcmVxdWVzdCBvZiBhIHVzZXIgYW5kIG1heSBoYXZlIGNyZWF0ZWQgb3IgbW9kaWZpZWQgdGhlIGZpbGUgY29udGVudHMubXNvZnR3YXJlQWdlbnShZG5hbWVmQ2xhdWRlcmFsbEFjdGlvbnNJbmNsdWRlZPUAAADIanVtYgAAAEBqdW1kY2JvcgARABCAAACqADibcRNjMnBhLmhhc2guZGF0YQAAAAAYYzJzaJenAmlQv3qoOMAnohsp9iMAAACAY2JvcqVjYWxnZnNoYTI1NmNwYWRNAAAAAAAAAAAAAAAAAGRoYXNoWCATlJrhuD8FXPetsPOvi0ybzgJ5Y9TOAAOV2E6jcRT1fmRuYW1lbmp1bWJmIG1hbmlmZXN0amV4Y2x1c2lvbnOBomVzdGFydBidZmxlbmd0aBkeBAAAAj5qdW1iAAAAJ2p1bWRjMmNsABEAEIAAAKoAOJtxA2MycGEuY2xhaW0udjIAAAACD2Nib3KlY2FsZ2ZzaGEyNTZpc2lnbmF0dXJleE1zZWxmI2p1bWJmPS9jMnBhL3VybjpjMnBhOmI1YjgyNTdiLTZhZjItNGI3ZC05NTJmLTBlNzAzOWM3NWE2Ny9jMnBhLnNpZ25hdHVyZWppbnN0YW5jZUlEeCx4bXA6aWlkOjhmZjAyZWU2LWU5YmItNGVlYS1hYWY0LTdmYzllNmU2ZGEwN3JjcmVhdGVkX2Fzc2VydGlvbnODomN1cmx4LXNlbGYjanVtYmY9YzJwYS5hc3NlcnRpb25zL2MycGEuaW5ncmVkaWVudC52M2RoYXNoWCDZk4YYlbOJIgNsbJ+RtNsP2ofCoLp+mbroln7sVyeRj6JjdXJseCpzZWxmI2p1bWJmPWMycGEuYXNzZXJ0aW9ucy9jMnBhLmFjdGlvbnMudjJkaGFzaFggCSRb1cxQLJGjYx3t81xRBFDZv6c2/Pyu/qXE+fsCCWuiY3VybHgpc2VsZiNqdW1iZj1jMnBhLmFzc2VydGlvbnMvYzJwYS5oYXNoLmRhdGFkaGFzaFggqG1ucL4iFb9iDpvXLgizTpJPQbVlURgde6RCfNp4KDB0Y2xhaW1fZ2VuZXJhdG9yX2luZm+jZG5hbWVvQW50aHJvcGljIEZpbGVzZ3ZlcnNpb25lMS4wLjBrc3BlY1ZlcnNpb25lMi40LjAAABA4anVtYgAAAChqdW1kYzJjcwARABCAAACqADibcQNjMnBhLnNpZ25hdHVyZQAAABAIY2JvctKEWQISogEmGCFZAgowggIGMIIBjaADAgECAhRA5aAK7sI50L64g/oGQgU9Z1UTADAKBggqhkjOPQQDAzBJMRcwFQYDVQQKEw5BbnRocm9waWMsIFBCQzEuMCwGA1UEAxMlQW50aHJvcGljIENvbnRlbnQgQ3JlZGVudGlhbHMgUm9vdCBDQTAeFw0yNjA4MDcxODQzNTZaFw0yODA4MDYxOTQzNTZaMEQxFzAVBgNVBAoTDkFudGhyb3BpYywgUEJDMSkwJwYDVQQDEyBBbnRocm9waWMgQ2xhdWRlIENvbnRlbnQgU2lnbmluZzBZMBMGByqGSM49AgEGCCqGSM49AwEHA0IABJh6CmvLUBgFFNU0vUKlOVtE6djd17L5SuwX0LemFisBM3dkd/3cyjxFA3Qo5S46fX0/ihY0VZ7mfb9KF703t5OjWDBWMA4GA1UdDwEB/wQEAwIHgDAVBgNVHSUEDjAMBgorBgEEAYPoXgIBMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUzlHiBIFOZFsj+OPEz5o+nMHXXMIwCgYIKoZIzj0EAwMDZwAwZAIwMXMdFJ4BetLLVY7ORuE9noqbbAZOZn/aArXyTwFAZfKrPzxF2vPoJNf1+UCdg1XGAjBwX1zd9WGqYkqmL5SFqw1QySjr1zJfpJM9+1rdDwSPLMOPOjKuiXjoU/pUUeG9RwmhY3BhZFkNngAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAPZYQKgDo0Z0y2oNkfEzLRs3ICCeIO+qN5JvfY6LQ5hx3nSrME7zDJiS4gzmFaPKuY/dF+6lTeA25C4uZI+MSP4F1eI=Toy results: where preference tuning pays, and where it over-optimises (true reward; reference 0.80, perfect about 2.0)Beta sweep (3,000 clean pairs): true reward | KL from referencebeta 0.05RLHF 2.14 DPO 2.13 KL 1.46beta 0.1RLHF 2.13 DPO 2.13 KL 1.41beta 0.3RLHF 2.06 DPO 2.07 KL 1.08beta 1.0RLHF 1.65 DPO 1.68 KL 0.37beta 3.0RLHF 1.18 DPO 1.21 KL 0.07Pairs of data (beta 0.3): true reward, DPO vs perfect30 pairs1.36 (perfect 2.08)100 pairs1.65 (perfect 2.08)300 pairs1.82 (perfect 2.08)1000 pairs2.04 (perfect 2.08)3000 pairs2.07 (perfect 2.08)Label noise (1,000 pairs, beta 0.3): true reward0% flipped2.0410% flipped1.9320% flipped1.6930% flipped1.2840% flipped0.83Best of N with a learned reward model (300 pairs)N = 10.80N = 21.28N = 41.60N = 81.76N = 161.83N = 321.81Over-optimisation, worst of 20 runs (100 noisy pairs, beta 0.02): the learned reward keeps rising, the true reward fallsstep 0learned reward 0.20TRUE reward 0.85step 100learned reward 3.24TRUE reward -0.39step 200learned reward 3.25TRUE reward -0.39step 300learned reward 3.26TRUE reward -0.39step 400learned reward 3.26TRUE reward -0.40With the same settings 60% of 20 runs ended worse than the reference. A larger beta, more or cleaner data and a held-out check reduce this.Blue bars = RLHF with a fitted reward model; green = DPO; both match the closed form when data is plentiful and the reward model is not over-regularised.
Beta, data, noise, best-of-N and over-optimisation in the toy.
  1. The KL leash is the main knob; it trades reward gain for distance from a model you trust.
  2. RLHF and DPO share an optimum; differences in practice come from optimisation, calibration and engineering, not the objective.
  3. Data quality limits everything: 1,000 pairs with 30 percent flipped labels (1.28) did worse than 30 clean pairs (1.36).
  4. The proxy can fail silently: the learned reward rose while the true reward fell.

Design rules

  • Start from a strong SFT model and keep it as the reference.
  • Collect comparisons, not scores, and measure labeler agreement (a figure of about 73 percent was reported for InstructGPT).
  • Pick beta by a held-out evaluation, not by the training reward.
  • Watch for over-optimisation: track the proxy and an independent check together, and stop early.
  • Check reward-model calibration (scale and regularisation) before concluding that one method is better.
  • Filter noisy or contradictory pairs; audit a sample by hand.
  • Prefer DPO for simplicity; use RLHF or best-of-N when you need a reusable scorer.
  • Do not trust the reward model on prompts unlike its training data.
  • Run a regression eval (safety, tool use, hallucination) before and after; preference tuning can improve overall preference while making one of these worse.
  • Version datasets, reward models and reference models together.

Interview questions, with the follow-ups

Explain RLHF and DPO, and how they differ.

"RLHF trains a reward model on comparisons with a Bradley-Terry loss, then optimises the policy, usually with PPO, to maximise reward minus beta times KL from the reference. DPO starts from the closed-form optimum of that same objective, pi* proportional to pi_ref times exp(reward over beta), and trains the policy directly on the pairs with a logistic loss on the log-ratio difference, so it needs no reward model and no sampling loop. In my toy, with a well-fitted reward model, both reached the same true reward." Follow-up: "When would you still use RLHF?" When you want a reusable reward model (ranking, filtering, best-of-N) or online data from the current policy.

What does beta do?

"It weights the KL penalty. Large beta keeps the policy near the reference; small beta lets it chase the reward and its errors. In my toy beta 0.05 gave 2.14 true reward with a KL of 1.5 and beta 3 gave 1.18 with a KL of 0.06." Follow-up: "How do you choose it?" On a held-out evaluation that is independent of the reward model, watching for the point where gains stop.

What is reward hacking or over-optimisation?

"Optimising a proxy until it diverges from the real goal. In my toy, with 100 noisy pairs and a small beta, the learned reward rose from 0.2 to 3.2 while the true reward fell below zero in the worst run, and 60 percent of runs ended worse than the starting model." Follow-up: "How do you mitigate it?" Larger KL penalty, more and cleaner data, reward-model ensembles or uncertainty, early stopping on a held-out check, and mixing in verifiable rewards.

Why train on comparisons instead of scores?

"People disagree on absolute scores and drift over time, but they agree more often on which of two responses is better; the Bradley-Terry loss turns rankings into a scoring model. About 73 percent agreement was reported among InstructGPT labelers, so labels are still noisy, which my toy shows costs a lot of reward at 30 percent flips." Follow-up: "What does noise do?" It lowers the achievable reward and can make the reward model confidently wrong.

Common mistakes

  • Comparing RLHF and DPO without matching reward-model calibration (my first run did this).
  • Evaluating on the reward model that the policy was trained against.
  • Tiny beta with a weak reward model.
  • Skipping the reference model or letting it drift.
  • Training on contradictory or noisy preference pairs without auditing.
  • Ignoring regressions in safety, tool use and factuality.
  • Treating the implicit DPO reward as an absolute score (it is defined up to a per-prompt constant).
  • Assuming PPO behaves like exact gradients; real RL adds variance and instability.
  • Using best-of-N with a poorly calibrated reward model and large N.
  • Assuming preferences are universal.

What the example does not do

  • No language model: a lookup-table policy over 10 responses per prompt, with no generalisation between prompts.
  • No PPO and no sampling noise, so none of RL's instability or variance appears.
  • True rewards are random, and the labelers are perfect Bradley-Terry samplers (plus optional flipped labels); real labels have bias, length effects and annotator disagreement.
  • The reward model is also tabular, so it cannot extrapolate to unseen prompts or responses, which is where real over-optimisation often starts.
  • Ten random worlds per setting, a single reference policy family, and hand-chosen learning rates.
  • No offline-versus-online distinction, no length bias, no RLAIF.
  • Optional real experiment (I did not run this): with a small open model and a public preference dataset, train DPO at beta 0.05, 0.1 and 0.5 using a library such as TRL, then compare win rate against the SFT model by a held-out judge and check for length and refusal drift.

Check yourself

What do beta in RLHF and beta in DPO control? (How far the tuned policy may move from the reference model. Small beta lets it gain more reward and drift further; large beta keeps it close. In my toy the perfect policy at beta 0.05 had KL 1.52 from the reference, at beta 3 only 0.06.)

For the curious A learned reward is a proxy, and it can be gamed

With 100 pairs, 30 percent label noise and beta 0.02, 60 percent of 20 runs ended worse than the reference on the true reward; in the worst run the learned reward rose from 0.20 to 3.24 while the true reward fell from 0.85 to -0.39. My first run also showed DPO beating RLHF, and the cause was my reward model's L2 setting, not the method. The policy is a table, there is no PPO and no language model.

What you will use daily at work

Practice What it looks like
Collect comparisons, not scores And measure labeler agreement.
Hold out a check the reward model never saw Human review or a verifier.
Use best-of-N first It needs no training, only N times the inference.
Run a regression eval before and after Safety, tool use, hallucination, format.

Draft: confirm which preference-tuning methods your provider or library supports and their data format.

What you must be able to say (one page)

  • What it is. After supervised fine-tuning (SFT), preference tuning pushes the model toward outputs people prefer, using comparisons (this response beat that one) instead of single gold answers. The main methods are RLHF (used for InstructGPT and Llama 2), DPO (used for Llama 3) and RLAIF (preferences from AI feedback). Meta moved from RLHF in Llama 2 to DPO in Llama 3, which is a simpler training loop.
  • RLHF has two parts: (1) train a reward model that scores (prompt, response); (2) optimise the model to produce responses the reward model scores highly, usually with PPO. I did not implement PPO; my toy uses the exact policy gradient on a small softmax policy.
  • Why comparisons. Pointwise scores vary between labelers and over time; comparing two responses is easier. Figures I recall from published reports: about $3.50 per comparison against $25 per written response (Llama 2 report) and labeler agreement of about 73 percent (InstructGPT). I did not re-check them. A ranking of three responses gives three pairs.
  • The reward-model loss: for a prompt x, winner y_w and loser y_l, minimise -log sigmoid(r(x, y_w) - r(x, y_l)). This is the Bradley-Terry model: P(w beats l) = sigmoid(difference in rewards). Only differences matter.
  • The objective RLHF optimises: maximise expected reward minus beta times the KL divergence from the reference (SFT) model. The maximiser has a **closed form: pi(y|x) proportional to pi_ref(y|x) x exp(r(x,y) / beta) In my toy, exact-gradient RLHF wh the true reward matched it: largest probability difference 8.3e-3, 2.0e-3, 1e-4 and 6.5e-8** at beta 0.1, 0.3, 1 and 3 (6,000 steps; the small-beta runs had not fully converged).
  • beta is a leash. In my toy (3,000 clean pairs, mean of 10 random worlds, reference reward 0.80): the perfect KL-regularised policy gets true reward 2.16, 2.16, 2.08, 1.69 and 1.21 at beta 0.05, 0.1, 0.3, 1 and 3, with KL from the reference of 1.52, 1.43, 1.05, 0.36 and 0.06. Small beta gains more reward but moves far from the reference and trusts the reward more.
  • DPO skips the reward model. The loss is -log sigmoid(beta x [(log pi(w) - log ref(w)) - (log pi(l) - log ref(l))]). It comes from the closed form: the optimal policy implies a reward beta x log(pi / pi_ref) (up to a per-prompt constant), so you can fit the policy directly. In my toy the softmax normalisers cancel in the gradient, so the update on a pair is simply "raise the winner's logit, lower the loser's, weighted by sigmoid(-z)". Its implicit reward correlated 0.92 with the true reward (the fitted reward model: 0.93).
  • RLHF and DPO matched in my toy, once the reward model was fitted fairly. True reward, RLHF / DPO: beta 0.3: 2.06 / 2.07; beta 1: 1.65 / 1.68; with 100 pairs 1.65 / 1.65; with 1,000 pairs 2.03 / 2.04. My first run showed DPO well ahead (2.07 against 1.66 at 3,000 pairs) and the cause was my reward model's L2 regularisation (it shrank rewards, so RLHF behaved as if beta were larger). With L2 0.1, 0.01, 0.001 and 0.0001 the RLHF result was 1.01, 1.65, 2.00 and 2.03. Lesson: in RLHF the reward model's scale and calibration matter; a comparison between methods can be an artefact of such a setting.
  • Noisy labels hurt both. With 1,000 pairs and 0, 10, 20, 30 and 40 percent of labels flipped, DPO reached 2.04, 1.93, 1.69, 1.28 and 0.83. A reward-model pairwise accuracy ceiling is below 100 percent even for the true reward: in my toy, 0.77 (learned model with 300 pairs: 0.71), because Bradley-Terry labels are random.
  • Over-optimisation (Goodhart). A learned reward is a proxy. With 100 pairs, 30 percent label noise and a small beta (0.02), 60 percent of 20 runs ended worse than the reference on the true reward. In the worst run the learned reward rose from 0.20 to 3.24 while the true reward fell from 0.85 to -0.39. Remedies: a larger beta, more and cleaner data, a held-out check, reward-model ensembles (module 4 returns to this).
  • Best-of-N is the cheap alternative (sample several responses and keep the one the reward model scores highest; some production teams reportedly stop there and never run RL). With a reward model fitted on 300 pairs, true reward by N: 0.80, 1.28, 1.60, 1.76, 1.83 and 1.81 for N = 1, 2, 4, 8, 16, 32. It plateaus, and can slip, because the reward model's errors are exploited as N grows. It costs N times the inference.
  • Honest warnings. There are debates on why RLHF and DPO work. Overall human preference can improve while a specific failure such as hallucination does not, so measure each one. Perplexity can rise after post-training. The goal itself assumes a universal human preference exists.
  • What to say in an interview: "Preference tuning optimises a policy toward human comparisons while staying near a reference model. RLHF fits a Bradley-Terry reward model and then runs RL with a KL penalty; DPO derives the same optimum in closed form and trains on the pairs directly, without a reward model or sampling loop. beta sets how far the policy may move. I watch for reward over-optimisation, noisy labels and calibration, and I evaluate on held-out human or verifier judgements, not on the reward model."
  • Honest limits: a tabular softmax policy (6 prompts, 10 responses), random true rewards, simulated labelers; no language model, no PPO, no sampling noise in the policy updates.