Preference tuning: RLHF and DPO: Training stages

Choosing the better of two answers is easy, and it can be used to train the model. Includes a DPO loss you can compute by hand.

After SFT the model answers, but it might answer rudely, unsafely or at needless length. Writing the perfect answer is hard. Choosing the better of two is easy. Preference tuning uses that.

Preference data

Triples of a prompt, a chosen (better) response and a rejected (worse) response, labelled by people or by another model.

RLHF (reinforcement learning from human feedback)

First train a reward model to predict which of two answers a human prefers. Then adjust the language model, by a trial-and-reward method called reinforcement learning, to produce answers that score high with the reward model, while staying close to the SFT model so it does not drift into nonsense.

The reward model rests on a simple formula. If answer A has score rA and B has rB, the probability that a person prefers A is sigmoid(rA - rB). The sigmoid function squeezes any number into the range 0 to 1:

preference_probability(2.0, 0.0)   # 0.88: a 2-point reward gap means about 88% preference

DPO (direct preference optimisation)

A simpler alternative: skip the reward model and the reinforcement-learning loop, and train directly on the pairs. The model is rewarded for making the chosen answer more likely than a frozen reference copy does, and the rejected answer less likely.

def dpo_loss(logp_chosen, logp_rejected, ref_logp_chosen, ref_logp_rejected, beta=0.1):
    margin = beta * ((logp_chosen - ref_logp_chosen) - (logp_rejected - ref_logp_rejected))
    return -math.log(sigmoid(margin))

logp means the log-probability the model gives to a whole answer (a negative number; closer to 0 means more likely). The reference is a frozen copy of the model before tuning. A loss is the number training tries to make smaller. Here are three real cases:

Situation	Inputs (chosen, rejected, ref chosen, ref rejected)	Loss
Policy equals reference (nothing learned yet)	-5, -7, -5, -7	0.693 (that is ln 2)
Policy likes chosen more and rejected less than reference	-4, -9, -5, -7	0.554 (lower: good)
Policy moved the wrong way	-6, -5, -5, -7	0.854 (higher: bad)

Check the middle row by hand: the chosen answer rose by 1 (from -5 to -4), the rejected fell by 2 (from -7 to -9), so the margin is 0.1 x (1 - (-2)) = 0.3, and the loss is -ln(sigmoid(0.3)) = 0.554.

beta controls how strongly the model is held near the reference. For one pair (chosen -10, rejected -12, reference -11 and -11) I get these losses when the policy moved the right way and the wrong way:

beta	Loss if policy moved right	Loss if policy moved wrong
0.01	0.683	0.703
0.1	0.598	0.798
1	0.127	2.127
5	0.000	10.000

A larger beta makes the loss react much more strongly to how far the policy has moved from the reference.

A model tuned this way tends to answer more helpfully, but also to agree with the user (module 7 covers this "sycophancy"), because people often prefer agreeable answers.

Check yourself

Why is the loss 0.693 when the policy equals the reference?
What is the reference model for?