The RL-free alignment loss that still hides a reward model

Suppose people compare two answers and choose the better one. Traditional reinforcement learning from human feedback, or RLHF, first trains a model to predict those choices, then uses reinforcement learning to make a language model chase that predicted reward.

Direct Preference Optimization (DPO) collapses those two stages into one familiar operation: binary classification. Show the model a preferred answer and a rejected answer, then teach one log-probability ratio to beat another. No rollout loop or separately deployed reward network is needed.

A policy is simply the model's probability distribution over possible answers. DPO compares the trained policy with a frozenreference policy, usually the supervised fine-tuned model it started from. For one answer, the implicit reward is β log(π / πref). The temperature-like constant β controls how strongly deviations from the reference count.

For a preferred/rejected pair, any prompt-only normalization cancels. What remains is the improvement in log odds over the reference. A sigmoid turns that margin into the probability that the pair is correctly ordered; negative log probability gives an ordinary logistic loss.

def dpo_pair(logp_win, logp_lose, logr_win, logr_lose, beta):
    margin = beta * ((logp_win - logp_lose) - (logr_win - logr_lose))
    probability = 1 / (1 + exp(-margin))
    loss = -log(probability)
    weight = 1 - probability
    return margin, probability, loss, weight

The site runs the numerically stable TypeScript core; Python and C++ above are faithful translations of the same four quantities.

Here is the catch. Imagine the trained policy gives the preferred answer 80% probability. That sounds decisive. But if the reference already gave it 90%, the trained policy has moved away from the preference. Our reconstruction assigns the pair only 30.8%probability of being correctly ordered and a training weight of 69.2%.

The gradient carries a dynamic weight: one minus the model's current pair probability. Wrongly ordered or barely separated pairs receive large updates; already-confident correct pairs fade. The paper reports that a naive probability-ratio objective without this weighting can make the language model degenerate.

A larger β makes the same relative log-odds change look more decisive inside the classifier. Operationally, however, β is tied to the KL penalty in the underlying RLHF objective, so comparing beta values without the reference and data distribution is incomplete.

For open-ended tasks there is no ground-truth reward, so the paper used GPT-4 to judge most wins. Its human study supported that proxy, but also exposed sensitivity: a concise judging prompt gave DPO a 54% win rate, a simpler prompt gave 47%, and humans gave 58% on the sampled comparison. The metric is another learned preference system, not an objective ruler.

• Swap the reference model while holding every preference pair fixed.
• Inject contradictory annotators and test the Bradley-Terry fit.
• Audit results with judges that prefer different lengths and styles.

To inspect the sigmoid classifier at the heart of DPO, poke at the interactive logistic regression page. For the sequence probabilities it compares, visit the transformer page.