The reasoning model that cannot learn when every answer ties

Ask a language model the same maths question eight times. Give each correct answer a 1 and each wrong answer a 0. Group Relative Policy Optimization (GRPO) turns those eight scores into a local ranking: above-average answers are reinforced and below-average answers are suppressed. Reinforcement learning (RL) simply means improving a policy from rewards rather than from a supplied correct next token.

The striking part is what the method removes: unlike actor–critic RL, GRPO does not train a separate value model to predict how good a partial answer will be. The other sampled answers provide the baseline instead.

For rewards r₁…r₈, equation 3 subtracts the group mean and divides by the group's population standard deviation. The result is an advantage: a dimensionless measure of how surprising each answer's reward was inside this particular group.

def advantages(rewards, eps=1e-8):
    mean = sum(rewards) / len(rewards)
    variance = sum((r - mean) ** 2 for r in rewards) / len(rewards)
    std = variance ** 0.5
    if std <= eps:
        return [0.0 for _ in rewards]
    return [(r - mean) / std for r in rewards]

Normalization is scale-invariant: rewards [0, 1, 2] and [0, 10, 20] produce identical advantages. What matters is position within the group. With one correct answer among seven failures, the winner receives √7 advantage while each failure receives −1/√7. GRPO focuses strongly on rare evidence that the model can solve the prompt at all.

This creates a curriculum automatically only when the sampling temperature, model ability, and prompt difficulty produce mixed outcomes. Too easy or too hard, and the group stops teaching. Group size therefore controls more than compute cost: it changes the chance of observing a useful disagreement.

GRPO reuses sampled answers for several updates. The probability ratio says how much more or less likely the new policy makes an answer than the policy that sampled it. As in PPO, clipping limits the reward for moving that ratio beyond 1 ± ε. For a positive advantage, extra probability above 1.2 stops helping; for a negative advantage, dropping below 0.8 stops helping.

A sampled KL penalty also discourages drifting too far from a fixed reference model. KL divergence is a measure of distributional change; here it acts like a tether while the reward pushes the policy.

R1-Zero used rule-based accuracy rewards for verifiable problems and a format reward for the requested reasoning structure. The authors deliberately avoided a learned neural reward model because it could be exploited during large-scale RL—a failure commonly called reward hacking.

• Track the fraction of unanimous groups by prompt difficulty.
• Compare fixed groups of eight with adaptive resampling after a tie.
• Measure gains at equal inference budgets, not pass@1 against cons@64.

GRPO changes how a transformer is trained, not its forward pass. To poke at the attention machinery underneath the policy, visit the interactive transformer walkthrough.