The model got better by moving its answer into a box

Suppose a base language model writes “The answer is B,” while its reinforcement-learned successor writes exactly the same verdict as\\boxed{B}. If the benchmark reads only the box, the second model gets the point and the first gets zero. Nothing about the answer improved; only its address changed.

An answer extractor converts free-form model text into a short prediction such as A, B, C, or D. The toy set below is deliberately small and exact. Both models state two correct answers, one wrong answer, and one unusable or wrong answer. A box-only parser nevertheless reports a 50-point gain because it cannot see the base model's prose.

The paper formalizes the same comparison at scale. Let the reported gain be the RL-minus-base accuracy difference under the original harness extractor. Let the fixed gain be the difference after a single broader extractor reads both sets of stored outputs. Their difference is the format component; dividing by the reported gain gives the format share.

def format_share(reported_gain, fixed_gain):
    if reported_gain <= 0:
        raise ValueError("reported gain must be positive")
    return (reported_gain - fixed_gain) / reported_gain

share = format_share(22.5, 4.0)

Across 46 eligible comparisons on MMLU, MMLU-Pro, and ARC-Challenge, the harness extractors report an average 22.5-point gain. The fixed extractor finds 4.0 points on the same generated outputs. The remaining 18.5 points are attributed to format, so 18.5 / 22.5 = 82.2%.

The mechanism is visible directly. Among outputs where the fixed extractor reads an answer, 26.6% of base-model responses contain a box, compared with 58.3% of RL-model responses. Most of the remaining base answers already appear in ordinary phrases such as “Answer:” or “the answer is.”

The headline pool is carefully filtered: four SimpleRL pairs built from three 7B base models, three out-of-domain multiple-choice benchmarks, two prompts, and only comparisons with a clearly positive reported gain and adequate extraction. Three Qwen pairs contribute 98.5% of the summed reported gain. Changing the pooling rule lowers the share to 75%; independent LLM readers estimate 77%.

That reversal is the post's central restraint. The artifact is real, reproducible, and large in the 7B SimpleRL setting. It is not a law of RLVR, a proof that reasoning never improved, or a correction factor to paste onto every benchmark. Model scale, base-model habits, prompting, training recipe, and extractor reach all interact.

Save raw responses and publish the exact extraction code. Score every checkpoint with at least two readers: the release harness and a fixed, model-agnostic reader. Report extraction rate beside accuracy, manually adjudicate disagreements without revealing model identity, and run a prompt that does not reward the target wrapper. For multiple choice, direct answer matching or constrained decoding can remove much of the parser surface—but may measure a different deployment behavior.

Next, test whether conclusions survive across model families and sizes, especially when the base model already follows formatting instructions. Separate in-domain math, out-of-domain multiple choice, and free-form tasks rather than pooling them. Finally, measure semantic correctness before asking whether RL changed the style in which correctness is expressed.

RLVR updates a neural language model, so the mechanism connects naturally to the site's interactive Transformer walkthrough. The broader lesson applies to every classifier pipeline: the last parsing function is part of the model you evaluate.