A model improves from getting 80% of tokens right to 90%. That sounds gradual. Now require every token in a 20-token answer to be correct: its expected exact match jumps from about 1% to 12%. The model changed smoothly; the ruler bent the curve.
Schaeffer, Miranda, and Koyejo argue that some reported emergent abilities—capabilities said to appear sharply and unpredictably only at large scale—are this kind of measurement effect. Their claim is about fixed model outputs scored differently, not about denying that bigger models can acquire new abilities.
One exponent makes the cliff
Let p be the chance of producing one correct token. Under the paper's simplifying independence assumption, all L tokens are correct with probability pᴸ. Expected token errors instead equalL(1−p). One changes geometrically with answer length; the other changes linearly.
def score_pair(p_token, length):
exact_match = p_token ** length
expected_errors = length * (1 - p_token)
return exact_match, expected_errorsThe TypeScript implementation above drives every illustrative chart below; Python and C++ are faithful translations.
A smooth scaling law can look like a phase change
The paper's toy model assumes per-token cross-entropy falls as a power law with parameter count. Cross-entropy is the penalty for assigning too little probability to the correct token. Turning that smooth loss into per-token correctness remains smooth; raising it to the twentieth power creates the apparent takeoff.
The benchmark can be too small to see progress
A test set of N answers measures accuracy only in steps of1/N. If a model's true exact-match probability is below that step, observing zero successes is ordinary sampling noise, not evidence of zero ability. Long answers make this worse because pᴸ becomes tiny.
What the paper actually found
| Analysis | Paper-reported finding | Why it matters |
|---|---|---|
| Two dominant metrics | >92% of annotated BIG-Bench emergent abilities | Exact string match or multiple-choice grade |
| Preferred BIG-Bench metrics | 39 examined; at most 5 showed emergence | The effect clustered by metric, not only task |
| GPT arithmetic re-evaluation | All four model sizes above chance with more data | The apparent zero plateau was resolution-limited |
Multiple-choice grading creates a second discontinuity: whichever option has the largest probability gets 1, all others get 0. A continuous Brier score instead measures the squared error of the full probability forecast. The paper found that changing to continuous or approximately linear metrics often removed the apparent emergence while retaining graded improvement.
The caveat cuts both ways
So exact match is not a bad metric when exactness is the real requirement. It is a bad microscope for forecasting smooth scaling unless it is paired with token-level scores, confidence-sensitive metrics, enough test examples, and dense coverage of model sizes.
What I would probe next
- Report exact match and token error on the same fixed outputs.
- Attach confidence intervals that expose the zero-success uncertainty.
- Pre-register metrics before searching many tasks for dramatic cliffs.
The measurement story sits above the model itself. To inspect the attention computation whose outputs these benchmarks score, poke at the interactive transformer page.
References
- Rylan Schaeffer, Brando Miranda, Sanmi Koyejo (2023). Are Emergent Abilities of Large Language Models a Mirage?. NeurIPS 2023
- Jason Wei et al. (2022). Emergent Abilities of Large Language Models. Transactions on Machine Learning Research