The 4-bit model that passes the average—and forgets the latest value

Imagine telling an assistant that your meeting moved from 2 pm to 3 pm, then to 4 pm. The answer is unambiguous and sits immediately above the question. Yet language models often retrieve an older value. This is proactive interference: earlier information obstructs recall of a later update bound to the same key.

Shahrabi-Farahani and Rahmati ask whether post-training quantization makes that failure worse. Post-training quantization compresses already-trained weights for inference; their INT4 arm uses bitsandbytes NF4, a 4-bit format, with double quantization. Across Qwen, Mistral, and Phi, the aggregate score barely tells a story. The targeted overwrite probe does: at each model's informative high-interference level, INT4 loses 6.6 to 12.7 percentage points against FP16.

Each prompt chronologically rebinds one subject–attribute pair: “Ava's favorite color is red,” then blue, then green, and so on. The model must output only the final value. The interference level k is the number of updates. Four semantic attributes—mood, favorite color, favorite animal, and occupation—draw distractors from one meaning-rich category. Temperature, stock price, and page count form a numeric control.

This is not the usual “needle in a haystack” test. There is no single fact buried among unrelated text; there are several once-valid answers for the same question. Prompts contain at most a few hundred tokens, generation is greedy, and the newest update is closest to the query. A wrong earlier value is evidence of interference, not ambiguity.

For a fixed model and seed, every precision sees the same subject, distractors, order, and gold answer. That makes the comparison paired: the useful question is not only how many items each model got right, but which exact items changed outcome. The authors verified zero pairing mismatches across 23,075 unique trials.

McNemar's exact test conditions on disagreements. Under the null that neither precision is favored, each discordant item is a fair coin: FP16-only or INT4-only. Qwen has 40 in the first direction and only 2 in the second, giving odds 20:1 and an exact two-sided p-value of 4.1e-10. An unpaired accuracy test discards which prompt flipped and is therefore less powerful.

def exact_mcnemar(reference_only, compressed_only):
    discordant = reference_only + compressed_only
    if discordant == 0:
        return 1.0
    tail = min(reference_only, compressed_only)
    probability = 0.0
    for successes in range(tail + 1):
        probability += comb(discordant, successes) * 0.5 ** discordant
    return min(1.0, 2 * probability)

The TypeScript version shown above is the same core routine used for the displayed p-value; Python and C++ are line-for-line translations. For 40 versus 2 discordances, it doubles the lower Binomial(42, 0.5) tail. The full post never simulates paper trials or invents model outputs: every empirical accuracy is quoted from the paper.

A mixed-effects logistic regression uses all interference levels and controls for model, seed, and attribute. Its key interaction is INT4 × log₂(k). For semantic values the coefficient is −0.046: each doubling of overwrites multiplies INT4's odds relative to FP16 by exp(−0.046). For numeric values the coefficient is +0.053. That opposite sign is difficult to explain as generic quantization noise.

Numeric accuracy still declines with more overwrites, so the control is not trivially easy. But at k = 32, 64, and 96 its FP16–INT4 differences are individually non-significant and not consistently signed. Token length is an imperfectly matched design detail—the numeric pool was not prefiltered to single tokens—but the authors found a flat token-length mix across levels and retained the sign reversal within the dominant token-length stratum.

The authors replayed their seeded trial generator and checked whether each error matched one of the earlier overwritten values. Between 99.8% and 99.9% of errors did. The model is not emitting random nonsense; it is selecting a fact that used to be correct for the same key.

The intrusion rate rises from 21.5% at FP16 to 23.2% at INT8 and 24.6% at INT4. Mean intrusion recency barely changes—13.8, 13.2, and 13.2 positions back—so quantization changes how often a stale candidate wins, not which age of candidate wins. This supports a representational-coarsening interpretation, but does not prove it: the paper does not probe residual-stream or attention representations directly.

The Interference Endurance Score (IES) integrates accuracy over a log₂ interference axis and divides by the tested range. Reported INT4 losses are 1.82% for Qwen, 5.09% for Mistral, and 5.16% for Phi. Qwen's 1.82% looks mild because its curves remain at or above 96% through k = 32; averaging those saturated points with the k = 64 collapse dilutes the local 12.7-point failure.

The appendix adds an unusually candid warning: word attributes have different tokenizer-filtered vocabulary caps, and the authors did not independently verify how their IES implementation pools levels unsupported by every attribute. IES is useful directional evidence, not a substitute for the predeclared per-level paired analysis.

• Build paired prompts from your actual update-heavy workflows.
• Stratify by semantic similarity, overwrite count, attribute, and model family.
• Report worst relevant slices beside aggregate accuracy and normalized area.
• Classify errors as stale real values versus unrelated outputs.
• Compare quantizer families, not only bit widths.
• Test 8k+ contexts, alternative decoding, activation, and KV-cache quantization.

This is a failure of a deployed Transformer's in-context state tracking, not training-time catastrophic forgetting. It also complements the earlier BitNet walkthrough: BitNet asks what native low-bit training really stores; this paper asks which behavior a frozen model loses when a deployment library compresses its backbone after training.

“Aggregate performance is preserved” is a statement about an average over tasks and regimes. It cannot certify a narrow capability that matters to your product. Here, 69,225 controlled trials reveal a failure concentrated in semantically confusable overwrites, while the pooled score remains reassuring. Before buying memory savings, test the memory behavior you depend on.