The attention map looks sparse. The cache still needs 61% of its tokens.

Long-context language models keep a key–value cache: the keys and values for earlier tokens that every new query may attend to. The cache makes decoding fast, but it grows with the prompt. A tempting compression rule is to inspect an attention heat map, keep the few bright cells, and evict the dim tail.

Yang and Li show why that visual argument skips three contracts. A few conspicuous weights need not contain most of the probability mass. Retained mass does not determine the output without the omitted value vectors. And reproducing one attention output does not guarantee that a later query—or the task—needs the same state. Their CertKV method turns this taxonomy into a competitive training-free compressor, but the paper is most valuable for spelling out what its own evidence does not certify.

Attention first turns logits—query–key similarity scores—into softmax weights, then returns a weighted sum of value vectors. The paper separates four quantities: threshold count, mass coverage, output fidelity, and task utility. Each arrow needs new information:

1. A threshold count needs the full ranked score-gap profile to imply mass coverage.
2. Coverage needs omitted values to imply output fidelity.
3. One output needs future-query and task structure to imply useful retained state.

Let G(j) be the gap between the largest logit and rank j. If G(j) grows faster than log j, exponentiated tail weights are summable and a fixed budget can cover a target mass. Below that unit-slope boundary, the budget grows with context length. The all-rank condition matters: the paper's 4K fit missed the measured 128K coverage requirement by more than 1.5×.

For a realized attention row, split indices into a kept set S and its tail. The exact output is simply the kept weighted contribution plus the omitted weighted value sum. That tail sum is the unique additive correction. Omitted mass alone only bounds error: equal mass can reinforce in one direction or cancel to zero.

def fixed_row(weights, values, keep):
    kept = [0.0] * len(values[0])
    tail = [0.0] * len(values[0])
    keep_set = set(keep)
    for j in range(len(weights)):
        target = kept if j in keep_set else tail
        for d in range(len(values[j])):
            target[d] += weights[j] * values[j][d]
    full = [kept[d] + tail[d] for d in range(len(kept))]
    return kept, tail, full

The code is the theorem's accounting identity without an ML library. In a hand-verified row with weights [0.5, 0.25, 0.125, 0.125], keeping two tokens retains 75% mass. The computed kept contribution is [1, 0.5], the tail is [0.5, 0.5], and their sum is the exact output [1.5, 1].

CertKV keeps high-scoring real tokens, reserves one synthetic tail-summary slot per KV head, and allocates remaining slots toward heads whose omitted values have high mass-weighted centered dispersion. A tail gets more memory when it carries more importance mass or its values are harder to summarize with one centroid. Crucially, the synthetic slot counts against the same physical budget as a real token.

On all 503 aligned LongBench-v2 examples per setting, CertKV's compressed point estimate was top-two in seven of nine model-by-ratio settings. After Holm correction across 36 paired comparisons, however, only Llama at 2× versus PyramidKV was decisive: +8.75 points, 95% CI [3.58, 13.92], adjusted p = 0.0407. That supports a competitive operating point, not a universal winner.

Physical accounting did work: at Llama 64K and a nominal 10× budget, persistent KV storage fell from 8.59 to 0.86 GB and decode peak memory from 23.11 to 16.34 GiB. The authors correctly limit the inference—this validates storage accounting, not latency gains or superiority to other methods at the same budget.

Exact attention is an estimator, not the target itself. In the paper's planted-token model, a useful token can enter top-k before full softmax gives it much mass. Renormalizing the kept set then suppresses a diffuse noisy bulk and can help a retrieval target. An aggregation target may instead need that bulk, making exact attention better.

The controlled experiment produced the predicted single crossover. The analogous observable did not predict a sign change on downstream tasks; exact and top-k stayed statistically indistinguishable on the frozen dense model. This negative transfer result is the right warning: a clean toy mechanism is a probe, not a calibrated production policy.

• Measure a target-mass budget, not only the number of weights above a threshold.
• Repeat the coverage profile at deployment context lengths before fixing cache size.
• Log omitted weighted values and compare cancellation with worst-case mass bounds.
• Charge synthetic summaries, metadata, and safeguards to the physical cache budget.
• Use paired uncertainty and multiplicity correction before declaring a method winner.
• Separate realized-row reconstruction from unseen-query and multilayer stability.

The broader Transformer lesson is that attention weights are routing coefficients, not explanations of what information may safely disappear. A sparse heat map is evidence worth investigating; it is not a systems contract.