The training example that moved the weights—and vanished from the loss

Suppose you want to know whether one paragraph mattered to a language model. The clean experiment sounds simple: train one model with that paragraph and an otherwise identical model without it. In practice, that means duplicating an entire pretraining run while controlling every other source of variation.

Speck and Shepard actually ran that counterfactual. One row was enough for the model to predict the inserted passage better 50 steps later. By the end, that advantage was no longer detectable—yet the final weights remained far from their uninjected twins. “Did this example matter?” has incompatible answers unless the question names both a time and a metric.

The authors trained 32 GPT-2 models with 124 million parameters: three injection conditions plus an uninjected twin, each at eight seeds. Within a seed, all four models were bit-for-bit identical through step 199. At step 200—the peak learning rate—one 194-token passage replaced part of one row in a 256-row batch. Training then continued deterministically to step 9,535.

The two fluent passages had almost the same full-batch gradient-delta norm: 0.010698 versus 0.010683, a 0.14% difference. One subject appeared four times elsewhere in the corpus; the other was fabricated. A random-character arm tested a different gradient direction. The design isolates one data intervention much more tightly than an influence estimate does.

This timing issue is not cosmetic. The paper's per-step arm-versus-twin loss difference first falls, then rises by roughly two orders of magnitude near step 300, then slowly decays. A probe at step 210 would tell a different story from one at step 300 or at the endpoint.

Euclidean distance is straight-line length between two parameter vectors. It notices every coordinate change, whether or not that change alters useful predictions. The injected models finished 235.03 units from their twins; models with different seeds averaged 532.88 units apart. Normalizing like with like gives 44.1%.

At the midpoint, the normalized distance was 0.407; at the end it was 0.441. Thus 92.3% of the final displacement was already settled halfway through training, even though the passage-specific advantage later became undetectable.

Interpolation loss barrier mixes the two finished weight vectors at 21 evenly spaced points. At each mixture, it measures held-out cross-entropy and subtracts the straight chord between the endpoint losses. The largest rise above that chord is the barrier.

def interpolation_barrier(left, right, loss, points=21):
    left_loss = loss(left)
    right_loss = loss(right)
    barrier = 0.0
    for i in range(points):
        alpha = i / (points - 1)
        weights = [(1 - alpha) * a + alpha * b for a, b in zip(left, right)]
        chord = (1 - alpha) * left_loss + alpha * right_loss
        barrier = max(barrier, loss(weights) - chord)
    return barrier

The injection-to-twin barrier was 0.1479, versus 4.9303 between seeds: 3.0% of its seed floor. A tiny from-scratch example shows why distance and barrier need not agree. Under the flat-basin loss L([x, y]) = x², moving from [0, 0] to [0, 10] travels 10 parameter units while the entire interpolation path has loss zero, so its barrier is 0. Large movement can stay inside one low-loss basin.

The strongest result is about operationalization: a concrete readout used as a stand-in for “influence.” Passage loss detects short-lived learning. Euclidean distance detects persistent relocation. Held-out loss, interpolation barrier, and layerwise representation similarity detect little endpoint separation. None alone exhausts every behavior the two models could differ on.

• Record influence during training, not only at the final checkpoint.
• Report power or an MDE beside every null result.
• Pair runs within seed so ordinary seed variance does not swamp one-row effects.
• Use behavioral and geometric readouts, and state what each one is blind to.
• Do not infer that content never matters from two passages at 124M-parameter scale.

The attested passage was also true while the fabricated one was false, so the study cannot separate attestation from truth. The passage probe was exploratory, the barrier may miss a narrow peak between grid points, and the conclusions come from one architecture, corpus, injection time, and scale. For the underlying model family, visit the Transformer and neural-network pages.