The causal feature score that moves when the token moves

A sparse autoencoder, or SAE, turns a language model's dense internal activation into a small set of active features. Researchers inspect those features to give names to what the model may be computing: a language, a topic, a syntactic pattern, perhaps a factual concept.

To test whether one feature matters, they remove its contribution at a token and measure how much the model's next-token distribution changes. That sounds causal. Noël's paper finds that the number often describes the chosen token as much as the feature—and the SAE being evaluated usually chooses that token itself.

At position t, a feature has activation a and decoder direction d. Zero ablation changes the residual stream from h to h − ad. The paper scores the intervention with KL divergence between the original and ablated next-token distributions. A larger value means the intervention changed the prediction more.

But the same feature fires at many tokens—a median of 103 positions for the released dictionaries studied. The common convention selects the token where that SAE's activation is largest. Fit another SAE and its activation ranking changes, so the experiment moves.

Our deterministic example has four versions of one feature across five positions. Each arm selects a different activation maximum, so pairwise position agreement is 0.0%. The underlying ablation effects are almost identical between arms at any fixed position; only the position itself changes the effect strongly.

The paper's arm-symmetric rule keeps positions where the feature is active in every arm, then chooses the position whose minimum activation across arms is largest. No single dictionary becomes the reference. The critical change is simpler than the ranking rule: after choosing once, evaluate every dictionary at that same position.

def shared_position(activations_by_arm):
    positions = len(activations_by_arm[0])
    best_position, best_minimum = -1, float("-inf")
    for position in range(positions):
        minimum = min(arm[position] for arm in activations_by_arm)
        if minimum > 0 and minimum > best_minimum:
            best_position, best_minimum = position, minimum
    if best_position < 0:
        raise ValueError("no position is active in every arm")
    return best_position

The live charts run the validated TypeScript implementation; Python and C++ reproduce the same loop. The paper also tested a less selective shared rule based on the union of each arm's top positions. The variance collapse remained, although Gemma-3's controlled estimate rose from 2.4% to 6.3%. The exact shared-position rule still matters.

The reliability coefficient Eρ² asks whether a feature ranking would survive a new set of SAE arms. It is latent variance divided by latent variance plus the feature-by-arm interaction and residual variance per arm. Reliability rises automatically when the interaction term falls; the paper correctly treats that rise as arithmetic, not a second independent discovery.

The filter was applied symmetrically to both measurement modes, so it cannot manufacture the within-subpopulation collapse. In fact, excluded features carried more interaction variance, suggesting the reported confound is a lower bound. But external validity remains narrower than the headline: two Gemma models, one architecture family, shared-initialization SAEs trained on 12 million tokens, and zero-ablation magnitude metrics only.

The paper is unusually candid about corpus sensitivity. Five candidate claims were withdrawn and two mechanisms falsified when evaluation grew from 96 to 384 sequences. Even statistically significant, replicated effects with plausible stories did not survive more contexts. Its surviving position result did—and disagreement worsened as the corpus grew.

• Report the token position and how many positions were measured per feature.
• Use one shared position set when comparing dictionaries.
• Report effect per unit perturbation norm, not only raw KL magnitude.
• Exclude special tokens before pooling and verify the expected reconstruction quality.
• Show a distribution over positions or contexts instead of compressing it to one scalar.

SAEs operate on a Transformer's residual stream. Explore how those representations are produced on the interactive Transformer page.