When a huge hidden-state error changes almost nothing

Two sequence models can update their hidden states in radically different ways and still make nearly identical predictions. Wen, Liu, Zhou, Zeng, and Su prove that this is not merely possible: in a finite-state hidden Markov model, one explicit update gap grows without bound while the decoded predictive KL divergence goes to zero.

A hidden Markov model (HMM) assumes an unobserved mode that changes over time and emits noisy observations. A filter maintains a probability for each mode. The surprising part is where the metric is measured: logits can drift far apart after an update while softmax maps both vectors into almost the same confident categorical belief.

Adding one constant to every logit changes nothing after softmax. The paper therefore centers every vector by subtracting its mean and measures differences through pairwise log odds. For K latent modes, the meaningful state lives in K − 1 dimensions.

The HMM stays in its current mode with probability 1 − q and switches uniformly to another mode with probability q. Rare switching makes the natural confidence scale L_K(q) = log((K − 1)(1 − q)/q) grow. Exact Bayesian mixing and the paper's radial tanh update both saturate on this scale, but they approach the boundary differently.

def exact_update(h, q):
    return 2 * atanh((1 - 2*q) * tanh(h / 2))

def radial_update(h, q):
    L = log((1 - q) / q)
    return L * tanh((1 - 2*q) * h / L)

def bernoulli_kl(exact, approx):
    p, r = sigmoid(exact), sigmoid(approx)
    return p*log(p/r) + (1-p)*log((1-p)/(1-r))

The binary formulas above expose the mechanism without a machine-learning library. Python is the default; TypeScript and C++ are line-for-line translations. The figures call the tested K-state core implementation, not pasted chart values.

For each fixed K, the paper chooses centered logits z = L(1/4, −1/4, 0, …, 0). As q falls, L rises. The quotient distance compares every pairwise log-odds change from exact and radial mixing, ignoring the common shift that softmax cannot see.

Imagine two thermometers that disagree by ten degrees but both read far above the boiling point. For the decision “is it boiling?”, their disagreement barely matters. Here both filters become highly confident in the same mode. Softmax curvature becomes tiny inside that common confidence cone, suppressing a logit-space discrepancy that remains large in absolute units.

The paper formalizes this with a bound proportional to R²e^−m, where R bounds the range of the logit discrepancy and m is the shared winning margin. Polynomial growth in R can lose to exponential decay in decoder sensitivity. This is the missing link between representation distance and prediction loss.

This does not invalidate the theorem: the path-level convergence proof controls the full stationary expectation, including rare paths. It does change what the simulation establishes. The displayed K = 2, 4, 8 subset was also chosen after a broader grid was evaluated; four of five original configurations passed the directional criterion, while one skewed K = 8 setting had an unresolved log-KL slope.

The learned-model evidence is narrower still. End-to-end training met the authors' transfer criterion at four of six q values rather than the prespecified five. At horizons up to 2,048, the analytic tanh recurrence remained stable, but one- and two-block Mamba variants did not establish the same mechanism. A proved hand-designed update is not evidence that a named architecture will discover it.

If an internal-state metric predicts downstream failure, test the whole bridge: how often separating states occur under the deployment path law, how sensitive the decoder is there, and whether the chosen loss exposes the difference. Report task loss beside representation distance, stratify rare transitions, reverse asymmetric divergences, and extend evaluation far beyond the training horizon.

The paper fixes K before q approaches zero, assumes distinct scalar Gaussian means, symmetric switching, stationary initialization, and a logarithmic horizon. It proves a counterexample, not a general license for pruning or compression. For the decoder mechanics, continue with the site's logistic-regression walkthrough; for the sequence-model connection, see the transformer walkthrough.