The drift score that knows when it cannot know

Your fraud model starts missing a new batch. Did customer behavior move into a different part of feature space, or did the relationship between features and fraud actually change? The first is covariate shift: P(X) changes. The second is mechanism or concept drift: P(Y | X) changes. They demand different responses, but most drift scores mix them together.

Oda's Conditional Jensen–Shannon Discrepancy (CJSD) asks a crisp question: after the input X is known, how much extra information does the label Y reveal about which dataset the row came from? Two ordinary probabilistic classifiers estimate the answer. The important catch is in the paper's anti-coupling inequality: if X already gives the dataset away, there is almost no information left for Y to add. A tiny CJSD can mean “same mechanism,” or “no overlap, so we cannot tell.”

Pool equally many examples from datasets A and B and attach a dataset indicator Z. Train one discriminator T₁ to predict Z from X. Train a second T₂ to predict Z from the pair (X, Y). A discriminator here is just a probabilistic binary classifier; a logistic regression works for simple boundaries, while a tree or boosted model can represent nonlinear ones.

If Y carries no dataset-specific information once X is fixed, T₂ cannot improve on T₁. If the labeling rule changed, the joint classifier should predict the dataset better. With cross-entropy (log loss), the population identity is

CJSD = I(Z; Y | X) = CE(Z | X) − CE(Z | X, Y).

Conditional mutual information I(Z; Y | X) is the information Y adds about Z after X is already known. The subtraction is evaluated on held-out folds, so neither classifier scores points it trained on. The two losses are paired row by row, which also gives a useful variance estimate.

def held_out_cjs(rows, eps=1e-6):
    total = 0.0
    for z, p_x, p_xy in rows:
        p_x = min(1 - eps, max(eps, p_x))
        p_xy = min(1 - eps, max(eps, p_xy))
        loss_x = -log(p_x if z == 0 else 1 - p_x)
        loss_xy = -log(p_xy if z == 0 else 1 - p_xy)
        total += loss_x - loss_xy
    return total / len(rows)

The from-scratch function above is the paper's held-out estimator. On a two-row sanity check where T₁ is uninformative at 0.5 and T₂ assigns 0.8 to the correct dataset, it returns 0.470 nats, exactly ln(1.6). In a real audit, the probabilities must come from cross-fitted models, and the two source samples are balanced so P(Z) = 1/2.

The total Jensen–Shannon difference between the two joint datasets is I(Z; X, Y). The mutual information chain rule splits it without a residual:

I(Z; X, Y) = I(Z; X) + I(Z; Y | X).

The first term is a covariate axis: how identifiable the dataset is from inputs alone. The second is CJSD, the functional axis. Under pure covariate shift, the conditional label distributions match at every observable X, so CJSD is exactly zero no matter how far P(X) moves. Unlike exchanging two task-specific predictors, nothing is asked to extrapolate off its training support.

That normalization gives CJSD an intuitive ruler. If two deterministic labelers disagree on 25% of an otherwise shared input distribution, the score is 0.25 ln 2, and the highlighted bar reads 25%. This is an identity, not a fit to the paper's empirical datasets.

Because Z is binary and balanced, all the information about it fits inside ln 2 nats—one bit. Therefore

CJSD ≤ ln 2 − I(Z; X).

The two axes are additive, not independent coordinates. As the input distributions become easy to separate, the maximum observable functional signal collapses. This is not a defect a larger classifier can repair; it is an identifiability boundary.

Let ε₁ be T₁'s excess log loss above the best possible X-only discriminator, and ε₂ the corresponding excess for T₂. The fitted population loss gap is CJSD + ε₁ − ε₂. Its bias has no guaranteed sign. The paper's one-sided result is sharper: an inflated alarm is bounded only by ε₁, while an overly reassuring clearance is bounded only by ε₂.

In the main experiments, matched architectures tended to make ε₂ larger and shrink the estimate toward zero. That was an empirical regularity, not a theorem. The appendix constructs a pure scale shift where linear logistic T₁ cannot represent its quadratic target: ε₁ = 0.1492 exceeds ε₂ = 0.1255, creating a false +0.0237 functional signal even though true CJSD is zero. A flexible discriminator removes the artifact.

• Balance A and B so the ln 2 normalization remains exact.
• Cross-fit both discriminators; do not score their training rows.
• Clip probabilities before log loss, and report the clipping threshold.
• Audit calibration and capacity separately for T₁ and T₂.
• Read CJSD beside the covariate axis and an overlap diagnostic.

Under a pure support shift, the paper's exchange score rose from 0.001 to 0.346 while estimated CJSD stayed between −0.002 and 0.002. In the dimensionality study, the kNN plug-in dropped to 0.72 AUC at d = 64 and reached chance from d = 256. With 12,000 samples per side, a histogram-gradient-boosting discriminator reached 1.00 across the tested grid through d = 512. That comparison supports a practical claim about the estimator, not a claim that CJSD discovered a different population quantity than conditional mutual information.

Before using CJSD as a production gate, sweep discriminator families, folds, probability clipping, and source balance. Plot the score against I(Z; X); challenge every low CJSD with deliberately reduced overlap; and calibrate thresholds on matched null pairs. Also remember what the paper does not claim: CJSD tests no-adaptation reuse, not whether fine-tuning will transfer well. Differences outside shared support are irrelevant to the former and may be decisive for the latter.