← Blog/blog/shape-stability-reference-level-trap

The stable model that changes when you rename the baseline

An insurer retrains a pricing model after new claims arrive. Predictions barely move, yet the curves shown to actuaries may change enough to undermine trust. Côté and Côté propose measuring that structural change directly, feature by feature, instead of looking only at final premiums.

Their Mean Squared Shape function distance, or MSS, is elegant: evaluate every pre- and post-update shape on the same policies, square the differences, and average. But the score compares representations, not just functions. Before the subtraction, those representations must speak the same coordinate language.

01

Stability lives on the additive scale

A shape function maps one feature—or one pairwise interaction—to its contribution on the model's link scale. A functionally transparent model adds an intercept, univariate shapes, and at most bivariate interaction shapes before applying its link function. Generalized linear models, generalized additive models, and explainable boosting machines all fit this form.

For each feature, MSS averages the squared difference between its two shapes over the evaluation policies. Adding the feature contributions yields model-level MSS; taking its square root gives RMSS in the units of the link scale. The same terms can instead be grouped by policy, so one equation supports model-, observation-, and feature-level diagnosis.

def mss(before, after):
    n = len(next(iter(before.values())))
    by_feature = {
        name: sum((b - a) ** 2 for a, b in zip(before[name], after[name])) / n
        for name in before
    }
    return sum(by_feature.values()), by_feature

def rebase(intercept, shape, reference_index):
    offset = shape[reference_index]
    return intercept + offset, [value - offset for value in shape]
02

The baseline-category counterexample

Consider a territory shape with contributions [0, 10, 20] and intercept 100. If territory B becomes the reference, the identical model is written with intercept 110 and shape [−10, 0, 10]. The three predictions do not change. Raw RMSS is nevertheless 10. Rebasing both snapshots to territory A makes RMSS exactly 0.

raw RMSSprediction gap
Deterministic equivalent-model test. A, B, and C are alternative reference territories. RMSS changes only because the same effect is reparameterized; the maximum prediction gap remains zero. These are illustrative values, not paper-reported results.
03

Once aligned, the decomposition earns its keep

After alignment, the metric can identify where a real update changed the model. In this second toy example, territory contributes 5/3 to MSS and vehicle age contributes 1/6, for total MSS 1.833and RMSS 1.354. The chart is generated from the same pure function tested against those fractions.

Feature-level MSS for a hand-checkable aligned example. Territory changes from [0, 10, 20] to [0, 11, 18]; vehicle age changes from [−1, 0, 1] to [−1.5, 0, 1.5]. Illustrative, deterministic values.
04

What the insurance benchmark found

The authors emulate periodic updates on three public auto-insurance datasets and model claim frequency, severity, and loss cost. GLMs were usually the most stable; explainable boosting machines, or EBMs, offered a stronger performance–stability compromise. Structural instability concentrated in sparse feature tails and in interactions selected in only one of the two model versions.

Anchoring the post-update model on the pre-update predictions—called sequential modelling—cut EBM RMSS by roughly half or more in eight of nine dataset–response combinations. The lone exception increased RMSS by 3.5%. Predictive deviance changed little in most cases, although the beMTPL97 loss-cost case rose 2.145%.

Dataset / responseRMSS changeDeviance change
norauto / frequency-68.8%+0.054%
norauto / severity+3.5%+0.255%
norauto / loss cost-54.4%+0.060%
beMTPL97 / frequency-52.2%+0.099%
beMTPL97 / severity-70.6%+0.791%
beMTPL97 / loss cost-59.3%+2.145%
pg15training / frequency-56.2%+0.681%
pg15training / severity-61.4%+0.071%
pg15training / loss cost-49.5%+0.765%
Paper-reported relative changes for EBM under sequential modelling (Table 3). Negative RMSS means improved structural stability; lower deviance is better.
05

What not to conclude

Low MSS does not prove that a model is accurate, fair, or causally correct. It says that aligned additive components changed little on the chosen evaluation distribution. Sparse tails receive little empirical weight, even though the paper finds some of its largest shape divergence there. Conversely, a large MSS can coexist with small prediction changes when feature-level movements cancel.

The case study is one emulated update, not a long production history. Its public datasets are large but static; the experiment cannot establish how the strategies behave under repeated drift. XGBoost is included for prediction and performance comparisons, but depths above two introduce higher-order interactions, so the proposed structural score is not computed for it.

06

What to probe next

Repeat the update many times and report tail-weighted MSS beside the ordinary empirical score. Align categorical references, center numerical shapes on the same population, and save those conventions with every model artifact. Then inspect feature contributions before accepting a reassuring model-level average. The site's linear modelpage explains the GLM baseline; XGBoost shows the boosted-tree machinery behind the black-box comparator.

References

  1. Côté, F., & Côté, M.-P. (2026). Quantifying structural stability of transparent models for general insurance pricing. arXiv:2611.17308
  2. Breiman, L. (2001). Statistical Modeling: The Two Cultures. Statistical Science 16(3)