The benchmark that changes one winner in three

Daily PM10 is the concentration of airborne particles smaller than ten micrometres. Monahan and da Silva forecast it one to seven days ahead at 425 European background stations and 365 U.S. monitors. Their primary candidates are persistence, SARIMA, and XGBoost: yesterday's value, a classical seasonal time-series model, and a nonlinear boosted-tree model.

The experiment changes the evaluation protocol while holding stations, forecast horizons, and candidates fixed. A static split fits once and scores a later block. Rolling-origin evaluation repeatedly advances time and refits, closer to how a deployed forecaster would be updated. Neither is a clerical detail: changing between them selects a different winner at 35.3% of European stations and 31.5% of U.S. monitors.

At each horizon, the paper orders models from lowest to highest loss and compares two complete rankings with Kendall's tau. Tau is 1 when every pair agrees, −1 under complete reversal, and values in between when only some pairs invert. The Protocol Sensitivity Score (PSS) averages 1 − tau across horizons, so it runs from 0 to 2.

def kendall_tau(a, b):
    pos = {model: i for i, model in enumerate(b)}
    concordant, discordant = 0, 0
    for i in range(len(a)):
        for j in range(i + 1, len(a)):
            if pos[a[i]] < pos[a[j]]:
                concordant += 1
            else:
                discordant += 1
    return (concordant - discordant) / (concordant + discordant)

def pss(rankings_a, rankings_b, weights):
    return sum(
        w * (1 - kendall_tau(a, b))
        for a, b, w in zip(rankings_a, rankings_b, weights)
    )

The site runs the TypeScript implementation; Python and C++ are faithful translations. With three models and seven equally weighted horizons, one pair inversion changes the average PSS by 0.095. With nine models it changes by only 0.0079. That resolution change matters when comparing candidate-set sizes.

The spatial bootstrap is important. Nearby monitors share weather and emissions, so 425 rows are not 425 independent experiments. The paper blocks European stations by country and checks alternative geographic blocks; the interval ordering remains intact.

This is the paper's most reusable lesson. If two validation schemes score different dates, a ranking can move because those dates are simply easier for a different model. Pairing turns “same task” from a verbal promise into a checkable set equality over date–horizon cells.

Expanding from three to nine models raises the European winner-swap rate from 35.3% to 60.2%, but mean PSS falls from 0.801 to 0.540. There is no contradiction. Many lower-ranked pairwise relations remain stable while close contenders trade places at the top. Kendall's tau weights every pair; deployment cares disproportionately about rank one.

Repeat the audit on classification folds, covariate-shift benchmarks, and foundation-model harnesses; pre-register tie handling; preserve example identity across every contrast; and report both full-ranking displacement and Top-1 swaps. For time series, also vary the amount and recency of training data separately from the refit schedule so those mechanisms do not travel together.

The primary nonlinear candidate is explained on the XGBoost model page. For a complementary case where point forecasts depend on what information existed at evaluation time, see the forecast-provenance walkthrough.