← Blog/blog/series-wise-cv-target-mismatch

The best error estimate can still pick the worse forecaster

A global forecaster is one model trained across many time series: stores, meters, products, patients, or sensors. The usual validation split holds out each series' future but lets every entity appear in training. Cerqueira, Oliveira, and Roque ask a different question: what if tomorrow's population contains entirely new series?

Their answer separates two jobs that are often treated as identical. Cross-validation can estimate the magnitude of future error, or it can rank candidate models well enough to choose one. The split that wins the first job does not necessarily win the second.

01

One dataset has two axes to hold out

Time-wise cross-validation preserves chronology: train on the past and validate on the future of the same entities. Series-wise cross-validation also separates entities, so no validation series appears in training. Each held-out series still uses its own history as forecasting context, but its parameters never influenced the fitted model.

The paper evaluates 10 neural architectures on 11 datasets. Its final out-set contains the last two forecast horizons from every series, while only 70% of series appear in the development in-set. The other 30% are genuinely new at evaluation time. That design is not a neutral detail; it defines the population that the validation procedure should target.

02

A closer estimate can rank models incorrectly

Consider a specialist that is excellent on familiar series but brittle on new ones, and a transferable model that gives up a little accuracy on familiar entities. Their deployment loss is just a mixture of those two regimes. The tested TypeScript below drives the next figure; Python and C++ are faithful translations.

def mixed_population_loss(known_loss, new_loss, new_share):
    if not 0.0 <= new_share <= 1.0:
        raise ValueError("new_share must be between zero and one")
    return (1.0 - new_share) * known_loss + new_share * new_loss
specialist: best on known seriestransferable: steady on new series
Core-computed illustrative losses. The specialist wins while fewer than 22.2% of deployment series are new; beyond that point, the transferable model wins. These are seeded-free toy values, not paper-reported losses.

A time-wise split sees the left edge and selects the specialist. A series-wise split sees the right edge and selects the transferable model. At the paper's 30% new-series deployment mix, the time-wise choice incurs 0.014 regret; the series-wise choice incurs 0.000. Neither split is universally correct. Each is correct for a different target population.

03

The paper separates estimation from selection

MASE—mean absolute scaled error—divides forecast MAE by a naive-baseline MAE, so lower is better. MAPEE is the absolute gap between a validation estimate and the true held-out MASE. Time-wise CV achieved the best MAPEE, yet series holdout selected the oracle-best architecture more often and K-Monte-Carlo CV achieved less than half its regret.

Validation methodMAPEE ↓Under-estimatesSelection accuracy ↑Regret ↓
Time-wise CV0.10932.4%27.3%0.034
Series holdout0.14137.3%36.4%0.019
Series K-MC-CV0.14026.7%30.3%0.016
Paper-reported averages across 11 datasets and three seeds. A method can estimate every candidate's loss closely while still getting the ordering wrong near the minimum.
Paper-reported probability of selecting the oracle-best architecture. With 10 candidates, random selection would average 10%. Holdout leads despite having the worst MAPEE of these three methods.
04

The benefit reverses with the population

In the evolving-population experiment, time-wise CV had the worst average downstream MASE, 0.883. Series K-MC-CV reached 0.857—a 2.94% reduction. But with a complete in-set, where every series was already known, time-wise CV reached 0.849 and beat the series-wise alternatives.

Final populationTime-CV MASEHoldout MASEK-MC-CV MASE
30% new series0.8830.8580.857
No new series0.8490.8640.859
Paper-reported average downstream MASE. The winning validation axis changes when the deployment population changes.
Core-computed relative reduction from the paper's reported evolving-population MASE values, using Time-CV as the baseline.
05

What to probe next

Vary the unseen-series fraction instead of fixing it at 30%, and stratify the series split by scale, frequency, geography, or product family. A random entity holdout can still be easy if near-duplicates land on both sides. Confidence intervals over selection regret would also show whether the small average MASE gap is stable enough to justify more training runs.

Finally, combine entity shift with gradual within-series drift. The paper isolates one axis cleanly, but real deployments often move on both. Its DLinear candidate connects to the linearmodel page, while PatchTST and TFT connect to the transformer mechanics.

References

  1. Cerqueira, V., Oliveira, M., and Roque, L. (2026). Series-wise Cross-validation for Global Time Series Forecasting. arXiv
  2. Bates, S., Hastie, T., and Tibshirani, R. (2024). Cross-validation: what does it estimate and how well does it do it?. Journal of the American Statistical Association 119(546)
  3. Zeng, A., Chen, M., Zhang, L., and Xu, Q. (2023). Are Transformers Effective for Time Series Forecasting?. AAAI 2023