← Blog/blog/hurricane-discussion-three-hours-late

The hurricane forecast discussion helps—but it arrives three hours late

A hurricane forecaster's prose can contain something the numerical fields do not: a judgment that a storm will resist shear, a synthesis of aircraft observations, or a warning that rapid strengthening is likely. Azizi and colleagues ask whether a model can turn that prose into a better intensity forecast.

Retrospectively, yes. Adding National Hurricane Center discussion text to 13 numerical predictors cuts the five-model ensemble's 24-hour wind mean absolute error (MAE)—the average absolute miss—from 9.89 to 8.90 knots. That is a 10.0% reduction. But the text is usually issued 3 hours after the numerical valid time it is paired with. The result establishes predictive information, not a forecast that could have existed at that earlier time.

01

What the model combines

The study covers 637 Atlantic and eastern North Pacific tropical cyclones from 2002–2023. Each case predicts the change in maximum sustained wind and minimum sea-level pressure 24 to 120 hours ahead. The numerical branch receives 13 SHIPS predictors such as current wind, pressure, sea-surface temperature, humidity, and wind shear. Optional ERA5 inputs are two storm-centered 10-metre wind fields.

The text branch embeds the storm-specific NHC discussion. Neural models concatenate projected text features with tabular features; a temporal variant also processes eight six-hourly wind frames with a gated recurrent unit. The main split trains on 2002–2019, validates on 2020–2021, and tests on whole storms from 2022–2023, so no storm crosses a split.

02

Compare errors in pairs, resample storms

Every test case has two predictions from fixed ensembles: tabular-only and tabular-plus-discussion. The paired gain is the first model's MAE minus the second's. Positive means the discussion helped. Forecasts within one cyclone are dependent, so resampling individual six-hour cases would pretend there is more independent evidence than there is.

The paper instead draws whole storms with replacement and keeps every row from each drawn storm. Its reported percentile interval is conditional on the fitted ensembles: it captures which storms landed in the test sample, but not uncertainty from tuning, checkpoint selection, or trying additional baselines.

def storm_bootstrap_gain(rows, repeats, seed):
    rng = Random(seed)
    storm_ids = sorted({row.storm_id for row in rows})
    blocks = {sid: [row for row in rows if row.storm_id == sid] for sid in storm_ids}
    gains = []
    for _ in range(repeats):
        sample = []
        for _ in storm_ids:
            sid = storm_ids[rng.randrange(len(storm_ids))]
            sample.extend(blocks[sid])
        mae_t = mean(abs(row.tabular - row.observed) for row in sample)
        mae_td = mean(abs(row.with_discussion - row.observed) for row in sample)
        gains.append(mae_t - mae_td)
    return gains
03

A seeded block-bootstrap audit

The small audit below has six synthetic storms and two forecasts per storm. It is not a reproduction of the private prediction files. Its job is narrower: make the resampling unit visible. Every one of 400 seeded draws samples six storm IDs, then carries both rows of each selected storm into the paired MAE calculation.

seeded storm-block gain
Quantiles of paired MAE gain from 400 seeded whole-storm resamples. The 2.5th–97.5th percentile interval is [0.08, 3.00] knots. Illustrative synthetic data computed by the pure core module.
04

The gain is real in one comparison, not universal

ComparisonWithout / referenceWith discussionReading95% storm interval / caveat
Primary 24 h ensemble9.89 kt8.90 kt0.99 kt[0.32, 1.70]
Temporal embedding + XGBoost10.56 kt9.71 kt0.59 kt paired[−0.11, 1.33]
Official NHC forecast context7.69 kt official8.97 kt T+Dofficial lowernot a parity claim
Paper-reported wind MAE comparisons. The official forecast row uses the same available forecast rows for context, but the retrospective inputs are not information-matched to operations.

The official NHC forecast is also more accurate than the multimodal model on the matched operational-aid comparison: 7.69 versus 8.97 knots. That does not invalidate the ablation. It does prevent the ablation from being sold as operational parity.

05

Ninety percent overall can hide the dangerous misses

The paper calibrates 90% intervals using whole-storm blocks and evaluates held-out storms. Marginal wind coverage for tabular-plus-discussion is 0.917, yet storm-by-storm coverage ranges sharply. Otis reaches only 70%, with 26.1-knot point MAE; its largest rapid-intensification changes remain outside the band.

Paper-reported per-storm coverage for the primary 24-hour tabular-plus-discussion intervals (Figure 13). The nominal target is 90%; aggregate coverage does not guarantee conditional coverage.

Rapid intensification means an observed wind increase of at least 30 knots in 24 hours. All eight interval methods fall below their 90% target on that realized subset. The diagnostic reuses 97 unique cases across many held-out folds, and it identifies rapid intensification only after it happened; it is neither a pre-forecast risk classifier nor a future conditional-coverage guarantee.

06

Why the timing caveat is the mechanism

Consider a 06Z numerical state matched to a 09Z discussion. The text may mention reconnaissance, later guidance, or the official forecast itself. If a model improves, three explanations remain entangled: expert reasoning added value, text restated strong existing guidance, or information simply arrived later. The experiment cannot distinguish them because it never reconstructs the exact information set available at 06Z.

This is not ordinary train/test leakage: the temporal split is sensible. It is availability leakage between modalities inside one test example. A deployable multimodal benchmark needs an as-of timestamp for every observation, analysis field, narrative, and label revision.

07

What to probe next

Re-run the comparison on a future season with model choices frozen. Construct each case at the actual discussion issue time, roll numerical predictors forward to that same time, and replace retrospective fields with archived real-time analyses. Report performance against the official forecast, stratify by a pre-declared rapid-intensification risk group, and keep storms—not rows—as the unit of uncertainty.

For the mechanics behind the model branches, the site's transformer, neural-network, and XGBoost pages build the encoders and nonlinear regressors from scratch. The important lesson travels beyond weather: multimodal performance is meaningful only when all modalities share an honest availability time.

References

  1. Azizi, I., Tam, F. I.-H., Boldi, M.-O., Chavez-Demoulin, V., & Beucler, T. (2026). Quantifying the added value of human forecaster discussions for tropical cyclone intensity prediction. arXiv:2611.21602 [cs.LG]
  2. Gibbs, I., & Candès, E. (2021). Adaptive conformal inference under distribution shift. Advances in Neural Information Processing Systems 34