When one linear layer beat the Transformer, read closely

A forecast here means: given the last L readings of a series (the look-back window), predict the next H readings (the horizon). “Long-term” forecasting pushes H out to 96, 336, even 720 steps — days of hourly data at once. It is the problem behind every capacity plan and every “how much power will the grid need Tuesday” question.

After Transformers conquered language, the field spent 2020–2022 porting them to this task, one ornate variant after another — Informer, Autoformer, FEDformer — each adding machinery to make attention affordable over long sequences, each claiming the new state of the art. Then Zeng, Chen, Zhang and Xu asked a rude question: had anyone checked a linear baseline? They wrote a model with no attention, no layers, no nonlinearity — a single matrix that multiplies the look-back window and out comes the whole forecast — and it beat all of them, “surprisingly… in all cases.”

The subtlety is in one word: direct. Most classical forecasters are autoregressive — predict one step, feed it back, predict the next — so errors compound down the horizon. LTSF-Linear instead maps the look-back to all H future steps in a single matrix multiply. Each future step gets its own row of weights over the whole window; step 200 can lean on lag 47 directly if that is where its signal lives. Crucially the weights are ordered: lag 1 and lag 24 have different columns. Contrast that with self-attention, which is permutation-invariant at its core — shuffle the inputs and the attention weights follow them — so it has to bolt ordering back on with positional encodings. The paper's thesis in one line: for a signal where order is the entire content, starting from an order-blind mechanism is starting in a hole.

The strong variant, DLinear, adds one idea from 1950s forecasting: split the window into a slow trend (a centered moving average) and the seasonal remainder, run a separate linear layer on each, and add the two forecasts. The decomposition is four lines.

def series_decomp(window, kernel):          # kernel is odd (paper uses 25)
    h = (kernel - 1) // 2
    n = len(window)
    trend = []
    for i in range(n):
        s = 0.0
        for j in range(-h, h + 1):
            idx = min(max(i + j, 0), n - 1)  # replicate-pad the two ends
            s += window[idx]
        trend.append(s / kernel)             # centered moving average
    seasonal = [window[i] - trend[i] for i in range(n)]
    return trend, seasonal                   # smooth drift + the oscillation left over

Then the forecast is those two linear maps, summed. Set the two weight matrices equal and the split dissolves back into a plain linear layer — so DLinear is strictly a superset of Linear, free to treat drift and oscillation differently.

def dlinear_forecast(window, W_seasonal, W_trend, kernel):
    trend, seasonal = series_decomp(window, kernel)
    # each layer maps the WHOLE look-back straight to the WHOLE horizon, in one shot
    out_s = W_seasonal @ seasonal        # W_seasonal: H x L
    out_t = W_trend    @ trend           # W_trend:    H x L
    return out_s + out_t                 # DLinear. set W_seasonal = W_trend -> plain Linear

We never train these matrices with an optimiser. A one-layer linear network minimising mean-squared error has a closed-form optimum — the same least-squares solve the linear-regression page performs, only widened to many inputs and many outputs. The site runs the TypeScript above (weights stored transposed so one Cholesky factorisation serves every horizon column); the Python and C++ are line-for-line translations. Every number in the charts below is produced by that code, per the site's formula = code rule.

# The "layer" is fit in closed form: the least-squares optimum an MSE-trained
# one-layer linear net converges to. Stack the training windows into X (features)
# and their futures into Y (targets), then solve the ridge normal equations.
#   Linear   -> features are the raw window          (L columns)
#   DLinear  -> features are [seasonal, trend]        (2L columns)
A = X.T @ X + z * I          # (P x P), P = L or 2L
W = cholesky_solve(A, X.T @ Y)   # (P x H); one factorisation, every horizon column
# No optimiser, no epochs, no learning rate. It is the /linear page's engine, widened.

The LTSF benchmarks (electricity, traffic, weather) are, underneath, strongly periodic. So we forecast a seeded stand-in: a gentle upward trend plus a fast daily-style cycle (period 24) and a slow weekly one (period 168), with Gaussian noise — all drawn through the site's seeded RNG. We fit on the first 72% and forecast the held-out tail it never saw. Here is one test window: the model sees the blue history, then has to draw the next 32 steps.

Scored over every test window, the gap is not subtle. Persistence posts a mean-squared error of 5.94; the linear layer, 0.247 — a 95.8% reduction. On the paper's real data the same model beat the Transformers by comparable margins (their numbers are in §07, labelled as theirs). A matrix multiply, doing the work of a research programme.

DLinear is the variant the paper leads with, and the decomposition looks like where the intelligence lives. So test it directly: fit a plain Linear layer on the raw window, fit DLinear with the trend/seasonal split, score both on the same held-out data.

Here is the paper's sharpest experiment, and the one that actually indicts attention. Feed the models a longer look-back — more history — and see who uses it. A competent forecaster should improve: more context, more of the periodic structure in view. The paper reports that the Transformers do not — their error stays flat or drifts up as the window grows, because self-attention drowns in the added tokens. The linear models improve steadily. We can compute the linear half honestly:

Why does a line drink up context that a Transformer chokes on? A sinusoid obeys a linear recurrence — its next value is a fixed linear combination of its past values — so a linear map with a long-enough window can lock onto the exact phase and reproduce the cycle indefinitely. Attention, built to find which faraway token to copy, has no such affinity for smooth periodic extrapolation; more positions mostly mean more ways to be distracted. Order-blindness, again, costing exactly where order is everything.

Every weight in that matrix is estimated from the training window and then frozen. The model is not reasoning about the future; it is asserting the future will oscillate the way the past did — same periods, same phase relationships. On the stationary benchmarks that assumption holds beautifully, which is precisely why a fixed linear map can look miraculous. So we break it in the one way a real series will: partway through, we change the dominant period from 24 to 40 — a regime shift, the market equivalent of a new supply schedule or a change in behaviour — and forecast across it with the same recipe.

None of this makes the paper wrong; it makes it precise. Its claim is about the LTSF benchmarks as they exist, and on those it is correct and important. The caveat is only that the win is inseparable from a property of the data — a stable spectrum — that the model never checks and cannot defend.

For honesty, here is the paper's headline table — real MSE on the standard datasets, not our synthetic stand-in. Lower is better; DLinear (highlighted) is beneath every Transformer in every row.

The paper landed as a provocation, and the field answered it — not by defending the old Transformers but by fixing them. Within a year PatchTST (Nie et al., ICLR 2023) gave attention two things it had been missing: it chops the series into patches (so a token is a little sub-window, not a single timestamp) and forecasts each channel independently. That put a Transformer back on top of DLinear across the same benchmarks. The lesson was never “linear wins” — it was “your baseline and your tokenisation were wrong,” and both camps were better for the fight.

• Does the regime-shift break survive a trend-only decomposition? Our seasonal layer bets on the old period. Freezing the seasonal layer and letting only the moving-average trend forecast across the shift would isolate how much of DLinear's robustness is real versus borrowed from stationarity.
• Where does linearity actually snap? Swap the clean cycle for a nonlinear, state-dependent one and a straight matrix must fail. A small neural net on the same windows would show exactly how much curvature the benchmarks were hiding — and whether the Transformers were solving a problem the data never posed.
• The honest baseline habit. The durable takeaway is methodological: before crowning a complex model, fit the one-line one. It is the forecasting cousin of this site's linear-regression refrain — a line through the data is not the answer, but it is the number every fancier answer has to beat.

The engine underneath all of this is the plainest fit there is — least squares, a matrix solved once. Go widen it in your head on the linear-regression page: picture the inputs as a strip of recent history and the output as the next stretch of it. LTSF-Linear is a genuinely humbling result — a subfield out-run by its own missing baseline. Just remember what the line is really promising: that tomorrow keeps yesterday's rhythm.