The virtue of complexity, read closely

Market timing means changing how much you hold in the market based on a forecast: lean in when you expect gains, step out when you don't. The classic scorecard is the Sharpe ratio — average return divided by its volatility, so it rewards return per unit of risk, not raw return. A buy-and-hold S&P investor earns a Sharpe around 0.4 annually; beating that with timing is supposed to be nearly impossible.

For fifteen years the reference point was Goyal and Welch (2008): they ran the standard predictors — dividend yield, the term spread, volatility, and a dozen more — and found the out-of-sample R² (how much of next month's return variance the forecast explains, tested on data the model never saw) was reliably negative. A negative R² means you would have done better ignoring the model and guessing the historical average. Their verdict: return prediction is “a failed endeavor.”

The paper's lever is model complexity, defined as c = P / T: the number of parameters P divided by the number of training observations T. Orthodox statistics says keep c well below 1 — with more parameters than data points (c > 1) a linear model can fit the training set exactly, including its noise, and should generalize terribly.

But machine learning kept noticing something strange, formalized by Belkin et al. (2019) as “double descent.” As you add parameters, test error first falls, then spikes to a peak right at the interpolation boundary (c = 1, where the model just barely fits the data), and then — counterintuitively — falls again as you push into the over-parameterized regime. More parameters past the danger point can help. This paper is that idea carried into finance.

You cannot get P > T from 15 predictors and a linear model directly — you only have 15 slopes. So the paper manufactures features. Take the predictor vector G (15 numbers) and pass it through random Fourier features (RFF): draw a random weight vector ω ~ N(0, I) and emit a sin/cos pair of a random linear combination. Draw as many weights as you like and you have as many features as you like — the same trick used to approximate a kernel. It is literally a wide two-layer neural net whose first layer is frozen at random and never trained.

import numpy as np

def rff(g, omegas, gamma):
    # Kelly-Malamud-Zhou eq. (20): each weight w ~ N(0, I) makes a pair
    #   S_i = [ sin(gamma * w . g), cos(gamma * w . g) ].
    # P features from P/2 weights — a wide net with a FIXED first layer.
    out = []
    for w in omegas:
        z = gamma * float(np.dot(w, g))
        out += [np.sin(z), np.cos(z)]
    return out

Those P features go into ridge regression — ordinary least squares with a penalty z on the size of the coefficients. Ridge is the same regularized linear fit you can bend on the linear-regression page; here it is the entire learning algorithm. Complexity just means stacking more random features in front of it.

# Primal ridge: (X^T X + z I) b = X^T y      -- textbook, a P x P solve.
def ridge_primal(X, y, z):
    P = X.shape[1]
    return np.linalg.solve(X.T @ X + z * np.eye(P), X.T @ y)

# Dual ridge: b = X^T (X X^T + z I)^-1 y      -- IDENTICAL (Woodbury), but a
# T x T solve. When P >> T (high complexity), this is the cheap one.
def ridge_dual(X, y, z):
    T = X.shape[0]
    return X.T @ np.linalg.solve(X @ X.T + z * np.eye(T), y)

We can't ship 95 years of CRSP data, so the charts below run the paper's exact machinery on a seeded, illustrative market: persistent predictors and a small nonlinear predictable signal (a sin and a product term) buried in noise — the kind of structure a linear model is blind to but random features can bend toward. Everything is drawn by vocCurves() in src/core/complexity/, with a one-year training window (T = 12) exactly as in the paper.

# One recursive out-of-sample step at month t, training window length T:
#   train on pairs (features[i], return[i+1]) for i in [t-T, t-1]
#   forecast  f = b . features[t]
#   timing return = f * return[t+1]          # bet proportional to the forecast
#
# Complexity is c = P / T. The market's expected return is normalised to 0,
# so a positive average of (f * return) is pure timing skill.

Now the forecast-accuracy view — the number Goyal and Welch stared at. Near c = 1, the near-ridgeless model interpolates the 12 training months and its out-of-sample forecasts explode: R² craters far below −100% (our seeded run bottoms near −1200%). Push complexity higher and it climbs back. Add shrinkage and the spike is tamed entirely — the orange line barely leaves zero.

On real data (CRSP, 1926–2020, 15 monthly predictors, RFF up to P = 12,000), the authors put the orthodoxy and the complex model side by side. Table I is the clearest contrast — a one-year window, forecasting the monthly market return:

The high-complexity Sharpe holds up across training windows, and it survives the honest adjustment for buy-and-hold exposure: information ratios (alpha per unit of tracking error) land around 0.3 with alpha t-statistics of 2.6–2.9.

“More parameters than data, and it beats the market” is the headline. Read closely and the mechanism is more sobering than magical.

Two more things worth holding onto. First, the danger zone is exactly where a cautious practitioner would sit: the linear model at c ≈ 1 didn't just underperform, it produced a 98σ loss. “A little complexity” is the worst place to be — you want either genuinely simple or genuinely over-parameterized with shrinkage, not the boundary between them. Second, the results are gross of transaction costs, and a strategy that leans on the size of an all-but-random forecast can demand real turnover.

• Cost the turnover. The reported Sharpes are pre-cost. Re-run the timing strategy net of a realistic spread and see how much of the 0.47 survives — the honest number for anyone who'd trade it.
• Ablate the recession signal. If most of the edge is “get out before recessions,” a simple two-state model gated on a recession probability should capture much of it. How much complexity is left doing real work after that?
• Bandwidth honesty. Our toy uses γ = 0.5 to match its predictor scale; the paper uses γ = 2 on standardized predictors and reports insensitivity to it. Sweeping γ is a clean way to see how much the effect leans on the random-feature construction versus the shrinkage.

The whole engine is a regularized linear fit with random features bolted on the front. Go turn the shrinkage knob on the linear-regression page, then come back and reread Fig 2 — the ridge penalty is doing more of the work than the word “complexity” lets on.