LGBM for stock returns, read closely

The task is deliberately narrow: using the last 10 days of data, predict a stock's next-day return (not price — return, which is far harder to fake through autocorrelation). The dataset is every S&P 500 company plus the index itself, 2014–2017, about 497,000 rows. Eighteen features per row: raw OHLCV, the S&P 500 index as a market-context signal, and a handful of technical indicators.

The indicators are the interesting part, because they're small enough to implement and watch. Here are two of them — the chart below is drawn by this code, not a screenshot. (Toggle the language: the site runs the TypeScript; the Python and C++ are line-for-line translations.)

def bollinger(prices, n, k=2):
    mid = sma(prices, n)
    bands = []
    for i in range(len(prices)):
        if i < n - 1:
            bands.append((None, None, None))
            continue
        m = mid[i]
        var = sum((prices[j] - m) ** 2 for j in range(i - n + 1, i + 1)) / n
        sd = var ** 0.5                       # population sd over the window
        bands.append((m, m + k * sd, m - k * sd))
    return bands

RSI is the other workhorse — Wilder's smoothed ratio of average gains to average losses, squashed into 0–100. Above 70 is “overbought,” below 30 “oversold.”

def rsi(prices, n):
    out = [None] * len(prices)
    if len(prices) <= n:
        return out
    gain = sum(max(prices[i] - prices[i - 1], 0) for i in range(1, n + 1))
    loss = sum(max(prices[i - 1] - prices[i], 0) for i in range(1, n + 1))
    avg_gain, avg_loss = gain / n, loss / n          # Wilder's seed
    out[n] = 100 - 100 / (1 + avg_gain / avg_loss)
    for i in range(n + 1, len(prices)):
        ch = prices[i] - prices[i - 1]
        avg_gain = (avg_gain * (n - 1) + max(ch, 0)) / n
        avg_loss = (avg_loss * (n - 1) + max(-ch, 0)) / n
        out[i] = 100 - 100 / (1 + avg_gain / avg_loss)
    return out

With time series, the split is everything. Shuffle the rows and you leak the future into training; your R² soars and means nothing. The paper uses expanding-window cross-validation: each fold trains only on the past and tests on the next unseen block.

# Expanding-window time-series CV: every fold trains only on the past.
def time_series_splits(n, folds):
    size = n // (folds + 1)
    splits = []
    for f in range(1, folds + 1):
        splits.append({
            "train": range(0, f * size),              # everything up to here
            "test": range(f * size, (f + 1) * size),  # the next block, unseen
        })
    return splits

Across the five models, LightGBM has the lowest error and the highest explained variance on next-day returns. Gradient boosting on tabular features beating an LSTM is the expected outcome here, and it's worth internalizing: for tabular data, boosted trees remain the one to beat.

The second half turns predicted returns into a 20-stock portfolio, weighted for risk parity and optimized for alpha. The headline numbers look strong: alpha of +10.6% over the benchmark, +19.1% total return.

To its credit, the paper computes the Kelly criterion — the growth-optimal bet size given a win probability and payoff. It plugs in the model's R² as the win probability:

# f* = p - (1 - p) / b   ->   negative means: do not bet.
p = 0.7377   # win prob, taken from the model's R^2
b = 1        # net odds
f = p - (1 - p) / b   # = -0.6327

• R² on returns is fragile — a model that always predicts “tiny positive” scores well when most days are tiny and positive. Directional accuracy (did we get the sign?) is the number that would move Kelly.
• Risk parity here collapsed to equal 5% weights. With a real covariance estimate the weights and the VaR would both shift; that's a clean follow-up experiment.
• The honest headline isn't “LGBM predicts the market.” It's “LGBM explains next-day return variance well in-sample, and even then the bet doesn't clear the Kelly bar.”

The model at the center of this paper is on this site — go bend the LightGBM decision surface yourself, then come back and reread the R² table.