Hierarchical risk parity, read closely

Start with the problem HRP is answering. You hold N assets; you know (roughly) how their returns move together — the covariance matrix Σ, whose diagonal is each asset's variance and whose off-diagonals say how each pair co-moves. The classic question, Markowitz's, is: what mix minimises the portfolio's variance? The answer is a clean one-liner,

w = Σ⁻¹𝟙 / (𝟙ᵀΣ⁻¹𝟙),

and that Σ⁻¹ — the matrix inverse — is where the trouble lives. When two assets are highly correlated, Σ is close to singular (nearly non-invertible), and inverting it divides by numbers close to zero. Michaud called mean- variance optimisation an “error maximiser”: the inverse loads up on the smallest eigenvalues of Σ — the directions estimated least reliably from finite data — and turns their noise into enormous offsetting long and short positions. The book looks like it has tiny variance on the data you fitted, because it has fitted the noise. Show it a fresh month and the bet unwinds.

We can watch this happen. Take N = 10 assets that all share the same pairwise correlation ρ. By symmetry the true minimum-variance portfolio is just equal weight — leverage 1, no shorting. So any leverage above 1 in the estimated book is pure estimation error. We raise ρ from 0 to 0.9, estimate Σ from a short, seeded return sample each time, and measure the gross leverage (the sum of absolute weights) the optimiser produces.

HRP's escape is to never form Σ⁻¹ at all. It uses the covariance only to answer local questions — “which assets are relatives?” and “which of these two halves is riskier?” — and builds the weights bottom-up from those. Three stages.

Stage 1 — cluster the correlation matrix into a tree. First turn correlations into distances. Two assets that move together should be “close.” The metric López de Prado uses is dᵢⱼ = √((1 − ρᵢⱼ)/2): a correlation of 1 gives distance 0, a correlation of −1 gives distance 1, uncorrelated gives √0.5. Feed that distance matrix to agglomerative hierarchical clustering — repeatedly merge the two closest clusters — and you get a dendrogram, a binary tree whose leaves are assets and whose branches are the nested groups they fall into. (This is the same recursive binary splitting the decision-tree page animates, run on assets instead of data points.)

import math

def corr_dist(corr):                     # HRP Stage 1: correlations -> distances
    n = len(corr)
    return [[math.sqrt((1 - corr[i][j]) / 2)   # a proper metric, d in [0, 1]
             for j in range(n)] for i in range(n)]

def ivp_weights(cov, idx):               # inverse-variance, normalised to sum 1
    inv = [1 / cov[i][i] for i in idx]
    s = sum(inv)
    return [v / s for v in inv]

def cluster_var(cov, idx):               # variance of that sub-portfolio: w' V w
    w = ivp_weights(cov, idx)
    return sum(w[a] * cov[idx[a]][idx[b]] * w[b]
               for a in range(len(idx)) for b in range(len(idx)))

Stage 2 — quasi-diagonalisation. Read the leaves of that tree left to right and reorder the covariance matrix into that sequence. Because relatives were merged early, the big correlations now cluster along the diagonal — a “quasi-diagonal” matrix. Nothing is transformed or thrown away (unlike PCA); the rows are only permuted, so every number still means what it did. This ordering is what lets the next stage split the portfolio along its natural seams.

Stage 3 — recursive bisection. Now allocate. Start with the whole (reordered) list holding 100% of the capital, and split it in half. Compute each half's variance as an inverse-variance sub-portfolio — this is the only place variance enters, and it is a weighted sum, never an inverse — then hand the riskier half proportionally less capital via α = 1 − Vₗₑ𝒻ₜ/(Vₗₑ𝒻ₜ + Vᵣᵢ𝓰ₕₜ). Recurse into each half until you reach single assets. The result is long-only, sums to 1, and never inverted Σ.

def hrp_bisect(cov, order):              # order = quasi-diagonal leaf order (Stage 2)
    w = [1.0] * len(cov)
    def recurse(items):
        if len(items) <= 1: return
        mid = len(items) // 2
        left, right = items[:mid], items[mid:]
        v_l, v_r = cluster_var(cov, left), cluster_var(cov, right)
        alpha = 1 - v_l / (v_l + v_r)    # give the riskier half LESS capital
        for i in left:  w[i] *= alpha
        for i in right: w[i] *= 1 - alpha
        recurse(left); recurse(right)
    recurse(order)
    return w                             # long-only, sums to 1 — V was never inverted

For contrast, here is the whole minimum-variance optimiser — one inversion, and the fragility of Fig 1 rides in on that single line. The site runs the TypeScript of all of these to draw the charts below; the Python and C++ are line-for-line translations.

def min_variance(cov):                   # the optimiser Markowitz hands you
    inv = inverse(cov)                   # <-- invert the ESTIMATED covariance
    ones = [1.0] * len(cov)
    raw = mat_vec(inv, ones)             # Sigma^-1 . 1
    total = sum(raw)                     # 1' Sigma^-1 . 1
    return [r / total for r in raw]      # w = Sigma^-1 1 / (1' Sigma^-1 1)

Now the paper's headline claim, which sounds impossible at first: HRP produces lower risk out-of-sample than the minimum-variance optimiser — even though minimising variance is literally the optimiser's job. The trick is the phrase “out-of- sample.” We build a seeded world of 12 assets in three correlated blocks of very different risk (a big, tightly-correlated, low-volatility cluster; a smaller, loosely-correlated, high-volatility one; and an almost-independent pair). Then, for each of 150 draws, we estimate Σ on one 44-period sample, build every book from it, and measure the risk they actually deliver on an independent sample drawn from the same world.

“Outperform out-of-sample” is doing a lot of quiet work in the title, and it is easy to over-read. Three cautions the result deserves:

• HRP is not minimising variance — and it doesn't claim to. On the very data it was built from, minimum-variance beats it every time (Fig 2's in-sample column). HRP's win is conditional on estimation error: shrink the noise — more history, fewer assets, a better covariance estimator — and the gap narrows, because the optimiser was only ever losing to its own inverted noise. HRP is insurance against a bad Σ, not a free lunch on a good one.
• It never looks at expected returns. Every method here allocates on risk alone; none forecasts a single return. So “outperform” means lower realised risk, not higher profit. That is genuinely useful — a steadier denominator lifts the Sharpe ratio (return per unit of risk) — but it is a different claim from “makes more money,” and pairing HRP with a return signal is a separate, unsolved problem.
• The clustering is a modelling choice, lightly examined. We used single- linkage, like the paper. Single linkage is prone to chaining — one bridging asset can string unrelated groups into a straggly cluster — and the whole quasi-diagonal order, and thus the weights, can shift if you swap in average- or Ward-linkage. The paper doesn't probe this sensitivity, and it is the first place a live book would surprise you.

• Vary the linkage and re-run Fig 2. Swap single- for average- and Ward- linkage and watch how much the out-of-sample volatility moves. If HRP's edge is robust, the choice shouldn't matter much; if it swings, the clustering — not the risk-splitting — is doing the work.
• Shrink the covariance and let minimum-variance fight back. Replace the raw sample Σ with a Ledoit–Wolf shrinkage estimate before inverting. Shrinkage is the other answer to the error-maximisation problem; seeing how much of HRP's gap it closes tells you whether HRP is buying something a better estimate can't.
• Stress the correlation structure over time. Our two samples are drawn from one fixed world. Real correlations regime-shift — and in a crisis they all rush toward 1, the exact ρ → 1 corner where Fig 1 shows the inversion blowing up. HRP's tree would collapse toward a single cluster too; measuring how gracefully it degrades is the test that matters for a real book.

HRP is a lovely example of a machine-learning idea — hierarchical clustering — quietly fixing a sixty-year-old finance problem, not by optimising harder but by refusing to invert a matrix it can't trust. The recursive bisection at its heart is the same tree-splitting you can watch grow on the decision-tree page; picture it running on a correlation matrix instead of a scatter of points, handing each branch a slice of risk on the way down. Just remember what it bought you — robustness to a bad estimate — and what it never promised: a view on where returns are going.