Why trees beat deep nets on tables — until you rotate

Tabular data is the spreadsheet kind: rows are examples, columns are named features — age, price, blood pressure, a one-hot category. It is the most common data in the world and the least glamorous, and for a decade the reliable way to model it has been gradient-boosted trees (XGBoost, LightGBM): ensembles of small decision trees, each fixing the last one's errors. Deep learning conquered images, audio, and text, but on tables it keeps losing to a technique from the 2000s.

Grinsztajn, Oyallon, and Varoquaux set out to say why, carefully. They built a benchmark of 45 tabular datasets (~10,000 samples each), spent on the order of 20,000 compute hours tuning every model fairly, and confirmed the folklore: tree ensembles top neural nets — MLPs, ResNets, and the transformer-flavored FT-Transformer — on medium-sized tables. The contribution isn't the leaderboard. It's the autopsy: three concrete properties of tabular data that trees handle and nets don't.

Each reason is a mismatch between a neural net's built-in assumptions — its inductive bias, the shape of solution it prefers before seeing data — and what tabular data actually looks like.

Here is the intuition the rotation test makes concrete. A decision tree draws only horizontal and vertical lines. If the true boundary is horizontal or vertical, one cut nails it. If the boundary is diagonal, the tree has to approximate it with a staircase — many cuts, each stealing a little data, generalizing a little worse:

So when a feature carries individual meaning — when the axes are the “right” ones, the way age and income are — the tree's stubborn axis-alignment is exactly what you want. Rotate the data and you scramble that meaning: every new axis is a blend of old ones, the crisp boundaries go diagonal, and the tree is forced to staircase everything. A rotation-invariant learner, by contrast, can't even tell you rotated it.

We can't ship the paper's 45 datasets, so we build the cleanest toy that isolates the mechanism: a checkerboard. Points fall in a 3×3 grid on the plane, colored by the parity of their cell — an aggressively axis-aligned, non-smooth target. Then we pit two learners with opposite biases against it: the site's own decision tree (axis-aligned) and a k-nearest-neighbors classifier using Euclidean distance (rotation-invariant, because a rotation preserves every distance exactly — a fact a unit test in tabular.test.ts pins to 1e-9).

import math

def cell_parity(x, y, k):
    cx = int(((x + 1) / 2) * k)      # column of the k x k grid (x in [-1, 1])
    cy = int(((y + 1) / 2) * k)      # row
    return (cx + cy) % 2             # 0/1 checkerboard label

def rotate(pts, theta):             # turn the whole cloud about the origin
    c, s = math.cos(theta), math.sin(theta)
    return [(x*c - y*s, x*s + y*c, lbl) for (x, y, lbl) in pts]

The experiment: rotate the whole dataset by an angle θ, retrain both learners on the rotated data, and measure out-of-sample accuracy. Sweep θ from 0° to 90°. (The site runs the TypeScript; the Python and C++ are line-for-line translations. The tree is the exact buildTree from the model pages.)

# Two learners with opposite inductive biases:
#   tree -> axis-aligned splits   (reads the coordinate frame)
#   kNN  -> Euclidean distance     (ignores it; rotation-invariant)
def knn_predict(train, q, k):
    nn = sorted(train, key=lambda p: (p.x-q.x)**2 + (p.y-q.y)**2)[:k]
    return round(sum(p.label for p in nn) / k)   # majority vote

def accuracy_at(theta, train, test):
    tr, te = rotate(train, theta), rotate(test, theta)
    tree = build_tree(tr, depth=6)               # the /tree page's grower
    a_tree = mean(predict(tree, q) == q.label for q in te)
    a_knn  = mean(knn_predict(tr, q, 15) == q.label for q in te)
    return a_tree, a_knn

That crossing is the paper's Figure 6 in miniature. On its real datasets the same move — “random rotations reverse the performance order: NNs are now above tree-based models” — is what proves the point: the tree's edge was its use of the coordinate frame. Take the frame away and the edge goes with it.

The headline invites the wrong takeaway — “trees are the better model class” — and a practical footgun follows from it.

And the second finding compounds the first: MLPs are hurt by uninformative features, which real tables are full of. A rotation is the worst of both worlds for them in reverse — it spreads whatever signal exists across all columns, which happens to help a net that was going to mix them anyway, while destroying the sparse, column-local structure a tree feeds on. The two biases aren't just different; they're matched to different data geometries.

• Find the crossover angle. Our tree falls below k-NN somewhere between on-axis and ~68°. Sweeping finely for the exact angle where the ranking flips would quantify “how much frame” the tree is really living on for a given dataset.
• Cost a real PCA. Take a tabular benchmark where XGBoost wins, apply PCA whitening, and re-fit. If the caveat above is right, the tree's accuracy should sag toward the net's — a clean, falsifiable pipeline experiment.
• Give the tree a better frame, not a worse one. The flip side of rotation-hurts is rotation-could-help if you rotate toward the boundary. Learning a small orthogonal transform that makes boundaries axis-aligned, then boosting, is the tree-friendly mirror image of what a net does for free.

The learner in Fig 2 is the same one on the decision-tree page — go draw a diagonal boundary by hand and watch it staircase, then imagine a random rotation doing that to every clean cut at once. That staircase is the whole reason deep learning still hasn't taken the table.