Trees & ensembles · /forest

Random forest

Bagging plus one crucial twist: each split flips a seeded coin and considers only one feature, decorrelating the trees. The loss curve tracks out-of-bag error — validation you get for free.

class 0class 1filled = train, hollow = testhover to probe · click to pin

step 0OOB err —train acc —test acc —

Data

datasetpoints150noise0.15seedsplitmouse

Forest prediction

\overset{p}{ˉ} (x) = \frac{1}{M} m = 1 \sum M p_{m} (x) = \overset{p}{ˉ}

Feature subsampling

features tried per split = m of d = 2

With m = 1, each split flips a coin between x₁ and x₂ — this decorrelates the trees, which is what separates a random forest from plain bagging.

Out-of-bag error

OOB = \frac{1}{∣ B ∣} i \in B \sum 1 [\overset{y}{^}_{i}^{oob} \neq = y_{i}] = e

A free validation estimate: every point is out-of-bag for ≈ 37% of the trees. OOB points are ringed on the canvas.

What's simplified here

With d = 2 features, "random subspace" means one random feature per split (m = 1); real forests use m ≈ √d of many features.
OOB error counts only train points left out of at least one bag, judged solely by trees that never saw them.
No feature importance in v1.