Trees & ensembles · /bagging

Bagging

Each Step trains one tree on a fresh bootstrap sample. The heatmap is the average of every learner's vote — open Internals to see the individual learners and knock them out of the vote.

class 0class 1filled = train, hollow = testhover to probe · click to pin

step 0train err —train acc —test acc —

Data

datasetpoints150noise0.15seedsplitmouse

Bootstrap sampling

n^{'} = ⌈ ρ n ⌉ = n^{'} P (omitted) = (1 - \frac{1}{n})^{n^{'}} = p

Faded points in the learner strip were never drawn for that learner; larger points were drawn multiple times.

Aggregate prediction

\overset{p}{ˉ} (x) = \frac{1}{M} m = 1 \sum M p_{m} (x) = \overset{p}{ˉ}

Why averaging helps

Var (\frac{1}{M} \sum_{m} f_{m}) = \frac{σ ^{2}}{M} (if the f_{m} were independent)

Bootstrapped trees are correlated, so the true reduction is smaller — but the direction holds.

What's simplified here

Base learners are the same CART trees as the tree page (Gini, min 2 samples per leaf).
Aggregation averages class probabilities (soft voting); majority vote differs only at exact ties.
Bootstrap draws come from the train split only — the test points never enter a bag.