The categorical encoder that refuses to read its own label

A category such as user ID, city, or product code has no natural numeric order. One-hot encoding gives every value a column, which becomes unwieldy when there are millions of values. A compact alternative istarget encoding: replace each category with the average label observed for it.

That average can quietly include the row being encoded. CatBoost's central ordering principle fixes the leak: invent a random order and let each training row see only labels from earlier rows. The feature still learns whether a category tends to click or default, but it cannot copy its own answer.

For a row's category, sum the earlier labels, add a smoothed prior, and divide by the earlier count plus the prior strength. Crucially, update the running totals after emitting the row's feature. This is the offline-data version of an online learner that cannot look into the future.

def ordered_encode(categories, targets, order, prior, strength):
    encoded = [0.0] * len(categories)
    sums, counts = {}, {}
    for i in order:
        category = categories[i]
        total = sums.get(category, 0.0)
        count = counts.get(category, 0)
        encoded[i] = (total + strength * prior) / (count + strength)
        sums[category] = total + targets[i]
        counts[category] = count + 1
    return encoded

The charts run the tested TypeScript core. Python and C++ are faithful translations of the same history-only loop. CatBoost uses multiple permutations and applies the ordering principle to boosting gradients too; this reconstruction isolates its categorical statistic.

This is target leakage: information unavailable at prediction time sneaks into a training feature. Cross-validation does not rescue an encoding calculated once on the full dataset; the encoding itself must be fitted inside each fold or constructed without self-labels.

Ordered encoding cannot know a category's target rate before its first appearance. The first occurrence gets the prior; later rows gradually incorporate real history. In contrast, the leaky full-data mean is constant from the first row because it uses labels from the entire sequence—including the future.

The table keeps CatBoost's ordered boosting mode fixed and swaps only the categorical target statistic. Positive numbers are worse than ordered target statistics. Greedy encoding is dramatically fragile on several high-cardinality datasets; holdout avoids the leakage but sacrifices labeled rows for either encoding or training.

In the broader benchmark, CatBoost's tuned log loss on Amazon was 0.139; LightGBM and XGBoost were each about 17% worse. That is not a universal ranking: the paper's gains vary sharply by dataset, and its ordered mode was about 1.7× slower than CatBoost's plain mode in the reported timing experiment.

• Measure performance by category frequency, not only overall loss.
• Shift the test mix toward categories unseen during training.
• Fit priors per fold and audit every preprocessing boundary.

To see how the encoded number becomes a split, poke at the interactive decision-tree page. For the additive ensemble around those trees, compare the XGBoost and LightGBM pages.