One checkpoint cannot serve every noise level

Early stopping sounds like one decision: save each epoch, watch validation error, and keep the checkpoint with the lowest score. Rudenko and colleagues show why that decision can be internally contradictory. In tabular regression, examples with noisy targets may start overfitting while cleaner regions are still learning. The “best” global epoch is then a compromise, not a universal optimum.

The authors compare 18 regression objectives on 41 established tabular datasets, with as many as 100 tuning trials per dataset-objective pair. Most alternatives do not consistently beat a carefully tuned mean squared error (MSE) baseline. Quantile loss is the exception that transfers most reliably across four neural backbones.

A conditional quantile is the target value below which a chosen fraction τ of outcomes should fall. The pinball loss penalizes underprediction by τ and overprediction by 1 − τ. At τ = 0.8, predicting 2 for a target of 5 costs 2.4, while predicting 5 for a target of 2 costs only 0.6. A grid of such asymmetric predictors sketches the conditional distribution.

The paper constructs matched synthetic datasets with the same features and noiseless target function. Only the training-label noise changes: one version has constant variance, while the other has input-dependent variance. On the heteroskedastic version, high-noise points prefer earlier checkpoints and low-noise points prefer later ones. The separation disappears in the constant- noise control.

def select_epochs(validation_mse):
    cohorts = len(validation_mse)
    epochs = len(validation_mse[0])
    global_curve = []
    for epoch in range(epochs):
        total = 0.0
        for cohort in range(cohorts):
            total += validation_mse[cohort][epoch]
        global_curve.append(total / cohorts)
    global_epoch = min(range(epochs), key=lambda e: global_curve[e])
    cohort_epochs = []
    for cohort in range(cohorts):
        best = min(range(epochs), key=lambda e: validation_mse[cohort][e])
        cohort_epochs.append(best)
    return global_epoch, cohort_epochs

The site runs the TypeScript version of that selector. The deterministic toy curves below are illustrative, not extracted from the paper. Their clean region bottoms out at epoch 56; the noisy region at epoch 28; averaging both validation curves chooses epoch 44. That middle checkpoint is defensible, yet optimal for neither group.

Splitting validation data into five or ten cohorts gives each checkpoint decision fewer examples. That alone can make selection noisy. The authors therefore compare uncertainty cohorts with random cohorts of the same size. On the seven largest real datasets, uncertainty grouping improves median test RMSE while random grouping does not. The gain is real, but it is measured in tenths of a percent.

In the small hand-checkable matrix used by this page, global selection routes both cohorts to checkpoint 2, whereas separate selection chooses 3 and 1. The routed test RMSE falls from 0.346 to 0.324. This is only a mechanism trace; the table and bar chart above are the paper's empirical result.

Standard quantile regression compares its predicted distribution with a point mass at the observed label. Adaptive Quantile Regression (AdaQR) replaces that point with a Gaussian centered on the label. A second output head predicts the Gaussian's scale for each input, while a penalty prevents unlimited smoothing. Under the paper's Gaussian assumptions, the optimal smoothing variance tracks the target-noise variance. The scale head is training-only, so inference still averages predicted quantiles and adds no test-time cost.

The practical lesson is diagnostic: plot validation error by uncertainty group before replacing a loss or backbone. If minima separate, one scalar early- stopping score hides a learning-allocation problem. The same check works for a plain neural network, and the clean linear baseline on the linear-regression page is a useful control before adding a distributional head.

Next, test whether the conflict survives uncertainty estimators other than the paper's CatBoost proxy, and whether sample reweighting or cohort-balanced batches align the minima without storing many checkpoints. Report both aggregate RMSE and cohort curves: an objective can improve the headline score while leaving the worst mismatch intact.