When a model says 90%, read closely

When a model outputs “90%,” it is making a promise: of all the times I say 90%, I'm right about nine times in ten. A model that keeps this promise is calibrated. Calibration is separate from accuracy — a weather model that says “70% rain” and is right 70% of those days is perfectly calibrated even though it's wrong 30% of the time. And it matters wherever a number downstream reads the probability rather than just the label: a medical triage threshold, a fraud-review queue, a trading signal sized by conviction, a human deciding whether to trust the machine. A confidently wrong model is worse than an honestly uncertain one.

Guo, Pleiss, Sun, and Weinberger open with an uncomfortable observation. A LeNet from 1998 was well calibrated: its average confidence tracked its accuracy. A ResNet from 2016 is far more accurate — and far more over-confident, piling probability mass near 100% while getting a meaningful fraction of those wrong. The very things that made nets better classifiers (more depth, more width, batch normalization, less weight decay) made their probabilities worse. Progress on accuracy quietly came with a regression in honesty.

To fix miscalibration you first have to score it. The paper's instrument is the Expected Calibration Error. Sort every prediction into bins by its confidence — the [80%, 90%) bin, the [90%, 100%) bin, and so on. In each bin, compare two numbers: the model's average confidence and its actual accuracy. If they match, the bin kept its promise. ECE is the average gap, weighted by how many predictions fall in each bin:

def ece(confidences, correct, n_bins=15):
    # Guo et al. 2017: bin predictions by confidence, then average the
    # gap |accuracy - confidence| weighted by how many land in each bin.
    n = len(confidences)
    acc_sum = [0.0] * n_bins
    conf_sum = [0.0] * n_bins
    count = [0] * n_bins
    for c, ok in zip(confidences, correct):
        b = min(int(c * n_bins), n_bins - 1)   # c == 1 -> last bin
        count[b] += 1
        acc_sum[b] += 1.0 if ok else 0.0
        conf_sum[b] += c
    e = 0.0
    for b in range(n_bins):
        if count[b] == 0:
            continue
        acc = acc_sum[b] / count[b]
        conf = conf_sum[b] / count[b]
        e += (count[b] / n) * abs(acc - conf)
    return e

We can't ship the paper's ImageNet models, so we build a seeded stand-in that has the same disease. It emits calibrated latent logits — probabilities that would keep their promise — and then multiplies every logit by 2×. Scaling logits sharpens the softmax (pushes the winning probability toward 1) without changing which class wins, so the model's accuracy is untouched while its confidence inflates. That is over-confidence in a test tube. On 6,000 examples over 10 classes it lands at 66.2% accuracy but an average confidence of 82.5% — a gap you can see bin by bin.

Our 16.3% is not a toy number: the paper reports a ResNet-110 on CIFAR-100 at 16.53% ECE — the same order of over-confidence, from a real network. The disease is real and the metric catches it.

Here is the paper's surprise. You don't need a new architecture or retraining. Take the trained network's logits z and, before the softmax, divide them all by a single scalar temperature T:

NLL as a function of T is unimodal, so we don't even need gradients — a golden-section search converges in a fraction of a second. This is the whole optimizer:

def nll(logits, labels, T):
    # mean negative log-likelihood of the true labels under softmax(z / T)
    s = 0.0
    for z, y in zip(logits, labels):
        p = softmax([v / T for v in z])
        s += -math.log(max(p[y], 1e-12))
    return s / len(logits)

def fit_temperature(logits, labels, lo=0.05, hi=10.0, iters=100):
    # NLL(T) is unimodal in T; golden-section search needs no gradients.
    phi = (5 ** 0.5 - 1) / 2
    a, b = lo, hi
    c, d = b - phi * (b - a), a + phi * (b - a)
    fc, fd = nll(logits, labels, c), nll(logits, labels, d)
    for _ in range(iters):
        if fc < fd:
            b, d, fd = d, c, fc
            c = b - phi * (b - a); fc = nll(logits, labels, c)
        else:
            a, c, fc = c, d, fd
            d = a + phi * (b - a); fd = nll(logits, labels, d)
    return (a + b) / 2

Fit on our over-confident model, the search returns T = 2.00 — recovering, to two digits, the 2× we used to break it. (That's not luck: NLL is a proper scoring rule, minimized by the true distribution, and our sharpening was exactly a division by 1/2.) Softening the logits by that T pulls the orange line up onto the diagonal:

“Mostly” is doing quiet work in that last sentence, and the paper is careful where its popularizers are not. Two cracks are worth forcing open.

First: ECE is a fragile number. It depends on a binning choice you made, not just on the model. Once a model is roughly calibrated, the leftover ECE is dominated by how many bins you drew. Here is our calibrated model's ECE measured with different bin counts — same predictions, same model, and the “error” more than quintuples just by slicing finer:

Second, and sharper: temperature scaling optimizes NLL, not calibration. It finds the single T that makes the likelihood happiest — which equals good calibration only when the miscalibration is uniform across the input space. Break that assumption and the one knob has to compromise. To show it, we mix two populations: one over-confident (sharpness 3×), one under-confident (sharpness 0.5×), the way an easy subgroup and a hard subgroup differ in a real dataset. We fit one global temperature on the pool and look at each group.

It's worth asking why this is a modern-net problem at all. A plain logistic regression trained by minimizing log-loss is calibrated almost by construction: log-loss is a proper scoring rule, and a linear model doesn't have the capacity to memorize the training set into 99.99% confidence the way an over-parameterized net does. The paper's temperature scaling is, tellingly, just one-parameter Platt scaling — the same trick used to calibrate SVM outputs for decades — reduced to its barest form. The wheel of calibration turns back to logistic regression at the end: temperature scaling literally fits a one-parameter logistic map on top of the network's logits.

• Post-2020 is a different story. Minderer et al. (2021) revisited this and found the trend reversed: the newest models — vision transformers, MLP-Mixer, the non-convolutional ones — are among the best calibrated out of the box, and degrade less under distribution shift. The 2017 “modern nets are miscalibrated” headline is itself now dated; architecture, not just size, drives calibration.
• Measure the tails, not the average. The Fig 4 failure is invisible to a single ECE. Reporting ECE per subgroup, or a metric that penalizes worst-bin rather than average gap, would catch the compromise the global knob makes.
• Let T vary. If uniform miscalibration is what makes one temperature work, heterogeneous miscalibration calls for a temperature that's a function of the input — which is just a small calibration head, trading the one-parameter guarantee for coverage of the groups a scalar can't serve.

The softmax at the end of the neural-net page is the exact operation temperature scaling reaches into — raise its temperature and watch a spiky distribution melt toward uniform. That one division is the difference between a model that knows what it doesn't know and one that doesn't.