The 96-model ensemble that gets worse when everyone votes

Test-time compute sounds simple: run a tabular foundation model more times, then combine the answers. Ning and colleagues show why that slogan is incomplete. A pool of 96 configurations improves only after validation labels decide who gets a vote. Let everyone vote equally and error rises instead.

Each candidate is one prediction configuration: a different checkpoint, context construction, view, or adaptation choice. Greedy ensemble selection starts empty. For 40 rounds it tentatively adds every candidate, keeps the one with the lowest validation log loss, and allows repeated picks. Selection frequency becomes the final weight.

def select(labels, members, rounds):
    counts = [0] * len(members)
    total = [0.0] * len(labels)
    for step in range(rounds):
        losses = []
        for member in members:
            trial = [(s + p) / (step + 1) for s, p in zip(total, member)]
            losses.append(log_loss(labels, trial))
        best = min(range(len(losses)), key=lambda i: losses[i])
        counts[best] += 1
        total = [s + p for s, p in zip(total, members[best])]
    return [count / rounds for count in counts]

This is supervised fitting even though no model parameters move. The labels used by the selector therefore belong in the statistical budget, not merely in an engineering footnote.

The most important table is not the headline score. The authors rerun selection with fractions of the validation set. With only one eighth of the labels, every tested pool size remains worse than the default model. A larger pool shrinks the damage, but it does not cross zero. Only at one quarter does the 16-member pool become marginally useful.

The next chart is deterministic and illustrative, not a reproduction of a paper dataset. One candidate is reliably moderate. Another is nearly perfect on the first four validation rows and confidently wrong later. A third is cautious. The selector sees prefixes of 2, 4, 8, 12, and 16 labels; its weights are then scored on the same fixed 16-row evaluation set. The small-prefix validation score looks excellent precisely when the evaluation score is worst. The equally weighted toy ensemble has log loss 0.583.

Cheap native views deliver +21 Elo in 1.66 GPU-hours. The 96-configuration selection sweep delivers +67 Elo but consumes 247.5 GPU-hours; combining mechanisms reaches +93 Elo at 253.1 GPU-hours. A separate frozen TabFM baseline scores higher than that combined system while using 19.3 GPU-hours. Test-time scaling is therefore a mechanism-and-budget choice, not a generic promise that more inference wins.

Gains are uneven too: across methods, the five largest positive dataset gains account for 46–70% of all positive error reduction. The paper's attempt to predict which datasets benefit reaches cross-validated R² of only 0.012 at best. The average gain is real, but advance routing remains unreliable.

Reserve a second, untouched audit set for choosing the selector itself; plot gain against validation examples rather than fractions; compare greedy selection with shrinkage toward uniform or the default; and report compute-adjusted regret when candidate generation and label acquisition share one budget. Finally, test whether the chosen weights survive dataset shift.