Eight LLM samples can cost five times more

Test-time scaling asks a model for several candidate answers and then chooses among them. A common version is self-consistency: sample independent reasoning paths and return the plurality answer. The candidate count N sounds like a complete budget. It is not.

Eight candidates can arrive in one call as a batch of eight, two calls of four, four calls of two, or eight serial calls of one. Kashaniyan and Jannesari hold N at eight and change only that schedule. On A100 GPUs, the serial endpoint consumes 4.64× the gross device energy for Phi-3 and 4.86× for Qwen, while P95 latency grows 5.77× and 6.12×.

Let S = (b₁,…,bC), where bc is the number of candidates in call c. Then N = Σbc, while C records how many times the inference engine is entered. All four schedules below generate eight logical candidates and repeat a 125-token prompt 1,000 logical token-times. Their execution shapes are different.

Why does serial execution lose? A larger batch exposes more independent decoding work to the GPU at once. Smaller calls repeat call overhead and prompt processing while offering less concurrent work. The experiment measures the end-to-end result; it does not separately identify how much each mechanism contributes.

The paper also increases batched N from one to eight. Energy per generated token falls because the batch uses the device more efficiently, yet total joules per query rise because the query asks for more output. Those statements are compatible: a cheaper token can still produce a more expensive request.

The accuracy curve contains a smaller but revealing protocol trap. Ties are resolved by the earliest candidate. With two disagreeing candidates, both have one vote, so the first answer wins. Therefore N=2 is mechanically identical to N=1 for every prompt—not merely similar on average.

In the deterministic toy below, the first two candidates disagree on both prompts. Accuracy stays at 0% from N=1 to N=2, then jumps to 100% at N=3 when a majority forms. The first toy prompt returns A → A → B as N grows from one to three.

Energy is the difference between synchronized NVML cumulative device-energy readings around the measured query interval. It includes generation, answer extraction, and voting, but excludes model loading, warm-up, reporting-time token counting, and final grading. Idle GPU power is not subtracted, and CPU, memory, cooling, and the rest of the node are outside the boundary.

The fixed-N schedules also sample candidates independently. Their mean generated-token volume differs by only 0.8% for Phi-3 and 1.0% for Qwen, and length-stratified analyses preserve the pattern, but this is not a token-identical replay. The authors correctly frame the result as the cost of the complete schedule.

def summarize(schedule, energy_j, latency_s, generated_tokens):
    if not schedule or any(b <= 0 for b in schedule):
        raise ValueError('positive batch sizes required')
    candidates = sum(schedule)
    return {
        'candidates': candidates,
        'calls': len(schedule),
        'max_batch': max(schedule),
        'average_power_w': energy_j / latency_s,
        'candidates_per_s': candidates / latency_s,
        'joules_per_token': energy_j / generated_tokens,
    }

def plurality_vote(answers):
    counts = {answer: answers.count(answer) for answer in answers}
    return max(answers, key=lambda answer: counts[answer])

The practical rule is conditional: for independent candidates, when memory permits, use fewer calls with larger batches. It need not survive unchanged under continuous batching, dependent search, speculative decoding, multi-GPU inference, larger models, or strict serving constraints. The exact ratios come from two small models, A100/V100 GPUs, and Hugging Face batch-scheduled generation.

A useful next experiment would replay identical sampled continuations or controlled token lengths across schedules, then decompose prompt processing, launch overhead, decoding utilization, and idle draw. A serving study should also add concurrent traffic and memory limits—the conditions under which one giant batch may be impossible or harmful to queueing latency.

For the model-side mechanics behind batched attention and decoding, continue with the Transformer walkthrough. The systems lesson is simpler: candidate count describes the logical search; schedule describes how the machine pays for it.