The LLM forecast that pays you whichever return happens

Suppose a language model prices two disjoint stock-return bins at 40.00% and 38.33%, then prices their union at 81.67%. No market outcome is needed to see the contradiction: the union should cost exactly the sum, 78.33%. Buy the two parts and sell the overpriced whole. The event payoffs cancel, leaving money in every possible state.

Andrews and Sarkar turn that idea into a label-free evaluation of probabilistic forecasts. They ask 15 language models about logically related return events and solve a linear program for the largest guaranteed profit. The result is more than a consistency spot-check—but its meaning depends on the event set and the way the questions are presented.

A binary contract costs the stated probability p, pays one if its event happens, and pays zero otherwise. A positive stake buys it; a negative stake sells it. A Dutch book is a portfolio whose net payoff stays positive over every logically possible outcome, exposing forecasts that cannot all come from one probability distribution.

The portfolio buys one-third of each narrow bin and sells one-third of their union. Its gross stake—the sum of absolute positions—is exactly one. When either bin occurs, the matching long pays one-third and the union short loses one-third. When neither occurs, neither side pays. Contract payoffs therefore cancel in all three atoms; only the inconsistent purchase prices remain.

def disjoint_union_book(p_a, p_b, p_union):
    mismatch = p_union - p_a - p_b
    direction = 1.0 if mismatch >= 0.0 else -1.0
    stakes = [direction / 3.0, direction / 3.0, -direction / 3.0]
    incidence = [[1, 0, 0], [0, 1, 0], [1, 1, 0]]
    payoffs = []
    for atom in range(3):
        payoff = sum(stakes[i] * (incidence[i][atom] - [p_a, p_b, p_union][i]) for i in range(3))
        payoffs.append(payoff)
    return stakes, payoffs, min(payoffs)

The minimum bar is the guaranteed profit: 0.01113. The same number is also the smallest maximum adjustment needed to repair these three forecasts. The core moves both narrow-bin probabilities to 0.4111 and 0.3944, and the union to 0.8056.

For many events, the paper builds an incidence matrix: rows are queried events, columns are the finest mutually exclusive outcomes implied by them, and each cell says whether an event occurs in that outcome. The linear program chooses stakes with total absolute size at most one and maximizes the worst column payoff. By duality, that profit equals the sup-norm distance from the forecast vector to the nearest coherent probability distribution.

Mean profit ranges from 0.002067 for GPT-OSS-120B to 0.1994 for Mistral Small 3.2, while Brier scores span only 0.1966 to 0.2157. The rankings still broadly agree—the paper reports a Spearman correlation of 0.9143—but the coherence measure separates the models much more sharply.

The exact three-event score forms a V around the coherent union price. Moving the union quote away from 0.7833 exposes more guaranteed profit, even though the two component quotes stay fixed.

Under the baseline, each event is asked in a separate session. Pairing an event with its complement reduces incoherence; asking for all 14 next-day events in one query drives it almost to the four-decimal rounding floor. Irrelevant context moves the other way: a user's 70% gut estimate or positive emotion increases measured incoherence, sometimes by an order of magnitude.

That is the most actionable result. The model may contain enough local structure to produce a coherent panel when the dependencies are visible, yet fail when users ask isolated questions. A better follow-up would fix one event set, randomize only elicitation structure, test more domains, and compare raw forecasts with a projection onto the coherent set. Accuracy and calibration should remain separate axes: projection can remove an arbitrage without making a forecast true.

The implementation bridge is the Transformer page. The architecture generates each forecast, but coherence is a property of the set of outputs. It cannot be diagnosed from one probability in isolation.