The Bayesian audit that can average violations away

A policy can match Thompson sampling's reward curve without sampling from a Bayesian posterior. Chen and colleagues test that distinction in decision-pretrained transformers: sequence models trained to choose good actions in bandit problems, then adapted to a new problem only through their context. The models often act like posterior samplers. The paper asks whether the mechanism survives two necessary audits.

In a bandit, an agent repeatedly chooses an arm and observes its reward. Thompson sampling draws a possible environment from the Bayesian posterior, then chooses the action that is best in that draw. For Gaussian and Bernoulli arms, the posterior depends on each arm's observation count and reward sum—not the order in which those rewards arrived. Those counts and sums are sufficient statistics: a compressed history containing all information used by the assumed likelihood.

def stats(context, arms):
    counts = [0] * arms
    sums = [0.0] * arms
    for arm, reward in context:
        counts[arm] += 1
        sums[arm] += reward
    return counts, sums

def paired_difference(original, transformed):
    return [p - q for p, q in zip(original, transformed)]

The authors transform a context while preserving those statistics. They either sort action–reward pairs by reward or replace rewards while keeping each arm's total fixed. A genuine posterior sampler should return the same next-action distribution. The paired difference in the code is therefore zero under the exact condition.

Exact invariance is pointwise: every context and its transformed partner should produce equal action probabilities. Testing that equality literally would reject for tiny approximation errors, so the paper uses a paired Hotelling test of whether the average probability difference is zero. This is a deliberate, pragmatic relaxation.

In the actual experiments, this caveat does not rescue DPT. Across Gaussian, Bernoulli, and linear bandits, every one of the 56 reported sufficient-statistic tests rejects at the 5% level. The bars count rejections by domain and deployment mode; they reproduce the paper's test grid rather than a synthetic effect size.

Posterior probabilities obey a second necessary condition. Before seeing more data, the expected future posterior equals the current posterior. Applied to actions, the average next-action probability after model-generated continuations should equal today's probability. This is the martingale property: beliefs may move after evidence, but they should not drift predictably in expectation.

Ordinary DPT predicts actions but not rewards, so it cannot generate the joint continuations needed for this audit. The paper trains DPT′ with an added reward prediction and GDPT as a generative decision transformer over action–reward pairs. Using 5,000 environments or continuations per test, every DPT′ martingale test rejects. GDPT rejects 78 of 80; the only non-rejections are online Bernoulli cases with p-values 0.099 and 0.094.

A transformer with positional encodings can react to a reorder even when a Bayesian posterior would not. The authors therefore remove positional encodings, replace causal attention with bidirectional attention, and mean-pool the context. The ablated DPT becomes exactly invariant to sorting, as its architecture requires. Yet all 12 reward-replacement tests still reject.

Report the paired-difference distribution alongside its mean: maximum gaps, quantiles, and context-conditioned slices would reveal cancellation. Pre-register transformations whose distributions differ from the originals, and test several sufficient-statistic-preserving families. For martingales, separate model misspecification in the reward head from drift in the action head.

The present evidence is intentionally narrow: stateless bandits, specific latent likelihoods, trained model classes, and finite tests. Extending the audit to state-dependent control will require a careful definition of what the posterior conditions on. For the sequence-model mechanics behind these policies, continue with the Transformer walkthrough.