The reasoning probe that predicts the question, not the attempt

Imagine a monitor that sees only the question—not one token of the model's reasoning—and still predicts whether the answer will be correct. That sounds impossible until the benchmark mixes easy and hard questions. A question-level difficulty score can then look like a trajectory detector merely because easy questions produce more correct attempts.

Bulut audits two popular claims about long language-model reasoning: that a decisive breakthrough appears mid-trace, and that an early hidden state predicts which attempt will succeed. Both weaken sharply after matching compute and question identity. The paper's most useful lesson is therefore an evaluation rule: if the deployment decision is about one attempt, the positive and negative examples must come from the same problem.

The area under the receiver operating characteristic curve (AUROC) is the probability that a randomly chosen positive receives a higher score than a randomly chosen negative, with half credit for a tie. That interpretation makes the confound visible: pooled evaluation usually draws the two attempts from different questions. Within-question evaluation draws both from the same question.

def auc(labels, scores):
    positives = [s for y, s in zip(labels, scores) if y == 1]
    negatives = [s for y, s in zip(labels, scores) if y == 0]
    if not positives or not negatives:
        raise ValueError("both labels are required")
    credit = 0.0
    for positive in positives:
        for negative in negatives:
            credit += (positive > negative) + 0.5 * (positive == negative)
    return credit / (len(positives) * len(negatives))

Here is the smallest useful counterexample. An easy question has two passes and one failure; a hard question has one pass and two failures. Give every attempt its question's fixed score: 0.9 for easy and 0.1 for hard. The score contains no attempt information whatsoever.

The toy's pooled AUROC is 0.667, but the within-question value is 0.500—exactly the tie baseline of 0.5. A high pooled number is not false arithmetic. It answers the easier, different question: “Was this probably an easy problem?”

For attempt i, the trace-blind baseline averages the labels of the other attempts on the same problem. Leaving out the current label prevents direct target leakage. But within a question, removing a pass lowers the score while removing a failure raises it. The positive therefore receives the lower score by construction.

The early hidden-state probe reaches pooled AUROC 0.849 on the matched subset. On 22 questions that contain both passes and failures, however, failure-weighted AUROC is 0.496 and pair-weighted AUROC is 0.515. Both confidence intervals cover chance, and all ten tested token anchors from 4 through 512 show the same null pattern. This is no detected gain—not a proof that useful early signals never exist.

The trajectory audit saves prefixes and launches fresh continuations. A prefix-limited event requires those continuations to beat a fresh restart given the same remaining token budget. Without that matched restart, a late prefix receives more total compute and can look causally special even when restarting works just as well.

Of 178 cells, 84 remain unsolved, 43 are instant, 37 are budget-limited, eight terminal, and five have no crossing. Exactly one is prefix-limited. At its saved token 896, all four continuations pass while one of four matched restarts passes, for an advantage of 0.75. That is the interesting exception, not evidence that every long trace contains a reusable breakthrough.

The primary crossing rule labels a prefix successful when at least three of four continuations pass. Even if the true pass probability is only 0.5, that event occurs 5/16 of the time. Requiring six of eight reduces the same false-threshold probability to 37/256, but does not eliminate sampling uncertainty.

At a true rate of 0.5, the two plotted probabilities are 0.313 and 0.145. This is why the paper treats the lone prefix-limited cell as modest evidence and reports sensitivity analyses instead of promoting it into a universal phase transition.

The study is deliberately narrow: two small 4-bit models, one math family, four to eight continuations per checkpoint, and one generation seed. Token matching also does not match floating-point operations, latency, or key-value-cache cost. The public full-precision traces were rescored through a quantized model with 0.906 top-1 fidelity, and their original sampling temperature was unavailable.

• Report pooled and within-problem AUROC side by side.
• Include a question-only or question-ID negative control.
• Match continuation and restart compute at the decision point.
• Increase repetitions before assigning a sharp crossing token.
• Pre-register an equivalence margin if the goal is to claim no effect.

The implementation bridge is the logistic-regression page for scoring and the transformer page for the hidden states. A flexible probe can learn the right separator and still answer the wrong scientific question. Before trusting its AUROC, specify whether the comparison is across problems, within the same problem, or at a fixed compute budget.