The benchmark score that moves 42 points without changing one answer

A financial language model writes exactly the same seven answers. One evaluator says four models lose to a constant predictor; another says those same four beat it. No weights, prompts, generations, or labels changed. Only the rule that extracts a class name from the generated text did.

Cao, Gang, and Veluri audit FLARE, the task suite behind the PI XIU and FinBen financial-LLM benchmarks. Their useful mechanism is a candidate set: collect every allowed label that appears in a generation, then ask how much freedom the unreported extraction rule has. The paper's most important caveat is easy to miss: zero ambiguity is not automatically good. A model that names no label also earns a zero.

FLARE turns classification into generation: a transformerwrites free text, and a harness searches that text for an allowed label. The benchmark path checks labels in dataset-card order and returns the first one found anywhere. The lm-evaluation-harness path returns the label whose match appears first in the text.

Suppose the choices are [negative, neutral, positive] and the model writes “Positive, not negative.” Card order extracts negative; text order extracts positive. Both outputs are faithful to their rules. The benchmark score is therefore partly a property of an undocumented parser.

For generation gᵢ and choice set C, define Sᵢ as the labels in C that occur in gᵢ after lowercasing. Coverage is the fraction with |Sᵢ| > 0. The ambiguity rate M is the fraction with |Sᵢ| ≥ 2. Any admissible rule must choose from Sᵢ and may abstain only when it is empty, so two such rules can differ in accuracy by at most M.

With the answer key, call an ambiguous item contested when its gold label is also in Sᵢ. The contested rate M* is exactly the width of the reachable accuracy interval. An ambiguous item whose candidates all miss the gold is wrong under every admissible rule, so M* can be smaller than M.

def diagnostics(generations, gold, choices):
    covered = ambiguous = contested = forced_correct = 0
    for text, answer in zip(generations, gold):
        text = text.lower()
        candidates = [c for c in choices if c.lower() in text]
        covered += len(candidates) > 0
        ambiguous += len(candidates) >= 2
        if len(candidates) >= 2:
            contested += answer in candidates
        elif candidates and candidates[0] == answer:
            forced_correct += 1
    n = len(generations)
    return {
        'coverage': covered / n,
        'ambiguity': ambiguous / n,
        'contested': contested / n,
        'reachable': (forced_correct / n, (forced_correct + contested) / n),
    }

The code is a from-scratch transcription of that counting argument. It does not train a model or run a second evaluator. The lower endpoint counts unambiguous correct items; the upper endpoint adds contested items. Both are divided by the full test-set size, matching the paper's convention that an unparsed item is wrong.

Qwen2.5-7B is the clean control: full coverage, no ambiguity, and 0.7639 under both rules. The instruction-tuned 14B model is the opposite: its admissible interval reaches from 0.3258 to 0.8113, straddling the majority floor. Its score cannot establish whether it beats a constant predictor without naming the parser.

A majority-class floor is the accuracy of always predicting the most common label. It is not a sophisticated baseline; that is precisely why failing to beat it matters. Of 192 sweep cells, the paper reports 15 clearing their floor, 24 tying, and 153 below. On the three binary tasks, none of 144 cells clears.

The sharper failure is a default label. On German credit, a model with zero parseable outputs scores 0.0000 under the ordinary classification path. The shipped task override supplies the majority label on almost every item and reports 0.6600 on the same generations. That is not recovered model knowledge; it is the benchmark filling in an answer.

The bound applies to rules that select a label actually present in the generation. A bare-letter fallback, synonym table, LLM judge, or default label can return something outside Sᵢ. The paper gives a brutal example: a certified zero-ambiguity German-credit cell scores 0.0000 under every admissible rule but 0.6550 under a shipped bare-letter fallback, which mistakes feature codes such as A11 for answer A.

So “certified” never means globally safe. It means invariant within a declared rule family, at a declared prompt arm and generation budget. The family is an assumption, not a theorem about every conceivable evaluator.

The practical specification is compact: record prompt formatting and generation budget; report coverage beside accuracy; report M and the reachable interval; and show the majority floor, stratified when the dataset declares subgroups. None requires generating a second answer.

Next, test semantic extractors against human labels, include rules outside the substring family, and measure whether conclusions survive exact-answer prompting. For benchmark users, the immediate question is simpler: if the score moves, did the model change—or only the function that reads it?