← Blog/blog/oracle-feature-acquisition-recoverability-ceiling

The oracle that can only be copied halfway

Imagine a doctor deciding which test to order next. Buying every measurement wastes money, time, and sometimes privacy. Active feature acquisition asks a model to choose the next measurement separately for each patient. Meng and colleagues make that choice teachable: during training, an oracle may peek at the true diagnosis and every unbought test, then a student learns to imitate its ranking using only information available in clinic.

The headline is efficient accuracy. The deeper result is a limit: at the initial free-feature state, the paper estimates that only 29.7%–51.3% of the oracle's next-test choices are recoverable from deployment-visible information. A stronger student cannot reconstruct a distinction that its input does not contain.

01

Turn a diagnosis into a ranking

For each unobserved test, the oracle temporarily adds its true value, runs the fixed downstream predictor, and computes a Brier score: squared distance between the predicted class probabilities and the one-hot true label. Lower is better. A softmax over negative scores turns the candidates into a full ranking target rather than a brittle single winner.

def oracle_target(candidate_probs, label, temperature):
    scores = []
    for probs in candidate_probs:
        score = 0.0
        for index, probability in enumerate(probs):
            target = 1.0 if index == label else 0.0
            score += (probability - target) ** 2
        scores.append(score)
    logits = [-score / temperature for score in scores]
    shift = max(logits)
    weights = [exp(logit - shift) for logit in logits]
    total = sum(weights)
    return [weight / total for weight in weights]

The small example below is illustrative, not paper data. The three candidate tests produce Brier scores 0.335, 0.076, 0.485. Test 2 is best. Temperature controls how much of the near-miss information survives: at low temperature the target approaches one-hot; at high temperature it becomes nearly uniform.

test 1test 2test 3
Core-computed soft teacher targets for three illustrative candidate tests. Horizontal positions are temperatures 0.10, 0.25, 0.50, 1.00, and 2.00.
02

Better teachers help, but labels add less than expected

The paper holds the student architecture and masking recipe fixed, then swaps the teacher. Random targets are worst. An entropy teacher may inspect the unobserved test value but not the label. The Brier teacher may inspect both. Brier wins on all three reported datasets, yet the label-blind entropy teacher closes most of the gap.

Core-computed lift in mean-across-budgets accuracy over the random-teacher student, from the paper's Table 1. Values are proportions: 0.022 equals 2.2 percentage points.
03

The ceiling is a conditional mode

Given the same visible state, the hidden label and hidden test values can make the oracle pick several different actions. From the student's view the teacher is therefore a distribution P(action | visible state). The best deterministic student can select only its most probable action. Averaging that maximum probability across states gives the agreement ceiling.

The paper estimates the unknown conditional distribution with teacher-action agreement among nearby examples. Its appendix reports the bottleneck free-tier state below. “Above chance” is our core-computed normalization: zero is uniform guessing and one is perfect recovery.

DatasetPaid actionsChanceConsistencyAbove chanceUnrecoverable
Cube110.0910.2970.2270.703
Adult120.0830.3850.3290.615
Heart110.0910.4180.3600.582
MiniBooNE110.0910.4280.3710.572
PhysioNet90.1110.4440.3750.556
Glioma60.1670.5130.4160.487
Paper-reported kNN teacher consistency and action counts, with derived chance-normalized and unrecoverable shares. Consistency is an empirical diagnostic, not a proved universal bound.
Paper-reported estimated recoverability at the initial free-feature state. Even the highest bar leaves 48.7% of teacher actions unexplained by visible information.
04

Why another Transformer barely moves the limit

The authors try a wider MLP, a Transformer, cross-attention, a different downstream predictor, changed costs, and even an in-sample student trained on the test fold's labels. Architectures move mean accuracy by roughly one point at most, while in-sample training closes only 16%–54% of the oracle gap. That pattern supports an information bottleneck more than an optimization failure.

05

The oracle is greedy and the budget is local

Each teacher score looks only one acquisition ahead. That is clean and cheap, but a test with little immediate value may unlock a valuable second test. The paper notes that lookahead gains are dataset-dependent and leaves trajectory-level teachers for future work. Deployment also has no STOP action: unused budget cannot be transferred to another patient. Neither choice is neutral in a hospital with shared capacity.

06

What to probe next

Compare three teachers under a shared global testing budget: the paper's myopic Brier oracle, a two-step cost-sensitive oracle, and a teacher allowed to stop. Report predictive utility, calibration, subgroup test burden, and the same recoverability diagnostic. If the stronger teacher becomes less imitable, better privileged decisions may paradoxically create a worse deployed policy.

The student is a small mask-conditioned network; the site's neural-network page exposes that computation. The candidate ranking is also a multiclass probability problem, so the logistic-regression page is a useful simpler bridge to softmax and cross-entropy.

References

  1. Meng, Y., Ding, Z., Grossman, R. L., & Chen, Y. (2026). Active Feature Acquisition by Distilling an Oracle Teacher. arXiv:2611.26598 [cs.LG]
  2. Brier, G. W. (1950). Verification of Forecasts Expressed in Terms of Probability. Monthly Weather Review, 78(1), 1–3