Imagine a doctor deciding which test to order next. Buying every measurement wastes money, time, and sometimes privacy. Active feature acquisition asks a model to choose the next measurement separately for each patient. Meng and colleagues make that choice teachable: during training, an oracle may peek at the true diagnosis and every unbought test, then a student learns to imitate its ranking using only information available in clinic.
The headline is efficient accuracy. The deeper result is a limit: at the initial free-feature state, the paper estimates that only 29.7%–51.3% of the oracle's next-test choices are recoverable from deployment-visible information. A stronger student cannot reconstruct a distinction that its input does not contain.
Turn a diagnosis into a ranking
For each unobserved test, the oracle temporarily adds its true value, runs the fixed downstream predictor, and computes a Brier score: squared distance between the predicted class probabilities and the one-hot true label. Lower is better. A softmax over negative scores turns the candidates into a full ranking target rather than a brittle single winner.
def oracle_target(candidate_probs, label, temperature):
scores = []
for probs in candidate_probs:
score = 0.0
for index, probability in enumerate(probs):
target = 1.0 if index == label else 0.0
score += (probability - target) ** 2
scores.append(score)
logits = [-score / temperature for score in scores]
shift = max(logits)
weights = [exp(logit - shift) for logit in logits]
total = sum(weights)
return [weight / total for weight in weights]The small example below is illustrative, not paper data. The three candidate tests produce Brier scores 0.335, 0.076, 0.485. Test 2 is best. Temperature controls how much of the near-miss information survives: at low temperature the target approaches one-hot; at high temperature it becomes nearly uniform.
Better teachers help, but labels add less than expected
The paper holds the student architecture and masking recipe fixed, then swaps the teacher. Random targets are worst. An entropy teacher may inspect the unobserved test value but not the label. The Brier teacher may inspect both. Brier wins on all three reported datasets, yet the label-blind entropy teacher closes most of the gap.
The ceiling is a conditional mode
Given the same visible state, the hidden label and hidden test values can make the oracle pick several different actions. From the student's view the teacher is therefore a distribution P(action | visible state). The best deterministic student can select only its most probable action. Averaging that maximum probability across states gives the agreement ceiling.
The paper estimates the unknown conditional distribution with teacher-action agreement among nearby examples. Its appendix reports the bottleneck free-tier state below. “Above chance” is our core-computed normalization: zero is uniform guessing and one is perfect recovery.
| Dataset | Paid actions | Chance | Consistency | Above chance | Unrecoverable |
|---|---|---|---|---|---|
| Cube | 11 | 0.091 | 0.297 | 0.227 | 0.703 |
| Adult | 12 | 0.083 | 0.385 | 0.329 | 0.615 |
| Heart | 11 | 0.091 | 0.418 | 0.360 | 0.582 |
| MiniBooNE | 11 | 0.091 | 0.428 | 0.371 | 0.572 |
| PhysioNet | 9 | 0.111 | 0.444 | 0.375 | 0.556 |
| Glioma | 6 | 0.167 | 0.513 | 0.416 | 0.487 |
Why another Transformer barely moves the limit
The authors try a wider MLP, a Transformer, cross-attention, a different downstream predictor, changed costs, and even an in-sample student trained on the test fold's labels. Architectures move mean accuracy by roughly one point at most, while in-sample training closes only 16%–54% of the oracle gap. That pattern supports an information bottleneck more than an optimization failure.
The oracle is greedy and the budget is local
Each teacher score looks only one acquisition ahead. That is clean and cheap, but a test with little immediate value may unlock a valuable second test. The paper notes that lookahead gains are dataset-dependent and leaves trajectory-level teachers for future work. Deployment also has no STOP action: unused budget cannot be transferred to another patient. Neither choice is neutral in a hospital with shared capacity.
What to probe next
Compare three teachers under a shared global testing budget: the paper's myopic Brier oracle, a two-step cost-sensitive oracle, and a teacher allowed to stop. Report predictive utility, calibration, subgroup test burden, and the same recoverability diagnostic. If the stronger teacher becomes less imitable, better privileged decisions may paradoxically create a worse deployed policy.
The student is a small mask-conditioned network; the site's neural-network page exposes that computation. The candidate ranking is also a multiclass probability problem, so the logistic-regression page is a useful simpler bridge to softmax and cross-entropy.
References
- Meng, Y., Ding, Z., Grossman, R. L., & Chen, Y. (2026). Active Feature Acquisition by Distilling an Oracle Teacher. arXiv:2611.26598 [cs.LG]
- Brier, G. W. (1950). Verification of Forecasts Expressed in Terms of Probability. Monthly Weather Review, 78(1), 1–3