The privacy score that rises when a patient has more records

A model can look only mildly vulnerable on average while a small group of patients is unusually easy to identify as training members. Vrettos and colleagues audit that tail across 7,789 medical-AI models trained on six public datasets spanning imaging, electrocardiograms, tabular records, and clinical text. Their patient-level view is the important move. Its numerical meaning needs equal care.

A membership-inference attack asks whether a record helped train a model. For one record, the audit compares confidence from models that trained on it with confidence from models that did not. The attack AUC is the probability that a randomly selected member-model confidence exceeds a randomly selected non-member-model confidence, with half credit for ties. An AUC of 0.5 means chance ranking; 1 means perfect separation.

def record_auc(member_scores, nonmember_scores):
    wins = 0.0
    for member in member_scores:
        for nonmember in nonmember_scores:
            wins += (member > nonmember) + 0.5 * (member == nonmember)
    return wins / (len(member_scores) * len(nonmember_scores))

def patient_risk(record_aucs):
    return max(record_aucs)

The second function is deliberately short: patient risk is the maximum record AUC. That aggregation finds a vulnerable record that a population average can hide. On the ISIC skin-lesion task, the paper reports a population attack AUC of 0.56 while 70.8% of patients cross the patient-level 0.7 threshold.

Under exchangeable continuous scores, a record's Mann–Whitney AUC has an exact finite-sample null distribution. The core computes it by enumerating the rank-sum distribution with dynamic programming. If one record has null probability p of exceeding 0.7, then under an independence illustration a patient with r records has probability 1 − (1 − p)^r that at least one record crosses the line.

On NIH ChestX-ray14, changing only the random seed barely changes the exposed fraction: 88% in the reference run and 89% in the repeat. Strong weight decay reduces it to 30%. Across the study, weight decay cuts the exposed fraction by 1.38× to 2.95×, compared with only 1.07× from reseeding.

Fine-tuning scope matters too. For the ECGFounder task, training only a new prediction head exposes 22.5% of patients; partial fine-tuning reaches 30.4%; full fine-tuning reaches 54.6%. The full model rises from 10.9% at epoch 2 to 54.6% at epoch 20. More adaptation is useful, but it also gives the model more capacity and time to encode patient-specific evidence.

The paper's DataConfessor screen trains four inexpensive proxy models, ranks patients by proxy risk, and flags the top 20% before the expensive reference audit. Across datasets, those flags recover 72.7% to 100% of the eventual top-5% high-risk tail—3.6× to 5× random recovery—in 1.4 to 7.4 GPU-minutes.

First, replace the independent-record illustration with a patient-level permutation that preserves within-patient correlation. Second, report calibrated excess risk with uncertainty, not only the threshold count. Third, test whether atypicality—the paper finds large standardized differences of 1.78 to 4.18—predicts exposure after controlling for record count and subgroup prevalence. That association is useful for triage but is not yet causal.

Finally, compare regularization with a formally private training method at matched clinical utility. The improved ISIC recipe raises held-out AUROC by 0.174 and reduces low-false-positive attack sensitivity 13-fold, yet the authors correctly stop short of claiming a privacy guarantee. For the classifier mechanics behind an attack score, explore the logistic-regression walkthrough.