The sleep-apnea detector with 29 missing ROC curves

A mattress can listen to breathing without a wearable. Tiny pressure changes—called a ballistocardiogram, or BCG—carry respiratory motion, heartbeat, posture, and movement. Campero Jurado and colleagues turn those mixed signals into a strong sleep-apnea event detector.

The headline is excellent: Histogram Gradient Boosting reaches 0.969 ROC AUC while holding out one patient at a time. The evaluation detail that should travel with that number is less prominent: 29 held-out patients had no clean windows, so their ROC curves did not exist.

The authors record 512 pressure sensors at 50 Hz. They sum the whole mat and three body regions, average five sensors with strong respiratory periodicity, and build one adaptively preprocessed channel. That last channel high-pass filters drift, selects sensors by autocorrelation and spectral concentration, flips sensors with opposite polarity, and averages them.

Normal adult breathing concentrates pressure variation near 0.1–0.4 Hz—roughly 6–24 breaths per minute. The implementation computes a discrete Fourier transform directly, squares each complex magnitude, and integrates power inside that band. No signal-processing or ML library is involved.

In the clinical data, breathing-band power alone receives 30.3% of total Random Forest impurity importance across six channels. Six FFT shape descriptors on the preprocessed channel add 15.1%. Simple area and curve-length features supply another 21.5%.

ROC AUC has a wonderfully concrete meaning: choose one positive and one negative example at random; AUC is the probability that the model ranks the positive higher, with half credit for a tie. The tested implementation below uses that definition directly.

def roc_auc(labels, scores):
    pos = [s for y, s in zip(labels, scores) if y == 1]
    neg = [s for y, s in zip(labels, scores) if y == 0]
    if not pos or not neg:
        return None
    wins = 0.0
    for p in pos:
        for n in neg:
            wins += 1.0 if p > n else 0.5 if p == n else 0.0
    return wins / (len(pos) * len(neg))

The four-example toy scores produce AUC 1. But when a held-out patient contributes only respiratory-event windows, there is no negative to pair with a positive. The result is therefore null, not zero, one, or 0.5.

The title says spectral features “dominate,” but the ranking uses Random Forest mean decrease in impurity (MDI). Correlated versions of the same breathing-band feature appear across six channels, and continuous features offer many split candidates. MDI can distribute or inflate credit under both conditions. The authors explicitly warn that it can favor high-cardinality features and does not measure causal importance.

That distinction matters for the proposed compact detector. “These 48 features receive 67% of MDI” does not prove that removing the other 143 preserves 90% of predictive performance. A grouped ablation—retraining without each family inside every patient-held-out fold—would answer the engineering question directly.

• Publish each fold's class counts and the exact denominator for every metric.
• Report macro averages over metrics defined for one-class folds, plus pooled patient-level confusion counts.
• Replace MDI-only ranking with grouped permutation tests and nested leave-group-out retraining.
• Separate obstructive, hypopneic, central, and mixed events instead of pooling them.
• Repeat across homes, mattress types, body positions, and co-sleeping conditions.

The Random Forest and LightGBM pages explain the two nonlinear model families behind the result. The broader lesson is about clinical metrics: before admiring an average, count who was mathematically able to enter it.