The ICU model that mistakes measurement for physiology

A lab value records a patient. The decision to order that lab also records the hospital. In intensive care, frequent lactate or creatinine checks may signal clinical concern even before the values themselves change. A model can exploit that pattern—and silently learn a local monitoring policy alongside physiology.

Pike separates those signals in ICU mortality prediction. Physiological trajectories transfer from one hospital system to another. Features describing when and how often clinicians measured the patient help locally, but their advantage vanishes after transfer and reverses when measurement availability changes. Crucially, the timing experiments hold observation count fixed.

The target is in-hospital mortality predicted 48 hours after ICU entry. R1 is a latest-value snapshot. R2 adds physiological severity, trend, and variability summaries: whether a value is worsening, recovering, or unstable. R3 adds the observation process: measurement frequency, timing, and time since the last observation.

The tiers are nested, but they are not a controlled one-feature intervention. R1, R2, and R3 contain 42, 334, and 514 shared features. Adding a tier changes interactions and regularisation as well as adding information, a limitation the paper states explicitly.

import math

def binary_log_loss(label, prediction):
    p = min(1 - 1e-15, max(1e-15, prediction))
    return -(label * math.log(p) + (1 - label) * math.log(1 - p))

def paired_log_loss_difference(labels, candidate, baseline):
    candidate_loss = sum(binary_log_loss(y, p) for y, p in zip(labels, candidate)) / len(labels)
    baseline_loss = sum(binary_log_loss(y, p) for y, p in zip(labels, baseline)) / len(labels)
    return candidate_loss - baseline_loss

def milli_loss(difference):
    return 1000 * difference

The site runs the TypeScript version above; Python and C++ are faithful translations. The tested core returns ln(2) for a 50% forecast, 0.10536 for a correct 90% forecast, and a negative candidate-minus-baseline difference when the candidate is better. Charts express small differences in thousandths of log loss without rounding the stored values.

The core check confirms the contrast: the local eICU interval clearly excludes zero (yes), while the transferred R3 interval does not (no). Relative to R1, R2 reduces eICU loss by 3.06%. R3 then reduces R2 loss by only 0.04%.

The key experiment removes observations and rebuilds every derived feature. For each patient and variable, timing comparisons retain the same number of measurements but select them randomly, prefer recent measurements, or spread them through the window. A separate test removes the final 12 hours and compares it with random removal of the same count.

Spreading observations through the window improves R3 by 0.03280, versus 0.00703 for R2. Removing the final 12 hours harms R3 by 0.01947, versus 0.00822 for R2. The observation-pattern representation is not merely sensitive to how much data remains; it is especially sensitive to where the data sits in time.

On complete local eICU records, R3 improves on R2 by −0.00257. At 50% retention the gain nearly disappears (−0.00042). At 25% retention it reverses to +0.00326, and a final-12-hour outage widens the harm to +0.01081. This is the post's central result: a predictive workflow signal can be locally real and operationally fragile at the same time.

Retraining R3 on 25%-retained records restores its advantage under that restriction: loss falls by 0.01509 relative to complete-record training, and beats a similarly retrained R2 by 0.00336. But the restricted model is 0.01609 worse when evaluated on complete records. Matching the update to one regime trades away performance in another.

A calibrated blend of R2 and R3 responds more smoothly. As availability falls, the fitted coefficient on R3 drops from 0.503 on complete records to 0.325 at 50% and 0.199 at 25%. Recalibration recovers a modest R3 advantage under random thinning, but it cannot reverse the final-12-hour outage: R3 remains worse by +0.00392.

Start with a simple logistic-regression model whose coefficients can be audited, then compare snapshot, trajectory, and observation-process groups under hospital-grouped validation. Add ashallow XGBoost-style model only after the measurement-shift tests are fixed. The paper's penalised logistic regression exception is a reminder that the conclusion is not architecture-free.

Report performance across a grid of observation regimes, not one external cohort: random thinning, terminal outages, delayed labs, and site-specific schedules. Track calibration and fixed alert capacity alongside log loss. Finally, prospectively define which monitoring changes are plausible at deployment. A model that uses the act of measurement must be versioned with the workflow that gives that act meaning.