← Blog/blog/ecmo-balance-score-panel-dependence

The safer policy score that changes when a new rival enters

A reinforcement-learning system for life support should not discover a clever action only because the historical record barely contains it. Darvishvand and colleagues tackle that problem for pediatric extracorporeal membrane oxygenation, or ECMO: a machine temporarily oxygenates blood for critically ill children while clinicians adjust circuit and ventilator controls.

Their method has a strong, auditable idea. Improve on clinicians only where the data contain enough similar state–action examples; elsewhere, retain the empirical clinician policy. But the single number used to summarize “performance plus safety” has a quieter dependency: its scale is rebuilt from every comparator in the experiment.

01

Constrain improvement where evidence is thin

Offline reinforcement learning learns only from logged trajectories; it cannot try actions and observe the result. Standard fitted Q-iteration, or FQI, repeatedly regresses Bellman targets and chooses the action with the largest estimated long-run value. Sparse actions are dangerous because extrapolation error can make them look better than anything clinicians commonly did.

The authors cluster training states, count each action inside each cluster, and call an action supported when its count reaches threshold N∧. Unsupported actions keep their clinician probability. The total clinician probability on supported actions moves to the supported action with the largest Q estimate. If none is supported, the complete clinician distribution is returned unchanged.

def constrained_policy(baseline, q, counts, threshold):
    supported = [i for i, count in enumerate(counts) if count >= threshold]
    if not supported:
        return baseline[:]
    best = max(supported, key=lambda i: q[i])
    mass = sum(baseline[i] for i in supported)
    return [baseline[i] if i not in supported else mass if i == best else 0
            for i in range(len(baseline))]

def balance(value, disagreement, values, disagreements):
    v = (value - min(values)) / (max(values) - min(values))
    a = (max(disagreements) - disagreement) / (max(disagreements) - min(disagreements))
    return 0 if v + a == 0 else 2 * v * a / (v + a)
02

A three-action audit you can do by hand

In this deterministic illustration, clinician probabilities are [0.2, 0.5, 0.3], Q estimates are [9, 4, 7], and counts are [2, 12, 15] with threshold 10. Action A has the tempting Q value 9 but lacks support, so its 0.2 probability is frozen. The supported mass 0.8 moves from B and C to C, the better supported action. The projected value is 7.4.

Support-constrained action probabilities from the pure core function. This is a hand-checkable illustration, not patient data or a paper-reported result.
03

How the Balance Score is assembled

Fitted Q-evaluation, or FQE, estimates a candidate policy's value from held-out trajectories using another fitted Bellman recursion. Disagreement is the fraction of held-out states where the policy's most likely action differs from the recorded action. The paper min–max normalizes FQE upward and disagreement downward, then takes their harmonic mean. A near-zero axis drags down the combined score.

The catch is in those min and max terms. For a toy panel with a safe low-value endpoint and a risky high-value endpoint, the focal policy has normalized value 0.600, conservatism 0.714, and score 0.652. Add a rival with value 20 and the same maximum disagreement: none of the focal policy's raw numbers change, yet its score falls.

same policy's Balance Score
Comparison-set sensitivity of one unchanged toy policy. Panel 1 contains value endpoints 0 and 10; Panel 2 adds a value-20 rival. The focal policy remains value 6, disagreement 0.3. Illustrative deterministic values.
04

What the 78-patient study reported

The retrospective cohort contains 78 pediatric ECMO trajectories and more than 6,000 hours. Four independently modelled controls are encoded as increase, decrease, or stay the same. Patient-level three-fold cross-validation keeps each child out of their training fold; support uses 100 training-fold clusters, and both FQI and FQE use boosted-tree regressors.

Unconstrained FQI had the highest estimated value and by far the largest disagreement. Imitation learning stayed closest to clinicians but had the lowest estimated value. At threshold 5, FQI-SC sat between them: 1.28 FQE units above imitation learning and 71% less disagreement than FQI. Its reported 0.65 Balance Score was 38% above FQI's 0.47 after rounding.

PolicyAverage FQE ↑Disagreement ↓Balance Score ↑
FQI−0.18 ± 0.280.680.47
Imitation learning−4.32 ± 0.560.060.50
FQI-SC (N∧ = 5)−3.04 ± 0.200.200.65
Paper-reported aggregate results (Table 2). FQE is mean ± standard deviation across head–fold evaluations. The authors compute normalization bounds over the full policy–head–fold panel, so the aggregate rows alone cannot reproduce the scores.
05

What not to conclude

This is not evidence that an algorithm should control ECMO. FQE can be biased when its model is misspecified, and the reward combines clinician-specified physiological ranges with a sparse neurological injury component. The cohort is small, retrospective, single-setting, and excludes children with congenital heart disease. Unmeasured confounding can affect both clinician actions and outcomes.

Count support is also representation-dependent: K-means with 100 clusters decides which continuous states are “similar,” and the four controls are optimized independently rather than as one joint action. A supported action is merely frequent enough under this partition. It is not certified safe, and closeness to historical practice can preserve historical biases.

06

What to probe next

Report the two raw axes and their uncertainty before any scalar score. Freeze clinically meaningful normalization anchors in advance, then rerun the ranking under alternative anchors and with new comparators. Stress-test cluster count, support threshold, reward weights, and FQE models; add external hospitals and clinician review before contemplating prospective evaluation. The site's XGBoost page explains the tree ensemble used for value fitting, while decision trees provide the from-scratch building block.

References

  1. Darvishvand, F. G., Mathur, S., Prabhakar, N., Skinner, M. A., Soni, A., Shah, N., Sanford, E. L., Raman, L., & Natarajan, S. (2026). Support-Constrained Offline Policy Improvement for Pediatric ECMO from Limited Clinical Data. World Models for High-Stakes Health / arXiv:2611.17134
  2. Laroche, R., Trichelair, P., & Tachet des Combes, R. (2019). Safe Policy Improvement with Baseline Bootstrapping. ICML