← Blog/blog/offline-rl-action-outcome-coupling

The offline-RL log that can pass overlap and still be useless

Before training an offline reinforcement-learning system, ask a more basic question: do different logged actions produce meaningfully different expected outcomes? Dang and colleagues call that quantity action–outcome coupling. If it is tiny, better overlap, a larger neural network, and a sharper evaluator cannot create policy value.

The paper turns coupling into a ceiling measured in dollars per decision. Its most important warning comes next: a contextual learner that chooses an action separately in many state cells can look confidently profitable on completely action-independent data when the same rows choose and evaluate the policy.

01

Measure the response surface, not model skill

Let μ(s, a) be the expected reward after taking action a in state s. Under a reference action design, coupling κ averages the within-state variance of μ(s, a). κ = 0 means every supported action has the same expected outcome at almost every state. This is a property of the decision problem and reference design—not of a particular learner.

def action_outcome_coupling(states):
    coupling = 0.0
    for state in states:
        mean = sum(state.action_means) / len(state.action_means)
        variance = sum((value - mean) ** 2 for value in state.action_means) / len(state.action_means)
        coupling += state.weight * variance
    return coupling

def uniform_policy_ceiling(coupling, actions):
    return sqrt((actions - 1) * coupling)

For K uniformly weighted actions, the paper bounds the best contextual policy's improvement over the uniform reference by √((K − 1)κ). The inequality works in only one direction: a ceiling below implementation cost is a reason to stop; a large ceiling is not permission to deploy.

02

Marginal action means can erase the opportunity

In the hand-verified example below, action A pays 2 in state 1 and 0 in state 2; action B does the reverse. Each action's marginal mean is 1, so a fixed-rule comparison finds no signal. Yet choosing by state gains 1 unit. The full coupling is 1.0, the main-effect coupling is 0.0, and both the contextual gain and sharp two-action ceiling are 1.0.

coupling κvalue ceiling
A deterministic two-state illustration. Scaling state-specific action contrasts by x makes coupling grow as x², while the uniform-policy value ceiling grows linearly. Values are computed by the pure core module; they are not paper data.
03

Reuse turns local selection into fake value

Suppose actions truly have no effect. In each state cell, sampling noise still makes one action's fitted mean larger. A greedy learner selects that lucky action. Evaluating it on the same rows then reuses the same positive noise. With one global comparison, the asymptotic false-licence rate stays at the nominal 5%. Repeating selection across cells compounds the optimizer's curse.

Paper-reported limiting false-licence rates at nominal α = 0.05 under balanced two-action logging. Rates come from 400,000 draws of the paper's limiting statistic; the core module performs the exact table lookup used by this chart.

At 10 state cells the false approval rate is 82.3%; at 30 it rounds to 100%. Even with only 5% probability on the rarer action, the paper reports 98.2% at 30 cells. Cross-fitting—choosing on one fold and evaluating on an independent fold—returns the rate to the nominal level in the theorem's setting.

04

The real logs expose three different failure modes

Corpus / rewardAudit findingPaper verdict
OBD randomised e-commerce (click)fixed-rule coupling clears threshold; 26/30 seeds certifyTier 3 [economic flag]
DataCo logistics (on-time)target is an identity of the action in 97.55% of rowsTier 0: refuse
Olist marketplace (review)coupling stays below threshold; learner certifies 29/30 seedsTier 0: refuse
Paper-reported audit outcomes. Tier 3 permits deployment only inside the named policy class; the economic flag still warns that the upper-bound business case is unfavorable under the paper's cost inputs.
05

The audit is a routing rule, not a theorem of deployability

The proposed ladder routes each dataset–reward–reference-design–policy-class combination to Refuse, Predict, Shadow, or Deploy. It first rejects malformed targets and missing identification, checks whether coupling survives richer state adjustment, compares it with a calibrated detectability threshold, and only then considers a disjoint-fold learner certificate and economics.

Several steps remain design choices. The “learnability frontier” is explicitly a calibrated conjecture, not a theorem. State binning changes both measured coupling and its threshold. The three applications are single-step logs, so the sequential continuation-coupling condition is proposed but not exercised. And the ceiling is partial-equilibrium: capacity or congestion can make one order's action change another order's outcome.

06

What to probe next

Pre-register the reward, state resolution, reference action design, and policy class. Run coupling before tuning a learner. Use an honest split or rolling origin for every fit-and-evaluate loop, repeat the entire pipeline under within-stratum action permutations, and report both the monetary ceiling and the achieved fraction of it. If coupling is below detection, spend the next budget on controlled exploration—not on another architecture.

The site's decision-tree, random-forest, and XGBoost pages explain the flexible learners that can model state–action interactions. This paper supplies the prior question: whether those interactions represent an actionable response at all.

References

  1. Dang, T., Phung, T. H., Nguyen, S. L., & Nguyen, T. (2026). Action-outcome coupling: a pre-investment feasibility audit for offline reinforcement learning in supply chain operations. arXiv:2611.22921 [cs.LG]
  2. Chernozhukov, V., Chetverikov, D., Demirer, M., et al. (2018). Double/debiased machine learning for treatment and structural parameters. The Econometrics Journal, 21(1), C1–C68