Before training an offline reinforcement-learning system, ask a more basic question: do different logged actions produce meaningfully different expected outcomes? Dang and colleagues call that quantity action–outcome coupling. If it is tiny, better overlap, a larger neural network, and a sharper evaluator cannot create policy value.
The paper turns coupling into a ceiling measured in dollars per decision. Its most important warning comes next: a contextual learner that chooses an action separately in many state cells can look confidently profitable on completely action-independent data when the same rows choose and evaluate the policy.
Measure the response surface, not model skill
Let μ(s, a) be the expected reward after taking action a in state s. Under a reference action design, coupling κ averages the within-state variance of μ(s, a). κ = 0 means every supported action has the same expected outcome at almost every state. This is a property of the decision problem and reference design—not of a particular learner.
def action_outcome_coupling(states):
coupling = 0.0
for state in states:
mean = sum(state.action_means) / len(state.action_means)
variance = sum((value - mean) ** 2 for value in state.action_means) / len(state.action_means)
coupling += state.weight * variance
return coupling
def uniform_policy_ceiling(coupling, actions):
return sqrt((actions - 1) * coupling)For K uniformly weighted actions, the paper bounds the best contextual policy's improvement over the uniform reference by √((K − 1)κ). The inequality works in only one direction: a ceiling below implementation cost is a reason to stop; a large ceiling is not permission to deploy.
Marginal action means can erase the opportunity
In the hand-verified example below, action A pays 2 in state 1 and 0 in state 2; action B does the reverse. Each action's marginal mean is 1, so a fixed-rule comparison finds no signal. Yet choosing by state gains 1 unit. The full coupling is 1.0, the main-effect coupling is 0.0, and both the contextual gain and sharp two-action ceiling are 1.0.
Reuse turns local selection into fake value
Suppose actions truly have no effect. In each state cell, sampling noise still makes one action's fitted mean larger. A greedy learner selects that lucky action. Evaluating it on the same rows then reuses the same positive noise. With one global comparison, the asymptotic false-licence rate stays at the nominal 5%. Repeating selection across cells compounds the optimizer's curse.
At 10 state cells the false approval rate is 82.3%; at 30 it rounds to 100%. Even with only 5% probability on the rarer action, the paper reports 98.2% at 30 cells. Cross-fitting—choosing on one fold and evaluating on an independent fold—returns the rate to the nominal level in the theorem's setting.
The real logs expose three different failure modes
| Corpus / reward | Audit finding | Paper verdict |
|---|---|---|
| OBD randomised e-commerce (click) | fixed-rule coupling clears threshold; 26/30 seeds certify | Tier 3 [economic flag] |
| DataCo logistics (on-time) | target is an identity of the action in 97.55% of rows | Tier 0: refuse |
| Olist marketplace (review) | coupling stays below threshold; learner certifies 29/30 seeds | Tier 0: refuse |
The audit is a routing rule, not a theorem of deployability
The proposed ladder routes each dataset–reward–reference-design–policy-class combination to Refuse, Predict, Shadow, or Deploy. It first rejects malformed targets and missing identification, checks whether coupling survives richer state adjustment, compares it with a calibrated detectability threshold, and only then considers a disjoint-fold learner certificate and economics.
Several steps remain design choices. The “learnability frontier” is explicitly a calibrated conjecture, not a theorem. State binning changes both measured coupling and its threshold. The three applications are single-step logs, so the sequential continuation-coupling condition is proposed but not exercised. And the ceiling is partial-equilibrium: capacity or congestion can make one order's action change another order's outcome.
What to probe next
Pre-register the reward, state resolution, reference action design, and policy class. Run coupling before tuning a learner. Use an honest split or rolling origin for every fit-and-evaluate loop, repeat the entire pipeline under within-stratum action permutations, and report both the monetary ceiling and the achieved fraction of it. If coupling is below detection, spend the next budget on controlled exploration—not on another architecture.
The site's decision-tree, random-forest, and XGBoost pages explain the flexible learners that can model state–action interactions. This paper supplies the prior question: whether those interactions represent an actionable response at all.
References
- Dang, T., Phung, T. H., Nguyen, S. L., & Nguyen, T. (2026). Action-outcome coupling: a pre-investment feasibility audit for offline reinforcement learning in supply chain operations. arXiv:2611.22921 [cs.LG]
- Chernozhukov, V., Chetverikov, D., Demirer, M., et al. (2018). Double/debiased machine learning for treatment and structural parameters. The Econometrics Journal, 21(1), C1–C68