The energy-pricing DQN that cannot change tomorrow

A reinforcement-learning agent sounds well matched to an electricity market: quote a price now, observe what happens, and learn how today's decision changes tomorrow. But in Benalcazar and colleagues' simulator, the solar output, household demand, and battery schedule are fixed before pricing begins. The agent changes money flows, not the next physical state.

That turns the paper into a useful lesson about choosing the smallest learning problem that matches the environment. The Deep Q-Network (DQN) can still learn a state-dependent price, but its sequential machinery cannot discover long-run control effects that the simulator does not contain. Meanwhile three transparent, budget-balanced rules keep more value inside the community and beat every learned policy in the only direct comparison.

A prosumer both consumes electricity and exports surplus rooftop solar. At each hour, internal trade is limited by the smaller of local surplus and residual demand. A buyer will not rationally pay more than the grid import tariff; a seller will not rationally accept less than the grid export tariff. Those two prices form the corridor.

A budget-balanced mechanism quotes one internal price, so every euro paid by a buyer reaches a seller. The learned mechanisms quote separate buy and sell prices. Their difference is retained by the market operator and counted as a community cost.

def settle(supply, demand, grid_buy, grid_sell, quote_buy, quote_sell):
    valid = (grid_sell < quote_sell <= quote_buy < grid_buy)
    volume = min(supply, demand) if valid else 0.0
    buyer_payment = volume * quote_buy
    seller_revenue = volume * quote_sell
    operator_spread = buyer_payment - seller_revenue
    saving = volume * (grid_buy - grid_sell) - operator_spread
    return volume, operator_spread, saving

The rule earns €0.60 of community saving. The two-price quote earns €0.40 and sends €0.20 to the operator. No neural-network prediction can beat the balanced rule on the same physical exchange unless it changes how much energy clears.

The supply–demand ratio (SDR) is local surplus divided by local demand. The learned variant does not output an arbitrary price. It chooses a sensitivity parameter for a bounded curve based on the previous hour's SDR, keeping prices inside the grid corridor. The fixed control uses one sensitivity pair; the DQN chooses among nine.

This structure explains the strongest within-family result. RL-SDR-L and the fixed SDR control clear exactly the same 7,131 kWh in the solar-only case. Learning reduces the annual internal spread from €210.46 to €92.49. The €117.97 reduction equals their community-savings difference exactly. The DQN improves settlement, not energy routing.

A DQN estimates an action value as immediate reward plus discounted future value. Here, the action does not affect the next load, solar output, battery state, or hour. For a fixed current state, every candidate action therefore receives the same future-value addition. Adding the same number cannot change which action is largest.

The greedy index is 1 before bootstrapping and 1 after adding a discounted future value of five. In this environment, the DQN is functionally solving a contextual bandit: observe the current context, choose one price action, receive an immediate reward. That does not make the learned mapping invalid. It makes long-horizon credit assignment unnecessary.

RL-Multiplier looks especially weak, but its action space carries much of the blame. Of the 3,197 kWh it fails to settle, 1,768 kWh occurs when no candidate multiplier pair can produce a valid spread at the prevailing tariffs. Another 1,429 kWh is lost when a feasible pair exists but the learned policy selects an inverted one. Architecture and action design are entangled in the result.

Adding fixed battery schedules raises RL-SDR-L's reported saving from €734.23 to €978.52. But the common reference cost excludes batteries, while the battery-equipped baseline already costs €105.76 less before peer-to-peer settlement. Charging energy was committed upstream and is not debited from simulated solar surplus or imports.

The core subtracts €4,642.05 from the €4,747.81 no-battery reference, giving €105.76 before settlement. The remaining €872.76 is the internal-market component. The authors correctly avoid comparing that combined bar with a rule-based storage result because none was run; their budget-balance argument predicts the rules would retain the advantage on the same fixed exchange.

A stronger follow-up would compare a contextual bandit, supervised action-value regression, and the DQN on held-out weather and load years; rerun every policy across seeds; give all mechanisms the same storage trajectory; debit charging energy; and report operator revenue separately from participant welfare. If prices are later allowed to affect battery dispatch or demand response, sequential RL becomes substantively useful rather than merely a solver choice.

The implementation bridge is the neural-network page: the DQN's two ReLU layers are ordinary function approximation. The deeper lesson is upstream of the network. Before reaching for RL, ask whether today's action changes tomorrow's state. If it does not, test the simpler bandit and rule-based baselines first.