Eight options can hide one broken learner

An option in hierarchical reinforcement learning is a policy for primitive actions plus a rule for when to hand control back. The usual story is temporal abstraction: learn reusable behaviours that run for several steps. Liu and Jing ask a sharper question. When Option-Critic improves as its option count grows, is duration actually doing the work?

Their answer is no in the benchmark that introduced the method. Forcing every option to terminate after every primitive action preserves the option-count curve. The lower-level policies instead suffer a local exploration failure, and multiple options cover one another's frozen mistakes.

FourRooms is a small gridworld with walls, doorways, and a goal. Because every state can be enumerated, the authors solve the Markov decision process exactly and know which actions are acceptable. A frozen option is necrotic at a state when almost none of its probability mass lies on those acceptable actions.

Each state belongs to one of three classes: every option is necrotic; at least one option is viable but the router selects a necrotic one; or the router selects a viable option. That partition separates a coverage failure from a routing failure.

def audit_coverage(necrotic, selected):
    states, options = len(necrotic), len(necrotic[0])
    single = sum(sum(row) for row in necrotic) / (states * options)
    joint = sum(all(row) for row in necrotic) / states
    misses = sum(not all(row) and row[pick] for row, pick in zip(necrotic, selected)) / states
    correct = 1.0 - joint - misses
    return single, joint, misses, correct

Option-Critic uses a softmax policy inside each option. Its original tabular setting divides preferences by temperature 0.001 and uses learning rate 0.25, so an advantage is effectively multiplied by 250 before it reaches the logits. Starting from four equally likely actions, a positive advantage of only 0.008 makes the sampled action exceed 70% probability; 0.032 puts it at 99.899% (99.9% when rounded).

The trap then seals itself. The score-function update contains one-hot(action) minus policy probability. Once the chosen action is near probability one, its gradient nearly vanishes; alternatives still have a useful gradient but are almost never sampled. With sparse reward, the first apparent winner can be arbitrary.

The decisive experiment forks every training run after episode 500. One branch continues unchanged. The other executes a uniformly random lower-level action 10% of the time. Both are later evaluated with that exploration switched off, so the comparison measures what was learned, not random behaviour at test time.

With one option, expected steps fall from 306.66 to 55.42 and all 350 seeds improve. Under constant step cost, exploration takes one option from 963.21 to 36.50 steps and raises frozen success probability from 0.037 to 0.999. More options then become overhead rather than rescue.

Solving joint = single^κ turns the rounded rates into about 6.2 independent attempts at K=8; the paper reports κ=6.20 from its full-precision data. Shared trajectories and a shared critic correlate failures, so eight trained options are not eight independent chances. They are still far from identical copies.

The learned termination test compares the value of continuing the current option with the value of reselecting. If both sides read the same estimates, continuing can never exceed their maximum, so the test fires at every step. It is pathwise identical to always terminating.

If the selector explores using optimistic values while termination reads raw critic values, the test can veto the very option whose uncertainty caused it to be proposed. The paper constructs an instance with linear regret for that rule while always terminating has logarithmic regret. But this is a worst-case separation, not a claim that interruption is worse on every instance.

• Measure failure overlap directly before paying for more options.
• Repair lower-level exploration first, then rerun the K ablation.
• Track router misses after coverage improves; the bottleneck moves upstairs.
• Test diversity objectives by how much they raise effective κ at fixed K.
• Repeat the audit with termination objectives not trained solely on return.

This is the same accounting instinct behind bagging: an ensemble is useful when member errors are not perfectly aligned. The neural-network page supplies the softmax and gradient machinery underneath the continuous experiments. The portable lesson is sharper than “more capacity helps”: fix the base learner, then ask whether the architectural prior still earns its keep.