Online adaptation looks best when the baseline trains too little—or too much

Imagine installing a load forecaster in a smart building. Seasons, occupancy, and sensors drift, so the model keeps learning after deployment. Its online updates cut error by 20%. Success—unless the frozen comparison model was stopped too early and the adapter merely finished its training.

Fujimoto and Nishi study that measurement problem across six public multivariate streams. Their most useful result is not a new forecasting layer. It is a warning that the apparent value of adaptation is a ratio whose denominator—the static baseline—moves with the warmup budget.

Online adaptation means updating a deployed model from newly revealed observations. Concept drift means the relationship generating those observations changes over time. A valid prequential test—predict, then evaluate, then learn—ensures the current truth cannot help predict itself.

def score_then_adapt(stream, weight, rate):
    trace = []
    for x, target in stream:
        prediction = weight * x
        error = (prediction - target) ** 2
        gradient = 2 * x * (prediction - target)
        weight = weight - rate * gradient
        trace.append((prediction, error, weight))
    return trace

The paper uses non-overlapping windows whose stride equals the forecast horizon. Each horizon is scored before it enters a gradient. As an audit, the authors deliberately restore overlapping-target leakage: on ETTh2/PatchTST, the result flips from −15.6% with delayed truth to +24.2% when future scored targets leak into earlier updates. Relative to a static MSE of 1, those benefits correspond to adapted errors 1.156 and 0.758.

Adaptation benefit is 100 × (static MSE − adapted MSE) / static MSE. Make the static model worse and the same adapted error earns a larger percentage. Under-warm the base model and online learning gets credit for completing ordinary training. Over-warm it on old data and it can specialize to the pre-drift regime, again enlarging the gap after drift.

In the small illustrative candidate set used by this page, validation alone selects 1,000 steps. The selection function never reads test MSE. In the paper, validation selection also withholds its freshest slice from training and does not refit afterward, leaving the deployed model about 73 days stale on the hourly and 15-minute series and about 14 days stale on Appliances.

A second fairness trap appears when SGD with momentum and Adam share the same 10⁻³ online learning rate. Adam falls below the static baseline in 174 of 360 cells, versus 29 for SGD+m. At a fixed 10⁻⁴, Adam instead wins 322 cells. Selecting each rate through a pre-drift online rehearsal makes Adam win 310 of 360 cells and reduces its negative cases to four.

Rehearsal is selection, not certification. Seven rehearsed SGD+m cells and four Adam cells still lose after drift. A conservative rule choosing the smallest rate within 2% of best validation MSE removes all negative cells in this grid, but changes 337 of 360 choices and costs a median 3.6 benefit points.

The resource axis counts a gradient buffer plus optimizer state: 8 bytes per trainable parameter for fp32 SGD+m and 12 for Adam. For an illustrative 85,700-parameter model, the tested arithmetic yields the bars below, closely matching the paper's rounded full-PatchTST points.

That exclusion matters. Latency is measured at batch one on an A100, not on the claimed class of device, and the energy figures multiply latency by an assumed 5 W. The authors explicitly say Jetson-, Raspberry-Pi-, or meter-class validation is required before calling the recipe deployable end to end.

• Refit after selecting warmup and measure whether fresher training changes the adaptation gap.
• Expand the capped warmup grid for the 42 boundary selections.
• Stress rehearsed rates on abrupt drift that is absent from the validation slice.
• Measure peak resident memory, latency, and energy on the actual target device.
• Separate energy channels from weather and indoor-sensor channels on Appliances.

The linear regression page explains the family behind DLinear, while the Transformer page supplies the backbone intuition for PatchTST. The durable lesson is simpler: an adapter's gain is only as honest as the frozen model in its denominator.