A language model predicts the next token. Shabashev asks whether a model trained on individual exchange messages can predict a more useful object: where the market's fair price will be a few hundred milliseconds later. The answer is yes—but the most important result is what that accuracy still cannot buy.
The 300M-parameter decoder is pretrained on 2.76 billion Level-3 order-flow tokens, then adapted with low-rank adapters, or LoRA, to forecast microprice return and realized volatility. At 200 ms it beats a carefully engineered gradient- boosted baseline. Yet the forecasted moves are sub-basis-point while crossing the spread and paying a high-volume taker fee costs about 2 basis points.
The target moves before the midpoint does
Level-3 data records individual order additions, cancellations, and changes. The model turns each event into one token. Its target is the microprice: the best bid and ask weighted by the quantity resting on the opposite side. A thin ask and deep bid pull fair value toward the ask. With bid 100, ask 101, bid depth 3, and ask depth 1, the tested implementation gives 100.75.
This matters because the ordinary midpoint is exactly unchanged in 74% of the paper's 200 ms windows. Microprice supplies a continuous, imbalance-aware target while the quoted price is sticky. The paper also checks the same predictions against future midpoint moves, where the result becomes stronger rather than disappearing.
def microprice(bid, ask, bid_qty, ask_qty):
total_qty = bid_qty + ask_qty
return (bid * ask_qty + ask * bid_qty) / total_qty
def one_way_cost_bps(spread_bps, taker_fee_bps):
return spread_bps / 2 + taker_fee_bps
def expected_edge_bps(accuracy, move_bps, cost_bps):
signed_hit_rate = 2 * accuracy - 1
return signed_hit_rate * move_bps - cost_bpsPretraining helps where order flow is fastest
The adapted foundation model reaches direction correlations 0.44, 0.45, and 0.39 at 0.2, 0.5, and 1 second. Gradient boosting on ten-level order-flow imbalance, several book-imbalance depths, and recent return reaches 0.39, 0.37, and 0.31. The paired gaps are 0.05, 0.08, 0.08; all three reported 95% intervals exclude zero.
| Horizon | Best OBI | Gradient boost | FM + LoRA | Model − GBM |
|---|---|---|---|---|
| 0.2 s | 0.23 | 0.39 | 0.44 | +0.05 |
| 0.5 s | 0.24 | 0.37 | 0.45 | +0.08 |
| 1 s | 0.23 | 0.31 | 0.39 | +0.08 |
Accuracy cannot cross a larger cost barrier
A directional bet with hit rate a on a symmetric move m has gross expectation (2a−1)m. Subtract the one-way cost: half the spread plus the taker fee. Using the paper's median model spread of 1.84 bp and its roughly 1 bp high-volume fee gives 1.92 bp. The paper only says the forecasted moves are sub-basis-point, so the chart below uses clearly labeled illustrative magnitudes—not paper measurements—and the reported 82% sign accuracy on price-moving 200 ms windows.
At an illustrative 0.8 bp move, break-even accuracy would be 170%—mathematically impossible. Even perfect direction loses 1.12 bp. This is the central distinction: prediction quality and economic value live on different scales.
Where the result is fragile
The 200 ms Pearson estimate is heavy-tail sensitive: the top 5% of absolute-return windows contribute 64% of prediction–return covariance, while rank correlation is much tighter. The fine-tune covers one asset and two days; all five evaluations stay on one venue and instrument inside four weeks. Volatility comparisons also change by regime, and beyond a few seconds classical methods match or beat the adapted model.
Token time is another load-bearing detail. Reconstructing timestamps by summing retained inter-arrival tokens drifts about 2.25 seconds—225% of a one-second target—because dropped events still advance the exchange clock. The paper keeps true exchange timestamps; an implementation that does not can accidentally move the “future” target into its context.
What to probe next
Evaluate the signal as an execution decision: condition fill probability and adverse selection on the forecast, include queue position and end-to-end network latency, and compare saved spread against missed fills. Retrain across assets and regimes, then report net execution cost rather than directional PnL. The site's Transformer page explains the backbone, while the gradient-boosting page is the right reference for the strongest handcrafted-feature baseline.
References
- Shabashev, F. (2026). Fine-Tuning an Order-Flow Foundation Model for Sub-Second Forecasting. arXiv:2611.14117
- Stoikov, S. (2018). The Micro-Price: A High-Frequency Estimator of Future Prices. Quantitative Finance 18(12)
- Hu, E. J. et al. (2022). LoRA: Low-Rank Adaptation of Large Language Models. ICLR 2022