The finance leaderboard that weights news after prices move

Suppose two stock forecasters each call three of six four-week moves correctly. Ordinary directional accuracy calls them tied. XForecast can separate them sharply because it gives more weight to cases where GPT-tagged news agrees with the price move that eventually happened.

That design is intentional: the benchmark wants to emphasize episodes where news appears informative. But the realized future enters twice—once as the target and again when constructing its importance weight. The score is therefore not plain out-of-sample accuracy, and it is not direct proof that a model learned a causal or incremental news signal.

At prediction time, a model may use prices and paired text only through four weeks before the target close. A hit is one when the sign of the predicted price change matches the sign of the realized price change. The competition requires supplied text embeddings, so a price-only model cannot enter the final ranking.

The paper decomposes paired news into atomic statements. GPT tags each statement as positive or negative for the company. After the future price direction is realized, the instance weight is the fraction of those tags that agree with that realized direction. Weighted hit rate is then the weighted mean of the zero-one hits.

def news_alignment_weight(statement_dirs, realized_dir):
    if not statement_dirs:
        raise ValueError("at least one statement is required")
    matches = sum(direction == realized_dir for direction in statement_dirs)
    return matches / len(statement_dirs)

def weighted_hit_rate(hits, weights):
    if len(hits) != len(weights) or not hits:
        raise ValueError("equal non-empty arrays are required")
    denominator = sum(weights)
    if denominator <= 0:
        raise ValueError("weights need positive mass")
    return sum(hit * weight for hit, weight in zip(hits, weights)) / denominator

The TypeScript implementation above is the one used by every chart in this post. It contains no model library and no randomness. A hand-checked example with hits [1, 0, 1] and weights [1, 2, 3] produces 4/6. The statement-weight function also makes the outcome dependency explicit: tags [+, +, −, +] weigh 0.75 if the stock rises, but 0.25 if it falls.

Consider six illustrative cases with alignment weights 1.0, 0.8, 0.6, 0.4, 0.2, and 0.0. Model A gets the first three right; model B gets the last three right. Both have ordinary hit rate 50%. Under the paper's weighted aggregation, A scores 80% and B scores 20%.

Weight concentration also changes the amount of evidence behind a score. The six illustrative cases have a Kish effective sample size of only 4.09 even before accounting for correlation among dates or stocks. XForecast does not report this diagnostic, but a leaderboard should: two metrics with the same row count can have very different effective support.

The challenge establishes that news-grounded forecasting can be run at realistic scale under a fixed temporal split and reproducible Kaggle notebooks. It also shows that strong solutions need not share one architecture: selected high-ranking entries used diffusion models, inverted Transformers, temporal neural networks, tree ensembles, and tabular foundation models. Many pooled same-date information to estimate common market or sector effects.

The paper does not publish a full leaderboard table, price-only ablation, uncertainty intervals, or a controlled comparison of raw hit rate against weighted hit rate. Its five inspected solutions are explicitly not claimed to represent all 51 teams. Because price-only submissions were excluded from final ranking, the competition cannot isolate the incremental value of text from its published ranking alone.

First, report ordinary and weighted hit rate side by side, plus weight quantiles and effective sample size. Recompute weights with multiple independent taggers and prompts. If rankings move materially, the benchmark is partly measuring evaluator choice. A pre-outcome alternative would weight cases using only news availability, novelty, or independently defined event importance.

Second, run four ablations: prices only, text only, shuffled text, and correctly aligned text. A shuffled-text model that retains its score is a warning that the evaluation subset—not news use—is doing the work. Compare a transparent LightGBM fusion baseline with a cross-attention Transformer, and evaluate both on the exact same frozen cases.

Finally, convert directions to a declared portfolio rule with costs and confidence intervals clustered by date. The challenge is valuable because it forces text and prices into one reproducible protocol. The next version should make equally explicit which population its headline score describes.