← Blog/blog/pyrite-pairwise-ranking-gap

The 74% pose ranker that can still miss the best pose

A protein–ligand pose is one proposed 3D arrangement of a small molecule inside a protein pocket. A cofolding model can generate many plausible poses; the practical problem is choosing which one to inspect, simulate, or synthesize first.

Pyrite turns that problem into 1,500 pairwise choices. Its best hybrid agent asks an LLM to invent features, then gives those features to logistic regression. It reaches 74% accuracy on the curated post-AF3 split, ahead of expert chemists. The buried question is whether winning pairs produces the ordered candidate list a discovery pipeline needs.

01

What Pyrite actually measures

Each example shows two poses from the same protein–ligand query and asks which lies closer to the known crystal structure. The paper balances the binary task and counts abstentions as wrong. This makes per-pair accuracy clean and comparable, but it does not test whether all candidates can be combined into a coherent total order.

Paper-reported curated-split accuracies. The best PyriteFeaturizer result uses an LLM to generate features and a logistic regression to combine them; the expert bar uses decisive annotations only.
MethodPost-AF3 curatedPost-AF3 all
Experts, decisive only57%—
Vanilla logistic regression64%60%
Best direct reasoner67%67%
Best LLM featurizer74%71%
Paper-reported pairwise accuracies. Tree ensembles were not significantly different from the vanilla logistic baseline; the strongest result comes from LLM-guided feature discovery.
02

How pairwise votes become a list

A minimal aggregator gives each pose one point per predicted pairwise win, then sorts by points. This is a Copeland-style ranking. The tested TypeScript below drives the diagnostic; Python and C++ are faithful translations.

def pairwise_accuracy(comparisons):
    if not comparisons:
        raise ValueError('comparisons must not be empty')
    correct = sum(predicted == truth for predicted, truth in comparisons)
    return correct / len(comparisons)

def rank_by_pairwise_wins(candidate_count, pairs):
    scores = [0] * candidate_count
    for left, right, winner in pairs:
        if winner not in (left, right):
            raise ValueError('winner must belong to pair')
        scores[winner] += 1
    return sorted(range(candidate_count), key=lambda i: (-scores[i], i))
One wrong pairPair accuracyErrors touching bestRanked firstTop-1 recall
Error: best vs second83.3%1Pose 20%
Error: third vs fourth83.3%0Pose 1100%
Deterministic four-pose illustration computed by the core module. Pose 1 is truly best. Both rankers make exactly one error, but only the error involving Pose 1 changes the winner.
errors involving the best posesame number of errors elsewhere
Deterministic six-pose illustration. Errors concentrated on comparisons involving the true best pose erase its five pairwise wins; the same number of errors among lower-ranked poses leaves its score unchanged.
03

The benchmark itself changes the heuristic

Pyrite keeps the top 25 poses selected by MM-GBSA before constructing difficult pairs. Inside that already-filtered set, choosing the lower MM-GBSA energy is anti-correlated with the answer: 35% accuracy on the full post-AF3 split. The paper explicitly warns that flipping the sign is not a general physical law—the relationship changes with how many candidates survive the initial filter.

04

What the 74% does not yet establish

Training uses pre-AF3 structures and evaluation uses post-AF3 structures, so calendar is a useful leakage guard but also a possible distribution shift. The candidates come from one proprietary cofolding model and diversity sampler. Most importantly, the paper evaluates pairs, not the end-to-end top-k enrichment, regret, or cost of a full ranking pipeline.

The next experiment should group every candidate set by complex, score all pairs without reusing them across folds, aggregate them into a list, and report top-1/top-k recovery plus cycle rates. Repeat that protocol on poses from several public generators. The site's logistic regression page explains the auditable downstream model; decision trees provide the nonlinear baseline the paper found statistically tied with it.

References

  1. Srivastava, M., Lim, M. H., Keys, A. M., Murugan, P., and Gruver, N. (2026). Pyrite: Building and Benchmarking Protein-Ligand Pose Ranking Agents. NeurIPS 2026 Workshop on Agentic AI for Biological Discovery; arXiv:2611.14511
  2. Cox, D. R. (1958). The Regression Analysis of Binary Sequences. Journal of the Royal Statistical Society: Series B 20(2)
  3. Copeland, A. H. (1951). A Reasonable Social Welfare Function. University of Michigan Seminar on Applications of Mathematics to the Social Sciences