The survival metric that says every model tied

Two survival models can rank every patient in exactly the same order while disagreeing radically about whether anyone is likely to relapse. A concordance score calls that a tie. A clinician deciding whether to treat does not have that luxury: 20% and 90% are different decisions.

In survival analysis, the target is time until an event. A record is right-censored when observation ends before the event, so we know only that its time lies later. Competing risks adds mutually exclusive causes: after a first event such as death from one cause, another cause cannot be the first event.

The useful prediction is a cumulative incidence function: the probability that cause k happens by time t, before any rival cause. The C-index asks whether earlier cases receive higher risk than comparable later cases. The Brier score also asks whether the probabilities have the right magnitude. With censoring, the paper uses inverse-probability weights; our four-person demonstration has no censoring before the chosen horizon, so every weight is one.

def brier_score(labels, probabilities):
    total = 0.0
    for y, p in zip(labels, probabilities):
        total += (y - p) ** 2
    return total / len(labels)

def concordance(times, causes, risks, cause):
    score, pairs = 0.0, 0
    for i in range(len(times)):
        if causes[i] != cause:
            continue
        for j in range(len(times)):
            if times[i] >= times[j]:
                continue
            pairs += 1
            score += 1.0 if risks[i] > risks[j] else 0.5 if risks[i] == risks[j] else 0.0
    return score / pairs

The TypeScript version above is the code executed on this page; the Python and C++ tabs are line-for-line translations. Patient A has the first cause at time 1, B at time 2, C has a competing cause at time 3, and D remains event-free through time 4. Both forecasts rank A, B, C, D in that order for cause 1.

The inflated model's Brier score is almost nine times larger, yet its concordance is identical. Adding a constant to every log-odds value preserves all rankings. The orange curve below therefore changes while the purple curve cannot.

The paper crosses three losses with four backbones under the same 150-iteration randomized-search budget. Its main loss, CRNLL, models cumulative incidence directly and reweights observations for censoring. The authors compare spreads in Brier-score gain while holding one axis fixed. On the ten competing-risks datasets, the median loss effect is 0.053 versus 0.027 for the backbone. On the ten single-event datasets, the order reverses slightly.

The cleanest same-backbone comparison is DeepHit's MLP. Replacing its own hazard-plus-ranking loss with CRNLL raises the median Brier-score gain over the covariate-free Aalen–Johansen baseline from 0.12 to 0.26. The architecture did not change.

Each loss–backbone cell is searched twice: once selecting by integrated Brier score and once by C-index. This gives the ranking loss every chance to prove that the default tuning criterion held it back. It does not. Selecting the ranking-augmented loss on C-index improves median C-index by 0.0018 while worsening the Brier score by 0.0117. Its median ranking weight rises from 0.41 to 0.71, so the search really does lean harder on ordering—it simply purchases little ordering and loses probability accuracy.

The benchmark is broad, not universal. Ten of its 20 tabular datasets are single-event, training sets are capped at 50,000 rows, and the three image datasets use frozen encoders. Its proper-score guarantees also rely on correctly estimated censoring weights and suitable independence assumptions. A covariate-conditional Cox censoring model converges in only 35 of 94 tested cells, so the paper uses a more stable marginal Kaplan–Meier estimate. That is a defensible engineering choice, not a proof that censoring is covariate-independent.

Next, report a proper score first and keep concordance as a diagnostic. Stratify results by competing-risk versus single-event data, publish the censoring model and its failures, and test whether the probability gain changes real treatment or maintenance decisions. For neural survival models, the shared representation still comes from the same building blocks explored in the site's interactive neural-network walkthrough; this paper's lesson is that the output loss and evaluation rule deserve equal scrutiny.