Fixing the sample size cannot fix bad sampling weights

Suppose a thousand human labels are affordable. A common active inference design gives every record its own sampling probability and flips all of those coins independently. The expected number of labels can be exactly one thousand; the invoice cannot. Skalnes and Oberst replace that random-sized draw with conditional Poisson sampling (CPS), which keeps the weighted preference but returns exactly the requested number of records.

In Poisson sampling, unit i is included independently with probability πᵢ. The expected sample size is Σπᵢ, but its variance is Σπᵢ(1−πᵢ). Overshooting can be infeasible when labels require a fixed contract, laboratory batch, or annotation queue; undershooting wastes capacity.

The deterministic toy population below has three units and a target of two labels. The core first scales odds 1:2:3 so the independent coins have expected size two. PS still returns zero, one, two, or three labels. CPS conditions on exactly two.

CPS is easiest to describe as rejection sampling: draw the independent coins, reject any draw whose size is not n, and keep the first size-n draw. Efficient implementations use dynamic programming, but the conditional distribution is the same. If the underlying coin odds are oᵢ = pᵢ/(1−pᵢ), a size-n subset S receives probability proportional to ∏ᵢ∈S oᵢ.

def cps_distribution(odds, size):
    subsets = combinations(range(len(odds)), size)
    masses = [product(odds[i] for i in s) for s in subsets]
    total = sum(masses)
    return [(s, mass / total) for s, mass in zip(subsets, masses)]

def ht_mean(values, selected, inclusion):
    weighted = sum(values[i] / inclusion[i] for i in selected)
    return weighted / len(values)

Here is the implementation trap. Conditioning changes each unit's marginal inclusion probability. The orange bars are the independent coin probabilities after scaling them to an expected size of two; the purple bars are the probabilities that each unit actually appears after conditioning. The paper uses Chen's calibration map to choose underlying coins that recover a requested final vector.

A Horvitz–Thompson estimator divides each sampled value by its marginal probability of inclusion, then divides the weighted total by population size. It is design-unbiased when those probabilities match the actual sampling design. For the toy values 10, 20, and 40, the true mean is 23⅓. Averaging the estimator over all three CPS subsets recovers that value when it uses the final CPS marginals. Reusing the pre-conditioning coin probabilities produces about 23.93 instead.

Active statistical inference uses a cheap prediction f(x) on every unit, then labels a subset to correct residuals wᵢ = yᵢ−f(xᵢ). For mean estimation, the paper's asymptotic PS variance scale is the average of wᵢ²(1−πᵢ)/πᵢ. CPS subtracts πᵢβ from each residual first. The exact gap is a square divided by Σπᵢ(1−πᵢ), so it cannot be negative.

With a uniform policy and centered residuals, β is zero and the two designs coincide asymptotically. Shift every residual in the same direction—as a miscalibrated predictor can—and PS estimates move with its random sample size. CPS removes precisely that component.

Across 5,000 seeds and budgets 100, 300, 500, 700, and 1,000, CPS is never significantly worse than PS in the paper's experiment table. It has significantly lower RMSE in six settings. The largest reductions occur when the response mean or prediction residual has been shifted, exactly where random sample size becomes most correlated with the estimate.

The shifted Pew classifier keeps the same ranking ability but adds 3 to every logit, badly damaging calibration. CPS cuts PS RMSE by 16% for Biden approval and 13% for Trump approval. Yet the same learned sampling policy over-predicts Trump approval as 0.65 when the true mean is 0.30. Against uniform simple random sampling, CPS is then 86% worse.

Audit selection weights out of sample before spending the label budget. Report calibration and ranking separately, because the shifted Pew experiment preserves AUROC while breaking the policy. Compare against uniform CPS, stress-test clipped or blended policies, and show how much performance changes when a small pilot batch recalibrates f.

The computational boundary matters too. The paper's dynamic program costs O(nN) time and space to build. Its largest reported build takes 16.82 seconds at n=1,000, but n=10,000 and N=10 million would require about 10¹¹ table entries—roughly 800 GB in double precision. PS has no such table. Fixed-size inference is statistically attractive, not free at arbitrary scale.

For the model that often supplies these probabilities, continue with the logistic-regression walkthrough. Its probability calibration matters here even when its ranking metric looks unchanged.