How Many Forecasts to Evaluate a Forecaster? Brier Score Power Analysis

AbstractTwo forecasters who differ by a realistic amount (one perfectly calibrated, one whose log-odds are 50% too extreme) differ in expected Brier score by only 0.004. In 2,000 simulated comparisons per sample size, a paired test detects that difference with 80% power only somewhere between 800 and 1,600 resolved questions; at 100 questions it succeeds 15% of the time. Worse, when questions resolve in correlated clusters (20 contracts on one election night), the naive test reports a significant difference between two equally skilled forecasters 32% of the time instead of 5%. Leaderboards built on a few hundred questions are mostly noise.

The size of the gap#

On 200,000 simulated questions with true probabilities drawn from Beta(0.7, 0.7), the Brier scores of five forecasters were:

Table 1. Scores on 200,000 questions (research/scoring.py, seed 5). Uncertainty term = 0.25 for all.
Forecaster	Brier	Log loss	Reliability	Resolution
Truth	0.1463	0.4454	0.00002	0.1029
Overconfident	0.1501	0.4678	0.00350	0.1020
Underconfident	0.1528	0.4693	0.00645	0.1027
Noisy	0.1559	0.4713	0.00067	0.0938
Climatology (always 0.5)	0.2500	0.6931	0.00000	0.0000

The overconfident forecaster is meaningfully worse, with a reliability term 175 times the truth's. Yet its Brier score is only 0.0038 higher. Per question, the Brier difference has a standard deviation far larger than its mean, because the outcome noise $(f - o)^{2}$ dwarfs the skill difference.

Power, simulated#

For each sample size $N$ we drew 2,000 independent sets of $N$ questions, scored the truth and a competitor on the same questions, and ran a one-sided paired $t$ -test at 5% on the per-question Brier differences $d_{i} = (f_{i}^{c} - o_{i})^{2} - (f_{i}^{*} - o_{i})^{2}$ .

Table 2. Probability of detecting that the truth beats each competitor (2,000 replications per cell).
Questions N	Overconfident (log-odds × 1.5)	Underconfident (log-odds × 0.6)	Noisy (log-odds + N(0, 0.6²))
25	5%	20%	12%
50	10%	25%	17%
100	15%	36%	27%
200	25%	53%	44%
400	43%	75%	69%
800	71%	93%	92%
1,600	92%	100%	100%

OverconfidentUnderconfidentNoisy

Figure 1. Power rises roughly with √N. 80% power requires roughly 500 questions against the underconfident or noisy forecaster and around 1,000 against the overconfident one.

The rule of thumb that falls out: to distinguish forecasters who differ by realistic amounts, you need on the order of a thousand independent resolved questions. A season of a forecasting tournament with 100 questions can only separate the very good from the very bad.

A back-of-envelope formula#

The paired test has approximate power $Φ (N d ˉ / s_{d} - z_{0.95})$ , where $d ˉ$ and $s_{d}$ are the mean and standard deviation of the per-question score difference. Inverting for 80% power:

N_{80} \approx (\frac{( z _{0.95} + z _{0.80} ) s _{d}}{d ˉ})^{2} = (\frac{2.49 s _{d}}{d ˉ})^{2}

(1)

Table 3. Per-question effect sizes estimated on 400,000 questions (research/effect_sizes.py) and equation (1). These agree with the simulated power in Table 2.
Competitor	Mean difference d̄	SD per question s_d	Effect d̄ / s_d	N for 80% power
Overconfident (log-odds × 1.5)	0.0039	0.0503	0.078	1,027
Underconfident (log-odds × 0.6)	0.0065	0.0577	0.112	490
Noisy (log-odds + N(0, 0.6²))	0.0096	0.0908	0.106	552

The per-question standard deviation is 9–13 times the mean difference. Estimate both quantities on whatever history you have before launching an evaluation, not after.

Correlated questions break the test#

The test above assumes independent questions. Event markets violate this constantly: twenty contracts on the same election night, a ladder of strikes on one inflation print, a dozen games in one weekend affected by the same weather. When questions share a common shock, their score differences are correlated and the effective sample size is much smaller than the count.

We simulated 400 questions arranged in clusters of size $m$ , with outcomes driven partly by a cluster-level shock, and compared two forecasters of identical skill whose errors also had a cluster component. A correct test should reject 5% of the time.

Table 4. Nominal level 5%, 2,000 replications per row.
Cluster size m	Clusters	False-positive rate of naive paired t-test
1	400	5.1%
5	80	11.0%
20	20	32.3%
50	8	48.4%

With clusters of 20, one in three comparisons between equally good forecasters declares a winner. The standard correction is to treat the cluster as the unit: average differences within each cluster and test across clusters, or use cluster-robust standard errors. The design effect $1 + (m - 1) ρ$ , with $ρ$ the within-cluster correlation of score differences, tells you how much smaller your effective sample is.

Peeking: the leaderboard problem#

Leaderboards are checked continuously, and someone declares a winner as soon as one forecaster looks significantly ahead. That is a sequential test run as if it were a fixed-sample one. We simulated two forecasters of identical skill on up to 1,000 independent questions (research/peeking.py, 4,000 replications), testing at 5% every $s$ questions and stopping at the first significant result:

Table 5. False-positive rate of repeated paired t-tests between equally skilled forecasters (nominal 5%).
Check every	Number of looks	Chance of declaring a false winner
1000 questions	1	5.4%
500 questions	2	8.7%
200 questions	5	14.1%
100 questions	10	20.8%
50 questions	20	26.2%
25 questions	40	32.0%

Checking every 25 questions declares a winner between identical forecasters 32% of the time. The remedies are standard: fix the evaluation horizon in advance, or use a sequential procedure designed for continuous monitoring, such as alpha-spending boundaries or always-valid confidence sequences, which widen the threshold at early looks.

Combine this with clustering (Table 4) and the uncomfortable conclusion is that most informal claims of "model A beats model B" on a live leaderboard are not evidence of anything.

What we do with this#

We never promote or retire a model on fewer than several hundred independent resolved contracts, and we count clusters (events), not contracts.
Comparisons are always paired: same questions, same timestamps. Unpaired comparisons need several times more data.
We report the score difference with a cluster-robust interval, not two scores side by side.
When data are scarce we compare models on the market's own price as a benchmark, at many timestamps per contract. That adds information per question but not independent questions, and the clustering correction must account for it.

Questions#

How many forecasts do you need to evaluate a forecaster?

To distinguish realistic differences in skill (such as moderate overconfidence) with 80% power, roughly 500 to 1,000 independent resolved questions in our simulations. With 100 questions, detection rates were 15–36%.

Why are forecasting leaderboards noisy?

Because outcome randomness dominates skill differences on any single question. The Brier difference between a calibrated and an overconfident forecaster was 0.004 per question in our simulation, while its standard deviation per question was about 13 times larger.

How do correlated questions affect forecast evaluation?

They reduce the effective sample size. With clusters of 20 correlated questions, a naive paired t-test falsely declared a winner between equally skilled forecasters 32% of the time instead of 5%. Test at the cluster level or use cluster-robust standard errors.

References#

Murphy, A. H. (1973). A new vector partition of the probability score. Journal of Applied Meteorology, 12(4), 595–600.
Gneiting, T. and Raftery, A. E. (2007). Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102(477), 359–378.
Diebold, F. X. and Mariano, R. S. (1995). Comparing predictive accuracy. Journal of Business & Economic Statistics, 13(3), 253–263.

DisclosureThis note is research and education. It is not investment advice or an offer to trade, and nothing in it describes the firm's own results. Code and outputs: research/scoring.py in our research repository, reproducible with a fixed seed.