Brier Score: Definition, Formula & Example | Prediction Market Glossary

DefinitionThe mean squared difference between probability forecasts and what happened (1 or 0). Lower is better; 0 is perfect and always forecasting 50% scores 0.25.

Formula#

BS = \frac{1}{N} i = 1 \sum N (f_{i} - o_{i})^{2}

where $f_{i}$ is the forecast probability and $o_{i} \in {0, 1}$ the outcome. Introduced by Brier (1950) for weather forecasts.

Worked example#

Three forecasts: 0.8 (happened), 0.3 (did not), 0.6 (did not). Errors squared: $0.04$ , $0.09$ , $0.36$ . Brier score $= 0.49/3 \approx 0.163$ .

What is a good Brier score?#

It depends entirely on the questions. On coin-flip questions nobody can beat 0.25; on near-certain questions 0.02 may be mediocre. Compare Brier scores only on the same question set, or use a skill score relative to a baseline: $BSS = 1 - BS / BS_{ref}$ . In our simulations a perfectly calibrated forecaster scored 0.146 and a moderately overconfident one 0.150. Separating them reliably took about 1,000 questions (power analysis).

Properties#

Strictly proper: honest forecasts maximise the expected score (proof).
Bounded: no single forecast can cost more than 1, unlike log loss.
Decomposable into reliability, resolution and uncertainty (Murphy 1973): see calibration and the Brier score.

Questions#

What is the Brier score?

The mean squared error between probability forecasts and binary outcomes. Lower is better; a forecaster who always says 50% scores 0.25.

Is a lower Brier score better?

Yes. 0 is a perfect score. Compare scores only across forecasters evaluated on the same questions.