Methods
Forecast calibration: Brier score, log loss and why markets keep winning
If every forecast ends in a 0 or a 1, you can grade every forecaster honestly. Here is how we do it, and what it says about markets.
Why probabilistic forecasts need special scoring
A forecaster who says "70% chance of rain" is neither right nor wrong when it rains. They are right on average if it rains on about 70% of the days they say 70%. Grading probabilistic forecasts requires scoring rules that reward both honesty and information, and that cannot be gamed by shading forecasts toward 0 or 1.
Event markets are an ideal laboratory. Every contract resolves to exactly 0 or 1, every price is a timestamped probability, and there are tens of thousands of resolved contracts to learn from.
The Brier score
The Brier score is the mean squared error between forecast probabilities and outcomes:
Lower is better. A forecaster who always says 50% scores 0.25. A perfect forecaster scores 0. Typical skilled forecasts on genuinely uncertain questions land somewhere between 0.05 and 0.20, depending heavily on how hard the questions are, so Brier scores are only comparable on the same set of questions.
The Brier score is a proper scoring rule: your expected score is optimised by reporting your true belief. That property is what makes it suitable for evaluating models and people alike.
Log loss and why it punishes overconfidence
Log loss (also called the logarithmic or ignorance score) is:
It is also proper, but unlike Brier it is unbounded. Saying 1% on something that happens costs ln(100) ≈ 4.6, while saying 0.1% costs ≈ 6.9. Log loss is therefore far harsher on confident mistakes at the tails, which is exactly where trading losses concentrate. We report both, and we pay particular attention to log loss on contracts priced below 10¢ or above 90¢.
Calibration and resolution: the decomposition
The Brier score splits into three interpretable parts (the Murphy decomposition):
- Reliability (lower is better) measures calibration: when you say 70%, does it happen 70% of the time?
- Resolution (higher is better) measures discrimination: do your forecasts actually separate events that happen from those that do not?
- Uncertainty is a property of the questions, not the forecaster: the base-rate variance.
A forecaster who always predicts the base rate is perfectly calibrated and completely useless: zero resolution. Good forecasting means being calibrated and decisive. Most of the craft is improving resolution without sacrificing reliability.
Reading a reliability diagram
The most useful single picture of a forecaster is the reliability diagram: bin forecasts by predicted probability and plot the observed frequency in each bin. Perfect calibration lies on the diagonal.
Points above the diagonal at the low end mean events you call unlikely happen more often than you say: you are overconfident in "no". Points below the diagonal at the high end are the mirror image. An S-shaped curve means systematic overconfidence; a flattened curve means underconfidence. Always look at the bin counts; the tails often have few observations and wide error bars.
Why liquid markets are hard to beat
Studies across elections, economics and sport repeatedly find that liquid prediction market prices are well calibrated and competitive with, or better than, polls, expert panels and many statistical models. There are good reasons:
- Markets aggregate many independent views, weighting them by conviction (and capital).
- Participants who are systematically wrong lose money and influence; those who are right gain both.
- Prices update continuously as information arrives, rather than on a publication schedule.
The weak points are equally well documented: thin markets, the favourite-longshot bias at the tails, slow updating in low-attention contracts, and occasional distortions from participants with non-financial motives. That is precisely where professional liquidity and disciplined models add value, both to the firm and to the accuracy of the price.
We do not try to beat the market everywhere. We try to know, precisely, where it is weakest.
Common pitfalls when scoring forecasts
Scoring rules are simple to compute and easy to misuse. The mistakes we see most often, including in published work:
- Comparing scores across question sets. A Brier score of 0.08 on near-certain questions can be worse than 0.18 on coin-flips. Compare forecasters only on identical questions, or use skill scores relative to a shared baseline.
- Scoring only the final forecast. A forecaster who is right only the day before resolution is less useful than one who is right a month earlier. Score forecasts at fixed horizons (for example 30, 7 and 1 day before resolution) and report each separately.
- Ignoring look-ahead bias. Backtests that use data revised after the fact, or contract rules clarified after listing, flatter models. Everything must be point-in-time.
- Too few tail observations. Calibration at 2% or 98% needs hundreds of observations per bin to say anything. Report confidence intervals, or pool bins at the tails.
- Treating correlated questions as independent. Fifty contracts on one election night are closer to one observation than fifty. Cluster by theme when estimating uncertainty in scores.
- Selecting on attention. Scoring only the questions people talked about over-represents surprises. Score every question a model covered.
Scoring a model against the market
For a trading firm the relevant benchmark is not the base rate; it is the market price. A model that scores 0.10 against a market that scores 0.09 on the same questions at the same timestamps is, for trading purposes, worse than useless: trading on it would systematically lose to the price.
The useful statistic is the score difference, computed contract by contract and timestamp by timestamp, then aggregated with clustering by theme. We also look at where the difference comes from: categories, horizons, price ranges and liquidity regimes. A model that loses to the market overall can still be valuable if it reliably beats it in one niche, such as low-attention contracts, long horizons, or the minutes after a release, and capital can be allocated accordingly.
Finally, the score difference must be translated into money. A small improvement in Brier score on contracts trading near 50¢ with tight spreads may be worth nothing after costs; the same improvement on wide, illiquid contracts may be worth a great deal. Calibration is necessary, not sufficient.
How we use this in practice
- Every model publishes a probability for every contract it covers, with a timestamp, before the market resolves.
- On resolution, the scoring service computes Brier, log loss and reliability per model, per category and per horizon.
- Each model is compared with the market's own price at the same timestamps. A model that does not beat the market somewhere does not get capital.
- Recalibration layers (isotonic or Platt-style) are fitted on rolling windows to correct systematic drift, and their own out-of-sample performance is scored too.
The same discipline applies to people. Researchers and forecasters on the desk log probabilities for the questions they cover, and those forecasts are scored exactly like the models'. It is a humbling exercise and an invaluable one: it shows where human judgement adds something a model cannot (reading an ambiguous rule, recognising an unprecedented situation) and where it merely adds noise. Over time, the scores decide which questions are handed to models, which to people, and which are best left to the market.
Frequently asked questions
What is a good Brier score?
It depends on the difficulty of the questions. Always predicting 50% scores 0.25; skilled forecasts on uncertain questions typically score between about 0.05 and 0.20. Compare Brier scores only on the same question set.
What is the difference between Brier score and log loss?
Both are proper scoring rules. Brier is bounded mean squared error; log loss is unbounded and penalises confident mistakes at the tails far more heavily.
Are prediction markets more accurate than polls?
Research generally finds that liquid prediction markets are well calibrated and at least competitive with polls and expert forecasts, though thin markets and tail prices show known biases.
Related: Pricing event contracts · Our forecasting approach · Glossary