Proper Scoring Rules Explained: Brier, Log, Spherical & Why Linear Fails

AbstractA scoring rule is proper if a forecaster maximises their expected score by reporting their true belief. We prove properness for the Brier and logarithmic rules, give the Savage characterisation (every proper rule comes from a convex function), and verify numerically that under a true probability of 0.7 the Brier, log and spherical rules are maximised at a report of exactly 0.70, while the intuitive linear rule is maximised at the boundary (0.99). The choice between proper rules is then about which errors you want to punish: log loss is 17 times harsher than Brier on a 1% forecast that comes true.

Definition#

For a binary event, a scoring rule $S (q, o)$ assigns a reward to a reported probability $q \in [0, 1]$ when the outcome is $o \in {0, 1}$ . If the forecaster's true belief is $p$ , their expected score from reporting $q$ is

S ˉ (q; p) = p S (q, 1) + (1 - p) S (q, 0)

(1)

The rule is proper if $S ˉ (q; p) \leq S ˉ (p; p)$ for all $q$ , and strictly proper if equality holds only at $q = p$ . Properness is what makes a score usable for paying or ranking forecasters: it removes any incentive to shade.

Brier and log are strictly proper#

Brier (quadratic). As a reward, $S (q, o) = - (q - o)^{2}$ . Then

S ˉ (q; p) = - p (1 - q)^{2} - (1 - p) q^{2}, \frac{\partial S ˉ}{\partial q} = 2 p (1 - q) - 2 (1 - p) q = 2 (p - q)

(2)

which is zero only at $q = p$ , and the second derivative is $- 2$ . Strictly proper.

Logarithmic. $S (q, 1) = ln q, S (q, 0) = ln (1 - q)$ :

\frac{\partial S ˉ}{\partial q} = \frac{p}{q} - \frac{1 - p}{1 - q} = 0 ⟺ q = p

(3)

Also strictly proper. Its expected score at the truth is $- H (p)$ , the negative entropy: the log score is information-theoretic, and the gap $S ˉ (p; p) - S ˉ (q; p)$ is exactly the Kullback–Leibler divergence $D_{KL} (p ∥ q)$ .

Linear, the trap. The rule $S (q, o) = q$ if $o = 1$ , $1 - q$ if $o = 0$ looks natural ("score the probability you gave to what happened"). But $S ˉ = pq + (1 - p) (1 - q)$ is linear in $q$ , so it is maximised at $q = 1$ whenever $p > \frac{1}{2}$ . It rewards exaggeration.

A numerical check#

We evaluated equation (1) on a grid of reports for true $p = 0.7$ under four rules (normalised to [0, 1] for display).

Brierlogsphericallinear (improper)

Figure 1. Expected score as a function of the report. The three proper rules peak at the truth; the linear rule increases all the way to the boundary.

Table 1. Numerical maximisers on a 0.01 grid (research/scoring.py).
Rule	Score if Yes	Score if No	Report that maximises expected score (p = 0.7)
Brier	$- (1 - q)^{2}$	$- q^{2}$	0.70
Logarithmic	$ln q$	$ln (1 - q)$	0.70
Spherical	$q / q^{2} + (1 - q)^{2}$	$(1 - q) / q^{2} + (1 - q)^{2}$	0.70
Linear	$q$	$1 - q$	0.99 (grid edge)

Where proper rules come from#

Every proper scoring rule for a binary event can be generated from a convex function $G$ on [0, 1] (Savage 1971; Gneiting and Raftery 2007):

S (q, o) = G (q) + G^{'} (q) (o - q)

(4)

$G (p)$ is the expected score of an honest forecaster with belief $p$ , and the score is the tangent line to $G$ at $q$ , evaluated at the outcome. Because a convex function lies above its tangents, reporting anything other than $p$ can only lower the expected score. Brier corresponds to $G (p) = - p (1 - p)$ (up to constants), log to $G (p) = p ln p + (1 - p) ln (1 - p)$ . The construction also explains the link to markets: Hanson's LMSR is the market version of the rule generated by negative entropy (see LMSR, derived).

Beyond binary: scoring a strike ladder#

Many event markets are ladders (CPI at least 2.8%, 2.9%, 3.0% and so on). Scoring each strike as a separate binary forecast throws away the ordering: a forecast that put its mass one bucket away from the outcome is treated the same as one that missed by five. The ranked probability score uses the ordering. With cumulative forecast $F_{k} = Pr (X \leq K_{k})$ and cumulative outcome $O_{k} = 1 {x \leq K_{k}}$ ,

RPS = k = 1 \sum K (F_{k} - O_{k})^{2}

(5)

It is strictly proper and has a convenient property for event traders: a ladder's prices already are a cumulative distribution, since $F_{k} = 1 - P (X \geq K_{k + 1})$ . The RPS of a market ladder can therefore be computed directly from the Yes prices, with no bucketing step, and compared with a model's RPS at the same timestamp. Its continuous limit is the CRPS used in weather forecasting (Gneiting and Raftery 2007).

Worked example: a five-strike ladder at 2.7–3.1% priced (cumulatively) at 0.10, 0.30, 0.65, 0.88, 0.97 when the print is 2.9%. The outcome vector is $(0, 0, 1, 1, 1)$ , so $RPS = 0.1 0^{2} + 0.3 0^{2} + 0.3 5^{2} + 0.1 2^{2} + 0.0 3^{2} = 0.2378$ . A ladder that put the same mass one bucket higher would score $0.2369$ on the same outcome. Per-strike Brier scores would not distinguish the two as cleanly.

Scoring in practice: horizons, clipping and market benchmarks#

Score at fixed horizons. A forecast made an hour before resolution is easy. Score at fixed lead times (30 days, 7 days, 1 day) and report each separately; a single "final forecast" score rewards forecasters who wait.

Clipping breaks properness, slightly. Log loss is infinite at $q = 0$ for an event that happens, so implementations clip forecasts to $[ε, 1 - ε]$ . The clipped rule is no longer strictly proper for beliefs below $ε$ . With $ε = 1 0^{- 4}$ that rarely matters; with $ε = 0.01$ , as in some leaderboards, it makes every forecast below 1% equivalent and removes the incentive to distinguish 0.1% from 1%. For tail contracts that is exactly the distinction that matters.

Benchmark against the market. For trading, the useful number is a skill score relative to the market price at the same timestamp,

SS = 1 - \frac{BS _{model}}{BS _{market}}

(6)

A positive skill score means the model's probabilities were closer to outcomes than the prices were. A model with a good absolute Brier score and a negative skill score is worse than the market it would trade against. Because $SS$ is a ratio of noisy averages, its sampling error is large; see how many forecasts it takes before believing it.

Choosing between proper rules#

Proper rules agree on what an honest forecast is and disagree on how much each kind of error costs. The most practical difference is at the tails:

Table 2. Penalty for a single forecast of q when the event occurs.
Forecast for an event that happens	Brier penalty	Log penalty	Log / Brier
50%	0.2500	0.693	2.8×
20%	0.6400	1.609	2.5×
10%	0.8100	2.303	2.8×
5%	0.9025	2.996	3.3×
1%	0.9801	4.605	4.7×

Brier caps the damage of any single forecast at 1; log loss is unbounded. A forecaster who says 1% on something that happens loses 4.6 log-units, more than six "50%" forecasts that go wrong. Which you want depends on the decision the forecast feeds:

Trading tail contracts: use log loss. Mispricing a 1% event as 0.1% is a 10× error in the price of the contract, and log loss reflects that; Brier barely notices.
Ranking many forecasters on mixed questions: Brier is more robust to a single catastrophic forecast and easier to explain, and its Murphy decomposition separates calibration from resolution.
Never use accuracy (whether the forecast was on the right side of 50%) or the linear rule to evaluate probabilities. Both are improper.

On our simulated forecasters (200,000 questions), Brier and log loss rank them identically: truth 0.1463 / 0.4454, overconfident 0.1501 / 0.4678, underconfident 0.1528 / 0.4693, noisy 0.1559 / 0.4713. Rankings can diverge on small samples dominated by a few extreme forecasts, which is exactly when the choice matters.

Questions#

What is a proper scoring rule?

A rule for scoring probability forecasts under which a forecaster maximises their expected score by reporting their true belief. The Brier score, logarithmic score and spherical score are strictly proper.

Is accuracy a proper scoring rule?

No. Scoring a forecast as right if it was on the correct side of 50% ignores the probability entirely and gives no reward for calibration. The linear rule, which scores the probability assigned to the outcome, is also improper: it rewards reporting 0 or 1.

Should I use Brier score or log loss?

Use log loss when errors on rare events are costly (such as pricing tail contracts), because it heavily penalises confident mistakes. Use Brier when you want a bounded, robust score for ranking forecasters across mixed questions.

References#

Brier, G. W. (1950). Verification of forecasts expressed in terms of probability. Monthly Weather Review, 78(1), 1–3.
Savage, L. J. (1971). Elicitation of personal probabilities and expectations. Journal of the American Statistical Association, 66(336), 783–801.
Gneiting, T. and Raftery, A. E. (2007). Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102(477), 359–378.

DisclosureThis note is research and education. It is not investment advice or an offer to trade, and nothing in it describes the firm's own results. Code and outputs: research/scoring.py in our research repository, reproducible with a fixed seed.