proper scoring rule

A scoring rule is a function used to evaluate forecasts. Let $\mathcal{X}$ be an outcome space of interest, and $\Delta (\mathcal{X})$ the set of distributions over $\mathcal{X}$. The goal of a forecaster is to produce some $P\in \Delta (\mathcal{X})$ which is “good”, in some sense.

To measure whether $P$ is good, we introduce a scoring rule $S:\Delta (\mathcal{X})\times \mathcal{X}\to \mathbb{R}$, which takes in the forecaster’s distribution $P$ and an outcome $x\in \mathcal{X}$ and says how good $P$ was. Usually $S$ is taken to be “positively oriented,” meaning larger values of $S$ imply better forecasts.

Given $S$, the expected score between $P,Q\in \Delta (\mathcal{X})$ is

$$S(P\Vert Q):={\mathbb{E}}_{x\sim Q}S(P,x).$$

A scoring rule $S$ is proper if for all $Q\in \Delta (\mathcal{X})$,

$$Q\in {\text{argmax}}_{P\in \Delta (\mathcal{X})}S(P\Vert Q).$$

In words: Suppose you as the forecaster knew that the “true” distribution was $Q$. If $S$ is a proper scoring rule, it means that you can play $Q$ to maximize your score. This is so intuitive it’s almost painful.

Examples of proper scoring rules for binary forecasts are (note that $P=p\in [0,1]$ in this case):

• Brier score: $S(p,x)=1-(p-x{)}^2$
• Spherical score: $S(p,x)=\frac{py+(1-p)(1-x)}{\sqrt{p^2+(1-p{)}^2}}$
• Logarithmic score: $S(p,x)=x\log (p)+(1-x)\log (1-p)$
• 0-1 score: $S(p,x)=x1\{p\ge 0.5\}+(1-x)ind\{p\le 0.5\}$.