calibration

Given a distribution $P$ over feature-label pairs $(X,Y)$, the goal of mean calibration is to produce a model $f$ such that

$$v\approx \mathbb{E}[Y|f(x)=v],$$

and the goal of quantile calibration is to produce a model such that

$$q\approx \mathbb{P}[y\le v|f(x)=v].$$

The most interesting setting for calibration is online calibration.

Patching

Most research into calibration is not trying to generate a new model $f$ which is calibrated, but instead supposes we are given an uncalibrated model $f$ and asks whether we can we “fix” it to make it more calibrated? Yes, yes we can.

We have a target average calibration error $K_2$ of $\alpha$. Let

$$v^{\prime }={\text{argmax}}_v\mathbb{P}(f(x)=v)\left(v-{\mathbb{E}}_P[y|f(x)=v]\right).$$

Then we simply set $\hat{f}(x)=v^{\prime }$ if $f(x)=v$ and $\hat{f}(x)=f(x)$ otherwise. Repeating this procedure leads to a model with $\le \alpha$ calibration error. Moreover, it also reduces squared error. You can show that this runs for $T\le m/\alpha$ iterations where $m=|\text{range}(f)|$.

There’s also a one-shot algorithm that accomplishes this goal. If $c(v)=\mathbb{E}[y|f(x)=v]$, then simply let $\hat{f}(x)=c(f(x))$. Then $\hat{f}$ has $K_2=0$.

Obviously, these assume access to the true distribution. One can approximate the means with the empirical distribution and then obtain bounds on the calibration error using standard concentration inequalities.

Similar arguments apply if we’re interested in quantile calibration. There’s also a one-shot version.