conformal p-value

Given scalar observations $Y_1,\dots ,Y_n$, the conformal p-value of a new observation $Y$ is

$$Q(Y)=\frac{1+{\sum }_{i\le n}1\{Y_i\le Y\}}{n+1}.$$

If $(Y_1,\dots ,Y_n,Y)$ is exchangeable, then $Q(Y)$ is equally distributed among $\frac{1}{n+1},\dots ,\frac{n}{n+1}$. (This assumes that $Y_i\ne Y_j$ almost surely. If this doesn’t hold then add random uniform noise to each observation.)

Many procedures are based on this p-value, including conformal prediction and permutation testing. It has also been used for selection by prediction, testing for outliers, and conditional testing. An e-value and e-process were proposed which sequentialize the conformal p-value, see permutation testing by betting.