two-sample testing

In hypothesis testing, given a stream of paired observations $(X_t,Y_t)$, $t=1,2,\dots ,n$, where $X_t\sim P_X$ and $Y_t\sim P_Y$, two-sample testing asks $H_0:P_X=P_Y$, $H_1:P_X\ne P_Y$. This can of course be asked in the sequential setting or the batch setting.

Approaches:

• testing by betting—two-sample testing (sequential, nonparametric)
• Original Kernel MMD test in the batch setting (nonparametric)
• Two-sample t-test or z-test (parametric)
• General framework for two-sample testing, but seems to be beat by testing by betting (sequential, nonparametric)
• One can build confidence sequences for the KS distance or the kernel MMD from confidence sequences for convex functionals, and then use the duality between hypothesis tests and CIs to form hypothesis tests. This is more conservative than testing directly.