contextual bandit

todo

A slightly more general setting that the multiarmed bandit. As in that problem we have a set of actions $\mathcal{A}$. The reward of each action now depends on an additional context vector, $s$. The formal setup is as follows.

For $t=1,\dots ,T$:

• We are presented with a context vector $s$
• We select an action $a$ according to our policy ${\pi }_t$ (which can depend on $s$ and the history until the current moment)
• We see reward $r_t\sim R(a,s)$.

Our goal is to minimize regret:

$$\mathbb{E}[Reg(T)]=\sum _{t=1}^T(\underset{a}{\max }[{\mathbb{E}}_{S}R(a,S)]-r_t).$$

Known results