The Dirichlet process is a commonly used prior for the task of cdf estimation in Bayesian nonparametrics. It was invented by Thomas Ferguson in 1973.
The Dirichlet process is a distribution over distributions. Specifically, for a base distribution and a concentration parameter , a DP prior generates random distributions . Intuitively, controls how tightly the prior is around , similarly to the variance of the normal distribution.
Sampling from the prior.
We use the “stick breaking” procedure: First draw $\beta_k\sim \text{Beta}(1,\alpha), then set
which satisfies . Then draw independently, and finally set . In practice we stop after some finite number .
Sampling from the marginal
As usual, can draw (using stick breaking), and then draw from which is a draw from the marginal.
An alternative method which doesn’t involve two steps is called the “Chinese restaurant process”. Here we draw , and then
where is the empirical distribution on . (So there will very likely be duplicates in the sample). It’s called the Chinese restaurant process since we can imagine it as follows: the -th customer walks into the restaurant, and sits at each table proportional to the number of people already at that table. With some probability he sits at a new table.
Sampling from the posterior
Turns out that the posterior is also a Dirichlet distribution, namely if and has prior , then the posterior is where
where again is the empirical cdf on the points.