Keywords: gibbs sampler |  metropolis |  mcmc |  metropolis-hastings | 


We show that Gibbs is a special case of a MH sampler with proposal distribution a conditional on one variable holding all others constant.

We need to show that Gibbs Sampling converges to our target distribution. We start with a componentwise MH update.

We shall use the following notation, assuming that $x$ is multidimensional: $x_{k}^i$ is the $k$th component of $x$ at $i$th step, while $x_{-k}^i$ is all other componets of $x$ at the same step.

Now consider the following proposal function:

In other words, you propose from the conditional distribution of the current component, given a fixed $x_{old}$ value of the other components. You keep all the other component values the same. If you change any of the old component values, the proposal probability is 0. This should remind you of componentwise updating in MH (and block, as $k$ could represent a block). Now you see where this conditional takes you in the $x_k$ direction.

Under this proposal distribution, let us see what our MH acceptance probability term:

gives us. We use $p(x^*) = p(x_{-k}^*, x_k^*) = p(x_{k}^* \vert x_{-k}^*)p(x_{-k}^*)$ to obtain

Putting in the definition of the proposal, we get

Now, because of the componentwise update, $x_{-k}^* = x_{-k}^i$ and thus we get the middle 2 terms in the numerator and denominator cancelling, and the 1st and 3rd terms from the numerator cancelling the 3rd and 1st terms respectively. Thus our acceptance probability is 1.

We have constructed a MH sampler with no rejection!!