Interactive visualization of Mixture of Gaussians, the Law of Total Expectation and the Law of Total Variance

The laws of total expectation and variance hold for any probability distribution, but are illustrated here with a Gaussian mixture, which is amenable to a nice visual decomposition.

This demo probably needs Firefox or Chrome to work.

Mixture of Gaussians

A data point's Y is one of three components, and its X is drawn from that Gaussian component. In generative notation, each data point has two draws — first Y, then X conditioned on that y:
\begin{align} Y &\sim Discrete(p_1, p_2, p_3) \\ X|y &\sim Normal(\mu_y, \sigma^2_y) \end{align}
We are interested in the marginal X. Its density function is
\[ f(x) = \sum_y\ p(y)\ f_{Normal}(x; \mu_y, \sigma^2_y) \ \ \propto\ \sum_y\ p(y)\ \sigma^{-1}_y e^{ -(x-\mu_y)^2/{2 \sigma^2_y}} \]
Below is a plot of this marginal density. The sum across the three components is represented as a stacked area graph, so each individual color is a single Gaussian density.
\( X \)
The goal is to break down the expectation and variance of the marginal X, by conditioning on Y.

Law of Total Expectation

\( E[X] = E( E[X|Y] ) \)

Decompose the mean by the means of each component. We show calculations progressing downward. Every colored bar represents the mean for one component, and the final bar represents the marginal mean.

\( X \)
each X|y
\( E[X|y] \)
sized by p(y)
\( E(E[X|Y]) = \) \( \sum_y\ p(y)\ E[X|y] \)
mean of above (weighted by choice of Y)
\( = E[X] = \)
↓ Click on a parameter to manipulate it.
Prior Location and scale
\[p(y)\] \[ E[X|y] \] \[ V[X|y] \]
y=1 A P s1
y=2 B Q s2
y=3 C R s3

Law of Total Variance

Horizontal bars represent variance.

\( V[X] = \) \( V(E[X|Y]) \) \( + \) \( E(V[X|Y]) \)
variance as
variance of component means
("explaned variance")
and average component variance
("unexplained variance")

\( X \)
\( X \)
\( E[X|y] \) \( V[X|y] \)
\( V(E[X|Y]) = \) \( \sum_y\ p(y)\ (E[X|y] - E[X])^2 \)
var of above (over choice of Y)
\( E(V[X|Y]) = \) \( \sum_y\ p(y)\ V[X|y] \)
mean of above (over choice of Y)
\( V(E[X|Y]) = \)
\( E(V[X|Y]) = \)
\( V[X] = \)


Plotting with Protovis. Also used MathJax and jQuery. Wikipedia articles: There are also many references to Gaussian mixtures on the web; many are tutorials on fitting the parameters to data, which is not a concern here.

Created Dec. 2010 by Brendan O'Connor.


blog comments powered by Disqus