The laws of total expectation and variance hold for any probability distribution, but are
illustrated here with a Gaussian mixture, which is amenable to a nice visual decomposition.
This demo probably needs Firefox or Chrome to work.
Mixture of Gaussians
A data point's Y is one of three components, and its X is drawn from that Gaussian component. In generative notation, each data point has two draws — first Y, then X conditioned on that y:
Below is a plot of this marginal density. The sum across the three components is represented as a stacked area graph, so each individual color is a single Gaussian density.
\( X \)
The goal is to break down the expectation and variance of the marginal X, by conditioning on Y.
Law of Total Expectation
\( E[X] = E( E[X|Y] ) \)
Decompose the mean by the means of each component. We show calculations progressing downward. Every colored bar represents the mean for one component, and the final bar represents the marginal mean.
\( X \)
stacking each X|y
\( E[X|y] \)
sized by p(y)
\( E(E[X|Y]) = \)
\( \sum_y\ p(y)\ E[X|y] \)
mean of above (weighted by choice of Y)
\( = E[X] = \)
↓ Click on a parameter to manipulate it.
Prior
Location and scale
\[p(y)\]
\[ E[X|y] \]
\[ V[X|y] \]
y=1
A
P
s1
y=2
B
Q
s2
y=3
C
R
s3
Law of Total Variance
Horizontal bars represent variance.
\( V[X] = \)
\( V(E[X|Y]) \)
\( + \)
\( E(V[X|Y]) \)
Decompose variance as
variance of component means ("explaned variance")
and
average component variance ("unexplained variance")