Postscript version of this page

STAT 801: Mathematical Statistics

The Multivariate Normal Distribution

Def'n: $Z \in R^1 \sim N(0,1)$ iff

$$f_Z(z) = \frac{1}{\sqrt{2\pi}} e^{-z^2/2}$$

Def'n: $Z \in R^p \sim MVN(0,I)$ if and only if $Z=(Z_1,\ldots,Z_p)^t$ with the $Z_i$ independent and each $Z_i\sim N(0,1)$.

In this case according to our theorem

$$f_Z(z_1,\ldots,z_p) = \prod \frac{1}{\sqrt{2\pi}} e^{-z_i^2/2}$$

$$= (2\pi)^{-p/2} \exp\{ -z^t z/2\} \, ;$$

superscript $t$ denotes matrix transpose.

Def'n: $X\in R^p$ has a multivariate normal distribution if it has the same distribution as $AZ+\mu$ for some $\mu\in R^p$, some $p\times p$ matrix of constants $A$ and $Z\sim MVN(0,I)$.

If the matrix $A$ is singular then $X$ will not have a density.

If $A$ is invertible then we can derive the multivariate normal density by the change of variables formula:

$$X=AZ+\mu \Leftrightarrow Z=A^{-1}(X-\mu)$$

$$\frac{\partial X}{\partial Z} = A \qquad \frac{\partial Z}{\partial X } = A^{-1} \, .$$

So

$$f_X(x) = f_Z(A^{-1}(x-\mu)) \vert \det(A^{-1})\vert$$

$$= \frac{ \exp\{-(x-\mu)^t (A^{-1})^t A^{-1} (x-\mu)/2\} }{(2\pi)^{p/2} \vert\det{A}\vert } \, .$$

Now define $\Sigma=AA^t$ and notice that

$$\Sigma^{-1} = (A^t)^{-1} A^{-1} = (A^{-1})^t A^{-1}$$

and

$$\det \Sigma = \det A \det A^t = (\det A)^2 \, .$$

Thus $f_X$ is

$$\frac{ \exp\{ -(x-\mu)^t \Sigma^{-1} (x-\mu) /2 \}}{ (2\pi)^{p/2} (\det\Sigma)^{1/2} } \, ;$$

the $MVN(\mu,\Sigma)$ density.

Note density is the same for all $A$ such that $AA^t=\Sigma$. This justifies the notation $MVN(\mu,\Sigma)$.

For which vectors $\mu$ and matrices $\Sigma$ is this a density? Any $\mu$ but if $x \in R^p$ then

$$x^t \Sigma x = x^t A A^t x$$

$$= (A^t x)^t (A^t x)$$

$$= \sum_1^p y_i^2 \ge 0$$

where $y=A^t x$.

Inequality strict except for $y=0$ which is equivalent to $x=0$. Thus $\Sigma$ is a positive definite symmetric matrix.

Conversely, if $\Sigma$ is a positive definite symmetric matrix then there is a square invertible matrix $A$ such that $AA^t=\Sigma$ so that there is a $MVN(\mu,\Sigma)$ distribution. ($A$ can be found via the Cholesky decomposition, e.g.)

More generally $X$ has MVN distribution if it has the same distribution as $AZ+\mu$ (no restriction that $A$ be non-singular). When $A$ is singular $X$ will not have a density: $\exists a$ such that $P(a^t X = a^t \mu) =1$; $X$ is confined to a hyperplane. Still true that the distribution of $X$ depends only on the matrix $\Sigma=AA^t$: if $AA^t = BB^t$ then $AZ+\mu$ and $BZ+\mu$ have the same distribution.

Properties of the $MVN$ distribution

1: All margins are multivariate normal: if

$$X = \left[\begin{array}{c} X_1\\ X_2\end{array} \right]$$

$$\mu = \left[\begin{array}{c} \mu_1\\ \mu_2\end{array} \right]$$

and

$$\Sigma = \left[\begin{array}{cc} \Sigma_{11} & \Sigma_{12} \\ \Sigma_{21} & \Sigma_{22} \end{array} \right]$$

then $X\sim MVN(\mu,\Sigma) \Rightarrow X_1\sim MVN(\mu_1,\Sigma_{11})$.

2: All conditionals are normal: the conditional distribution of $X_1$ given $X_2=x_2$ is

$$MVN(\mu_1+\Sigma_{12}\Sigma_{22}^{-1} (x_2-\mu_2),\Sigma_{11}-\Sigma_{12}\Sigma_{22}^{-1}\Sigma_{21})\, .$$

3: $MX+\nu \sim MVN(M\mu+\nu, M \Sigma M^t)$: affine transformation of MVN is normal.