Postscript version of this file

STAT 801 Lecture 8

Reading for Today's Lecture: ?

Goals of Today's Lecture:

• Establish distribution theory for iid normal samples.

Today's notes

Last time I listed basic properties of multivariate normals and began to prove

Theorem 1   Suppose $X_1,\ldots,X_n$ are independent $N(\mu,\sigma^2)$ random variables. Then

1.

The sample mean $\bar X$ and the sample variance s² are independent.

2.

$n^{1/2}(\bar{X} - \mu)/\sigma \sim N(0,1)$

3.

$(n-1)s^2/\sigma^2 \sim \chi^2_{n-1}$

4.

$n^{1/2}(\bar{X} - \mu)/s \sim t_{n-1}$

Since each X_i can be written as $\sigma Z_i + \mu$ where the Z_i are iid N(0,1) we can derive this theorem from

Theorem 2   Suppose $Z_1,\ldots,Z_n$ are independent N(0,1) random variables. Then

1.

The sample mean $\bar Z$ and the sample variance s² are independent.

2.

$n^{1/2}\bar{Z} \sim N(0,1)$

3.

$(n-1)s^2 = \sum(Z_i-\bar{Z})^2 \sim \chi^2_{n-1}$

4.

$\frac{n^{1/2}\bar{Z}}{s} \sim t_{n-1}$

Proof

Statement 1: The variable s² is a function of the vector $(Z_1-\bar{Z}, \ldots, Z_n -\bar{Z})$. I claim that this vector is independent of $\bar Z$. One way to prove independence is to factor the joint density of

$$(\bar{Z}, Z_1-\bar{Z}, \ldots, Z_n -\bar{Z})$$

but this wont work because this vector has n+1 components and we only started with n values of Z. The key point is this: since $\sum (Z_i - \bar{Z}) = 0$ we can write

$$Z_n - \bar{Z} = -\left[ (Z_1-\bar{Z}) + \cdots +(Z_{n-1} -\bar{Z})\right]$$

So I will consider the joint density of

$$Y = (\bar{Z}, Z_1-\bar{Z}, \ldots, Z_{n-1}-\bar{Z})$$

Note that

$$Y =\left[\begin{array}{cccc} \frac{1}{\sqrt{n}} & \frac{1}{\s... ...in{array}{c} Z_1 \\ Z_2 \\ \vdots \\ Z_n \end{array}\right]$$

or letting M denote the matrix

Y=MZ

It follows that $Y\sim MVN(0,MM^t)$ so we need to compute MM^t:
$$\begin{align*}MM^t & = \left[\begin{array}{c\vert cccc} 1 & 0 & 0 & \cdots & 0 \... ...egin{array}{c\vert c} 1 & 0 \\ \hline \\ 0 & Q \end{array}\right] \end{align*}$$
(Q is just notation for whatever is the lower corner of the matrix. The formula is an example of working with a partitioned matrix.)

Now it is easy to solve for Z from Y because Z_i = n^-1/2Y₁+Y_i+1for $1 \le i \le n-1$. To get Z_n we use the identity

$$\sum_{i=1}^n (Z_i-\bar{Z}) = 0$$

to see that $Z_n = - \sum_{i=2}^n Y_i + n^{-1/2}Y_1$. This proves that M is invertible and we find

$$\Sigma^{-1} \equiv (MM^t)^{-1} = \left[\begin{array}{c\vert c} 1 & 0 \\ \hline \\ 0 & Q^{-1} \end{array}\right]$$

Now we use the change of variables formula to compute the density of Y. It is helpful to let ${\bf y}_2$ denote the n-1 vector whose entries are $y_2,\ldots,y_n$. Note that

$$y^t\Sigma^{-1}y = y_1^2 + {\bf y}_2^t Q^{-1} {\bf y_2}$$

Then
$$\begin{align*}f_Y(y) =& (2\pi)^{-n/2} \exp[-y^t\Sigma^{-1}y/2]/\vert\det M\vert ... ...^{-(n-1)/2}\exp[-{\bf y}_2^t Q^{-1} {\bf y_2}/2]}{\vert\det M\vert} \end{align*}$$

Notice that this is a factorization into a function of y₁ times a function of $y_2,\ldots,y_n$. Thus $\sqrt{n}\bar{Z}$ is independent of $Z_1-\bar{Z},\ldots,Z_{n-1}-\bar{Z}$. Since s_Z² is a function of $Z_1-\bar{Z},\ldots,Z_{n-1}-\bar{Z}$ we see that $\sqrt{n}\bar{Z}$ and s_Z² are independent.

Furthermore the density of Y₁ is a multiple of the function of y₁ in the factorization above. But the factor in question is the standard normal density so $\sqrt{n}\bar{Z} \sim N(0,1)$.

We have now done the first 2 parts of the theorem. The third part is a homework exercise but I will outline the derivation of the $\chi^2$density.

Suppose that $Z_1,\ldots,Z_n$ are independent N(0,1). We define the $\chi^2_n$ distribution to be that of $U=Z_1^2 + \cdots + Z_n^2$. In Lecture 3 I derived the density of Z₁², that is, the $\chi_1^2$ density. In class I will derive the $\chi_2^2$ density. Here is the start of the general case, followed by details for n=3. Define angles $\theta_1,\ldots,\theta_{n-1}$ by
$$\begin{align*}Z_1 &= U^{1/2} \cos\theta_1 \\ Z_2 &= U^{1/2} \sin\theta_1\cos\th... ...\theta_{n-1} \\ Z_n &= U^{1/2} \sin\theta_1\cdots \sin\theta_{n-1} \end{align*}$$
(These are spherical co-ordinates in n dimensions. The $\theta$ values run from 0 to $\pi$ except for the last $\theta$ whose values run from 0 to $2\pi$.) Then note the following derivative formulas

$$\frac{\partial Z_i}{\partial U} = \frac{1}{2U} Z_i$$

and

$$\frac{\partial Z_i}{\partial\theta_j} = \left\{ \begin{array}... ...n\theta_i & j=i \\ Z_i\cot\theta_j & j < i \end{array}\right.$$

I now fix the case n=3 to clarify the formulas. The matrix of partial derivatives is

$$\left[\begin{array}{ccc} U^{-1/2} \cos\theta_1 /2 & -U^{1/2} ... ...theta_2 & U^{1/2} \sin\theta_1\cos\theta_2 \end{array}\right]$$

The determinant of this matrix may be found by adding $2U^{1/2}\cos\theta_j/\sin\theta_j$ times column 1 to column j+1 (which doesn't change the determinant). The resulting matrix is lower triangular with diagonal entries (after a small amount of algebra) $U^{-1/2} \cos\theta_1 /2$, $U^{1/2}\cos\theta_2/ \cos\theta_1$ and $U^{1/2} \sin\theta_1/\cos\theta_2$. We multiply these together to get

$$U^{1/2}\sin(\theta_1)/2$$

which is non-negative for all U and $\theta_1$. For general n we see that every term in the first column contains a factor U^-1/2/2 while every other entry has a factor U^1/2. Multiplying a column in a matrix by c multiplies the determinant by c so the Jacobian of the transformation is u^(n-1)/2u^-1/2/2 times some function, say h, which depends only on the angles. Thus the joint density of $U,\theta_1,\ldots \theta_{n-1}$ is

$$(2\pi)^{-n/2} \exp(-u/2) u^{(n-2)/2}h(\theta_1, \cdots, \theta_{n-1}) / 2$$

To compute the density of U we must do an n-1 dimensional multiple integral $d\theta_{n-1}\cdots d\theta_1$. We see that the answer has the form

$$cu^{(n-2)/2} \exp(-u/2)$$

for some c which we can evaluate by making

$$\int f_U(u) du = c \int u^{(n-2)/2} \exp(-u/2) du =1$$

Substitute y=u/2, du=2dy to see that

$$c 2^{(n-2)/2} 2 \int y^{(n-2)/2}e^{-y} dy = c 2^{(n-1)/2} \Gamma(n/2) = 1$$

so that the $\chi^2$ density is

$$\frac{1}{2\Gamma(n/2)} \left(\frac{u}{2}\right)^{(n-2)/2} e^{-u/2}$$

Finally the fourth part of the theorem is a consequence of the first 3 parts of the theorem and the definition of the $t_\nu$distribution, namely, that $T\sim t_\nu$ if it has the same distribution as

$$Z/\sqrt{U/\nu}$$

where $Z\sim N(0,1)$, $U\sim\chi^2_\nu$ and Z and U are independent.

However, I now derive the density of T in this definition:
$$\begin{align*}P(T \le t) &= P( Z \le t\sqrt{U/\nu}) \\ & = \int_0^\infty \int_{-\infty}^{t\sqrt{u/\nu}} f_Z(z)f_U(u) dz du \end{align*}$$
I can differentiate this with respect to t by simply differentiating the inner integral:

$$\frac{\partial}{\partial t}\int_{at}^{bt} f(x)dx = bf(bt)-af(at)$$

by the fundamental theorem of calculus. Hence

$$\frac{d}{dt} P(T \le t) = \int_0^\infty f_U(u) \sqrt{u/\nu}\frac{\exp[-t^2u/(2\nu)]}{\sqrt{2\pi}} du \, .$$

Now I plug in

$$f_U(u)= \frac{1}{2\Gamma(\nu/2)}(u/2)^{(\nu-2)/2} e^{-u/2}$$

to get

$$f_T(t) = \int_0^\infty \frac{1}{2\sqrt{\pi\nu}\Gamma(\nu/2)} (u/2)^{(\nu-1)/2} \exp[-u(1+t^2/\nu)/2] \, du \, .$$

Make the substitution $y=u(1+t^2/\nu)/2$, $dy=(1+t^2/\nu)du/2$ $(u/2)^{(\nu-1)/2}= [y/(1+t^2/\nu)]^{(\nu-1)/2}$to get

$$f_T(t) = \frac{1}{\sqrt{\pi\nu}\Gamma(\nu/2)}(1+t^2/\nu)^{-(\nu+1)/2} \int_0^\infty y^{(\nu-1)/2} e^{-y} dy$$

or

$$f_T(t)= \frac{\Gamma((\nu+1)/2)}{\sqrt{\pi\nu}\Gamma(\nu/2)}\frac{1}{(1+t^2/\nu)^{(\nu+1)/2}}$$