Postscript version of this file

STAT 801 Lecture 7

Reading for Today's Lecture:

Goals of Today's Lecture:

• Derive sampling distribution of sample mean and variance for iid normal samples.

Today's notes

Normal samples: Distribution Theory

Theorem 1   Suppose $X_1,\ldots,X_n$ are independent $N(\mu,\sigma^2)$ random variables. (That is each satisfies my definition above in 1 dimension.) Then

1.

The sample mean $\bar X$ and the sample variance s² are independent.

2.

$n^{1/2}(\bar{X} - \mu)/\sigma \sim N(0,1)$

3.

$(n-1)s^2/\sigma^2 \sim \chi^2_{n-1}$

4.

$n^{1/2}(\bar{X} - \mu)/s \sim t_{n-1}$

Proof: Let $Z_i=(X_i-\mu)/\sigma$. Then $Z_1,\ldots,Z_p$ are independent N(0,1) so $Z=(Z_1,\ldots,Z_p)^t$ is multivariate standard normal. Note that $\bar{X} = \sigma\bar{Z}+\mu$ and $s^2 = \sum(X_i-\bar{X})^2/(n-1) = \sigma^2 \sum(Z_i-\bar{Z})^2/(n-1)$Thus

$$\frac{n^{1/2}(\bar{X}-\mu)}{\sigma} = n^{1/2}\bar{Z}$$

$$\frac{(n-1)s^2}{\sigma^2} = \sum(Z_i-\bar{Z})^2$$

and

$$T=\frac{n^{1/2}(\bar{X} - \mu)}{s} = \frac{n^{1/2} \bar{Z}}{s_Z}$$

where $(n-1)s_Z^2 = \sum(Z_i-\bar{Z})^2$.

Thus we need only give a proof in the special case $\mu=0$ and $\sigma=1$.

Step 1: Define

$$Y=(\sqrt{n}\bar{Z}, Z_1-\bar{Z},\ldots,Z_{n-1}-\bar{Z})^t \, .$$

(This choice means Y has the same dimension as Z.) Now

$$Y =\left[\begin{array}{cccc} \frac{1}{\sqrt{n}} & \frac{1}{\s... ...in{array}{c} Z_1 \\ Z_2 \\ \vdots \\ Z_n \end{array}\right]$$

or letting M denote the matrix

Y=MZ

It follows that $Y\sim MVN(0,MM^t)$ so we need to compute MM^t:
$$\begin{align*}MM^t & = \left[\begin{array}{c\vert cccc} 1 & 0 & 0 & \cdots & 0 \... ...egin{array}{c\vert c} 1 & 0 \\ \hline \\ 0 & Q \end{array}\right] \end{align*}$$

Now it is easy to solve for Z from Y because Z_i = n^-1/2Y₁+Y_i+1for $1 \le i \le n-1$. To get Z_n we use the identity

$$\sum_{i=1}^n (Z_i-\bar{Z}) = 0$$

to see that $Z_n = - \sum_{i=2}^n Y_i + n^{-1/2}Y_1$. This proves that M is invertible and we find

$$\Sigma^{-1} \equiv (MM^t)^{-1} = \left[\begin{array}{c\vert c} 1 & 0 \\ \hline \\ 0 & Q^{-1} \end{array}\right]$$

Now we use the change of variables formula to compute the density of Y. It is helpful to let ${\bf y}_2$ denote the n-1 vector whose entries are $y_2,\ldots,y_n$. Note that

$$y^t\Sigma^{-1}y = y_1^2 + {\bf y}_2^t Q^{-1} {\bf y_2}$$

Then
$$\begin{align*}f_Y(y) =& (2\pi)^{-n/2} \exp[-y^t\Sigma^{-1}y/2]/\vert\det M\vert ... ...^{-(n-1)/2}\exp[-{\bf y}_2^t Q^{-1} {\bf y_2}/2]}{\vert\det M\vert} \end{align*}$$

Notice that this is a factorization into a function of y₁ times a function of $y_2,\ldots,y_n$. Thus $\sqrt{n}\bar{Z}$ is independent of $Z_1-\bar{Z},\ldots,Z_{n-1}-\bar{Z}$. Since s_Z² is a function of $Z_1-\bar{Z},\ldots,Z_{n-1}-\bar{Z}$ we see that $\sqrt{n}\bar{Z}$ and s_Z² are independent.

Furthermore the density of Y₁ is a multiple of the function of y₁ in the factorization above. But the factor in question is the standard normal density so $\sqrt{n}\bar{Z}N(0,1)$.

We have now done the first 2 parts of the theorem. The third part is a homework exercise but I will outline the derivation of the $\chi^2$density. In class I will do the special cae n=2 (which is notationally much simpler).

Suppose that $Z_1,\ldots,Z_n$ are independent N(0,1). We define the $\chi^2_n$ distribution to be that of $U=Z_1^2 + \cdots + Z_n^2$. Define angles $\theta_1,\ldots,\theta_{n-1}$ by
$$\begin{align*}Z_1 &= U^{1/2} \cos\theta_1 \\ Z_2 &= U^{1/2} \sin\theta_1\cos\th... ...\theta_{n-1} \\ Z_n &= U^{1/2} \sin\theta_1\cdots \sin\theta_{n-1} \end{align*}$$
(These are spherical co-ordinates in n dimensions. The $\theta$ values run from 0 to $\pi$ except for the last $\theta$ whose values run from 0 to $2\pi$.) Then note the following derivative formulas

$$\frac{\partial Z_i}{\partial U} = \frac{1}{2U} Z_i$$

and

$$\frac{\partial Z_i}{\partial\theta_j} = \left\{ \begin{array}... ...n\theta_i & j=i \\ Z_i\cot\theta_j & j < i \end{array}\right.$$

I now fix the case n=3 to clarify the formulas. The matrix of partial derivatives is

$$\left[\begin{array}{ccc} U^{-1/2} \cos\theta_1 /2 & -U^{1/2} ... ...theta_2 & U^{1/2} \sin\theta_1\cos\theta_2 \end{array}\right]$$

The determinant of this matrix may be found by adding $2U^{1/2}\cos\theta_j/\sin\theta_j$ times column 1 to column j+1 (which doesn't change the determinant). The resulting matrix is lower triangular with diagonal entries (after a small amount of algebra) $U^{-1/2} \cos\theta_1 /2$, $U^{1/2}\cos\theta_2/ \cos\theta_1$ and $U^{1/2} \sin\theta_1/\cos\theta_2$. We multiply these together to get

$$U^{1/2}\sin(\theta_1)/2$$

which is non-negative for all U and $\theta_1$. For general n we see that every term in the first column contains a factor U^-1/2/2 while every other entry has a factor U^1/2. Multiplying a column in a matrix by c multiplies the determinant by c so the Jacobian of the transformation is u^(n-1)/2u^-1/2/2 times some function, say h, which depends only on the angles. Thus the joint density of $U,\theta_1,\ldots \theta_{n-1}$ is

$$(2\pi)^{-n/2} \exp(-u/2) u^{(n-2)/2}h(\theta_1, \cdots, \theta_{n-1}) / 2$$

To compute the density of U we must do an n-1 dimensional multiple integral $d\theta_{n-1}\cdots d\theta_1$. We see that the answer has the form

$$cu^{(n-2)/2} \exp(-u/2)$$

for some c which we can evaluate by making

$$\int f_U(u) du = c \int u^{(n-2)/2} \exp(-u/2) du =1$$

Substitute y=u/2, du=2dy to see that

$$c 2^{(n-2)/2} 2 \int y^{(n-2)/2}e^{-y} dy = c 2^{(n-1)/2} \Gamma(n/2) = 1$$

so that the $\chi^2$ density is

$$\frac{1}{2\Gamma(n/2)} \left(\frac{u}{2}\right)^{(n-2)/2} e^{-u/2}$$

Finally the fourth part of the theorem is a consequence of the first 3 parts of the theorem and the definition of the $t_\nu$distribution, namely, that $T\sim t_\nu$ if it has the same distribution as

$$Z/\sqrt{U/\nu}$$

where $Z\sim N(0,1)$, $U\sim\chi^2_\nu$ and Z and U are independent.

However, I now derive the density of T in this definition:
$$\begin{align*}P(T \le t) &= P( Z \le t\sqrt{U/\nu}) \\ & = \int_0^\infty \int_{-\infty}^{t\sqrt{u/\nu}} f_Z(z)f_U(u) dz du \end{align*}$$
I can differentiate this with respect to t by simply differentiating the inner integral:

$$\frac{\partial}{\partial t}\int_{at}^{bt} f(x)dx = bf(bt)-af(at)$$

by the fundamental theorem of calculus. Hence

$$\frac{d}{dt} P(T \le t) = \int_0^\infty f_U(u) \sqrt{u/\nu}\frac{\exp[-t^2u/(2\nu)]}{\sqrt{2\pi}} du \, .$$

Now I plug in

$$f_U(u)= \frac{1}{2\Gamma(\nu/2)}(u/2)^{(\nu-2)/2} e^{-u/2}$$

to get

$$f_T(t) = \int_0^\infty \frac{1}{2\sqrt{\pi\nu}\Gamma(\nu/2)} (u/2)^{(\nu-1)/2} \exp[-u(1+t^2/\nu)/2] \, du \, .$$

Make the substitution $y=u(1+t^2/\nu)/2$, $dy=(1+t^2/\nu)du/2$ $(u/2)^{(\nu-1)/2}= [y/(1+t^2/\nu)]^{(\nu-1)/2}$to get

$$f_T(t) = \frac{1}{\sqrt{\pi\nu}\Gamma(\nu/2)}(1+t^2/\nu)^{-(\nu+1)/2} \int_0^\infty y^{(\nu-1)/2} e^{-y} dy$$

or

$$f_T(t)= \frac{\Gamma((\nu+1)/2)}{\sqrt{\pi\nu}\Gamma(\nu/2)}\frac{1}{(1+t^2/\nu)^{(\nu+1)/2}}$$