Postscript version of this page

STAT 801: Mathematical Statistics

Normal samples: Distribution Theory

Theorem: Suppose $X_1,\ldots,X_n$ are independent $N(\mu,\sigma^2)$ random variables. Then

1. $\bar X$ (sample mean)and $s^2$ (sample variance) independent.

2. $n^{1/2}(\bar{X} - \mu)/\sigma \sim N(0,1)$.

3. $(n-1)s^2/\sigma^2 \sim \chi^2_{n-1}$.

4. $n^{1/2}(\bar{X} - \mu)/s \sim t_{n-1}$.

Proof: Let $Z_i=(X_i-\mu)/\sigma$.

Then $Z_1,\ldots,Z_p$ are independent $N(0,1)$.

So $Z=(Z_1,\ldots,Z_p)^t$ is multivariate standard normal.

Note that $\bar{X} = \sigma\bar{Z}+\mu$ and $s^2 = \sum(X_i-\bar{X})^2/(n-1) = \sigma^2 \sum(Z_i-\bar{Z})^2/(n-1)$ Thus

$$\frac{n^{1/2}(\bar{X}-\mu)}{\sigma} = n^{1/2}\bar{Z}$$

$$\frac{(n-1)s^2}{\sigma^2} = \sum(Z_i-\bar{Z})^2$$

and

$$T=\frac{n^{1/2}(\bar{X} - \mu)}{s} = \frac{n^{1/2} \bar{Z}}{s_Z}$$

where $(n-1)s_Z^2 = \sum(Z_i-\bar{Z})^2$.

So: reduced to $\mu=0$ and $\sigma=1$.

Step 1: Define

$$Y=(\sqrt{n}\bar{Z}, Z_1-\bar{Z},\ldots,Z_{n-1}-\bar{Z})^t \, .$$

(So $Y$ has same dimension as $Z$.) Now

$$Y =\left[\begin{array}{cccc} \frac{1}{\sqrt{n}} & \frac{1}{\sqrt{... ...] \left[\begin{array}{c} Z_1 \\ Z_2 \\ \vdots \\ Z_n \end{array}\right]$$

or letting $M$ denote the matrix

$$Y=MZ \, .$$

It follows that $Y\sim MVN(0,MM^t)$ so we need to compute $MM^t$:

$$MM^t = \left[\begin{array}{c\vert cccc} 1 & 0 & 0 & \cdots & 0 \\ \hli... ...ots & -\frac{1}{n} \\ 0 & \vdots & \cdots & & 1-\frac{1}{n} \end{array} \right]$$

$$= \left[\begin{array}{c\vert c} 1 & 0 \\ \hline \\ 0 & Q \end{array} \right] \, .$$

Solve for $Z$ from $Y$: $Z_i = n^{-1/2}Y_1+Y_{i+1}$ for $1 \le i \le n-1$. Use the identity

$$\sum_{i=1}^n (Z_i-\bar{Z}) = 0$$

to get $Z_n = - \sum_{i=2}^n Y_i + n^{-1/2}Y_1$. So $M$ invertible:

$$\Sigma^{-1} \equiv (MM^t)^{-1} = \left[\begin{array}{c\vert c} 1 & 0 \\ \hline \\ 0 & Q^{-1} \end{array}\right]$$

Use change of variables to find $f_Y$. Let ${\bf y}_2$ denote vector whose entries are $y_2,\ldots,y_n$. Note that

$$y^t\Sigma^{-1}y = y_1^2 + {\bf y}_2^t Q^{-1} {\bf y}_2$$

Then

$$f_Y(y) = (2\pi)^{-n/2} \exp[-y^t\Sigma^{-1}y/2]/\vert\det M\vert$$

$$= \frac{1}{\sqrt{2\pi}} e^{-y_1^2/2} \times$$

$$\frac{(2\pi)^{-(n-1)/2}\exp[-{\bf y}_2^t Q^{-1} {\bf y}_2/2]}{\vert\det M\vert}$$

Note: $f_Y(y)$ is ftn of $y_1$ times a ftn of $y_2,\ldots,y_n$.

Thus $\sqrt{n}\bar{Z}$ is independent of $Z_1-\bar{Z},\ldots,Z_{n-1}-\bar{Z}$.

Since $s_Z^2$ is a function of $Z_1-\bar{Z},\ldots,Z_{n-1}-\bar{Z}$ we see that $\sqrt{n}\bar{Z}$ and $s_Z^2$ are independent.

Also, density of $Y_1$ is a multiple of the function of $y_1$ in the factorization above. But factor is standard normal density so $\sqrt{n}\bar{Z}\sim N(0,1)$.

First 2 parts done. Third part is a homework exercise.

Derivation of the $\chi^2$ density:

Suppose $Z_1,\ldots,Z_n$ are independent $N(0,1)$. Define $\chi^2_n$ distribution to be that of $U=Z_1^2 + \cdots + Z_n^2$. Define angles $\theta_1,\ldots,\theta_{n-1}$ by

$$Z_1 = U^{1/2} \cos\theta_1$$

$$Z_2 = U^{1/2} \sin\theta_1\cos\theta_2$$

$$\vdots = \vdots$$

$$Z_{n-1} = U^{1/2} \sin\theta_1\cdots \sin\theta_{n-2}\cos\theta_{n-1}$$

$$Z_n = U^{1/2} \sin\theta_1\cdots \sin\theta_{n-1}$$

(Spherical co-ordinates in $n$ dimensions. The $\theta$ values run from 0 to $\pi$ except last $\theta$ from 0 to $2\pi$.) Derivative formulas:

$$\frac{\partial Z_i}{\partial U} = \frac{1}{2U} Z_i$$

and

$$\frac{\partial Z_i}{\partial\theta_j} = \left\{ \begin{array}{ll}... ...>i \\ -Z_i\tan\theta_i & j=i \\ Z_i\cot\theta_j & j < i \end{array}\right.$$

Fix $n=3$ to clarify the formulas.

Use shorthand $R=\sqrt{U}$.

Matrix of partial derivatives is

$$\left[\begin{array}{ccc} \frac{\cos\theta_1}{2R} & -R \sin\theta_... ...R} & R cos\theta_1\sin\theta_2 & R \sin\theta_1\cos\theta_2 \end{array}\right]$$

Find determinant by adding $2U^{1/2}\cos\theta_j/\sin\theta_j$ times col 1 to col $j+1$ (no change in determinant).

Resulting matrix is lower triangular; diagonal entries $U^{-1/2} \cos\theta_1 /2$, $U^{1/2}\cos\theta_2/ \cos\theta_1$ and $U^{1/2} \sin\theta_1/\cos\theta_2$.

We multiply these together to get

$$U^{1/2}\sin(\theta_1)/2$$

(non-negative for all $U$ and $\theta_1$).

General $n$: every term in the first column contains a factor $U^{-1/2}/2$ while every other entry has a factor $U^{1/2}$.

FACT: Multiplying a column in a matrix by $c$ multiplies the determinant by $c$.

SO: Jacobian of the transformation is $u^{(n-1)/2}u^{-1/2}/2$ times some function, say $h$, which depends only on the angles.

Thus the joint density of $U,\theta_1,\ldots \theta_{n-1}$ is

$$(2\pi)^{-n/2} \exp(-u/2) u^{(n-2)/2}h(\theta_1, \cdots, \theta_{n-1}) / 2$$

To compute the density of $U$ we must do an $n-1$ dimensional multiple integral $d\theta_{n-1}\cdots d\theta_1$.

Answer has the form

$$cu^{(n-2)/2} \exp(-u/2)$$

for some $c$.

Evaluate $c$ by making

$$\int f_U(u) du = c \int u^{(n-2)/2} \exp(-u/2) du =1$$

Substitute $y=u/2$, $du=2dy$ to see that

$$c 2^{n/2} \int y^{(n-2)/2}e^{-y} dy = c 2^{n/2} \Gamma(n/2) = 1$$

CONCLUSION: the $\chi^2_n$ density is

$$\frac{1}{2\Gamma(n/2)} \left(\frac{u}{2}\right)^{(n-2)/2} e^{-u/2} 1(u>0) \, .$$

Fourth part: consequence of first 3 parts and def'n of $t_\nu$ distribution.

Defn: $T\sim t_\nu$ if $T$ has same distribution as

$$Z/\sqrt{U/\nu}$$

where $Z\sim N(0,1)$, $U\sim\chi^2_\nu$ and $Z$ and $U$ are independent.

Derive density of $T$ in this definition:

$$P(T \le t) = P( Z \le t\sqrt{U/\nu})$$

$$= \int_0^\infty \int_{-\infty}^{t\sqrt{u/\nu}} f_Z(z)f_U(u) dz du$$

Differentiate wrt $t$ by differentiating inner integral:

$$\frac{\partial}{\partial t}\int_{at}^{bt} f(x)dx = bf(bt)-af(at)$$

by fundamental thm of calculus. Hence

$$\frac{d}{dt} P(T \le t) = \int_0^\infty f_U(u) \left(\frac{u}{\nu}\right)^{1/2} \frac{\exp[-t^2u/(2\nu)]}{\sqrt{2\pi}} du$$

Plug in

$$f_U(u)= \frac{1}{2\Gamma(\nu/2)}(u/2)^{(\nu-2)/2} e^{-u/2}$$

to get

$$f_T(t) = \frac{\int_0^\infty (u/2)^{(\nu-1)/2} \exp[-u(1+t^2/\nu)/2]du }{2\sqrt{\pi\nu}\Gamma(\nu/2)} \, .$$

Substitute $y=u(1+t^2/\nu)/2$, to get

$$dy=(1+t^2/\nu)du/2$$

$$(u/2)^{(\nu-1)/2}= [y/(1+t^2/\nu)]^{(\nu-1)/2}$$

leading to

$$f_T(t) = \frac{(1+t^2/\nu)^{-(\nu+1)/2} }{\sqrt{\pi\nu}\Gamma(\nu/2)} \int_0^\infty y^{(\nu-1)/2} e^{-y} dy$$

or

$$f_T(t)= \frac{\Gamma((\nu+1)/2) }{\sqrt{\pi\nu}\Gamma(\nu/2)} \frac{1}{(1+t^2/\nu)^{(\nu+1)/2}}\ ,.$$

Postscript version of this page