STAT 450

Lecture 30

Goals for today:

• Introduce concepts of Hypothesis Testing

• Define power, level, most powerful.

• State and prove the Neyman Pearson Lemma

Hypothesis Testing

Null hypothesis: $H_o: \theta\in \Theta_0$
Alternative hypothesis: $H_1: \theta\in\Theta_1$
Rejection region: $\{x: \text{reject if see $x$ }\}$.
Test function: $\phi(x) = 1(\text{reject if see $x$ })$.
Power function: $\pi(\theta) = E_\theta(\phi(X))$
Type I error: incorrectly ``reject'' H<sub>o</sub>.
Type II error: incorrectly ``accept'' H_o.
Simple hypothesis: $\Theta_i$ has cardinality 1.

Simple versus Simple tests

For simple hypotheses we denote $\beta=P_1(\text{accept}) =1-\pi(\theta_1)$and $\alpha = P_0(\text{accept}) - \pi(\theta_0)$.

Theorem: For each fixed $\lambda$ the quantity $\beta+\lambda \alpha$ is minimized by any $\phi$ which has

$$\phi(x) =\left\{\begin{array}{ll} 1 & \frac{f_1(x)}{f_0(x)} >... ...bda \\ 0 & \frac{f_1(x)}{f_0(x)} < \lambda \end{array}\right.$$

Most Powerful Level $\alpha_0$ test maximizes $1-\beta$subject to $\alpha \le \alpha_0$.

Example: X is Binomial(5,p) p₀=1/2, p₁=3/4. Most powerful level 0.05 test $\phi(x) =1(x=5)$. Most powerful level 0.0625 test $\phi(x) =1(x\in \{0,5\})$. But X=0 is evidence for p=1/2 not for p=3/4.

Solution: Let $0 \le \phi(x) \le 1$. If $\phi(x)=0.6$ then if we see X=x we toss coin (P(H) =0.6) and reject if get heads.

Definition: A hypothesis test is a function $\phi(x)$whose values are always in [0,1]. If we observe X=x then we choose H₁ with conditional probability $\phi(X)$. In this case we have

$$\pi(\theta) = E_\theta(\phi(X))$$

$$\alpha = E_0(\phi(X))$$

and

$$\beta = E_1(1-\phi(X))$$

Now to example: try

$$\phi(x) = \begin{cases}1 & x=5 \\ 1/5 & x=4 \\ 0 & \text{otherwise} \end{cases}$$

Then
$$\begin{align*}\alpha & = \text{E}_0(\phi(X)) \\ & = P_0(X=5) + (1/5) P_0(X=4) \\ & = \frac{1}{32} + \frac{1}{5}\cdot \frac{5}{32} \\ & = 0.0625 \end{align*}$$
and
$$\begin{align*}\beta & = \text{E}_1(1-\phi(X)) \\ & = 1-\left[ P_1(X=5) + (1/5) ... ...{4}\right)^5 - \left(\frac{1}{5}\cdot \frac{5\cdot 3^4}{4^5}\right) \end{align*}$$

It can be verified that this test has the same level as the test $1(x\in\{0,5\})$and smaller probability of Type II error.

Definition: A hypothesis test is a function $\phi(x)$whose values are always in [0,1]. If we observe X=x then we choose H₁ with conditional probability $\phi(X)$. In this case we have

$$\pi(\theta) = E_\theta(\phi(X))$$

$$\alpha = E_0(\phi(X))$$

and

$$\beta = E_1(\phi(X))$$

The Neyman Pearson Lemma

Theorem: In testing f₀ against f₁ the probability $\beta$ of a type II error is minimized, subject to $\alpha \le \alpha_0$by the test function:

$$\phi(x) =\left\{\begin{array}{ll} 1 & \frac{f_1(x)}{f_0(x)} >... ...bda \\ 0 & \frac{f_1(x)}{f_0(x)} < \lambda \end{array}\right.$$

where $\lambda$ is the largest constant such that

$$P_0( \frac{f_1(x)}{f_0(x)} \ge \lambda) \ge \alpha_0$$

and

$$P_0( \frac{f_1(x)}{f_0(x)}\le \lambda) \ge 1-\alpha_0$$

and where $\gamma$ is any number chosen so that

$$E_0(\phi(X)) = P_0( \frac{f_1(x)}{f_0(x)} > \lambda) + \gamma P_0( \frac{f_1(x)}{f_0(x)} =\lambda) = \alpha_0$$

The value of $\gamma$ is unique if $P_0( \frac{f_1(x)}{f_0(x)} = \lambda) > 0$.

Example: In the Binomial(n,p) with p₀=1/2 and p₁=3/4the ratio f₁/f₀ is

3^x 2^-n

Now if n=5 then this ratio must be one of the numbers 1, 3, 9, 27, 81, 243 divided by 32. Suppose we have $\alpha = 0.05$. The value of $\lambda$ must be one of the possible values of f₁/f₀. If we try $\lambda = 243/32$ then

$$P_0(3^X 2^{-5} \ge 243/32) = P_0(X=5) = 1/32 < 0.05$$

and

$$P_0(3^X 2^{-5} \ge 81/32) = P_0(X \ge 4) = 6/32 > 0.05$$

This means that $\lambda=81/32$. Since

P₀(3^X 2^-5 > 81/32) =P₀( X=5) =1/32

we must solve

$$P_0(X=5) + \gamma P_0(X=4) = 0.05$$

for $\gamma$ and find

$$\gamma = \frac{0.05-1/32}{5/32}= 0.12$$

NOTE: No-one ever uses this procedure. Instead the value of $\alpha_0$used in discrete problems is chosen to be a possible value of the rejection probability when $\gamma=0$ (or $\gamma=1$). When the sample size is large you can come very close to any desired $\alpha_0$ with a non-randomized test.

If $\alpha_0=6/32$ then we can either take $\lambda$ to be 243/32 and $\gamma=1$ or $\lambda=81/32$ and $\gamma=0$. However, our definition of $\lambda$ in the theorem makes $\lambda=81/32$ and $\gamma=0$.

When the theorem is used for continuous distributions it can be the case that the cdf of f₁(X)/f₀(X) has a flat spot where it is equal to $1-\alpha_0$. This is the point of the word ``largest'' in the theorem.

Example: If $X_1,\ldots,X_n$ are iid $N(\mu,1)$ and we have $\mu_0=0$ and $\mu_1 >0$ then

$$\frac{f_1(X_1,\ldots,X_n)}{f_0(X_1,\ldots,X_n)} = \exp\{\mu_1 \sum X_i -n\mu_1^2/2 - \mu_0 \sum X_i + n\mu_0^2/2\}$$

which simplifies to

$$\exp\{\mu_1 \sum X_i -n\mu_1^2/2 \}$$

We now have to choose $\lambda$ so that

$$P_0(\exp\{\mu_1 \sum X_i -n\mu_1^2/2 \}> \lambda ) = \alpha_0$$

We can make it equal because in this case f₁(X)/f₀(X) has a continuous distribution. Rewrite the probability as

$$P_0(\sum X_i > [\log(\lambda) +n\mu_1^2/2]/\mu_1) =1-\Phi([\log(\lambda) +n\mu_1^2/2]/[n^{1/2}\mu_1])$$

If $z_\alpha$ is notation for the usual upper $\alpha$ critical point of the normal distribution then we find

$$z_{\alpha_0} = [\log(\lambda) +n\mu_1^2/2]/[n^{1/2}\mu_1]$$

which you can solve to get a formula for $\lambda$ in terms of $z_{\alpha_0}$, n and $\mu_1$.

The rejection region looks complicated: reject if a complicated statistic is larger than $\lambda$ which has a complicated formula. But in calculating $\lambda$ we re-expressed the rejection region in terms of

$$\frac{\sum X_i}{\sqrt{n}} > z_{\alpha_0}$$

The key feature is that this rejection region is the same for any $\mu_1 >0$. [WARNING: in the algebra above I used $\mu_1 >0$.] This is why the Neyman Pearson lemma is a lemma!

Definition: In the general problem of testing $\Theta_0$ against $\Theta_1$ the level of a test function $\phi$is

$$\alpha = \sup_{\theta\in\Theta_0}E_\theta(\phi(X))$$

The power function is

$$\pi(\theta) = E_\theta(\phi(X))$$

A test $\phi^*$ is a Uniformly Most Powerful level $\alpha_0$test if

1.

$\phi^*$ has level $\alpha \le \alpha_o$

2.

If $\phi$ has level $\alpha \le \alpha_0$ then for every $\theta\in \Theta_1$ we have

$$E_\theta(\phi(X)) \le E_\theta(\phi^*(X))$$

Application of the NP lemma: In the $N(\mu,1)$ model consider $\Theta_1=\{\mu>0\}$ and $\Theta_0=\{0\}$ or $\Theta_0=\{\mu \le 0\}$. The UMP level $\alpha_0$ test of $H_0: \mu\in\Theta_0$ against $H_1:\mu\in\Theta_1$ is

$$\phi(X_1,\ldots,X_n) = 1(n^{1/2}\bar{X} > z_{\alpha_0})$$

Proof: For either choice of $\Theta_0$ this test has level $\alpha_0$ because for $\mu\le 0$ we have
$$\begin{align*}P_\mu(n^{1/2}\bar{X} > z_{\alpha_0}) & = P_\mu(n^{1/2}(\bar{X}-\mu... ..._0}-n^{1/2}\mu) \\ & \le P(N(0,1) > z_{\alpha_0}) \\ & = \alpha_0 \end{align*}$$
(Notice the use of $\mu\le 0$. The central point is that the critical point is determined by the behaviour on the edge of the null hypothesis.)

Now if $\phi$ is any other level $\alpha_0$ test then we have

$$E_0(\phi(X_1,\ldots,X_n)) \le \alpha_0$$

Fix a $\mu > 0$. According to the NP lemma

$$E_\mu(\phi(X_1,\ldots,X_n)) \le E_\mu(\phi_\mu(X_1,\ldots,X_n))$$

where $\phi_\mu$ rejects if $f_\mu(X_1,\ldots,X_n)/f_0(X_1,\ldots,X_n) > \lambda$ for a suitable $\lambda$. But we just checked that this test had a rejection region of the form

$$n^{1/2}\bar{X} > z_{\alpha_0}$$

which is the rejection region of $\phi^*$. The NP lemma produces the same test for every $\mu > 0$ chosen as an alternative. So we have shown that $\phi_\mu=\phi^*$ for any $\mu > 0$.

The following will not be covered in class.

Here is a smaller example. There are 4 possible values of X and 2⁴ possible rejection regions. Here is a table of the levels for each possible rejection region R:

R	$\alpha$
{}	0
{3}, {0}	1/8
{0,3}	2/8
{1}, {2}	3/8
{0,1}, {0,2}, {1,3}, {2,3}	4/8
{0,1,3}, {0,2,3}	5/8
{1,2}	6/8
{0,1,3}, {0,2,3}	7/8
{0,1,2,3}

The best level 2/8 test has rejection region $\{0,3\}$ and $\beta = 1-[(3/4)^3+(1/4)^3] = 36/64$. If, instead, we permit randomization then we will find that the best level test rejects when X=3 and, when X=2 tosses a coin which has chance 1/3 of landing heads, then rejects if you get heads. The level of this test is 1/8+(1/3)(3/8) = 2/8 and the probability of a Type II error is $\beta =1-[(3/4)^3 +(1/3)(3)(3/4)^2(1/4)] = 28/64$.

End of material not covered in class

Proof of the Neyman Pearson lemma: Given a test $\phi$ with level strictly less than $\alpha_0$ we can define the test

$$\phi^*(x) = \frac{1-\alpha_0}{1-\alpha} \phi(x) + \frac{\alpha_0-\alpha}{1-\alpha}$$

has level $\alpha_0$ and $\beta$ smaller than that of $\phi$. Hence we may assume without loss that $\alpha=\alpha_0$ and minimize $\beta$ subject to $\alpha=\alpha_0$. However, the argument which follows doesn't actually need this.

Lagrange Multipliers

Suppose you want to minimize f(x) subject to g(x) = 0. Consider first the function

$$h_\lambda(x) = f(x) + \lambda g(x)$$

If $x_\lambda$ minimizes $h_\lambda$ then for any other x

$$f(x_\lambda) \le f(x) +\lambda[ g(x) - g(x_\lambda)]$$

Now suppose you can find a value of $\lambda$ such that the solution $x_\lambda$ has $g(x_\lambda) = 0$. Then for any x we have

$$f(x_\lambda) \le f(x) +\lambda g(x)$$

and for any x satisfying the constraint g(x) = 0we have

$$f(x_\lambda) \le f(x)$$

This proves that for this special value of $\lambda$ the quantity $x_\lambda$ minimizes f(x) subject to g(x)=0.

Notice that to find $x_\lambda$ you set the usual partial derivatives equal to 0; then to find the special $x_\lambda$ you add in the condition $g(x\lambda)=0$.

Proof of NP lemma

For each $\lambda> 0$ we have seen that $\phi_\lambda$ minimizes $\lambda\alpha+\beta$ where $\phi_\lambda=1(f_1(x)/f_0(x) \ge \lambda)$.

As $\lambda$ increases the level of $\phi_\lambda$ decreases from 1 when $\lambda=0$ to 0 when $\lambda = \infty$. There is thus a value $\lambda_0$ where for $\lambda < \lambda_0$ the level is less than $\alpha_0$ while for $\lambda > \lambda_0$ the level is at least $\alpha_0$. Temporarily let $\delta=P_0(f_1(X)/f_0(X) = \lambda_0)$. If $\delta = 0$ define $\phi=\phi_\lambda$. If $\delta > 0$ define

$$\phi(x) =\left\{\begin{array}{ll} 1 & \frac{f_1(x)}{f_0(x)} >... ...0 \\ 0 & \frac{f_1(x)}{f_0(x)} < \lambda_0 \end{array}\right.$$

where $P_0(f_1(X)/f_0(X) < \lambda_0)+\gamma\delta = \alpha_0$. You can check that $\gamma\in[0,1]$.

Now $\phi$ has level $\alpha_0$ and according to the theorem above minimizes $\alpha+\lambda_0\beta$. Suppose $\phi^*$ is some other test with level $\alpha^* \le \alpha_0$. Then

$$\lambda_0\alpha_\phi+ \beta_\phi \le \lambda_0\alpha_{\phi^*} + \beta_{\phi^*}$$

We can rearrange this as

$$\beta_{\phi^*} \ge \beta_\phi +(\alpha_\phi-\alpha_{\phi^*})\lambda_0$$

Since

$$\alpha_{\phi^*} \le \alpha_0 = \alpha_\phi$$

the second term is non-negative and

$$\beta_{\phi^*} \ge \beta_\phi$$

which proves the Neyman Pearson Lemma.

Definition: In the general problem of testing $\Theta_0$ against $\Theta_1$ the level of a test function $\phi$is

$$\alpha = \sup_{\theta\in\Theta_0}E_\theta(\phi(X))$$

The power function is

$$\pi(\theta) = E_\theta(\phi(X))$$

A test $\phi^*$ is a Uniformly Most Powerful level $\alpha_0$test if

1.

$\phi^*$ has level $\alpha \le \alpha_o$

2.

If $\phi$ has level $\alpha \le \alpha_0$ then for every $\theta\in \Theta_1$ we have

$$E_\theta(\phi(X)) \le E_\theta(\phi^*(X))$$

Application of the NP lemma: In the $N(\mu,1)$ model consider $\Theta_1=\{\mu>0\}$ and $\Theta_0=\{0\}$ or $\Theta_0=\{\mu \le 0\}$. The UMP level $\alpha_0$ test of $H_0: \mu\in\Theta_0$ against $H_1:\mu\in\Theta_1$ is

$$\phi(X_1,\ldots,X_n) = 1(n^{1/2}\bar{X} > z_{\alpha_0})$$

Proof: For either choice of $\Theta_0$ this test has level $\alpha_0$ because for $\mu\le 0$ we have
$$\begin{align*}P_\mu(n^{1/2}\bar{X} > z_{\alpha_0}) & = P_\mu(n^{1/2}(\bar{X}-\mu... ..._0}-n^{1/2}\mu) \\ & \le P(N(0,1) > z_{\alpha_0}) \\ & = \alpha_0 \end{align*}$$
(Notice the use of $\mu\le 0$. The central point is that the critical point is determined by the behaviour on the edge of the null hypothesis.)

Now if $\phi$ is any other level $\alpha_0$ test then we have

$$E_0(\phi(X_1,\ldots,X_n)) \le \alpha_0$$

Fix a $\mu > 0$. According to the NP lemma

$$E_\mu(\phi(X_1,\ldots,X_n)) \le E_\mu(\phi_\mu(X_1,\ldots,X_n))$$

where $\phi_\mu$ rejects if $f_\mu(X_1,\ldots,X_n)/f_0(X_1,\ldots,X_n) > \lambda$ for a suitable $\lambda$. But we just checked that this test had a rejection region of the form

$$n^{1/2}\bar{X} > z_{\alpha_0}$$

which is the rejection region of $\phi^*$. The NP lemma produces the same test for every $\mu > 0$ chosen as an alternative. So we have shown that $\phi_\mu=\phi^*$ for any $\mu > 0$.

This phenomenon is somewhat general. What happened was this. For any $\mu > \mu_0$ the likelihood ratio $f_\mu/f_0$ is an increasing function of $\sum X_i$. The rejection region of the NP test is thus always a region of the form $\sum X_i > k$. The value of the constant k is determined by the requirement that the test have level $\alpha_0$and this depends on $\mu_0$ not on $\mu_1$.

Definition: The family $f_\theta;\theta\in \Theta\subset R$has monotone likelikelood ratio with respect to a statistic T(X)if for each $\theta_1>\theta_0$ the likelihood ratio $f_{\theta_1}(X) / f_{\theta_0}(X)$ is a monotone increasing function of T(X).

Theorem: For a monotone likelihood ratio family the Uniformly Most Powerful level $\alpha$ test of $\theta \le \theta_0$(or of $\theta=\theta_0$) against the alternative $\theta>\theta_0$is

$$\phi(x) =\left\{\begin{array}{ll} 1 & T(x) > t_\alpha \\ \gamma & T(X)=t_\alpha \\ 0 & T(x) < t_\alpha \end{array}\right.$$

where $P_0(T(X) > t_\alpha)+\gamma P_0(T(X) = t_\alpha) = \alpha_0$.

A typical family where this will work is a one parameter exponential family. In almost any other problem the method doesn't work and there is no uniformly most powerful test. For instance to test $\mu=\mu_0$against the two sided alternative $\mu\neq\mu_0$ there is no UMP level $\alpha$ test. If there were its power at $\mu > \mu_0$ would have to be as high as that of the one sided level $\alpha$ test and so its rejection region would have to be the same as that test, rejecting for large positive values of $\bar{X} -\mu_0$. But it also has to have power as good as the one sided test for the alternative $\mu < \mu_0$ and so would have to reject for large negative values of $\bar{X} -\mu_0$. This would make its level too large.