Postscript version of these notes

STAT 801 Lecture 17

Reading for Today's Lecture:

Goals of Today's Lecture:

• Discuss techniques for proving UMVU property.

• Introduce sufficiency

• Introduce the Rao Blackwell theorem.

Today's notes

The Binomial(n,p) example

If $Y_1,\ldots,Y_n$ are iid Bernoulli(p) then $X=\sum Y_i$ is Binomial(n,p). We used various algebraic tactics to arrive at the following conclusions:

• The log likelihood is a function of X only and not the actual values of $Y_1,\ldots,Y_n$.

• There is only one function of X, namely, $\hat p = X/n$ which is an unbiased estimate of p.

• If $T(Y_1,\ldots, Y_n)$ is an unbiased estimate of p then the average value of $T(y_1,\ldots,y_n)$ over those $y_1,\ldots,y_n$ for which $\sum y_i = x$ is x/n.

• The conditional distribution of T given $\sum Y_i = x$ does not depend on p.

• If $T(Y_1,\ldots, Y_n)$ is an unbiased estimate of p then
  
  $${\rm Var}(T) = {\rm Var}(\hat p) + E[(T-\hat p)^2]$$
  
  

• $\hat p$ is the UMVUE of p.

This long, algebraically involved, method of proving that $\hat p = X/n$ is the UMVUE of p is one special case of a general tactic.

Sufficiency

In the binomial situation the conditional distribution of the data $Y_1,\ldots,Y_n$ given X is the same for all values of $\theta$; we say this conditional distribution is free of $\theta$.

Definition: A statistic T(X) is sufficient for the model $\{ P_\theta;\theta \in \Theta\}$ if the conditional distribution of the data X given T=t is free of $\theta$.

Intuition: Why do the data tell us about $\theta$? Because different values of $\theta$ give different distributions to X. If two different values of $\theta$ correspond to the same joint density or cdf for X then we cannot, even in principle, distinguish these two values of $\theta$ by examining X. We extend this notion to the following. If two values of $\theta$ give the same conditional distribution of X given T then observing T in addition to X does not improve our ability to distinguish the two values.

Mathematically Precise version of this intuition: If T(X) is a sufficient statistic then we can do the following. If S(X) is any estimate or confidence interval or whatever for a given problem but we only know the value of T then:

• Generate an observation X^* (via some sort of Monte Carlo program) from the conditional distribution of X given T.

• Use S(X^*) instead of S(X). Then S(X^*) has the same performance characteristics as S(X) because the distribution of X^* is the same as that of X.

You can carry out the first step only if the statistic T is sufficient; otherwise you need to know the true value of $\theta$ to generate X^*.

Example 1: If $Y_1,\ldots,Y_n$ are iid Bernoulli(p) then given $\sum Y_i = y$ the indexes of the y successes have the same chance of being any one of the $\dbinom{n}{y}$ possible subsets of $\{1,\ldots,n\}$. This chance does not depend on p so $T(Y_1,\ldots,Y_n) = \sum Y_i$ is a sufficient statistic.

example 2: If $X_1,\ldots,X_n$ are iid $N(\mu,1)$ then the joint distribution of $X_1,\ldots,X_n,\overline{X}$ is multivariate normal with mean vector whose entries are all $\mu$ and variance covariance matrix which can be partitioned as

$$\left[\begin{array}{cc} I_{n \times n} & {\bf 1}_n /n \\ {\bf 1}_n^t /n & 1/n \end{array}\right]$$

where ${\bf 1}_n$ is a column vector of n 1s and $I_{n \times n}$ is an $n \times n$ identity matrix.

You can now compute the conditional means and variances of X_i given $\overline{X}$ and use the fact that the conditional law is multivariate normal to prove that the conditional distribution of the data given $\overline{X} = x$ is multivariate normal with mean vector all of whose entries are x and variance-covariance matrix given by $I_{n\times n} - {\bf 1}_n{\bf 1}_n^t /n$. Since this does not depend on $\mu$ we find that $\overline{X}$ is sufficient.

WARNING: Whether or not a statistic is sufficient depends on the density function and on $\Theta$.

Rao Blackwell Theorem

Theorem: Suppose that S(X) is a sufficient statistic for some model $\{P_\theta,\theta\in\Theta\}$. If T is an estimate of some parameter $\phi(\theta)$ then:

1.

E(T|S) is a statistic.

2.

E(T|S) has the same bias as T; if T is unbiased so is E(T|S).

3.

${\rm Var}_\theta(E(T\vert S)) \le {\rm Var}_\theta(T)$ and the inequality is strict unless T is a function of S.

4.

The MSE of E(T|S) is no more than that of T.

Proof: It will be useful to review conditional distributions a bit more carefully at this point. The abstract definition of conditional expectation is this:

Definition: E(Y|X) is any function of X such that

$$E\left[R(X)E(Y\vert X)\right] = E\left[R(X) Y\right]$$

for any function R(X).

Definition: E(Y|X=x) is a function g(x) such that

g(X) = E(Y|X)

Fact: If X,Y has joint density f_X,Y(x,y) and conditional density f(y|x) then

$$g(x) = \int y f(y\vert x) dy$$

satisfies these definitions.

Proof of Fact:
$$\begin{align*}E(R(X)g(X)) & = \int R(x) g(x)f_X(x) dx \\ & = \int\int R(x) y f(... ...t x) dy dx \\ &= \int\int R(x)y f_{X,Y}(x,y) dy dx \\ &= E(R(X)Y) \end{align*}$$

You should simply think of E(Y|X) as being what you get when you average Y holding X fixed. It behaves like an ordinary expected value but where functions of X only are like constants.

Proof of the Rao Blackwell Theorem

Step 1: The definition of sufficiency is that the conditional distribution of X given S does not depend on $\theta$. This means that E(T(X)|S) does not depend on $\theta$.

Step 2: This step hinges on the following identity (called Adam's law by Jerzy Neyman - he used to say it comes before all the others)

E[E(Y|X)] =E(Y)

which is just the definition of E(Y|X) with $R(X) \equiv 1$.

From this we deduce that

$$E_\theta[E(T\vert S)] = E_\theta(T)$$

so that E(T|S) and T have the same bias. If T is unbiased then

$$E_\theta[E(T\vert S)] = E_\theta(T) = \phi(\theta)$$

so that E(T|S) is unbiased for $\phi$.

Step 3: This relies on the following very useful decomposition. (In regression courses we say that the total sum of squares is the sum of the regression sum of squares plus the residual sum of squares.)

$${\rm Var(Y)} = {\rm Var}(E(Y\vert X)) + E[{\rm Var}(Y\vert X)]$$

The conditional variance means

$${\rm Var}(Y\vert X) = E[ (Y-E(Y\vert X))^2\vert X]$$

This identity is just a matter of squaring out the right hand side

$${\rm Var}(E(Y\vert X)) =E[(E(Y\vert X)-E[E(Y\vert X)])^2] = E[(E(Y\vert X)-E(Y))^2]$$

and

$$E[{\rm Var}(Y\vert X)] = E[(Y-E(Y\vert X)^2]$$

Adding these together gives

E[Y² -2YE[Y|X]+2(E[Y|X])² -2E(Y)E[Y|X] + E²(Y)]

The middle term actually simplifies. First, remember that E(Y|X) is a function of X so can be treated as a constant when holding X fixed. This means

E[Y|X]E[Y|X] = E[YE(Y|X)|X]

and taking expectations gives

E[(E[Y|X])²] = E[E[YE(Y|X)|X]] =E[YE(Y|X)]

This makes the middle term above cancel with the second term. Moreover the fourth term simplifies

E[E(Y)E[Y|X]] = E(Y) E[E[Y|X]] =E²(Y)

so that

$${\rm Var}(E(Y\vert X)) + E[{\rm Var}(Y\vert X)] = E[Y^2] - E^2(Y)$$

We apply this to the Rao Blackwell theorem to get

$${\rm Var}_\theta(T) = {\rm Var}_\theta(E(T\vert S)) + E[(T-E(T\vert S))^2]$$

The second term is non negative so that the variance of E(T|S) must be no more than that of T and will be strictly less unless T=E(T|S). This would mean that T is already a function of S. Adding the squares of he biases of T (or of E(T|S) which is the same) gives the inequality for mean squared error.

Examples:

In the binomial problem Y₁(1-Y₂) is an unbiased estimate of p(1-p). We improve this by computing

E(Y₁(1-Y₂)|X)

We do this in two steps. First compute

E(Y₁(1-Y₂)|X=x)

Notice that the random variable Y₁(1-Y₂) is either 1 or 0 so its expected value is just the probabily it is equal to 1:
$$\begin{align*}E(Y_1(1-Y_2)\vert X=x) &= P(Y_1(1-Y_2) =1 \vert X=x) \\ & = P(Y_1... ...rac{\dbinom{n-2}{x-1}}{\dbinom{n}{x}} \\ & = \frac{x(n-x)}{n(n-1)} \end{align*}$$
This is simply $n\hat p(1-\hat p)/(n-1)$ (which can be bigger than 1/4 which is the maximum value of p(1-p)).

Example: If $X_1,\ldots,X_n$ are iid $N(\mu,1)$ then $\bar{X}$ is sufficient and X₁ is an unbiased estimate of $\mu$. Now
$$\begin{align*}E(X_1\vert\bar{X})& = E[X_1-\bar{X}+\bar{X}\vert\bar{X}] \\ & = E[X_1-\bar{X}\vert\bar{X}] + \bar{X} \\ & = \bar{X} \end{align*}$$
which is the UMVUE.

Finding Sufficient statistics

In the binomial example the log likelihood (at least the part depending on the parameters) was seen above to be a function of X (and not of the original data $Y_1,\ldots,Y_n$ as well). In the normal example the log likelihood is, ignoring terms which don't contain $\mu$,

$$\ell(\mu) = \mu \sum X_i - n\mu^2/2 = n\mu\bar{X} -n\mu^2/2 \, .$$

These are examples of the Factorization Criterion:

Theorem: If the model for data X has density $f(x,\theta)$ then the statistic S(X) is sufficient if and only if the density can be factored as

$$f(x,\theta) = g(s(x),\theta)h(x)$$

The theorem is proved by finding a statistic T(x) such that X is a one to one function of the pair S,T and applying the change of variables to the joint density of S and T. If the density factors then you get

$$f_{S,T}(s,t) =g(s,\theta) h(x(s,t))$$

from which we see that the conditional density of T given S=s does not depend on $\theta$. Thus the conditional distribution of (S,T) given S does not depend on $\theta$ and finally the conditional distribution of X given S does not depend on $\theta$. Conversely if S is sufficient then the conditional density of T given S has no $\theta$ in it and the joint density of S,T is

$$f_S(s,\theta) f_{T\vert S} (t\vert s)$$

Apply the change of variables formula to get the density of X to be

$$f_S(s(x),\theta) f_{T\vert S} (t(x)\vert s(x)) J(x)$$

where J is the jacobian. This factors.

Example: If $X_1,\ldots,X_n$ are iid $N(\mu,\sigma^2)$ then the joint density is

$$(2\pi)^{-n/2} \sigma^{-n} \exp\{-\sum X_i^2/(2\sigma^2) +\mu\sum X_i/\sigma^2 -n\mu^2/(2\sigma^2)\}$$

which is evidently a function of

$$\sum X_i^2, \sum X_i$$

This pair is a sufficient statistic. You can write this pair as a bijective function of $\bar{X}, \sum (X_i-\bar{X})^2$ so that this pair is also sufficient.