\input{../courseDefs}

\begin{center}
Tenth Set of Notes
\end{center}


\section{More Multiplicity: Shrinkage Estimators}
For references, see 
Stein, C., 1956. 
Inadmissibility of the usual 
estimator of the mean of a multivariate normal
distribution, {\em Proc. Third Berkeley Symp. Math. Stat. Probab., 1}, 197-206,
Univ. Calif. Press; 
James, W., and Stein, C., 1961.
Estimation with quadratic loss, {\em Proc. Fourth Berkeley Symp. Math. Stat. Probab., 1}, 
361-380, Univ. Calif. Press;
Stein, C., 1981. 
Estimation of the mean of a multivariate normal distribution,
{\em Ann. Stat., 9}, 1135-1151; 
Brandwein, A.C., and Strawderman, W.E., 1990.
Stein estimation: the spherically symmetric case, {\em Statistical Sci., 5}, 356-369;
Evans, S.N. and Stark, P.B., 1996. 
Shrinkage estimators, Skorokhod's problem, and stochastic
integration by parts, {\em Ann. Stat., 24}, 809-815.

We have looked briefly at testing multiple hypotheses and at finding confidence 
intervals for multiple parameters.
Here we consider estimating multiple parameters with squared error loss.

Let's recall a few results from earlier in the course:
A minimax estimator is Bayes for the least favorable prior and Bayes estimators are
admissible (if they are unique).

\subsection{Minimaxity of the sample mean for estimating a normal mean}

Let $X \sim N(\theta, \sigma^2)$, and let $\theta \in \Theta = \bfR$.
Then $X$ is admissible for $\theta \in \Theta$ under squared-error loss.

Before the proof, a few definitions and results.

The {\em Fisher Information} at $\theta$ is
\begin{eqnarray}
I(\theta) &=& E \left [ \frac{\partial}{\partial \theta} \log p_\theta(X) \right ]^2
\nonumber \\
&=& \int \left ( \frac{p'_\theta}{p_\theta} \right )^2 p_\theta d \mu .
\end{eqnarray}

The larger $I(\theta_0)$, the more easily  $\theta_0$ can be distinguished from
neighboring values.
The information for $n$ iid measurements is $n$ times the information for a single 
measurement (the factor $p_\theta$ occurs $n$ times).

For $N(\theta, \sigma^2)$, $I(\theta) = \sigma^{-2}$.

\begin{Theorem} The information inequality (see Lehmann, TPE, Theorem 6.4).
Let $\theta$ be a real-valued parameter; let $\Theta$ be an open interval;
suppose the distributions $\{P_\theta\}_{\theta \in \Theta}$ have common support;
assume that $p'_\theta(x) = \partial/\partial \theta p_\theta(x)$ exists and is
finite for all $x$ in the support of $P_\theta$ and all $\theta \in \Theta$; and
assume that 
\beq
E_\theta \left [ \frac{\partial}{\partial\theta} \log p_\theta(X) \right ] = 0.
\eeq
Let $\delta$ be any statistic with $E_\theta \delta^2 < \infty$ for which
the derivative w.r.t. $\theta$ of
\beq
E_\theta(\delta) = \int \delta p_\theta d \mu
\eeq
exists and can be obtained by differentiating under the integral.
Then
\beq
\Var_\theta(\delta) \ge \frac{\left [ 
\frac{\partial}{\partial\theta} E_\theta(\delta) \right ]^2}{I(\theta)} .
\eeq
\end{Theorem}

\begin{Theorem}
Let $\{ X_j \}_{j=1}^n$ be iid $N(\theta,1)$.
The sample mean
$\bar{X} = \frac{1}{n} \sum_{j=1}^n X_j$ is admissible and minimax under 
squared-error loss for $\theta$.
\end{Theorem}

\noindent
{\bf Proof.} (See Lehmann, TPE, pp265-267.) For any estimator $\delta$,
\beq
R(\theta, \delta) = E(\delta - \theta)^2
= \Var_\theta (\delta) + b^2(\theta)
\eeq
where $b(\theta)$ is the bias of $\delta$ at $\theta$ ($b(\theta) = E\delta(X) - \theta$).
By the information inequality,
\beq
R(\theta, \delta) \ge b^2(\theta) + \frac{[1 + b'(\theta)]^2}{nI(\theta)} .
\eeq
Here $I(\theta) = 1$.
For the sample mean, $R(\theta, \bar{X}) = 1/n$.
If the risk of $\delta$ is at most $1/n$ for all $\theta$, then
\beq \label{eq:bBound}
b^2(\theta) + \frac{[1 + b'(\theta)]^2}{n} \le \frac{1}{n}, \; \forall \theta .
\eeq
Now because $| b(\theta)| \le n^{-1/2}$, $b$ is bounded.
For $n=1$, we have 
\beq
1 + 2b'(\theta) + b'^2(\theta) \le 1,
\eeq
so $b'(\theta) \le 0$: $b$ is nonincreasing.
The boundedness of $b$ and the fact that $b' \le 0$
imply that $b' \rightarrow 0$ as $\theta \rightarrow \pm \infty$.
This, together with the bound \ref{eq:bBound} implies that
$b(\theta) \rightarrow 0$ as $\theta \rightarrow \pm \infty$; hence $b$ is constant,
and equal to zero.
Thus for any $\delta$,
\beq
R(\theta, \delta) \ge \frac{1}{n} \; \forall \theta .
\eeq
Thus $R(\theta, \delta) = 1/n$, so $\bar{X}$ is admissible, and minimax.

\subsection{Dimensions 3 and above: Stein's theorem}
Charles Stein (1956) showed the following surprising result.
Let $X \sim N(\theta, I)$, $\theta \in \Theta = \bfR^p$, $p \ge 3$.
Not only is $X$ not minimax for $\theta$ under squared-error loss, $X$ is inadmissible
for $\theta$.
Heuristically, in dimensions three and above, the ``noise'' that the normal adds to
the mean is more likely than not to result in a ``data vector'' further from the origin 
(larger in norm) than the parameter vector is. ``Shrinking'' the data towards the
origin by a small amount tends to give a more accurate estimator than simply reporting
the data.

A proof can be based on Stein's integration-by-parts Lemma, which we saw earlier
(fifth set of notes).

Let's take $X \sim N(\theta, I)$, $\theta \in \Theta = \bfR^p$, $p \ge 3$.
We seek to estimate $\theta$ under squared-error loss.
Define $\delta_a(x) \equiv (1 - a \| x \|^{-2} ) X$ (this is the James-Stein estimator;
Stein's original result was for $\delta_{a,b}(x) = (1 - a/(b + \|x\|^2))X$).
Note what $\delta_a(X)$ does: it estimates $\theta$ by a vector with the same
direction as $X$, but that is shorter than $X$ by a fraction $a$
(the estimator ``shrinks'' $X$ towards the origin). The
estimate can have sign opposite that of $X$ when $\|X\|$ is small; 
``positive part'' estimators,
of the form  $(1 - a \| x \|^{-2} )_+ X$, prevent
the coordinates from crossing zero. Recall that in the case of estimating a
bounded normal mean, we found that the optimal affine estimator ``shrinks''
towards the center of the interval $\theta$ is known to lie in.
The surprise here is that even without any knowledge of where in $\bfR^p$ $\theta$
might lie, it helps to shrink towards the origin.

\begin{Theorem} (James and Stein, 1961. See Brandwein and Strawderman, Thm. 4.1.)
For $a \in (0, 2(p-2))$, $\delta_a(X)$ dominates $X$ in squared-error loss.
The estimator $\delta_{p-2}$ has uniformly smallest risk in the class.
\end{Theorem}

\noindent
{\bf Proof.}
Let $Z \sim N(0,1)$.
The risk of $X$ is
\beq
E \| X - \theta \|^2 
 = p E Z^2 = p.
\eeq
Note that for $x \in \bfR^p$,
\beq
\frac{d}{dx_j} \| x \|^2 = 2x_j,
\eeq
so
\begin{eqnarray}
\frac{d}{dx_j} \| x \|^{-2} &=&
 \frac{d}{dx_j} (\| x \|^2 )^{-1} \nonumber \\
&=& - (\| x \|^2 )^{-2} 2x_j
\nonumber \\
&=& -2x_j \| x \|^{-4}
\end{eqnarray} 
Thus
\begin{eqnarray}
R(\theta, \delta_a) &=&
E \left \| (1 - a \|X \|^{-2})X - \theta \right \|^2
\nonumber \\
&=& E \| X - \theta \|^2 + a^2 E \|X\|^{-2} - 2aE \left ( X'(X-\theta)\|X\|^{-2} \right )
\nonumber \\
&=& p + a^2E \|X\|^{-2} - 2a\sum_{j=1}^p E \left ( X_j(X_j - \theta_j)\|X\|^{-2} \right )
\nonumber \\
&=& p + a^2 E \|X\|^{-2} - 2a\sum_{j=1}^p E \left ( \frac{d}{dx_j}(X_j \|X\|^{-2}) \right )
\mbox{ (by Stein's lemma, componentwise) }
\nonumber \\
&=& p + a^2 E \|X\|^{-2} - 2a\sum_{j=1}^p E \left ( \|X\|^{-2} - 2 X_j^2 \|X\|^{-4} \right )
\nonumber \\
&=& p + a^2 E \|X\|^{-2} - 2a p E\|X\|^{-2} - 4a E \left ( \|X\|^2 \|X\|^{-4} \right )
\nonumber \\
&=& p + a^2 E \|X\|^{-2} - 2a p E\|X\|^{-2} - 4a E \|X\|^{-2}
\nonumber \\
&=& p + (a^2 - 2a(p-2))E \|X\|^{-2}.
\end{eqnarray}
Note that $E \|X\|^{-2} > 0$.
For $a \in (0, 2(p-2))$, $a^2 - 2a(p-2)<0$, so for $a \in  (0, p-2]$, $\delta_a$ dominates
$X$ for all $\theta \in \bfR^p$. 
\beq
\arg \min_{a \in (0, 2(p-2))}\{a^2 - 2a(p-2)\} = p-2,
\eeq
so $\delta_{p-2}$ has the uniformly smallest risk in this class.
It can be shown further that $R(0, \delta_{p-2}) = 2$, $\forall p \ge 3$.

Note that there is nothing special about shrinking towards the origin:
shrinking towards any other $\theta' \in \bfR^p$ by $a \in (0, p-2]$
also dominates $X$ (take 
$\delta(x) = \theta' + (1 - (p-2)\| x - \theta'\|^{-2}) (x - \theta')$).

Now suppose we observe instead $\{ X_j \}_{j=1}^n$ iid $N(\theta, \sigma^2)$.
We still seek to estimate $\theta \in \bfR$ under squared-error loss.
The sample mean $\bar{X}$ is sufficient for $\theta$, so this is risk-equivalent to
estimating $\theta$ from $\bar{X} \sim N(\theta, \sigma^2/n)$; $\bar{X}$ is thus
admissible and minimax.

\subsection{Other Distributions}
Stein's result that the sample mean is inadmissible in higher dimensions has been
generalized in a variety of ways. 
The most general result so far is due to 
S.N. Evans and P.B. Stark  
(1996, Shrinkage estimators,
 Skorokhod's problem, and stochastic integration by parts,
{\em Ann. Stat., 24}, 809-815.),
who showed that in dimensions three and higher, the sample mean is inadmissible for the mean 
for all distributions that can be characterized as a stopping time of Brownian motion.
That condition is equivalent to saying that we observe $X = \theta + Z$ with $Z$ satisfying
$EZ = 0$, $EZ^2 < \infty$, and 
\beq \label{eq:evansCondition}
E\|Z+\theta \|^{2-d} \le \|\theta \|^{2-d}.
\eeq
This includes all symmetric distributions, as well as many others (including some supported on
fractal sets, for example).
The condition \ref{eq:evansCondition}, intuitively speaking, says that
the ``noise'' $Z$ on average makes $\|X\|^2 > \|\theta\|^2$ (not exactly---the exponent is
not 2), which is when one would
expect shrinking towards the origin to help.
The proof is essentially a generalization of Stein's ``integration by parts'' lemma to
stochastic integrals of Brownian motion.

Th amount of shrinkage that is optimal for other distributions depends on the distribution.
The results in Evans and Stark are for the estimator
\beq
\delta(x) = (1 - a(1 + \|x\|^2)^{-1})x,
\eeq
which is like Stein's original estimator with $b=1$,
but similar techniques work (with additional assumptions) for $\delta_a$.
They establish the existence of $a \in \bfR+$ sufficiently small that the
shrinkage estimator dominates $X$; in the special cases that the support of $Z$ is
within a ball, or does not intersect a ball, they construct values of $a$ that work.

\noindent
{\bf Exercise.}
In the normal linear regression model, we observe $X \sim N(A\cdot \theta, \sigma^2 I)$,
where $A$ is a known $n \times p$ matrix with $A'\cdot A$ nonsingular, 
$\theta \in \bfR^p$ is unknown, $X \in \bfR^n$, and $\sigma^2 > 0$.
The least-squares estimate of $\theta$ is
\beq
\hat{\theta}_{\mbox{LS}} = (A'\cdot A)^{-1} A'X .
\eeq
Consider estimating $\theta$ subject to squared-error ``prediction'' loss 
\beq
L(\hat{\theta}, \theta) = \| A\cdot \hat{\theta} - A \cdot \theta \|^2.
\eeq
Show that $\hat{\theta}_{\mbox{LS}}$ is inadmissible for $\theta$.

\end{document}


