\input{../courseDefs}

\begin{center}
Seventh Set of Notes
\end{center}


\section{Optimization}
For references, see D.G. Luenberger (1969) {\em Optimization by Vector Space Methods},
John Wiley and Sons, Inc., NY.; E.J. Anderson and P. Nash, 1987, 
{\em Linear Programming in Infinite-Dimensional
Spaces}, Wiley, NY; 
M.S. Bazaraa and C.M. Shetty, 1979, {\em Nonlinear Programming: Theory and Algorithms},
Wiley, NY;
Shor, 1985, {\em Minimization Methods for Non-Differentiable Functions},
Springer-Verlag, NY.

Many questions in statistical theory can be reduced to optimization problems,
sometimes in infinite-dimensional spaces (spaces of functions or measures, for
example).
For example, we have just seen in Donoho's work how the difficulty of 
certain minimax estimation
problems can be related to the modulus of continuity, whose computation is
an optimization problem over a convex subset of $\ell_2$.

Some unconstrained optimization problems with differentiable, convex
objective functions, are fairly straightforward to solve, {\em e.g.},
using the calculus of variations.
Solving differentiable, unconstrained, convex problems numerically is not
typically difficult (descent algorithms can be used), 
unless evaluating the objective functional or its derivative
is extremely computationally intensive.

Constrained problems, nondifferentiable problems, and nonconvex problems are typically
much harder.
Even for convex functions whose derivative exists almost everywhere, the steepest
descent algorithm can converge to a nonstationary point (see Shor, Ch. 2, \S2.1 for
an example).
Nondifferentiable objective functionals arise fairly frequently in statistics.
For example, the absolute value function is not differentiable at zero, so the problem of
finding the median as the solution of the optimization problem of minimizing the sum of
the absolute deviations (or of finding a multivariate
generalization of the median) is a convex, nondifferentiable optimization problem.
Similarly, the objective functionals for minimum $\ell_1$ and minimum 
$\ell_\infty$ regression are
nondifferentiable.

Some quite interesting statistical problems have convex objective functionals, 
but nonconvex constraints, such as signal recovery problems subject to constraints 
on the measure of the support of the signal (sparsity constraints).

Linear equality constraints (such as $\langle x, g \rangle = 0$) are fairly 
straightforward
to deal with; one can project the problem onto the subspace where 
the constraint is
satisfied. 
Linear inequalities are somewhat harder.
Two of the most useful tools for solving constrained 
infinite-dimensional optimization problems are Fenchel and Lagrange duality.

A {\em cone} in a real linear vector space $\calX$ is a set $P \subset \calX$ such that 
if $x \in P$, then
$\alpha x \in P$ for all $\alpha > 0$.
One can establish a partial order on a vector space $\calX$
with a convex cone $P$ (then called the
{\em positive cone}) by defining $x \ge y$ if $x-y \in P$.  
If $\calX$ is a topological space
(such as a normed space with topology inherited from the norm),
and if the interior of the positive cone $P$ is nonempty in the topology of $\calX$, 
we write
$x > y$ if $x-y \in P^\circ$, the interior of $P$.
(Note that this differs from the definition of $<$ we used for a totally ordered set, 
where
$<$ meant $\le$ but not $=$; here, $<$ 
derives from topological properties of the
positive cone that defines the order.)

If $\calX$ is a linear vector space, $\calX^*$ denotes the linear space of all linear 
functionals defined on $\calX$, and is called the algebraic dual space of $\calX$.  
If $\calX$ is a normed space, by default $\calX^*$ 
is the space of {\em bounded} linear
functionals on $\calX$, (called the normed dual space of $\calX$)
unless otherwise specified.
Denote by $\langle x^*, x\rangle$ the action of the linear functional $x^* \in \calX^*$ on
the element $x \in \calX$.
The natural mapping from a space $\calX$ to its second dual $\calX^{**}$ (the dual of its
dual) is $\langle x^{**}, x^*\rangle = \langle x^*, x\rangle$. Clearly each $x \in \calX$
gives a linear functional $x^{**}$ on $\calX^*$ this way.  If every (bounded) linear functional
on $\calX^*$ can be obtained this way ({\em i.e.}, if $\calX^{**} = \calX$), $\calX$ is
said to be {\em reflexive}.

The epigraph of a functional $f$ on a set $C \subset \calX$ is 
\beq
[f, C]_+ \equiv \{(t, x) \in \bfR \times \calX: x \in C,\; f(x) \le t \} .
\eeq
If $C$ is convex, $[f, C]_+$ is a convex set in
$\bfR \times \calX$ iff $f$ is a convex functional.
Similarly, let
\beq
[g, D]_- \equiv \{(t, x) \in \bfR \times \calX: x \in D,\; f(x) \ge r \}.
\eeq


{\bf Definition.}
A {\em linear variety} in a vector space $\calX$ is the translation of a subspace
$S$ in $\calX$. That is, if $S$ is a subspace, then for every $x \in \calX$,
$S+x$ is a linear variety.
A {\em hyperplane} $H$ in a vector space $\calX$ is a maximal proper linear variety;
that is, a linear variety such that if $Y$ is another linear variety in $\calX$ and
$H \subset Y$, then either $Y=H$ or $Y = X$.
A hyperplane can be characterized by a linear functional: every hyperplane can be written
as $\{ x \in \calX : \ip{x^*}{x} + b = 0 \}$ for some $x^* \in \calX^*$ and some
$b \in \bfR$.
Every set of the form $\{ x \in \calX : \ip{x^*}{x} + b = 0 \}$ is a hyperplane.
If $x^*$ is a nonzero linear functional on a normed vector space $\calX$, the hyperplanes
$\{ x : \ip{x^*}{x} + b = 0 \}$ are closed for every $b \in \bfR$ iff 
$x^*$ is continuous.

\begin{Theorem}
Separating hyperplane theorem.
Suppose that $C$, $D$ are convex subsets of a normed vector space $\calX$,
that $C$ contains interior points, and that $D$ contains no interior point of $C$.
Then there is a
a hyperplane {\em separating} the sets:
there is an element $x^* \in \calX^*$ s.t.
\beq
\sup_{x \in C} \ip{x^*}{x} \le \inf_{x \in D} \ip{x^*}{x}.
\eeq
Thus there is a number $b \in \bfR$ s.t.
\begin{eqnarray}
\ip{x^*}{x} + b \le 0 & \;\;&\forall x \in C \nonumber \\
\ip{x^*}{x} + b \ge 0 & \;\;&\forall x \in D \nonumber \\
\end{eqnarray}
\end{Theorem}

\subsection{Algebraic Duality}

We always take $\inf_{x \in \emptyset} f(x) = \infty$ and $\sup_{x \in \emptyset} f(x)
= - \infty$.

For any functional $f$ on a real linear vector space $\calX$ with dual $\calX^*$
and any sets $C, D \subset \calX$
consider the {\em value of the primal problem}
\beq
v(\calP) \equiv \inf_{x \in C \cap D} f(x) .
\eeq
Clearly, for any $x^* \in \calX^*$,
\begin{eqnarray}
v(\calP) &=& \inf_{x \in C \cap D} \{ f(x) - \langle x^*, x\rangle + \langle x^*, x\rangle \}
\nonumber
\\
&\ge& \inf_{x \in C \cap D} \ip{ x^*}{x} + 
\inf_{x \in C \cap D} \{ f(x) - \ip{ x^*}{x} \} 
\nonumber \\
&\ge& \inf_{x \in D} \ip{ x^*}{x} + \inf_{x \in C} \{f(x) - \ip{ x^*}{x} \}
\end{eqnarray}
This is true for all $x^*$, so
\begin{eqnarray}
\inf_{x \in C \cap D} f(x) 
&\ge& \sup_{x^* \in \calX^*}
\left \{ \inf_{x \in D} \ip{ x^*}{ x} + \inf_{x \in C} \{f(x) - \ip{ x^*}{ x} \}
\right \} \nonumber \\
&=& \sup_{x^* \in \calX^*} \{ D^*[x^*] + C^*[x^*] \},
\end{eqnarray}
where
\beq
C^*[x^*] \equiv \inf_{x \in C} \{ f(x) - \ip{x^*}{ x} \}
\eeq
and
\beq
D^*[x^*] \equiv \inf_{x \in D} \{ \ip{ x^*}{ x} \}.
\eeq
The only functionals $x^*$ it is worth considering are those for which
$C^*$ and $D^*$ are greater than $-\infty$.
Let 
\beq
\calC^* \equiv \left \{ x^* \in \calX^* : \inf_{x \in C} \{ f(x) - \ip{ x^*}{x} \}
> -\infty \right \},
\eeq
and
\beq
\calD^* \equiv \left \{ x^* \in \calX^* : \inf_{x \in D} \langle x^*, x\rangle 
> -\infty \right \}.
\eeq
Then wlog we can restrict attention to
\beq
\sup_{x^* \in \calC^* \cap \calD^*} \{ D^*[x^*] + C^*[x^*] \},
\eeq
with the supremum defined to be $-\infty$ if $\calC^* \cap \calD^* = \emptyset$.

In general, the sup over $\calX^*$ need not equal the inf over $\calX$.  When it does
not, there is said to be a ``duality gap.''

\noindent
{\bf Example.}
Suppose $\calX$ is a linear vector space of functions $x = x(t)$ on some fixed domain;
$\{ x_j^* \}_{j=1}^n \subset \calX^*$, $\{ x_j^* \}$ linearly independent;
$d: \calX \rightarrow \bfRn,$ $x \rightarrow (\ip{x_j^*}{x})_{j = 1}^n$;
$\Xi \subset \bfR^n$ is a bounded
subset of $\bfRn$, $D = \{ x \in \calX : d(x) \in \Xi \}$.
Then one can show that $\calD^* = \linSpan\{x_j^* \}_{j=1}^n$ ($D$ is a hypercylinder
constrained only in the directions ``aligned'' with an $x^*_j$; in other directions
$D$ is unconstrained, so a linear functional with a component in any direction
not in $\linSpan\{x_j^* \}_{j=1}^n$ is unbounded below on $D$).
Thus for any $f$ and $C$, the infinite-dimensional problem
\beq
\inf_{x \in C \cap D} f(x)
\eeq
is bounded from below by a finite-dimensional problem on $\linSpan\{x_j^* \}_{j=1}^n$.
For particular sets $\Xi$ and $C$, and particular functionals $f$, this can lead
to an easy solution for $v(\calP)$.  
Continuing the example, suppose that $C = \calX$, that $f(x) = \ip{x_0^*}{x}$,
and that
\beq
\Xi \equiv \{ \gamma \in \bfR^n : \| \gamma - \delta \| \le \epsilon \},
\eeq
for some fixed $\delta \in \bfR^n$ (an $\epsilon$-ball in $\bfR^n$ centered at $\delta$).
Then $\calC^* = x_0^*$, and we already saw that $\calD^* = \linSpan\{ x_j^* \}_{j=1}^n$,
so $\calC^* \cap \calD^* = \emptyset$ unless $x_0^* = \sum_{j=1}^n \alpha_j x_j^*$
for some sequence of constants $\alpha = (\alpha_j)_{j=1}^n$.
Given a linearly independent set 
$\{ x_j^* \}_{j=1}^n \subset \calX^*$, one can construct a linearly independent
set $\{ x_j \}_{j=1}^n \subset \calX$ such that
\beq
\ip{x_j^*}{x_k} = 1_{j=k} .
\eeq
If indeed $x_0^* = \sum_{j=1}^n \alpha_j x_j^*$, then
for $x^* \in \calC^* \cap \calD^*$
\beq
\inf_{x \in C} \ip{x_0^*}{x} - \ip{x^*}{x} = 0.
\eeq
For any $x \in D$, $d(x) = \delta + \nu$ with $\| \nu \| \le \epsilon$, so for
$x \in D$,
\begin{eqnarray}
\alpha \cdot d(x) &=& \alpha \cdot \delta + \alpha \cdot \nu \nonumber \\
&\ge& \alpha \cdot \delta - | \alpha \cdot \nu | \nonumber \\
&\ge& \alpha \cdot \delta - \|\alpha \| \|\nu \| \nonumber \\
&\ge& \alpha \cdot \delta - \epsilon \| \alpha \|.
\end{eqnarray}
This bound is in fact attained by setting
\beq
\beta = \delta - \epsilon \frac{\alpha}{\|\alpha\|}
\eeq
and taking $x = \sum_j \beta_j x_j$.
Thus
\beq
\inf_{x \in D} \ip{x^*}{x} = \alpha \cdot \delta - \epsilon\| \alpha \|.
\eeq
This gives us 
\beq
\inf_{x \in C \cap D} \ip{x_0^*}{x} = \left \{
\begin{array}{cc}
\alpha \cdot \delta - \epsilon\| \alpha \|, & x_0^* = \sum_{j=1}^n \alpha_j x_j^*
\\
- \infty, & \mbox{ otherwise }
\end{array}
\right .
\eeq
There is no duality gap in this problem.

See Stark (1992) {\em J. Geophys. Res., 97,} 14,055--14,082,
for more examples.


\subsection{Fenchel Duality}
Fenchel duality establishes conditions under which there is no duality gap in
the algebraic duality relation given in the previous section.
The conditions rely on topological considerations, and the version presented here
assumes that the spaces are normed, the sets are convex, {\em etc.} Note that
the definitions are changed a bit from the last section!

Let $\calX$ be a normed linear vector space with normed
dual space $\calX^*$, and let
$C$ and $D$ be convex subsets of $\calX$. Let $f : \calX \rightarrow \bfR$ be convex,
and let $g : \calX \rightarrow \bfR$ be concave.
The {\em conjugate set} $\calC^*$ of $C$ is
\beq
\calC^* \equiv \{ x^* \in \calX^* : \sup_{x \in C} 
[\ip{ x^*}{ x } - f(x) ] < \infty \}.
\eeq
The conjugate functional $f^*$ of $f$ is defined on $C^*$ by
\beq
f^*(x^*) = \sup_{x \in C} [\ip{ x^*}{x} - f(x) ] .
\eeq
The {\em conjugate set} $\calD^*$ of $D$ is
\beq
\calD^* \equiv \{ x^* \in \calX^* : \inf_{x \in D} [\ip{ x^*}{x} - g(x) ] 
> - \infty \}.
\eeq
The conjugate functional $g^*$ of $g$ is defined on $D^*$ by
\beq
g^*(x^*) = \inf_{x \in D} [\ip{ x^*}{x} - g(x) ] .
\eeq
One may show that $\calC^*$ and $\calD^*$ are convex subsets of $X^*$, that 
$f^*$ is a convex
functional, and that $g^*$ is a concave functional.



\begin{Theorem} Fenchel Duality (see Luenberger, 1969, \S7.12, Theorem 1).
Let $\calX$ be a normed space, $C$ and $D$ be convex subsets of $\calX$, $f$ a convex
functional on $\calX$, $g$ a concave functional on $\calX$. Suppose $C \cap D$ contains
points in the relative interior of $C$ and $D$, and that either 
$[f, C]_+$ or $[g, D]_-$ has nonempty interior. If
\beq
v(\calP) = \inf_{x \in C \cap D} \{f(x) - g(x) \} > - \infty ,
\eeq
then
\beq
v(\calP) = \max_{x^* \in C^* \cap D^*} \{ g^*(x^*) - f^*(x^*) \} .
\eeq
The max on the right is attained by some $x_0^* \in C^* \cap D^*$.
If the infimum is attained by some $x_0 \in C \cap D$, then
\beq
\max_{x \in C} [ \ip{x_0^*}{ x} - f(x)] = \ip{x_0^*}{ x_0} - f(x_0)
\eeq
and
\beq
\min_{x \in D} [ \ip{x_0^*}{ x} - g(x)] = 
\ip{x_0^*}{x_0} - g(x_0).
\eeq
\end{Theorem}

\noindent
{\bf Remark.} I cannot see where in the proof of the Fenchel Duality theorem
Luenberger uses the hypothesis that $C \cap D$ contains
points in the relative interior of $C$ and $D$. Furthermore, that hypothesis is
not necessarily satisfied in the example he gives to the min-max theorem of games---see
below
(a compact convex subset of a reflexive space does not necessarily have nonempty
interior).
Thanks to Jason Schweinsberg and Von Bing Yap for pointing out these problems!
Can anyone see how the hypothesis enters the proof?



\noindent
{\bf Example.}
The min-max theorem in game theory.
Let $\calX$ be a normed linear vector space and $\calX^*$ its normed dual.
Assume $\calX$ is reflexive.
Player $A$ selects a strategy $x$ from $\calA \subset \calX$ and player $B$ selects a
strategy from $\calB^* \subset \calX^*$, without knowledge of player $A$'s choice.
Player $A$ pays player $B$ $\ip{x^*}{x}$. 
Player $A$ thus wants to minimize
$\ip{x^*}{x}$, while player $B$ wants to maximize it.
Suppose both players seek a minimax strategy: one in which they lose least (gain most)
in the worst case.  Thus player $A$ seeks
\beq
v_- = \inf_{x \in \calA} \sup_{x^* \in \calB^*} \ip{x^*}{x},
\eeq
while
player $B$ seeks
\beq
v_+ = \sup_{x^* \in \calB^*} \inf_{x \in \calA}  \ip{x^*}{x}.
\eeq

\begin{Theorem} (Min-max theorem; see Luenberger, \S 7.13, Theorem 1.)
Suppose $\calA$ and $\calB^*$ are compact and convex.
Then
\beq
\inf_{x \in \calA} \sup_{x^* \in \calB^*} \ip{x^*}{x}
= \sup_{x^* \in \calB^*} \inf_{x \in \calA}  \ip{x^*}{x}.
\eeq
\end{Theorem}

\noindent
{\bf Proof.}
Define 
\begin{eqnarray}
f : \calX &\rightarrow& \bfR \nonumber \\
x &\mapsto &\max_{x^* \in \calB^*} \ip{x^*}{x}.
\end{eqnarray}
The max is attined because $\calB^*$ is compact (by assumption)
and $x$, viewed as an element of
$\calX^{**}$, is a continuous function on $\calB^*$.
By linearity and the convexity of $\calB^*$, for all $\alpha \in [0, 1]$,
\begin{eqnarray}
f(\alpha x + (1-\alpha) y) &=& \max_{x^* \in \calB^*} \ip{x^*}{\alpha x + (1-\alpha) y}
\nonumber \\
&=& \max_{x^* \in \calB^*} [\ip{x^*}{\alpha x} + \ip{x^*}{(1-\alpha) y}]
\nonumber \\
&= & \max_{x^* \in \calB^*} [\alpha \ip{x^*}{x} + (1-\alpha)\ip{x^*}{y}]
\nonumber \\
&\le& 
\max_{x^* \in \calB^*} \alpha \ip{x^*}{x} + \max_{x^* \in \calB^*} (1-\alpha)\ip{x^*}{y}
\nonumber \\
&=& \alpha f(x) + (1-\alpha) f(y) .
\end{eqnarray}

\noindent
{\bf Assignment.}
Show that $f$ is continuous, and that there exist points in the $C \cap D$ in the
relative interior of $C$ and $D$.

We seek $\min_{x \in \calA} f(x)$.
Apply Fenchel duality, identifying $C$ with $\calX$, $g=0$, and $D$ with $\calA$.
Because $D$ is a compact set, $\calD^* = \calX^*$,
and
\beq
g^*(x^*) = \min_{x \in \calA} \ip{x^*}{x} .
\eeq
Consider $\calC^*$.
Suppose $x_1^* \not\in \calB^*$. Then (by the separating hyperplane theorem) there
exists $x_1 \in \calX$ and $\alpha \in \bfR$ such that
\beq
\ip{x_1^*}{x_1} - \ip{x^*}{x_1} > \alpha > 0,\;\; \forall x^* \in \calB^*.
\eeq
Thus letting $x = a x_1$ with $a > 0$ sufficiently large, 
\beq
\ip{x_1^*}{x} - \max_{x^* \in \calB^*} \ip{x^*}{x} = \ip{x_1^*}{x} - f(x)
\eeq
can be made arbitrarily large,
so $x_1^* \not \in \calC^*$.
On the other hand, if $x_1^* \in \calB^*$,
\beq
\sup_{x \in \calX}
\ip{x_1^*}{x} - \max_{x^* \in \calB^*} \ip{x^*}{x} = 0,
\eeq
and is attained by $x = 0$.
Thus $\calC^* = \calB^*$.
The conditions of Fenchel duality apply, and hence
\beq
\min_{x \in \calA} f(x) = \max_{x^* \in \calB^* \cap \calX^*} g^*(x^*) =
\max_{x^* \in \calB^*} \min_{x \in \calA} \ip{x^*}{x} .
\eeq

\noindent
{\bf Example (of something): Seminorm minimization in a Hilbert Space, and Splines.}
{\em Regularized least squares} is an estimator sometimes used in nonparametric regression.
The idea is to find the regression function of minimum Sobolev norm 
(a norm that involves derivatives of the function) among those that
fit the data within an $\ell_2$ tolerance, or (equivalently) to find the model that 
fits the data best in an $\ell_2$ sense among those whose Sobolev norm does not exceed
some bound.

Recall that a real inner-product space is a real
linear vector space $\calX$ with an additional operation
$\langle \cdot , \cdot \rangle: \calX \times \calX \rightarrow \bfR$ with the properties
that for all $x, y, z \in \calX$ and all $\gamma \in \bfR$,
\begin{eqnarray}
\langle x, y \rangle &=& \langle y, x \rangle \nonumber \\
\langle x+y, z \rangle &=& \langle x, z \rangle + \langle y, z \rangle \nonumber \\
\langle \gamma x, y \rangle &=& \gamma \langle x, y \rangle \nonumber \\
\langle x, x \rangle &\ge& 0 \nonumber \\
\langle x, x \rangle = 0 \;&\Rightarrow&\; x = 0
\end{eqnarray}
The functional $\| x \| \equiv \sqrt{\langle x, x \rangle}$ is a
norm on a (real) inner-product space.
A real inner-product space that is complete in the norm topology 
({\em i.e.}, in which all Cauchy sequences have limits in the space)
is called a Hilbert space.
Hilbert spaces are their own normed dual spaces.
A normed vector space $\calX$ is {\em separable} if there is a countable 
set $\{ x_j \}_{j=1}^\infty$
such that every element $x \in \calX$ can be approximated arbitrarily well
(in the norm) by a finite linear combination of $\{ x_j \}_{j=1}^\infty$.
All separable Hilbert spaces are isomorphic to $\ell_2$.
Every real inner product space can be ``completed'' to form a Hilbert space
by considering elements to be equivalence classes of Cauchy sequences.

For any $x , y $ in an inner-product space $\calX$, 
$|\langle x, y \rangle | \le \| x \| \cdot \| y \|$.
Two elements $x$ and $y$ in an inner-product space are said to be {\em orthogonal}
if $\langle x, y\rangle = 0$; then we write $x \perp y$.
If $x \perp y$ then $\| x + y \|^2 = \| x \|^2 + \| y \|^2$.
An element $x$ of an inner product space is orthogonal to a subset $M$ of the same
inner product space if $x \perp y$ $\forall y \in M$; then we write
$x \perp M$. Two subsets $M, N$ of an inner product
space are orthogonal if $x \perp y$ $\forall x \in M$ and $\forall y \in N$; then we
write $M \perp N$.
The {\em orthogonal complement} of a set $M \subset \calX$ is
\beq
M^\perp = \{ x \in \calX : x \perp M \}.
\eeq
For any $M \subset \calX$, $M^\perp$ is a closed subspace.
If $M$ is a closed subspace of $\calX$, each $x \in \calX$ has a unique
decomposition $x = m + n$ where $m \in M$ and $n \in M^\perp$; we write
$\calX = M \bigoplus M^\perp$.
Every finite-dimensional subspace of a Hilbert space is closed.

\begin{Lemma}\label{lemma:finiteDminnorm}
Let $\{ x_j \}_{j=1}^n \subset \calX$ be a linearly independent set, 
$M = \linSpan\{ x_j \}_{j=1}^n$, and $\delta \in \bfR^n$.
Then
\beq
\arg \min_{x \in \calX} \{ \| x \| : \ip{x_j}{x} = \delta_j, \;\;j=1, \cdots, n \} \in M.
\eeq
The minimum is attained, and by a unique element of $M$.
\end{Lemma}

\noindent
{\bf Proof.}
$M$ is a closed subspace of $\calX$, so we can decompose every $x$ into a sum
$x = x_M + x_{M^\perp}$ with $x_M \in M$ and $x_{M^\perp} \in M^\perp$, and
$\| x \|^2 = \| x_M \|^2 + \| x_{M^\perp} \|^2$.
Now $\ip{x_j}{x_{M^\perp}} = 0$ for all $j$, so $\ip{x_j}{x} = \ip{x_j}{x_M}$.
Thus if $\ip{x_j}{x} = \delta_j$ $\forall j$, then $\ip{x_j}{x_M} = \delta_j$ $\forall j$,
but $\| x \| \ge \| x_M \|$. Thus it suffices to consider $x \in M$.
Because $\{x_j\}_{j=1}^n$ is a linearly independent set, there is a unique vector $x_0 \in M$
s.t. $\ip{x_j}{x_0} = \delta_j$. Even if $\{ x_j \}$ were not linearly independent,
that the minimum is attained would follow from the fact that $M$ is a closed subspace of
$\calX$, and therefore complete.  Any sequence of elements of $M$ for which 
the norm converges to the minimal value can be shown to be a Cauchy sequence
within $M$, so
its limit is in $M$. The uniqueness would then follow from the 
strict convexity of the norm in
Hilbert space.

\noindent
{\bf Remark.}
Essentially the same proof shows that if $M$ is a closed subspace and $x \in \calX$, 
\beq
\min_{m \in M} \| x - m \|
\eeq
is attained by an element $m_0$ of $M$, and that $x - m_0 \in M^\perp$. 

\noindent
{\bf Remark.}
Let $\ip{\bfx}{x} = ( \ip{x_j}{x})_{j=1}^n$.
It follows from Lemma \ref{lemma:finiteDminnorm} that if $\Xi \subset \bfR^n$, 
and $x_0$ solves
\beq
\min \{ \| x \| : \ip{\bfx}{x} \in \Xi \},
\eeq
then $x_0 \in M = \linSpan\{ x_j \}_{j=1}^n$.

Let $S$ be a closed subspace of $\calX$. Then
\beq
| x |_S = \min_{s \in S} \| x - s \|
\eeq
is a seminorm.

\noindent
{\bf Assignment.} Verify that for a closed subspace $S$
of a Hilbert space $\calX$, $| \cdot |_S$ is a seminorm.

Let $S = \linSpan \{ s_k \}_{k=1}^m$ be a finite-dimensional 
(and therefore closed) subspace of $\calX$.
Consider the problem of finding 
\beq
\min \{ | x |_S : \ip{\bfx}{x} = \delta \} ,
\eeq
with $\{ x_j \}_{j=1}^n$ linearly independent.
Any $x$ can be decomposed into $x_S + x_{S^\perp}$, with $x_S \in S$ and 
$x_{S^\perp} \in
S^\perp$.
Let $\ip{\bfx}{S} = \{  \ip{\bfx}{s}: s \in S \}$.
Taking $\Xi = \delta + \ip{\bfx}{S}$, the problem becomes
\beq
\min \{ \|x \| : \ip{\bfx}{x} \in \Xi \}.
\eeq
The optimal $x_0$ is thus a linear combination of $\{ x_j \}_{j=1}^n$, plus an element
of $S$; we restrict attention to the subspace of $\calX$ consisting of
linear combinations of elements of $S$ and $\{ x_j \}_{j=1}^n$.
Define $\Gamma$ to be the $n$ by $n$ matrix with elements $\Gamma_{ij} = \ip{x_i}{x_j}$.
Because $\{ x_j \}$ is linearly independent and $\ip{\cdot}{\cdot}$ is
symmetric, $\Gamma$ is a positive-definite symmetric
matrix.
Let $\Lambda$ be the $n$ by $m$ matrix with elements $\Lambda_{ij} = \ip{x_i}{s_j}$.
Write 
\beq
x = \gamma \cdot \bfx + \lambda \cdot \bfs = \sum_{j=1}^n \gamma_j x_j +
\sum_{j=1}^m \lambda_j s_j.
\eeq
The component of $x$ in $S$ does not contribute to $|x|_S$, so
$| x |_S^2 \le \gamma \cdot \Gamma \cdot \gamma$, with equality if
$\gamma \cdot \bfx \perp S$.
For the optimal $x$, that must hold; otherwise, the component could be absorbed into
$x_S$, thereby decreasing $|x|_S$.
Thus the seminorm minimization problem is to find
\begin{eqnarray}
v(\calP) &=&
\min_{\gamma \in \bfR^n, \lambda \in \bfR^m}
\{ \ip{\gamma\cdot \bfx}{\gamma \cdot \bfx}: 
\ip{\bfx}{\lambda \cdot \bfs + \gamma \cdot \bfx}
= \delta \}
\nonumber \\
&=& \min \{ \gamma \cdot \Gamma \cdot \gamma : \Gamma \cdot \gamma + \Lambda \cdot \lambda =
\delta \}
\end{eqnarray}
As just argued, for the optimal $x$, $\ip{s_j}{\gamma \cdot \bfx} = 0$, $j=1, \cdots, m$,
so $\Lambda^T \cdot \gamma = 0$.
This gives the coefficients of $x$ through solving
\beq
\left [ \begin{array}{cc}
\Gamma & \Lambda \\
\Lambda^T & 0
\end{array}
\right ]
\cdot
\left [
\begin{array}{c}
\gamma \\
\lambda
\end{array}
\right ]
=
\left [
\begin{array}{c}
\delta \\
0
\end{array}
\right ],
\eeq
which has a unique solution if the matrix on the left is nonsingular.
That holds if the projections of $\{s_j \}$ onto $\linSpan\{x_j \}$
are linearly independent.

This was for the case that $\ip{\bfx}{x} = \delta$.
If instead we require only $\ip{\bfx}{x} \in \Xi$, the same argument shows
that the optimal $x_0$ is a linear combination of 
$\{ x_j \}_{j=1}^n \cup \{ s_j \}_{j=1}^m$.


\subsection{Quadratic Smoothing Splines}

Let $\calX$ be the set of functions $x = x(t)$
on the interval $[0, 1]$ that are absolutely continuous, have
absolutely continuous first derivatives, and have Lebesgue square-integrable 
second derivatives.
Let $x' = x'(t) = dx/dt$, and $x'' = x''(t) = d^2x/dt^2$.
Endow $\calX$ with the norm
\beq
\| x \|^2 = x^2(0) + |x'(0)|^2 +
\int_0^1 \left ( x''(t)\right )^2 dt .
\eeq
This norm derives from the inner product
\beq
\langle x, y \rangle = x(0)y(0) + x'(0)y'(0) + \int_0^1 x''(t)y''(t) dt .
\eeq
({\em I.e.}, $\|x\| = \sqrt(\langle x, x\rangle)$.)

\noindent
{\bf Problem.}
Show that $\calX$ is indeed a Hilbert space: verify that $\langle \cdot, \cdot \rangle$
is an inner product, and that $\calX$ is complete w.r.t. the induced norm topology.

The space $\calX$ is in fact a reproducing kernel Hilbert space (RKHS).
A RKHS is a Hilbert space of functions $x = x(t)$ for which the functional that
evaluates each function $x$ at the point $t_0$ 
(the point-evaluator) is a bounded linear functional; that is, 
evaluating any $x$ at the point $t_0$ can be written as the inner product
of $x$ with a function $e_{t_0} \in \calX$.
(Clearly, point-evaluation is linear if the addition of functions and multiplication
of functions by scalars is defined in the
standard way:
\begin{eqnarray}
(x+y)(t) &=& x(t) + y(t), \;\; x, y \in \calX \nonumber \\
(\alpha x)(t) &=& \alpha (x(t)), \;\; x \in \calX,\; \alpha \in \bfR . \nonumber
\end{eqnarray}
The issue is whether this linear functional is bounded.)
In the present case, 
\begin{eqnarray}
x(t_0) &=& x(0) + x'(0)t_0 + \int_0^{t_0} du \int_0^u ds x''(s)
\nonumber \\
&=& x(0) + x'(0)t_0 + \int_0^1 du 1_{u \le t_0} \int_0^1 ds x''(s) 1_{s \le u} 
\nonumber \\
&=& x(0) + x'(0)t_0 + \int_0^1 ds x''(s) \int_0^1 du 1_{u \le t_0} 1_{u \ge s}
\nonumber \\
&=& x(0) + x'(0)t_0 + \int_0^1 ds x''(s) (t_0 - s)_+.
\end{eqnarray}
Formally, this looks like the inner product with an element $e_{t_0} \in \calX$
whose value at $0$ is 1, whose derivative at $0$ is $t_0$, and whose second
derivative is $e_{t_0}''(t) = (t_0 - t)_+$.
The question is whether $e(t) \in \calX$.
It is: it is absolutely continuous and has absolutely
continuous first derivative, and bounded
second derivative, so its $\calX$-norm is finite.
We have
\beq
e_{t_0}(t) = \left \{
\begin{array}{cc}
1 + t_0 t + t_0 \frac{t^2}{2} - \frac{t^3}{6} & t \le t_0 \\
1 + t_0 t + t \frac{t_0^2}{2} - \frac{t_0^3}{6} & t > t_0 .
\end{array}
\right .
\eeq
For any collection of distinct points $\{ t_j \}_{j=1}^n$, the set of functions
$\{ e_{t_j} \}_{j=1}^n \subset \calX$ is linearly independent.

Define the seminorm $| x |$ for $x \in \calX$ by
\beq
| x |^2 = \int_0^1 \left ( x''(t)\right )^2 dt.
\eeq
Let $S = \linSpan\{ 1, t \}$.
Then $S^\perp \equiv \{ x \in \calX: x(0) = x'(0) = 0 \}$.
The seminorm $ | \cdot |$ can be viewed as the square of the norm of the orthogonal
projection $P_{S^\perp}$ of $x$ onto the (closed) subspace $S^\perp$;
{\em i.e.},
\beq
| x | = \| P_{S^\perp} x \| = \inf_{\ell \in S} \| x - \ell \| = \| x - P_S x \|.
\eeq
This is a continuous functional on $\calX$:
consider a sequence $x_k \rightarrow x$ in the norm.
The operator norm of a Hilbert-space projection is unity; that is, for any
subspace $M \subset \calX$,
$\| P_M x \| \le 1 \times \| x \|$.
By the triangle inequality, $\| x + (y-x) \| \le \| x \| + \| y-x \|$,
and $\| y + (x-y) \| \le \| y + \| x-y \|$; together these imply that
$\left | \| x \| - \| y \| \right | \le \| x-y \|$.
Thus
\begin{eqnarray}
\left |\| P_{S^\perp} x_k \| - \| P_{S^\perp} x_j \| \right | &\le &
\| P_{S^\perp} x_k - P_{S^\perp} x_j \|
\nonumber \\
&=&
\| P_{S^\perp} (x_k - x_j) \| \nonumber \\
\le \| x_k - x_j \| \rightarrow 0
\end{eqnarray}
Now $S$ is finite dimensional, and therefore closed, so we have
$\calX = S \bigoplus S^\perp$. For any $x \in \calX$, let 
$x_S + x_{S^\perp}$ be the (unique) decomposition of $x$ into its components
$x_S \in  S$ and $x_{S^\perp} \in S^\perp$.
Then $| x | = \| x_{S\perp} \|$.
Let $e_j = e_{t_j}$, $j = 1, \cdots, n$. 
Let $\ip{\bfe}{x} = (\ip{e_j}{x})_{j=1}^n$.
Because each $e_j$ is a continuous linear functional on $\calX$ and the two-norm on
$\bfR^n$ is continuous, for any $\delta \in \bfR^n$ and any $\epsilon >0$,
\beq
D \equiv \{ x \in \calX: \| \bfex - \delta \| \le \epsilon \}
\eeq
is a closed convex subset of $\calX$.
The smoothing spline optimization problem is to
find a function $x_0$ that attains $v(\calP)$,
\beq
v(\calP) \equiv \inf_{x \in D} | x |,
\eeq
which is a seminorm minimization problem of the type just explored.
It follows that $v(\calP)$ is attained, and 
the optimal vector $x_0$ is a linear combination of
$1$, $t$, and $\{e_j\}_{j=1}^n$. 
That is, $x_0$ is an absolutely
continuous piecewise cubic function
with absolutely continuous first derivative and 
square-integrable second derivative.

To try to add some intuition, consider what the solution would be like were
$\{ e_j \}_{j=1}^n \perp S$.  Then there would be no advantage to including
an element of $S$ in the solution, and the optimal $x_0$ would just be a linear
combination of $\{ e_j \}$.  However, if the subspace spanned by $\{ e_j \}$ is not
orthogonal to $S$, a part of the data can be produced by an element of $S$.
The left over part of the data can still be produced using just a linear combination of
$\{ e_j \}$, and there is no advantage to going outside the subspace spanned by
$S \cup \{e_j \}$.  Define $\Gamma_{ij} = \ip{e_i}{e_j}$ and $\Lambda_{ij} = \ip{e_i}{s_j}$.
The seminorm ignores any component of $S$ remaining in the ``extra'' linear combination of 
$\{ e_j\}$, so the optimal solution is of the form $x_0 = x_{0S} + x_{0S^\perp}$,
where $x_{0S^\perp}= \alpha \cdot \bfe$ with 
$\Lambda \cdot \alpha = 0$.

We can also apply Fenchel duality to this problem.
Let $f(x) = | x |_S^2$, $C = \calX$, and $g(x) = 0$. 
Because $\{ e_j \}$ is a linearly independent subset of $\calX$ and
$f$ is continuous, provided $\epsilon > 0$, the
epigraph $[f, D]_+$ has nonempty relative interior.
The conjugate set $\calC^*$ of $C$ is
\begin{eqnarray}
\calC^* &\equiv& \{ x^* \in \calX^* : \sup_{x \in C} 
[\ip{ x^*}{ x } - f(x) ] < \infty \}
\nonumber \\
&=&  \{ x \in \calX : \sup_{y \in \calX} 
[\ip{x}{ y } - |y| ] < \infty \}
\nonumber \\
&=& S^\perp \cap B(1),
\end{eqnarray}
where $B(1) = \{ x \in \calX: \| x \| \le 1 \}$.
The dual functional of $f$ is 
\begin{eqnarray}
f^*(x^*) &=& \sup_{x \in C} [\ip{ x^*}{x} - f(x) ] 
\nonumber \\
&=& \sup_{x \in \calX} [\ip{x^*}{x} - | x |_S^2 ] 
\nonumber \\
&=& \left \{
\begin{array}{cc}
0, & x^* \in S^\perp \cap B(1) \\
\infty, & \mbox{ otherwise } ,
\end{array}
\right .
\end{eqnarray}
The conjugate set $\calD^*$ of $D$ is
\begin{eqnarray}
\calD^* &\equiv& \{ x \in \calX : \inf_{y \in D} [\ip{x}{y} - g(y) ] 
> - \infty \} 
\nonumber \\
&=& \{ x \in \calX : \inf_{y \in D} [\ip{ x}{y} ] 
> - \infty \} 
\nonumber \\
&=& \linSpan\{ e_j \}_{j=1}^n.
\end{eqnarray}
The conjugate functional $g^*$ of $g$ is
\begin{eqnarray}
g^*(x^*) &=& \inf_{x \in D} [\ip{ x^*}{x} - g(x) ] 
\nonumber \\
&=& \left \{
\begin{array}{cc}
-\infty, & x^* \not \in \linSpan\{e_j \}_{j=1}^n \\
\alpha \cdot \delta - \epsilon \| \alpha \|, & x^* = \sum_j \alpha_j e_j ,
\end{array}
\right .
\end{eqnarray}
on substituting the result
we obtained in the section on algebraic duality.
This leads to the dual problem 
\beq
v(\calD) = \max_{\alpha \in \bfR^n : \sum_j \alpha_j e_j \perp S; \| \alpha \cdot \bfe \| \le 1} 
\{ \alpha \cdot \delta - \epsilon \| \alpha \| \} .
\eeq
Taking $\Lambda$ and $\Gamma$ to be defined as above, this yields 
\beq
v(\calD) = \max_{\alpha \in \bfR^n : \alpha\cdot \Lambda = 0; \alpha \cdot \Gamma \cdot \alpha \le 1} 
\{ \alpha \cdot \delta - \epsilon \| \alpha \| \} .
\eeq
This maximum is attained by some $x_0^* = \sum_j \alpha_j e_j$. 
Because the value of the primal problem is also attained by some $x_0 \in \calX$, 
we can use the last part of the
Fenchel duality theorem to characterize $x_0$:
\begin{eqnarray}
\min_{x \in D} \ip{x_0^*}{x} &=& \alpha \cdot \delta - \epsilon \| \alpha \|
\nonumber \\
&=& \ip{x_0^*}{x_0},
\end{eqnarray}
and
\beq
\max_{x \in C} [ \ip{x_0^*}{ x} - |x|] = \ip{x_0^*}{ x_0} - |x_0| = 0.
\eeq
One can use these relations to find $x_0$ explicitly in terms of $\alpha$.
%We seek $x_0 = \sum_j \beta_j e_j$ such that
%\begin{eqnarray}
%c \equiv \alpha \cdot \delta - \epsilon \| \alpha \| &=&
%\ip{\sum_j \alpha_j e_j}{\sum_j \beta_j e_j} 
%\nonumber \\
%&=& \alpha \cdot \Gamma \cdot \beta ,
%\end{eqnarray}
%and 
%\beq
%0 = \ip{x_0^*}{x_0} - | x_0 | =
%\alpha \cdot \Gamma \cdot \beta. 
%\eeq
%The projection $P_S x$ of any $x \in \calX$ onto the subspace $S$ spanned by 
%$\{ s_j \}_{j=1}^m$ can be constructed as follows.
%Let $\Sigma$ be the $m$ by $m$ positive definite symmetric matrix with 
%elements $\Sigma_{ij} =
%\ip{s_i}{s_j}$.
%The rows  of $\Sigma^{-1/2}$ are the coefficients of an orthonormal basis for
%$S$ in terms of $\{ s_j \}_{j=1}^m$: Let $\hat{s}_j = \sum_k \Sigma_{jk}^{-1/2} s_k$.
%Then
%\begin{eqnarray}
%\ip{\hat{s}_j}{\hat{s}_\ell} &=&
%\ip{\sum_k \Sigma_{jk}^{-1/2} s_k}{\sum_i \Sigma_{\ell i}^{-1/2} s_i}
%\nonumber \\
%&=& \sum_k \sum_i \Sigma_{jk}^{-1/2} \Sigma_{\ell i}^{-1/2} \ip{s_k}{s_i}
%\nonumber \\
%&=& (\Sigma^{-1/2} \cdot \Sigma \cdot \sigma^{-1/2} )_{j\ell}
%\nonumber \\
%&=& 1_{j = \ell}.
%\end{eqnarray}
%Note that $\ip{\hat{s}_i}{s_j} = \Sigma_{ij}^{-1/2}$, so $\sigma \cdot \bfs =
%\sigma \cdot \Sigma^{-1/2} \cdot \bfhats$, where
%$\bfhats = (\hat{s}_j)_{j=1}^m$.
%The orthonormal basis for $S$ lets us write 
%\beq
%P_S x = \sum_j \ip{\hat{s}_j}{x} \hat{s}_j.
%\eeq
%For $x = \sum_j \beta_j e_j$, 
%\beq
%P_S x = \bfhats \cdot \Sigma^{-1/2} \cdot \Lambda \cdot \beta,
%\eeq
%and hence for $x = \sum_j \beta_j e_j$,
%\begin{eqnarray}
%| x_0 | = \| x_0 - P_S x_0 \| 
%&=& \| \bfx \cdot \beta - \bfhats \cdot \Sigma^{-1/2} \cdot \Lambda \cdot \beta \| \nonumber \\
%&=& \| x_0 - \sum_j
%\end{eqnarray}


\subsection{Lagrange Duality}
\begin{Theorem}
(See D.G. Luenberger, 1969. {\em Optimization by Vector Space Methods}, Wiley, NY., \S8.6,
Theorem 1.)
Let
\beq
\mu = \inf \{f(x): x \in C,\; G(x) \le 0 \in \calZ\},
\eeq
where $C$ is a convex subset of a linear
vector space $\calX$, $f$ is a real-valued convex functional on $C \subset \calX$, 
$G$ is a convex mapping
from $C$ into a normed space $\calZ$ with positive cone $P$, which is assumed to
have non-empty interior.
Let $\calZ^*$ be the normed dual space of $\calZ$ (the space of all bounded linear
functionals on $\calZ$), and let $P^*$ be the positive cone in $\calZ^*$ induced by
$P$ (the set of linear functionals $z^* \in \calZ^*$ that are 
nonnegative for all $z \in P$).
Suppose that $-\infty < \mu < \infty$, and that $\exists y \in C$ s.t.
$G(y) < c$ ($G(y)$ is in the interior of $\theta - P$).
For $z^* \in P^*$ define 
\beq
\phi(z^*) \equiv \inf_{x \in C} \{ f(x) + \ip{z^*}{G(x)}.
\eeq
Then 
\beq
\mu = \inf_{G(x) \le 0, \;x \in C} f(x) = \max_{z^* \ge 0} \phi(z^*)
\eeq
The maximum on the right is attained by some $z_0^* \ge 0$.
If the infimum on the left is attained by some $x_0 \in C$, then
\beq
\ip{z_0^*}{G(x_0)} = 0,
\eeq
and $x_0$ maximizes
\beq
f(x) + \ip{z_0^*}{G(x)}
\eeq
over all $x \in C$.
\end{Theorem}

\noindent
{\bf Example: Linear Programming}
Let $C$ be the positive orthant in $\bfR^n$.
Define $\calZ = \bfR^m$, endowed with the supremum norm 
$\| z \|_\infty = \max_{1\le i\le n} |z_i|$,
and the positive cone $P = \{z : z_j \ge 0, \;\;j=1, \ldots, m\}$.
Note that $P$ has nonempty interior in the sup-norm topology.
Let $c \in \bfR^m$.
The dual space of $\calZ$ is $\calZ^* = \bfR^m$; its induced positive cone
is the usual positive orthant.
Let $A$ be an $m$ by $n$ matrix, and let $G(x) = A \cdot x - c$.
Let $f(x) = f \cdot x$ for some $f \in \bfR^n$.
We seek
\beq
\mu = \inf_{x \ge 0 \in \bfR^n} \{ f\cdot x : A \cdot x - c \le 0 \} .
\eeq
This is of the canonical form given.

Suppose there exists $x \ge 0 \in \bfR^n$ s.t. $A\cdot x < 0$, and that $\mu$ is finite.
The theorem says that 
\beq
\mu = \max_{y \in \bfR^m: y \ge 0} \inf_{x \in \bfR^n: x \ge 0} 
\{ f\cdot x + y \cdot (A \cdot x - c)\}.
\eeq
The infimum is $-\infty$ unless $f + y \cdot A \ge 0$, in which case it is $-y \cdot c$.

The dual problem is thus
\beq
\max_{y \ge 0: -A^T \cdot y - f \le 0} - c \cdot y,
\eeq
which is another linear program.

\noindent
{\bf Example: Quadratic Programming.}
Let $\Omega = \bfR^n$, $b \in \bfR^n$, $Q$ a positive-definite, symmetric $n$ by $n$ matrix,
$A$ an $m$ by $n$ matrix, $c \in \bfR^m$.
We seek
\beq
v(\calP) = \min_{x \in \bfR^n : A\cdot x \le c } \frac{1}{2} x\cdot Q \cdot x 
- b \cdot x.
\eeq
This is called a quadratic program.
If there exists $x$ s.t. $A\cdot x < c$,
then
\beq
v(\calP) = \max_{\lambda \ge 0} \min_x \{ \frac{1}{2} x\cdot Q \cdot x - b \cdot x
+ \lambda \cdot (A\cdot x - c ) \}.
\eeq
In this case, the minimization problem is the unconstrained, and is
solved by
\beq
x = Q^{-1} (b - A \cdot \lambda ).
\eeq
Let $R = A Q^{-1} A^T$ and $d = c - A Q^{-1} b$.
Then
\beq
v(\calP) = \max_{\lambda \ge 0} \left \{
-\frac{1}{2} \lambda \cdot R \cdot \lambda - \lambda \cdot d - \frac{1}{2}
b \cdot Q^{-1} \cdot b \right \},
\eeq
which is another quadratic program. If $m < n$, the new (dual) program can be substantially
simpler than the original.

\end{document}
