\documentclass{article} 
\include{macros}

\usepackage{floatflt}

%\setlength{\oddsidemargin}{0.25 in}
%\setlength{\evensidemargin}{-0.25 in}
%\setlength{\topmargin}{-0.6 in}
%\setlength{\textwidth}{6.5 in}
%\setlength{\textheight}{8.5 in}
%\setlength{\headsep}{0.75 in}
%\setlength{\parindent}{0 in}
%\setlength{\parskip}{0.1 in}

%\newcommand{\lecture}[3]{
%   \pagestyle{myheadings}
%   \thispagestyle{plain}
%   \newpage
%   \setcounter{page}{1}
%   \noindent
%   \begin{center}
%   \framebox{
%      \vbox{\vspace{2mm}
%    \hbox to 6.28in { {\bf CS281A/Stat241A:~Statistical Learning Theory \hfill} }
%       \vspace{6mm}
%       \hbox to 6.28in { {\Large \hfill #1  \hfill} }
%       \vspace{6mm}
%       \hbox to 6.28in { {\it Lecturer: #2 \hfill Scribes: #3} }
%      \vspace{2mm}}
%   }
%   \end{center}
%   \markboth{#1}{#1}
%   \vspace*{4mm}
%}
\def\qed{\ifmmode\Box\else$\Box$\fi}
\begin{document}

\lecture{18}{Stopping Times and Martingales}{Sridhar Machiraju}{machi@cs}

\section{Stopping Times}

Assume that $\{\F_n\}$ is an increasing set of $\sigma-$fields. Recall that a stopping time $T$ is a 
random variable $T: \Omega \leftarrow \Z^+$ such that
$\forall~n < \infty$, one of the following equivalent conditions holds - 

\begin{center}
\begin{enumerate}
\item
$\{T = n\} \in \F_{n}$

\item
$\{T \leq n\} \in \F_{n}$

\item
$\{T \geq n\} \in \F_{n}$
\end{enumerate} 
\end{center}

{\bf Walds Identity - }
\rm Let $X_1 , X_2 , X_3 ,...$ be i.i.d. random variables with $E|X_i| < \infty$ and
$T$ be a stopping time of $\{\F_n\}$ where $\F_n = \sigma (X_1 , X_2 ,..., X_n )$
with $E(T) < \infty$. Let $S_n = X_1 + X_2 + ... + X_n$.
Then $ES_{T} = EX_1ET$.
\par\noindent
{\it Proof:}
\begin{eqnarray*}
ES_{T} = E(\sum_1^{T}{X_i}) = \sum_1^{\infty}{E(X_i 1_{T\ge i})}
= \sum_1^{\infty}{E(X_i 1_{T > i-1})} = \sum_1^{\infty}{EX_i E1_{T> i-1}}
=EX_1ET
\end{eqnarray*}
The final step is because $X_i$ and $1_{T>i-1}$ are independent~\qed.

While it seems as if common means
for the $X_is$ suffices instead of identical distribitions.
However this is not true. The reason for this is that we need $E|X_i| < \infty$
to hold without which the summation and integral (in the calculation of expected value
of $S_{T}$) cannot be exchanged. The following example illustrates this.

Define $X_i$ as $P(X_i=\pm 2^i) = \frac{1}{2}$. Let $T=\{$inf$~n: S_n\ge 1\}$.
Clearly, $P(T=n) = \frac{1}{2}^n,~ ET = 2 < \infty$ and $ES_T\ge 1$. However,
$EX_i=0$ which clearly violates Walds identity.


We now derive the classic {\bf Gamblers Ruin} formula using Walds identity. The problem is 
that of a random walk that starts at $X_0=a>0$ and $X_i$ is a symmetric simple random walk i.e.,
a probability of $\frac{1}{2}$ for both $1$ and $-1$. Define $T=\{$inf$n: a+S_n=0~ or~ a+S_n=b\}$. 
A practical explanation of this problem is that of a gambler starting with a capital of $a$. We are 
interested in the probability that the gambler wins $b$ before going broke. Formally, we want to 
calculate $P(a+S_T=b)$.
Now, $E(a+S_T)=bP(a+S_T=b)$ is also given by $a+ES_T$ which is $a$ by Walds identity. Hence,
the gambler earns $b$ before getting broke with a probability of $\frac{a}{b}$.

\section{Martingales}

Martingales are defined for a filtration i.e., an increasing sequence of $\sigma-$fields, 
$\F_n~(n=1,2....)$.
A sequence of random variables $M_n$ is adapted to this filtration of $M_n\in\F_n~(n=1,2\dots)$.
Such a filtration is a martingale (MG) (w.r.t. $\F_n$) if 
\begin{itemize}
\item
$E|M_n| < \infty$ and

\item
$E(M_{n+1}|\F_n) = M_n \forall n$.
\end{itemize}
We define $M_0=0$ for convenience and use this definition unless explicitly mentioning otherwise
in the rest of the course.
The filtration with finite means is a sub-martingale if $E(M_{n+1}|\F_n) \ge M_n$ and a 
super-martingale if $E(M_{n+1}|\F_n) \le M_n \forall n$. Note that in the case of martingales,
the second condition implies that $M_n$ is a filtration whereas this is not true in the case of 
sub-martingales and super-martingales.
Another definition that will be used later is that of a predictable sequence. A predictable sequence 
of random variables $M_n$ such that $M_n\in \F_{n-1}$. 

An example of a MG is $S_n=\sum_{i=1}^{n}X_i - nEX_1$ where $X_i$ is a sequence of i.i.d random variables.
This is because 
\begin{equation}
\nonumber E(S_{n+1}-(n+1)EX_1|\F_n) = E(S_n-(n+1)EX_1|\F_n) + E(X_{n+1}|\F_n) = S_n -(n+1)EX_1 + EX_{n+1} 
= M_n
\end{equation}
Also notice that if $X_n=M_n-M_{n-1}$ then $E(X_n|\F_{n-1}) = 0$. Similar results can easily be derived for
super-martingales and sub-martingales.
We will be considering two kinds of results involoving MGs. These are optional stopping theorems (maximal inequalities)
and convergence theorems.

Martingales and predictable sequences can be used in a natural way in gambling systems. 
If $X_n$ is the outcome of the $n^{th}$ bet and $H_n$ is the 
multiplier that the gambler places for this bet his/her earnings on this bet are $X_n\cdot H_n$. 
Since gamblers can place bets at time $n$ based upon the outcomes at times $1...n-1$, $H_n\in \F_{n-1}$ i.e.,
$H_n$ is predictable and $S_n$ is a martingale. Denoting the gamblers earnings after $n$ bets as a new 
variable $Y_n$ we get, assuming $X_0=0$,
\begin{equation}
Y_n = H_1\cdot X_1 +\dots H_n\cdot X_n = H_1(S_1-S_0) + \dots + H_n(S_n-S_{n-1}) = (H\cdot S)_n
\end{equation}
which is the MG-transform. $Y_n$ is an $\F_n$-martingale if $E(Y_n-Y_{n-1}|\F_n)=0$. But,
\begin{equation}
E(Y_n-Y_{n-1}|\F_n)=E(H_nX_n|\F_n)=H_nE(X_n|\F_n)
\end{equation}
The last equality holds only if $X_n$ and $X_nH_n$ are integrable. Since we assume that $X_n$ is integrable,
$X_nH_n$ is integrable if either $H_n$ is bounded or $X_n,H_n$ are in ${\cal L}^2$ (Cauchy Shwartz implies 
that $X_nH_n$ is integrable in this case).


Martingales in conjunction with stopping times are a neat way of modeling gamblers strategies. A martingale $M_n$
whose differences represent the outcomes at time $n$ may be bet upon by a gambler until he stops at some time.
This time is intuitively a stopping time $T$ because (by definition of stopping times) $T=n$ is measurable w.r.t
$\F_n$. We can thus define a new process $M_{n \wedge T}$.

\begin{theorem}
If $M_n$ is an $\F_n-$martingale and $T$ is a stopping time, then $M_{n \wedge T}$ is also an $\F_n-$martingale.
\end{theorem}
\begin{proof}
Using $H_n=1_{T>n-1}\in F_{n-1}$ in the MG-transform formula the result is achieved (note that $H$ is bounded).
\end{proof}

Now, if $M_n$ is a martingale and $T$ is a stopping time bounded by $b$, then 
\begin{equation}
E(M_T) = E(M_{T \wedge b}) = E(M_0)
\end{equation}
In the case of an unbounded stopping time $T$, we have that $M_{T \wedge n}\rightarrow M_T$~a.s.
Hence, if expectations and limits can be swapped as in 
\begin{equation}
E(lim M_{T \wedge n}) = lim E(M_{T \wedge n}) = E(M_0)
\end{equation}
we can calculate the L.H.S. But, this is not possible always. For instance, in case of a random symmetric walk
starting at $S_0=1$ and $T=(inf~n: S_n=0)$, we have $ES_T=0$ because $P(T<\infty)=1$. However,  
$1=ES_0=ES_{T \wedge n}$! As we will see later, uniform integrability is enough to justify the swapping
of the expectations and limits.
\end{document}

