It is much harder to get useful information about distributions (rather than mere expectations). Here are a few general results.
A few useful facts about stationary Markov chains are, to experts, just specializations of facts about arbitrary (i.e. not-necessarily-Markov) stationary processes. Here we give a bare-hands proof of one such fact, the relation between the distribution of return time to a subset and the distribution of first hitting time to from a stationary start. We start in discrete time.
For ,
where .
Proof.The first equality is obvious. Now let be the chain started with its stationary distribution . Then
establishing the Lemma.
We’ll give two consequences of Lemma 2.23. Summing over gives
which extends the familiar fact . Multiplying the identity of Lemma 2.23 by and summing gives
Appealing to Kac’s formula and rearranging,
(2.21) | |||||
(2.22) |
More generally, there is a relation between and .
In continuous time, the analog of Lemma 2.23 is
(2.23) |
where
Integrating over gives the analog of Kac’s formula
(2.24) |
Transform methods are useful in analyzing special examples, though that is not the main focus of this book. We record below just the simplest “transform fact”. We work in discrete time and use generating functions – the corresponding result in continuous time can be stated using Laplace transforms.
Define
Then
Analysis proof. Conditioning on gives
and so
Thus , and the lemma follows.
Probability proof. Let have geometric law , independent of the chain. Then
The distribution at time of the continuization of a discrete-time chain is most simply viewed as a Poisson mixture of the distributions . That is, where has Poisson distribution independent of . At greater length,
This holds because we can construct from by replacing the deterministic “time ” holds by random, exponential, holds between jumps, and then the number of jumps before time has Poisson distribution. Now write for the time of the ’th jump. Then the hitting time for the continuized chain is related to the hitting time of the discrete-time chain by . Though these two hitting time distributions are different, their expectations are the same, and their variances are related in a simple way. To see this, the conditional distribution of given is the distribution of the sum of independent ’s, so (using the notion of conditional expectation given a random variable)
Thus (for any initial distribution)
And the conditional variance formula ([133] p. 198)
tells us that
(2.25) | |||||