In discrete time, consider the number of visits to state before time . (Recall our convention is to count a visit at time but not at time .) For the stationary chain, we have (trivially)
It’s not hard to calculate the variance:
setting . This leads to the asymptotic result
(2.10) |
The fundamental matrix of (2.6) reappears in an apparently different context. Here is the more general result underlying (2.10). Take arbitrary functions and and center so that and . Write
and similarly for . Then
The contribution to the latter double sum from terms equals, putting ,
Collecting the other term and subtracting the twice-counted diagonal leads to the following result.
(2.11) |
where is the symmetric positive-definite matrix
(2.12) |
As often happens, the formulas simplify in continuous time. The asymptotic result (2.10) becomes
and the matrix occurring in (2.11) becomes
Of course these asymptotic variances appear in the central limit theorem for Markov chains.
For centered ,
The standard proofs (e.g. [133] p. 378) don’t yield any useful finite-time results, so we won’t present a proof. We return to this subject in Chapter 4 section 4.1 (yyy 10/11/94 version) in the context of reversible chains. In that context, getting finite-time bounds on the approximation (2.10) for variances is not hard, but getting informative finite-time bounds on the Normal approximation remains quite hard.
Remark. Here’s another way of seeing why asymptotic variances should relate (via ) to mean hitting times. Regard as counts in a renewal process; in the central limit theorem for renewal counts ([133] Exercise 2.4.13) the variance involves the variance of the inter-renewal time, and by (2.22) below this in turn relates to .