Title:
Monotonic convergence in an information-theoretic law of small numbers

Abstract:

The information-theoretic central limit theorem states that the entropy
(if it is ever finite) of the standardized partial sum of i.i.d. random
variables increases monotonically to the entropy of the standard normal. 
While the convergence of the entropy is well-known (Barron 1986), the
monotonicity is fully proved much later (Artstein et al. 2004).  In this
work we establish a discrete version of this monotonicity which involves
the thinning operation (the discrete analogue of scaling) and a Poisson limit
(the discrete counterpart of the normal).  Specifically, let $T_{1/n}
f^{*n}$ denote the probability mass function (pmf) obtained by applying an
$n$-fold convolution and then a $1/n$-thinning of $f$.  Then, as $n$ tends
to infinity, the relative entropy between $T_{1/n} f^{*n}$ and a Poisson
with the same mean decreases monotonically to zero, if it ever becomes
finite.  Moreover, if $f$ is ultra-log-concave, then the entropy
$H(T_{1/n} f^{*n})$ increases monotonically in $n$.  These results extend
the recent work of Kontoyiannis et al. (2005) and Harremo\"{e}s et al.
(2007, 2008).  Ingredients in the proofs include convexity, majorization,
and stochastic orders.  Related issues such as the convergence rate are
also discussed.