Memory Complexity of Estimating Entropy and Mutual Information

We observe an infinite sequence of independent identically distributed random variables X₁, X₂, . . . drawn from an unknown distribution p over [n], and our goal is to estimate the entropy H(p) = − E[log p(X)] within an ε-additive error.To that end, at each time point we are allowed to update a finite-state machine with S states, using a possibly randomized but time-invariant rule, where each state of the machine is assigned an entropy estimate. Our goal is to characterize the minimax memory complexity S^∗ of this problem, which is the minimal number of states for which the estimation task is feasible with probability at least 1 − δ asymptotically, uniformly in p.Specifically, we show that there exist universal constants C₁and C₂ such that S^∗ (Formula presented) for ε not too small, andS^∗ (Formula presented) for ε not too large. The upper bound is proved using approximate counting to estimate the logarithm of p, and a finite memory bias estimation machine to estimate the expectation operation. The lower bound is proved via a reductionof entropy estimation to uniformity testing. We also apply these results to derive bounds on the memory complexity of mutual information estimation.