Entropy and Data Compression Schemes

Some ways of defining the entropy of a process by observing a single typical output sequence as well as a new kind of Shannon-McMillan-Breiman theorem are presented. Here are two sample results: 1) For a stationary ergodic process let R_n(ξ = inf{k ≥ n: ξ_k+1ξ_k+2 ···ξ_k+n = ξ₁ξ₂ ··· ξ_n }, then a.s. lim_n→∞ (log R_n(ξ)/n = entropy of the process. 2) In the Lempel-Ziv parsing, a.s. for n sufficiently large most of ξ₁ ···ξ_nhas been parsed into blocks of size roughly, (log n)/h, where h is the entropy of the process.