Entropy Rate

Now lets say those X_i's are correlated

$$H(X)=\lim_{n\rightarrow\infty}H(X_n\vert X_{n-1}...X_1)$$

i.e. how much is new under the sum. What does history tell you about the present
For stationary processes we also have

$$H(X)=\lim_{n\rightarrow\infty} \frac{1}{n} H(\underline{X}^n)$$

Aside Memory source $\Rightarrow H(x)=H(X)$