Entropy coding

More precisely, the source coding theorem states that for any source distribution, the expected code length satisfies ${\displaystyle \operatorname {E} _{x\sim P}[\ell (d(x))]\geq \operatorname {E} _{x\sim P}[-\log _{b}(P(x))]}$, where ${\displaystyle \ell }$ is the number of symbols in a code word, ${\displaystyle d}$ is the coding function, ${\displaystyle b}$ is the number of symbols used to make output codes and ${\displaystyle P}$ is the probability of the source symbol. An entropy coding attempts to approach this lower bound.