the two theorems that built the information age
Entropy measures uncertainty. For a binary source emitting 1 with probability p:
H(p) = −p log₂ p − (1−p) log₂(1−p)
It peaks at 1 bit when p = ½ (a fair coin — maximally surprising) and falls to 0 when p = 0 or 1 (no surprise, no information). The left panel is this curve, with your source marked.
Source coding theorem: you cannot compress a source below H(p) bits per symbol on average — but you can get arbitrarily close. A biased coin (p = 0.1) carries only H = 0.47 bits/symbol, so it's compressible ~2×. This is the theoretical floor every compressor (ZIP, JPEG, FLAC) chases.
Noisy channel & capacity: send bits through a binary symmetric channel that flips each bit with probability f. The most information you can reliably push through, per use, is the capacity:
C = 1 − H(f)
At f = 0, C = 1 bit (perfect). At f = ½, C = 0 (output independent of input — useless). The mutual information I(X;Y) is how much the output actually tells you about the input for your specific source; capacity is its maximum over all sources.
Channel coding theorem (Shannon's masterpiece): as long as you transmit below capacity, there exist codes that drive the error rate to zero — no matter how noisy the channel. Above capacity, reliable communication is impossible. This was shocking: noise doesn't force errors, it just caps your rate. Modern turbo codes and LDPC codes operate within a fraction of a decibel of the Shannon limit; 5G, Wi-Fi, and the Voyager probes all ride on it.
Reach: entropy is now the common currency of thermodynamics (Boltzmann), machine learning (cross-entropy loss), finance, linguistics, and black-hole physics (the Bekenstein-Hawking entropy is information on the horizon). Shannon gave "information" a number.