Information Theory

Create a free account to open this chapter.

• Quantifying surprise ~8 min
• Why probability alone isn't enough (Hartley's measure and its limits) ~9 min
• Self-information ~10 min
• Why logarithms ~9 min
• Choice of base: bits, nats, and other units ~9 min
• Additivity of information for independent events ~10 min
• Information and expectation ~10 min
• A first widget-driven intuition check ~9 min

Create a free account to open this chapter.

• Entropy as expected information ~9 min
• Entropy of a discrete random variable ~9 min
• The binary entropy function ~8 min
• Bounds on entropy (0 to log k) ~9 min
• When entropy is maximized ~9 min
• When entropy is minimized ~8 min
• Entropy of common distributions ~10 min
• Entropy under deterministic transformations ~10 min
• Entropy and coding, a first preview ~9 min

Create a free account to open this chapter.

• Joint entropy ~11 min
• Conditional entropy ~11 min
• The chain rule for entropy ~10 min
• Conditioning reduces entropy ~12 min
• Entropy of independent variables ~9 min
• Mutual information, defined two ways ~12 min
• Mutual information as shared information ~12 min
• Conditional mutual information ~12 min
• Mutual information and independence ~9 min
• The data-processing inequality ~13 min

Create a free account to open this chapter.

• Kullback-Leibler divergence ~11 min
• Gibbs' inequality (non-negativity) ~10 min
• Why KL divergence isn't a metric ~10 min
• Asymmetry and the two directions of KL ~10 min
• Cross-entropy ~10 min
• The entropy/cross-entropy/KL relationship ~10 min
• KL divergence in model comparison ~10 min

Create a free account to open this chapter.

• Why compression is possible ~11 min
• Fixed-length vs. variable-length codes ~11 min
• Prefix (instantaneous) codes ~12 min
• The Kraft inequality ~12 min
• The McMillan inequality ~12 min
• Expected code length and redundancy ~11 min
• The entropy lower bound ~12 min

Create a free account to open this chapter.

• The Huffman algorithm ~11 min
• Why merging the two least-probable symbols is optimal ~13 min
• Building and reading a Huffman tree ~10 min
• When a distribution has more than one optimal tree ~10 min
• The Shannon-Fano algorithm ~11 min
• Huffman vs. Shannon-Fano ~10 min
• Arithmetic coding: the interval-narrowing idea ~11 min
• Encoding and decoding with arithmetic coding ~12 min
• Comparing the three methods on one source ~12 min

Create a free account to open this chapter.

• The problem of unknown source statistics ~9 min
• Adaptive Huffman coding and the sibling property ~13 min
• LZ77 ~12 min
• LZSS ~11 min
• LZ78 ~12 min
• LZW ~12 min
• Dictionary coding in practice (zip, gzip) ~10 min
• Compression method trade-offs ~11 min

Create a free account to open this chapter.

• Shannon's source coding theorem, stated ~10 min
• Typical sequences ~12 min
• The asymptotic equipartition property, two equivalent forms ~14 min
• Probability concentration as n grows ~12 min
• ~2^(nH(X)) typical sequences ~12 min
• The law of large numbers, revisited ~12 min
• Entropy as the fundamental compression limit ~14 min
• Lossless compression in the limit ~12 min

Create a free account to open this chapter.

• Differential entropy for continuous variables ~11 min
• Why differential entropy can be negative ~10 min
• Differential entropy of the uniform distribution ~10 min
• Differential entropy of the Gaussian ~12 min
• The Gaussian maximizes entropy for fixed variance ~13 min
• Joint and vector Gaussian entropy ~13 min
• Entropy under linear transformations ~12 min
• Continuous mutual information and relative entropy ~13 min
• Why Gaussians appear throughout information theory ~11 min

Create a free account to open this chapter.

• The source-encoder-channel-decoder-destination model ~10 min
• Discrete memoryless channels and transition probabilities ~12 min
• The binary symmetric channel ~12 min
• The binary erasure channel ~12 min
• Errors vs. erasures ~10 min
• Channel capacity as a mutual-information maximization ~14 min
• Capacity for symmetric and weakly symmetric channels ~16 min
• The capacity-achieving input distribution ~14 min
• Capacity vs. entropy ~10 min
• Channel capacity as an engineering budget ~12 min

Create a free account to open this chapter.

• Reliable communication, defined ~16 min
• Shannon's noisy-channel coding theorem, stated ~16 min
• Rates below capacity ~18 min
• Rates above capacity ~16 min
• Block length and error probability ~16 min
• Why the theorem is nonconstructive ~15 min

Create a free account to open this chapter.

• Motivation: detecting vs. correcting errors ~9 min
• Hamming distance and minimum distance ~13 min
• Repetition and parity codes ~11 min
• Linear codes as a vector-space idea ~12 min
• Generator matrices ~12 min
• Parity-check matrices ~11 min
• Syndrome decoding ~12 min
• Hamming codes ~11 min
• The Hamming, Singleton, Plotkin, and Gilbert-Varshamov bounds ~14 min
• Convolutional codes, a first look ~11 min
• CRC and error detection ~12 min

Create a free account to open this chapter.

• When lossless isn't the goal ~9 min
• Distortion measures (Hamming, squared error) ~10 min
• The rate-distortion function ~16 min
• Quantization ~13 min
• Vector quantization and transform coding ~12 min
• The discrete cosine transform ~15 min
• Rate vs. quality, quantitatively ~12 min
• JPEG-style compression, conceptually ~14 min

Create a free account to open this chapter.

• Entropy and secrecy ~12 min
• Perfect secrecy, defined ~10 min
• The one-time pad ~13 min
• Why key length matters ~13 min
• Information leakage ~12 min
• Shannon's secrecy result ~11 min

Create a free account to open this chapter.

• Maximum likelihood and information ~14 min
• The maximum entropy principle ~15 min
• KL divergence as model fitting ~14 min
• Cross-entropy loss ~13 min
• Fisher information ~13 min
• The Cramer-Rao bound ~15 min
• Information gain and decision trees ~14 min
• Mutual information for feature selection ~12 min
• Softmax and temperature ~13 min
• The information bottleneck, a first look ~12 min

Create a free account to open this chapter.

• The data processing inequality, formalized ~10 min
• Fano's inequality ~13 min
• Pinsker's inequality ~12 min
• Variational inference and the ELBO, a preview ~11 min
• Network information theory: multiple-access and broadcast channels, a preview ~10 min
• Distributed source coding, a preview ~10 min
• Quantum information, a first look ~12 min

Create a free account to open this chapter.

• Designing a complete communication system ~14 min
• Trade-offs across source coding, channel coding, and rate ~15 min
• The capstone lab widget ~18 min