Skip to content
Free public beta until December 4, 2026: every course is open, no card needed. After that, the first chapter of each course stays free and full access is a one-time $39 ($19 for up to 100 founding members chosen for helpful feedback). Your progress is kept. How the beta works

Chapter 0

Readiness: what information theory leans on

A convex bowl curve with a chord staying above it, next to a small three-state Markov chain diagram with arrows between states A, B, and C
A curve that bends one particular way, and a process with a one-step memory β€” both about to become load-bearing.

The Toolkit Entropy Leans On

Most of what this chapter reviews is genuinely review: random variables, joint distributions, and Bayes' theorem all come straight from the probability course, revisited here only to have them fresh before Chapter 1 leans on them immediately.

Two pieces are new, though, and they aren't a light touch β€” they're the actual mathematical engine behind nearly every theorem in this course. How a function bends, and what that bending guarantees about averages versus applying the function first, is what makes entropy provably non-negative, proves that averaging two distributions can't decrease entropy, and shows that relative entropy is never negative. Markov chains and their stationary distributions are what let Chapter 8 define a well-behaved notion of "entropy per symbol" for a source whose symbols depend on each other, not just an idealized independent stream.

The chapter closes by asking a question that sounds almost philosophical β€” why does information have to be measured with a logarithm at all? β€” and answers it directly: a logarithm isn't a stylistic choice, it's forced by requiring that learning two independent things gives exactly as much information as learning each one separately.

By the end, every tool Chapter 1 needs to define information itself will already be in hand.

  • No matches.