Chapter 1 is about to define the information in an event using a logarithm, and it can look like an arbitrary choice the first time it's seen. It isn't. A logarithm is close to the only function that can do the job, and this section is about seeing why before the formal definition arrives.
Think about why doubling matters differently for information than it does for probability. If two independent coin flips are each "1 bit" of information, then two flips together should be "2 bits" — addition. But the probability of both flips landing a specific way isn't \frac12+\frac12, it's \frac12\times\frac12=\frac14 — multiplication. Any honest information measure has to convert multiplication (how probabilities combine) into addition (how information combines), and that conversion — turning products into sums — is the defining property of a logarithm. It's not that someone picked \log because it looked good in a formula; \log is forced by requiring independent information to add up.
Suppose two independent events happen — say a coin flip and a die roll — with probabilities p and q. Whatever function f measures the information in an event, it's natural to demand two things of it:
- Independent information should add. Learning both outcomes gives exactly as much information as learning them one at a time and summing: f(pq) = f(p) + f(q), since the joint event has probability pq.
- Less-likely events carry more information. f should be decreasing in p — a rarer event is more surprising.
A function satisfying f(pq)=f(p)+f(q) for all p,q\in(0,1] is called logarithmic in structure, and — given a mild regularity condition like continuity — the only functions with this property are constant multiples of a logarithm, f(p) = k\log_b(p) for some base b and constant k. Combined with the decreasing requirement, k must be negative, which is exactly why the definition ahead is -\log p, not \log p.
Worked example
Checking the additivity requirement directly
-
1
Two fair, independent coins are flipped. Confirm that f(p) = -\log_2 p gives additive information: the information in "both land heads" equals the sum of the information in each individual flip.
-
2
Compute each piece. Each flip: p=\frac12, so f(\frac12) = -\log_2(\frac12) = 1 bit. Both flips together: p = \frac12 \times \frac12 = \frac14, so f(\frac14) = -\log_2(\frac14) = 2 bits.
-
3\boxed{f\!\left(\tfrac14\right) = 2 = 1+1 = f\!\left(\tfrac12\right)+f\!\left(\tfrac12\right)}
Additivity holds exactly, as the functional equation guarantees for any independent pair.
No power of 1/p fixes additivity
Drag a — for every value, f(0.25) = 1/0.25^a should equal f(0.5)+f(0.5) = 2/0.5^a for additivity. Check the two curves' values at x=0.25 and x=0.5 — the gap never closes.
Loading interactive…
The shape Chapter 1 is about to define
This is -log_2(p), decreasing and unbounded as p approaches 0 — drag the base b and see every choice keep the same decreasing shape, just rescaled.
Loading interactive…
Would this function satisfy f(pq) = f(p) + f(q)?
Sort each candidate function by whether it satisfies the additivity requirement for all p, q in (0,1].
Loading interactive…
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
What this task checks
Structure checked automatically. Whether your ideas are convincing isn't graded, so reread with the checklist.
The compiler says
A model answer
Correct answer
Also accepted:
Why
Why
Where this goes next
Chapter 1 picks a specific base for the logarithm — base 2, giving units of bits — but nothing in this section's argument forces that particular base. Any base gives a valid, additive information measure; the base only rescales the answer by a constant factor, which is exactly why "bits" and "nats" (Chapter 1's own units section) are just unit conversions of the same underlying quantity, the way inches and centimeters both measure the same length.
Check your understanding (counts toward your progress)
Why must an information measure satisfy f(pq) = f(p)+f(q) for independent events?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
That's a different requirement (the decreasing property), not what additivity is about.
Information isn't bounded at all — additivity has nothing to do with boundedness.
It follows directly from wanting independent learning events to combine by simple addition, not from arbitrary convention.
Why
Two independent events' JOINT probability multiplies (pq); requiring their information to simply ADD forces the function to convert products into sums — exactly what a logarithm does.
Confirm additivity numerically: for p=0.2 and q=0.5, compute -\log_2(pq) - \big[-\log_2(p) - \log_2(q)\big]. It should be exactly 0.
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Why
-\log_2(0.1) \approx 3.322 and -\log_2(0.2)-\log_2(0.5) \approx 2.322+1 = 3.322 — they match exactly, confirming additivity holds for this specific pair, as the functional-equation argument guarantees for every pair.
Correct answer
Also accepted:
Why
Compute -\log_2(pq) as a single number first, then separately compute -\log_2(p) - \log_2(q), and compare the two results before subtracting.
Which function, besides constant multiples of \log, satisfies f(pq)=f(p)+f(q) for all p,q\in(0,1] (with mild regularity like continuity)?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Why
The functional equation f(pq)=f(p)+f(q), together with continuity, forces f to be a constant multiple of a logarithm — none of the other listed functions satisfy it for every p,q.
Test each candidate against the functional equation directly: plug in two specific numbers for p and q and see whether f(pq) actually equals f(p)+f(q).
Practice: Why logarithms measure information (5 problems · doesn’t affect your progress)
'A rarer event should carry more information than a common one' describes which of the two requirements this section places on an information measure f?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Additivity is about how information from independent events COMBINES, not about how it compares across likely versus unlikely events.
This particular statement is about likelihood and surprise, not about how two independent pieces of information combine.
This is exactly the second requirement named in the technical box.
Why
The decreasing requirement is exactly this: less-likely events should carry more information, so f must decrease as p increases.
The decreasing (monotonicity) requirement
Which single property of logarithms is exactly what additivity, f(pq)=f(p)+f(q), demands?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
This isn't even a true property of logarithms — it garbles addition and multiplication.
This is the power rule, a different (true) property of logarithms, but not the one additivity is asking for.
True, but unrelated to converting products into sums — that's a separate fact about logarithms.
Why
\log(pq)=\log p+\log q is precisely 'turn a product into a sum,' the exact behavior additivity requires of f.
\log(pq) = \log p + \log q
Using f(p)=-\log_2 p, compute f(0.1) - f(0.5) — how many more bits does the rarer event (p=0.1) carry than the common one (p=0.5)?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Why
f(0.1)=-\log_2(0.1)\approx 3.322 and f(0.5)=-\log_2(0.5)=1, so the difference is 3.322-1=2.322.
Correct answer
Also accepted:
Why
Compute f(0.1)=-\log_2(0.1) and f(0.5)=-\log_2(0.5)=1 separately, then subtract.
The technical box requires 'mild regularity like continuity' before concluding f must be a constant multiple of \log. What is that regularity condition actually ruling out?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Continuity is precisely a hypothesis the conclusion depends on — without some regularity condition, the functional equation alone doesn't pin f down to a clean logarithm.
Negativity of f is unrelated to what continuity is ruling out — it comes from the separate decreasing requirement.
p or q equal to 1 is perfectly fine and gives f(1)=0 regardless of continuity — unrelated to this issue.
Why
The functional equation alone, with no regularity condition, technically admits wildly discontinuous solutions that aren't a constant multiple of \log; requiring continuity is what rules those pathological cases out and pins the answer down to genuine logarithms.
Bizarre, non-continuous solutions to f(pq)=f(p)+f(q) that satisfy the equation everywhere but aren't a clean multiple of a logarithm
Does the additivity requirement f(pq)=f(p)+f(q) by itself force f(1)=0, even before deciding on any particular base or constant k?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
The base only rescales f by a constant factor — it can't change whether f(1) is forced to be 0, which the equation settles on its own.
The functional equation holds for ALL p,q\in(0,1], including the case p=q=1.
This holds for every valid choice of f satisfying additivity, not just the base-2 case — it's a consequence of the equation itself.
Why
Setting p=q=1 in the functional equation gives f(1)=f(1)+f(1)=2f(1), which only holds if f(1)=0 — forced by additivity alone, matching 'a certain event carries zero information.'
Yes — set p=q=1: f(1)=f(1)+f(1) forces f(1)=0 directly from the equation alone