Entropy's most important properties — that it's never negative, that it's maximized by the uniform distribution, that averaging two distributions never decreases it — all come from one shared piece of machinery: convexity. It's worth having the definition and a feel for it before entropy itself shows up in Chapter 2.
Stand two points on the curve and stretch a rubber band between them. Convex means the actual curve sags below that rubber band everywhere in between — like the inside of a bowl. Concave means the curve bulges above the band — like the outside of a dome, or a hill. A straight road is the boundary case: the band lies exactly on top of it, neither sagging nor bulging.
The letters are the trap here: \log x curves the way it does — leveling off as x grows — because doubling a huge number matters far less than doubling a small one. That flattening-out shape is exactly a dome, so \log is concave, not convex, even though nothing about the word "logarithm" suggests a direction. When Chapter 4 needs -\log p(x) instead of \log p(x), the sign flip is what turns that dome upside down into a bowl.
A function g is convex on an interval [a,b] if, for every x_1, x_2 \in [a,b] and every \lambda \in [0,1], g\big(\lambda x_1 + (1-\lambda)x_2\big) \leq \lambda g(x_1) + (1-\lambda)g(x_2). The left side is the function's value at a point between x_1 and x_2; the right side is the corresponding point on the straight line (the chord) connecting (x_1, g(x_1)) and (x_2, g(x_2)). Convexity says the curve never rises above its own chord — it bends upward, or stays flat. A function g is concave on [a,b] if -g is convex there, equivalently if the inequality above reverses: the curve never falls below its chord.
When g is twice differentiable, this reduces to a sign check: g''(x) \geq 0 on the interval means convex, g''(x)\leq 0 means concave. x^2 and e^x are convex everywhere; \log x and \sqrt{x} are concave on x>0; a straight line is both (its chord and its graph coincide).
Chord above or below the curve?
Drag x_1 and x_2 apart — the second curve is the chord connecting them. Watch it stay above x^2 everywhere between the two points.
Loading interactive…
Worked example
Checking convexity with the second derivative
-
1
Is g(x) = x\log x (for x>0, base-2 logarithm) convex or concave? This exact function is about to reappear as the core of the log-sum inequality in Chapter 4.
-
2
Differentiate twice. g'(x) = \log x + \dfrac{1}{\ln 2} (product rule, plus the derivative of \log_2 x). Differentiating again, g''(x) = \dfrac{1}{x\ln 2}.
-
3
For x>0, \ln 2 > 0 and x>0, so g''(x) > 0 throughout the domain.
\boxed{g(x)=x\log x \text{ is convex on } (0,\infty)}
Worked example
A function that's neither, on its full domain
-
1
Is g(x) = x^3 convex or concave on (-\infty,\infty)?
-
2
g''(x) = 6x, which is negative for x<0 and positive for x>0 — it changes sign. g is concave on (-\infty, 0] and convex on [0,\infty), but neither convex nor concave on the whole real line.
-
3\boxed{\text{Convexity/concavity is a property of an interval, not always the whole domain.}}
A concave curve: the chord sits below
Drag x_1 and x_2 — the chord for \log x now stays BELOW the curve everywhere between them, the mirror image of the x^2 widget above.
Loading interactive…
Convex, concave, or neither on their natural domain?
Sort each function by its shape.
Loading interactive…
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
What this task checks
Structure checked automatically. Whether your ideas are convincing isn't graded, so reread with the checklist.
The compiler says
A model answer
Correct answer
Also accepted:
Why
Why
Where this goes next
Convexity isn't a detour — it's the single fact behind three separate results ahead: entropy's non-negativity (Chapter 2), why relative entropy can never be negative (Chapter 4), and why the rate-distortion function is well-behaved enough to actually minimize (Chapter 13). All three arguments have the same shape once the averages-versus-function inequality from the next section is in hand.
Check your understanding (counts toward your progress)
g(x) = -x\log_2 x (with g(0)=0) is the function inside Shannon entropy's formula. Given \frac{d^2}{dx^2}(x\log_2 x) = \frac{1}{x\ln 2}, is -x\log_2 x convex or concave on (0,1]?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
The negative sign flips the shape of x\log_2 x — check the sign of the second derivative of -x\log_2 x, not of x\log_2 x.
Why
Since \frac{d^2}{dx^2}(x\log_2 x) = \frac{1}{x\ln 2} > 0 on (0,1], negating it flips the sign: \frac{d^2}{dx^2}(-x\log_2 x) = -\frac{1}{x\ln 2} < 0, so -x\log_2 x is concave — exactly the shape that will make entropy maximized (not minimized) at the uniform distribution in Chapter 2.
The problem hands you \frac{d^2}{dx^2}(x\log_2 x) = \frac{1}{x\ln 2} directly — you don't need to differentiate from scratch, just apply the negative sign in front of -x\log_2 x to that known second derivative.
Differentiation is linear, so negating a function negates its second derivative too — work out the sign of \frac{1}{x\ln 2} on (0,1] first, then see what negating it does to that sign.
For g(x) = x^4, compute the second derivative g''(x) (using * for multiplication and ^ for exponents).
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Any equivalent form counts — 2x+3, 3 + 2·x, … Use the keypad below the field for fractions, roots, and other symbols.
Why
g'(x)=4x^3, g''(x)=12x^2 \geq 0 for every real x (it's a square), so g is convex on the whole real line.
Correct answer
Also accepted:
Why
Differentiate g(x)=x^4 once first to get g'(x), then differentiate that result again — don't skip straight to a second-derivative formula.
g(x) = \log_2 x on x>0 is:
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Its second derivative is -1/(x^2\ln 2), which is negative — that's concave, not convex.
A function is both only if it's linear; \log x curves, so it can't be both.
Why
g''(x) = -\dfrac{1}{x^2\ln 2} < 0 for all x>0, so \log x is concave — it flattens out as x grows, the signature dome shape.
Recall \frac{d}{dx}\log_2 x = \frac{1}{x\ln 2}, then differentiate once more and check only the sign of what results.
A function's chord between any two points lies entirely above the curve. What is the function?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Concave is the opposite — the chord lies BELOW the curve.
A linear function's chord coincides with the curve, it doesn't lie strictly above it.
Why
The chord staying above the curve everywhere between two points is exactly the definition of convex.
Practice: Convex and concave functions (5 problems · doesn’t affect your progress)
Without computing any derivative, which shape description matches e^x everywhere?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
That's the mirror-image description — concave curves have chords sitting below them, not above.
e^x genuinely curves; it isn't a straight line, so its chord and graph don't coincide.
Convexity is a property of the whole curve on its domain here, not something that flips depending on which two points are chosen.
Why
e^x bends upward at an increasing rate everywhere, so its chord always sits above the curve — convex on the whole real line.
A chord between two points sits above the curve (convex)
For h(x) = x^6, compute the second derivative h''(x) (using * for multiplication and ^ for exponents).
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Any equivalent form counts — 2x+3, 3 + 2·x, … Use the keypad below the field for fractions, roots, and other symbols.
Why
h'(x)=6x^5, so h''(x) = 30x^4.
Correct answer
Also accepted:
Why
Differentiate x^6 once to get h'(x), then differentiate that result once more.
h(x) = -x^2 is:
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Check the sign of h''(x) directly — it isn't positive here.
A negative second derivative everywhere is a clean case, not an ambiguous one.
Why
h''(x) = -2 < 0 everywhere, so h is concave — the negative sign flips x^2's bowl into a dome.
Compute h''(x) directly and check only its sign — the leading coefficient's sign carries straight through from h to h''.
Concave
h(x) = |x| isn't twice differentiable at x=0, so the second-derivative test doesn't apply there. Using the chord definition directly instead, is h convex on [-1,1]?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
A corner doesn't violate the chord definition — convexity never required smoothness, only that the chord stays on or above the curve, which still holds right through the sharp point.
The chord definition applies to any function, differentiable or not — the second-derivative test is just a shortcut for when it happens to apply.
Pick any two points spanning the corner, on either side of 0 — the chord between them still sits above the curve, so the restriction to [0,1] isn't needed.
Why
The chord definition doesn't require differentiability anywhere — check any two points on the V shape and the straight segment between them sits on or above the curve, so |x| is convex on the whole interval despite the corner.
Yes — every chord between two points on |x| lies on or above the V-shaped curve
If f and g are both convex on the same interval, is f+g guaranteed to be convex there too?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Try building a counterexample — the chord-inequality argument for why it DOES survive addition is hard to break.
Nothing in the chord-inequality argument requires the two functions to be equal to each other.
A function can't be both convex and concave unless it's linear, so this condition would rule out almost every genuine example.
Why
Add the two true chord inequalities for f and g at the same x_1,x_2,\lambda — the sum of two true inequalities pointing the same direction is still true, so f+g satisfies convexity's chord inequality too.
Yes — applying the chord inequality to f and g separately and adding them preserves the inequality