Every quantity this course defines — self-information, entropy, mutual information — is an expectation of a function of a random variable. None of that is new mathematics; it's the probability course's own machinery, pointed at a new function.
A probability mass function is a lookup table: pick an outcome, and it hands back the chance that outcome happens. Every entry in the table has to be zero or positive, and the whole table has to add up to exactly one — no probability left over, none borrowed from nowhere. A joint distribution is the same kind of lookup table for a pair of outcomes read off together. Instead of one column of chances for X alone, it's a whole grid: one cell for every combination of what X could be and what Y could be, and the entire grid still has to add up to one.
Two things you can do with that grid matter constantly from here on. First, if you only care about X and want to ignore Y, add up every cell in X's row — that collapses the grid back down to X's own plain distribution, its marginal. (The name comes from exactly that: on a printed table, the row total would sit written in the margin.) Second, if you're told Y's exact value and want to know what that implies about X, restrict to just the cells in that one column, then rescale them so they add back up to one — that rescaled slice is X's conditional distribution given Y's value. Splitting a joint chance this way — first the chance Y happens, then the chance X happens on top of that — is an ordinary "first this, then that," and it resurfaces almost immediately once entropy is defined for pairs.
Last, independence is what it means for two variables to genuinely have nothing to do with each other: knowing Y's value doesn't nudge your guess about X at all, so the joint grid is nothing more than the product of its own row totals and column totals, cell by cell. Two separate coin flips are independent this way. A coin and an echo of that same coin are not — knowing the echo tells you the original flip for free.
A discrete random variable X takes values in a finite or countable set, and its probability mass function (pmf) p(x) := P(X=x) assigns a probability to each value, subject to p(x)\geq 0 and \sum_x p(x)=1. When two random variables X and Y are observed together, their joint pmf p(x,y) := P(X=x,\,Y=y) assigns a probability to every combined outcome (x,y), with \sum_x\sum_y p(x,y)=1. Three facts about p(x,y) this course reaches for in nearly every later chapter:
Marginalization. X's own distribution is recovered by summing the joint pmf over every value of Y: p(x)=\sum_y p(x,y), and symmetrically p(y)=\sum_x p(x,y). This is the event \{X=x\} partitioned by every possible value of Y: P(X=x)=\sum_y P(X=x,\,Y=y).
Conditioning. For any y with p(y)>0, the conditional pmf p(x\mid y):=p(x,y)/p(y) is a valid pmf over x for that fixed y. Rearranged, p(x,y)=p(y)\,p(x\mid y) — probability's own chain rule, factoring a joint outcome as "first Y happens, then X happens given that."
Independence. X and Y are independent exactly when p(x,y)=p(x)p(y) for every pair (x,y) — equivalently, p(x\mid y)=p(x) for every y with p(y)>0: knowing Y changes nothing about X's distribution. Independence is the case where the chain-rule factorization above collapses to a plain product.
Quick review
Practice from earlier coursework, pulled in for this topic.
From Probability
Two dice are rolled. Let X = the sum. Find X(2,5).
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Why
X(2,5) = 2+5 = 7.
Correct answer
Also accepted:
Why
Apply X's rule directly to the two given values.
From Probability
A random variable is a function whose domain is...
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
The real numbers are the CODOMAIN (where values land), not the domain (what the function is applied to).
An event is a subset of the sample space, not what the random variable is defined on directly at the outcome level.
A probability is a single number, not a space of inputs a function could be defined on.
Why
A random variable X: S -> R has the sample space S as its domain and the real numbers as its codomain.
From Probability
Can two different outcomes be assigned the same value by a random variable?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Nothing in the definition of a function (or a random variable specifically) requires distinct inputs to produce distinct outputs.
This has nothing to do with the sample space's size -- even a small finite sample space routinely has several outcomes sharing one value.
Discrete random variables share values just as often -- rolling a 3 and a 4 on two different dice can both sum to 7.
Why
A random variable is an ordinary function, and functions need not be one-to-one -- many outcomes commonly share one value.
Think about whether a function is required to be one-to-one.
From Probability
A random variable assigns...
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
This reverses the direction -- the function maps FROM outcomes TO numbers, and several outcomes can share one number, so the reverse mapping isn't even well-defined.
Assigning a probability directly is what a probability MEASURE does; a random variable assigns a plain number.
Exactly one number, not a set of numbers, is assigned to each single outcome.
Why
X: S -> R assigns exactly one real number to every outcome in the sample space.
From Probability
Two dice are rolled. Let Y = the larger of the two values shown. Find Y(3,5).
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Why
The larger of 3 and 5 is 5.
Correct answer
Also accepted:
Why
Y picks out whichever of the two shown faces is bigger.
From Probability
A coin is flipped 3 times. Let X = the number of heads. Find X(HTH).
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Why
HTH has 2 heads (positions 1 and 3).
Correct answer
Also accepted:
Why
Count the heads in the sequence H, T, H directly.
Quick review
Practice from earlier coursework, pulled in for this topic.
From Probability
A joint table has one cell equal to -0.05. What does this mean?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
The total summing to 1 is necessary but not SUFFICIENT -- every individual cell must also be nonnegative.
A joint pdf still requires nonnegativity everywhere, not just a valid total.
Negative probability is never valid, in any distribution, discrete or continuous.
Why
A negative cell violates the nonnegativity condition directly, regardless of what the total sums to.
Check BOTH conditions separately: nonnegativity is not optional just because the total is 1.
From Probability
In a joint pmf table, what does the cell at row x, column y represent?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
P(X=x) alone is a marginal, found by summing a whole row, not reading one cell.
P(Y=y) alone is a marginal too, found by summing a whole column.
'Or' describes a union, a different (and larger) event than one specific cell.
Why
Each cell holds the joint probability of that exact pair of values occurring together.
One cell corresponds to one specific PAIR of values, not one variable alone.
From Probability
A joint table has cells 0.05, 0.35, 0.25, k, a four-cell table. Find k.
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Why
0.05+0.35+0.25+k=1, so k=1-0.65=0.35.
Correct answer
Also accepted:
Why
Same approach as before: the four cells must sum to exactly 1.
From Probability
A valid joint pmf must satisfy...
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Cells are almost never all equal -- there's no such requirement.
Row totals can be anything nonnegative; nothing requires them to differ.
A zero cell is allowed but never required.
Why
A joint pmf needs every cell nonnegative, with the whole grid summing to exactly 1 -- the same two conditions as a single pmf, applied per cell.
Compare this to the two conditions a single-variable pmf must satisfy.
From Probability
A joint table has 6 cells: 0.05, 0.10, 0.15, 0.20, 0.25, k, a larger six-cell table. Find k.
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Why
0.05+0.10+0.15+0.20+0.25=0.75, so k=1-0.75=0.25.
Correct answer
Also accepted:
Why
Add up the five given cells first, then find what's needed to reach 1.
From Probability
A joint table has cells 0.10, 0.20, 0.15, k. Find k.
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Why
0.10+0.20+0.15+k=1, so k=1-0.45=0.55.
Correct answer
Also accepted:
Why
Set the sum of all four cells equal to 1 and solve for k.
Independent, or not? Read it straight off the grid
Click Check independence — every cell should equal the product of its own row total and column total, confirming X and Y are independent in this table.
Loading interactive…
Worked example
Reading a joint table for what's ahead
-
1
A joint distribution for X\in\{0,1\} and Y\in\{0,1\} is given by p(0,0)=0.4, p(0,1)=0.1, p(1,0)=0.2, p(1,1)=0.3. Find p(X=0) and p(Y=0\mid X=1).
-
2
p(X=0) sums the row where X=0: p(0,0)+p(0,1) = 0.4+0.1 = 0.5. For the conditional, restrict to the column... no, the row where X=1: p(1,0)+p(1,1) = 0.2+0.3=0.5, so p(1,0)/0.5 = 0.2/0.5 = 0.4.
-
3\boxed{p(X=0)=0.5, \quad p(Y=0\mid X=1) = 0.4}
Click around the table from the worked example
Click the X=1 row, then the Y=0 column — confirm the readout matches the 0.4 found above.
Loading interactive…
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
What this task checks
Structure checked automatically. Whether your ideas are convincing isn't graded, so reread with the checklist.
The compiler says
A model answer
Correct answer
Also accepted:
Why
Worked example
Checking independence from a joint table
-
1
Are X and Y from the worked example above independent? Use p(X=0)=0.5, p(X=1)=0.5 (found above) together with the marginals of Y: p(Y=0)=p(0,0)+p(1,0)=0.4+0.2=0.6 and p(Y=1)=p(0,1)+p(1,1)=0.1+0.3=0.4.
-
2
Independence requires p(x,y)=p(x)p(y) at every cell, so check just one cell for a counterexample. p(X=0)p(Y=0) = 0.5 \times 0.6 = 0.30, but the table gives p(0,0)=0.4. The two numbers disagree.
-
3\boxed{p(0,0)=0.4 \neq p(X=0)p(Y=0)=0.30, \text{ so } X \text{ and } Y \text{ are NOT independent}}
One mismatched cell is enough to rule independence out — unlike checking a positive claim, a single counterexample settles a negative one. The independence widget earlier in this section shows a table that does pass the check at every cell.
A reminder: probability is bounded and normalized
Drag b — this weighted blend of two extreme shapes stays a legitimate function everywhere, the same way any valid mixture of two distributions stays a valid distribution.
Loading interactive…
Discrete or continuous?
Sort each random variable by whether it's discrete or continuous — the entropy definitions in Chapter 2 and Chapter 9 split along exactly this line.
Loading interactive…
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
What this task checks
Structure checked automatically. Whether your ideas are convincing isn't graded, so reread with the checklist.
The compiler says
A model answer
Correct answer
Also accepted:
Why
Why
Where this goes next
Chapter 2's definition of entropy is built by summing -p(x)\log p(x) over every outcome — literally E_X[-\log p(X)], an expectation, exactly like E[X] or E[X^2] from the probability course. Chapter 3 then reuses this section's joint-pmf machinery directly: joint entropy sums p(x,y)\log_2 p(x,y) over the whole grid, conditional entropy averages the conditional pmf p(x\mid y) defined above over every fixed value of y, the chain rule for entropy is this section's factorization p(x,y)=p(y)\,p(x\mid y) run through -\log_2 and averaged, and the independence test p(x,y)=p(x)p(y) reappears verbatim as the condition for joint entropy to split into a plain sum and, later, for the mutual information between X and Y to vanish entirely.
Check your understanding (counts toward your progress)
A joint distribution has p(0,0)=0.3, p(0,1)=0.2, p(1,0)=0.1, p(1,1)=0.4. What is p(X=1)?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
That's just p(1,0) alone — the marginal sums the WHOLE row where X=1, both Y values.
That's p(1,1) alone, not the full row for X=1.
That's p(X=0), the other row — you summed the wrong marginal.
Why
The marginal p(X=1) sums every cell in the X=1 row: p(1,0)+p(1,1) = 0.1+0.4=0.5.
Sum every cell in the row where X=1 — don't stop after just one of the two cells.
Using the same joint distribution (p(0,0)=0.3, p(0,1)=0.2, p(1,0)=0.1, p(1,1)=0.4), find p(Y=1 \mid X=1).
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Why
p(Y=1\mid X=1) = p(1,1)/p(X=1) = 0.4/0.5 = 0.8.
Correct answer
Also accepted:
Why
Condition means restrict to the row X=1 only, then divide p(1,1) by the sum of that whole row, not by 1.
Which of these is naturally modeled as a discrete random variable?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Voltage varies continuously over a range — that's a continuous random variable.
An exact arrival time can fall anywhere on a continuum, so it's continuous, not discrete.
SNR is a ratio of continuous power measurements — continuous, not discrete.
Why
A bit-error count only takes whole-number values 0,1,\ldots,100 — the defining trait of a discrete random variable.
Practice: Discrete random variables (4 problems · doesn’t affect your progress)
Which of these is naturally modeled as a discrete random variable: the number of dropped frames in a 10-second video stream, or the exact voltage read off a sensor?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
A sensor voltage can fall anywhere in a continuous range — that's the signature of a continuous random variable, not a discrete one.
Only the dropped-frame count is restricted to whole numbers — voltage is continuous.
Both quantities are genuinely random and unknown in advance — the question is only about discrete versus continuous, not about randomness itself.
Why
A dropped-frame count only takes whole-number values 0,1,\ldots,10 — the defining trait of a discrete random variable — while voltage varies continuously.
Number of dropped frames — whole-number counts only
A random variable representing the exact time (in seconds, to arbitrary precision) between two consecutive network pings is:
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Time measured to arbitrary precision can fall anywhere on a continuum — not restricted to whole numbers or any countable set.
It's a perfectly well-defined kind of random variable, just a continuous one.
Every network's inter-ping time is continuous by this same reasoning — it doesn't depend on network specifics.
Why
A time measured to arbitrary precision can take any value on a continuum, not just countably many — that's continuous, not discrete.
Continuous
A random variable counts how many of 20 coin flips land heads. Even though the physical flipping mechanics are continuous, why is the COUNT itself discrete?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Tangibility has nothing to do with it — what matters is the SET OF POSSIBLE VALUES the random variable itself can take.
Probability theory handles both discrete and continuous random variables — it isn't restricted to one or the other.
The random variable of interest is the COUNT, not the underlying physics — and a count is always confined to whole numbers.
Why
The random variable in question — the count of heads — only takes one of the 21 whole-number values 0,1,\ldots,20, no matter how continuous the physical coin-flipping process underneath it is.
Because the count can only be one of the 21 whole numbers 0 through 20, regardless of the underlying physical mechanics
A 'random variable' reports a customer's account status as one of the labels 'active', 'suspended', or 'closed' — not a number at all. Can it still be treated as a discrete random variable with a pmf?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
The technical definition only requires a finite or countable set of possible values — nothing requires those values to be numbers.
Categorical labels can absolutely have probabilities assigned, exactly the way a pmf assigns one to each possible outcome.
Converting to numbers is a bookkeeping convenience, not a requirement — the pmf works directly on the three labels as given.
Why
This section's technical box defines a discrete random variable as one taking values in a finite or countable SET — nothing in that definition requires the set to be numeric, so three category labels qualify just as well as three numbers.
Yes — a discrete random variable's values just need to be a finite or countable set; they don't have to be numbers
Practice: Random variables and joint distributions (5 problems · doesn’t affect your progress)
A joint pmf table has cells p(n,e) for N (retransmissions: 0 or 1) and E (checksum error: 0 or 1). To find the marginal p(N=0) from the table, which computation is correct?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
That adds down a column (N=0 and N=1 with E=0), not across the N=0 row.
That's just one cell of the N=0 row, not the whole row summed.
Marginalization sums cells, it never multiplies them.
Why
The marginal p(N=0) sums every cell in the row where N=0: p(0,0)+p(0,1), exactly the definition from this section's technical box.
Add p(0,0)+p(0,1)
For the table p(0,0)=0.5, p(0,1)=0.05, p(1,0)=0.1, p(1,1)=0.35, compute p(N=0).
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Why
p(N=0) = p(0,0)+p(0,1) = 0.5+0.05 = 0.55.
Correct answer
Also accepted:
Why
Using the same table (p(0,0)=0.5, p(0,1)=0.05, p(1,0)=0.1, p(1,1)=0.35), find p(E=1 \mid N=1) to three decimal places.
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Why
p(N=1) = p(1,0)+p(1,1) = 0.1+0.35 = 0.45, so p(E=1\mid N=1) = p(1,1)/p(N=1) = 0.35/0.45 \approx 0.778.
Correct answer
Also accepted:
Why
Restrict to the row where N=1 first, then divide the E=1 cell by the sum of that whole row.
For the same table (p(0,0)=0.5, p(0,1)=0.05, p(1,0)=0.1, p(1,1)=0.35), are N and E independent?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Check the actual numbers first — at least one cell fails to match the product of its marginals.
Comparing the two values of N's own marginal says nothing about independence, which needs p(n,e)=p(n)p(e) at every cell.
A joint table is exactly what's needed to test independence — compute the marginals from it, then compare.
Why
p(N=0)=0.55 and p(E=0)=p(0,0)+p(1,0)=0.6, so p(N=0)p(E=0)=0.33, but the table gives p(0,0)=0.5. The mismatch on this one cell is enough to rule independence out.
Compute both marginals p(N=0) and p(E=0) first, then check whether their product matches the table's p(0,0) cell exactly.
No — p(0,0)=0.5 doesn't match p(N=0)p(E=0)
A joint pmf table shows three of its four cells: p(0,0)=0.5, p(0,1)=0.05, p(1,0)=0.1. The fourth cell, p(1,1), is missing. Which principle alone is enough to determine it without any new information?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Nothing forces the row or column totals to be exactly 0.5 in general — that's not a rule of joint pmfs.
One equation (all cells sum to 1) is exactly enough to pin down one missing unknown.
There's no such rule connecting a missing cell's size to the other three.
Why
A joint pmf's cells always sum to 1, so p(1,1) = 1 - 0.5 - 0.05 - 0.1 = 0.35 — the normalization requirement alone determines the missing cell.
Every joint pmf's cells must sum to exactly 1