Chapters 12 and 13 stop treating a signal as a fixed, known sequence of
numbers. They start treating it as random instead. Sensor noise is one
case. So is the static in a communications channel, or the natural wobble
in a signal measured from a living body. This section teaches the four
numbers that describe a random quantity: expectation, variance,
covariance, and correlation. It uses the notation this course carries
forward. All four were first built in probability. See chapters 4-5
there for random variables and expectation, and chapter 9 for covariance
and correlation. Go there for the full treatment. What follows here is
everything chapters 12-13 need.
Quick review
Practice from earlier coursework, pulled in for this topic.
From Probability
A coin is flipped 3 times. Let X = the number of heads. Find X(HTH).
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Why
HTH has 2 heads (positions 1 and 3).
Correct answer
Also accepted:
Why
Count the heads in the sequence H, T, H directly.
From Probability
Two dice are rolled. Let Y = the larger of the two values shown. Find Y(3,5).
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Why
The larger of 3 and 5 is 5.
Correct answer
Also accepted:
Why
Y picks out whichever of the two shown faces is bigger.
Quick review
Practice from earlier coursework, pulled in for this topic.
From Probability
X takes value 2 with probability 0.5, value 6 with probability 0.5. Find Var(X).
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Why
mu = 4. Var(X) = (2-4)^2(0.5)+(6-4)^2(0.5) = 4(0.5)+4(0.5) = 4.
Correct answer
Also accepted:
Why
Find the mean first, then weight each squared deviation by its own probability.
From Probability
Using the widget in 'same mean, different spread' mode, click the distribution with the LARGER variance.
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
What this task checks
Structure checked automatically. Whether your ideas are convincing isn't graded, so reread with the checklist.
The compiler says
A model answer
Why
The 'Spread' distribution's values sit much further from the shared mean of 5, giving it a much larger variance.
Correct answer
Also accepted:
Why
Compare how far each distribution's own values sit from their shared mean of 5.
Quick review
Practice from earlier coursework, pulled in for this topic.
From Probability
Cov(X, Y) = E(XY) - E(X)E(Y). This formula is the continuous/discrete analog of...
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
The law of total probability is about weighted-average probabilities, not a covariance-style cross term.
Bayes' theorem inverts a conditional probability, unrelated to this algebraic shortcut.
The multiplication rule applies only under independence; covariance is defined whether or not X, Y are independent.
Why
Cov(X,Y)=E(XY)-E(X)E(Y) generalizes Var(X)=E(X^2)-E(X)^2 from one variable to two.
Set Y=X in the covariance formula -- what familiar formula does it become?
From Probability
For a pair with E(XY)=6, E(X)=3, E(Y)=1, find Cov(X, Y).
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Why
6-(3)(1)=6-3=3.
Correct answer
Also accepted:
Why
Same computational formula: E(XY) minus the product of the two means.
Think of a random variable as a machine with a button. You do not know which number it will spit out on any one press. You do know the whole menu of numbers it can hand you. You also know how often each one tends to come up.
A random signal is just many such machines side by side, one per time step. All of them are wired to the same source of chance, so they are not strangers to each other.
Expectation is the number the machine averages toward in the long run. Press the button forever, keep a tally, and that is the value the average settles on. There is a picture for it as well. Lay out a ruler and mark every value x the machine can produce. At each mark, pile on a weight equal to p_X(x), the probability of that value coming up. Now find the spot where the loaded ruler would balance on a fingertip. That spot is the expectation.
Linearity is about what happens if you tinker with the output. Double every result, or add a fixed amount to every result. The balance point doubles, or shifts, by exactly the same amount. Nothing strange happens.
A random variable assigns a number to each outcome of a random experiment. It is not a fixed value. It is a rule for what the value would be under each possible outcome. This course uses the idea in one specific way. A random signal X[n] is one random variable per time index n, all observed from the same underlying experiment. So X[3] and X[7] from the same random signal are two distinct random variables. Each has its own possible values and probabilities. A discrete random variable's probability mass function p_X(x) gives the probability of each value. Its expectation E[X] = \sum_x x\,p_X(x) is the long-run average value. Picture each outcome x as a weight p_X(x) placed on a number line. The expectation is then the balance point of that distribution. Expectation is linear, meaning E[aX+b]=aE[X]+b for constants a,b, exactly as it is for any weighted average.
E[X] as a literal balance point
Move the slider until the fulcrum looks balanced, then click Reveal E[X] to compare.
Loading interactive…
Variance answers one question. How far do the usual values wander from the average? Values packed in a tight cluster near the mean give a small variance. Values thrown far and wide give a large one.
Covariance is the two-variable version of that question. Watch two quantities at once. When the first one sits above its own average, what does the second one do? If it also tends to sit above its own average, the covariance is positive. If it tends to sit below, the covariance is negative. If it shows no steady pattern either way, the covariance is zero.
Correlation asks that very same question with the units stripped out. Its value always falls between -1 and 1. That makes it a fair yardstick. You can compare how strongly two things move together even when the two pairs of quantities are nothing alike.
Zero correlation has a narrow meaning: no straight-line pattern. It does not mean the two quantities have nothing to do with each other. A real link between them can bend, and a bent link can average out to zero correlation.
Calling a random signal "white" is a short way of saying two things. First, every sample looks the same on paper, with the same mean and the same variance. Second, knowing one sample's value tells you nothing about any other sample. Picture a long run of fair coin flips, each flip blind to the ones before it. The samples here behave like that, except each one can land anywhere in a range of values, not just on heads or tails.
Variance measures how spread out a random variable's values tend to be around its own mean \mu=E[X]: \operatorname{Var}(X) = E[(X-\mu)^2] = E[X^2]-\mu^2 The second form follows by expanding the square and applying linearity. Its square root, the standard deviation \sigma_X, is in the same units as X itself. Scaling a random variable scales its variance by the square of the scale factor, so \operatorname{Var}(aX)=a^2\operatorname{Var}(X).
For two random variables X,Y, the covariance \operatorname{Cov}(X,Y) = E[(X-\mu_X)(Y-\mu_Y)] = E[XY]-\mu_X\mu_Y measures how strongly they vary together. It is positive when they tend to move the same direction, and negative when they move in opposite directions. It is zero when there is no linear tendency either way. Dividing by both standard deviations gives the correlation \rho = \frac{\operatorname{Cov}(X,Y)}{\sigma_X\sigma_Y} \in [-1,1], a scale-free version of the same idea.
Independent random variables are always uncorrelated, with \rho=0. The converse can fail. Two variables can be uncorrelated and still not independent, so \rho=0 never proves independence on its own. A random signal is called white, informally, when its samples all share the same mean and variance, and every pair of different time indices is uncorrelated. No one sample's value then gives any linear hint about any other sample's value. Chapter 12 makes this precise.
A white noise signal, sample by sample
Every point on this stem plot is its own random variable, all with the same mean and variance and no correlation between different time indices -- the picture 'white' describes.
Loading interactive…
Dialing in a target correlation
Move the target-correlation slider, then click Resample. Watch the scatter cloud tighten into a line as |correlation| approaches 1, and spread into a shapeless blob as it approaches 0.
Loading interactive…
Worked example
Expectation and variance from a three-point pmf
-
1
A random variable X takes values 0,1,2 with probabilities 0.2,0.5,0.3. Find E[X] and \operatorname{Var}(X). Compute E[X] first, then E[X^2], and combine with the computational form of variance.
-
2
E[X] = 0(0.2)+1(0.5)+2(0.3) = 0+0.5+0.6 = 1.1. E[X^2] = 0^2(0.2)+1^2(0.5)+2^2(0.3) = 0+0.5+1.2 = 1.7, so \operatorname{Var}(X) = 1.7-(1.1)^2 = 1.7-1.21 = 0.49.
-
3\boxed{E[X]=1.1, \quad \operatorname{Var}(X)=0.49}
Chapter 12 computes exactly this pair of numbers for a random signal's samples, one time index at a time.
Loading interactive…
Worked example
Covariance of two signal samples
-
1
Two samples of a random signal have E[X[0]]=0, E[X[1]]=0, and E[X[0]X[1]]=0.6. Find \operatorname{Cov}(X[0],X[1]). Apply \operatorname{Cov}(X,Y)=E[XY]-\mu_X\mu_Y directly.
-
2
\operatorname{Cov}(X[0],X[1]) = 0.6 - (0)(0) = 0.6.
-
3\boxed{\operatorname{Cov}(X[0],X[1]) = 0.6}
Both means being zero simplifies the formula here, but the definition applies unchanged whenever the means aren't zero.
Loading interactive…
Worked example
Correlation, and the uncorrelated-versus-independent trap
-
1
Two signal samples have \operatorname{Cov}(X[0],X[1])=0.6, \operatorname{Var}(X[0])=1, and \operatorname{Var}(X[1])=4. Find their correlation, and state whether zero correlation would have proven the two samples independent. Divide covariance by the product of the two standard deviations.
-
2
\sigma_{X[0]}=\sqrt1=1 and \sigma_{X[1]}=\sqrt4=2, so \rho = \dfrac{0.6}{(1)(2)} = 0.3.
-
3\boxed{\rho = 0.3}
Had the covariance instead been 0, that would say only one thing: no straight-line tendency exists between X[0] and X[1]. It would not by itself rule out some other, non-linear dependence between them.
Loading interactive…
Where this goes next
Random signals, chapters 12-13
Chapter 12 treats a random signal as a sequence of jointly distributed random variables, one per time index. Its first question is whether that signal is white in the sense defined above. Chapter 13 then builds filters that cut a signal's variance. The parts of the signal that matter have to come through untouched. Every one of those designs is scored with exactly the expectation, variance, and covariance machinery taught here.
Check your understanding (counts toward your progress)
Two random variables are found to be uncorrelated, \rho=0. What can be concluded?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Independence always implies zero correlation, but the reverse implication can fail — this is the trap.
Zero correlation says nothing about either variable's own variance.
The covariance is well-defined and equal to zero — that's exactly what \rho=0 means.
Why
Zero correlation only rules out a linear relationship; a real but non-linear dependence can still produce \rho=0, so uncorrelated does not prove independent.
Independence is a one-way implication toward zero correlation — ask whether the reverse direction has to hold too.
Two random variables have E[XY]=1.5, E[X]=1, and E[Y]=1. Find \operatorname{Cov}(X,Y).
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Why
\operatorname{Cov}(X,Y) = E[XY]-\mu_X\mu_Y = 1.5-(1)(1) = 0.5.
Correct answer
Also accepted:
Why
Apply the covariance formula directly, substituting the three given values.
A random variable has E[X]=2 and E[X^2]=6. Find \operatorname{Var}(X).
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Why
\operatorname{Var}(X) = E[X^2]-\mu^2 = 6-2^2 = 6-4 = 2, using the computational form.
Correct answer
Also accepted:
Why
Substitute both given values into \operatorname{Var}(X)=E[X^2]-\mu^2 directly.
Practice: Expectation, variance, and correlation for random signals (2 problems · doesn’t affect your progress)
A random variable X takes values -1 and 1, each with probability 0.5. Find \operatorname{Var}(X).
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Why
E[X]=0 by symmetry, and E[X^2]=(-1)^2(0.5)+1^2(0.5)=1, so \operatorname{Var}(X)=E[X^2]-\mu^2=1-0=1.
Correct answer
Also accepted:
Why
A random signal is described as "white." What does that mean about X[3] and X[10]?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
White doesn't mean the samples are equal — each realization can differ; it constrains their statistics, not their individual values.
A white signal's samples all share the same variance, not different ones.
Quite a lot can be said — same mean, same variance, and zero correlation between the two, by the definition of white.
Why
A white random signal has every sample sharing the same mean and variance, with zero correlation between any two different time indices — exactly the relationship between X[3] and X[10] here.
They share the same mean and variance and are uncorrelated with each other