Every LLM in this course reads a whole passage of text at once. It lets any part of it inform any other part directly. That wasn't always how sequence models worked. This section recalls the older approach: reading a sequence one token at a time, through a single running summary. It also covers the specific reason that approach eventually gets replaced by attention (Section 6) -- the mechanism every model in this course actually uses.
Picture reading a long paragraph with a single sticky note. It's only one sentence long. You must keep rewriting it as you go. Each time you finish a new word, you cross out the old sentence, and write a new one that tries to sum up everything read so far. That sticky note is the hidden state h_t: one fixed-size summary, wiped and rewritten at every step. It's never allowed to grow, no matter how long the passage gets. f(x_t, h_{t-1}), decoded: "the new summary comes from combining the current word x_t with whatever the previous summary h_{t-1} already said." This process never looks further back than one step. The one-step-back summary is supposed to already hold everything worth remembering.
LSTM and GRU are smarter versions of that sticky note. They add a learned "gate": a small decision-maker. At each word, it chooses how much of the old note to keep, versus cross out -- instead of blindly replacing everything every time. That genuinely helps hold onto important details longer. But notice what didn't change. It's still one sticky note. It's still one fixed size. It still gets rewritten at every single word, gates or no gates.
Here's why that's a real problem, not just a small annoyance. Imagine that sticky note must sum up an entire 50-word sentence, by the time the last word is read. Now say the first word matters a lot for reading the last one -- a pronoun that points back many words later, say. By the time you've rewritten the note 49 times, how much of that first word's exact content do you honestly expect to survive? It's squeezed into the same fixed amount of space, the whole way through. This isn't because anyone did a bad job writing the note. A fixed-size note simply cannot hold a growing amount of detail without losing some of it. Attention (Section 6) fixes this with a very different idea. Instead of forcing everything through one note that keeps getting rewritten, it keeps every word's own note sitting on the table the whole time. Whatever needs information can look directly at whichever notes are actually relevant -- however far back they sit.
A recurrent network processes a sequence one token at a time. It keeps a single hidden state h_t, which gets replaced at every step: h_t = f(x_t, h_{t-1}), for some learned function f. Each new hidden state is computed from the current input and the previous hidden state alone -- never from the raw inputs further back. h_t acts as the network's entire running summary of everything the sequence has shown it so far. Its size is fixed by the network's own design, chosen before any sequence is read. It does not grow as the sequence gets longer.
The LSTM and GRU improve on the plain recurrence above. They add learned gates: small networks that decide, at every step, how much of the old hidden state to keep versus replace, and how much new input to let in at all. Both were built to fight the plain network's tendency to forget information from many steps back. But both still keep the same core commitment: one fixed-size hidden state, replaced at every step, gates or no gates.
That fixed-size commitment is the real problem attention exists to fix. An encoder built from any of the networks above produces a context vector c = h_T: the final hidden state, after reading all T tokens. It's meant to summarize an entire input sequence for whatever reads it next. The size of c is fixed by the design -- identical whether the input was 5 tokens or 500. Every update replaces the running summary, rather than adding to it. So an early token's influence on the final c gets a little weaker at each later step. For long enough sequences, this is not a training bug that can be tuned away. It's a direct consequence of forcing a growing sequence through one fixed-size vector. Sections 6-7 build the alternative. Instead of squeezing the whole sequence into one vector and throwing the rest away, keep every position's own copy of itself available. Let whatever needs information decide, per query, which positions to draw from, and how much.
Watch a running summary get overwritten, step by step
Click Next through all five steps. x_1 = 3 is a strong, clear signal -- watch how little of it is still visible in h_5 by the time the recurrence finishes.
Loading interactive…
Worked example
Basic: does the summary's size depend on sequence length?
-
1
An encoder's hidden state h_t always has dimension d=256. What is \dim(h_T) for a 4-token input, and for a 40-token input?
-
2
\dim(h_t) is a property of the network's own architecture, fixed before any sequence is read -- it never depends on T, the number of tokens actually processed.
-
3
\boxed{\dim(h_T) = 256 \text{ for both inputs}} -- the 40-token input has ten times as much to summarize, packed into the exact same amount of space as the 4-token one.
Worked example
Typical: tracing a running summary by hand
-
1
Using h_t = \tanh(0.5x_t + 0.5h_{t-1}) with h_0=0, and the sequence x_1=3,\ x_2=-1,\ x_3=2, compute h_1, h_2, h_3.
-
2
h_1 = \tanh(0.5(3) + 0.5(0)) = \tanh(1.5) \approx 0.9051.
-
3
h_2 = \tanh(0.5(-1) + 0.5(0.9051)) = \tanh(-0.0475) \approx -0.0474.
-
4
h_3 = \tanh(0.5(2) + 0.5(-0.0474)) = \tanh(0.9763) \approx \boxed{0.7513} -- notice how little trace of x_1=3, a strong input, remains recoverable from h_3 after only two more updates.
Worked example
Common trap: assuming LSTM and GRU gates remove the fixed-size limit
-
1
A student argues that since LSTM and GRU were specifically built to fix recurrent networks' memory problems, they no longer have a fixed-size bottleneck at all. Is that right?
-
2
Gates change how much of the old state survives an update -- they let the network learn to preserve important information longer than a plain recurrence would. They do not change the size of the hidden state itself, which stays fixed by architecture, exactly like the plain recurrence.
-
3
\boxed{\text{No -- LSTM/GRU reduce how quickly information decays, but the state is still one fixed-size vector, overwritten at every step}}. For long enough sequences, the same structural ceiling eventually applies; gates raise the ceiling, they don't remove it.
Where this goes next
Section 6, attention by hand
Every remaining section in this chapter removes the assumption this section just exposed. That assumption said: whatever reads a sequence can only see one fixed-size summary of it. Section 6 builds the fix: attention. It keeps every position's own copy available. Each query then decides which positions are most relevant, and draws from those. That's the mechanism every model in this course, starting Chapter 1, is built from.
Check your understanding (counts toward your progress)
What do LSTM and GRU gates actually change about a plain recurrence?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Gates change how information flows through the SAME fixed-size hidden state -- they don't change its size.
LSTM and GRU still maintain a hidden state -- gates control it, they don't eliminate it.
Gates control how much of the PAST is kept -- they have nothing to do with seeing future tokens.
Why
Gates are learned decision-makers controlling how much of the old state to preserve versus overwrite, fighting the plain recurrence's tendency to forget -- but the hidden state stays the identical fixed size either way.
Gates are about the FLOW of information into and out of the state, not the state's fixed size.
An encoder's hidden state always has dimension 64, regardless of input length. For a 12-token input, what is the dimension of the final context vector c = h_T?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Why
Dimension is a property of the network's own design, chosen before any sentence is read -- it stays 64 regardless of how many tokens (12, here) were actually processed.
Correct answer
Also accepted:
Why
Hidden-state dimension is fixed by the architecture before any sequence is read.
A 40-token sequence is processed by a recurrent network. Which best explains, mathematically, why the FIRST token's influence on the final hidden state tends to fade almost entirely by the end?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
There's no deliberate deletion step -- the fading happens as a side effect of repeated overwriting, not intentional removal.
A plain recurrence processes every token in order; it doesn't truncate to only the last 10.
Nothing forces the first token's own embedding to be zero -- the fading happens to its surviving INFLUENCE on later hidden states, not to the input itself.
Why
Because h_t is recomputed from h_{t-1} at every step and immediately overwrites the previous value, an early token's specific contribution keeps getting folded into (and diluted by) every subsequent update, with nothing preserving its exact original value.
The fading is a side effect of a fixed-size vector being repeatedly recomputed and overwritten, not a deliberate design choice.
Using h_t = tanh(0.5x_t + 0.5h_{t-1}) with h_0 = 0 and x_1 = 4, compute h_1. Round to 3 decimal places.
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Why
h_1 = \tanh(0.5(4) + 0.5(0)) = \tanh(2.0) \approx 0.964.
Correct answer
Also accepted:
Why
With h_0 = 0, the h_{t-1} term drops out entirely for the very first step.
In a recurrent network, what does the hidden state h_t represent?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
The recurrence specifically does NOT keep separate copies -- each new h_t overwrites the previous running summary.
The hidden state is an intermediate, per-step quantity, not the network's single final output.
h_t carries learned information about the sequence's content, not a plain step counter.
Why
h_t is one fixed-size running summary, recomputed and overwritten at every time step from the current input and the previous hidden state.
An encoder's context vector c = h_T always has dimension 128, regardless of input length. What happens to \dim(c) for a 100-token input versus a 5-token input?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Hidden-state dimension is fixed by the architecture before any sequence is read -- it never scales up with sequence length.
There's no per-token growth in dimension at all -- 5 tokens and 100 tokens produce the identical fixed size.
Dimension doesn't grow with token count in a fixed-size recurrent hidden state.
Why
Hidden-state dimension is fixed by the network's own design -- both inputs produce a 128-dimensional c, regardless of how many tokens were read to produce it.
Dimension is an architectural CHOICE made before any sentence is read, not a consequence of that sentence's length.
Check your understanding (counts toward your progress)
According to this section's applied block, what does Section 6 build on top of this section's own conclusion?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Infinite hidden-state size isn't the direction this course takes -- the fix is structural, not "bigger."
Nothing in this course's approach involves deleting tokens up front before processing.
Section 6 changes the underlying mechanism (attention), not just the speed of the same recurrence.
Why
Section 6 builds the actual attention mechanism this section has been motivating: every position's own representation stays available, and each query decides, by computed relevance, what to draw from.
What is the structural fix attention (Section 6) makes, compared to a recurrent network's fixed-size summary?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
No fixed dimension, however large, removes the ceiling entirely for unboundedly long sequences -- attention's fix is structural, not just "a bigger number."
Speed isn't the structural change attention makes -- what information stays available is.
Attention still represents tokens as vectors -- it changes how they're combined, not whether vectors are used at all.
Why
Attention's structural fix is keeping every position's own vector available rather than compressing the whole sequence into one repeatedly-overwritten summary, letting each query draw information from wherever it's actually needed.
The fix isn't a bigger fixed size -- it's not compressing into one summary at all.
A pronoun needs to refer back to a noun many words earlier in a long sentence. Which model is structurally more likely to lose track of that connection: a plain (ungated) recurrent network, or an LSTM?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Gates were built specifically to slow information decay, not speed it up -- LSTM should do relatively better here, not worse.
The two architectures process information differently (gated vs. ungated), so a mathematical guarantee of identical performance doesn't hold.
Fixed-size hidden states are precisely what causes long-range information to fade -- they're directly relevant here.
Why
A plain recurrence has no mechanism to protect important information from being overwritten -- an LSTM's gates can learn to preserve exactly this kind of long-range dependency better, though even it eventually faces the same fixed-size ceiling on long enough sequences.
Which architecture has a mechanism specifically built to help important information survive longer?
Do LSTM and GRU gates completely remove the fixed-size bottleneck of a plain recurrence?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Gated hidden states are still a fixed, chosen size -- gates change how information is preserved within that fixed size, not the size itself.
There's no sequence-length threshold at which gates suddenly remove the limit.
Gates were built specifically to fight information loss, not to make it worse -- they help, they just don't remove the ceiling entirely.
Why
LSTM and GRU raise how much information survives longer, but the hidden state remains one fixed-size vector -- for long enough sequences, the same structural ceiling still applies.
A student claims: "the fixed-size bottleneck is really just an engineering inconvenience -- with enough hidden-state dimensions, a recurrent network could hold onto everything, for any sequence length." What's wrong with this claim?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
There's no clean doubling relationship like that -- and more importantly, ANY fixed dimension still has a ceiling for long enough sequences.
Sequence information content grows with length, unbounded -- it isn't capped by a fixed constant.
There's no such restriction -- real recurrent networks use hidden states of hundreds or thousands of dimensions.
Why
A bigger hidden state genuinely holds more information and helps in practice, but it's still a fixed size chosen in advance -- there will always be some sequence long enough to exceed it, which is exactly why the fix has to be structural (attention), not just "pick a bigger number."
Ask: for ANY fixed dimension you pick, no matter how large, does a long enough sequence eventually exceed it?
Practice: Recurrent hidden states and the fixed-size bottleneck (5 problems · doesn’t affect your progress)
What happens to a recurrent network's hidden state at every new time step?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Why
The hidden state is a single running summary, recomputed from the current input and the previous value, overwriting that previous value each time.
It is recomputed and overwrites the previous value
Using h_t = tanh(0.4x_t + 0.6h_{t-1}) with h_0 = 0 and x_1 = 5, compute h_1 (round to 3 decimals).
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Why
h_1 = \tanh(0.4(5) + 0.6(0)) = \tanh(2.0) \approx 0.964.
Correct answer
Also accepted:
Why
With h_0 = 0, only the x_1 term contributes on the very first step.
Do LSTM gates change the SIZE of the hidden state, or how information flows through a fixed-size state?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Why
Gates control how much of the old state to keep versus overwrite -- the hidden state's SIZE stays exactly as fixed as a plain recurrence's.
How information flows through a fixed-size state
Why can't simply choosing a larger fixed hidden-state dimension fully solve the bottleneck for ANY possible sequence length?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Why
A bigger dimension raises the ceiling and genuinely helps, but for an unboundedly long sequence, some length will always eventually exceed any fixed capacity chosen in advance.
Ask: does the fix scale WITH sequence length, or is it still a fixed number chosen before any sequence is read?
For any fixed dimension, a long enough sequence's information content will eventually exceed it
What does attention (Section 6) keep available that a recurrent summary does not?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Why
Attention's structural fix is keeping every position's own vector available, rather than compressing the whole sequence into one repeatedly-overwritten summary.
Every position's own representation, rather than one compressed running summary