Section 1 used the phrase "language model" loosely, the way most conversation does. This section makes it precise. It defines one specific mathematical object. It gives one way of scoring how good a particular model is at its one job. And it shows the oldest working example -- a model with no neural network in it at all.
A language model, decoded plainly, is a machine. It looks at some text so far, and guesses what comes next. Each guess carries a probability. There's a 40% chance the next word is "mat," a 15% chance it's "floor," and so on, down a very long list. That list covers the model's entire vocabulary. "Probability distribution over sequences" sounds abstract. It just means the model has an opinion. That opinion is a number between 0 and 1. It says how likely any stretch of text is to be real, fluent language.
The big formula, P(w_1,\ldots,w_n) = \prod_{i=1}^n P(w_i \mid w_1,\ldots,w_{i-1}), says something much simpler in math notation. The chance of an entire sentence equals the chance of the first word. Multiply that by the chance of the second word, given the first. Multiply again by the chance of the third word, given the first two. Keep going, one word at a time. (\prod, the capital Greek letter pi, means "multiply everything that follows" -- the multiplication version of \sum. The vertical bar \mid reads as "given.") Nothing about this formula is specific to neural networks. It's just how probability always factors a sequence -- whether the thing computing each piece is a billion-parameter Transformer or a simple counting trick.
The n-gram model is that counting trick. It's worth seeing once in its full simplicity. Every later, fancier model this course studies is still doing the same job: predict the next token from context. It just uses a much better way of using that context. An n-gram model reads a huge pile of training text and counts. Every time it sees the word "the," it notes what comes right after, and how often. To guess what follows "the" in new text, it just reports those same fractions back. C(\cdot) means "count of." C(w_{i-1}, w_i) is "how many times did I see this exact pair of words next to each other." C(w_{i-1}) is "how many times did I see the first word at all." A bigram looks back one word. A Markov assumption is a blunt rule. It says: "only look back a fixed, small number of words, and pretend anything further back doesn't exist." It's crude. But it's a genuine, working language model -- and the direct ancestor of everything Sections 5-7 of this chapter build toward.
A language model is a probability distribution over sequences of tokens. Write a sequence as w_1, w_2, \ldots, w_n. A language model assigns a probability P(w_1, \ldots, w_n) to every possible sequence. Fluent, plausible text gets a higher score; text that doesn't read well gets a lower one. Estimating P(w_1,\ldots,w_n) directly, for an entire sequence at once, isn't practical -- even a short sequence has a huge number of possible versions. So every real language model instead uses the chain rule of probability. That rule factors the sequence into a product of next-token conditionals: P(w_1, \ldots, w_n) = \prod_{i=1}^n P(w_i \mid w_1, \ldots, w_{i-1}). This is exact, not an approximation. The chain rule holds for any joint distribution. It turns "score one whole sequence" into "predict one token at a time, each prediction conditioned on every token that came before it." That's the only thing a language model -- including every LLM in this course -- is actually trained to do. Everything else an LLM appears to do (answering questions, writing code, holding a conversation) is this one operation, repeated: predict the most likely next token, append it, repeat.
An n-gram model is the oldest working language model, and it needs no neural network at all. It estimates each conditional directly from counts in a training corpus. It uses a Markov assumption: a token's probability depends only on the previous n-1 tokens, not the full history. A bigram model (n=2) estimates P(w_i \mid w_{i-1}) = \frac{C(w_{i-1}, w_i)}{C(w_{i-1})}, where C(\cdot) counts occurrences in the training corpus -- the fraction of the time w_{i-1} is immediately followed by w_i. This is a genuine, if crude, language model. It factors sequence probability with the same chain rule above, just with a short conditioning window and no learned parameters -- only counts.
Read a next-token probability off a tiny bigram table
A toy corpus produces this joint distribution over (previous token, next token) pairs. Click a row, then a column, to read off the joint probability and the conditional bigram probability P(next token | previous token) -- exactly the fraction the formula above describes.
Loading interactive…
Formula
Perplexity
Take a sequence w_1, \ldots, w_n and a language model's conditional probabilities P(w_i \mid w_{<i}). Perplexity is then \operatorname{PPL} = \exp\left(-\frac{1}{n}\sum_{i=1}^n \log P(w_i \mid w_{<i})\right). This is the exponentiated average negative log-probability the model assigns to each token. It is an exponentiated average of per-token surprise. A perplexity of k means something specific. On average, the model was as uncertain at each step as if it had been choosing among k options at random. Lower perplexity is better. It means the model assigned higher probability to what actually came next.
Worked example
Basic: factoring a 3-token sequence
-
1
Write P(w_1, w_2, w_3) for the sequence "large language model" as a product of next-token conditionals. Use the chain rule, with no approximation.
-
2
Apply the chain rule once per extra token. Each factor is conditioned on everything placed before it.
-
3
\boxed{P(\text{large})\, P(\text{language}\mid\text{large})\, P(\text{model}\mid\text{large},\text{language})} -- three factors, exact, nothing approximated yet.
Worked example
Typical: one bigram probability from counts
-
1
A training corpus contains the word "language" 200 times total. It contains "language model" as a consecutive pair 60 times. Estimate P(\text{model}\mid\text{language}) under a bigram model.
-
2
Apply the bigram formula directly: divide the pair count by the first-word count.
-
3
\boxed{P(\text{model}\mid\text{language}) = 60/200 = 0.3}
Worked example
Typical: perplexity of a short test sequence
-
1
A trained model assigns these probabilities to a 4-token test sequence, each conditioned on its own earlier context: 0.6,\ 0.5,\ 0.8,\ 0.5. Use \operatorname{PPL} = \exp\!\left(-\frac1n\sum_i\log P(w_i\mid w_{<i})\right) to compute the sequence's perplexity.
-
2
Take the log of each probability: \log 0.6\approx-0.511, \log0.5\approx-0.693, \log0.8\approx-0.223, \log0.5\approx-0.693. Sum: \approx-2.120. Divide by n=4 and negate: 0.530.
-
3
\operatorname{PPL} = \exp(0.530) \approx \boxed{1.70} -- on average, this model is about as uncertain as choosing uniformly among 1.7 options at each token. That's a confident model on this test sequence.
Worked example
Common trap: a lower probability sequence isn't automatically a worse model
-
1
A model assigns a 12-token test sentence an overall sequence probability of 0.0001. A second model assigns a different, 4-token test sentence a probability of 0.01. Can you conclude the second model is better, from these two numbers alone?
-
2
Raw sequence probability shrinks automatically as a sequence gets longer, just from multiplying more factors together. A longer sequence having a smaller raw probability says nothing about how good the model's per-token guesses actually were.
-
3
\boxed{\text{No -- compare perplexity, not raw sequence probability}}. Perplexity's \exp(-\frac1n\sum\ldots) divides by the sequence length n, exactly so scores across different-length sequences become comparable. Raw probability alone confuses "long sequence" with "bad model."
Where this goes next
Chapter 20, evaluating language models
Perplexity, exactly as defined here, is the first metric Chapter 20 reaches for. It compares two language models on the same held-out text. Lower is better. It stays the standard score for "how surprised is this model by real text," for every model in this course -- from an n-gram baseline through a production-scale LLM.
Check your understanding (counts toward your progress)
What does a bigram model condition its prediction on?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Unlimited full-history conditioning is exactly what an n-gram model's Markov assumption avoids -- that's the whole reason it's a manageable count-based model.
A bigram model conditions on the immediately preceding token, not one further back.
A bigram model's whole mechanism is estimating P(w_i \mid w_{i-1}) -- context is exactly what it uses.
Why
A bigram model (n=2) is a first-order Markov model: it estimates the next token's probability from only the single previous token.
A training corpus contains the word "model" 80 times total, and "model predicts" as a consecutive pair 20 times. Under a bigram model, what is P(predicts | model), as a decimal?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Why
The bigram estimate is C(\text{model},\text{predicts})/C(\text{model}) = 20/80 = 0.25.
Correct answer
Also accepted:
Why
Divide the pair count by the count of the first word alone.
A test sentence contains a 3-word combination that never once appeared in the training corpus. Under the plain, unsmoothed bigram formula, what happens to that pair's estimated probability, and what does that do to the sequence's overall probability?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
The raw count-ratio formula gives an unseen pair a count of exactly 0, not a small nonzero estimate -- that's the sparsity problem, not a rounding effect.
The chain-rule product includes every factor; a zero factor anywhere makes the whole product zero, it doesn't get skipped.
An unseen pair has count 0 in the numerator, giving probability 0, not 1 -- there's no default-to-certain behavior in the plain formula.
Why
An unseen bigram has C(w_{i-1},w_i)=0, making its estimated probability exactly 0 -- and since the chain rule multiplies every factor together, a single zero factor sends the entire sequence's probability, and its perplexity, to 0 and infinity respectively.
The bigram formula divides a pair count by a word count -- what does a pair count of exactly 0 do to that fraction?
Which correctly factors P(w_1, w_2) using the chain rule?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
That would only be correct if w_1 and w_2 were independent -- the chain rule instead conditions each later token on everything before it.
The chain rule conditions forward (later tokens on earlier ones), not backward.
Probabilities of independent-seeming events combine by multiplication here, via the chain rule -- not by addition.
Why
The chain rule gives P(w_1,w_2) = P(w_1)\,P(w_2\mid w_1): the first token's own probability, times the second token's probability given the first.
The chain rule conditions each new token on everything placed before it, starting from the first token alone.
Why does every practical language model factor P(w_1,\ldots,w_n) using the chain rule instead of estimating it directly?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
The chain-rule factorization is exact, not an approximation -- it holds for any joint probability distribution.
Text quality isn't the reason for the factorization -- it's about making the probability genuinely computable at all.
The chain rule generalizes to a sequence of any length, applied once per additional token.
Why
Scoring an entire sequence directly is computationally hopeless given how many sequences exist; the chain rule turns it into a sequence of next-token predictions, exactly and without approximation.
Think about how many distinct sequences of, say, 20 tokens exist out of a 50,000-word vocabulary.
What object does a language model formally define?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
A language model scores every possible sequence with a probability -- it doesn't output one single 'correct' sentence.
Part-of-speech tagging is a different NLP task entirely, unrelated to what a language model is defined to be.
Counting words is bag-of-words (Section 3), a different and much simpler object.
Why
A language model is a probability distribution over token sequences, almost always factored into a product of next-token conditionals via the chain rule.
Check your understanding (counts toward your progress)
A trained model assigns these probabilities to a 3-token test sequence: 0.4, 0.9, 0.6. Compute the sequence's perplexity, rounded to 2 decimal places.
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Why
\log(0.4)\approx-0.916, \log(0.9)\approx-0.105, \log(0.6)\approx-0.511; sum \approx-1.532; average \approx-0.511; \exp(0.511)\approx1.65.
Correct answer
Also accepted:
Why
Take the log of each probability, average the three, negate, then exponentiate.
Why does perplexity normalize by the -1/n power rather than just reporting raw sequence probability?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Raw sequence probability is a genuine product of many probabilities less than 1 -- it's not fixed at exactly 1.
Perplexity is defined to be a positive number (it's an exponential); normalization isn't about allowing negative scores.
The -1/n normalization has a real mathematical purpose: making scores length-independent.
Why
Without the length normalization, a longer test sequence would always look 'worse' purely from having more chances to multiply in a probability less than 1, regardless of how good the model actually is.
A model has perplexity 8 on a test set whose real vocabulary has 40,000 words. What's the honest reading of that number?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Perplexity isn't an accuracy percentage -- it's an effective-branching-factor reading of average uncertainty.
Perplexity doesn't restrict which words the model can consider -- every vocabulary word still gets some probability.
The vocabulary size (40,000) and the perplexity score (8) are two separate numbers; one doesn't shrink the other.
Why
Perplexity of 8 means the model's average uncertainty at each step is comparable to a uniform guess among 8 options, even though the real vocabulary is far larger.
Perplexity's operational meaning is "effective branching factor," not a percentage or a vocabulary cutoff.
What does a lower perplexity score mean?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
That describes higher perplexity, not lower -- perplexity and surprise move in the same direction.
Perplexity measures prediction quality, not vocabulary size directly.
Perplexity is computed from a test sequence's assigned probabilities, not from training-corpus length.
Why
Lower perplexity means the model assigned higher probability, on average, to what actually came next -- less surprised, and a better score.
A trained model assigns probabilities 0.5, 0.5, 0.5, 0.5 to a 4-token test sequence, each conditioned on its own preceding context. Using PPL = exp(-（1/n）* sum of log P), compute the perplexity. Round to 1 decimal place.
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Why
Each \log(0.5)\approx-0.693; the average is -0.693; negating and exponentiating gives \exp(0.693) = 1/0.5 = 2.0 exactly.
Correct answer
Also accepted:
Why
With every probability identical, the average log-probability is just log(0.5) -- and exp(-log(0.5)) simplifies directly.
According to the perplexity-formula card recalled in this section, perplexity is the exponentiated average of what quantity?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Perplexity is computed from assigned probabilities, not from a right/wrong count.
Parameter count doesn't enter the perplexity formula at all.
Vocabulary size and sequence length aren't combined this way in the perplexity formula.
Why
\operatorname{PPL} = \exp\left(-\frac1n\sum_i \log P(w_i\mid w_{<i})\right) -- the exponentiated average negative log-probability assigned to each actual token.
Practice: Factoring sequences and scoring language models (5 problems · doesn’t affect your progress)
Which correctly factors P(w_1, w_2, w_3) using the chain rule, with no independence assumed?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Why
Each factor conditions on everything placed strictly before it -- the chain rule applied once per additional token, with no term dropped.
P(w_1)\,P(w_2\mid w_1)\,P(w_3\mid w_1,w_2)
A corpus has the word "cat" appearing 50 times, and "cat sat" as a consecutive pair 15 times. What is P(sat | cat) under a bigram model, as a decimal?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Why
Divide the pair count by the first-word count: 15/50 = 0.3.
Correct answer
Also accepted:
Why
A test sequence gets probabilities 0.9, 0.9, 0.9 from a model. Compute the perplexity (round to 2 decimals).
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Why
With every probability equal to 0.9, perplexity simplifies to 1/0.9 \approx 1.11.
Correct answer
Also accepted:
Why
Average the log-probabilities, negate, then exponentiate -- or notice every probability is identical, so PPL = 1/0.9 exactly.
What does a first-order Markov assumption (k=1) throw away that the full chain rule keeps?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Why
A first-order Markov assumption conditions only on the single immediately preceding token, discarding everything further back.
Every part of the history further back than one token
Model X gets perplexity 3 on Test Set A (500 tokens) and Model Y gets perplexity 3 on Test Set B (50,000 tokens). Is it valid to say the two models performed equally well on their respective test sets?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Why
Perplexity's own -1/n length normalization is specifically what makes scores across different-length test sequences comparable in the first place.
This is exactly why perplexity divides by n before exponentiating -- reread that part of the definition.
Yes -- perplexity is defined to be comparable regardless of test-set length, by construction