This chapter closes on two ideas. The first turns a decoder-only
Transformer's raw output into the actual next-token distribution -- the
thing Section 2 defined a language model to be. The second turns it
into one number: how wrong a prediction was. Both are deep-learning's
own machinery. This section recalls them at working depth, because
Chapter 3 builds directly on the first one, sentence by sentence.
A Transformer's very last step produces one raw number per vocabulary word, called a logit. Think of it as a raw score. Bigger means "I favor this word more." But the numbers themselves can be negative, huge, or tiny. Nothing forces them to behave like percentages yet. Softmax is the fixed conversion that turns that messy pile of scores into real shares: all positive, all adding up to exactly 100%. The trick is e^{z_i}. In other words: Euler's number e, raised to the power of the logit. That's always positive, no matter how negative z_i starts out. So every resulting share is real and valid -- never negative or broken. Divide each one by the total. That forces everything to add up to 100\%, no more, no less. This is the exact mechanism Section 2 assumed but never spelled out. "The model's next-token distribution" is softmax, applied to that step's logits.
Cross-entropy answers a different question. Once training knows what the correct next word actually was, how do you turn "the model's guesses" and "the one right answer" into a single number? That number should say how wrong the model was. The formula collapses to something much simpler than it looks: H(y,\hat y) = -\log \hat y_{\text{true}}. Decoded: "take the probability the model gave the actual correct word, and flip the sign on its log." If the model gave the correct word a probability near 1 -- very sure, and right -- -\log of a number near 1 is near 0. Almost no cost. If the model gave the correct word a probability near 0 -- sure, and wrong -- -\log of a number near 0 shoots up to a huge cost. That gap is the whole point. Cross-entropy costs a sure mistake far more than honest doubt. That's exactly the behavior you'd want from a training signal. Now notice the last line above. Section 2's perplexity is this exact number, as an average over every token, run back through \exp. Two different-looking formulas measure the identical thing underneath -- one used to train, one used to report a score a human can read.
A decoder-only Transformer's final layer produces one raw score per vocabulary entry, called a logit. It's any real number, larger for tokens the model currently favors. Nothing guarantees it's positive, or that the scores sum to anything meaningful across the vocabulary. Softmax turns a vector of logits z = (z_1,\ldots,z_V) (one entry per vocabulary token) into a genuine probability distribution: \operatorname{softmax}(z)_i = \frac{e^{z_i}}{\sum_{j=1}^V e^{z_j}}. Every output lies in (0,1), and the whole vector sums to exactly 1. Exponentiating first guarantees every term is strictly positive, no matter how negative a logit starts. Dividing by the sum forces the total to 1. This is precisely the next-token distribution Section 2's language-model definition assumes exists at every step. The logits are the Transformer's raw output. Softmax is what turns them into a valid distribution to sample or score from.
Cross-entropy is the standard loss function used to train a language model. Take a true next token, represented as a one-hot vector y: all zeros, except a 1 at the correct token's position. Take the model's predicted distribution, \hat y = \operatorname{softmax}(z). Then: H(y, \hat y) = -\sum_i y_i \log \hat y_i = -\log \hat y_{\text{true}}, This collapses to just the negative log-probability the model assigned to the actual correct token, since every other term is multiplied by y_i=0. Cross-entropy is small when the model assigned high probability to the correct token. It grows without bound as that probability shrinks toward 0. A confidently wrong guess is punished far more than an unsure one. Training a language model means minimizing the average cross-entropy across every predicted token in the training corpus. Section 2's perplexity is nothing more than this same average, run through \exp: \operatorname{PPL}=\exp(\text{average cross-entropy}).
Softmax turns four logits into four percentages
These are this section's own four scores for the next word after 'The weather today is ___.' Read the table: e^{2.5}, e^{1.0}, e^{0.5}, e^{-0.5}, and their sum. Drag c to shift every score by the same amount -- watch the bars stay fixed, since softmax only depends on the DIFFERENCES between logits.
Loading interactive…
Cross-entropy against this section's own softmax output
The dashed outline says the true next word is 'sunny.' Read H(y, y-hat): it should match roughly 0.34, this section's own worked example. Drag the guess so nearly all the mass sits on 'sunny,' then nearly all of it on 'windy,' and watch the loss fall toward 0 and then climb toward a large number.
Loading interactive…
Worked example
Basic: one softmax probability by hand
-
1
Three candidate next tokens have logits z=(1, 0, -1). Compute \operatorname{softmax}(z) for the first token.
-
2
e^1\approx2.7183, e^0=1, e^{-1}\approx0.3679. Sum \approx4.0862.
-
3
\boxed{\operatorname{softmax}(z)_1 \approx 2.7183/4.0862 \approx 0.665} -- about 66.5% of the total probability mass.
Loading interactive…
Worked example
Typical: the full four-way softmax
-
1
Using this section's own logits z=(2.5,\ 1.0,\ 0.5,\ -0.5) for (sunny, rainy, cloudy, windy), compute the full softmax distribution.
-
2
e^{2.5}\approx12.182, e^{1.0}\approx2.718, e^{0.5}\approx1.649, e^{-0.5}\approx0.607. Sum \approx17.156.
-
3
\boxed{(0.710,\ 0.158,\ 0.096,\ 0.035)} -- sums to 1.0, matching the widget above once its shift slider c sits at 0.
Worked example
Typical: cross-entropy from that same distribution
-
1
The actual next word was "sunny." Using the softmax output from the previous example, compute the cross-entropy loss.
-
2
Cross-entropy collapses to the negative log-probability of the true token: H(y,\hat y) = -\log(0.710).
-
3
\boxed{H(y,\hat y) \approx 0.342} -- a small loss, since the model gave the correct word a comfortably high probability.
Worked example
Common trap: assuming any wrong guess costs the same
-
1
Two training steps both predict the wrong word. Step A gave the correct word probability 0.30 (wrong, but it was the second-highest guess). Step B gave the correct word probability 0.001 (sure of a different word). Compare their cross-entropy losses.
-
2
H_A = -\log(0.30) \approx 1.204. H_B = -\log(0.001) \approx 6.908 -- roughly 5.7\times larger, even though both guesses were simply "wrong."
-
3
\boxed{\text{Step B is penalized far more heavily}} -- cross-entropy doesn't just check right-or-wrong. It scales the cost by how sure the model was, exactly the property named in the technical explanation above.
Where this goes next
Chapter 3 (sampling) and Chapter 4 (pretraining)
Chapter 3 starts from exactly this section's softmax output: the next-token distribution. It builds the sampling methods (greedy, temperature, top-k, top-p) that turn it into one chosen token. Chapter 4 uses cross-entropy differently. It averages that same loss over every token in a massive training corpus. That average is the actual target a foundation model's pretraining run tries to shrink. It's the same formula this section just worked by hand -- just applied at massive scale, across huge amounts of training text.
Check your understanding (counts toward your progress)
How does this section connect cross-entropy back to Section 2's perplexity?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
They're directly related, not coincidentally similar -- one is the exponentiated average of the other.
There's no simple reciprocal relationship -- the actual relationship is exponentiation of an AVERAGE, not a division.
It's the opposite in practice -- cross-entropy is what training actually minimizes, while perplexity is typically reported as a human-readable evaluation score.
Why
\operatorname{PPL} = \exp(\text{average cross-entropy}) -- the two formulas measure the identical underlying quantity (how surprised the model was by the real text), one used as a direct training loss, the other exponentiated into a more interpretable reported score.
Section 8's technical block states this relationship directly, right after defining cross-entropy.
Three logits are (1, 1, 1) -- all equal. What is softmax(z) for the first entry, as a decimal? Round to 3 decimal places.
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Why
Equal logits produce equal exponentials, so softmax splits probability evenly: 1/3 \approx 0.333 for each of the three entries.
Correct answer
Also accepted:
Why
When every logit is identical, softmax's output is uniform across all of them.
What does cross-entropy collapse to, when the true label is a one-hot vector?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Every term for an incorrect token is multiplied by y_i=0 in the formula, dropping out entirely -- only the correct token's term survives.
Cross-entropy uses the PROBABILITY (post-softmax), not the raw logit, and specifically its logarithm.
Cross-entropy as defined here is about one specific probability (the true token's), not the spread of the whole distribution.
Why
Since a one-hot y is zero everywhere except the true token, every other term in the sum vanishes, leaving just -\log \hat y_{\text{true}} -- the negative log-probability assigned to the correct answer.
In H(y,\hat y) = -\sum_i y_i \log \hat y_i, every term where y_i = 0 vanishes -- what's left?
What is a logit?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
A logit can be any real number, including negative -- it's softmax's job to turn logits into genuine probabilities.
The chosen token is decided by a sampling strategy (Chapter 3) applied AFTER softmax, not the logit itself.
Training-time counts are a separate quantity from a logit, which is the model's live output score.
Why
A logit is one raw, unnormalized score per vocabulary entry -- any real number, not yet a probability -- before softmax normalizes the entire set.
Why does softmax exponentiate the logits before dividing by their sum, rather than just dividing the raw logits by their sum directly?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Speed isn't the reason -- exponentiation is what makes the output a legitimate probability distribution at all.
There's a specific mathematical necessity here: probabilities can't be negative, and raw logits can be.
Exponentiating still preserves relative differences between logits -- outputs aren't forced to be equal.
Why
A raw logit can be any real number, including negative -- exponentiating first guarantees a strictly positive value for every term, which is required for the result to be a valid probability after normalization.
A logit can be negative. Can a probability be negative?
The model assigned probability 0.8 to the correct next token. Compute the cross-entropy loss, -log(0.8). Round to 3 decimal places.
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Why
-\log(0.8) \approx 0.223 -- a small loss, since the model assigned the correct token a fairly high probability.
Correct answer
Also accepted:
Why
Cross-entropy for a one-hot label is just the negative natural log of the true token's assigned probability.
Practice: From logits to probabilities to a training loss (5 problems · doesn’t affect your progress)
What guarantee does softmax make about its output?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Why
Softmax always produces a genuine probability distribution: every entry strictly between 0 and 1, the whole vector summing to exactly 1.
Every output is between 0 and 1, and all outputs sum to exactly 1
Two logits are (2, 0). Compute softmax's output for the first entry (round to 3 decimals).
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Why
e^2\approx7.389, e^0=1; sum \approx8.389; 7.389/8.389\approx0.881.
Correct answer
Also accepted:
Why
e^2 / (e^2 + e^0).
The model assigned the true next token probability 0.5. Compute the cross-entropy loss, -log(0.5) (round to 3 decimals).
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Why
-\log(0.5) \approx 0.693.
Correct answer
Also accepted:
Why
The natural log of 0.5 is negative; negating it gives the loss.
Why does cross-entropy penalize a confidently wrong prediction more than an uncertain one?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Why
As the true token's assigned probability shrinks toward 0, -\log(p) grows without bound -- confidently wrong predictions get a far steeper penalty than merely uncertain ones.
Because -log(p) grows without bound as p shrinks toward 0
How is perplexity related to cross-entropy, according to this section?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Why
PPL = exp(average cross-entropy) -- the same underlying quantity, one form used to train, the other to report a human-readable score.
This section states the relationship directly: PPL = exp(...) of what, exactly?
Perplexity is the exponentiated average cross-entropy across every predicted token