The Transformer block and decoder-only models

A Transformer block is one repeating station on a line of stations. Each station does two things, in order. First, self-attention (Section 6) lets every token gather information from every other token. Second, a feed-forward network -- an ordinary small network, the same kind deep-learning's early chapters build -- handles each token's own result a little further, on its own. Both steps are wrapped in a residual connection. In other words: "add the station's own input back onto its output." That's a backup. It guarantees information can always flow straight through a station, even if that station's own work turns out to add little. That turns out to matter a lot, once dozens of stations are stacked back to back. A full model is just many of these identical-shaped stations, chained together, each with its own separately learned settings.

Before the very first station, positional encoding stamps a pattern based on position onto each token. Attention by itself genuinely cannot tell "the cat sat" from "sat the cat." Without this step, both would produce the identical set of attention outputs, just swapped. Multi-head attention means several attention computations run side by side, inside one station, instead of just one. Each one is free to form its own sense of "what matters." It's like having several readers pay attention to the same sentence in different, useful ways at once -- instead of forcing one single reading approach on everybody.

Masking, in other words: simply zero out certain attention weights, before they're ever used. Do this by making their score negative infinity, so softmax gives them exactly zero. A causal mask blocks every token from seeing anything that comes after it. Word 5 can only look at words 1 through 5, never word 6 onward. Here's why that specific rule defines almost every LLM you've ever used. Training a model to predict the next word IS its entire job. At the actual moment it has to generate word 6 for real, word 6 doesn't exist yet to peek at. So training it to peek during practice would teach a skill it could never actually use. A decoder-only model is simply a stack of Transformer blocks that all use this causal rule, with nothing else attached. That's specifically the shape of GPT, Claude, Llama, and almost every large language model you can talk to right now. (A model built to only read, not generate -- BERT is the classic example -- drops the causal mask and looks both directions at once instead. A model built to transform one complete text into another, like a program that translates, keeps both a reading half and a writing half. This course only ever wires up the kind built to write new text.)

Two last, practical terms this chapter has now set up. Context window is the maximum number of tokens a model can hold in view at once -- think of it as a fixed-size working memory. KV caching is a bookkeeping trick. It avoids working out every earlier word's key and value again from scratch, each time a new word appears. Both terms come back in Chapter 3, once the wiring -- not just the design -- is the point.

A Transformer block wraps self-attention (Section 6) around two more pieces. Each piece is guarded by a residual connection -- add the sublayer's input back to its output, so information always has a clear path through -- and a normalization step. Self-attention runs first. Then comes a position-wise feed-forward network: an ordinary two-layer network, applied the same way, independently, to every token's own vector. A full model stacks N of these identical-shaped blocks. Each block's output matches its input in shape. That's exactly what lets block k feed block k+1, with no reshaping anywhere in the stack. Before the first block, positional encoding adds a position-dependent pattern to each token's embedding. Attention alone has no notion of token order. Without positional encoding, "the cat sat" and "sat the cat" would produce identical attention outputs. Multi-head attention runs several independent attention computations in parallel, inside each block, each with its own learned projections. That lets different heads specialize in different kinds of connections (nearby words, long-range reference), rather than forcing one single notion of what matters onto the whole layer.

Masking, from Section 6's pipeline, steps in between the score and softmax steps. A disallowed position's score gets overwritten with -\infty before softmax runs, so it receives exactly zero attention weight. A causal mask disallows every query position i from attending to any key position j>i -- future tokens. A decoder-only model stacks Transformer blocks using causal self-attention at every layer, with no separate encoder and nothing to cross-attend to. This is the design behind almost every LLM in this course. The pretraining objective is next-token prediction (Section 2): predict token t from tokens 1,\ldots,t-1 only. Training the model to peek at token t while predicting it would teach a habit it can never use at real generation time, when later tokens don't exist yet. This is one of three model families built from the same block. Encoder-only models (bidirectional, no masking) are good at understanding a complete input. Encoder-decoder models (both halves) are good at transformation tasks like translation. But every model this course wires into a system is decoder-only.

Finally, a decoder-only model's context window is the fixed maximum number of tokens it can attend over at once. KV caching is the standard technique that makes generating one token at a time fast enough, within that window. Each position's key and value vectors get computed once, and reused for every later step -- rather than recomputed from scratch at each new token.