Skip to content
Free public beta until December 4, 2026: every course is open, no card needed. After that, the first chapter of each course stays free and full access is a one-time $39 ($19 for up to 100 founding members chosen for helpful feedback). Your progress is kept. How the beta works

Chapter 0

Readiness -- from text to Transformers

A lit row of four boxes -- Text, Tokenizer, Transformer (the LLM), Text -- with a faded, dashed row of six boxes below it (retrieved knowledge, tools, an agent loop, fine-tuned weights, an eval harness, a served app) hinting at what later chapters add to the same diagram.
Four boxes are enough to build a Transformer that predicts the next token. Wiring it into something useful is the rest of this course.

A Model That Only Ever Predicts the Next Word

Every large language model this course wires into something useful is, underneath, the same object deep-learning chapters 17 through 21 already built: a Transformer trained to predict the next token given the ones before it. This chapter is not a re-teaching of that architecture. It is a fast, working-depth pass through the seven ideas the rest of this course leans on constantly -- where an LLM sits on the field's own map, language modeling as next-token prediction, why sparse word-count representations fall short, tokens and embeddings, why a fixed-size summary eventually breaks down, attention computed by hand on numbers small enough to check with a pencil, the decoder-only Transformer block every deployed model actually is, and softmax/cross-entropy one more time -- each restated here in this course's own voice and pointed at exactly where it gets used starting Chapter 01.

If a fact here feels familiar from deep-learning, that is intentional: this chapter still states it, still works a fresh numerical example, and still explains why it matters for this course's job, rather than sending you elsewhere to find it. The one genuinely new material is Section 3's primer on sparse, word-count representations (bag-of-words, TF-IDF) -- a classical idea deep-learning never needed and this course only needs at working depth, since the full treatment belongs to natural-language-processing.

Section 1 draws the map this whole field sits on and names what this course adds beyond a Transformer's own architecture. Section 2 makes "language model" precise: a probability distribution over token sequences, scored by perplexity. Section 3 is the one new primer. Section 4 recalls tokens and embeddings. Sections 5 through 7 recall recurrence, attention, and the decoder-only Transformer block -- the machinery every model in this course actually runs. Section 8 closes with softmax and cross-entropy, one more time, setting up Chapter 3's own treatment of how a model turns those probabilities into generated text. Deeper background and full derivations stay one pointer away in deep-learning; nothing in this chapter depends on you opening it.

  1. 0.1NLP, neural NLP, Transformers, and LLMs: the map ~14 min
  2. 0.2Language modeling as next-token prediction ~18 min
  3. 0.3Sparse representations recalled ~16 min
  4. 0.4Tokens and embeddings recalled ~15 min
  5. 0.5From recurrence to attention ~17 min
  6. 0.6Attention by hand ~24 min
  7. 0.7The Transformer block and decoder-only models ~20 min
  8. 0.8Softmax and cross-entropy recalled ~14 min
Start with 0.1 → Chapter quiz
Tokenization →
  • No matches.