Section 3 ended with a wall. Sparse, word-count vectors have no notion
of which words mean similar things. This section recalls two ideas that
get past that wall: tokens and embeddings. deep-learning chapter 17
built both, and this section recalls them at working depth. That way,
Sections 5-7 can build attention on top of them, without stopping to
re-derive either one. Chapter 1 of this course goes much deeper on
tokenization specifically (byte-pair encoding, worked by hand). This
section only owes you what's needed before then.
A token, in plain terms, is the smallest chunk of text a model actually "sees." It works with one piece at a time. It's tempting to assume a token is always one whole word. It usually isn't. A tokenizer (the tool that does the splitting) often chops a long or unusual word into a few smaller pieces, called subword units. It leaves short, common words alone. Every one of those pieces gets a token ID: an ordinary whole number. That number is its address in the tokenizer's fixed list of every piece it knows how to produce (its vocabulary). "Text goes in, a list of numbers comes out" is the entire job of a tokenizer. Nothing more mysterious than that happens at this stage.
An embedding answers the next question. Once a token has an ID -- just a number, like "4217" -- how does a network turn that into something it can compute with? The answer is a giant lookup table. Picture a spreadsheet with one row per vocabulary entry: tens of thousands of rows, and a few hundred columns of numbers per row. Token ID 4217 simply is row 4217 of that table. Looking up an embedding is nothing more than reading off one row -- no arithmetic involved. What makes those rows useful isn't the lookup itself. It's how the numbers in each row got there. They were learned from a huge amount of text, adjusted so that tokens used in similar situations end up with similar rows. "Excellent" and "outstanding," which bag-of-words treated as completely unrelated (Section 3), end up close together as vectors here. That's because they tend to show up in similar sentences. That's the entire fix for Section 3's wall. Instead of "identical word or nothing," embeddings give every token a position in space. Nearby positions mean related meaning.
A token is the atomic unit of text a language model actually reads and predicts. It isn't necessarily a whole word, and it isn't a single character either. A tokenizer is the fixed procedure that splits raw text into a sequence of tokens. It maps each token to an integer token ID, from its fixed vocabulary. Modern tokenizers use subword units: word pieces shorter than most whole words, but longer than single characters. So "unbelievable" might become three tokens, "un," "believ," "able," while a common word like "the" stays one token. Chapter 1 of this course builds the algorithm (byte-pair encoding) that produces this vocabulary. This section only needs the result: text goes in, a sequence of integer token IDs comes out.
An embedding is a learned, fixed-length vector representation of a
token. It's stored as one row of a large embedding table -- a matrix
with one row per vocabulary entry. Looking up a token's embedding is a
single table lookup by its token ID. There's no computation, just
reading off the matching row. A bag-of-words vector (Section 3) is
sparse: mostly zero, one dimension per vocabulary word. An embedding is
different -- it's dense. Every one of its (typically a few hundred)
dimensions carries real information, and the vector's length is fixed,
regardless of vocabulary size. Embeddings are learned during training,
so that tokens used in similar contexts end up with similar vectors.
Geometric closeness stands in for meaning-closeness -- precisely the
property Section 3 showed sparse vectors cannot provide.
deep-learning chapter 17 derives how that training works (word2vec,
skip-gram, negative sampling). This course reuses the resulting vectors,
without re-deriving the training procedure.
A small, precomputed toy embedding space
Pick a word from the dropdown -- the highlighted points are its nearest neighbors by cosine similarity (Chapter 2 covers this measurement in full; for now, just notice that nearby points are related in meaning). Try 'chip': its neighbors are a genuine mix from two different clusters, since it sits ambiguously between them.
Loading interactive…
Worked example
Basic: counting tokens, not characters or words
-
1
A tokenizer splits "unbelievable" into three tokens: "un," "believ," "able." It splits "the" into the single token "the." How many tokens does the sentence "the unbelievable result" produce?
-
2
Count tokens per word, and add them up. "The" is 1 token. "Unbelievable" is 3 tokens. "Result" is (assume) 1 token, since it's a common whole word.
-
3
\boxed{5\text{ tokens total}} -- three separate words, but five tokens, because tokenization doesn't guarantee one token per word.
Worked example
Typical: embedding lookup is addressing, not computing
-
1
A vocabulary has 50,000 entries. The embedding table is therefore a 50{,}000\times 300 matrix (one 300-number row per vocabulary entry). The token "language" has ID 8841. What operation retrieves its embedding? Does it require any multiplication?
-
2
An embedding lookup reads off one specific row of the table, by its index -- row 8841. It returns exactly those 300 numbers, unchanged.
-
3
\boxed{\text{Read row 8841 of the table; no multiplication or other computation is involved}} -- the "learning" in embeddings happens once, during training, when those 300 numbers per row get adjusted. Using a trained embedding afterward is pure lookup.
Worked example
Recognition: which representation is this?
-
1
A word's vector has length 300. Every one of those 300 numbers is some nonzero decimal value, not a count of anything. Is this a bag-of-words vector (Section 3), or an embedding?
-
2
Bag-of-words vectors are sparse: mostly zero, with one dimension per vocabulary word. They hold literal word counts. This vector is dense instead -- every entry is meaningfully nonzero, and it's fixed at 300 dimensions, regardless of vocabulary size.
-
3
\boxed{\text{An embedding}} -- both the density and the fixed, vocabulary-independent length are the tell.
Where this goes next
Chapter 1 (tokenization) and Chapter 2 (embeddings for retrieval)
This section stays brief on purpose. Chapter 1 builds byte-pair encoding by hand -- the actual algorithm that produces a tokenizer's vocabulary. Chapter 2 builds sentence- and document-level embeddings. It also covers the cosine-similarity measurement this section's widget only gestured at. Both ideas lead to dense retrieval. Every RAG system in this course, starting Chapter 9, depends on that mechanism.
Check your understanding (counts toward your progress)
What is an embedding, as this section defines it?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
A raw occurrence count is a bag-of-words entry (Section 3), not an embedding.
A token ID is just an index -- the embedding is the learned vector looked up at that index, a separate object.
Sparse, one-dimension-per-vocabulary-word describes bag-of-words, the opposite of a dense embedding.
Why
An embedding is a dense, fixed-length, learned vector -- one row of an embedding table, indexed by a token's ID.
What property do embeddings have that a bag-of-words vector structurally cannot provide?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Document length isn't the property distinguishing the two representations here.
Storage size per entry isn't the structural property this section is about -- geometric similarity is.
300 is a common practical choice, not a mathematical requirement -- embedding dimension is a design decision.
Why
Because embeddings are learned so that similar-context tokens land near each other geometrically, they give a genuine notion of semantic similarity -- exactly what Section 3 showed bag-of-words cannot provide.
A tokenizer splits "replaying" into the two tokens "replay" and "ing", and leaves "the" and "game" as single tokens each. How many total tokens does "replaying the game" produce?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Why
"replaying" is 2 tokens, "the" is 1 token, "game" is 1 token -- 2 + 1 + 1 = 4 tokens for three words.
Correct answer
Also accepted:
Why
Count tokens per word separately, then add them up.
Looking up a token's embedding, once the embedding table is already trained, requires what operation?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Training happens once, in advance -- looking up an already-trained embedding is just reading a stored row.
That describes a similarity computation (Chapter 2), not the lookup step itself.
Averaging every row would produce one shared vector for every token, which defeats the point of a per-token lookup.
Why
Once a table is trained, retrieving a token's embedding is pure lookup -- reading a specific row by its token ID -- with no computation involved.
Distinguish what happens DURING training from what happens when you simply USE an already-trained table.
A vocabulary has 30,000 entries, and each embedding is 200-dimensional. As a plain number (rows times columns), how many total numbers does the embedding table store?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Why
30{,}000 \times 200 = 6{,}000{,}000 -- one 200-number row per vocabulary entry.
Correct answer
Also accepted:
Why
Multiply the number of vocabulary rows by the number of dimensions per row.
What is a token, precisely?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Modern tokenizers routinely split long or unusual words into multiple subword pieces -- one token per word isn't guaranteed.
Character-level tokenization exists but isn't what most modern tokenizers use -- subword tokens are usually longer than a single character.
A token is far smaller than a full sentence -- a sentence is made of many tokens.
Why
A token is the smallest unit a tokenizer's vocabulary defines -- often a subword piece shorter than a whole word and longer than a single character.
Practice: Tokens, token IDs, and embedding lookups (5 problems · doesn’t affect your progress)
A tokenizer splits "rewriting" into "re", "writ", "ing" (3 tokens) and leaves "a" and "draft" as one token each. How many tokens does "rewriting a draft" produce in total?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Why
3 (rewriting) + 1 (a) + 1 (draft) = 5 tokens.
Correct answer
Also accepted:
Why
What does a token ID identify?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Why
A token ID is an index into the tokenizer's fixed vocabulary list -- a whole number identifying which vocabulary entry this is.
A specific entry's position (row) in the tokenizer's fixed vocabulary
Which best describes an embedding table?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Why
The embedding table is a matrix, one row per vocabulary entry -- looking up a token's embedding means reading its own specific row.
A matrix with one learned row per vocabulary entry, looked up by token ID
A vocabulary has 12,000 entries and each embedding is 128-dimensional. How many total numbers does the embedding table store?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Why
12{,}000 \times 128 = 1{,}536{,}000.
Correct answer
Also accepted:
Why
Multiply vocabulary size by embedding dimension.
Two words, "joyful" and "content," have similar meanings but are spelled completely differently and never appear next to each other in a training corpus. Under a WELL-TRAINED embedding, what would you expect?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Why
Embeddings are trained so that words used in SIMILAR CONTEXTS end up close together, even without ever appearing next to each other directly or sharing any spelling.
Embedding closeness comes from similar surrounding CONTEXT across the whole corpus, not from direct adjacency or shared spelling.
Their embedding vectors would likely be close together, since embeddings are trained to place similar-context words near each other