Before embeddings (Section 4) and before attention (Sections 5-6), there
was a much simpler way to turn text into numbers. You could just count
the words. This section is a working-depth primer on that idea. More
importantly, it shows exactly where the idea runs out of road.
natural-language-processing covers the full treatment. Here, you get
enough to understand why Chapter 10 of this course still reaches for it,
inside modern hybrid search.
Bag-of-words, decoded: take a sentence, and throw every word into an imaginary bag. Only count how many of each word ended up inside. Throw away the order they went in entirely. "The cat chased the dog" and "the dog chased the cat" end up as the exact same bag: two "the"s, one each of "cat," "chased," "dog." That's the "bag" in bag-of-words. It's a real, working way to compare two pieces of text -- documents that share a lot of the same words probably are about similar things. But it already gives away something a sentence's meaning genuinely depends on.
The resulting vector is called sparse -- a word this course hasn't used before. It means "mostly empty." Picture a checklist with fifty thousand items, one line per possible word in the entire language. A single short email only checks off maybe thirty of those lines. Every other line -- tens of thousands of them -- stays at exactly zero. That's sparse: a long list, almost entirely zeros, with just a few real values scattered through it.
TF-IDF fixes one specific problem with plain counting. Some words -- like "the," "is," "a" -- appear constantly in every document. They tell you nothing about what makes any one document special. TF-IDF down-weights exactly those words automatically. Look at the formula's second piece, \log(N/n_t). Decoded: N is "how many documents total," and n_t is "how many of them contain this specific word." That piece shrinks toward zero when a word shows up in nearly every document. It grows large when a word shows up in only a few. Multiply that against the word's plain count. A word like "the" nearly vanishes from the score. A rare, topic-specific word like "perplexity" stands out instead.
Here's where this whole approach runs into a wall. Bag-of-words and TF-IDF both only know whether two documents share the exact same words, spelled the exact same way. "The movie was great" and "the film was excellent" share almost no words in common under this scheme. Yet a person reading both would say they mean nearly the same thing. Nothing about counting words can capture that "great" and "excellent" are close in meaning. There's no notion of closeness between words here at all -- only "identical word" or "completely unrelated." Fixing exactly that gap -- giving words a notion of distance and similarity -- is the entire point of the embeddings Section 4 builds next.
Bag-of-words represents a piece of text as a vector of word counts over a fixed vocabulary. Position j of the vector holds how many times vocabulary word j appeared in the document. Every other word in the vocabulary contributes a 0. A vocabulary of 50,000 words means every document is a 50,000-dimensional vector. Any single short document uses only a tiny handful of those slots -- the vector is sparse: almost every entry is exactly zero. Bag-of-words discards word order entirely. "The model is fast" and "is the model fast" produce the identical count vector. Counting "how many times does each word appear" never records where it appeared.
TF-IDF (term frequency-inverse document frequency) refines plain counting with one idea. A word's raw count matters less than how distinctively it marks this document, compared to the rest of the corpus. Take a term t in document d, drawn from a corpus of N documents, where t appears in n_t of them: \operatorname{TF\text{-}IDF}(t, d) = \underbrace{\operatorname{count}(t, d)}_{\text{term frequency}} \times \underbrace{\log\frac{N}{n_t}}_{\text{inverse document frequency}}. A word appearing in every document (n_t = N) gets \log(N/N) = 0. That zeroes out its weight entirely -- exactly right, since a word every document shares (like "the") carries no information about what makes this document distinct. A word appearing in only a few documents gets a large inverse-document-frequency multiplier instead, boosting its weight. That's exactly right too: a rare, document-specific word is precisely what should drive a search or comparison.
Sparse representations fall short in three structural ways, not just in practice. First, word order is thrown away completely. Two sentences with opposite meanings, but the same words, become indistinguishable. Second, every slot is treated as independent. "Excellent" and "outstanding" share zero vector overlap, even though they mean nearly the same thing -- there is no notion of one word being similar to another, only identical or not. Third, the vector's length is the entire vocabulary size, which grows without bound as more text is seen. Sections 4-7 of this chapter build the alternative: a dense representation (most entries nonzero, far fewer slots) learned so that similar words land near each other geometrically. That gives exactly the two properties bag-of-words and TF-IDF structurally cannot provide.
Compute one TF-IDF weight by hand
Two documents, three vocabulary words. Click a document, then a word, to see its raw count -- then check the worked example below for how that count turns into a TF-IDF weight.
Loading interactive…
Worked example
Basic: a bag-of-words vector by hand
-
1
Over the 3-word vocabulary {"the", "cat", "dog"}, write the bag-of-words vector for the sentence "the cat and the dog."
-
2
Count each vocabulary word's occurrences in the sentence. Ignore any word (like "and") that's outside the vocabulary.
-
3
\boxed{(\text{the}: 2,\ \text{cat}: 1,\ \text{dog}: 1)} -- "and" is simply dropped, since it isn't one of the three tracked dimensions.
Worked example
Typical: computing a TF-IDF weight
-
1
Use the widget's document 1 above, "the model predicts the next token." Compute \operatorname{TF\text{-}IDF}(\text{the}, \text{doc }1) across a 2-document corpus, where "the" appears in both documents (N=2, n_{\text{the}}=2).
-
2
Term frequency is the raw count from the widget: \operatorname{count} (\text{the}, \text{doc }1) = 2. Inverse document frequency: \log(N/n_t) = \log(2/2) = \log(1) = 0.
-
3
\boxed{\operatorname{TF\text{-}IDF}(\text{the},\text{doc }1) = 2\times0 = 0} -- "the" appears in every document in this corpus, so TF-IDF correctly zeroes its weight out, no matter how often it's repeated.
Worked example
Common trap: assuming similar meaning gives a similar vector
-
1
A search system, built on plain TF-IDF vectors, gets the query "car repair." It must rank two documents. Document A contains the phrase "car repair" verbatim. Document B is entirely about "automobile maintenance," and never once uses the words "car" or "repair." Which document does a TF-IDF search rank higher? Is that the right answer?
-
2
TF-IDF vectors only overlap on exact shared vocabulary words. Document A shares both query words with the query. Document B shares zero words with the query -- however relevant its topic actually is.
-
3
\boxed{\text{TF-IDF ranks Document A far higher, even though Document B may be just as relevant}} -- this is exactly the structural gap named above: no notion of word similarity, only exact overlap. Dense embeddings (Section 4) close exactly this gap.
Where this goes next
Chapter 10, hybrid retrieval
Sparse methods are not a footnote this course leaves behind. Chapter 10 builds real hybrid search systems around them. Those systems combine a sparse method (BM25, a refinement of the TF-IDF weighting derived here) with the dense retrieval Section 4 adds. The two get fused together, because each is strong exactly where the other is weak. Sparse methods excel at exact keyword and rare-term matches. Dense methods excel at meaning.
Check your understanding (counts toward your progress)
A word appears 5 times in a document, and appears in 4 of a corpus's 100 documents total. Compute its TF-IDF weight: count * log(N / n_t). Use natural log, and round your answer to 1 decimal place.
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Why
\log(100/4) = \log(25) \approx 3.219; 5 \times 3.219 \approx 16.1 -- a high weight, since the word is both frequent in this document and rare across the corpus.
Correct answer
Also accepted:
Why
First compute log(100 / 4), then multiply by the raw count of 5.
"The dog chased the cat" and "the cat chased the dog" produce identical bag-of-words vectors. Why?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Bag-of-words has no notion of meaning or real-world events -- it only counts word occurrences.
Bag-of-words treats every distinct word as its own separate dimension -- "dog" and "cat" are not merged.
Two different sentences of the same length with different words produce different vectors -- it's specifically the shared word counts here that make these two match.
Why
Both sentences share the identical multiset of words -- two "the"s, one "dog," one "cat," one "chased" -- and bag-of-words only records counts, never order, so the two vectors match exactly.
Compare exactly what each sentence's word COUNTS are, ignoring the order the words appear in.
What does "sparse" describe about a bag-of-words vector?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Sparse vectors actually have MANY dimensions (one per vocabulary word) -- "sparse" describes how few of them are nonzero, not the total count.
Bag-of-words vectors can hold whole-number counts greater than 1, not only values between 0 and 1.
Bag-of-words is deterministic -- the same document always produces the identical vector.
Why
Sparse means mostly zero: a 50,000-dimension vocabulary vector for a short document has only a handful of nonzero entries, the rest exactly zero.
Over the vocabulary ("a", "model", "language"), which bag-of-words vector correctly represents "a language model is a model"?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Bag-of-words counts every OCCURRENCE, not just whether a word appears at all -- "a" and "model" each appear twice, not once.
"is" is outside the 3-word vocabulary given, so it contributes no entry at all -- the vector stays exactly 3-dimensional.
This miscounts "a" (which appears twice, not once) and "language" (which appears once, not twice).
Why
"a" appears twice, "model" appears twice, "language" appears once -- giving (2, 2, 1) in the vocabulary's own order.
Count each vocabulary word's occurrences in the sentence, ignoring words outside the vocabulary.
What does a bag-of-words vector's position j actually hold?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Bag-of-words discards positional information entirely -- position j of the vector is a word-count slot, not a location.
That kind of learned similarity is what a dense embedding provides (Section 4), not a bag-of-words vector.
Document length isn't what a single vector position encodes.
Why
Bag-of-words is a vector of raw word counts, one dimension per vocabulary word -- position j counts how many times word j appeared, nothing about order or position in the sentence.
In a corpus of N = 10 documents, a word appears in exactly 10 of them (every single document). Compute log(N / n_t) for this word -- its inverse-document-frequency factor. (Use natural log; round to 2 decimals.)
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Why
N/n_t = 10/10 = 1, and \log(1) = 0 -- a word appearing in every document gets its TF-IDF weight zeroed out entirely, regardless of its raw count.
Correct answer
Also accepted:
Why
N and n_t are the same number here -- what does that make the ratio N/n_t?
Check your understanding (counts toward your progress)
Where does this course actually still use sparse, TF-IDF-style representations, according to this section's own forward pointer?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
The applied block names a specific, real later use -- it isn't purely historical in this course.
LLMs compute internally over dense embeddings, not sparse bag-of-words vectors.
Tokenization (Chapter 1) is a separate topic from TF-IDF-style sparse document representations.
Why
Chapter 10 builds hybrid retrieval systems that combine a sparse method (BM25, a TF-IDF refinement) with dense retrieval, fused together, since each is strong exactly where the other is weak.
"Great" and "excellent" mean nearly the same thing, but a bag-of-words search treats a document containing only "excellent" as completely unrelated to a query for "great." Why?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
The two words are close synonyms, not opposites -- the failure isn't about their actual meaning.
Vector length (equal to vocabulary size) isn't the limiting factor here -- both words would each get their own dimension just fine.
There's no deliberate spelling-correction step causing this -- it's a structural property of how bag-of-words represents words.
Why
Every distinct word gets its own independent dimension in a bag-of-words vector, with no built-in notion that two different words could mean similar things -- only exact-word overlap counts as a match at all.
Ask what bag-of-words actually compares: identical words, or something about MEANING?
A keyword search built on TF-IDF ranks a document containing the literal phrase "machine learning" far above a document about "statistical pattern recognition," even though both documents cover nearly identical material. What is the most accurate diagnosis?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
TF-IDF search has a real, named structural gap (Section 3) -- it isn't guaranteed accurate by construction.
This is exactly the general, structural gap this section names, not a one-off bug in a single implementation.
Phrase length isn't what's driving the ranking difference here -- shared vocabulary is.
Why
This is the structural gap Section 3 names directly: TF-IDF (and bag-of-words generally) has no notion of word or phrase similarity, only exact shared vocabulary -- exactly the gap dense embeddings (Section 4) exist to close.
Think about what TF-IDF vectors can and can't detect between two DIFFERENT sets of words describing the same idea.
Which of these is a genuine structural limitation of bag-of-words, not just a matter of choosing better weighting (like TF-IDF)?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
TF-IDF weighting specifically fixes over-weighting of common words -- that's a weighting problem, not a structural one.
TF-IDF specifically boosts rare words' weight -- again, a weighting fix, not structural.
Sparse vectors are usually stored efficiently (only the nonzero entries) -- raw memory isn't the core structural limitation named in this section.
Why
TF-IDF weighting can fix how much weight a word gets, but nothing about reweighting counts restores the word-order information bag-of-words throws away from the start.
Ask which limitation survives even after switching from plain counts to TF-IDF weighting.
According to this section, what does TF-IDF specifically add on top of plain bag-of-words counting?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
TF-IDF still discards word order -- that limitation isn't what it addresses.
TF-IDF has no synonym detection -- it still only responds to exact shared vocabulary.
TF-IDF vectors are still exactly as long as the vocabulary -- dimension doesn't shrink.
Why
TF-IDF's one improvement over plain counting is weighting: words common across the whole corpus get down-weighted (even to exactly zero), and words that are rare and distinctive get boosted.
Practice: Bag-of-words and TF-IDF by hand (5 problems · doesn’t affect your progress)
Over the vocabulary ("run", "jump", "swim"), which bag-of-words vector represents "run and jump and jump"?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Why
"run" appears once, "jump" appears twice, "swim" appears zero times: (1, 2, 0).
(1, 2, 0)
"Cats chase mice" and "mice chase cats" produce what relationship between their bag-of-words vectors?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Why
Bag-of-words discards word order entirely -- both sentences share the identical word counts, so their vectors are identical.
Identical vectors
A word appears in 20 of a corpus's 200 documents. Compute log(N / n_t) using natural log, rounded to 2 decimals.
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Why
\log(200/20) = \log(10) \approx 2.303.
Correct answer
Also accepted:
Why
N/n_t = 200/20 = 10; take the natural log of 10.
Which structural limitation does switching from plain bag-of-words to TF-IDF weighting NOT fix?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Why
TF-IDF only changes how counts are WEIGHTED -- it never restores the word-order information that was already thrown away at the bag-of-words step.
Word order being discarded entirely
A search for "automobile" fails to return a highly relevant document that only ever uses the word "car." What kind of representation would most directly fix this, according to this section?
Right on try — credit. Right — the answer had already been shown, so no credit this time. Next try is worth Answer shown after three tries — this one no longer earns credit. You chose to see the answer — this one earns no credit.
Why
This is exactly the structural gap Section 3 names: sparse methods have no notion of word similarity, only exact overlap -- dense embeddings are built specifically to close that gap.
A bigger vocabulary or a reweighted TF-IDF score still only responds to EXACT word matches -- what's needed is a notion of similarity between different words.
A dense embedding, where semantically similar words land near each other geometrically