Recipe

Search a gallery of embeddings for nearest neighbors

Use when: you have a query embedding and a gallery of precomputed embeddings, and must decide HOW to search

Goal: the correct (or a known-accurate approximate) top-k neighbor set, using a strategy sized to the gallery

Taught in Retrieval pipelines and why generic embeddings win

1. 1

   Use cosine similarity directly. Or normalize every embedding to unit length first, and use Euclidean distance instead. Section 2 showed these give the identical ranking.

   Why: The encoder's training optimizes direction, not raw length. Mismatching the measure to that silently reintroduces the exact ranking distortion Section 2's worked example demonstrated.

2. 2

   Compare the query embedding to every single gallery embedding, exactly, with no shortcuts, and keep the top k scores.

   Example

   A gallery of 9 embeddings (this chapter's own gallery widget) takes exactly 9 checks per query. That's small enough that brute force is not just correct -- it's already fast.

   Why: This guarantees the TRUE top-k neighbors, with zero approximation error, since nothing about the search is skipped.

3. 3

   Check brute-force search's cost: roughly, one check per gallery image, per query. Weigh that against how large the gallery actually is, and how quickly an answer is needed.

   • the gallery is small enough, or the latency budget loose enough, that comparing every embedding stays fast — stop here. Brute force is the answer -- reaching for anything more complex would be unearned complexity.
   • the gallery is large enough (many millions of images) that comparing every single embedding, for every query, is too slow — continue to the next step.

   Why: An approximate index (the next step) trades away an exactness guarantee for speed. That trade is only worth making once brute force has actually become the slow point, not by default.

4. 4

   Build a search structure over the gallery that organizes embeddings. Most of them then never need to be individually compared against a given query. Do this by partitioning the space. Or build a graph that lets search jump toward the right neighborhood, instead of scanning every point. This trades a small, tunable chance of missing the true nearest neighbor, for a large speed gain.

   Why: Brute-force search's own cost scales directly with gallery size. Once the gallery is large enough, it becomes the actual slow point in the whole pipeline. An index is a deliberate, named trade: a small amount of accuracy, for a large amount of speed. It is not a free upgrade.

5. 5

   Take a held-out sample of queries. Compare the approximate index's top-k results against the true top-k from brute-force search, over that same gallery. Record how often -- and by how much -- they disagree.

   Why: An approximate method's entire value depends on knowing its actual error rate. Shipping one unchecked means an unmeasured, unknown accuracy/speed trade-off. That defeats the point of choosing the trade deliberately, back in Step 4.

6. 6

   Hand back the k highest-similarity gallery items found, from whichever of Steps 2 or 4 you used.

   Why: This is the same output the image-retrieval recipe's own last step consumes. This recipe supplies the ranking; that one supplies the encoder and the gallery.