The scale problem, and the trade that solves it
You have embeddings. Now you need to find the nearest ones, fast.
The honest approach is brute force: compare the query vector against every vector you have, keep the closest. It's exact, trivially correct, and about fifteen lines of code.
It's also completely fine at small scale. At 10,000 chunks, a numpy array and a loop will genuinely do — this deserves saying, because a lot of teams install a distributed vector database to search a corpus that fits in RAM.
At 10 million chunks × 1,536 dimensions, every query becomes billions of multiplications. Now it's a problem.
ANN: approximate nearest neighbour
A vector database — Pinecone, Weaviate, Qdrant, pgvector, FAISS — exists to make that fast, and it does it by making one specific trade:
It stops looking for the exact nearest neighbours and starts looking for almost certainly the nearest neighbours.
The dominant index is HNSW, and the idea is elegant. Build a layered graph over your vectors. The top layer has few nodes with long-range links; lower layers get denser and more local. A search drops in at the top, greedily hops toward the query, then descends a layer and refines.
It's how you find a house without surveying every house in the country: take the motorway, then the A-road, then the street.
The recall knob
Search harder (HNSW exposes this as efSearch) and you find more of the true
neighbours, more slowly. Search less and you're faster and occasionally miss
one.
95-99% recall at a fraction of the latency is the normal operating point. For RAG that's usually invisible: missing the 5th-best chunk out of 50 rarely changes the answer. Know that the knob exists, though — when retrieval quality is mysteriously mediocre, someone may have tuned it for a benchmark.