Meaning gets an address
Last topic left us with a problem. Your text is now a list of ID numbers —
token 2543, token 464. Those IDs are arbitrary. Token 2543 isn't bigger
than token 464, or more positive, or more anything. It's a name badge, not a
measurement.
A model that only had name badges could never notice that "cat" and "kitten" have something to do with each other. So we need to replace each badge with something richer: an embedding.
The city of words
Picture a city where every word lives at an address, and the streets are organised by meaning.
Walk to the cat district and you'll find kitten next door, feline across
the street, and dog a couple of blocks away — different, but clearly the same
side of town. Tuesday is miles away in another neighbourhood entirely.
That's an embedding space. Every token is a point, and distance means dissimilarity. Things that show up in similar situations end up near each other.
The only difference from a real city: a real city has 2 dimensions (north-south, east-west). An embedding space has hundreds or thousands — 768, 1536, 3072. Every token is a long list of numbers, one per dimension:
"cat" → [0.21, -1.04, 0.88, 0.02, ... ] ← ~1,536 numbers
You can't picture 1,536 dimensions. Nobody can. But you don't need to — the maths of "how far apart are these two points" works identically whether there are 2 dimensions or 2,000. Trust the city analogy and let the arithmetic handle the rest.
Nobody assigned these addresses
There's no dictionary in here. No human labelled cat as "animal, furry,
domestic". The addresses were learned — nudged into place over billions of
training examples by one relentless piece of feedback: words that appear in
similar contexts should live near each other.
Do that long enough and structure falls out on its own. Animals cluster. Countries cluster. Verbs of motion cluster. Not because anyone drew the neighbourhoods, but because language itself has that shape, and the model is tracing it.