What Are Embeddings? King − Man + Woman, Tested for Real
In Putting It Together, step 2 was "each token id picks one row of the embedding table: 248,320 rows of 2,048 numbers". I called that row the token's vector and moved on. This article is about what those numbers mean, and why they're one of the most useful ideas in all of AI.
The idea
An embedding turns something (a word, a sentence, an image) into a list of numbers, a vector, so that things with similar meanings get similar numbers. "Cat" and "dog" end up near each other; "cat" and "keyboard" end up far apart.
Nobody writes these numbers by hand. Like every weight in this series, they start random, and training moves them until words used in similar contexts sit close together. "Similar" is measured with cosine similarity: how much two vectors point in the same direction. 1.0 means the same direction; 0 means unrelated.
You can see it in the map below (part 2): 54 words, each a list of 50 numbers, squashed onto a flat picture. Countries cluster with countries, numbers with numbers, animals with animals. And when I ask for the closest words to "salvador", the real vectors answer: guatemala, honduras, peru, nicaragua.
The party trick: king − man + woman
The most famous demo of embeddings comes from the word2vec research of 2013: take the vector for king, subtract man, add woman, and the closest word to the result is queen. The idea is that "royalty" and "gender" become directions in the space, and you can move along them with arithmetic.
I tested it on two kinds of real vectors.
Classic word vectors: GloVe (Stanford, 2014; 400,000 words × 50 numbers, trained on Wikipedia and news text):
| Arithmetic | Closest word | Then |
|---|---|---|
| king − man + woman | queen (0.85) | throne, prince |
| walked − walk + swim | swam (0.80) | rowed, raced |
| bigger − big + small | larger (0.92) | smaller (0.89), large |
| Paris − France + Spain | "aires" (0.84), "buenos" (0.84) | madrid third (0.81) |
It works, mostly, and the failures are interesting. "Bigger − big + small" lands on larger before smaller: the "comparative" direction is stronger than "small". And Paris − France + Spain gives Buenos Aires before Madrid: a Spanish-speaking capital, just the wrong one. (GloVe splits "Buenos Aires" into two words, so they both appear.)
A modern LLM's embedding table: Qwen2.5-0.5B (151,936 rows × 896 numbers, the first step of the model from the attention article):
| Arithmetic | Closest token | Then |
|---|---|---|
| king − man + woman | " queen" (0.66) | "King", " kings" |
| bigger − big + small | " smaller" (0.76) | " larger" |
| walked − walk + swim | " Swim" (0.64) | " swimming", " swims"… and 游泳 (Chinese for "swim") |
The LLM table also finds queen, and gets smaller right where GloVe didn't. But it can't produce swam: its neighbours are spellings and forms of "swim", including the Chinese word. And I couldn't test Spanish at all: "rey" and "reina" aren't single tokens in Qwen's vocabulary, they're split into pieces, exactly as the tokenizer article showed for other languages. An embedding table only has rows for whole tokens.
The lesson: the analogy trick is real, but it's a side effect, not a guarantee. Use it to understand embeddings, not to build on.
Part 1 runs the arithmetic in your browser, on 2,025 real GloVe vectors. Try "paris − france + japan".
Where embeddings become useful: sentences
Single words are a demo. The useful version embeds a whole sentence into one vector, with a model trained for exactly that. I used EmbeddingGemma (768 numbers per sentence) and the sentence from the tokenizer article, in four languages:
| English | Spanish | Russian | Chinese | |
|---|---|---|---|---|
| English: The children went to school early because it was raining. | 1.00 | 0.91 | 0.89 | 0.89 |
| Spanish | 0.91 | 1.00 | 0.97 | 0.96 |
| Russian | 0.89 | 0.97 | 1.00 | 0.98 |
| Chinese | 0.89 | 0.96 | 0.98 | 1.00 |
Same meaning, 0.89 to 0.98, even though a tokenizer cuts these four sentences into completely different pieces (in Qwen3.5's: 11 tokens in English, 15 in Russian, 9 in Chinese). And an English paraphrase that shares no words with the original, "The kids headed to class ahead of time since it was wet outside", scores 0.90. Unrelated sentences (about small models, or the price of coffee) score 0.57 to 0.69.
Two honest notes. Similarity never drops to zero for real sentences; what matters is the gap between related and unrelated. And curiously, Spanish, Russian and Chinese sit closer to each other (0.96–0.98) than to the English original (0.89–0.91). I can't tell you why from this test.
This is what powers semantic search: embed every document once, embed the question, return the documents whose vectors are closest. A Spanish question can find an English answer, because the vectors meet in meaning, not in words. It's the "library" half of the design in my offline tutor series, and the "R" in RAG.
Putting it with the rest of the series
- A token is a piece of text (tokenizers).
- Its embedding is a row of numbers, a tensor row, learned during training (weights).
- Attention lets those vectors look at each other and mix (attention).
- Perceptron layers transform them (perceptrons).
- At the end, the same kind of vectors are compared against the whole vocabulary to pick the next word (putting it together).
Embeddings are the language all those parts speak: meaning as a position in space.
How much to trust this
- A handful of analogies and one set of sentences. Enough to show the behaviour, not to rank models.
- The map is a 2-D shadow of 50 dimensions. Closeness on the picture is approximate; the "closest words" lists are computed in all 50.
- Different models, different spaces. GloVe, Qwen's table and EmbeddingGemma were each trained for different jobs, and their vectors can't be compared with each other, only within the same model.
The one-line mental model
An embedding is meaning turned into a position: similar things land close together, so you can search, compare and even do a little arithmetic with words, in any language the model has learned.