Embeddings

Token IDs → vectors

Sources: The Llama 3 Herd of Models — Grattafiori et al., 2024; Efficient Estimation of Word Representations in Vector Space — Mikolov et al., 2013; Using the Output Embedding to Improve Language Models — Press & Wolf, 2017

We now have a list of integers. That’s still not useful for a neural network: the integers are arbitrary indices, and the network needs to understand that “king” and “queen” are related while “king” and “banana” are not. We need each integer to become an object that can carry meaning: a list of real numbers. That object is called an embeddingembeddingA dense vector representation of a token (typically d=2k–8k floats). Similar tokens get nearby vectors.See in glossary →.

A giant lookup table

The first layer of the model is, structurally, the simplest one of all: an embedding matrixembedding matrixA table with one row per vocabulary entry. Looking up a token = indexing into this matrix.See in glossary →. It is a 2-D table with one row per token in the vocabulary and one column per dimension of the model’s “hidden size”, a number conventionally written $d_{\text{model}}$ (read “d-model”).

For Llama-3-8B, the vocabulary has 128,256 entries and $d_{\text{model}} = 4096$. So the embedding matrix is 128,256 × 4096: a little over half a billion numbers, around a gigabyte all on its own. To turn a token ID into a vector, you just go to that row of the table. For example, if a tokenizer assigns “Hello” the ID 9906, its lookup returns row 9906: a vector of 4,096 floats. No arithmetic at all, just indexing.

What does an embedding mean?

A 4,096-dimensional vector is hard to picture. But the rough intuition is: each dimension represents something about the token. One dimension might roughly correspond to “is this a noun?”, another to “is this related to royalty?”, another to “is this a proper noun?”, and so on. Real embeddings don’t have such clean axes (the meaningful directions are tangled across many dimensions) but the geometric facts are striking. Some learned embedding tables do show semantic neighborhoods. The famous king − man + woman ≈ queen analogy was an empirical result from static Word2Vec-style word embeddings. Treat it as useful intuition rather than a guarantee for a modern LLM’s input embeddings: their entries are token-specific and become contextual only after the transformer layers.

The values are learned. During training the embedding matrix starts random and gets nudged so that the model is better at predicting next tokens. The objective does not directly tell the model which tokens should be close; instead, tokens that help make similar predictions can receive related training signals, and useful geometric patterns can emerge.

Try it

Below is a deterministic, fake embedding: the values are seeded from the token text so similar words won’t actually land near each other; they’ll just look like noise. But the shape is real: a token in, a row of d-many floats out. Visualized as colored bars (warm orange-red = negative, cyan = positive, brightness = magnitude, dark slate ≈ zero), this is what a token actually looks like to the rest of the model.

In the real network, the residual-stream vectors and final hidden state have $d_{\text{model}}$ floats. But not every internal vector has that exact shape: each attention head uses a smaller head dimension, and the MLP temporarily expands to a larger width. The rest of the model repeatedly transforms and combines these representations.

A common confusion: input embedding vs output projection

There is one detail worth flagging now because it confuses everyone the first time. The model also needs to produce token outputs: at the end, a vector of $d_{\text{model}}$ floats must become one score for every token in the vocabulary. It does this with an output projection, often called the LM headLM headLanguage-Model head — the final linear projection from hidden states (d_model) back to vocab size, producing logits over every token. "Head" because it sits atop the transformer stack like the head of a body; "LM" because it's the layer specialized for the language-modeling (next-token-prediction) objective.See in glossary →. Its weights are commonly stored with shape vocab × $d_{\text{model}}$.

Many models save memory by tying the input embedding and output-projection weights. Llama models do not tie them; they use separate matrices. Tying reduces parameter count, while its effect on quality depends on the architecture and training setup. We’ll come back to this when we talk about the LM head in section 9.

Embeddings give every token a vector. We now have everything we need to look at the first real operation the model does on those vectors, the one that lets information flow between tokens and is responsible for almost everything interesting an LLM does. That operation is attention, and it’s the topic of the next section.