Attention Is All You Need
Lesson 2Positional embeddings & encodings
Telling the model where each token sits in the sequence
Positional embeddings, telling the model where each token sits
Where we are: about to invent attention
The whole point of this paper is a new mechanism, self-attention, a way for every word to look at every other word and decide what to pay attention to. We will build it carefully over the next few lessons.
But attention does not operate on words. It operates on numbers, specifically on vectors. Before a single dot product can run, the sentence the cat sat has to become a stack of vectors. That conversion is the job of two things working together:
- A token embedding, which captures what each word is.
- A positional embedding, which captures where each word sits.
This lesson is about that input layer, the thing attention eats. If the words "embedding vector" are already a little fuzzy, stop here, the next section is for you. Everything downstream rests on it.
What is an embedding vector?
An embedding is a list of real numbers that stands in for a discrete thing (here, a token) so a neural network can do math on it. That's it, it is a vector, a point in space.
A few questions almost everyone has at this point, answered directly:
Is an embedding computed by an algorithm, like hashing?
No. It is a lookup, not a calculation. There is a big table E with one row per vocabulary token. Token id 312 just reads row 312. The values in that table are learned during training by gradient descent, the same backprop that trains every other weight. Nobody writes a formula for what goes in a row; the model discovers it.
Is "embedding" the same as "encoding"?
Loosely people use them interchangeably, but there's a useful distinction. Encoding is any turning-into-numbers (a one-hot vector is an encoding; ASCII is an encoding). Embedding usually means a learned, dense, low-dimensional encoding where distance carries meaning. The original paper calls the position vectors "positional encodings" partly because the classic version is a fixed sin/cos formula, not learned. Same slot in the pipeline, different recipe.
What size should the vector be, and what's inside it?
The width is a hyperparameter you choose, called d_model (512 in the paper; 768 in GPT-2 small; up to ~12k in the largest models). Inside are just real numbers, learned features. No single slot has a human-assigned meaning like "slot 3 = animal-ness." Meaning is smeared across the whole vector, which is why we can only interpret embeddings by comparing them (cosine similarity), not by reading one number.
If my input sequence is 100,000 tokens long, is my embedding 100,000 × d?
This is the single most common confusion, so read slowly. The embedding table size depends on the vocabulary, not the sequence length. The table E is |V| × d (the 2017 paper used a shared byte-pair vocabulary of about 37,000 tokens with d = 512), fixed for the whole model. A specific input of length L becomes an L × d matrix by looking up L rows. So a 100,000-token sequence produces a 100,000 × d matrix, not 100,000 separate copies of the table, and d stays exactly the same no matter how long the sequence is. Sequence length and embedding width are independent dials. (The paper itself worked on sentence-length sequences; 100k is just an extreme to make the point.)
The cleanest way to feel this is to see it. Pick a token below and watch it become a vector. Notice how the dense embedding stays narrow (d) while the one-hot encoding balloons with the vocabulary, and how only the embedding lets similar words sit close together.
From token ID to vector
A tokenizer first turns text into integer ids; here each id becomes a vector, because attention runs on vectors, never on integers. Compare the two ways to make that jump: one-hot (sparse, high-dim, no semantics) vs a learned embedding (dense, low-dim, semantically clustered, the matrix E the transformer trains).
⚠️ Hand-picked 15-word vocab below. Real vocabularies in the 2017 paper were ~37,000 byte-pair tokens; we shrink it here so the cosine-similarity demo stays legible.
Pick a token
Selected: cat → vocab index 0
One-hot encoding
ℝ15 · all zeros, single 1 at index 0
Scales to |V| per token. No semantic relationship between any two tokens.
Learned embedding
ℝ4 · dense values · E[0]
Stays d ≪ |V| regardless of vocabulary size. Encodes meaning along each dim.
Nearest neighbours by cosine similarity
Only embeddings cluster like this; one-hot vectors have cosine similarity 0 between every pair.
one-hot(i) = ei ∈ { 0, 1 }|V|
embed(i) = E[i] ∈ ℝd, d ≪ |V|
| Quantity | Shape | Depends on |
|---|---|---|
| Embedding table E | |V| × d | Vocabulary size, never the input |
| One input sequence | L × d | How many tokens you fed in |
| Width d (d_model) | scalar | A fixed design choice, independent of both |
Now we have the one fact attention needs: every token is a vector of width d, and a sentence is a matrix of shape L × d. With that in hand, the original problem of this lesson finally makes sense.
The problem: attention is blind to order
Self-attention compares tokens using inner products and softmax, it treats the sequence as a set of vectors unless you tell it otherwise. Swap two rows of embeddings and, without position information, the attention pattern can be the same: the mechanism is permutation-invariant over positions. Real language is not, cat sat the is not the cat sat.
So we inject where each token lives in the sequence using positional encodings or positional embeddings: continuous vectors in ℝᵈ (same d = d_model as token embeddings), element-wise added to token embeddings. That sum is what flows into the first transformer block and, after layer updates, into attention, but the position signal itself is not query, key, or value.
The Lex AI walkthrough at
Attention is All You Need, Lex AI
shows the same pipeline: token embeddings + positional information → a matrix Z that is the true input to the stack, then separate projections build Q, K, V.
Not token IDs, continuous vectors for “place”
Token embeddings answer which symbol (lookup row i in the vocabulary table). Positional vectors answer which slot in the sequence (0, 1, 2, …).
- You could imagine a discrete position index, but what the network consumes is still a vector: either a fixed formula (sin/cos per dimension) or a learned row from a position table.
- Every component of that vector is a real number, so position is represented in continuous space, not as a single categorical ID passed through the model the same way as a token ID before embedding.
That matters for relative structure: with sinusoidal encodings especially, the same wave pattern at different pos shifts phase; linear layers can form combinations that behave like offsets between positions (the original paper motivated this). So “position” is not just a one-hot slot, it is a dense code the model can rotate and mix.
What positional encodings are not
| Object | Role |
|---|---|
| Token embedding | Meaning of the vocabulary item (learned lookup E). |
| Positional encoding / embedding | Place in the sequence, sinusoidal or learned vector, same length d_model. |
| Q, K, V | Later: linear projections of the current layer input for attention, recomputed every layer and head. |
Positional vectors are not Q, K, or V. They are added into the residual stream (at least at the bottom); Q/K/V are derived from that stream when attention runs. | |
The fusion step: add, then attend
For each position pos:
- x_pos, token embedding (from E).
- p_pos, positional encoding vector (sinusoidal or learned).
- z_pos, input to the first sublayer; after layer norm / blocks, later tensors still carry mixed semantic + positional information.
Before the first attention layer
Z = token embeddings + positional encodings → then Linear maps produce Q, K, V
So positional information enters before Q/K/V exist. The attention sublayer sees a representation that already encodes both “what token” and “where.”
Tap through the sentence below to watch the two halves fuse. The token row depends only on the word, the positional row depends only on the slot, and their sum z is what flows into the stack. The heatmap underneath is the full positional table: slow waves on the left, fast waves on the right, a unique fingerprint per row.
Positional encoding, made visible
A token embedding says what the word is. It says nothing about where the word sits. The positional vector fills that gap, and we simply add the two. Tap a word to watch the fusion.
The sequence
every vector here lives in ℝ16, same width, so addition just works
The positional table, as a heatmap
Each row is one position; each column is one dimension. Colour = the sin/cos value (teal +1 ··· rose −1). Left columns are slow waves (they barely change down the rows); right columns are fast waves. Together they give every position a unique fingerprint.
The highlighted row is pos 1, the same vector plotted as cells in the fusion above.
PE(pos, 2i) = sin(pos / 100002i/d) · PE(pos, 2i+1) = cos(pos / 100002i/d)
Sinusoidal positional encoding (sine & cosine waves)
Vaswani et al. (2017) defined fixed (non-learned) encodings using sines and cosines at different frequencies across dimensions. For position pos and dimension index i (paired into even/odd channels):
Intuition (high level):
- Low indices i → slow waves along position (smooth across long spans).
- High i → fast waves (finer local distinction between nearby positions).
- Each position gets a unique d_model-dimensional pattern of sines and cosines, a continuous “fingerprint” of place.
- Because sin and cos satisfy shift identities, the model can linearly recover relative relationships (e.g. differences in pos) from these features, without storing position as a single discrete number inside the vector.
These values are precomputed from pos and i; they are not token weights from the vocabulary table. The Lex AI material describes the same idea: mathematical waves of different frequencies added to each embedding dimension so the model gains order awareness without giving up parallel attention over positions.
Learned positional embeddings
Many models (e.g. GPT-2/3/4, BERT-style) use learned vectors P with shape (L_max, d_model): position pos selects row pos, still a continuous vector, still added to x_pos, but trained end-to-end instead of fixed sines/cosines.
| Sinusoidal (fixed) | Learned | |
|---|---|---|
| Parameters | None (formula only) | L_max × d_model trainable |
| Length generalization | Often better beyond train length (relative structure) | Strong when inference length ≤ L_max |
| Implementation | Slightly more code | Simple Embedding(max_len, d_model) |
Whether fixed or learned, the interface is the same: same width as token embeddings, addition, then the stack.
Why addition (not concatenation in the classic design)?
The original design adds so the dimension stays d_model and a single stream carries semantics + place. (Other architectures concatenate or fuse differently; RoPE, ALiBi, etc. change how position is injected, same underlying problem: break permutation symmetry of attention.)
Takeaway
- Attention is order-blind without position; positional encodings fix that.
- They are continuous d_model vectors (waves or learned rows), not token IDs, not the embedding matrix E, not Q/K/V.
- Token embedding + positional vector → Z, and Z is what the transformer starts from before projections build Q, K, V for the attention sublayer.
Next: how Q, K, V mix information, attention.
Check your understanding
1 / 7Why does the transformer architecture need explicit positional encodings?
Practice it yourself
Build the input layer attention eats — fuse position in, see why it's needed, and assemble Z. Real Python, hidden tests, no setup.