Dot product, angle, projection

Here is a grid. Each cell scores one word against another.

Thirty-six numbers, and every one of them was produced by the same operation. Not thirty-six rules. One operation, run once per pair.

Learn that operation and you have the thing that a neural network layer does, that a search engine does, and that every attention score in every transformer is made of. It is also the number Length, distance, normalization kept quoting as an angle without ever computing.

Computing the dot product

Take two vectors. Multiply them slot by slot, then add up the results.

That is the whole calculation. One number out, no matter how many slots went in.

If it feels arbitrary, good. Nothing about "multiply pairwise and add" explains why the answer would mean anything. Sorting that out is the rest of the lesson.

The dot product as a projection

Point a light straight down onto vector b and look at the shadow a casts along it. Measure that shadow, multiply it by how long b is, and you get a number.

That should have nothing to do with multiplying slots and adding. Watch both at once.

Drag either arrow anywhere you like. The recipe's total and the shadow's measurement are computed separately, from scratch, and they never come apart.

That is the fact worth carrying out of this lesson. The arithmetic version and the geometric version are one number wearing two outfits, so you can compute it the easy way and think about it the useful way.

The green bar is the shadow, and it is the part that carries the meaning. It measures how much of a points along b.

What the sign of the dot product means

Since the shadow can fall either way along b, the score has a sign, and each case says something specific.

Swing a slowly from pointing with b to pointing against it.

• Positive: they lean the same way. The more they agree, the bigger it gets.
• Zero: exactly perpendicular. a casts no shadow on b at all.
• Negative: they lean apart by more than a right angle. The shadow falls behind the origin.

That middle case earns its own word. Two vectors whose dot product is zero are called orthogonal, which is the precise version of "these two have nothing to do with each other." It comes back later in this course, and it is the reason models sometimes want directions that do not interfere.

Why vector length distorts the dot product

Now a problem, and it is one you have been quietly walking past since Course 0 — Warm-Up.

Keep b pointing exactly where it points, and just make it longer.

Push the slider. The angle does not move at all. The score climbs anyway.

Nothing became a better match there. b just got bigger. The raw dot product mixes together two different things: how well the arrows agree, and how long they happen to be. For a match score, only the first one was wanted.

This is the same problem from the other side, when the ruler kept picking whichever word had the biggest vector.

Cosine similarity: dividing by both lengths

Divide by both lengths. That is it.

The result is called the cosine of the angle between them, and it is trapped between -1 and 1. One means pointing identically, zero means perpendicular, minus one means dead opposite. Stretch either arrow as much as you like and this number does not flinch, because both lengths were divided out.

So the two scores answer two questions:

• Dot product: how much do these agree, counting size?
• Cosine: how much do these agree, ignoring size?

And now the loose end from Length, distance, normalization can be tied. That lesson showed that once every vector has length one, ranking by distance and ranking by direction give the same order, and promised the reason later. Here it is: if both vectors already have length one, dividing by their lengths divides by one, so the cosine and the dot product are the same number. That is why production retrieval normalizes everything once at the door and then just uses dot products. It gets the angle for free.

In code

The last two lines print the same number, which is the whole argument of the previous section in two lines of code.

Cosine similarity on word vectors

Same six words, with the division now available.

On the raw scores, some words dominate their whole row simply by being longer. Divide by both lengths and that flattens out, leaving the pairs that genuinely point the same way.

Two places you will meet this grid again:

A neural network layer. A layer holds a stack of learned pattern vectors and scores its input against every one of them. One dot product per pattern, all at once. Course 0 showed you this as W @ x without saying that every entry of the output is one of these.

Attention. A transformer scores every token against every other token to decide what to pay attention to, and each of those scores is a dot product between two vectors. It does more afterwards, and that is a whole course later in PROF, Attention Is All You Need. But the grid at its heart is the one above.

That stack of pattern vectors, and what it means to apply a whole grid of numbers to a vector at once, is where this course goes next. It turns out a matrix is not a grid of numbers at all. It is a function.