Length, distance, normalization

Twice now you have been handed a list of nearest words and asked to accept it. What a vector actually is said similar words sit close together. Adding, scaling, span said the analogy lands nearest queen.

Neither lesson ever said what near means. Here is why that was worth flagging.

Both lists come from the same numbers, computed two different ways, and they disagree about first place. One says the nearest thing to Germany is Berlin. The other says it is Italy.

Neither is a bug. They are answering different questions, and by the end of this lesson you will be able to say which question you want answered.

Vector length: the L2 norm

Start with the simplest version of the question. A vector points from the origin to somewhere. How far is that?

You already know this one, from a triangle.

The two slots are the sides. The vector is the hypotenuse. Square the sides, add them, take the square root.

Now the part that matters more than the formula. Nothing in that recipe cares how many slots there are. Three slots, square all three and add. A hundred slots, square all hundred and add.

You settled this when the arrow gave up at four dimensions and the list did not. The picture stops. The arithmetic does not.

Distance as the length of the difference

Distance is not a new idea. It is the length of the gap.

Subtract one vector from the other, which you did when you first added and scaled them, then measure how long the result is.

Drag either point and watch. This is the ordinary sense of distance, the one a ruler gives you, and it is what the left-hand list at the top of this lesson used.

The L1 norm: distance in city blocks

Before the hard case, here is an easy one that makes the same point.

You are in a city laid out in blocks. How far is it from here to there? A helicopter takes the straight line. A taxi cannot drive through buildings, so it goes across and then up.

Both numbers are correct. The straight line is the L2 distance, the one from the triangle. The path along the streets is the L1 distance, and you get it by adding up the sizes of the gaps in each slot with no squaring at all.

L1 is not a worse L2. If you are dispatching taxis, L1 is the right answer and L2 is a fantasy.

Hold onto the shape of that: two defensible answers, and the task decides.

Normalising to a unit vector

One more tool before the hard case.

Take any vector and divide it by its own length. Every slot shrinks by the same factor, so the arrow keeps pointing exactly where it pointed, and its length becomes one. That is normalizing, and the result is called a unit vector.

Press the button. Both arrows jump onto the dashed circle and stay pointing where they were.

Something was destroyed there, and it is worth naming. Normalizing throws away magnitude and keeps direction. Whether that is vandalism or exactly what you wanted depends entirely on what the magnitude meant.

When distance and cosine disagree

Now the case the lesson has been building toward.

Below there is a query and two candidates. A is close by but points off at an angle. B is further away but points almost exactly where the query points.

The ruler picks whichever candidate is physically closest. Direction ignores distance entirely and picks whichever one points most similarly.

They disagree constantly, and now the Germany result makes sense. In the numbers a language model learned, germany and berlin sit close together because those two words turn up in the same sentences all the time. But germany and italy point in nearly the same direction, because they play the same role in language: both are countries, used the same way, surrounded by the same kinds of words.

So the two lists are answering two different questions:

• By the ruler: what is most strongly associated with Germany? Berlin.
• By direction: what is most like Germany, in kind? Italy.

Search engines and recommendation systems care overwhelmingly about the second question. That is not an arbitrary preference. In real embeddings the length of a vector tends to track how often a word appeared rather than what it means, so a ruler-based search quietly rewards popularity. Direction ignores that.

Normalise once, then use the dot product

Here is the move that follows from everything above, and it is one line of code.

a and b point the same way and differ only in length, so the ruler calls them far apart while direction calls them identical. Normalize both and the difference disappears, because it was never about meaning in the first place.

That is why nearly every vector database and retrieval system normalizes everything on the way in. Once every vector has length one, the ruler and direction rank results in the same order, and you no longer have to keep choosing. You make the decision once, at the door.

There is a reason those two rankings coincide on unit vectors, and it is a good one. It needs a tool you meet next: the dot product.

Norms in kNN and weight decay

Nearest neighbours. A kNN classifier labels a new point by looking at the closest examples it has seen. Every word in that sentence after "closest" depends on which distance you picked, so the choice is part of the model, not a detail underneath it.

Weight decay. Training a model often adds a penalty for large weights, and that penalty is a norm. Penalise the L2 norm and weights shrink smoothly toward zero. Penalise the L1 norm and many weights become exactly zero instead, which is a genuinely different behaviour from the same idea measured differently. Later, in Foundations of Regression, you will meet both properly.

Comparing neighbours by distance and cosine

Try other words. Roughly a fifth of the ones shipped here disagree about first place.

You can now say what each column means, which is more than you could when this page loaded.

One thread is still hanging. All lesson, direction has been measured as an angle on a screen and never as a number you could compute. There is a single operation that produces that number, and it turns out to be the most important operation in all of deep learning.

That is Dot product, angle, projection.