Adding, scaling, span

Last lesson you met the object. A word is a hundred numbers, and words used in similar ways sit near each other.

Here is something stranger. Take the vector for king, subtract the vector for man, add the vector for woman, and look at which word the result lands nearest.

That is arithmetic. On meaning. Nobody programmed a rule about royalty and gender, and yet subtracting one word and adding another moves you exactly where you would hope.

You are going to spend this lesson learning the two operations that make it work. There are only two, and you already know both of them.

Scaling a vector

Multiply a vector by a number. Every component gets multiplied, so the arrow keeps its direction and changes its length.

Drag the slider through zero and watch what happens on the way past.

Nothing dramatic happens at zero. Negative numbers are not a separate rule, they just point the arrow the other way. Two and a half is a stretch, one half is a shrink, minus one is a turn.

Keep this in mind for later. When a model multiplies an input by a weight, that number is doing exactly this. Training a model is, in large part, searching for the right numbers to scale things by.

Adding two vectors

Add two vectors by adding them slot by slot. The picture is better than the arithmetic though.

Walk along a. Then, from wherever you landed, walk along b. Where you end up is a + b.

Notice the sum is usually shorter than the two lengths added together. Two arrows at an angle partly work against each other, and only when they point exactly the same way do their lengths simply add. That observation comes back in Length, distance, normalization, where length gets a definition.

Linear combinations: scaling and adding together

Scaling and adding are the only two operations in this entire course. Put them together and you get the move that everything else is built from:

Take some of a, take some of b, add the results. Written down, that is c₁a + c₂b, and it is called a linear combination.

Below, the two dials are c₁ and c₂. There is a target. Land on it.

You can get there. It takes a bit of fiddling, but the target is reachable, and there is exactly one pair of dial settings that hits it.

Which raises the real question of this lesson. Is everything reachable?

The span of two vectors

Same two vectors, same two dials. Now every point you could possibly reach is shaded in.

The whole plane. With just two arrows and two dials, there is no point on the page you cannot land on.

That shaded set has a name. The span of a group of vectors is every point you can reach by scaling them and adding the results. Not the vectors themselves. Everything they can get to.

Now do something to it. Drag b slowly around until it points along the same line as a, and watch the shading.

It collapses. Not gradually, but at one moment: the plane becomes a line, and the target you reached a minute ago is now sitting off that line, unreachable no matter how you turn the dials. Try it. The dials cannot get you there.

Here is why. If b points along a, then b is already something you could have built out of a alone. It tells you nothing new. Adding a second dial that only repeats the first dial's job does not open up a single new direction, so you are stuck on one line forever.

Linear independence and basis

Everything above already happened on the screen. These are just the names.

Span is the reachable set, which you have been staring at.

Vectors are linearly independent when none of them is reachable from the others. Independent means each one genuinely adds a direction. The moment b lined up with a, they stopped being independent.

A basis is a minimal set that spans the space: enough vectors to reach everything, with none of them redundant. Two independent vectors are a basis for the plane. Three would be one too many, because in a plane the third is always reachable from the other two.

That last point generalises, and it is worth holding onto. In a space of n dimensions, any n + 1 vectors are guaranteed to have a redundant member. You never have to check.

Why stacked linear layers collapse into one

Here is the payoff, and it settles something the warm-up course left hanging.

A neural network layer multiplies the input by a matrix. Stack two layers and the input gets multiplied by one matrix, then by another. It seems obvious that two layers can do more than one.

They cannot. Run this.

They agree to every decimal place, and not by luck. W2 @ W1 is itself just a matrix, so whatever two stacked linear layers can compute, some single layer computes too. Ten of them would collapse the same way. Depth without something else in between buys you nothing at all.

That is a span argument. The reachable set of a linear layer is fixed by its structure, and stacking more of them does not enlarge it, exactly as adding a redundant b did not enlarge the shaded region.

The missing ingredient is a small non-straight step between the layers, called an activation. Below, the same network trains with activations on and off. Off, the boundary is always one straight line no matter how many layers you add.

You will build activations properly in Deep Neural Networks. For now it is enough to know why they have to exist: without them, depth is an illusion.

The word analogy as a linear combination

Now the analogy is readable. Change any of the three words.

Subtracting man and adding woman is a linear combination with coefficients of one, minus one and one. It works because directions in this space carry meaning. Roughly, one direction tracks gender and another tracks royalty, so moving along one while holding the other lands you on queen.

Now try brother - man + woman. You would expect sister, and you get daughter instead.

Nothing is broken. These directions are approximate, learned from how words happen to co-occur in text, not carved out exactly. brother sits closer to the family cluster than to a clean gender axis, so the arithmetic slides sideways. Treat these analogies as evidence that the space has structure, not as proof that it is exact.

One thing is still missing, and it is the same thing that was missing at the end of What a vector actually is. Every time the word nearest appeared here, it went undefined. You have been trusting a notion of distance that has not been written down once.

That is next.