Lesson 0006

How the stack learns

Stacking fixed what one straight line couldn't draw. It also quietly broke the way learning worked. This lesson is the fix for the fix.

One layer: one suspect

A single perceptron makes the final answer itself. So when it's wrong, the mistake is easy to assign: the weights that made the call are the ones that get corrected. No calculus — just nudge them toward what would have been right.

Concretely: an email with four suspicious links comes in. The perceptron says normal; the training label says spam. Every weight that pushed the score toward "normal" gets nudged the other way. Next time, that email lands closer to spam.

Stack layers, lose the suspect

Now stack. Early layers don't make the final answer — they pass intermediate signals forward, and later layers build on them. When the answer comes out wrong, the last layer made the call — but it made it out of parts the earlier layers handed it.

So which earlier weight gets corrected, and by how much?

Sharpen your pencil: can the simple nudge rule handle it alone?

Backpropagation: send the error backward

The fix needs one idea from math. A derivative answers a simple question: if I nudge this a little, how much does the output move? The chain rule strings derivatives together through chained steps, so a nudge at the start can be traced through everything downstream.

Backpropagation uses exactly that: it sends the error backward through the network and uses the chain rule to estimate each weight's share of responsibility. Every weight — even one buried layers away from the answer — gets a number: this much of the miss was yours. Then every weight adjusts by its share.

The percentages are the chain rule's output: each stage's estimated share of the miss, so even the weights far from the answer know how much to adjust.

Why the hard step had to go

One catch: blame travels along slopes. The perceptron's hard step has none — flat, flat, flat, then one instant jump. Nudge a weight and the output usually doesn't move at all… then suddenly flips. Ask "how much did the output move?" and the answer is zero, almost everywhere. Zero signal, nothing to learn from.

A smooth activation — sigmoid, tanh, ReLU — behaves like a dimmer knob instead: tiny turn, tiny change. Now every nudge answers "did that help or hurt?", and blame can flow backward through every layer. Smooth activations aren't a stylistic upgrade. They're what makes deep stacks trainable at all.

Side by side

Check for understanding