Lesson 0005
One family at a time is a lie by omission — the field didn't arrive at today's models by picking the "best" model. It arrived by hitting a wall, backing up, and stacking. Here's that wall, and the stack.
Each picture has two kinds of points: blue and red. Your job is simple: could one straight line put all the blue points on one side and all the red points on the other?
Real-world version: each dot is one email from the training data, placed by two measurements — across: how many suspicious links it contains; up: how many of its words scream urgency. The color isn't part of the email — it's the answer tag we already know for that training example (Lesson 0001's y): a blue dot marks an email that turned out to be normal, a red dot one that turned out to be spam. The straight line you're trying to draw is the classifier.
If one straight line cannot separate the two colors, then any method limited to one straight cut will fail. More practice examples will not change that shape limit.
Now name the machinery. Every model family in this lesson enters the same pipeline from Lesson 0001:
Linear regression, kNN, perceptrons, and neural networks are different fills for the left-hand box.
Sort each item: is it a model family (a shape of candidate functions, chosen before any data), or something else?
Frank Rosenblatt wasn't trying to build a faster calculator. He was a psychologist, and the question underneath the formula below was stranger than anything in it: how little machinery does it take before learning appears? That question flipped the field's whole approach — not can we program intelligence with explicit rules?, but can a machine learn its own internal connections from experience?
Rosenblatt's idea began as a psychologist's question: what is the simplest physical system that could learn from experience? The perceptron was his answer — not a hand-programmed rule machine, but a network whose connections changed when it made mistakes.
Intelligence as instructions supplied by the programmer.
Intelligence as connections adjusted by experience.
That's the whole radical move, and it's what the formula below actually encodes: not a rule someone wrote down, but a set of weights w that starts wrong and gets corrected by its own mistakes. Lesson 0001 gave you y = wTx + b for a number. Slap a perceptron's step function on the same score and you get a class instead: ŷ = h(wTx + b). Same line underneath — just a different question asked of it.
z = -2.1 -> h(z) = 0
z = 3.4 -> h(z) = 1
The perceptron still computes the same kind of weighted score, z = wTx + b. The new piece is the threshold function h: negative scores become class 0, nonnegative scores become class 1. That one addition is the whole jump from "predict a number" to "predict a class."
No perceptron — trained any way you like, for as long as you like — can solve the XOR case above. That's not a training failure. It's a shape limit: a single straight cut simply cannot wall off two interleaved corners from the other two.
A model can only express the boundaries its own geometry allows. No amount of data, and no cleverer training, rescues the wrong family — you have to change the shape.
One straight cut fails. Stacking changes the shape: perceptrons feed their 0/1 answers into more perceptrons, so later layers can combine simple cuts into corners and curves.
Top row: the anatomy. Bottom row: the boundary that anatomy can draw. Notice the ring dataset from the exercise above — unsolvable in panel one, solved in panel three.
A perceptron's output can be another perceptron's input. That's the entire trick — nothing inside any single unit changed. In notation, an MLP is h(ℓ) = σ(W(ℓ)h(ℓ-1) + b(ℓ)): each layer transforms the previous layer's outputs.
A handful of words get thrown around as if they're synonyms. They're not — they're nested, and the pipeline from the top of this lesson already gives you the vocabulary to tell them apart:
| Term | What it actually is |
|---|---|
| ML | The field: improve performance on a task from experience (Lesson 0001's T/E/P). |
| ML pipeline | The two-phase process — D --A--> f^_D, then x --f^_D--> s --δ--> a — that applies no matter which family fills in f. |
| Model | Generic word for one candidate function f. Ambiguous on its own — always ask "family, or fitted?" |
| Neural network | One model family among several (alongside linear/softmax regression, kNN) — stacked layers instead of one line. |
Two more words belong on this list — transformer and GPT — but they need attention (literally) before they mean anything. Lesson 0007 picks up exactly there.
Answer from memory first — that's the part that makes it stick.