Lesson 0002
You already have a feel for linear regression — one straight line, fitted by sliding two knobs. This lesson builds a feel for two more model families the same way: play first, name it after.
Short answer: the shape you're willing to consider is a decision you make before touching any data — same as choosing "only straight lines" was for linear regression. What differs is how much of the data each family keeps around after training. That split is the one new idea this lesson is really about.
Before picking any model family, the slides insist on one rule: figure out the shape of a correct answer before you go looking for a family to produce it. Think of a kids' shape-sorter toy — round hole, star hole, square hole. You look at your block's shape first, then find the matching hole. Nobody jams a star block through the round hole and hopes for the best.
That's the whole reason softmax regression exists as a separate family rather than just reusing linear regression: the output space S is part of what a model family even is (recall 𝓕 = {f : X → S} from the notation decoder) — get S right first, and the choice of family follows from it, not the other way around.
Linear regression predicts one number. But MNIST digit recognition isn't "how much" — it's "which one of 10 categories." Picture a talent show with judges: each contestant (each possible digit) gets a raw score reflecting how much evidence there is for them. A high score isn't a percentage yet — it's just "more evidence than the others." Softmax is the rule that turns those raw scores into percentages that always add up to 100%, so you can compare them directly.
Below are the actual raw scores from the slides for a handwritten "7" being classified — drag them and watch the probabilities update. Notice the total always lands on 100%, no matter what you set the sliders to.
Softmax regression is parametric, same family as linear regression — it just needs one set of coefficients per category instead of one overall. Ten digits means ten small coefficient vectors, but still a small, fixed, data-independent-in-size bundle of numbers. Compress, store, discard the original data.
Now for something structurally different. Imagine houses on a map, colored by neighborhood type — blue for one kind, orange for another. You want to guess the type of a brand-new address just dropped on the map (the ring outline). kNN's rule: look at its K closest existing neighbors, and go with whichever color wins the majority vote among them. No coefficients, no formula fitting — just "who's nearby, and what are they?"
Click anywhere on the map below to drop the query point, and drag K to change how many neighbors get a vote.
Unlike the two regressions above, kNN's "training" barely does anything — it just remembers the dataset (plus the choices you made: how many neighbors K, which distance measure, how ties are broken). All the real work happens at prediction time, walking through the stored examples fresh for every query. That's why it's nonparametric: nothing was compressed away.
| Model family | Fitted state | Parametric? |
|---|---|---|
| Linear regression | a handful of coefficients | Yes — compresses data away |
| Softmax regression | one coefficient vector per category | Yes — compresses data away |
| kNN | the training data itself | No — keeps everything |
Try from memory first.
An Introduction to Statistical Learning covers kNN classification in Ch. 2 and 4, and multi-class/softmax regression in Ch. 4 — the free official PDF. See RESOURCES.md for more, including StatQuest videos that walk through the same ideas visually if you want a second explanation.