Lesson 0002
You already have a feel for linear regression — one straight line, fitted by sliding two knobs. This lesson builds a feel for two more model families the same way: play first, name it after. (One of them, softmax regression, turns out to secretly be that same line — just run several times over.)
Before you've seen a single row of data, STOR 323 asks you to commit to a model family — linear regression, softmax regression, kNN — and only then hand it real examples to fit. That should feel uncomfortable. Isn't picking the family supposed to come after you've looked at the data, not before? What if you commit to the wrong one?
Hold onto that discomfort — it's the right instinct, and this lesson resolves it properly in a minute. For now, notice what you can already answer with zero data in hand: what shape does the correct answer have to take? A house price is a number. A digit is one of ten unordered categories. That's a fact about the problem itself, not about the data — and it already rules out most of the choice before you've collected a single example.
No data collected yet in either case — just the problem statement.
Five more problems, zero data attached to any of them. For each, pick the shape a correct answer has to take — a plain number, a probability between 0 and 1, or one of several unordered categories.
Not the output shape — that's fixed by the problem, as you just saw, not chosen by the model family. What differs is something else entirely: how much of the training data each family keeps around after it's done training. That's a separate question from output shape, and it's the one new idea the rest of this lesson is about.
Before picking any model family, the slides insist on one rule: figure out the shape of a correct answer before you go looking for a family to produce it. Think of a kids' shape-sorter toy — round hole, star hole, square hole. You look at your block's shape first, then find the matching hole. Nobody jams a star block through the round hole and hopes for the best.
That's the whole reason softmax regression exists as its own family instead of just reusing linear regression: a model family isn't only a formula — it's a formula plus a promise about what shape of answer it's allowed to produce. Change the shape you need, and you've changed which family even qualifies, not just tweaked the one you already had. Get the shape right first, and the right family follows on its own — never the other way around.
First, the bridge from what you already trust. You've already met β₀ + xᵀβ — that's linear regression's whole formula. Softmax regression doesn't throw that away; it just runs ten copies of it side by side, one per digit, each with its own coefficients: zk(x) = βk0 + xᵀβk for each digit k. Same formula, same fitting machinery — just ten of them instead of one, each producing its own raw score for its own digit.
So where does the name come from? A plain max would just hand the single largest of those ten scores 100% of the vote and give everyone else 0% — a hard, all-or-nothing cutoff (that's actually what argmax already does, further back in lesson 1). The trouble with a hard cutoff: nudge a score slightly and the winner either doesn't move at all, or flips instantly — there's no "give" to learn from. Softmax is the soft, smoothed-out version of that same idea: scores still compete for a shrinking 100%, but a close second place keeps a meaningful share instead of getting zeroed out. That smoothness is exactly what training needs — the same gradual, nudge-and-check fitting process used to find linear regression's coefficients can be used here too, which a hard max would never allow.
Every digit gets its own linear regression, run on the same x. Softmax takes all their raw scores at once and turns them into probabilities — p(7) + p(1) + p(9) = 100%, always, no matter what the raw scores were.
Linear regression predicts one number. But MNIST digit recognition isn't "how much" — it's "which one of 10 categories." Picture a talent show with judges: each contestant (each possible digit) gets a raw score reflecting how much evidence there is for them. A high score isn't a percentage yet — it's just "more evidence than the others." Softmax is the rule that turns those raw scores into percentages that always add up to 100%, so you can compare them directly.
Below are the actual raw scores from the slides for a handwritten "7" being classified — drag them and watch the probabilities update. Notice the total always lands on 100%, no matter what you set the sliders to.
Softmax regression is parametric, same family as linear regression — it just needs one set of coefficients per category instead of one overall. Ten digits means ten small coefficient vectors, but still a small, fixed, data-independent-in-size bundle of numbers. Compress, store, discard the original data.
Now for something structurally different. Imagine houses on a map, colored by neighborhood type — blue for one kind, orange for another. You want to guess the type of a brand-new address just dropped on the map (the ring outline). kNN's rule: look at its K closest existing neighbors, and go with whichever color wins the majority vote among them. No coefficients, no formula fitting — just "who's nearby, and what are they?"
Click anywhere on the map below to drop the query point, and drag K to change how many neighbors get a vote.
Unlike the two regressions above, kNN's "training" barely does anything — it just remembers the dataset (plus the choices you made: how many neighbors K, which distance measure, how ties are broken). All the real work happens at prediction time, walking through the stored examples fresh for every query. That's why it's nonparametric: nothing was compressed away.
| Model family | Fitted state | Parametric? |
|---|---|---|
| Linear regression | a handful of coefficients | Yes — compresses data away |
| Softmax regression | one coefficient vector per category | Yes — compresses data away |
| kNN | the training data itself | No — keeps everything |
Try from memory first.
An Introduction to Statistical Learning covers kNN classification in Ch. 2 and 4, and multi-class/softmax regression in Ch. 4 — the free official PDF. See RESOURCES.md for more, including StatQuest videos that walk through the same ideas visually if you want a second explanation.