Lesson 0003
The model is done training. Now: how do you know if it's any good — and "good for what"?
You wouldn't use a bathroom scale to check someone's temperature, or a ruler to weigh a suitcase. The right measuring tool depends entirely on what kind of thing you're measuring. Model evaluation works the same way — the slides put it as: evaluation follows the object being evaluated.
| What the model outputs | Question you're asking | Right tool | What it misses |
|---|---|---|---|
| A number (house price) | How far off? | MSE | Whether the downstream action was any good |
| A score or ranking | How well-ordered? | ROC / AUC | Error costs, capacity limits |
| A yes/no decision at threshold t | Which errors happen? | Confusion matrix | Who actually bears each error |
No single metric answers every question — that's not a limitation to work around, it's the whole point of matching the tool to the object.
You already met this one. Back in lesson 0001's house-price slider, the "total error" number ticking up and down as you dragged the sliders — that was MSE the whole time. It's just the average of every dot's squared miss: predict $305k, actual sale was $300k, that one point misses by $5k, squared to 25 (in $000s²). Add up every point's squared miss, divide by how many points — that's the whole idea. Simple, but it only ever answers "how far off, on average" — it has nothing to say about what happens *after* the prediction, which is exactly where the next two tools pick up.
Picture a doctor's screening test. The model outputs a score from 0 (probably healthy) to 1 (probably sick) for each patient. But at some point, someone has to actually decide: refer this person for a follow-up, or not? That decision needs a cutoff — a threshold t. Everyone scoring at or above t gets flagged; everyone below doesn't. Move the threshold below, and drag it around:
Four things can happen to any one person, and they have names: flag someone who's actually sick — a true positive (TP), good; flag someone who's actually healthy — a false positive (FP), a wasted follow-up; miss someone who's actually sick — a false negative (FN), a missed diagnosis; correctly clear someone healthy — a true negative (TN), good. That 2×2 grid of counts is the confusion matrix. TPR = TP/(TP+FN) asks "of everyone actually sick, what fraction did we catch?" FPR = FP/(FP+TN) asks "of everyone actually healthy, what fraction did we wrongly alarm?"
Notice the little plot next to the sliders. Every time you move the threshold, you get one (FPR, TPR) point — and as you sweep the threshold across its whole range, those points trace out a curve. That curve is the ROC curve (receiver operating characteristic — an old radar-engineering name that stuck). It answers a different question than any single confusion matrix does: not "how good is this one threshold," but "how well does this model rank sick patients above healthy ones, across every possible threshold at once?"
A model that's a coin-flip would trace the diagonal dashed line — no better than guessing. A model that ranks perfectly (every sick patient scored higher than every healthy one) would hug the top-left corner. AUC (area under that curve) compresses the whole curve into one number: 0.5 is coin-flip, 1.0 is perfect ranking. It's a great "is this model any good at all" summary — but notice it never tells you which single threshold to actually deploy. That's still a separate decision.
Look back at the "what it misses" column in the first table. MSE, ROC/AUC, and the confusion matrix all describe the model — but none of them say who should bear the cost of an error, or which threshold is fair to deploy. We already worked through a concrete case of this in the notation decoder: the exact same threshold, applied to the exact same model, produced a false-alarm rate of 40% in one group and 20% in another. Equal rule ≠ equal burden — and no metric on this page can tell you that on its own. You have to go looking for it, by group.
This closes the loop from lesson 0001: f̂_D is fixed once training ends. Everything from "score" onward — which metric you check, where you set the threshold, what action follows — is still an open, ongoing choice. The model is fixed; the policy is not. The slides end Lecture 3 on two open questions worth sitting with: who chooses the rule, and using what principle? There's no formula for that — it's a judgment call, made by whoever deploys the system.
Try from memory first.
An Introduction to Statistical Learning, Ch. 4, covers classification metrics and ROC curves formally. For the fairness/group angle, see Fairness and Machine Learning — both are already in RESOURCES.md.