Lesson 0001
The one diagram that everything else in this course hangs off. If you remember only one thing from Lecture 3, remember this.
STOR 323 defines machine learning operationally: performance on a task T improves through experience E, measured by performance P. For MNIST digit recognition: T = recognize a handwritten digit, E = labeled examples, P = error on new digits. Nothing mystical — just data in, better performance out.
The whole course splits into two phases that are easy to blur together if you don't separate them explicitly. Training happens once, ahead of time. Deployment happens over and over, on new inputs.
Training: a fixed learning algorithm A reads a dataset D and returns a fitted model f^D.
D is not one thing — it's a whole set of labeled pairs: D = {(x₁,y₁), …, (xₙ,yₙ)}, n examples where the true outcome y was already known at training time. That matters for what comes next.
Deployment: a new input x is predicted into a score s, then decided by a rule δ into an action a.
Here, x is a single brand-new input, arriving after training is finished — and critically, its true y is exactly what you don't know yet. That's the entire reason deployment needs a model at all: if you already knew y, there'd be nothing to predict.
"a" is easy to skim past as a formality. It helps to see it vary across problems — the action is never the score itself, it's a separate, chosen response to the score, and the same score can lead to different actions depending on who's using it.
| Problem | The score s | The action a |
|---|---|---|
| House price model | "$350,000" — a predicted number | List the house at that price? Approve a loan up to that amount? Flag it as underpriced for a buyer? Nothing happens until someone picks one. |
| Handwritten digit recognizer | "91% confident this is a 7" | Route this envelope to the bin for that zip code. Credit a bank deposit for that amount. Send it to a human if confidence is too low. |
| Medical screening | "0.62 probability of disease" | Refer for follow-up testing, or don't. |
s is always just information — a number or a probability. Nothing changes in the world because of s alone. What changes the world is a, and a always comes from a separate rule δ, chosen by a person, not computed by the model. Two systems could see the exact same score and take completely different actions, because they're running different δ.
s is the model's guess, not a fact about the world. Look what's missing if you try to fold a deployment case straight back into training:
(x, s) is an input plus the model's own opinion about it. Training on that would mean training the model on its own guesses — reinforcing whatever it already believes, right or wrong, with nothing to correct it against reality. The one thing missing is y — the actual outcome, confirmed independently of the model (e.g. the patient's diagnosis, verified later by an actual test, not by asking f^D again). Only (x, y) — never (x, s) — is a legitimate new training example, and turning deployment cases into future training data is always a deliberate, separate decision, never automatic.
Don't take the rule on faith — work it. Each case below is a real deployment moment. Decide: does it belong in D, or not? Click a verdict and see why.
A junior engineer says: "We don't have enough labeled data to retrain this quarter. But we do have a year of deployment logs — every input x, every score s the model gave it. Let's just use the model's own scores as the labels. That's a year of free training data."
The fitted model depends on the algorithm and the data. The decision rule depends on neither.
Change A, or change D, and f^D changes too — it's built from both.
δ only acts on the score s, after the model is already fixed. Changing it retrains nothing — no new data, no new algorithm, not even a new f^D.
That's why STOR 323 keeps asking "which one moved?" — mix them up and you'll retrain a model to fix what was actually a bad threshold, or leave a bad threshold in place because retraining "didn't help."
Answer from memory before checking — that effort is what makes it stick.
Primary source for this lecture's formalism: An Introduction to Statistical Learning (James, Witten, Hastie, Tibshirani) — free official PDF, Ch. 2 covers exactly this train/predict framing. See RESOURCES.md for more.