learning a decision rule
Lesson 0014 · Homework 1, Problem 3
Softmax, log loss, and the action after prediction
Lesson 0013 covered log loss for two classes. Here, softmax is just the high-school move of turning evidence points into percentages, then remembering that digit labels are categories, not positions on a ruler.
Summary before you start
One image becomes three separate objects.
logitsraw evidence points
softmaxpoints / total points
rulesscore truth or act
x_i
pixels
28 x 28 grayscale grid
→
z_c
logits
raw evidence
→
p_c
softmax
percent share
-log(p_true)training loss
argmax / reviewdeployment action
Do not mix these up
The same vector has two different jobs.
argmax(p)
decision rule
Pick the single most likely class when the system needs one final answer.
-log(p_y)
training signal
Score the probability assigned to the true class. It does not ask whether argmax was right.
Key contrast: a model can get argmax wrong and still have lower log loss than a clueless baseline if it put more probability on the truth than the baseline did.
Why MNIST matters
A benchmark turns opinions into comparable results.
MNIST is a fixed, shared set of 60,000+ labeled handwritten digits assembled in 1998. Its real contribution was standardization: every method ran on the same images with the same scoring, so researchers could compare SVMs, k-NN, and neural nets head to head. Like the Perceptron in 1958 and AlexNet on ImageNet in 2012, the method mattered because a shared yardstick made progress visible.
same digits, same scoring
deep CNNs + GPUs on ImageNet
Concrete before symbolic
An MNIST row is a grid first, then a vector.
A real MNIST image is a 28 x 28 grid of grayscale pixels. Below is a simplified 10 x 10 digit so the storage pattern is visible: the model receives numbers, not the word "seven."
Flattening
This is 100 numbers, not a picture. A real MNIST digit is the same idea at 28 x 28 = 784 numbers.
x_i = [255, 255, ..., 0, 0, 128, 255, 128, ...]
Row 1 first, then row 2, then row 3. For the real dataset, x_i in R^784 and y_i in {0, 1, ..., 9}.
The central trap
Digit labels are bucket names, not ruler marks.
MNIST labels look numeric, but they are categories. A 7 is not seven units more of something than a 0. That means "nearest digit" is the wrong way to measure the error.
Wrong: distance on a line
0123456789
|7 - 6| = 1
This makes 6 look "almost correct" because it is numerically near 7.
Right: separate buckets
0123456789
truth bucket = 7
The model is judged by probability on the true bucket, not by how close the label numbers are.
Classification error is not nearest-number error: if the truth is 7, guessing 6 and guessing 2 are both wrong labels. Log loss asks a better question: how much probability did the model put in the 7 bucket?
High-school math bridge
Softmax turns evidence points into percentages.
Pretend four digit buckets receive evidence points. Converting points to percentages is ordinary arithmetic: divide each bucket by the total. Softmax adds one extra step before that: it turns raw logits into positive points using e^z.
Evidence points
A3
B1
C5
D1
Percentages
A30%
B10%
C50%
D10%
log 3 → 3 points
0 → 1 point
log 5 → 5 points
0 → 1 point
Why squared error fails here
Do not regress on the digit codes.
Fitting one scalar output directly to 0-9 would silently assume a false thing: that predicting 6 when the truth is 7 is a small error, and predicting 0 when the truth is 7 is a big one. The labels are names, so the model needs one output per class.
Wrong output
one number on a fake digit ruler
Right output
ten unordered classes, ten logits
Softmax regression sidesteps the false number-line assumption by giving each class its own logit. See Lesson 0002 for the full softmax setup and the glossary's logit / argmax entries.
Try it
Move the evidence, watch the percentages change.
The sliders are raw logits. The chart is the percentage share after softmax. Raise one class and it takes a larger share of the total, but all probabilities still add to 100%.
logits
log 3, 0, log 5, 0
evidence points
3 + 1 + 5 + 1 = 10
probabilities
0.3, 0.1, 0.5, 0.1
decision
argmax = C
For Homework 1(b): the actual logits are z1 = log 2, z3 = log 4, and zero for the other 8 classes. Same denominator-then-divide procedure; different numbers and ten classes total.
Contrastive minimal pair
Log loss punishes low confidence in the truth.
Suppose the true label is A here. The model's argmax is C, so classification error says it is wrong. But its probability on A is 0.3, which is still better than a uniform four-class baseline's 0.25.
Model
-log(0.3) ~= 1.204
Uniform baseline
-log(0.25) ~= 1.386
Deployment policy
The probability vector is not the action rule.
Homework 1(d) adds a policy on top of the model: auto-enter a digit only when max_c p_c(x) >= 0.80; otherwise route to a person. The demo's max probability is 0.5, so this image goes to human review even though the model has an argmax opinion.
max p = 0.50
review
0.80 auto-enter threshold
This is the same distinction Lesson 0012 drew for advising thresholds: f^D produces scores or probabilities; delta is the separate rule that decides what to actually do.
Same structure, new context
Evaluate on new writers, not just new images.
"Represents new writers" is the same held-out design question as Problem 1(d) in Lesson 0012. If the same person contributes images to both train and holdout, the model may be recognizing that writer's style instead of generalizing to a person it has never seen.
Wrong
same writer in train + test
Right
hold out entire writers
Too high, and the automation is not automating much.
This is the subset nobody double-checks.
Data quantity trap
Thousands of images is not the same as enough writers.
Lesson 0012's sample-size discussion applies here with the count relabeled. A classical rule like "32 is enough" says nothing about whether the model has seen enough distinct handwriting styles to generalize to a writer it has never encountered.
Same row count
60,000 images could come from a narrow writer poolRelevant generalization count
distinct writers checked by writer-grouped holdout errorNo magic ratio
100 writers. 80/20? 90/10? Something else?
As in Lesson 0012's split-ratio section, the right answer is a reasoning process, not a memorized number.
80/2080 train20 holdout
90/1090 train10 holdout
70/3070 train30 holdout
k-foldrotate writersheld out once
Too little holdout
10 writers
noisy estimate
Too little training
70 writers
weaker model
Often useful
grouped k-fold
rotate whole writers
Whatever ratio is picked, the 0.80 auto-enter rule is only as trustworthy as the holdout. If writer images leak across train and holdout, the probability vectors can look more confident than they will on a genuinely new writer.
Retrieval practice
Check your recall.
Answer from memory before checking.
MNIST digit labels 0-9 are best treated as what kind of variable?
What rule converts a probability vector into a single predicted class?
Go deeper
Formal references.
An Introduction to Statistical Learning, Ch. 4 covers multiclass classification and softmax formally. StatQuest "Softmax" and "cross entropy" videos build the intuition visually first. See RESOURCES.md.
Want to compute the actual 10-class version with the homework's z1 = log 2, z3 = log 4 together, step by step? Just ask your teaching agent.