Last year's students
Lesson 0012 · Homework 1, Problem 1
Thresholds and policy
Sage trains on last year's students. The model outputs a score. Next year's students get predicted before the support problem happens, so the university can act in time.
Visual walkthrough
Watch the validation mistake happen.
The last two scenes show why random row splitting is the wrong target: weekly rows from the same student can leak across train and holdout. The fix is to hold out whole students, and ideally a future entering class.
Problem 1(a)
One pipeline. Three different jobs.
The fitted model produces s_i. The threshold rule produces a_i. Moving t changes the action rule, not the trained model.
Do not mix these up
Last year's students have answers. Next year's students do not yet.
Next year's students
How the dataset is built
Last year's students become training data when the year is over.
Sage records only what can be observed while the year is still happening. The same student can appear in several weekly rows. Once the year ends, the outcome label is attached to each of those rows, and the fitted model can score next year's students before the problem happens.
Same student, several weekly rows
week 6
PID 730123456
classes 3 of 5, late work 2
week 9
PID 730123456
classes 1 of 5, late work 4
week 12
PID 730123456
classes 2 of 5, late work 3
Labelnot known yet
After the year ends
PID730123456
Same observationsx_i
Label added from recordsy_i = 1
Meaningneeded support
Incoming student, scored by the model
PID891246357
Model scores_i = 0.62
Thresholdt = 0.50
Decisioncontact now
Benefithelp before trouble grows
Why this matters: a correct prediction can trigger help before the student falls behind, instead of after the problem has already grown.
Problem 1(d)
Same PID means same student.
Validate on students the model has never seen.
Wrong
same PID in train + test
Right
whole students, or next cohort
Risky design
Randomly split weekly rows
Better design
Hold out new students
Answer: split by student ID, or hold out next year's entering class. Random rows can put one student's earlier weeks in training and later weeks in test.
A tempting but wrong analogy
"32 is enough" is a statistics answer, not an ML answer.
Sage remembers a stats course: once you have a sample size of about 32, the sampling distribution of the mean is close enough to normal, and a bigger sample mostly buys you a tighter confidence interval on a fixed, low-dimensional quantity — a mean, a proportion, a slope. Sage reasons that 100 student records should be more than plenty.
Why the analogy breaks: a classical sample-size rule assumes you already know the shape of what you're estimating (one mean, one proportion) and are just asking how precisely a sample of that size pins it down. Fitting
f^D is a different kind of problem — you're searching over a whole family of functions for one that captures how features relate to the outcome, and you don't know in advance how much data that search needs. A model too flexible for the data available will fit noise in those 100 records instead of the real pattern, and no formula tells you the right amount up front the way the CLT tells you 32 is enough for a mean.
This is exactly why the holdout set exists: not to compute a textbook confidence interval, but to give an honest, out-of-sample check on whether f^D has actually learned something transferable or has quietly memorized last year's roster. If the holdout error is much worse than the training error, that's the empirical signal that more data (or a simpler model) is needed — a diagnosis a sample-size formula can't give you, because there was never a fixed target quantity to solve for.
Now pick a number
100 student records. 80/20? 90/10? Something else?
After grouping by student, choose the train/holdout balance.
80/2080 train20 holdout
90/1090 train10 holdout
70/3070 train30 holdout
k-foldrotate groupsevery student held out once
Too little holdout
10 students
noisy estimate
Too little training
70 students
weaker model
Often useful
grouped k-fold
rotate whole students
Takeaway: no magic ratio. Use enough holdout students to trust the estimate and enough training students to resemble deployment.
Where does the threshold fit in?
The threshold t doesn't decide the split — but a bad split corrupts your choice of t.
Grouping by student and choosing a split ratio happen before scoring; the threshold is chosen afterward, by looking at TPR/FPR tradeoffs on the holdout scores (as in the panel above). The two questions are separate — but not independent. If a random-row split let copies of the same student leak into both train and holdout, the holdout scores s_i would look more separable than they really are, so whatever threshold you pick by staring at those scores would look better in validation than it performs on genuinely new students next year. A leaky holdout doesn't just inflate accuracy — it inflates confidence in a specific, deployed decision rule δ_t. Get the holdout design right first; only then is the threshold tuning on it trustworthy.
Problems 1(b) and 1(c)
Move the cutoff. Watch the mistakes move.
The dots are the held-out sample from the homework. Drag the threshold, then compare the two worked checkpoints. This is the part where prediction becomes action before the student needs support.
t = 0.50
TP2
FP2
FN2
TN2
TPR = 2/4 = 0.50
FPR = 2/4 = 0.50
t = 0.30
TP3
FP3
FN1
TN1
TPR = 3/4 = 0.75
FPR = 3/4 = 0.75
The policy tradeoff
Lower threshold: more help, more false alarms.
Benefit
fewer missed students
Cost
more unnecessary contacts
Capacity changes deployment
Only four calls this week.
The scores stay fixed. The office adds a practical rule: sort by score, then contact the top four it can actually handle.
10.90contact
20.75contact
30.65contact
40.55contact
50.45wait
60.35wait
70.25wait
80.10wait
What stays fixed?
Fixed
A, D, f^D, scores s_i
Changesthreshold t, decision rule δ_t, actions a_i
Validationhold out whole students, not random weekly rows
Stuck on the threshold counts, the capacity rule, or the validation split? Ask your teaching agent to walk through one panel.