Lesson 0003
The model is finished. Now: how do you know if it's good — and good for what? This is where knowing stops being enough and someone has to act.
A scale weighs; a clock times; neither can do the other's job, and asking a clock "how heavy" is a category error, not a hard question. STOR 323 makes the same point about evaluating models: the metric follows the object being evaluated. A number gets measured by how far off it is (MSE). A ranking gets measured by how well-ordered it is (ROC/AUC). A yes-or-no decision gets measured by which specific errors it makes (the confusion matrix). Using the wrong one doesn't just give a worse answer — it answers a question nobody asked.
MSE you've effectively already met: back in lesson 1, the "error" number that dropped as you tuned the impression-line was exactly this — the average squared distance between the impression and what actually happened. It answers "how far off," and nothing more. The richer question is what comes next.
A screening test returns a score between 0 and 1 — call it the model's honest best guess. But a number alone refers no one for treatment. Someone has to draw a line: score above this, act; below, don't. That line is a threshold, t, and choosing it is not a discovery about the world. It's a decision.
Choosing t is precisely this kind of act. You have the right to set the threshold — that part is yours. You do not have the right to which specific patient turns out to be a false alarm or a missed case — the fruits. Move the threshold below to see the shape of the choice:
Every choice of t gives one confusion matrix, one pair of error rates. But there's a question underneath all of them: independent of where you draw the line, is this model any good at ranking the sick above the healthy at all? Sweeping t across its whole range traces the ROC curve; AUC compresses that whole sweep into a single number.
Look at the small plot beside the sliders above: as you swept t, it traced a trail — not one point, the whole curve. That trail is what a witness's view of the model looks like: not attached to any single threshold's verdict, but taking in the entire range of what the model could say. AUC is what remains when you refuse to over-identify with any one decision. It answers "how good is the seeing," not "what was decided" — and, tellingly, it cannot tell you where to set t. That question is still yours; sākṣī bhāva was never a substitute for acting, only for clarity before acting.
Here is the sharpest turn in the lecture. Apply the exact same threshold — the exact same rule — to two different groups of patients. Group A: 40% of healthy patients get a false alarm. Group B: 20%. Same model, same t, same intention. Different real burden.
Adhikāra-bheda is a claim about teaching, not about statistics — the mechanism here is different (base rates and score distributions, not spiritual readiness). But the structural insight transfers cleanly: uniform treatment, applied without attention to differing starting conditions, does not produce uniform outcomes. "Same rule for everyone" sounds like fairness. It is not the same claim as "same burden for everyone," and STOR 323 states this as plainly as the tradition does: equal rule ≠ equal burden.
Answer from memory before checking.
Formal side: An Introduction to Statistical Learning Ch. 4 on ROC curves, and Fairness and Machine Learning for the group-fairness material properly developed. Philosophical side: the Gītā's second chapter in full is worth reading past verse 2.47 — the whole chapter is Kṛṣṇa building the case for right action under uncertainty. See RESOURCES.md.