The compressed vocabulary from Lecture 3. Extend this file as later lectures add terms — don't redefine terms elsewhere.
| Term | Meaning |
|---|
| Task T / Experience E / Performance P | The operational definition of "learning": performance on T improves through experience E, measured by P. |
| Learning algorithm A | A fixed procedure. Reads a dataset D, returns a fitted map: A(D) = f^_D. |
| Fitted model f^_D | Data-dependent state produced by training. Stores what was learned; changes when D changes. |
| Training | Phase 1: examples → fit → reusable fitted map. D --A--> f^_D |
| Deployment | Phase 2: new input becomes an action. x --f^_D--> s --δ--> a (predict, then decide, then act). |
| Decision rule δ | Maps a score/probability s to an action a. Belongs to the deployed system, not the model. |
| Term | Meaning |
|---|
| Model family 𝓕 | The set of candidate functions {f_θ : X → S | θ ∈ Θ} considered before seeing data. Limits what can possibly be learned. |
| Fitted state | What a specific model family stores after training — e.g. coefficients (linear regression), 10 coefficient vectors (softmax regression), or the training set + a metric (kNN). |
| Parametric vs. nonparametric | Parametric models (linear/softmax regression) compress D into a fixed-size parameter vector. Nonparametric models (kNN) keep D itself as the fitted state. |
| Softmax | Converts per-class logit scores z_k(x) into a probability distribution over classes: p_k = e^{z_k} / Σ e^{z_j}. |
| Logit | A raw per-class score before softmax normalization — "evidence, not probability." |
| Argmax | Converts a probability vector into a single predicted label by picking the highest-probability class. |
| kNN (k-nearest neighbors) | Classifies a query point by majority vote among its K nearest training points, under a chosen distance metric. |
| Term | Meaning |
|---|
| MSE | Mean squared error — average squared distance between predicted and actual numeric values. Answers "how far?" Misses downstream action cost. |
| ROC / AUC | Receiver operating characteristic curve / area under it — measures how well a score ranks positives above negatives across all thresholds. Answers "how well ordered?" |
| Confusion matrix | Counts of TP / FP / FN / TN at one fixed decision threshold. Answers "which errors, and how many?" |
| Threshold t | The cutoff on a score s(x) that turns it into a binary action: δ_t(s) = 1{s ≥ t}. Changing t trades TPR against FPR — it does not change the underlying model. |
| TPR (true positive rate / sensitivity) | TP / (TP + FN) — fraction of actual positives correctly flagged. |
| FPR (false positive rate) | FP / (FP + TN) — fraction of actual negatives incorrectly flagged. |
| Group fairness gap | The same threshold t applied to the same model can produce very different FPR/FNR across subgroups G = g. "Equal rule ≠ equal burden." |
| Term | Meaning |
|---|
| Linear separability | Whether one straight boundary can split two classes perfectly. If not, no perceptron — however trained — can solve the task. |
| Perceptron | Linear regression's classification cousin: z = wTx + b, then a step function h(z) = 1{z ≥ 0} turns the score into a class. |
| Activation function | Turns a weighted sum into a signal — a step (perceptron), or something smoother (sigmoid, tanh, ReLU) in a neural network. |
| Multi-layer perceptron (MLP) | Perceptrons stacked in layers, each fully connected to the next: h(ℓ) = σ(W(ℓ)h(ℓ-1) + b(ℓ)). Bends the boundary a single perceptron can't. |
| Neural network | The broader model family that MLPs belong to — stacked, learned layers, of which many architectures exist. |
| Credit assignment | The problem stacking creates: the output was wrong, but which interior weight — layers upstream, not wired to the answer — deserves the blame? The single-layer perceptron rule can't answer it. |
| Backpropagation | The training algorithm that solved credit assignment: use the chain rule to compute each weight's share of the blame, passing the error backward through the layers. Needs smooth activations — it flows along slopes, and a hard step has none. |
| Attention | Inside a transformer, the mechanism that decides which other positions in the input should influence a given position's meaning — not just its immediate neighbor. |
| Transformer | A neural network architecture built for sequences (text, etc.), stacking attention and feed-forward blocks. Still just a model family: {f_θ}, unfitted until trained. |
| GPT (Generative Pre-trained Transformer) | Not a family — a specific fitted model, f^_D, produced by training the transformer family on a huge text dataset D via next-word prediction. |
| Token | The small chunk of text a language model actually counts — roughly ¾ of a word on average, not a whole word. Training-set size and cost are usually measured in tokens, not words. |
| Term | Meaning |
|---|
| Sigmoid σ(z) | 1/(1+e−z) — squashes any real number into (0,1). σ(0) = 0.5 always, regardless of the data. |
| Log loss L(β) | Average of −[y·log p + (1−y)·log(1−p)] over all training points — "how wrong, on average" for probability predictions, the same role MSE plays for numeric ones. |
| Gradient L′(β) | Which way, and how strongly, to move β to reduce L(β). Zero at a minimum; its sign tells gradient descent which direction to step. |
| Gradient descent step | β(1) = β(0) − η·L′(β(0)) — subtract the gradient, scaled by a learning rate η, from the current parameter. |
| Learning rate η | A hyperparameter controlling step size in gradient descent — not fitted from data. |
| Separation (separability) | When a single feature (or line) perfectly divides the classes. Under separation, unregularized logistic regression drives |β| toward infinity chasing ever-lower loss. |
| L2 regularization / ridge penalty | λ‖β‖₂² added to the loss. Favors many small coefficients over a few large ones ("distributed shrinkage") — geometrically, a round constraint region. |
| L1 regularization / lasso penalty | λ‖β‖₁ added to the loss. More readily produces exact zeros — geometrically, a cornered (diamond) constraint region whose corners sit on the axes. |
| Hyperparameter | A value set before training (e.g. λ, η) rather than fit from data. Chosen via a validation split; the test split stays untouched for final evaluation. |
| Multiclass log loss | −log py(x) — penalizes low probability on the true class only. Can disagree with classification error, since error only checks whether argmax was correct. |
| Capacity constraint | A limit on how many flagged cases can actually be acted on, independent of the model or threshold — handled by ranking and acting on the top-k, not by retraining. |
| Grouped (by-entity) split | A held-out split where every row belonging to one entity (a student, a writer) goes entirely into training or entirely into holdout — never both — so evaluation doesn't leak "I've seen this one before." |
Something here unclear, or you spot an error? Ask your teaching agent — that's what it's for.