Lesson 0013 · Homework 1, Problem 2

Log loss and gradient descent

Six pictures: squeeze a score, price a guess, step downhill, separate accuracy from loss, see why β runs away, then add the regularization wall.

The machine

One row through the calculator.

No mystery machine first: pick one high-school-number row, turn the β dial, multiply, squash, price the guess, then step the dial downhill.

one training row x = 2 y = 1
beta dial β = 1.5 0 big β
raw score βx = 1.5 × 2 = 3
sigmoid squash σ(3) = 0.95
log-loss price, because y = 1 -log(.95) = .05
same row with beta = 0 βx = 0 × 2 = 0 σ(0) = .50 -log(.50) = .69
one downhill dial step L′(β) = -0.40 η = 0.25 1.5 - .25(-.40) = 1.6

1. Sigmoid

Same button. Three raw scores.

Call the raw score z = βx. Put any z into the sigmoid calculator. Big negative becomes almost 0. Zero becomes exactly .5. Big positive becomes almost 1.

calculator rows
z = -6 press σ .002
z = 0 press σ .500
z = +6 press σ .998
σ(z) = 1 / (1 + e-z)

The input z can be any raw score. The output is always a probability.

number line becomes probability z=-6 z=0 z=+6 0 .5 1 .002 .500 .998 σ(0) = .5 negative raw score squeezes down positive raw score squeezes up raw score z = βx

2. Log loss

MSE's cousin for categories.

One training row has an input x_i and a known answer y_i. The model does not output three separate predictions; it outputs one probability list for the possible labels.

number prediction: MSE
input x_i house facts
→
known label y_i price = 80
truth = 80 guess = 50 distance = 30 MSE price = (80 - 50)^2 = 900
off by 2
4
off by 10
100
off by 30
900
category prediction: log loss
input x_i image pixels
→
known label y_i Cat
model prediction for that same input one probability list, not three rows
Cat
.90
Dog
.07
Fish
.03
log loss turns that probability into a penalty y_i = Cat, so use p(Cat) -log(.90) = .11 penalty
p(Cat) = .90
penalty .11
p(Cat) = .50
penalty .69
p(Cat) = .10
penalty 2.30
one row, two different things known answer in the data: y_i = Cat model's guess for same input: p(Cat)=.90, p(Dog)=.07, p(Fish)=.03 log loss checks the known answer's probability: p(Cat)
MSE: square the numeric miss log loss: -log(probability on the correct label) confident wrong = model put little probability on Cat average over rows = L(β)

Before gradient descent

Learning means changing beta.

The row does not change. The human knobs do not change during this run. The model learns by changing its beta values, then recomputing the prediction for the same row.

1
The training row gives the problem
x_i = 3input value
y_i = 1known answer
not learnedalready in the data
2
The human chooses setup knobs
η = .25step size
λ = .10regularization
t = .50classification cutoff
3
The model starts with beta values
β0 = -1intercept / bias
β1 = 2weight on x
these are learnablegradient descent moves them
4
Now do ordinary arithmetic
β0-1
+
β1x_i2 × 3 = 6
=
raw score = 5then sigmoid makes p
5
Gradient descent changes beta, then the prediction changes
β1 = 2.00before step
L′(β1) = -.40slope says increase
β1 ← 2.10after step
before learning step same x_i = 3 β1 = 2.00 raw score = -1 + 2×3 = 5.00 p = σ(5.00) = .993
→
after learning step same x_i = 3 β1 = 2.10 raw score = -1 + 2.1×3 = 5.30 p = σ(5.30) = .995

Same input, new model: x_i = 3 stays fixed, β1 changes from 2.00 to 2.10, so the prediction changes from σ(5.00)=.993 to σ(5.30)=.995.

3. Gradient descent

How does the model know which way to change beta?

The goal is not to move for its own sake. The goal is to lower log loss. Gradient descent is the rule that tells beta which direction to move next.

try betaβ = 2.00
measure lossL(β) is the penalty
read slopeL′(β) says which way
change betaβ ← β - ηL′(β)
β change beta to lower loss wrong direction raises loss β_new = β - η·L′(β)
purpose of the demo below watch one beta-learning step the curve changes because β changes the loss number should go down

Purpose: log loss tells the model how bad the current beta is. Gradient descent uses the slope of that loss to choose the next beta. Then the same rows are predicted again with the new beta.

4. Accuracy is not loss

One scoreboard can say perfect while another still moves.

Accuracy counts threshold decisions. Log loss keeps caring about the probability margin. Training accuracy also says nothing by itself about new data.

classification accuracy threshold t = .50 0 0 1 1 4 / 4 correct on training rows
log loss .60 .80 .95 all correct, but the price can still shrink

5. Separated data

Perfect separation makes a downhill road with no floor.

When every negative x has y=0 and every positive x has y=1, bigger β keeps making every probability more confident and still correct. Plain log loss keeps decreasing; β keeps running.

data view all y=0 left all y=1 right clean split
loss view no finite bottom β → ∞

6. Regularization

Add a wall so the valley has a real bottom.

λβ² is a spring attached to the β dial. Pull β too far and the spring pulls back. The combined objective has a finite minimum.

two forces log loss falls λβ² rises finite minimum
which knob is which?
β

learned directly by gradient descent

λ

chosen outside training with validation data

train β pick λ test once validation split tunes λ; test split stays untouched
objective: L(β) + λβ² β is fitted λ is a hyperparameter

Homework map

Use the same pictures on the assigned four points.

The demo used (-3,0), (-1,0), (1,1), (3,1). Homework 1 uses (-2,0), (-1,0), (1,1), (2,1). Same machine; different arithmetic.

p_iσ(βx_i)
L(0)average price
L′(0)slope
β updatestep downhill

Check

Three small locks.

Answer from memory before checking.

At β = 0, what does σ(βx) output?
Is λ fitted directly, or is it a hyperparameter?
Which split chooses λ?
Want one panel turned into a numeric walkthrough? Ask your teaching agent to trace the four homework rows through the same visual chain.
← Problem 1 Next: Problem 3 → Glossary Notation decoder