Lesson 0016 · Homework 1, Bonus
Two ReLU units — the same activation function from Lesson 0005 — combine into a shape neither one can make alone.
ReLU(z) = max(0, z): pass positive input through unchanged, clip anything negative to zero. Lesson 0005 introduced it as one of the smooth-enough activations that made stacked layers trainable; here, two of them get wired together in the simplest possible way — just added.
h₁(x) = ReLU(x) fires only for positive x. h₂(x) = ReLU(−x) fires only for negative x — it's the same gate, just fed the negated input. Their sum, s(x) = h₁(x) + h₂(x), is active on both sides.
Slide x negative and h₁ goes to 0 while h₂ takes over; slide it positive and the roles swap. Exactly one of the two is ever nonzero (except right at x = 0, where both are 0) — so their sum always equals whichever one is active, which is just the distance from zero: s(x) = |x|.
s(x) = |x|, the same relationship the demo above shows live for any x.
An affine score ax + b is a single straight line — its slope is fixed, one number, everywhere. |x| needs slope −1 for x < 0 and slope +1 for x > 0: two different slopes, meeting at a sharp corner at x = 0. No single a can be both −1 and +1 at once, so no affine function reproduces a V over the whole real line.
Each ReLU unit contributes a flat-then-sloped piece — h₁ is flat at 0 for x < 0, then slope 1 for x > 0; h₂ is the mirror image. Adding two piecewise-linear pieces with different breakpoints and slopes is exactly how a stack of simple units builds a shape no single one of them has — the same "shape and stacking" idea Lesson 0005 introduces with the XOR problem, just in one dimension instead of two.
Answer from memory before checking.
The homework's last part asks you to invent your own plausible campus quantity, centered so x = 0 is "ordinary," where unusually large values in either direction might trigger the same follow-up action — and to name one ethical or operational judgment the network's equations alone can't settle. That's a reasoning question the syllabus wants in your own words; this lesson deliberately doesn't supply an example to swap in. If it helps to think out loud about what "centered at zero" even means here, or what "the equations can't settle X" is getting at, ask your teaching agent — just don't take a ready-made answer from it into the write-up.
Lesson 0005's XOR section is the direct prerequisite here — same "one unit can't, two can" logic, worked out for a 2D boundary instead of a 1D absolute value. See RESOURCES.md for general neural-network intuition sources.