Lesson 0009
Same training idea, wildly bigger and more varied D — and patterns become visible that no person could inspect example by example.
At small scale, next-word prediction looks like autocomplete. At huge scale, the same objective begins to expose structure: grammar, facts, style, code, translation, and conversation patterns. The model did not get a new goal; scale changed what the goal could express.
Lesson 0007's table already told you what GPT is: the transformer family, run through D --A--> f^_D, with A = predict-the-next-word. That's the whole objective — the same one whether D is a small pile of text or the trillion-plus-word piles from Lesson 0008. Nobody added a "write working code" objective, or a "translate French" objective. A never changed. What changed was the size of D — and, as D grew by orders of magnitude, the fitted model's behavior stopped looking like autocomplete and started looking like something else.
A neural network does not mathematically require the internet. It can train from books, lab data, medical scans, weather records, private documents, or anything else that has been digitized. But modern models needed machine-readable data at huge scale, and the internet made that practical: text, code, images, audio, video, captions, forums, manuals, translations, and reference material already sitting in digital form.
So the internet was not the model's brain. It was the giant library, media archive, and loading dock. It made enough varied data available for training to stop looking like memorization and start looking like pattern extraction.
learns repeated structure, not "the internet"
Did GPT read every written word in human history? No. Training data is enormous, but it is still a sampled, filtered, licensed, scraped, cleaned, and deduplicated subset. Much writing was never digitized, never public, never included, or was removed during filtering. And modern AI is broader than English text: wherever data has repeated structure, a model may be able to learn useful internal representations.
Sticky idea: the model did not read "all human writing." It trained on giant, uneven, filtered data mixtures: huge enough to learn surprising patterns, incomplete enough to still have blind spots.
Predicting the next word plausibly requires modeling a lot more than grammar, once the training text is varied enough. To guess the next token after "The French word for 'cat' is" well, a model has to pick up translation-like structure. To guess the next token in working Python, it has to pick up code structure. The same broader lesson applies outside text: images have edges and objects, audio has rhythm and speech sounds, video has motion, and medical scans have visual patterns. The model is not looking for English. It is looking for repeated structure.
At scale, models can find patterns no human could inspect manually. That does not make them universally smarter than humans; it makes them extremely strong pattern learners when the data contains repeated structure.
Sort each capability: was it the model's explicit training objective, or structure learned as a byproduct of training over huge, varied data?
A model can only express what its shape and its training allow (Lesson 0005's lesson, restated). Scale does not make magic. It makes repeated structure visible to the training process.
"Reasoning-like behavior" is a careful phrase, not a claim that the model reasons the way a person does. Lesson 0007's Box quote still applies at any scale: the model is still wrong in specific, sometimes surprising ways — confidently wrong, not just occasionally wrong — because nothing about scaling up next-word prediction gives the model a way to check its guesses against reality. Bigger D changes what the model can produce. It doesn't change what kind of thing the model is.