Lesson 0008
"Trained on a huge pile of text" is the kind of phrase that sounds precise and means nothing. Before Lesson 0009 asks what huge scale does, this lesson asks how huge it actually is.
GPT-3 (2020) was trained on roughly 1 trillion words of text — about 300–500 billion tokens, depending how you count. Reading at a typical adult's silent reading speed (about 250 words per minute), 8 hours a day, every single day with no days off — how long would it take one person to read that much?
That was 2020. By 2024, Meta's Llama 3 was trained on roughly 15 trillion tokens — 7× more than its own predecessor, Llama 2, and on the order of 30–50× GPT-3's token count, just four years later. The "huge pile of text" a lesson casually gestures at isn't a fixed size. It's a number that has kept climbing by an order of magnitude every few years.
Figures above are sourced, not estimated from memory: GPT-3's token/word counts and training corpus composition are widely reported (Common Crawl + WebText2 + book corpora + Wikipedia, ~570GB filtered text); Llama 3's 15-trillion-token figure and its 7× jump over Llama 2 are from Meta's official Llama 3 announcement; the 250 wpm adult silent-reading-speed figure is from a meta-analysis of 190 studies (Brysbaert, 2019, reporting 238–260 wpm for adult English text).