Commonplace

There Are No New Ideas in AI… Only New Datasets

substack.com · saved

“LLMs in four breakthroughs

  1. Deep neural networks: Deep neural networks first took off after the AlexNet model won an image recognition competition in 2012

  2. Transformers + LLMs: in 2017 Google proposed transformers in Attention Is All You Need, which led to BERT (Google, 2018) and the original GPT (OpenAI, 2018)

  3. RLHF: first proposed (to my knowledge) in the InstructGPT paper from OpenAI in 2022

  4. Reasoning: in 2024 OpenAI released O1, which led to DeepSeek R1

…

It’s not crazy to argue that all the underlying mechanisms of these breakthroughs existed in the 1990s, if not before. We’re applying relatively simple neural network architectures and doing either supervised learning (1 and 2) or reinforcement learning (3 and 4).”

“But there’s a missing piece here: each of these four breakthroughs enabled us to learn from a new data source:

  1. AlexNet and its follow-ups unlocked ImageNet, a large database of class-labeled images that drove fifteen years of progress in computer vision

  2. Transformers unlocked training on “The Internet” and a race to download, categorize, and parse all the text on The Web (which it seems we’ve mostly done by now)

  3. RLHF allowed us to learn from human labels indicating what “good text” is (mostly a vibes thing)

  4. Reasoning seems to let us learn from “verifiers”, things like calculators and compilers that can evaluate the outputs of language models”

Thus the conclusion that AI breakthroughs are simply about unlocking new sources of training data.

“This discovered equivalence is really profound because it hints that there is an upper bound to what we might learn from a given dataset. All the training tricks and model upgrades in the world won’t get around the cold hard fact that there is only so much you can learn from a given dataset.”

← All articles