The inventor of the bitter lesson has a bitter lesson for LLM scaling enthusiasts

The inventor of the bitter lesson has a bitter lesson for LLM enthusiasts. Other researchers agree. In a recent post, Turing Award winner Richard Sutton says raw compute alone isn't enough for intelligence. AI science needs clever new methods and is "far from finished." His post sparked wide discussion in the AI community on X, a sign that Sutton struck a nerve.

In his famous 2019 essay "The Bitter Lesson," Sutton argued that general methods scaling with compute beat sophisticated human-engineered approaches eventually. What matters are ideas that grow with compute rather than rely on human assumptions. Sutton's position isn't new. In earlier statements, he criticized the limits of simply scaling large language models. He called synthetic data a "big mistake" because the real world is infinitely complex and any simulation of it is only "microscopic." Human expertise as a bottleneck prevents true scaling.

Instead, he called for agents with world models that continuously learn from their own experience. The world is massively more complex than what internet text can capture, and today's language models don't keep learning after training,…