Richard Sutton – Father of RL thinks LLMs are a dead end

About this video

• Video Title: Richard Sutton – Father of RL thinks LLMs are a dead end
• Channel: Dwarkesh Patel
• Speakers: Dwarkesh Patel, Richard Sutton
• Duration: 01:07:09

Overview

This video features a conversation between Dwarkesh Patel and Richard Sutton, a pioneer in reinforcement learning (RL). They discuss the limitations of Large Language Models (LLMs) from an RL perspective, emphasizing the importance of learning from experience and having goals. Sutton argues that LLMs primarily focus on mimicking human language rather than understanding the world or acting purposefully. The discussion also touches upon the "Bitter Lesson" in AI, the role of computation, the nature of intelligence, and the future of AI succession.

Key takeaways

• LLMs vs. RL: Sutton posits that LLMs are primarily about mimicking human language and behavior, whereas RL is fundamentally about understanding the world and achieving goals through experience.
• Importance of Experience: True intelligence, according to Sutton, comes from learning through direct experience and interaction with the world, not just from passively analyzing vast amounts of text data.
• The Role of Goals: A core component of intelligence is having a goal. Sutton argues that LLMs lack a substantive goal beyond next-token prediction, which doesn't drive them to alter the world or learn from outcomes in a meaningful way.
• The "Bitter Lesson": Sutton's influential essay suggests that general-purpose learning methods, particularly those leveraging computation and scaling, tend to outperform methods that rely heavily on human knowledge or handcrafted features. He sees LLMs as potentially being a temporary success that will eventually be superseded by more scalable, experience-driven approaches.
• Generalization Challenges: Current deep learning methods, including LLMs, struggle with true generalization. They excel at solving problems similar to those in their training data but often fail when faced with novel situations or catastrophic forgetting.
• AI Succession: The conversation explores the inevitable transition of control to AI or AI-augmented humans, driven by the increasing intelligence and resource acquisition capabilities of these systems.