About this video

• Video Title: The Fundamental Problem With LLMs – Richard Sutton
• Channel: Dwarkesh Clips
• Speakers: Dwarkesh Patel, Richard Sutton
• Duration: 00:05:39

Overview

This video discusses the fundamental differences between Reinforcement Learning (RL) and Large Language Models (LLMs) in the context of Artificial Intelligence. Richard Sutton argues that RL is more aligned with the core concept of intelligence, which involves understanding and acting in the world based on experience and goals, whereas LLMs primarily focus on mimicking human language and behavior without a genuine understanding of the world or a defined goal.

Key takeaways

• Distinction between RL and LLMs: RL focuses on understanding the world and acting based on experience and rewards, while LLMs mimic human language and actions, lacking a true understanding or objective.
• LLMs and World Models: The video questions whether LLMs, despite processing vast amounts of text, actually build robust world models. Sutton argues they primarily mimic human output rather than predict real-world occurrences.
• The Importance of Goals and Ground Truth: Sutton emphasizes that RL has a clear goal (maximizing reward), providing ground truth for learning. LLMs, in contrast, lack a defined goal or ground truth, making it difficult for them to determine the "right" thing to say or do.
• Learning from Experience vs. Imitation: True intelligence, according to Sutton, comes from learning from direct experience and seeing the consequences of actions. LLMs learn from observing what people say and do, which is imitation, not genuine learning from interaction with the world.
• The Limitation of LLM Priors: The idea that LLM outputs can serve as a "prior" for future learning is challenged because LLMs lack a definition of "actual knowledge" or a goal to guide this prior.

According to Richard Sutton, LLMs lack "ground truth" because there is no definition of what the "right" thing to say or do is within their framework. Unlike reinforcement learning, where the goal is to achieve rewards (providing a clear metric for correctness), LLMs don't have an explicit goal. Therefore, they cannot truly assess whether their output is correct or not, and consequently, they cannot have prior knowledge that is grounded in truth.

The key difference lies in their learning mechanisms: RL agents learn from direct experience by interacting with the world, taking actions, and observing the consequences to maximize rewards. In contrast, LLMs learn by mimicking human output from vast amounts of text, essentially learning "what a person did" in a given situation, without direct experience or a defined goal to guide their actions.