Yann LeCun on What Comes After LLMs

About this video

• Video Title: Yann LeCun on What Comes After LLMs
• Channel: Unsupervised Learning: With Jacob Effron
• Speakers: Yann LeCun, Jacob Effron
• Duration: 01:21:57

Overview

This video features a conversation between AI pioneer Yann LeCun and host Jacob Effron. LeCun outlines his belief that Large Language Models (LLMs) are not a viable path to human-level intelligence, despite their current utility. He discusses his alternative JEPA (Joint Embedding Predictive Architecture) for AI development, which focuses on understanding the real world through abstract representations rather than pixel-level predictions. The discussion also covers LeCun's departure from Meta, his new company AMI, his thoughts on the AI industry's focus on LLMs, the concept of sovereign AI with his Tapestry project, and the inherent safety concerns associated with LLMs.

Key takeaways

• LLMs are not the path to human-level intelligence: LeCun argues that LLMs excel at language manipulation but fail to understand the physical world, predict consequences of actions, or plan effectively, which he considers essential for true intelligence.
• JEPA architecture for real-world understanding: LeCun's proposed Joint Embedding Predictive Architecture (JEPA) aims to learn abstract representations of the world, enabling better prediction and planning, which he believes is crucial for applications like robotics and industrial control.
• Critique of current AI industry trends: LeCun observes a "herd behavior" in Silicon Valley, with many companies focusing on LLMs despite his belief in the limitations of this approach and the potential of alternative architectures like JEPA.
• Sovereign AI and Tapestry project: LeCun discusses his Tapestry project, which aims to create an open, fine-tunable AI platform that respects data sovereignty, allowing different cultures and languages to be represented without being dominated by US or Chinese AI models.
• LLMs are intrinsically unsafe: LeCun asserts that LLMs, in their current form, cannot be made reliably safe due to their inability to predict action consequences and their susceptibility to hallucinations, unlike architectures with world models that can incorporate safety constraints.
• Departure from Meta and FAIR's legacy: LeCun clarifies his role at Meta, emphasizing his focus on establishing the FAIR research lab and its contributions, and explains that his departure was due to a misalignment in strategic focus towards LLMs, which he did not believe was the future of AI.

Yann LeCun identifies several core limitations of LLMs that, in his view, prevent them from achieving human-level intelligence:

• Inability to Predict Consequences: LLMs lack the ability to anticipate the outcomes of their own actions. This is seen as a fundamental gap, as humans and animals naturally predict the results of their behaviors.
• Lack of Planning Capability: Related to the above, LLMs do not inherently possess planning abilities. LeCun contrasts this with how humans plan by searching for sequences of actions to achieve goals. LLMs, in their current form, predict the next token rather than engaging in strategic planning.
• Inadequacy for Real-World Understanding: LeCun argues that LLMs are not well-suited for understanding the physical, complex, high-dimensional, and continuous nature of the real world. Their success is primarily in domains where language itself is the substrate of reasoning, such as mathematics and code, not in modeling the messy realities of physical interaction.
• Susceptibility to Hallucinations: LeCun states that LLMs cannot be made reliably safe because they cannot be prevented from generating incorrect or fabricated information (hallucinating).
• Limited Generalization without Explicit Training: While LLMs can perform tasks they've been trained on, they struggle with novel situations or tasks they haven't encountered. This contrasts with systems that possess world models, which could potentially generalize to new problems with less explicit training data.

The JEPA (Joint Embedding Predictive Architecture) differs from current LLM approaches in several key ways, focusing on a different paradigm for AI development:

Differences from LLMs:

• Focus on Representation Learning vs. Pixel Prediction: LLMs are primarily trained on discrete sequences of tokens (language). JEPA, on the other hand, is designed to learn representations of the world by predicting future states in a representation space, rather than generating pixel-level predictions. LeCun argues that predicting pixels is an inefficient and often ineffective way to model complex reality.
• World Models and Prediction: JEPA is fundamentally a world model. Its goal is to allow an AI system to anticipate the consequences of its own actions by understanding the underlying dynamics of the world. LLMs, LeCun contends, do not inherently build these kinds of predictive world models.
• Abstraction over Direct Prediction: While LLMs predict the next token in a sequence, JEPA aims to capture more abstract, semantic information about the world. This allows for a more generalized understanding that isn't tied to specific output formats like text.
• Planning and Reasoning: By building world models that predict consequences, JEPA aims to enable planning and reasoning, which are seen as essential for intelligent behavior and are currently lacking in LLMs.
• Data Efficiency: LeCun suggests that architectures like JEPA, which build world models, have the potential to be more data-efficient, learning general principles that can be applied to new tasks with less specific training data, unlike the extensive data requirements for LLMs.

Intended Applications:

LeCun envisions JEPA being applicable to a wide range of tasks that require a deeper understanding of the real world beyond language:

• Robotics: For robots to effectively interact with and manipulate their environment, they need to predict the outcomes of their actions. JEPA's world modeling capabilities are seen as crucial for developing more capable and adaptable robots.
• Industrial Applications: In complex systems like manufacturing lines, power plants, or chemical processes, JEPA could be used to model the dynamics and predict the consequences of changes in control variables, allowing for optimal control and problem-solving.
• Healthcare: Developing personalized treatment plans for complex diseases or understanding cellular processes (like directing stem cells to become specific cell types) requires a sophisticated understanding of biological and physiological dynamics, which JEPA aims to provide.
• General Intelligence: Ultimately, LeCun believes that JEPA is a step towards building more general artificial intelligence that can truly understand and reason about the world, moving beyond the limitations of language-centric models.

The motivation behind Yann LeCun's Tapestry project stems from a concern about the dominance of AI models developed in the US and China, and the potential for these models to impose specific cultural, linguistic, and value systems on users globally.

Motivation for Tapestry:

• AI Sovereignty: Many countries outside of the US and China desire a degree of sovereignty over their AI development and deployment. They don't want their citizens' information consumption and AI interactions to be solely dictated by the biases and perspectives embedded in models trained by a few dominant tech hubs.
• Catering to Diverse Needs: Users worldwide speak different languages, have diverse cultures, and hold varying value systems and political opinions. Current AI assistants, often trained on data predominantly from the US or China, may not adequately represent or understand these nuances, leading to suboptimal or even alienating user experiences.
• Openness and Customization: LeCun believes that for AI to truly serve humanity broadly, there needs to be a platform that is open and allows for customization. This enables individuals, communities, and nations to fine-tune AI models to align with their specific needs, languages, cultures, and values.

How Tapestry Aims to Achieve Sovereign AI:

Tapestry proposes to build a foundation AI model that is as capable as proprietary models but remains open and accessible for global fine-tuning. It aims to achieve this through a federated learning-like approach:

• Open Foundation Model: The core idea is to create a large, powerful AI model that is not controlled by a single entity or nation.
• Decentralized Training and Data Control: Contributors worldwide can use their own data and computing resources to train or fine-tune the model. Crucially, they would not need to share their raw data with others. Instead, they would contribute parameter vectors or updates to a global consensus model.
• Federated Learning Principles: This approach resembles federated learning, where a central model is improved by insights learned from distributed data without the data itself leaving its local environment. Contributors would update their local models and then share their model's learned parameters, which would then be aggregated into a global model.
• Convergence to a Global Model: Through this iterative process, the contributions from diverse sources would converge into a single, powerful global model that encapsulates a wide range of world knowledge and culture.
• Fine-tuning for Local Needs: Once this robust, open foundation model exists, individuals or groups can then fine-tune it further to cater to their specific linguistic, cultural, or ideological needs, thereby achieving a form of AI sovereignty.
• Avoiding Proprietary "Walled Gardens": This approach is seen as a counter to the trend of increasingly closed-source AI models, drawing parallels to the rise of Linux and open-source software that eventually dominated proprietary operating systems and internet infrastructure due to their openness and adaptability.

Yann LeCun believes LLMs are "intrinsically unsafe" primarily due to two fundamental limitations:

• Inability to Predict Consequences: LeCun argues that LLMs cannot reliably predict the consequences of their actions. If an LLM were to be used in an agentic system (an AI that can take actions in the real world), this lack of predictive capability means it could take actions with unforeseen and potentially dangerous outcomes.
• Unreliability and Hallucinations: LLMs are prone to "hallucinations," meaning they can generate plausible-sounding but factually incorrect or nonsensical information. This inherent unreliability makes them unsafe for critical applications where accuracy is paramount. Even if trained to avoid certain behaviors, LeCun suggests there's always a "gap between training error and test error," meaning a prompt could potentially cause the LLM to behave in unintended or harmful ways.

Alternatives and Mitigation Strategies:

LeCun proposes moving away from LLMs towards a different architectural paradigm that he believes can be made safer and more reliable:

• Objective-Driven AI with World Models: His proposed alternative is to build AI systems that are "objective-driven." These systems would operate based on a clearly defined objective (e.g., "accomplish this task") and possess an explicit world model.
• Predictive Capabilities: The world model allows the AI to predict the consequences of potential actions. By simulating actions and their outcomes based on this model, the AI can then plan a sequence of actions that best achieves its objective.
• Incorporating Safety Constraints: Crucially, these objective-driven systems can have safety constraints built directly into their cost functions or objectives. This means the AI is designed from the ground up to satisfy conditions like "do not harm anyone" or "do not cause damage." LeCun emphasizes that these constraints are part of the system's fundamental design, making them more robust than trying to "trick" an LLM into behaving safely through prompts alone.
• JEPA Architecture: The JEPA (Joint Embedding Predictive Architecture) is LeCun's proposed framework for building these world models. By learning abstract representations and predicting in this space, JEPA aims to create systems that can understand and interact with the world more effectively and safely than LLMs. The idea is that by having a model of the world's dynamics, the AI can reason about the impact of its actions, thereby reducing the likelihood of dangerous or unintended consequences.

Here are some topics and tags to explore this video in detail:

Topics and Tags

Topic	Tags
AI Architectures	JEPA, World Models, Representation Learning, Self-Supervised Learning, Predictive Models, Embodied AI
Large Language Models (LLMs)	Limitations of LLMs, LLM Safety, LLM Scalability, LLM vs. World Models
Future of AI	Artificial General Intelligence (AGI), Intelligent Systems, AI Capabilities, AI Evolution
AI Ethics and Safety	AI Safety, AI Risk, Responsible AI, AI Control, Intrinsic Safety
AI Industry Trends	Silicon Valley, Tech Industry, AI Research Labs, Open Source AI, Proprietary AI, Innovation
Machine Learning Concepts	Predictive Modeling, Reinforcement Learning, Imitation Learning, Contrastive Learning, Representation Collapse
Specific Companies & Projects	AMI Labs, Tapestry, FAIR (Meta AI), OpenAI, Google DeepMind, Mistral AI
Robotics and Embodied AI	Robotics Models, Robot Manipulation, Autonomous Systems, Embodied Intelligence
AI Development Philosophy	Contrarian Views, Research vs. Product, Data Efficiency, Scientific Vision
Yann LeCun's Contributions	Turing Award, AI Pioneer, Self-Supervised Learning, Neural Networks
AI in Specific Domains	Healthcare AI, Industrial Automation, Manufacturing, Robotics
AI Governance and Society	AI Regulation, Sovereign AI, Cultural Bias in AI, AI Impact on Society

You're interested in the topic of Robotics within the context of this video. Yann LeCun discusses robotics in relation to the limitations of current AI approaches and the potential of his proposed JEPA architecture.

Here are some points related to robotics from the video:

• Problems with Current Robotics Models: LeCun suggests that many current robotics models, especially those relying on imitation learning or LLM-based approaches, are brittle and require vast amounts of task-specific data. He points out that these models struggle with generalization and reliability in real-world scenarios.
• Need for World Models in Robotics: He argues that for robots to become truly useful and adaptable, they need a world model. This model would allow them to predict the consequences of their actions in complex, dynamic environments, enabling them to plan and execute tasks more effectively without needing exhaustive pre-programmed data for every situation.
• JEPA's Role in Robotics: The JEPA architecture is proposed as a solution for developing these world models. By learning abstract representations and predicting in that space, JEPA aims to give robots a better understanding of physics and causality, leading to more robust and generalizable capabilities.
• Comparison to LLM-based Robotics: LeCun contrasts the data-hungry, imitation-based methods used in some current robotics demonstrations with the potential of JEPA-based systems, which he believes will require less data and exhibit greater generalization.
• Data Efficiency for Robotics: He highlights that while current robotics demonstrations might seem impressive, they are often trained on massive datasets. JEPA-based approaches, by learning world models, could potentially reduce the data burden significantly, making AI development for robotics more efficient.

Would you like to explore any of these points further, or perhaps discuss other aspects of robotics as they relate to the video?