Yann LeCun: Models of SSL (April 29, 2025)

About this Video

• Video Title: Yann LeCun: Models of SSL (April 29, 2025)
• Channel: Flatiron Institute
• Speakers: Yann LeCun
• Duration: 00:48:40

Introduction

This video features Yann LeCun discussing self-supervised learning (SSL) and its potential to achieve human-level AI. He critiques the reliance on Large Language Models (LLMs) alone, arguing that SSL, which models how humans and animals learn through observation, is crucial for understanding the world and planning actions. The lecture focuses on energy-based models as a superior alternative to density estimation for SSL and explores various methods for training these models and preventing collapse.

Key Takeaways

1. Limitations of LLMs: LeCun argues that LLMs alone are insufficient for achieving human-level AI, highlighting the vast difference in data processing capacity between humans and current LLMs.
2. Energy-Based Models for SSL: He advocates for energy-based models as a more tractable approach to SSL compared to density estimation in high-dimensional spaces. These models focus on learning energy functions that capture dependencies between variables, enabling inference through optimization.
3. Collapse Prevention: LeCun discusses the critical issue of "collapse" in SSL models, where the model ignores input and produces constant outputs. He contrasts contrastive and regularized methods for preventing this, favoring regularized methods for high-dimensional data like images and video.
4. Joint Embedding Predictive Architectures (JPAs): He proposes JPAs as a superior alternative to generative architectures for video and image data. JPAs learn representations of inputs and make predictions in representation space, simplifying the prediction problem.
5. World Models for Planning: LeCun suggests JPAs are ideal for building world models that predict future states based on actions, enabling AI systems to plan actions to achieve goals through optimization. This model aligns with human system-two reasoning.