About this video

• Video Title: But how do AI images and videos actually work? | Guest video by @WelchLabsVideo
• Channel: 3Blue1Brown
• Speakers: [None explicitly named in transcript, but the guest creator is Stephen Welsh]
• Duration: 00:37:20

Overview

This video explains the underlying mechanisms of AI image and video generation, focusing on diffusion models. It breaks down the process by first introducing CLIP (Contrastive Language–Image Pre-training) and its concept of a shared embedding space, then explains diffusion models and DDPM (Denoising Diffusion Probabilistic Models), detailing how they learn to reverse a noise-adding process by predicting vector fields. Finally, it discusses how CLIP and diffusion models are combined for text-to-image/video generation using techniques like conditioning, guidance, and negative prompts.

Key takeaways

• Diffusion Models: These AI systems generate images and videos by reversing a process similar to Brownian motion, starting with random noise and gradually refining it into a coherent output.
• CLIP (Contrastive Language–Image Pre-training): CLIP creates a shared high-dimensional space where images and their corresponding text descriptions are represented by similar vectors, enabling a connection between language and visual concepts.
• Learning Vector Fields: Diffusion models learn a "score function" or vector field that, for any given point (image) and time step, indicates the direction to move to remove noise and reconstruct the original data.
• DDPM vs. DDIM: DDPM involves adding noise at each step of generation, which helps capture the full data distribution. DDIM offers a faster, deterministic approach by using an ordinary differential equation derived from the DDPM process, allowing for fewer steps and higher quality without random noise addition.
• Conditioning, Guidance, and Negative Prompts: These techniques are used to steer the diffusion process based on text prompts. Conditioning integrates text information into the model, guidance amplifies the influence of the desired prompt, and negative prompts actively steer the generation away from undesired features.

The transcript doesn't directly address why all AI images might look like "AI images" even when appearing photorealistic initially. However, it does provide clues that could contribute to this phenomenon:

• Averages and Blurriness: The video explains that without the random noise step in the diffusion process (as seen in the comparison between DDPM and DDIM without noise), generated images tend to converge towards the "average" or "mean" of the dataset. In the space of images, these averages often appear blurry and less distinct, lacking the specific details that make a photorealistic image unique and believable.
• The Manifold of Realistic Images: The transcript mentions that in high-dimensional image space, the generation process might not perfectly reach the "manifold of realistic images." This suggests that generated images might exist in a space that is close to, but not exactly on, the distribution of real-world photographs, leading to subtle imperfections.
• Subtle Artifacts from Guidance/Conditioning: While guidance and conditioning (including negative prompts) are powerful tools for prompt adherence, they are still based on learned patterns. If the model's understanding of these patterns or the "vector fields" it learns from isn't perfect, it can lead to subtle, unnatural artifacts that a discerning eye might pick up on, even if the overall image is photorealistic.
• "Cartoonish" Elements: The example of the "astronaut on a horse" video without a negative prompt shows parts of the scene becoming "cartoonish." This indicates that even with advanced techniques, the model might still struggle with certain aspects of realism, leading to a tell-tale "AI look."

Essentially, while AI models are getting incredibly good at mimicking reality, the underlying processes (averaging, not quite reaching the true data manifold, or subtle misinterpretations of prompts) can still introduce tell-tale signs that differentiate them from genuine photographs.