Generative models

How AI images and videos actually work: diffusion, CLIP, and guidance

A text-to-video model starts from pure noise and a transformer sculpts it into a scene. The reason that works traces back to Brownian motion, a contrastive trick from 2021, and one more idea that makes prompts actually stick.

Creator: Welch Labs (guest video on 3Blue1Brown)Runtime: 37 minVideo published: July 25, 20251.9M views when summarized

Watch on YouTube

TL;DR

The short version

Diffusion = Brownian motion run backwards. A transformer turns random-noise pixels into video step by step, the time-reverse of particles diffusing in physics.
CLIP trains a text and an image encoder to land matching captions and pictures at the same point in a 512-dimensional space — so 'me with a hat' minus 'me' lands on the word hat.
DDPM works for two non-obvious reasons: it predicts the total noise added (not one step), and it re-injects random noise during generation. Remove that noise and you get a blurry average.
DDIM rewrites generation as a deterministic ODE (via Fokker-Planck), hitting the same final distribution in far fewer steps with no random kicks.
Classifier-free guidance subtracts the unconditioned vector field from the conditioned one and amplifies the difference — that's what makes a prompted tree actually appear and grow.

01 · Generative models

Noise into video, one pass at a time

Open up the source code of an open-source video model like Wan 2.1 and the first thing it does is call a random number generator: a video where every pixel intensity is chosen at random. That pure-noise clip goes into a transformer — the same architecture behind ChatGPT — but instead of text it outputs another video, still mostly noise with faint structure. Add that to the noise, pass it back in, repeat fifty times, and you get a sharp, realistic scene.

Pure noise on the left, transformer in the middle, the first hints of structure on the right. Run the loop ~50 times and the noise resolves into video.00:01:41

The connectionThis iterative denoising is mathematically equivalent to Brownian motion — particles diffusing — with time run backwards and in very high-dimensional space. The physics isn't a metaphor; it hands you real algorithms.

02 · CLIP

Training text and images into one space

In February 2021 OpenAI released CLIP, trained on 400 million image-caption pairs scraped from the internet. It's two models — one encodes text, one encodes images — and each emits a length-512 vector. The training objective: a picture and its caption should produce similar vectors. Across a batch, arrange image vectors as columns and text vectors as rows; the diagonal holds the matching pairs.

A cat, a dog, and a man, each run through the image encoder; their captions run through the text encoder. Maximize similarity down the diagonal, minimize it everywhere off it.00:04:26

What the C stands forContrastive. The model learns by contrasting matching pairs against non-matching ones, measuring similarity as the cosine of the angle between vectors — same direction, score of one.

03 · Embedding space

Doing arithmetic on ideas

The learned geometry has a strange property. Take a photo of yourself wearing a hat and one without, encode both, and subtract. Then search a few hundred common words for the text vector closest to that difference. The top match, at cosine similarity 0.165, is hat — followed by cap and helmet.

The difference vector between "me with a hat" and "me without" is a real direction in the embedding space — and it points at the word hat.00:06:49

The learned geometry of CLIP's embedding space allows us to operate mathematically on the pure ideas or concepts in our images and text.— Welch Labs

But CLIP only runs one direction: it maps images and text into the space. It can't generate anything from a vector. That's the missing half.

04 · DDPM

The two things that surprised everyone

A few weeks after the GPT-3 paper, a Berkeley team published Denoising Diffusion Probabilistic Models — DDPM — the first method to generate genuinely high-quality images by reversing a noising process. The intuitive version (train the model to undo one noise step at a time) does not work, and almost no modern model does it that way.

The DDPM paper's generated faces. The algorithms behind them break two intuitions at once.00:08:36

Predict the total noiseThe model isn't trained to reverse one step. It takes a clean image x0, adds scaled noise epsilon, and learns to predict all the noise at once — effectively jumping straight back to the original.
Add noise while generatingAt each generation step, after the network predicts a cleaner image, you add fresh random noise before feeding it back. Remove that one line of code and a prompt for a tree returns a tiny, sad, blurry tree.

05 · Vector fields

The toy spiral that explains it

Shrink images to two pixels and each one becomes a point on a scatter plot. Lay the training data out in a spiral. Adding noise to an image is a random step; adding lots of noise sends each point on a random walk away from the spiral — exactly the Brownian motion that gives diffusion models their name.

A cat being noised step by step, mapped onto a 2D toy dataset: each "+Noise" is a step in the random walk away from the spiral.00:13:13

Asking the model to reverse the walk means: given a point far from the spiral, point back toward it. Across many walks through the same neighborhood the random directions average out, and a clear signal — pointing home — remains.

06 · Score function

Why the random noise matters

Training the model to predict the total noise (the vector straight back to x0) instead of a single step turns out to be mathematically equivalent — but with far lower variance, so the model learns much faster. What it learns, for every point and every noise level, is a direction pointing toward more likely, less noisy data: the score function.

The learned vector field. Every arrow points back toward the spiral; a single generated point follows it home.00:18:14

Why noise helps, not hurtsThe model only learns the mean of the reverse distribution. To actually sample from that distribution you have to add back zero-mean Gaussian noise at each step — which is exactly what DDPM does. Skip it and every point collapses to the average of the data, and in image space, averages look blurry.

07 · DDIM

Deterministic, in far fewer steps

DDPM's cost was the large number of steps, each a full pass through a big network. A pair of papers from Stanford and Google Brain showed you can drop the random noise entirely. Write the DDPM process as a stochastic differential equation, then use the Fokker-Planck equation from statistical mechanics to find an ordinary differential equation — no random term — that yields the same final distribution.

DDIM's deterministic trajectories: a cloud of points follows the vector field's contours and lands cleanly on the spiral, no random kicks.00:24:16

The catch and the winDDIM doesn't change training at all, and produces high-quality images in far fewer steps, completely deterministically. The theory only promises the same distribution of outputs, not the same individual images. Wan uses a generalization of this called flow matching.

08 · Conditioning

Reversing CLIP to obey a prompt

CLIP and diffusion fit together. A diffusion model can invert the CLIP image encoder to generate images; the CLIP text encoder's output can steer that generation toward what you asked for. In 2022 OpenAI did exactly this — unCLIP, better known as DALL·E 2 — feeding the text vector in as an extra input so the model uses it to denoise more accurately.

The unCLIP / DALL·E 2 paper. Feed the text vector in — via cross-attention, by appending it, or both — and the model learns to denoise with the prompt as context.00:26:24

But conditioning alone falls short. Prompt Stable Diffusion (the open-source contemporary from Heidelberg) for a tree in the desert with conditioning only, and you get a desert and a shadow — no tree.

09 · Classifier-free guidance

Making the tree actually grow

The fix: train the model to sometimes see no class or text at all, so it can produce an unconditioned vector field pointing at the data in general. Subtract that from the conditioned field, and the difference isolates the direction toward what you asked for. Amplify it by a scaling factor alpha.

As the guidance scale alpha climbs from 1 to ~25, the prompted tree literally appears and grows in size and detail.00:33:24

As we use guidance to point our model's vector field more in the direction of our prompt, our tree literally grows in size and detail in our images.— Welch Labs

Wan pushes this further with a negative prompt — writing out the features it doesn't want (extra fingers, walking backwards), passed to the encoder in Chinese, then subtracted and amplified to steer away.

10 · The whole machine

Three pieces that shouldn't fit

CLIP for a shared space of ideas, diffusion for generation, guidance to steer — three ideas from three groups, built largely from simple geometric intuitions that somehow survive into spaces of staggering dimension.

The three pillars: CLIP, diffusion models, prompt guidance.00:34:42

The resulting models feel like a fundamentally new class of machine. To create incredibly lifelike images and video, you no longer need a camera. All you need is language.— Welch Labs