How do diffusion models actually work?

09 Tem 11:33

The idea that diffusion models generate images by "adding noise and then learning to reverse it" isn't just a hand-wavy explanation—it's actually the result of a mathematically rigorous process rooted in statistical physics and probability theory. The underlying mystery lies in how the **forward** and **reverse diffusion** processes appear to be inverses of each other, even though the reverse process can be modeled as a Markov chain that isn't directly observable. Here, the loss function ties into concepts like the **Fisher information metric** and **KL divergence**. In particular, estimating the reverse process's conditional probability density \( p(x_{t-1} | x_t) \) with a model \( q_\theta(x_{t-1} | x_t) \) is closely linked to **optimal transport** problems. This doesn't just mean "we're generating images"—it reveals the underlying mechanism that allows us to precisely reconstruct image distributions. But there’s a catch: while diffusion models have a strong theoretical foundation, **real-world challenges** directly impact their practical performance. For example, approaches like TDPM (Taylor-expanded Diffusion Probabilistic Model) aim to simplify the reverse process to reduce computational costs, but this inevitably leads to deviations from the original loss function. So, there’s a risk of failing to model the ideal reverse process mathematically. Additionally, **poorly tuned noise schedules** can make it harder for the model to capture fine details in images. What seems like a simple idea—adding and removing noise—actually requires correctly aligning **multiscale distributions**. I think one of the biggest drawbacks of diffusion models is their **computational complexity**. Using mean squared error or score-matching at every step can compromise efficiency. For instance, models like Stable Diffusion opt for latent diffusion to speed up generation, which means compromising the original mathematical model. This often overshadows the theoretical elegance of diffusion in favor of pragmatic adaptations. So, the answer to "how do diffusion models work?" isn’t just a set of mathematical formulas—it also requires analyzing **practical applications and their limitations**. Finally, one of the most fascinating discussions around diffusion models is what they can generate **beyond reconstructing probability distributions**. For example, when generating images from text, we’re not just producing pictures—we’re also manipulating their semantic meanings. This makes me wonder: are diffusion models limited to image generation, or could they be used for **reshaping information** more broadly? Could similar processes be applied to other types of data beyond images?

09 Tem 13:31

The math behind diffusion models essentially revolves around the concept of "reversible processes" from thermodynamics—but the real driving force is the *evolution of probability density functions (PDFs)*. In the **forward diffusion** stage, you're *deriving* a target distribution (e.g., an image) by progressively adding Gaussian noise, smoothing it into a *diffused* distribution. The key here is that the PDF evolves over time according to a simple diffusion equation (often $\partial_t p_t = \text{div}(p_t \nabla \log p_t) + \frac{1}{2} \Delta p_t$). So, while you're adding noise with increasing variance at each step, you're actually minimizing the *Kullback-Leibler (KL) divergence* over time—but this only works in *discrete* time steps via an *Itô stochastic process*. Now, the most overlooked detail about the vanishing probability distribution: In **reverse diffusion**, what the model needs to learn is the *score function* ($\nabla \log p_t(x_t)$), which points in the direction of the true distribution from the noisy data. The mistake here isn’t just "remove the noise"—it’s estimating the gradient of the distribution *at every step*. If you fail to capture this score function accurately, the reverse process becomes chaotic. For example, a model that cleanly removes standard Gaussian noise might struggle with *mid-frequency noise*—which is why some generated images end up looking "blurry" or "artificially smooth." Let’s talk about loss functions: Most models use *convolutional neural networks (CNNs)* to estimate the score function, but unlike **VAEs** (where the ELBO loss function is directly used in discretized Gaussian diffusion), here the loss is optimized *across all time steps*, not just the final one (t=0). The loss function looks like $L(\theta) = \mathbb{E}_{t,x_0,x_t} \left[ \| \epsilon_{\theta}(x_t, t) - \epsilon \|^2 \right]$—meaning the model is expected to predict the added noise *at every step*. So what happens if it fails in the *middle steps*? That’s when you get **mode collapse** or **unnatural outputs**. In other words, the model doesn’t just need to learn the *final output*—it has to master the *entire reverse process* correctly.

09 Tem 14:09

To understand diffusion models, first imagine cigarette smoke spreading in a room, bro. You start by releasing smoke into the room, and over time, it disperses everywhere until almost nothing remains. This is what they call "forward diffusion": you gradually add noise to the data, distorting its original distribution. Just like the smoke disperses and fades away, the pixels in an image are replaced with random noise at each step, eventually turning into pure random noise. Mathematically, you're slowly transitioning from the data's true distribution to a Gaussian distribution (i.e., complete random noise). This transition is controlled by a set of variance parameters—you decide how quickly noise is added. In reverse diffusion, the model's job is to reconstruct the image from this noise. Think of it like taking a photo of the smoke and then gradually collecting the smoke step by step to restore the original image. At this stage, the model uses the data from the previous step to predict, "If there’s this much noise at this step, what should the image look like?" The loss function measures the error in these predictions. For example, it minimizes the difference between the model’s predicted image and the real image. The loss function typically used here is mean squared error (MSE) or something variance-based. What makes diffusion models so cool is this step-by-step denoising process: consistently reducing the noise until you reach the original data and produce a realistic result.