I'd like to learn more about the diffusion process underlying Stable Diffusion. In principle, what mathematical steps does the image generation process go through? For example, how is the noise addition/removal cycle optimized? Could you provide a detailed explanation?
What's the underlying logic behind diffusion models?
👁️ 8 views💬 3 replies❤️ 0 likes
3 Replies
Diffusion models operate based on the mathematical representation of the "diffusion" process in thermodynamics. Fundamentally, their goal—such as in Stable Diffusion—is to transform an initially pure noise image into a target image by gradually "cleaning" it step by step. This process consists of two main phases: **forward diffusion (noise addition)** and **reverse diffusion (noise removal)**. In the forward phase, the model gradually adds Gaussian noise to the image, while in the reverse phase, a convolutional neural network (usually U-Net-based) predicts and "reverses" this noise. The critical point here is that the model must learn to generate a *reverse* conditional probability distribution of the noise addition process.
Optimizing the noise addition/removal cycle involves **convolutional networks (CNNs) and attention mechanisms**. At each step, the model attempts to reach a "cleaner" version by subtracting the predicted noise from the previous noisy state. The loss function typically used is **mean squared error (MSE)**, though some variants employ different strategies to preserve noise diversity. For example, models like **DDPM (Denoising Diffusion Probabilistic Models)** balance image quality and diversity by adjusting the variance at each step. The success of optimization depends on how accurately the model predicts noise and how realistic the final image appears.
Some diffusion models, such as those in Stable Diffusion, introduce an additional layer: the noise addition/removal process operates in a lower-dimensional latent space. This reduces computational costs and helps the model better capture structural features of images. While latent space processing reduces the computational power needed for image reconstruction, it’s important to remember that the core noise addition/removal process follows the same principles. In short, the magic of diffusion models lies in their ability to generate complex patterns and details from simple noise.
Pff, last night when I tried to explain diffusion, all I got was image corruption 😅 Every time I'd ask, "Why is this model giving me mashed potatoes instead of a cat?" with my teammate. But as you said, getting that noise back is pure magic! I almost burned my bedsheets working on the loss function's gradient in my prototype 🔥
When I first started working with diffusion models, everything seemed like pure magic to me. The system could turn a black image into a realistic portrait overnight. The first confusing part was the term "diffusion" itself — contrary to what the name suggests, it’s actually a process that gradually removes noise rather than spreading it. For my first project, I started by adding noise to a simple MNIST digit and then removing it. Despite its simplicity, seeing the tensors gradually clean up with each step was fascinating.
As for optimization, the biggest difference came from properly tuning skip connections and learning rates. In the U-Net architecture, using tempered noise levels and "time embedding" helps the model understand how much noise to remove at each stage. From my experience, experimenting with PyTorch’s diffusers library was the fastest way to learn — I’m sure I picked up a lot from both the source code and hands-on practice.