Can someone explain how the latent diffusion process works inside Stable Diffusion? Specifically, how does the model compress images into a latent representation, apply the denoising steps there, and then decode back to pixel space? I'm curious about the benefits of operating in latent space versus raw pixel space, and how the training objectives differ. Any explanations or resources would be appreciated.
Understanding the role of the latent space in Stable Diffusion models
👁️ 109 views💬 3 replies❤️ 0 likes
3 Replies
I'm curious about the specifics of the VAE encoder—what constraints or loss terms guide it to retain certain features while discarding others when compressing an image into the latent space? Also, does the denoising UNet operate on the same latent dimensions throughout training, or are there any intermediate reshapes?
Stable Diffusion first compresses a 512 × 512 RGB image into a much smaller latent tensor (typically 4×4×64) using a pre-trained VAE encoder. The encoder learns a probabilistic mapping p(z|x) so that the latent z captures the high-level structure while discarding fine-grained pixel noise. The diffusion model then operates entirely in this latent space: it adds Gaussian noise to z over a fixed number of steps, trains a UNet to predict and subtract that noise, and finally produces a clean latent ẑ that should decode back to the original image. The decoder part of the VAE (the VAE decoder) takes ẑ and reconstructs the pixel-space picture, restoring the details that the encoder omitted.
Operating in latent space is similar to what people do with GANs like StyleGAN, where the generator works on a low-dimensional latent vector instead of raw pixels. The key advantage is efficiency—training and inference are roughly four times faster and require far less GPU memory because the UNet sees a 1/64-sized feature map instead of the full image. Moreover, the training objective is a standard denoising score-matching loss on the latent distribution, whereas a VAE-GAN hybrid usually adds an adversarial term to enforce realism. In practice, this means Stable Diffusion can generate high-resolution images with fewer diffusion steps while still preserving the flexibility of conditional generation (text prompts, inpainting, etc.) that pure GANs struggle with. If you're looking for a concrete comparison, check out the “latent diffusion vs pixel-space diffusion” section in the original paper—they show a 3–5× speedup with comparable fidelity.
Exactly, when I tried Stable Diffusion, I noticed that the encoder converts the image into a smaller latent representation, then the UNet applies denoising in this latent space, and finally the decoder brings it back to pixel space. Working in latent space reduces memory consumption and time, while the training objective is to minimize the MSE difference between the noisy representation and the original representation.