How are noise sampling and latent space mapping implemented in Stable Diffusion?

08 Ağu 01:33

In Stable Diffusion, the default noise sampling follows a standard normal distribution \( \mathcal{N}(0,\mathbf{I}) \), meaning each latent dimension independently samples from an identical Gaussian distribution. During sampling, noise is gradually injected into the latent space according to a predefined noise schedule, typically determined by linear, cosine, or adaptive β-sequences. The implementation details can be found in `ldm/models/diffusion/ddpm.py`. The chosen β values directly influence the denoising magnitude at each step, with cosine schedules (as described in the paper *Improved Denoising Diffusion Probabilistic Models*) reducing the required steps while maintaining generation quality. The latent space dimensions are not arbitrary but are determined by the model's encoder. For example, in the official 1.5 version, the VAE encoder downsamples a 512×512 pixel image by a factor of 8, resulting in a 64×64 latent grid with 4×64 = 256 channels (i.e., `channel=4`, with height and width each divided by 8). This 4-channel structure originates from the *Latent Diffusion Models* paper, which projects noise into a relatively low-dimensional latent space to reduce computational load while preserving semantic information. For higher resolutions, you can adjust the VAE's downsample factor (e.g., changing it to 4×) or increase the channel count, though this significantly increases VRAM usage. The denoising step scheduling mechanism essentially functions as a sampler, with common options including DDIM, Euler-a, and DPM-solver. These samplers approximate the reverse diffusion process using different numerical integration methods on the same β-schedule, determining how noise is predicted and images are updated at each step. DDIM achieves good results in 25-50 steps through non-stochastic, step-by-step progression, while Euler-a retains randomness while providing smoother trajectories, making it suitable for scenes requiring finer details. Their implementations are located in `ldm/models/diffusion/ddim.py` and `ldm/models/diffusion/euler.py`, respectively, where interactions between `sample_loop` and `predict_x0` should be noted. For a deeper dive into implementation details, consider the following approach: 1. Read the original paper *High-Resolution Image Synthesis with Latent Diffusion Models* (including formula derivations). 2. Explore the official CompVis GitHub (`stable-diffusion` repository), focusing on the `autoencoder`, `unet`, and `sampler` modules. 3. Refer to HuggingFace Diffusers' documentation and annotations, particularly the implementations of `NoiseScheduler` and `SchedulerMixin`. 4. If interested in scheduling strategies, experiment with comparisons between "cosine-beta schedule" and "scaled-linear schedule," observing changes in PSNR/SSIM across different sampling steps. This approach ensures a strong grasp of theory while validating hypotheses at the code level.