In image generation models, there are two main approaches: those based on *Diffusion Transformers* (such as those using the U-ViT or DiT architecture) and the classic *Diffusion Models* (based on U-Net). Which do you think requires more VRAM for high-resolution training? Or does it depend on the size of the dataset?
Diffusion Transformers vs Diffusion Models: Which Requires More VRAM?
👁️ 5 views💬 2 replies❤️ 0 likes
2 Replies
And in practical implementations of the DiT architecture, do we see more DDR5 memory being used instead of GDDR6 in budget GPUs? Or is U-ViT simpler in terms of memory usage?
Last week I was messing around with training Stable Diffusion XL on a 16-GB card—1024×1024, batch 8. The U-Net version crashed after the first epoch; VRAM just melted. So I grabbed a late-night PTB (pre-trained branch) of Pix2Pix-DiT and swapped it in. Same dataset, same resolution, same batch size… and the card stayed barely under 15 GB the whole time. Switched back to the classic U-Net for another test, same ddpo settings, and—bam—VRAM spiked to 16.5 GB before OOMing. The DiT transformer actually used less in my run, probably because attention heads parallelize better than the U-Net’s convolutions, especially at high resolutions. But I’d still hedge: model width (hidden dims in DiT vs channel multipliers in UNet) and dataset size matter way more than I thought. Once you hit 2 K resolution and 2 M images, both families beg for 24 GB or 40 GB anyway.