How is layer normalization applied in diffusion models?

Layer normalization is applied in diffusion models to stabilize training by normalizing activations within neural network layers, particularly in the iterative noise prediction process. Diffusion models work by gradually adding noise to data and then learning to reverse this through a series of denoising steps. The core architecture, often a U-Net, processes noisy inputs at each step, and layer normalization helps maintain consistent signal scales across these steps. Unlike batch normalization, which depends on batch statistics, layer normalization uses per-sample statistics, making it suitable for scenarios with varying noise levels and small batch sizes common in diffusion training.

In practice, layer normalization is integrated into the residual blocks of the U-Net architecture. For example, in each downsampling or upsampling block, normalization is applied before the convolutional layers or attention mechanisms. A typical sequence might involve: (1) a convolutional layer, (2) layer normalization, (3) a non-linear activation (e.g., SiLU), and (4) a skip connection. In transformer-based diffusion variants, layer normalization is applied before multi-head self-attention or feed-forward layers to standardize inputs. Time step embeddings, which condition the model on the current denoising step, are often injected after normalization to avoid disrupting the normalized feature statistics. This placement ensures stable gradient flow during backpropagation across hundreds or thousands of denoising steps.

The primary benefit of layer normalization in diffusion models is its ability to handle varying noise magnitudes across timesteps. For instance, early denoising steps process heavily noised data, while later steps work with nearly clean data. Layer normalization adapts to these shifts without relying on batch-wide statistics, which could be inconsistent. This contributes to faster convergence and more stable training compared to alternatives like batch normalization. While some diffusion implementations (e.g., DDPM) use group normalization for computational efficiency, layer normalization remains a flexible choice, especially in architectures combining convolutional and attention layers, where feature scales can vary significantly between components.