How do you design the neural network for the reverse diffusion step?

Designing a neural network for the reverse diffusion step involves creating a model that can iteratively remove noise from data by learning the structure of the target distribution. The core idea is to train a network to predict the noise or the clean data at each step of the denoising process. This is typically done using a U-Net architecture, which is well-suited for capturing both local and global context through its encoder-decoder structure with skip connections. The network takes a noisy input and a timestep (indicating the current step in the diffusion process) and outputs an estimate of the noise or the clean data. Key components include residual blocks, attention mechanisms, and time embedding layers to condition the model on the current diffusion step.

The U-Net’s encoder reduces spatial resolution while increasing feature depth, capturing high-level patterns. The decoder then reconstructs the data by upsampling and combining features with skip connections from the encoder, preserving fine-grained details. For example, in image generation, each block might consist of convolutional layers, group normalization, and a SiLU activation. Timestep conditioning is achieved by projecting the timestep into an embedding vector, which is added to the feature maps at each layer. Attention layers, such as self-attention or cross-attention, are often inserted in the middle of the U-Net to model long-range dependencies. This setup allows the network to adapt its behavior based on the noise level (timestep) and focus on relevant structures in the data.

Practical considerations include balancing model capacity with computational efficiency. For instance, smaller U-Nets may suffice for low-resolution data, while high-resolution tasks require deeper architectures or techniques like checkpointing to manage memory. Training involves minimizing a loss function (e.g., mean squared error) between the predicted and actual noise. To improve stability, techniques like gradient clipping or exponential moving averages (EMA) of model weights are often used. For example, in DDPM (Denoising Diffusion Probabilistic Models), the network predicts the noise component at each step, and the loss is weighted uniformly across timesteps. Developers should also experiment with variance schedules (linear, cosine) for adding noise, as this impacts how the network learns to denoise incrementally. Testing with simplified datasets (e.g., MNIST) before scaling to complex data helps validate the design.