How do diffusion models handle different types of noise during sampling?

Diffusion models handle different types of noise during sampling by iteratively refining a noisy input into a structured output, guided by a trained neural network. The process involves reversing a predefined forward noising process, where the model learns to predict and remove noise at each step. At every sampling iteration, the model estimates the noise present in the data and subtracts it, gradually transforming random noise into a coherent sample. This is done using a predefined noise schedule, which dictates how much noise is present at each timestep. The model uses this schedule to adjust its predictions based on the current noise level, ensuring it operates correctly across varying stages of the denoising process.

The noise schedule plays a critical role in determining how the model handles different noise magnitudes. For example, some implementations use a linear schedule, where noise decreases uniformly over time, while others use a cosine schedule, which slows noise reduction early and late in the process. These schedules are integrated into the model’s training, allowing it to condition its predictions on the current timestep. For instance, a U-Net architecture might use embeddings of the timestep to modulate its layers, ensuring each step’s operations align with the expected noise level. This conditioning enables the model to adapt its behavior—predicting coarse structure in early high-noise steps and fine details in later low-noise steps. The choice of schedule impacts sample quality and speed, with some schedules enabling fewer sampling steps without sacrificing results.

During sampling, the model also relies on specific algorithms to manage stochasticity and determinism. For example, the DDPM sampler adds a small amount of random noise at each step to explore the data distribution, while DDIM uses a deterministic approach by fixing the noise, enabling faster sampling with fewer steps. Developers can choose between these methods based on their needs: stochastic samplers often produce diverse samples but require more steps, while deterministic samplers trade some diversity for efficiency. Additionally, techniques like classifier guidance or latent space adjustments allow fine-tuning noise handling by incorporating external signals (e.g., class labels) to steer the denoising process. These mechanisms ensure diffusion models remain flexible in managing noise, balancing speed, quality, and control depending on the use case.