How do diffusion models compare to score-based generative models?

Diffusion models and score-based generative models are closely related approaches for generating data like images or audio. Both work by iteratively refining random noise into structured outputs, but they differ in their mathematical frameworks and training objectives. Diffusion models focus on learning a step-by-step process to reverse the gradual addition of noise to data. For example, in image generation, a diffusion model might start with a noisy image and progressively “denoise” it over many steps. Score-based models, on the other hand, learn to estimate the gradient (or “score”) of the data distribution, which guides the generation process by indicating how to adjust random noise to align it with real data patterns.

The primary technical difference lies in how they model the data. Diffusion models explicitly define a forward process (adding noise over time) and a reverse process (removing noise). They train a neural network to predict the noise at each step, minimizing the difference between predicted and actual noise. Score-based models avoid modeling the noise directly and instead train a network to estimate the score—the direction in which data points should move to match the training distribution. For instance, a score-based model might use Langevin dynamics, a sampling technique that uses the score to iteratively refine random noise into plausible data by following the gradient of the data distribution. While both approaches involve iterative refinement, diffusion models are more prescriptive in their noise schedule, while score-based methods are more flexible in how they traverse the data space.

In practice, the two approaches share many similarities and can even be unified under a single framework. For example, recent work has shown that diffusion models can be viewed as a specific case of score-based models when the noise schedule is constrained. Tools like Denoising Diffusion Probabilistic Models (DDPM) and Noise-Conditional Score Networks (NCSN) illustrate this overlap: DDPMs predict noise, while NCSNs predict scores, but both rely on iterative denoising. Developers might choose diffusion models for their simplicity in implementation (e.g., fixed noise schedules) or score-based methods for their flexibility in handling complex data manifolds. Libraries like PyTorch and TensorFlow support both approaches, and performance often depends on details like the choice of network architecture or sampling steps. Ultimately, the choice between them may come down to the specific use case and the desired balance between theoretical elegance and practical tuning.