What is the difference between discrete and continuous diffusion models?

Discrete and continuous diffusion models are two approaches to generating data by iteratively adding and removing noise. The key difference lies in the type of data they handle and how they model the diffusion process. Discrete diffusion models operate on data with distinct, separate states, such as text tokens, categorical variables, or binary values. Continuous diffusion models, on the other hand, work with real-valued data like images, audio, or sensor readings, where values exist on a smooth spectrum. This distinction shapes how noise is applied and reversed during training and sampling.

In discrete diffusion, the process involves transitioning between discrete states over time. For example, in text generation, a token might be randomly masked or replaced with another token at each diffusion step. These transitions are governed by a predefined schedule or transition matrix that specifies probabilities for moving between states (e.g., keeping a token, replacing it, or masking it). A classic example is the “mask-predict” approach used in language models, where tokens are gradually revealed during sampling. Discrete models often rely on techniques like absorbing states (where tokens are masked) or uniform transitions between vocabulary items. In contrast, continuous diffusion models add Gaussian noise to data incrementally, following a predefined noise schedule. For instance, an image pixel’s value might be perturbed by small amounts of noise at each step until it becomes pure noise. During sampling, the model learns to reverse this process by predicting and subtracting the noise at each step. This approach is commonly used in models like DDPM (Denoising Diffusion Probabilistic Models) for image generation.

The practical implications for developers revolve around implementation and use cases. Discrete models are better suited for tasks involving categorical data, such as text generation or tabular data imputation. They often require fewer computational resources for certain operations (e.g., lookup tables for transition probabilities) but may struggle with high-dimensional data due to the combinatorial explosion of possible states. Continuous models excel in domains like image or audio synthesis, where gradients can be computed efficiently through neural networks. However, they demand careful tuning of noise schedules and may require more training data to learn complex distributions. For example, Stable Diffusion uses a continuous approach to generate high-resolution images by modeling pixel-level noise, while discrete models like D3PM (Discrete Denoising Diffusion Probabilistic Models) are applied to text by diffusing token-level changes. Developers should choose based on data type, scalability needs, and the trade-offs between interpretability (discrete transitions) and flexibility (continuous gradients).