What are some common datasets used to benchmark diffusion models?

Diffusion models are often benchmarked using established datasets that test their ability to generate high-quality, diverse outputs. Three categories of datasets are commonly used: general-purpose image datasets, text-conditioned datasets, and specialized or high-resolution collections. Each serves distinct evaluation purposes, from measuring basic generative performance to testing conditional synthesis and scalability.

First, general-purpose image datasets like CIFAR-10, ImageNet, and LSUN are widely used. CIFAR-10, with 60,000 32x32-pixel images across 10 classes, is a lightweight option for quick experimentation. Its low resolution makes it practical for testing architectural ideas or training efficiency. ImageNet, containing 1.3 million higher-resolution images (typically downscaled to 256x256 for training) across 1,000 classes, evaluates a model’s ability to handle diversity and scale. LSUN (Large-scale Scene Understanding) focuses on specific scene categories like bedrooms or churches, testing how well models generate structured, complex layouts. These datasets are paired with metrics like Fréchet Inception Distance (FID) or Inception Score (IS), which compare generated images to real data using pre-trained classifiers.

For text-to-image tasks, MS-COCO and subsets of LAION-5B are common benchmarks. MS-COCO includes 330,000 images with human-annotated captions, enabling evaluation of how well models align generated images with textual prompts. LAION-5B, a web-crawled dataset with 5 billion image-text pairs, is often used for training large models like Stable Diffusion, but smaller curated subsets (e.g., LAION-Aesthetics) are used for benchmarking to reduce computational costs. These datasets test a model’s ability to handle diverse prompts and generate semantically consistent outputs. Evaluation often involves metrics like CLIP Score, which measures the alignment between generated images and input text using a multimodal embedding model.

Finally, high-resolution or domain-specific datasets like FFHQ (Flickr-Faces-HQ) and CelebA-HQ are used to test scalability and specialization. FFHQ contains 70,000 high-quality face images at 1024x1024 resolution, challenging models to generate fine details like hair and skin texture. CelebA-HQ, a refined version of the original CelebA dataset, includes 30,000 aligned face images and is often used for tasks like attribute-conditional generation (e.g., adding glasses or changing hair color). These datasets push models to handle larger dimensions and domain-specific features. Benchmarks here might focus on perceptual quality (via user studies) or task-specific metrics, such as accuracy in preserving requested attributes during editing. Together, these datasets provide a comprehensive framework for evaluating diffusion models across tasks and complexities.