What are the common benchmarks used to evaluate zero-shot learning models?

Zero-shot learning (ZSL) models are evaluated using benchmarks designed to test their ability to generalize to unseen classes. Common datasets include CUB-200-2011 (Caltech-UCSD Birds), SUN (Scene Understanding), and AWA2 (Animals with Attributes 2). These datasets split classes into “seen” (used during training) and “unseen” (used only during testing). For example, CUB-200-2011 contains 200 bird species, with 150 seen and 50 unseen classes, and provides detailed attribute annotations (e.g., wing color) to link visual features to class descriptions. AWA2 includes 50 animal classes (40 seen, 10 unseen) with 85 attributes per class, such as habitat or fur texture. SUN covers 717 scene categories (645 seen, 72 unseen), focusing on contextual relationships. These datasets emphasize fine-grained distinctions, making them challenging for models to generalize without overfitting to training data.

Evaluation protocols vary, but most benchmarks follow two settings: traditional ZSL (testing only on unseen classes) and generalized ZSL (GZSL) (testing on both seen and unseen). Traditional ZSL uses top-1 accuracy on unseen classes, while GZSL employs the harmonic mean of seen and unseen accuracies to balance performance. For instance, in AWA2, a model might achieve 70% accuracy on unseen classes in traditional ZSL but drop to 40% in GZSL due to bias toward seen classes. Standardized splits, like those introduced by Xian et al., prevent data leakage by ensuring unseen classes are excluded from training, validation, or hyperparameter tuning. This standardization allows fair comparisons across methods.

Beyond image classification, benchmarks like Zero-Shot ImageNet (ZS-IMNET) test scalability by using subsets of ImageNet classes (e.g., 1,000 seen and 20,000 unseen). Text-based ZSL tasks, such as CLIP-style evaluations, use text prompts (e.g., “a photo of a zebra”) to align images and textual descriptions. Semantic representations like Word2Vec or GloVe embeddings are often used to encode class relationships in NLP-focused ZSL tasks (e.g., zero-shot text classification). These benchmarks stress the model’s ability to leverage auxiliary information (attributes, text) to bridge seen and unseen classes, ensuring robustness in real-world scenarios where new categories emerge frequently.