How do ANN benchmark datasets and evaluations account for different distance metrics? (Do they typically assume Euclidean distance, or do they evaluate algorithms under multiple metrics?)

ANN (Approximate Nearest Neighbors) benchmark datasets and evaluations often prioritize Euclidean distance (L2) as a default metric, but many frameworks and studies also test algorithms under multiple distance metrics to ensure broader applicability. While Euclidean is widely used due to its simplicity and prevalence in machine learning, benchmarks like ann-benchmarks and datasets such as SIFT-1M or GloVe include support for cosine similarity, Manhattan distance (L1), and other metrics. This reflects real-world scenarios where different applications—like text similarity (cosine) or geospatial data (Manhattan)—require varying distance measures. However, not all datasets or evaluations are explicitly designed to test all metrics, and some assume a default unless explicitly configured otherwise.

Datasets commonly used in ANN benchmarks are often preprocessed or selected to align with specific metrics. For example, the MNIST dataset (image data) is typically evaluated with Euclidean distance, while GloVe (word embeddings) is normalized for cosine similarity. The choice of metric can influence how data is indexed or preprocessed. For instance, cosine similarity requires vectors to be normalized to unit length, which effectively converts it to a Euclidean distance problem on a unit sphere. Benchmarks like ann-benchmarks handle this by allowing users to specify preprocessing steps (e.g., normalization) and distance metrics during evaluation. However, some datasets lack explicit guidance on metric choice, leaving it to developers to select the appropriate one based on their use case.

Evaluations of ANN algorithms often test multiple metrics to assess versatility. For example, FAISS (a popular ANN library) provides optimized indices for both L2 and inner product (related to cosine) distances, and benchmarks compare their performance across these metrics. Similarly, the HNSW algorithm allows swapping distance functions, enabling tests under L1, L2, or custom metrics. However, not all algorithms support every metric equally—some are optimized for specific ones. This means benchmarks must clearly report which metrics were tested and how they impacted results. Developers should verify whether a benchmark’s evaluation aligns with their target metric, as algorithm performance can vary significantly. For instance, a method optimized for L2 might underperform on Manhattan distance due to differences in how distances are calculated and indexed.