Why is tail latency (p95/p99) often more important than average latency for evaluating the performance of a vector search in user-facing applications?

Tail latency (p95/p99) is prioritized over average latency in user-facing vector search applications because it directly reflects the worst-case user experience. Average latency smooths over outliers, but p95 and p99 capture the slowest 5% and 1% of requests, which are critical for applications where consistency matters. For example, if a recommendation system has an average latency of 50ms but a p99 of 2 seconds, 1% of users will experience noticeable delays, leading to frustration or abandonment. In contrast, optimizing for p95/p99 ensures that even under imperfect conditions—like traffic spikes or hardware variability—the majority of users receive fast, predictable responses.

Vector search workloads are inherently variable, making tail latency a better indicator of real-world performance. Unlike simple key-value lookups, vector searches involve computationally heavy operations like nearest-neighbor searches in high-dimensional spaces. These operations can vary widely based on query complexity, data distribution, or indexing strategies. For instance, a hierarchical navigable small world (HNSW) index might perform well for most queries but occasionally traverse suboptimal paths, causing sporadic delays. Similarly, hardware factors like cache misses or background processes on a server can unpredictably slow down a small fraction of requests. By focusing on p95/p99, developers identify and address these edge cases, such as optimizing index traversal or isolating resource-intensive workloads, which average metrics might overlook.

User-facing applications also demand strict service-level agreements (SLAs) for reliability. For example, an e-commerce site using vector search for product recommendations can’t afford even 1% of users waiting seconds for results during peak shopping periods. Tail latency metrics help teams set realistic SLAs and design systems that handle load gracefully. Techniques like request hedging (sending duplicate requests to multiple nodes and using the first response) or sharding data to reduce index size per node are often employed to mitigate tail latency. By measuring and optimizing p95/p99, developers ensure that performance improvements translate to better user retention and satisfaction, rather than just statistical averages that don’t reflect real-world usage patterns.