How is query throughput (QPS, queries per second) measured for vector search, and what factors most directly impact achieving a high QPS in a vector database?

Measuring QPS in Vector Search Query throughput (QPS) for vector search is measured by benchmarking how many search requests a system can process per second under realistic conditions. This involves sending a stream of queries to the database and tracking the total completed within a one-second window. For accuracy, tests typically use representative query volumes, dataset sizes (e.g., 1M–100M vectors), and vector dimensions (e.g., 256–1024 dimensions). Tools like FAISS benchmarks or custom load-testing scripts simulate real-world scenarios, including varying query complexity (exact vs. approximate search) and filtering constraints. For example, a test might measure QPS when searching a 10M-vector dataset with 512-dimensional embeddings using an HNSW index, while logging latency percentiles to ensure consistent performance.

Key Factors Impacting High QPS The most direct factors are indexing algorithms, hardware resources, and query design. Index structures like HNSW, IVF, or disk-based ANN indices balance speed and accuracy differently. For instance, HNSW prioritizes low latency but requires more memory, while IVF scales better with distributed clusters. Hardware-wise, GPUs accelerate vector computations for large batches, and fast memory (e.g., NVMe SSDs) reduces I/O bottlenecks. Query design also matters: filtering by metadata before vector search (e.g., “product category = electronics”) reduces the search space. Additionally, tuning parameters like the number of probes (IVF) or search depth (HNSW) directly affects QPS. For example, increasing IVF probes from 10 to 20 might improve recall but lower QPS by 30%.

Optimization Trade-offs and System Design Achieving high QPS often involves trade-offs between accuracy, latency, and resource usage. Sharding data across nodes allows parallel query execution but adds network overhead. Batch processing (e.g., handling 100 queries at once) improves GPU utilization but increases latency for individual queries. Caching frequently accessed vectors in RAM or using quantization (e.g., 8-bit floats instead of 32-bit) reduces memory footprint but may lower precision. For example, a hybrid approach might use IVF to cluster vectors on disk, cache 20% of hot data in memory, and deploy GPUs for batched queries, achieving 5,000 QPS with 95% recall. The optimal setup depends on specific use cases—e-commerce recommendations might prioritize high QPS with moderate accuracy, while medical diagnostics could favor precision over speed.