Milvus 2.1 Benchmark Test Report

This report shows the major test results of Milvus 2.1, covering the performances of data insertion, index building, and vector similarity search. The tests aim to provide a benchmark against which the performances of future Milvus releases can be measured.

Terminology

Click to see the details of the terms used in the test

Term	Description
nq	Number of vectors to be searched in one search request
topk	Number of the nearest vectors to be retrieved for each vector (in nq) in a search request
ef	A search parameter specific to HNSW index
RT	Response time from sending the request to receiving the response
QPS	Number of search requests that are successfully processed per second

Test environment

All tests are performed under the following environments.

Hardware environment

Hardware	Specification
CPU	Intel(R) Xeon(R) Gold 6226R CPU @ 2.90GHz
Memory	16*\32 GB RDIMM, 3200 MT/s
SSD	SATA 6 Gbps

Software environment

Software	Version
Milvus	2.1.0-20220729-dcd6c9e5 2.0.2-20220401-898533c5
Milvus GO SDK	v2.1.1-0.20220801085923-509bbbbc89eb

Deployment scheme

Milvus instance (standalone or cluster) in each test is deployed via Helm on a Kubernetes cluster based on physical or virtual machines.
Configurations of the tested Milvus instances merely vary in the number of CPU cores, the size of memory, and the number of replicas (worker nodes), which only applies to Milvus cluster.
Unspecified configurations are default configurations.
Milvus dependencies (MinIO, Pulsar, and etcd) store data on the local SSD in each node.
Search requests are sent to the Milvus instances via Milvus GO SDK.

Data sets

The test uses the open source dataset SIFT (128 dimensions) from ANN-Benchmarks.

Test pipeline

Start a Milvus instance by Helm with respective server configurations as listed in each test.
Connect to the Milvus instance via Milvus GO SDK and get the corresponding test results.
Create a collection.
Insert 1 million SIFT vectors. Build an HNSW index and configure the index parameters by setting M=8 and efConstruction=200.
Load the collection.

Test results

The test conducts a concurrent search on the the prepared data in the created collection and records the search performance metrics. The test result shows:

When search parameters nq=1, topk=1, ef=64, search concurrency is 400 and the duration of concurrency is 5 hours.

The detailed performance test results of Milvus 2.1 cluster, standalone, and Milvus 2.0.2 standalone is shown below.

Milvus 2.1 cluster

Server configurations (cluster)

image:
  all:
    tag: 2.1.0-20220729-dcd6c9e5
queryNode:
  replicas: 1
  resources:
    limits:
      cpu: "12.0"
      memory: 8Gi

Search performance

QPS	RT(TP99) / ms	RT(TP95) / ms	fail/s
6904	59	58	0

Milvus 2.1 standalone

Server configurations (standalone)

image:
  all:
    tag: 2.1.0-20220729-dcd6c9e5
standalone:
  replicas: 1
  resources:
    limits:
      cpu: "12.0"
      memory: 16Gi

Search performance

QPS	RT(TP99) / ms	RT(TP95) / ms	fail/s
4287	104	103	0

Milvus 2.0.2 standalone

Server configurations (standalone)

image:
  all:
    tag: 2.0.2-20220401-898533c5
standalone:
  replicas: 1
  resources:
    limits:
      cpu: "12.0"
      memory: 16Gi

Search performance

QPS	RT(TP99) / ms	RT(TP95) / ms	fail/s
658	756	748	0

Summary

2.1_qps — A comparison of the QPS test results.

2.1_rt — A comparison of the RT test results.

In the current test scenario, the QPS of Milvus cluster is better than that of Milvus standalone under 1 million dataset.
In the current test scenario, the QPS of Milvus 2.1 standalone is better than that of Milvus 2.0.2 standalone under 1 million dataset.

What's next

Learn how to perform a Milvus 2.1 benchmark by yourself.