Milvus is an open-source project, and hence is free-of-charge.
Please adhere to Apache License 2.0, when using Milvus for reproduction or distribution purposes.
No, it does not.
Yes. To update a vector, you can delete it and then insert a new one.
By deploying Mishards, a cluster sharding middleware for Milvus, you can process datasets of up to a 100-billion scale.
Vectors imported into Milvus are stored locally at milvus/db/tables/.
Metadata can be stored in either MySQL or SQLite. See Manage Metadata with MySQL for more information.
Milvus stores vectors and indexes directly in the disk as files, not in SQLite or MySQL. It uses SQLite or MySQL to store metadata of the vectors instead.
No, we only support storing metadata using SQLite or MySQL.
Python SDKs corresponding to Milvus v0.9.0 or later have a connection pool. There is no upper limit on the default number of connections in a connection pool.
As of Milvus v0.7.0, we have provided Milvus Enterprise Manager as a graphical tool for managing Milvus.
We do not have a dedicated tool as yet. You can call
get_entity_by_id to get the intended vectors by ID.
Milvus stores and processes each dimension of a vector in single-precision floating-point format (accurate to seven decimal places). Therefore, if the original format of each dimension is double-precision floating-point (accurate to sixteen decimal places), you will see a precision loss.
Either way is fine. But please note that entity IDs in the same collection must be either user-generated or Milvus-generated. Can't be both.
Yes, you can. If you insert vectors with an existing ID, you would end up having duplicate IDs.
Entity IDs must be non-negative 64-bit integers.
Vectors inserted each time must not exceed 256 MB.
Why is the
top1 result of a vector search not the search vector itself, if the metric type is inner product?
This occurs if you have not normalized the vectors when using inner product as the distance metric.
Does the size of a collection affect vector searches in one of its partitions, especially when it holds up to 100 million vectors?
No. If you have specified partitions when conducting a vector search, Milvus searches the specified partitions only.
Does Milvus load the whole collection to the memory if I search only certain partitions in that collection?
No, Milvus only loads the partitions to search.
Yes. But the parallelism processing mechanism varies with Milvus versions.
Suppose a collection has multiple segments, then when a query request comes in:
CPU-only Milvus processes the segment reading tasks and the segment searching tasks in pipeline.
On top of the abovementioned pipeline mechanism, GPU-enabled Milvus distributes the segments among the available GPUs.
See How Does Milvus Schedule Query Tasks for more information.
It depends on your scenario. See How to Choose an Index in Milvus for more information.
No. A collection can have only one index type at a time.
No. Although a collection can hold various types of data, the same collection can use only one index type.
Yes. When the inserted vectors grow to a specified volume, Milvus creates a new segment and starts to create an index file for it at the same time. The building of the new index file does not affect the existing index files.
No, they have the same recall rate for the same dataset.
IVF_FLAT index divides a vector space into
nlist clusters. If you keep the default value of
nlist as 16384, Milvus compares the distances between the target vector and the centers of all 16384 clusters to get
nprobe nearest clusters. Then Milvus compares the distances between the target vector and the vectors in the selected clusters to get the nearest vectors. Unlike IVF_FLAT, FLAT directly compares the distances between the target vector and each and every vector.
Therefore, when the total number of vectors approximately equals
nlist, IVF_FLAT and FLAT has little difference in the way of calculation required and search performance. But as the number of vectors grows to two times, three times, or n times of
nlist, IVF_FLAT index begins to show increasingly greater advantages.
See How to Choose an Index in Milvus for more information.
Why do I see a surge in memory usage when conducting a vector search immediately after an index is created?
This is because:
Milvus loads the newly created index file to the memory for the vector search.
The original vector files used to create the index are not yet released from the memory, because the size of original vector files and the index file has not exceeded the upper limit specified by
No, you cannot.
Milvus automatically flushes data to disk at intervals of one second.
If I have set
preload_collection, does Milvus service start only after all collections are loaded to the memory?
Yes. If you have set
preload_collection in server_config.yaml, Milvus' service is not available until it loads all specified collections.
Milvus loads inserted data to the memory and automatically flushes data from memory to the disk at fixed intervals. You can call
flush to manually trigger this operation.
We recommend that you configure write nodes to using GPU-enabled Milvus and read nodes to using CPU-only Milvus. If you can have only one write node, you can configure this node to using GPU-enabled Milvus for creating indexes and configure read nodes to using CPU-only Milvus.
No, it does not.
Normalization refers to the process of converting an embedding (vector) so that its norm equals 1. If you use Inner Product to calculate embeddings similarities, you must normalize your embeddings. After normalization, inner product equals cosine similarity.
See Wikipedia for more information.
Why do I get different results using Euclidean distance (L2) and inner product (IP) as the distance metric?
Check if the vectors are normalized. If not, you need to normalize the vectors first. Theoretically speaking, similarities worked out by L2 are different from similarities worked out by IP, if the vectors are not normalized.
There is no limit on the number of collections. The upper limit on the number of partitions in a collection is 4096.
Among the indexes that Milvus supports, IVF_FLAT and IVF_SQ8 implement the k-means clustering method. A data space is divided into
nlist clusters and the inserted vectors are distributed to these clusters. Milvus then selects the
nprobe nearest clusters and compares the distances between the target vector and all vectors in the selected clusters to return the final results.
k are large and
nprobe is small, the amount of vectors in the
nprobe clusters may be less than
k. Therefore, when you search for the
topk nearest vectors, the number of returned vectors is less than k.
To avoid this, try setting
nprobe larger and
See Index Types for more information.
Milvus can support vectors with up to 32,768 dimensions.
Data inserted to Milvus is first written into memory. This limit is to avoid over-occupation of memory resources.
You can add the following section to the configuration file server_config.yaml:
engine_config: search_combine_nq: 1