
[Milvus](https://milvus.io/intro) is an open-source vector database designed specifically for AI applications. Whether you're working on machine learning, deep learning, or any other AI-related project, Milvus offers a robust and efficient way to handle large-scale vector data.

Now, with the [model module integration](https://milvus.io/docs/embeddings.md) in PyMilvus, the Python SDK for Milvus, it's even easier to add Embedding and Reranking models. This integration simplifies transforming your data into searchable vectors or reranking results for more accurate outcomes, such as in [Retrieval Augmented Generation (RAG)](https://zilliz.com/learn/Retrieval-Augmented-Generation).

In this blog, we will review dense embedding models, sparse embedding models, and re-rankers and demonstrate how to use them in practice using [Milvus Lite](https://milvus.io/blog/introducing-milvus-lite.md), a lightweight version of Milvus that can run locally in your Python applications. 


## Dense vs Sparse Embeddings 

Before we walk you through how to use our integrations, let’s look at two main categories of vector embeddings. 

[Vector Embeddings](https://zilliz.com/glossary/vector-embeddings) generally fall into two main categories: [**Dense Embeddings** and **Sparse Embeddings**](https://zilliz.com/learn/sparse-and-dense-embeddings).

- Dense Embeddings are high-dimensional vectors in which most or all elements are non-zero, making them ideal for encoding text semantics or fuzzy meaning.

- Sparse Embeddings are high-dimensional vectors with many zero elements, better suited for encoding exact or adjacent concepts.

Milvus supports both types of embeddings and offers hybrid search. [Hybrid Search](https://zilliz.com/blog/hybrid-search-with-milvus) allows you to conduct searches across various vector fields within the same collection. These vectors can represent different facets of data, use diverse embedding models, or employ distinct data processing methods, combining the results using re-rankers.

## How to Use Our Embedding and Reranking Integrations

In the following sections, we’ll demonstrate three practical examples of using our integrations to generate embeddings and conduct vector searches. 

### Example 1: Use the Default Embedding Function to Generate Dense Vectors 

You must install the `pymilvus` client with the `model` package to use embedding and reranking functions with Milvus. 

```
pip install "pymilvus[model]"
```

This step will install [Milvus Lite](https://milvus.io/docs/quickstart.md), allowing you to run Milvus locally within your Python application. It also includes the model subpackage, which includes all utilities for Embedding and reranking.

The model subpackage supports various embedding models, including those from OpenAI, [Sentence Transformers](https://zilliz.com/learn/Sentence-Transformers-for-Long-Form-Text), [BGE-M3](https://zilliz.com/learn/bge-m3-and-splade-two-machine-learning-models-for-generating-sparse-embeddings), BM25, [SPLADE](https://zilliz.com/learn/bge-m3-and-splade-two-machine-learning-models-for-generating-sparse-embeddings), and Jina AI pre-trained models.

This example uses the `DefaultEmbeddingFunction`, based on the `all-MiniLM-L6-v2` Sentence Transformer model for simplicity. The model is about 70MB and will be downloaded during the first use:

```
from pymilvus import model

# This will download "all-MiniLM-L6-v2", a lightweight model.
ef = model.DefaultEmbeddingFunction()

# Data from which embeddings are to be generated
docs = [
   "Artificial intelligence was founded as an academic discipline in 1956.",
   "Alan Turing was the first person to conduct substantial research in AI.",
   "Born in Maida Vale, London, Turing was raised in southern England.",
]

embeddings = ef.encode_documents(docs)

print("Embeddings:", embeddings)
# Print dimension and shape of embeddings
print("Dim:", ef.dim, embeddings[0].shape)
```

The expected output should be something like the following: 

```
Embeddings: [array([-3.09392996e-02, -1.80662833e-02,  1.34775648e-02,  2.77156215e-02,
      -4.86349640e-03, -3.12581174e-02, -3.55921760e-02,  5.76934684e-03,
       2.80773244e-03,  1.35783911e-01,  3.59678417e-02,  6.17732145e-02,
...
      -4.61330153e-02, -4.85207550e-02,  3.13997865e-02,  7.82178566e-02,
      -4.75336798e-02,  5.21207601e-02,  9.04406682e-02, -5.36676683e-02],
     dtype=float32)]
Dim: 384 (384,)
```


### Example 2: Generate Sparse Vectors Using The BM25 Model

BM25 is a well-known method that uses word occurrence frequencies to determine the relevance between queries and documents. In this example, we’ll show how to use `BM25EmbeddingFunction` to generate sparse embeddings for queries and documents.

In BM25, it's important to calculate the statistics in your documents to obtain the IDF (Inverse Document Frequency), which can represent the patterns in your documents. The IDF measures how much information a word provides, whether it's common or rare across all documents.

```
from pymilvus.model.sparse import BM25EmbeddingFunction

# 1. Prepare a small corpus to search
docs = [
   "Artificial intelligence was founded as an academic discipline in 1956.",
   "Alan Turing was the first person to conduct substantial research in AI.",
   "Born in Maida Vale, London, Turing was raised in southern England.",
]
query = "Where was Turing born?"
bm25_ef = BM25EmbeddingFunction()

# 2. Fit the corpus to get BM25 model parameters on your documents.
bm25_ef.fit(docs)

# 3. Store the fitted parameters to expedite future processing.
bm25_ef.save("bm25_params.json")

# 4. Load the saved params
new_bm25_ef = BM25EmbeddingFunction()
new_bm25_ef.load("bm25_params.json")

docs_embeddings = new_bm25_ef.encode_documents(docs)
query_embeddings = new_bm25_ef.encode_queries([query])
print("Dim:", new_bm25_ef.dim, list(docs_embeddings)[0].shape)
```

### Example 3: Using a ReRanker 

A search system aims to find the most relevant results quickly and efficiently. Traditionally, methods like BM25 or TF-IDF have been used to rank search results based on keyword matching. Recent methods, such as embedding-based cosine similarity, are straightforward but can sometimes miss the subtleties of language and, most importantly, the interaction between documents and a query's intent.

This is where using a [re-ranker](https://zilliz.com/learn/optimize-rag-with-rerankers-the-role-and-tradeoffs) helps. A re-ranker is an advanced AI model that takes the initial set of results from a search—often provided by an embeddings/token-based search—and re-evaluates them to ensure they align more closely with the user's intent. It looks beyond the surface-level matching of terms to consider the deeper interaction between the search query and the content of the documents.

For this example, we’ll use the [Jina AI Reranker](https://milvus.io/docs/integrate_with_jina.md).



```
from pymilvus.model.reranker import JinaRerankFunction

jina_api_key = "<YOUR_JINA_API_KEY>"

rf = JinaRerankFunction("jina-reranker-v1-base-en", jina_api_key)

query = "What event in 1956 marked the official birth of artificial intelligence as a discipline?"

documents = [
   "In 1950, Alan Turing published his seminal paper, 'Computing Machinery and Intelligence,' proposing the Turing Test as a criterion of intelligence, a foundational concept in the philosophy and development of artificial intelligence.",
   "The Dartmouth Conference in 1956 is considered the birthplace of artificial intelligence as a field; here, John McCarthy and others coined the term 'artificial intelligence' and laid out its basic goals.",
   "In 1951, British mathematician and computer scientist Alan Turing also developed the first program designed to play chess, demonstrating an early example of AI in game strategy.",
   "The invention of the Logic Theorist by Allen Newell, Herbert A. Simon, and Cliff Shaw in 1955 marked the creation of the first true AI program, which was capable of solving logic problems, akin to proving mathematical theorems."
]

results = rf(query, documents)

for result in results:
   print(f"Index: {result.index}")
   print(f"Score: {result.score:.6f}")
   print(f"Text: {result.text}\n")
```

The expected output is similar to the following:

```
Index: 1
Score: 0.937096
Text: The Dartmouth Conference in 1956 is considered the birthplace of artificial intelligence as a field; here, John McCarthy and others coined the term 'artificial intelligence' and laid out its basic goals.

Index: 3
Score: 0.354210
Text: The invention of the Logic Theorist by Allen Newell, Herbert A. Simon, and Cliff Shaw in 1955 marked the creation of the first true AI program, which was capable of solving logic problems, akin to proving mathematical theorems.

Index: 0
Score: 0.349866
Text: In 1950, Alan Turing published his seminal paper, 'Computing Machinery and Intelligence,' proposing the Turing Test as a criterion of intelligence, a foundational concept in the philosophy and development of artificial intelligence.

Index: 2
Score: 0.272896
Text: In 1951, British mathematician and computer scientist Alan Turing also developed the first program designed to play chess, demonstrating an early example of AI in game strategy.
```

## Star Us On GitHub and Join Our Discord!

If you liked this blog post, consider starring Milvus on [GitHub](https://github.com/milvus-io/milvus), and feel free to join our [Discord](https://discord.gg/FG6hMJStWu)! 💙


Introducing PyMilvus Integration with Embedding Models


*This post was originally published on [Intel's Medium Channel](https://medium.com/intel-tech/optimize-vector-databases-enhance-rag-driven-generative-ai-90c10416cb9c) and is reposted here with permission.*


<br>

Two methods to optimize your vector database when using RAG


![](https://miro.medium.com/v2/resize:fit:1400/1*FRWBVwOHPYFDIVTp_ylZNQ.jpeg)


Photo by [Ilya Pavlov](https://unsplash.com/@ilyapavlov?utm_content=creditCopyText\&utm_medium=referral\&utm_source=unsplash) on [Unsplash](https://unsplash.com/photos/monitor-showing-java-programming-OqtafYT5kTw?utm_content=creditCopyText\&utm_medium=referral\&utm_source=unsplash)



By Cathy Zhang and Dr. Malini Bhandaru
Contributors: Lin Yang and Changyan Liu


Generative AI (GenAI) models, which are seeing exponential adoption in our daily lives, are being improved by [retrieval-augmented generation (RAG)](https://www.techtarget.com/searchenterpriseai/definition/retrieval-augmented-generation), a technique used to enhance response accuracy and reliability by fetching facts from external sources. RAG helps a regular [large language model (LLM)](https://www.techtarget.com/whatis/definition/large-language-model-LLM) understand context and reduce [hallucinations](https://en.wikipedia.org/wiki/Hallucination_\(artificial_intelligence\)) by leveraging a giant database of unstructured data stored as vectors — a mathematical presentation that helps capture context and relationships between data.


RAG helps to retrieve more contextual information and thus generate better responses, but the vector databases they rely on are getting ever larger to provide rich content to draw upon. Just as trillion-parameter LLMs are on the horizon, vector databases of billions of vectors are not far behind. As optimization engineers, we were curious to see if we could make vector databases more performant, load data faster, and create indices faster to ensure retrieval speed even as new data is added. Doing so would not only result in reduced user wait time, but also make RAG-based AI solutions a little more sustainable.


In this article, you’ll learn more about vector databases and their benchmarking frameworks, datasets to tackle different aspects, and the tools used for performance analysis — everything you need to start optimizing vector databases. We will also share our optimization achievements on two popular vector database solutions to inspire you on your optimization journey of performance and sustainability impact.




## Understanding Vector Databases


Unlike traditional relational or non-relational databases where data is stored in a structured manner, a vector database contains a mathematical representation of individual data items, called a vector, constructed using an embedding or transformation function. The vector commonly represents features or semantic meanings and can be short or long. Vector databases do vector retrieval by similarity search using a distance metric (where closer means the results are more similar) such as [Euclidean, dot product, or cosine similarity](https://www.pinecone.io/learn/vector-similarity/).


To accelerate the retrieval process, the vector data is organized using an indexing mechanism. Examples of these organization methods include flat structures, [inverted file (IVF),](https://arxiv.org/abs/2002.09094) [Hierarchical Navigable Small Worlds (HNSW),](https://arxiv.org/abs/1603.09320) and [locality-sensitive hashing (LSH)](https://en.wikipedia.org/wiki/Locality-sensitive_hashing), among others. Each of these methods contributes to the efficiency and effectiveness of retrieving similar vectors when needed.


Let’s examine how you would use a vector database in a GenAI system. Figure 1 illustrates both the loading of data into a vector database and using it in the context of a GenAI application. When you input your prompt, it undergoes a transformation process identical to the one used to generate vectors in the database. This transformed vector prompt is then used to retrieve similar vectors from the vector database. These retrieved items essentially serve as conversational memory, furnishing contextual history for prompts, akin to how LLMs operate. This feature proves particularly advantageous in natural language processing, computer vision, recommendation systems, and other domains requiring semantic comprehension and data matching. Your initial prompt is subsequently “merged” with the retrieved elements, supplying context, and assisting the LLM in formulating responses based on the provided context rather than solely relying on its original training data.


![](https://miro.medium.com/v2/resize:fit:1400/1*zQj_YJdWc2xKB6Vv89lzDQ.jpeg)


Figure 1. A RAG application architecture.


Vectors are stored and indexed for speedy retrieval. Vector databases come in two main flavors, traditional databases that have been extended to store vectors, and purpose-built vector databases. Some examples of traditional databases that provide vector support are [Redis](https://redis.io/), [pgvector](https://github.com/pgvector/pgvector), [Elasticsearch](https://www.elastic.co/elasticsearch), and [OpenSearch](https://opensearch.org/). Examples of purpose-built vector databases include proprietary solutions [Zilliz](https://zilliz.com/) and [Pinecone](https://www.pinecone.io/), and open source projects [Milvus](https://milvus.io/), [Weaviate](https://weaviate.io/), [Qdrant](https://qdrant.tech/), [Faiss](https://github.com/facebookresearch/faiss), and [Chroma](https://www.trychroma.com/). You can learn more about vector databases on GitHub via [LangChain ](https://github.com/langchain-ai/langchain/tree/master/libs/langchain/langchain/vectorstores)and [OpenAI Cookbook](https://github.com/openai/openai-cookbook/tree/main/examples/vector_databases).


We’ll take a closer look at one from each category, Milvus and Redis.




## Improving Performance


Before diving into the optimizations, let’s review how vector databases are evaluated, some evaluation frameworks, and available performance analysis tools.




### Performance Metrics


Let’s look at key metrics that can help you measure vector database performance.


- **Load latency** measures the time required to load data into the vector database’s memory and build an index. An index is a data structure used to efficiently organize and retrieve vector data based on its similarity or distance. Types of [in-memory indices](https://milvus.io/docs/index.md#In-memory-Index) include [flat index](https://thedataquarry.com/posts/vector-db-3/#flat-indexes), [IVF\_FLAT](https://supabase.com/docs/guides/ai/vector-indexes/ivf-indexes), [IVF\_PQ, HNSW](https://towardsdatascience.com/ivfpq-hnsw-for-billion-scale-similarity-search-89ff2f89d90e), [scalable nearest neighbors (ScaNN),](https://github.com/google-research/google-research/tree/master/scann)and [DiskANN](https://milvus.io/docs/disk_index.md).
- **Recall** is the proportion of true matches, or relevant items, found in the [Top K](https://redis.io/docs/data-types/probabilistic/top-k/) results retrieved by the search algorithm. Higher recall values indicate better retrieval of relevant items.
- **Queries per second (QPS)** is the rate at which the vector database can process incoming queries. Higher QPS values imply better query processing capability and system throughput.




### Benchmarking Frameworks


![](https://miro.medium.com/v2/resize:fit:920/1*mssEjZAuXg6nf-pad67rHA.jpeg)


Figure 2. The vector database benchmarking framework.


Benchmarking a vector database requires a vector database server and clients. In our performance tests, we used two popular open source tools.


- [**VectorDBBench**](https://github.com/zilliztech/VectorDBBench/tree/main)**:** Developed and open sourced by Zilliz, VectorDBBench helps test different vector databases with different index types and provides a convenient web interface.
- [**vector-db-benchmark**](https://github.com/qdrant/vector-db-benchmark/tree/master)**:** Developed and open sourced by Qdrant, vector-db-benchmark helps test several typical vector databases for the [HNSW](https://www.datastax.com/guides/hierarchical-navigable-small-worlds) index type. It runs tests through the command line and provides a [Docker Compose](https://docs.docker.com/compose/) __file to simplify starting server components.


![](https://miro.medium.com/v2/resize:fit:1400/1*NpHHEFV0TxRMse83hK6H1A.jpeg)


Figure 3. An example vector-db-benchmark command used to run the benchmark test.


But the benchmark framework is only part of the equation. We need data that exercises different aspects of the vector database solution itself, such as its ability to handle large volumes of data, various vector sizes, and speed of retrieval.With that, let’s look at some available public datasets.




### Open Datasets to Exercise Vector Databases


Large datasets are good candidates to test load latency and resource allocation. Some datasets have high dimensional data and are good for testing speed of computing similarity.


Datasets range from a dimension of 25 to a dimension of 2048. The [LAION](https://laion.ai/) dataset, an open image collection, has been used for training very large visual and language deep-neural models like stable diffusion generative models. OpenAI’s dataset of 5M vectors, each with a dimension of 1536, was created by VectorDBBench by running OpenAI on [raw data](https://huggingface.co/datasets/allenai/c4). Given each vector element is of type FLOAT, to save the vectors alone, approximately 29 GB (5M \* 1536 \* 4) of memory is needed, plus a similar amount extra to hold indices and other metadata for a total of 58 GB of memory for testing. When using the vector-db-benchmark tool, ensure adequate disk storage to save results.


To test for load latency, we needed a large collection of vectors, which [deep-image-96-angular](https://docs.hippo.transwarp.io/docs/performance-dataset) offers. To test performance of index generation and similarity computation, high dimensional vectors provide more stress. To this end we chose the 500K dataset of 1536 dimension vectors.




### Performance Tools


We’ve covered ways to stress the system to identify metrics of interest, but let’s examine what’s happening at a lower level: How busy is the computing unit, memory consumption, waits on locks, and more? These provide clues to databasebehavior, particularly useful in identifying problem areas.


The Linux [top](https://www.redhat.com/sysadmin/interpret-top-output) utility provides system-performance information. However, the [perf](https://perf.wiki.kernel.org/index.php/Main_Page) tool in Linux provides a deeper set of insights. To learn more, we also recommend reading [Linux perf examples](https://www.brendangregg.com/perf.html) and the [Intel top-down microarchitecture analysis method](https://www.intel.com/content/www/us/en/docs/vtune-profiler/cookbook/2023-0/top-down-microarchitecture-analysis-method.html). Yet another tool is the [Intel® vTune™ Profiler](https://www.intel.com/content/www/us/en/developer/tools/oneapi/vtune-profiler.html), which is useful when optimizing not just application but also system performance and configuration for a variety of workloads spanning HPC, cloud, IoT, media, storage, and more.




## Milvus Vector Database Optimizations


Let’s walk through some examples of how we attempted to improve the performance of the Milvus vector database.




### Reducing Memory Movement Overhead in Datanode Buffer Write


Milvus’s write path proxies write data into a log broker via _MsgStream_. The data nodes then consume the data, converting and storing it into segments. Segments will merge the newly inserted data. The merge logic allocates a new buffer to hold/move both the old data and the new data to be inserted and then returns the new buffer as old data for the next data merge. This results in the old data getting successively larger, which in turn makes data movement slower. Perf profiles showed a high overhead for this logic.


![](https://miro.medium.com/v2/resize:fit:1400/1*Az4dMVBcGmdeyKNrwpR19g.jpeg)


Figure 4. Merging and moving data in the vector database generates a high-performance overhead.


We changed the _merge buffer_ logic to directly append the new data to be inserted into the old data, avoiding allocating a new buffer and moving the large old data. Perf profiles confirm that there is no overhead to this logic. The microcode metrics _metric\_CPU operating frequency_ and _metric\_CPU utilization_ indicate an improvement that is consistent with the system not having to wait for the long memory movement anymore. Load latency improved by more than 60 percent. The improvement is captured on [GitHub](https://github.com/milvus-io/milvus/pull/26839).


![](https://miro.medium.com/v2/resize:fit:1400/1*MmaUtBTdqmMvC5MlQ8V0wQ.jpeg)


Figure 5. With less copying we see a performance improvement of more than 50 percent in load latency.




### Inverted Index Building with Reduced Memory Allocation Overhead


The Milvus search engine, [Knowhere](https://milvus.io/docs/knowhere.md), employs the [Elkan k-means algorithm](https://www.vlfeat.org/api/kmeans-fundamentals.html#kmeans-elkan) to train cluster data for creating [inverted file (IVF) indices](https://milvus.io/docs/v1.1.1/index.md). Each round of data training defines an iteration count. The larger the count, the better the training results. However, it also implies that the Elkan algorithm will be called more frequently.


The Elkan algorithm handles memory allocation and deallocation each time it’s executed. Specifically, it allocates memory to store half the size of symmetric matrix data, excluding the diagonal elements. In Knowhere, the symmetric matrix dimension used by the Elkan algorithm is set to 1024, resulting in a memory size of approximately 2 MB. This means for each training round Elkan repeatedly allocates and deallocates 2 MB memory.


Perf profiling data indicated frequent large memory allocation activity. In fact, it triggered [virtual memory area (VMA)](https://www.oreilly.com/library/view/linux-device-drivers/9781785280009/4759692f-43fb-4066-86b2-76a90f0707a2.xhtml)allocation, physical page allocation, page map setup, and updating of memory cgroup statistics in the kernel. This pattern of large memory allocation/deallocation activity can, in some situations, also aggravate memory fragmentation. This is a significant tax.


The _IndexFlatElkan_ structure is specifically designed and constructed to support the Elkan algorithm. Each data training process will have an _IndexFlatElkan_ instance initialized. To mitigate the performance impact resulting from frequent memory allocation and deallocation in the Elkan algorithm, we refactored the code logic, moving the memory management outside of the Elkan algorithm function up into the construction process of _IndexFlatElkan_. This enables memory allocation to occur only once during the initialization phase while serving all subsequent Elkan algorithm function calls from the current data training process and helps to improve load latency by around 3 percent. Find the [Knowhere patch here](https://github.com/zilliztech/knowhere/pull/280).




## Redis Vector Search Acceleration through Software Prefetch


Redis, a popular traditional in-memory key-value data store, recently began supporting vector search. To go beyond a typical key-value store, it offers extensibility modules; the [RediSearch](https://github.com/RediSearch/RediSearch) module facilitates the storage and search of vectors directly within Redis.


For vector similarity search, Redis supports two algorithms, namely brute force and HNSW. The HNSW algorithm is specifically crafted for efficiently locating approximate nearest neighbors in high-dimensional spaces. It uses a priority queue named _candidate\_set_ to manage all vector candidates for distance computing.


Each vector candidate encompasses substantial metadata in addition to the vector data. As a result, when loading a candidate from memory it can cause data cache misses, which incur processing delays. Our optimization introduces software prefetching to proactively load the next candidate while processing the current one. This enhancement has resulted in a 2 to 3 percent throughput improvement for vector similarity searches in a single instance Redis setup. The patch is in the process of being upstreamed.




## GCC Default Behavior Change to Prevent Mixed Assembly Code Penalties


To drive maximum performance, frequently used sections of code are often handwritten in assembly. However, when different segments of code are written either by different people or at different points in time, the instructions used may come from incompatible assembly instruction sets such as [Intel® Advanced Vector Extensions 512 (Intel® AVX-512)](https://www.intel.com/content/www/us/en/architecture-and-technology/avx-512-overview.html) and [Streaming SIMD Extensions (SSE)](https://en.wikipedia.org/wiki/Streaming_SIMD_Extensions). If not compiled appropriately, the mixed code results in a performance penalty. [Learn more about mixing Intel AVX and SSE instructions here](https://www.intel.com/content/dam/develop/external/us/en/documents/11mc12-avoiding-2bavx-sse-2btransition-2bpenalties-2brh-2bfinal-809104.pdf).


You can easily determine if you’re using mixed-mode assembly code and have not compiled the code with _VZEROUPPER_, incurring the performance penalty. It can be observed through a perf command like _sudo perf stat -e ‘assists.sse\_avx\_mix/event/event=0xc1,umask=0x10/’ \<workload>_. If your OS doesn’t have support for the event, use _cpu/event=0xc1,umask=0x10,name=assists\_sse\_avx\_mix/_.


The Clang compiler by default inserts _VZEROUPPER_, avoiding any mixed mode penalty. But the GCC compiler only inserted _VZEROUPPER_ when the -O2 or -O3 compiler flags were specified. We contacted the GCC team and explained the issue and they now, by default, correctly handle mixed mode assembly code.



## Start Optimizing Your Vector Databases


Vector databases are playing an integral role in GenAI, and they are growing ever larger to generate higher-quality responses. With respect to optimization, AI applications are no different from other software applications in that they reveal their secrets when one employs standard performance analysis tools along with benchmark frameworks and stress input.

Using these tools, we uncovered performance traps pertaining to unnecessary memory allocation, failing to prefetch instructions, and using incorrect compiler options. Based on our findings, we upstreamed enhancements to Milvus, Knowhere, Redis, and the GCC compiler to help make AI a little more performant and sustainable. Vector databases are an important class of applications worthy of your optimization efforts. We hope this article helps you get started.


Optimize Vector Databases, Enhance RAG-Driven Generative AI



## Introduction

Milvus is a distributed vector database that aims to store, index and manage massive embedding vectors. Its ability to efficiently index and search through trillions of vectors makes Milvus a go-to choice for AI and machine learning workloads.

Kubernetes (K8s), on the other hand, excels in managing and scaling containerized applications. It provides features like auto-scaling, self-healing, and load balancing, which are crucial for maintaining high availability and performance in production environments.


## Why Use Them Together? 

K8s can automatically scale the Milvus clusters based on the workload. As your data grows or the number of queries increases, K8s can spin up more Milvus instances to handle the load, ensuring your applications remain responsive.

One of the standout features of K8s is its horizontal scaling, which makes expanding your Milvus cluster a breeze. As your dataset grows, K8s effortlessly accommodates this growth, making it a straightforward and efficient solution.

In addition, the ability to handle queries also scales horizontally with K8s. As the query load increases, K8s can deploy more Milvus instances to handle the increased similarity search queries, ensuring low latency responses even under heavy loads.


## Prerequisites & Setting Up K8s

### Prerequisites

- **Docker** - Ensure Docker is installed on your system.

- **Kubernetes** - Have a Kubernetes cluster ready. You can use `minikube` for local development or a cloud provider's Kubernetes service for production environments.

- **Helm** - Install Helm, a package manager for Kubernetes, to help you manage Kubernetes applications, you can check our documentation to see how to do that <https://milvus.io/docs/install_cluster-helm.md>

- **Kubectl** - Install `kubectl`, a command-line tool for interacting with Kubernetes clusters, to deploy applications, inspect and manage cluster resources, and view logs.

### Setting Up K8s

After installing everything needed to run a K8s cluster, and if you used `minikube`, start your cluster with: 

```
minikube start
```

Check the status of your K8s cluster with:

```
kubectl cluster-info
```

### Deploying Milvus on K8s

For this deployment, we're opting for Milvus in cluster-mode to leverage its full distributed capabilities. We'll be using Helm, to streamline the installation process.

**1. Helm Installation Command** 

```
helm install my-milvus milvus/milvus --set pulsar.enabled=false --set kafka.enabled=true
```

This command installs Milvus on your K8s cluster with Kafka enabled and Pulsar disabled. Kafka serves as the messaging system within Milvus, handling data streaming between different components. Disabling Pulsar and enabling Kafka tailors the deployment to our specific messaging preferences and requirements.

**2. Port Forwarding**

To access Milvus from your local machine, create a port forward: `kubectl port-forward svc/my-milvus 27017:19530`.

This command maps port `19530` from the Milvus service `svc/my-milvus` to the same port on your local machine, allowing you to connect to Milvus using local tools. If you leave the local port unspecified (as in `:19530`), K8s will allocate an available port, making it dynamic. Ensure you note the allocated local port if you choose this method.

**3. Verifying the Deployment:**

```
kubectl get pods 

NAME                                    READY   STATUS    RESTARTS   AGE
my-milvus-datacoord-595b996bd4-zprpd    1/1     Running   0          85m
my-milvus-datanode-d9d555785-47nkt      1/1     Running   0          85m
my-milvus-etcd-0                        1/1     Running   0          84m
my-milvus-etcd-1                        1/1     Running   0          85m
my-milvus-etcd-2                        1/1     Running   0          85m
my-milvus-indexcoord-65bc68968c-6jg6q   1/1     Running   0          85m
my-milvus-indexnode-54586f55d-z9vx4     1/1     Running   0          85m
my-milvus-kafka-0                       1/1     Running   0          85m
my-milvus-kafka-1                       1/1     Running   0          85m
my-milvus-kafka-2                       1/1     Running   0          85m
my-milvus-minio-0                       1/1     Running   0          96m
my-milvus-minio-1                       1/1     Running   0          96m
my-milvus-minio-2                       1/1     Running   0          96m
my-milvus-minio-3                       1/1     Running   0          96m
my-milvus-proxy-76bb7d497f-sqwvd        1/1     Running   0          85m
my-milvus-querycoord-6f4c7b7598-b6twj   1/1     Running   0          85m
my-milvus-querynode-677bdf485b-ktc6m    1/1     Running   0          85m
my-milvus-rootcoord-7498fddfd8-v5zw8    1/1     Running   0          85m
my-milvus-zookeeper-0                   1/1     Running   0          85m
my-milvus-zookeeper-1                   1/1     Running   0          85m
my-milvus-zookeeper-2                   1/1     Running   0          85m
```


You should see a list of pods similar to the output above, all in the Running state. This indicates that your Milvus cluster is operational. Specifically, look for the 1/1 under the `READY` column, which signifies that each pod is fully ready and running. If any pods are not in the Running state, you may need to investigate further to ensure a successful deployment.

With your Milvus cluster deployed and all components confirmed running, you're now ready to proceed to data ingestion and indexing. This will involve connecting to your Milvus instance, creating collections, and inserting vectors for search and retrieval.


## Data Ingestion and Indexing

To start ingesting and indexing data in our Milvus cluster, we'll use the pymilvus SDK. There are two installation options:

- Basic SDK: `pip install pymilvus`

- For rich text embeddings and advanced models: `pip install pymilvus[model]`

Time to insert data in our cluster, we’ll be using `pymilvus`, you can either install the SDK only with `pip install pymilvus` or if you want to extract rich text embeddings, you can also use `PyMilvus Models` by installing `pip install pymilvus[model]`. 

### Connecting and Creating a Collection:

First, connect to your Milvus instance using the port you forwarded earlier. Ensure the URI matches the local port assigned by K8s:

```
from pymilvus import MilvusClient

client = MilvusClient(
        uri="http://127.0.0.1:52070",
    )

client.create_collection(collection_name="quick_setup", dimension=5)
```

The `dimension=5` parameter defines the vector size for this collection, essential for the vector search capabilities.


### Insert Data

Here's how to insert an initial set of data, where each vector represents an item, and the color field adds a descriptive attribute:

```
data=[
    {"id": 0, "vector": [0.3580376395471989, -0.6023495712049978, 0.18414012509913835, -0.26286205330961354, 0.9029438446296592], "color": "pink_8682"},
    {"id": 1, "vector": [0.19886812562848388, 0.06023560599112088, 0.6976963061752597, 0.2614474506242501, 0.838729485096104], "color": "red_7025"},
    {"id": 2, "vector": [0.43742130801983836, -0.5597502546264526, 0.6457887650909682, 0.7894058910881185, 0.20785793220625592], "color": "orange_6781"},
    {"id": 3, "vector": [0.3172005263489739, 0.9719044792798428, -0.36981146090600725, -0.4860894583077995, 0.95791889146345], "color": "pink_9298"},
    {"id": 4, "vector": [0.4452349528804562, -0.8757026943054742, 0.8220779437047674, 0.46406290649483184, 0.30337481143159106], "color": "red_4794"},
    {"id": 5, "vector": [0.985825131989184, -0.8144651566660419, 0.6299267002202009, 0.1206906911183383, -0.1446277761879955], "color": "yellow_4222"},
    {"id": 6, "vector": [0.8371977790571115, -0.015764369584852833, -0.31062937026679327, -0.562666951622192, -0.8984947637863987], "color": "red_9392"},
    {"id": 7, "vector": [-0.33445148015177995, -0.2567135004164067, 0.8987539745369246, 0.9402995886420709, 0.5378064918413052], "color": "grey_8510"},
    {"id": 8, "vector": [0.39524717779832685, 0.4000257286739164, -0.5890507376891594, -0.8650502298996872, -0.6140360785406336], "color": "white_9381"},
    {"id": 9, "vector": [0.5718280481994695, 0.24070317428066512, -0.3737913482606834, -0.06726932177492717, -0.6980531615588608], "color": "purple_4976"}
]

res = client.insert(
    collection_name="quick_setup",
    data=data
)

print(res)
```


The provided code assumes that you have created a collection in the Quick Setup manner. As shown in the above code,

The data to insert is organized into a list of dictionaries, where each dictionary represents a data record, termed as an entity.

Each dictionary contains a non-schema-defined field named color.

Each dictionary contains the keys corresponding to both pre-defined and dynamic fields.


### Insert Even More Data

```
colors = ["green", "blue", "yellow", "red", "black", "white", "purple", "pink", "orange", "brown", "grey"]
data = [ {
    "id": i, 
    "vector": [ random.uniform(-1, 1) for _ in range(5) ], 
    "color": f"{random.choice(colors)}_{str(random.randint(1000, 9999))}" 
} for i in range(1000) ]

res = client.insert(
    collection_name="quick_setup",
    data=data[10:]
)

print(res)
```


## Similarity Search

After populating the collection, you can perform a similarity search to find vectors close to a query vector. The value of the query_vectors variable is a list containing a sub-list of floats. The sub-list represents a vector embedding of 5 dimensions.

```
query_vectors = [
    [0.041732933, 0.013779674, -0.027564144, -0.013061441, 0.009748648]
]

res = client.search(
    collection_name="quick_setup",     # target collection
    data=query_vectors,                # query vectors
    limit=3,                           # number of returned entities
)

print(res)
```


This query searches for the top 3 vectors most similar to our query vector, demonstrating Milvus's powerful search capabilities.


## Uninstall Milvus from K8s

Once you are done with this tutorial, feel free to uninstall Milvus from your K8s cluster with:`helm uninstall my-milvus`.

This command will remove all Milvus components deployed in the `my-milvus` release, freeing up cluster resources.

## Conclusion

- Deploying Milvus on a Kubernetes cluster showcases the scalability and flexibility of vector databases in handling AI and machine learning workloads. Through this tutorial, you've learned the basics of setting up Milvus with Helm, creating a collection, and performing data ingestion and similarity searches.

- Installing Milvus on a Kubernetes cluster with Helm should be straightforward. To go deeper into scaling Milvus clusters for larger datasets or more intensive workloads, our documentation offers detailed guidance <https://milvus.io/docs/scaleout.md>

Feel free to check out the code on [Github](https://github.com/stephen37/K8s-tutorial-milvus), check out [Milvus](https://github.com/milvus-io/milvus), experiment with different configurations and use cases, and share your experiences with the community by joining our [Discord](https://discord.gg/FG6hMJStWu).


Milvus Metadata Management (1) How to View Metadata

How to View Metadata

SQLite

MySQL

相关博客

Content

Keep Reading

Introducing PyMilvus Integration with Embedding Models

Optimize Vector Databases, Enhance RAG-Driven Generative AI

Getting started with Milvus cluster and K8s