
_"We initially built our semantic search on pgvector instead of Milvus because all our relational data was already in PostgreSQL,"_ recalls Alex, CTO of an enterprise AI SaaS startup. _"But as soon as we hit product-market fit, our growth ran into serious hurdles on the engineering side. It quickly became clear that pgvector wasn’t designed for scalability. Simple tasks such as rolling out schema updates across multiple shards turned into tedious, error-prone processes that consumed days of engineering effort. When we reached 100 million vector embeddings, query latency spiked to over a second, something far beyond what our customers would tolerate. After moving to Milvus, sharding manually felt like stepping into the stone age. It’s no fun juggling shard servers as if they were fragile artifacts. No company should have to endure that."_


## A Common Challenge for AI Companies

Alex's experience isn't unique to pgvector users. Whether you're using pgvector, Qdrant, Weaviate, or any other vector database that relies on manual sharding, the scaling challenges remain the same. What starts as a manageable solution quickly turns into a tech debt as data volumes grow.

For startups today, **scalability isn't optional—it's mission-critical**. This is especially true for AI products powered by Large Language Models(LLM) and vector databases, where the leap from early adoption to exponential growth can happen overnight. Achieving product-market fit often triggers a surge in user growth, overwhelming data inflows, and skyrocketing query demands. But if the database infrastructure can't keep up, slow queries and operational inefficiencies can stall momentum and hinder business success.

A short-term technical decision could lead to long-term bottleneck, forcing engineering teams to constantly address urgent performance issues, database crashes, and system failures instead of focusing on innovation. The worst-case scenario? A costly, time-consuming database re-architecture—precisely when a company should be scaling.


## Isn’t Sharding a Natural Solution to Scalability?

Scalability can be addressed in multiple ways. The most straightforward approach, **Scaling Up**, involves enhancing a single machine’s resources by adding more CPU, memory, or storage to accommodate growing data volumes. While simple, this method has clear limitations. In a Kubernetes environment, for example, large pods are inefficient, and relying on a single node increases the risk of failure, potentially leading to significant downtime.

When Scaling Up is no longer viable, businesses naturally turn to **Scaling Out**, distributing data across multiple servers. At first glance, **sharding** appears to be a simple solution—splitting a database into smaller, independent databases to increase capacity and enable multiple writable primary nodes.

However, while conceptually straightforward, sharding quickly becomes a complex challenge in practice. Most applications are initially designed to work with a single, unified database. The moment a vector database is divided into multiple shards, every part of the application that interacts with data must be modified or entirely rewritten, introducing significant development overhead. Designing an effective sharding strategy becomes crucial, as does implementing routing logic to ensure data is directed to the correct shard. Managing atomic transactions across multiple shards often requires restructuring applications to avoid cross-shard operations. Additionally, failure scenarios must be handled gracefully to prevent disruptions when certain shards become unavailable.


## Why Manual Sharding Becomes a Burden

_"We originally estimated implementing manual sharding for our pgvector database would take two engineers about six months,"_ Alex remembers. _"What we didn't realize was that those engineers would_ **_always_** _be needed. Every schema change, data rebalancing operation, or scaling decision required their specialized expertise. We were essentially committing to a permanent 'sharding team' just to keep our database running."_

Real-world challenges with sharded vector databases include:

1. **Data Distribution Imbalance (Hotspots)**: In multi-tenant use cases, data distribution can range from hundreds to billions of vectors per tenant. This imbalance creates hotspots where certain shards become overloaded while others sit idle.

2. **The Resharding Headache**: Choosing the right number of shards is nearly impossible. Too few leads to frequent and costly resharding operations. Too many creates unnecessary metadata overhead, increasing complexity and reducing performance.

3. **Schema Change Complexity**: Many vector databases implement sharding by managing multiple underlying databases. This makes synchronizing schema changes across shards cumbersome and error-prone, slowing development cycles.

4. **Resource Waste**: In storage-compute coupled databases, you must meticulously allocate resources across every node while anticipating future growth. Typically, when resource utilization reaches 60-70%, you need to start planning for resharding.

Simply put, **managing shards manually is bad for your business**. Instead of locking your engineering team into constant shard management, consider investing in a vector database designed to scale automatically—without the operational burden.


## How Milvus Solves the Scalability Problem

Many developers—from startups to enterprises—have recognized the significant overhead associated with manual database sharding. Milvus takes a fundamentally different approach, enabling seamless scaling from millions to billions of vectors without the complexity.


### Automated Scaling Without the Tech Debt

Milvus leverages Kubernetes and a disaggregated storage-compute architecture to support seamless expansion. This design enables:

- Rapid scaling in response to changing demands

- Automatic load balancing across all available nodes

- Independent resource allocation, letting you adjust compute, memory, and storage separately

- Consistent high performance, even during periods of rapid growth


### Distributed Architecture Designed from the Ground Up

Milvus achieves its scaling capabilities through two key innovations:

**Segment-Based Architecture:** At its core, Milvus organizes data into "segments"—the smallest units of data management:

- Growing Segments reside on StreamNodes, optimizing data freshness for real-time queries

- Sealed Segments are managed by QueryNodes, utilizing powerful indexes to accelerate search

- These segments are evenly distributed across nodes to optimize parallel processing

**Two-Layer Routing**: Unlike traditional databases where each shard lives on a single machine, Milvus distributes data in one shard dynamically across multiple nodes:

- Each shard can store over 1 billion data points

- Segments within each shard are automatically balanced across machines

- Expanding collections is as simple as increasing the number of shards

- The upcoming Milvus 3.0 will introduce dynamic shard splitting, eliminating even this minimal manual step


### Query Processing at Scale

When executing a query, Milvus follows an efficient process:

1. The Proxy identifies relevant shards for the requested collection

2. The Proxy gathers data from both StreamNodes and QueryNodes

3. StreamNodes handle real-time data while QueryNodes process historical data concurrently

4. Results are aggregated and returned to the user


![](https://assets.zilliz.com/Query_Processing_at_Scale_5792dc9e37.png)

## A Different Engineering Experience

_"When scalability is built into the database itself, all those headaches just... disappear,"_ says Alex, reflecting on his team's transition to Milvus. _"My engineers are back to building features customers love instead of babysitting database shards."_

If you're grappling with the engineering burden of manual sharding, performance bottlenecks at scale, or the daunting prospect of database migrations, it's time to rethink your approach. Visit our [docs page](https://milvus.io/docs/overview.md#What-Makes-Milvus-so-Scalable) to learn more about Milvus architecture, or experience effortless scalability firsthand with fully-managed Milvus at [zilliz.com/cloud](https://zilliz.com/cloud).

With the right vector database foundation, your innovation knows no limits.


Why Manual Sharding is a Bad Idea for Vector Database And How to Fix It



Building a performant [vector database](https://zilliz.com/learn/what-is-vector-database) like Milvus that scales to billions of vectors and handles web-scale traffic is no simple feat. It requires the careful, intelligent design of a distributed system. Necessarily, there will be a trade-off between performance vs simplicity in the internals of a system like this.

While we have tried to well balance this trade-off, some aspects of the internals have remained opaque. This article aims to dispel any mystery around how Milvus breaks down data insertion, indexing, and serving across nodes. Understanding these processes at a high level is essential for effectively optimizing query performance, system stability, and debugging-related issues.

So, let’s take a stroll in a day in the life of Dave, the Milvus datum. Imagine you insert Dave into your collection in a [Milvus Distributed deployment](https://milvus.io/docs/install-overview.md#Milvus-Distributed) (see the diagram below). As far as you are concerned, he goes directly into the collection. Behind the scenes, however, many steps occur across independent sub-systems.

![](https://assets.zilliz.com/a_day_in_the_life_of_a_milvus_datum_ca279f7f59.png)


## Proxy Nodes and the Message Queue

![](https://assets.zilliz.com/Proxy_Nodes_and_the_Message_Queue_03a0fde0c5.png)

Initially, you call the MilvusClient object, for example, via the PyMilvus library, and send an `_insert()`_ request to a _proxy node_. Proxy nodes are the gateway between the user and the database system, performing operations like load balancing on incoming traffic and collating multiple outputs before they are returned to the user.

A hash function is applied to the item’s primary key to determine which _channel_ to send it to. Channels, implemented with either Pulsar or Kafka topics, represent a holding ground for streaming data, which can then be sent onwards to subscribers of the channel.


## Data Nodes, Segments, and Chunks

![](https://assets.zilliz.com/Data_Nodes_Segments_and_Chunks_ae122dd1ac.png)

After the data has been sent to the appropriate channel, the channel then sends it to the corresponding segment in the _data node_. Data nodes are responsible for storing and managing data buffers called _growing segments_. There is one growing segment per shard.

As data is inserted into a segment, the segment grows towards a maximum size, defaulting to 122MB. During this time, smaller parts of the segment, by default 16MB and known as _chunks_, are pushed to persistent storage, for example, using AWS’s S3 or other compatible storage like MinIO. Each chunk is a physical file on the object storage and there is a separate file per field. See the figure above illustrating the file hierarchy on object storage.

So to summarize, a collection’s data is split across data nodes, within which it is split into segments for buffering, which are further split into per-field chunks for persistent storage. The two diagrams above make this clearer. By dividing the incoming data in this way, we fully exploit the cluster’s parallelism of network bandwidth, compute, and storage.


## Sealing, Merging, and Compacting Segments

![](https://assets.zilliz.com/Sealing_Merging_and_Compacting_Segments_d5a6a37261.png)

Thus far we have told the story of how our friendly datum Dave makes his way from an `_insert()`_ query into persistent storage. Of course, his story does not end there. There are further steps to make the search and indexing process more efficient. By managing the size and number of segments, the system fully exploits the cluster’s parallelism.

Once a segment reaches its maximum size on a data node, by default 122MB, it is said to be _sealed_. What this means is that the buffer on the data node is cleared to make way for a new segment, and the corresponding chunks in persistent storage are marked as belonging to a closed segment.

The data nodes periodically look for smaller sealed segments and merge them into larger ones until they have reached a maximum size of 1GB (by default) per segment. Recall that when an item is deleted in Milvus, it is simply marked with a deletion flag - think of it as Death Row for Dave. When the number of deleted items in a segment passes a given threshold, by default 20%, the segment is reduced in size, an operation we call _compaction_.


Indexing and Searching through Segments

![](https://assets.zilliz.com/Indexing_and_Searching_through_Segments_478c0067be.png)

![](https://assets.zilliz.com/Indexing_and_Searching_through_Segments_1_0c31b5a340.png)

There is an additional node type, the _index node_, that is responsible for building indexes for sealed segments. When the segment is sealed, the data node sends a request for an index node to construct an index. The index node then sends the completed index to object storage. Each sealed segment has its own index stored in a separate file. You can examine this file manually by accessing the bucket - see the figure above for the file hierarchy.

Query nodes - not only data nodes - subscribe to the message queue topics for the corresponding shards. The growing segments are replicated on the query nodes, and the node loads into memory sealed segments belonging to the collection as required. It builds an index for each growing segment as data comes in, and loads the finalized indexes for sealed segments from the data store.

Imagine now that you call the MilvusClient object with a _search()_ request that encompasses Dave. After being routed to all query nodes via the proxy node, each query node performs a vector similarity search (or another one of the search methods like query, range search, or grouping search), iterating over the segments one by one. The results are collated across nodes in a MapReduce-like fashion and sent back to the user, Dave being happy to find himself reunited with you at last.


## Discussion

We have covered a day in the life of Dave the datum, both for `_insert()`_ and `_search()`_ operations. Other operations like `_delete()`_ and `_upsert()`_ work similarly. Inevitably, we have had to simplify our discussion and omit finer details. On the whole, though, you should now have a sufficient picture of how Milvus is designed for parallelism across nodes in a distributed system to be robust and efficient, and how you can use this for optimization and debugging. 

_An important takeaway from this article: Milvus is designed with a separation of concerns across node types. Each node type has a specific, mutually exclusive function, and there is a separation of storage and compute._ The result is that each component can be scaled independently with parameters tweakable according to the use case and traffic patterns. For example, you can scale the number of query nodes to serve increased traffic without scaling data and index nodes. With that flexibility, there are Milvus users that handle billions of vectors and serve web-scale traffic, with sub-100ms query latency.

You can also enjoy the benefits of Milvus’ distributed design without even deploying a distributed cluster through [Zilliz Cloud](https://zilliz.com/cloud), a fully managed service of Milvus. [Sign up today for the free-tier of Zilliz Cloud and put Dave into action!](https://cloud.zilliz.com/signup)


A Day in the Life of a Milvus Datum

AI for Smarter Browsing: Filtering Web Content with Pixtral, Milvus, and Browser Use




![deep researcher.gif](https://assets.zilliz.com/deep_researcher_a0170dadd0.gif)

In the previous post, [_“I Built a Deep Research with Open Source—and So Can You!”_](https://milvus.io/blog/i-built-a-deep-research-with-open-source-so-can-you.md), we explained some of the principles underlying research agents and constructed a simple prototype that generates detailed reports on a given topic or question. The article and corresponding notebook demonstrated the fundamental concepts of _tool use_, _query decomposition_, _reasoning_, and _reflection_. The example in our previous post, in contrast to OpenAI’s Deep Research, ran locally, using only open-source models and tools like [Milvus](https://milvus.io/docs) and LangChain. (I encourage you to read the [above article](https://milvus.io/blog/i-built-a-deep-research-with-open-source-so-can-you.md) before continuing.) 

In the following weeks, there was an explosion of interest in understanding and reproducing OpenAI’s Deep Research. See, for example, [Perplexity Deep Research](https://www.perplexity.ai/hub/blog/introducing-perplexity-deep-research) and [Hugging Face's Open DeepResearch](https://huggingface.co/blog/open-deep-research). These tools differ in architecture and methodology although sharing an objective: iteratively research a topic or question by surfing the web or internal documents and output a detailed, informed, and well-structured report. Importantly, the underlying agent automates reasoning about what action to take at each intermediate step.

In this post, we build upon our previous post and present Zilliz’s [DeepSearcher](https://github.com/zilliztech/deep-searcher) open-source project. Our agent demonstrates additional concepts: _query routing, conditional execution flow_, and _web crawling as a tool_. It is presented as a Python library and command-line tool rather than a Jupyter notebook and is more fully-featured than our previous post. For example, it can input multiple source documents and can set the embedding model and vector database used via a configuration file. While still relatively simple, DeepSearcher is a great showcase of agentic RAG and is a further step towards a state-of-the-art AI applications.

Additionally, we explore the need for faster and more efficient inference services. Reasoning models make use of “inference scaling”, that is, extra computation, to improve their output, and that combined with the fact that a single report may require hundreds or thousands of LLM calls results in inference bandwidth being the primary bottleneck. We use the [DeepSeek-R1 reasoning model on SambaNova’s custom-built hardware](https://sambanova.ai/press/fastest-deepseek-r1-671b-with-highest-efficiency), which is twice as fast in output tokens-per-second as the nearest competitor (see figure below).

SambaNova Cloud also provides inference-as-a-service for other open-source models including Llama 3.x, Qwen2.5, and QwQ. The inference service runs on SambaNova’s custom chip called the reconfigurable dataflow unit (RDU), which is specially designed for efficient inference on Generative AI models, lowering cost and increasing inference speed. [Find out more on their website.](https://sambanova.ai/technology/sn40l-rdu-ai-chip)

![Output speed- deepseek r1.png](https://assets.zilliz.com/Output_speed_deepseek_r1_d820329f0a.png)


## DeepSearcher Architecture



The architecture of [DeepSearcher](https://github.com/zilliztech/deep-searcher) follows our previous post by breaking the problem up into four steps - _define/refine the question_, _research_, _analyze_, _synthesize_ - although this time with some overlap. We go through each step, highlighting [DeepSearcher](https://github.com/zilliztech/deep-searcher)’s improvements.

![deepsearcher architecture.png](https://assets.zilliz.com/deepsearcher_architecture_088c7066d1.png)

### Define and Refine the Question

```txt
Break down the original query into new sub queries: [
  'How has the cultural impact and societal relevance of The Simpsons evolved from its debut to the present?',
  'What changes in character development, humor, and storytelling styles have occurred across different seasons of The Simpsons?', 
  'How has the animation style and production technology of The Simpsons changed over time?',
  'How have audience demographics, reception, and ratings of The Simpsons shifted throughout its run?']
```


In the design of DeepSearcher, the boundaries between researching and refining the question are blurred. The initial user query is decomposed into sub-queries, much like the previous post. See above for initial subqueries produced from the query “How has The Simpsons changed over time?”. However, the following research step will continue to refine the question as needed.


### Research and Analyze

Having broken down the query into sub-queries, the research portion of the agent begins. It has, roughly speaking, four steps: _routing_, _search_, _reflection, and conditional repeat_.


#### Routing

Our database contains multiple tables or collections from different sources. It would be more efficient if we could restrict our semantic search to only those sources that are relevant to the query at hand. A query router prompts an LLM to decide from which collections information should be retrieved.

Here is the method to form the query routing prompt:

```python
def get_vector_db_search_prompt(
    question: str,
    collection_names: List[str],
    collection_descriptions: List[str],
    context: List[str] = None,
):
    sections = []
    # common prompt
    common_prompt = f"""You are an advanced AI problem analyst. Use your reasoning ability and historical conversation information, based on all the existing data sets, to get absolutely accurate answers to the following questions, and generate a suitable question for each data set according to the data set description that may be related to the question.

Question: {question}
"""
    sections.append(common_prompt)
    
    # data set prompt
    data_set = []
    for i, collection_name in enumerate(collection_names):
        data_set.append(f"{collection_name}: {collection_descriptions[i]}")
    data_set_prompt = f"""The following is all the data set information. The format of data set information is data set name: data set description.

Data Sets And Descriptions:
"""
    sections.append(data_set_prompt + "\n".join(data_set))
    
    # context prompt
    if context:
        context_prompt = f"""The following is a condensed version of the historical conversation. This information needs to be combined in this analysis to generate questions that are closer to the answer. You must not generate the same or similar questions for the same data set, nor can you regenerate questions for data sets that have been determined to be unrelated.

Historical Conversation:
"""
        sections.append(context_prompt + "\n".join(context))
    
    # response prompt
    response_prompt = f"""Based on the above, you can only select a few datasets from the following dataset list to generate appropriate related questions for the selected datasets in order to solve the above problems. The output format is json, where the key is the name of the dataset and the value is the corresponding generated question.

Data Sets:
"""
    sections.append(response_prompt + "\n".join(collection_names))
    
    footer = """Respond exclusively in valid JSON format matching exact JSON schema.

Critical Requirements:
- Include ONLY ONE action type
- Never add unsupported keys
- Exclude all non-JSON text, markdown, or explanations
- Maintain strict JSON syntax"""
    sections.append(footer)
    return "\n\n".join(sections)
```



We make the LLM return structured output as JSON in order to easily convert its output to a decision on what to do next.


#### Search

Having selected various database collections via the previous step, the search step performs a similarity search with [Milvus](https://milvus.io/docs). Much like the previous post, the source data has been specified in advance, chunked, embedded, and stored in the vector database. For DeepSearcher, the data sources, both local and online, must be manually specified. We leave online search for future work.


#### Reflection

Unlike the previous post, DeepSearcher illustrates a true form of agentic reflection, inputting the prior outputs as context into a prompt that “reflects” on whether the questions asked so far and the relevant retrieved chunks contain any informational gaps. This can be seen as an analysis step.

Here is the method to create the prompt:


```python
def get_reflect_prompt(
   question: str,
   mini_questions: List[str],
   mini_chuncks: List[str],
):
    mini_chunk_str = ""
    for i, chunk in enumerate(mini_chuncks):
        mini_chunk_str += f"""<chunk_{i}>\n{chunk}\n</chunk_{i}>\n"""
    reflect_prompt = f"""Determine whether additional search queries are needed based on the original query, previous sub queries, and all retrieved document chunks. If further research is required, provide a Python list of up to 3 search queries. If no further research is required, return an empty list.

If the original query is to write a report, then you prefer to generate some further queries, instead return an empty list.

    Original Query: {question}
    Previous Sub Queries: {mini_questions}
    Related Chunks: 
    {mini_chunk_str}
    """
   
    
    footer = """Respond exclusively in valid List of str format without any other text."""
    return reflect_prompt + footer
```



Once more, we make the LLM return structured output, this time as Python-interpretable data.

Here is an example of new sub-queries “discovered” by reflection after answering the initial sub-queries above:

```
New search queries for next iteration: [
  "How have changes in The Simpsons' voice cast and production team influenced the show's evolution over different seasons?",
  "What role has The Simpsons' satire and social commentary played in its adaptation to contemporary issues across decades?",
  'How has The Simpsons addressed and incorporated shifts in media consumption, such as streaming services, into its distribution and content strategies?']
```



#### Conditional Repeat

Unlike our previous post, DeepSearcher illustrates conditional execution flow. After reflecting on whether the questions and answers so far are complete, if there are additional questions to be asked the agent repeats the above steps. Importantly, the execution flow (a while loop) is a function of the LLM output rather than being hard-coded. In this case there is only a binary choice: _repeat research_ or _generate a report_. In more complex agents there may be several such as: _follow hyperlink_, _retrieve chunks, store in memory, reflect_ etc. In this way, the question continues to be refined as the agent sees fit until it decides to exit the loop and generate the report. In our Simpsons example, DeepSearcher performs two more rounds of filling the gaps with extra sub-queries.


### Synthesize

Finally, the fully decomposed question and retrieved chunks are synthesized into a report with a single prompt. Here is the code to create the prompt:

```python
def get_final_answer_prompt(
   question: str, 
   mini_questions: List[str],
   mini_chuncks: List[str],
):
    mini_chunk_str = ""
    for i, chunk in enumerate(mini_chuncks):
        mini_chunk_str += f"""<chunk_{i}>\n{chunk}\n</chunk_{i}>\n"""
    summary_prompt = f"""You are an AI content analysis expert, good at summarizing content. Please summarize a specific and detailed answer or report based on the previous queries and the retrieved document chunks.

    Original Query: {question}
    Previous Sub Queries: {mini_questions}
    Related Chunks: 
    {mini_chunk_str}
    """
    return summary_prompt
```


This approach has the advantage over our prototype, which analyzed each question separately and simply concatenated the output, of producing a report where all sections are consistent with each other, i.e., containing no repeated or contradictory information. A more complex system could combine aspects of both, using a conditional execution flow to structure the report, summarize, rewrite, reflect and pivot, and so on, which we leave for future work.


## Results

Here is a sample from the report generated by the query “How has The Simpsons changed over time?” with DeepSeek-R1 passing the Wikipedia page on The Simpsons as source material:

```txt
Report: The Evolution of The Simpsons (1989–Present)
1. Cultural Impact and Societal Relevance
The Simpsons debuted as a subversive critique of American middle-class life, gaining notoriety for its bold satire in the 1990s. Initially a countercultural phenomenon, it challenged norms with episodes tackling religion, politics, and consumerism. Over time, its cultural dominance waned as competitors like South Park and Family Guy pushed boundaries further. By the 2010s, the show transitioned from trendsetter to nostalgic institution, balancing legacy appeal with attempts to address modern issues like climate change and LGBTQ+ rights, albeit with less societal resonance.
…
Conclusion
The Simpsons evolved from a radical satire to a television institution, navigating shifts in technology, politics, and audience expectations. While its golden-age brilliance remains unmatched, its adaptability—through streaming, updated humor, and global outreach—secures its place as a cultural touchstone. The show’s longevity reflects both nostalgia and a pragmatic embrace of change, even as it grapples with the challenges of relevance in a fragmented media landscape.
```


Find [the full report here](https://drive.google.com/file/d/1GE3rvxFFTKqro67ctTkknryUf-ojhduN/view?usp=sharing), and [a report produced by DeepSearcher with GPT-4o mini](https://drive.google.com/file/d/1EGd16sJDNFnssk9yTd5o9jzbizrY_NS_/view?usp=sharing) for comparison.


## Discussion

We presented [DeepSearcher](https://github.com/zilliztech/deep-searcher), an agent for performing research and writing reports. Our system is built upon the idea in our previous article, adding features like conditional execution flow, query routing, and an improved interface. We switched from local inference with a small 4-bit quantized reasoning model to an online inference service for the massive DeepSeek-R1 model, qualitatively improving our output report. DeepSearcher works with most inference services like OpenAI, Gemini, DeepSeek and Grok 3 (coming soon!).

Reasoning models, especially as used in research agents, are inference-heavy, and we were fortunate to be able to use the fastest offering of DeepSeek-R1 from SambaNova running on their custom hardware. For our demonstration query, we made sixty-five calls to SambaNova’s DeepSeek-R1 inference service, inputting around 25k tokens, outputting 22k tokens, and costing $0.30. We were impressed with the speed of inference given that the model contains 671-billion parameters and is 3/4 of a terabyte large. [Find out more details here!](https://sambanova.ai/press/fastest-deepseek-r1-671b-with-highest-efficiency)

We will continue to iterate on this work in future posts, examining additional agentic concepts and the design space of research agents. In the meanwhile, we invite everyone to try out [DeepSearcher](https://github.com/zilliztech/deep-searcher), [star us on GitHub](https://github.com/zilliztech/deep-searcher), and share your feedback!


## Resources

- [**Zilliz’s DeepSearcher**](https://github.com/zilliztech/deep-searcher)

- Background reading: [**_“I Built a Deep Research with Open Source—and So Can You!”_**](https://milvus.io/blog/i-built-a-deep-research-with-open-source-so-can-you.md)

- _“_[__SambaNova Launches the Fastest DeepSeek-R1 671B with the Highest Efficiency__](https://sambanova.ai/press/fastest-deepseek-r1-671b-with-highest-efficiency)_”_

- DeepSearcher: [DeepSeek-R1 report on The Simpsons](https://drive.google.com/file/d/1GE3rvxFFTKqro67ctTkknryUf-ojhduN/view?usp=sharing)

- DeepSearcher: [GPT-4o mini report on The Simpsons](https://drive.google.com/file/d/1EGd16sJDNFnssk9yTd5o9jzbizrY_NS_/view?usp=sharing)

- [Milvus Open-Source Vector Database](https://milvus.io/docs)


Introducing DeepSearcher: A Local Open Source Deep Research


## Overview

As Milvus flourished in 2024 with major releases and a thriving open-source ecosystem, a hidden treasure trove of user insights was quietly forming in our community on [Discord](https://discord.gg/xwqmFDURcz). This compilation of community discussions presented a unique opportunity to understand our users' challenges firsthand. Intrigued by this untapped resource, I embarked on a comprehensive analysis of every discussion thread from the year, searching for patterns that could help us compile a frequently asked questions resource for Milvus users.

![](https://assets.zilliz.com/top_image_6bbdbe8caa.png)

My analysis revealed three primary areas where users consistently sought guidance: **Performance Optimization**, **Deployment Strategies**, and **Data Management**. Users frequently discussed how to fine-tune Milvus for production environments and track performance metrics effectively. When it came to deployment, the community grappled with selecting appropriate deployments, choosing optimal search indices, and resolving issues in distributed setups. The data management conversations centered around service-to-service data migration strategies and the selection of embedding models.

Let’s examine each of these areas in more detail.

## Deployment

![](https://assets.zilliz.com/deployment_c951c46339.png)

Milvus provides flexible deployment modes to fit various use cases. However, some users do find it challenging to find the right choice, and want to feel comfortable that they are doing so “correctly.”


### Which deployment type should I choose?

A very frequent question is which deployment to choose out of Milvus [Lite](https://milvus.io/docs/milvus_lite.md), [Standalone](https://milvus.io/docs/prerequisite-docker.md), and [Distributed](https://milvus.io/docs/prerequisite-helm.md). The answer primarily depends on how large your vector database needs to be and how much traffic it will serve: 


#### Milvus Lite

When prototyping on your local system with up to a few million vectors, or looking for an embedded vector db for unit testing and CI/CD, you can use Milvus Lite. Note that some more advanced features like full-text search are not yet available within Milvus Lite but coming soon.


#### Milvus Standalone

If your system needs to serve production traffic and / or you need to store between a few million and a hundred-million vectors, you should use Milvus Standalone, which packs all components of Milvus into a single Docker image. There is a variation which just takes its persistent storage (minio) and metadata store (etcd) dependencies out as separate images. 


#### Milvus Distributed

For any larger scale deployments serving production traffic, like serving billions of vectors at thousands of QPS, you should use Milvus Distributed. Some users may want to perform offline batch processing at scale, for example, for data deduplication or record linkage, and the future version of Milvus 3.0 will provide a more efficient way of doing this by what we term a vector lake.


#### Fully Managed Service

For developers who want to focus on the application development without worrying about DevOps, [Zilliz Cloud](https://cloud.zilliz.com/signup) is the fully managed Milvus that offers a free tier.

See [“Overview of Milvus Deployments”](https://milvus.io/docs/install-overview.md#Choose-the-Right-Deployment-for-Your-Use-Case) for more information.


### How much memory, storage, and compute will I require?

This question comes up a lot, not only for existing Milvus users but also those who are considering whether Milvus is appropriate for their application. The exact combination of how much memory, storage, and compute a deployment will require depends on complex interaction of factors.

Vector embeddings differ in dimensionality due to the model that is used. And some vector search indexes are stored entirely in memory, whereas others store data to disk. Also, many search indexes are able to store a compressed (quantized) copy of the embeddings and require additional memory for graph data structures. These are just a few factors that affect the memory and storage.


#### Milvus Resource Sizing Tool

Luckily, Zilliz (the team that maintains Milvus) has built [a resource sizing tool](https://milvus.io/tools/sizing) that does a fantastic job of answering this question. Input your vector dimensionality, index type, deployment options, and so on and the tool estimates CPU, memory, and storage needed across the various types of Milvus nodes and its dependencies. Your mileage may vary so a real load testing with your data and sample traffic is always a good idea. 


### Which vector index or distance metric should I choose?

Many users are uncertain which index they should choose and how to set the hyperparameters. First, it is always possible to defer the choice of index type to Milvus by selecting AUTOINDEX. If you wish to select a specific index type, however, a few rules of thumb provide a starting point.


#### In-Memory Indexes

Would you like to pay the cost to fit your index entirely into memory? An in-memory index is typically the fastest but also expensive. See [“In-memory indexes”](https://milvus.io/docs/index.md?tab=floating) for a list of the ones supported by Milvus and the tradeoffs they make in terms of latency, memory, and recall.

Keep in mind that your index size is not simply the number of vectors times their dimensionality and floating point size. Most indexes quantize the vectors to reduce memory usage, but require memory for additional data structures. Other non-vector data (scalar) and their index also takes up memory space.


#### On-Disk Indexes

When your index does not fit in memory, you can use one of the [“On-disk indexes”](https://milvus.io/docs/disk_index.md) provided by Milvus. Two choices with very different latency/resource tradeoffs are [DiskANN](https://milvus.io/docs/disk_index.md) and [MMap](https://milvus.io/docs/mmap.md#MMap-enabled-Data-Storage). 

DiskANN stores a highly compressed copy of the vectors in memory, and the uncompressed vectors and graph search structures on disk. It uses some clever ideas to search the vector space while minimizing disk reads and takes advantage of the fast random access speed of SSDs. For minimum latency, the SSD must be connected via NVMe rather than SATA for best I/O performance. 

Technically speaking, MMap is not an index type, but refers to the use of virtual memory with an in-memory index. With virtual memory, pages can be swapped between disk and RAM as required, which allows a much larger index to be used efficiently if the access patterns are such that only a small portion of the data is used at a time.

DiskANN has excellent and consistent latency. MMap has even better latency when it is accessing a page in-memory, but frequent page-swapping will cause latency spikes. Thus MMap can have a higher variability in latency, depending on the memory access patterns.


#### GPU Indexes

A third option is to construct [an index using GPU memory and compute](https://milvus.io/docs/gpu_index.md). Milvus’ GPU support is contributed by the Nvidia [RAPIDS](https://rapids.ai/) team. GPU vector search may have lower latency than a corresponding CPU search, although it usually takes hundreds or thousands of search QPS to fully exploit the parallelism of GPU. Also, GPUs typically have less memory than the CPU RAM and are more costly to run.


#### Distance Metrics

An easier question to answer is which distance metric should you choose to measure similarity between vectors. It is recommended to choose the same distance metric that your embedding model was trained with, which is typically COSINE (or IP when inputs are normalized). The source of your model (e.g. the model page on HuggingFace) will provide clarification on which distance metric was used. Zilliz also put together a convenient [table](https://zilliz.com/ai-models) to look that up.

To summarize, I think a lot of the uncertainty around index choice revolves around uncertainty about how these choices affect the latency/resource usage/recall tradeoff of your deployment. I recommend using the rules of thumb above to decide between in-memory, on-disk, or GPU indexes, and then using the tradeoff guidelines given in the Milvus documentation to pick a particular one.


### Can you fix my broken Milvus Distributed deployment?

Many questions revolve around issues getting a Milvus Distributed deployment up and running, with questions relating to configuration, tooling, and debugging logs. It’s hard to give a single fix as each question seems different from the last, although luckily Milvus has [a vibrant Discord](https://milvus.io/discord) where you can seek help, and we also offer [1-on-1 office hours with an expert](https://milvus.io/blog/join-milvus-office-hours-to-get-support-from-vectordb-experts.md).


### How do I deploy Milvus on Windows?

A question that has come up several times is how to deploy Milvus on Windows machines. Based on your feedback, we have rewritten the documentation for this: see [Run Milvus in Docker (Windows)](https://milvus.io/docs/install_standalone-windows.md) for how to do this, using [Windows Subsystem for Linux 2 (WSL2)](https://learn.microsoft.com/en-us/windows/wsl/about#what-is-wsl-2).


## Performance and Profiling

![](https://assets.zilliz.com/Performance_and_Profiling_481975ea1d.png)

Having chosen a deployment type and got it running, users want to feel comfortable that they have made optimal decisions and would like to profile their deployment’s performance and state. Many questions related to how to profile performance, observe state, and get an insight into what and why. 


### How do I measure performance?

Users want to check metrics related to the performance of their deployment so they can understand and remedy bottlenecks. Metrics mentioned include average query latency, distribution of latencies, query volume, memory usage, disk storage, and so on. While obtaining these metrics with [the legacy monitoring system](https://milvus.io/docs/monitor_overview.md) has been challenging, Milvus 2.5 introduces a new system called [WebUI](https://milvus.io/docs/milvus-webui.md#Milvus-WebUI) (feedback welcome!), which allows you to access all this information from a user-friendly web interface.


### What’s happening inside Milvus right now (i.e. observe state)?

Relatedly, users want to observe the internal state of their deployment. Issues raised include understanding why a search index is taking so long to build, how to determine if the cluster is healthy, and understanding how a query is executed across nodes. Many of these questions can be answered with the new [WebUI](https://milvus.io/docs/milvus-webui.md#Milvus-WebUI) that gives transparency to what the system is doing internally.


### How does some (complex) aspect of the internals work?

Advanced users often want some understanding of Milvus internals, for example, having an understanding of the sealing of segments or memory management. The underlying goal is typically to improve performance and sometimes to debug issues. The documentation, particularly under the sections “Concepts” and “Administration Guide" is helpful here, for instance see the pages [“Milvus Architecture Overview”](https://milvus.io/docs/architecture_overview.md) and [“Clustering Compaction”](https://milvus.io/docs/clustering-compaction.md). We will continue to improve the documentation on Milvus internals, make it easier to understand, and welcome any feedback or requests via [Discord](https://milvus.io/discord).


### Which embedding model should I choose?

A question related to performance that has come up multiple times in meetups, office hours, and on Discord is how to choose an embedding model. This is a difficult question to give a definitive answer although we recommend starting with default models like [all-MiniLM-L6-v2](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2). 

Similar to the choice of search index, there are tradeoffs between compute, storage, and recall. An embedding model with larger output dimension will require more storage, all else held equal, although probably result in higher recall of relevant items. Larger embedding models, for a fixed dimension, typically outperform smaller ones in terms of recall, although at the cost of increased compute and time. Leaderboards that rank embedding model performance such as [MTEB](https://huggingface.co/spaces/mteb/leaderboard) are based on benchmarks that may not align with your specific data and task.

So, it does not make sense to think of a “best” embedding model. Start with one that has acceptable recall and meets your compute and time budget for calculating embeddings. Further optimizations like fine-tuning on your data or exploring the compute/recall tradeoff empirically can be deferred to after you have a working system in production.


## Data Management

![](https://assets.zilliz.com/Data_Management_aa2d1159bd.png)

How to move data into and out of a Milvus deployment is another main theme in the Discord discussions, which is no surprise given how central this task is to putting an application into production.


### How do I migrate data from X to Milvus? How do I migrate data from Standalone to Distributed? How do I migrate from 2.4.x to 2.5.x?

A new user commonly wants to get existing data into Milvus from another platform, including traditional search engines like [Elasticsearch](https://docs.zilliz.com/docs/migrate-from-elasticsearch) and other vector databases like [Pinecone](https://docs.zilliz.com/docs/migrate-from-pinecone) or [Qdrant](https://docs.zilliz.com/docs/migrate-from-qdrant). Existing users may also want to migrate their data from one Milvus deployment to another, or [from self-hosted Milvus to fully managed Zilliz Cloud](https://docs.zilliz.com/docs/migrate-from-milvus).

The [Vector Transport Service (VTS)](https://github.com/zilliztech/vts) and the managed [Migration](https://docs.zilliz.com/docs/migrations) service on Zilliz Cloud are designed for this purpose.


### How do I save and load data backups? How do I export data from Milvus?

Milvus has a dedicated tool, [milvus-backup](https://github.com/zilliztech/milvus-backup), to take snapshots on permanent storage and restore them.


## Next Steps

I hope this has given you some pointers on how to tackle common challenges faced when building with a vector database. This definitely helped us to take another look at our documentation and feature roadmap to keep working on things that can help our community best succeed with Milvus. A key takeaway that I would like to emphasize, is that your choices put you within different points of a tradeoff space between compute, storage, latency, and recall. _You cannot maximize all of these performance criteria simultaneously - there is no “optimal” deployment. Yet, by understanding more about how vector search and distributed database systems work you can make an informed decision._

After trawling through the large number of posts from 2024, it got me thinking: why should a human do this? Has not Generative AI promised to solve such a task of crunching large amounts of text and extracting insight? Join me in the second part of this blog post (coming soon), where I investigate the design and implementation of _a multi-agent system for extracting insight from discussion forums._

Thanks again and hope to see you in the community [Discord](https://milvus.io/discord) and our next [Unstructured Data](https://lu.ma/unstructured-data-meetup) meetups. For more hands-on assistance, we welcome you to book a [1-on-1 office hour](https://milvus.io/blog/join-milvus-office-hours-to-get-support-from-vectordb-experts.md). _Your feedback is essential to improving Milvus!_


What Milvus Users Taught Us in 2024


![](https://assets.zilliz.com/deep_research_blog_image_95225226eb.png)

Well actually, a minimally scoped agent that can reason, plan, use tools, etc. to perform research using Wikipedia. Still, not bad for a few hours of work…

Unless you reside under a rock, in a cave, or in a remote mountain monastery, you will have heard about OpenAI’s release of _Deep Research_ on Feb 2, 2025. This new product promises to revolutionize how we answer questions requiring the synthesis of large amounts of diverse information.

You type in your query, select the Deep Research option, and the platform autonomously searches the web, performs reasoning on what it discovers, and synthesizes multiple sources into a coherent, fully-cited report. It takes several orders of magnitude longer to produce its output relative to a standard chatbot, but the result is more detailed, more informed, and more nuanced.


## How does it work?

But how does this technology work, and why is Deep Research a noticeable improvement over previous attempts (like Google’s _Deep Research_ - incoming trademark dispute alert)? We’ll leave the latter for a future post. As for the former, there is no doubt much “secret sauce” underlying Deep Research. We can glean a few details from OpenAI’s release post, which I summarize.

**Deep Research exploits recent advances in foundation models specialized for reasoning tasks:**

- “...fine-tuned on the upcoming OpenAI o3 reasoning model…”

- “...leverages reasoning to search, interpret, and analyze massive amounts of text…”

**Deep Research makes use of a sophisticated agentic workflow with planning, reflection, and memory:**

- “...learned to plan and execute a multi-step trajectory…”

- “...backtracking and reacting to real-time information…”

- “...pivoting as needed in reaction to information it encounters…”

**Deep Research is trained on proprietary data, using several types of fine-tuning, which is likely a key component in its performance:**

- “...trained using end-to-end reinforcement learning on hard browsing and reasoning tasks across a range of domains…”

- “...optimized for web browsing and data analysis…”

The exact design of the agentic workflow is a secret, however, we can build something ourselves based on well-established ideas about how to structure agents.

**One note before we begin**: It is easy to be swept away by Generative AI fever, especially when a new product that seems a step-improvement is released. However, Deep Research, as OpenAI acknowledges, has limitations common to Generative AI technology. We should remember to think critically about the output in that it may contain false facts (“hallucinations”), incorrect formatting and citations, and vary significantly in quality based on the random seed.


## Can I build my own?

Why certainly! Let’s build our own “Deep Research”, running locally and with open-source tools. We’ll be armed with just a basic knowledge of Generative AI, common sense, a couple of spare hours, a GPU, and the open-source [Milvus](https://milvus.io/docs), [DeepSeek R1](https://huggingface.co/unsloth/DeepSeek-R1-Distill-Llama-8B-unsloth-bnb-4bit), and [LangChain](https://python.langchain.com/docs/introduction/).

We cannot hope to replicate OpenAI’s performance of course, but our prototype will minimally demonstrate some of the key ideas likely underlying their technology, combining advances in reasoning models with advances in agentic workflows. Importantly, and unlike OpenAI, we will be using only open-source tools, and be able to deploy our system locally - open-source certainly provides us great flexibility!

We will make a few simplifying assumptions to reduce the scope of our project:

- We will use an open-source reasoning mode distilled then [quantized](https://zilliz.com/learn/unlock-power-of-vector-quantization-techniques-for-efficient-data-compression-and-retrieval) for 4-bits that can be run locally.

- We will not perform additional fine-tuning on our reasoning model ourselves.

- The only tool our agent has is the ability to download and read a Wikipedia page and perform separate RAG queries (we will not have access to the entire web).

- Our agent will only process text data, not images, PDFs, etc.

- Our agent will not backtrack or consider pivots.

- Our agent will (not yet) control its execution flow based on its output.

- Wikipedia contains the truth, the whole truth and nothing but the truth.

We will use [Milvus](https://milvus.io/docs) for our vector database, [DeepSeek R1](https://huggingface.co/unsloth/DeepSeek-R1-Distill-Llama-8B-unsloth-bnb-4bit) as our reasoning model, and [LangChain](https://python.langchain.com/docs/introduction/) to implement RAG. Let’s get started! 


# A Minimal Agent for Online Research

We will use our mental model of how humans conduct research to design the agentic workflow:


### Define/Refine Question

Research starts by defining a question. We take the question to be the user’s query, however, we use our reasoning model to ensure the question is expressed in a way that is specific, clear, and focused. That is, our first step is to rewrite the prompt and extract any subqueries or subquestions. We make effective use of our foundation models specialization for reasoning, and a simple method for JSON structured output.

Here is an example reasoning trace as DeepSeek refines the question “How has the cast changed over time?”:

```text
<think>

Alright, so I need to break down the question "How has the cast changed over time?" related to the evolution of The Simpsons. Let me think about what aspects are involved here.

First, I know that the cast has changed a lot, but I need to figure out the intermediate questions to approach this. The main question is about changes in the cast over time, so I should consider different areas that contribute to this change.

I guess the first sub-question would be about the original cast members. Who were the main voices and how did they evolve? Then, there might be new cast additions over the years, so another sub-question about that.

Also, some original voice actors have left, so I should include a sub-question about departures. Then, new voice actors joining would be another point.

The show has been popular for a long time, so recurring roles changing might be another aspect. Additionally, the role of the show in society might have influenced casting choices, so a sub-question about that.

Lastly, the overall impact on the cast's careers could be another angle. So, I should list these as sub-questions to cover all aspects.

</think>
```

### Search

Next, we conduct a “literature review” of Wikipedia articles. For now, we read a single article and leave navigating links to a future iteration. We discovered during prototyping that link exploration can become very expensive if each link requires a call to the reasoning model. We parse the article, and store its data in our vector database, Milvus, akin to taking notes.

Here is a code snippet showing how we store our Wikipedia page in Milvus using its LangChain integration:

```python
wiki_wiki = wikipediaapi.Wikipedia(user_agent='MilvusDeepResearchBot (<insert your email>)', language='en')
page_py = wiki_wiki.page(page_title)

text_splitter = RecursiveCharacterTextSplitter(chunk_size=2000, chunk_overlap=200)
docs = text_splitter.create_documents([page_py.text])

vectorstore = Milvus.from_documents(  # or Zilliz.from_documents
    documents=docs,
    embedding=embeddings,
    connection_args={
        "uri": "./milvus_demo.db",
    },
    drop_old=True, 
    index_params={
        "metric_type": "COSINE",
        "index_type": "FLAT",  
        "params": {},
    },
)
```

### Analyze

The agent returns to its questions and answers them based on the relevant information in the document. We will leave a multi-step analysis/reflection workflow for future work, as well as any critical thinking on the credibility and bias of our sources.

Here is a code snippet illustrating constructing a RAG with LangChain and answering our subquestions separately.

```python
# Define the RAG chain for response generation
rag_chain = (
    {"context": retriever | format_docs, "question": RunnablePassthrough()}
    | prompt
    | llm
    | StrOutputParser()
)

# Prompt the RAG for each question
answers = {}
total = len(leaves(breakdown))

pbar = tqdm(total=total)
for k, v in breakdown.items():
    if v == []:
        print(k)
        answers[k] = rag_chain.invoke(k).split('</think>')[-1].strip()
        pbar.update(1)
    else:
        for q in v:
            print(q)
            answers[q] = rag_chain.invoke(q).split('</think>')[-1].strip()
            pbar.update(1)
```

### Synthesize

After the agent has performed its research, it creates a structured outline, or rather, a skeleton, of its findings to summarize in a report. It then completes each section, filling it in with a section title and the corresponding content. We leave a more sophisticated workflow with reflection, reordering, and rewriting for a future iteration. This part of the agent involves planning, tool usage, and memory.

See [accompanying notebook](https://drive.google.com/file/d/1waKX_NTgiY-47bYE0cI6qD8Cjn3zjrL6/view?usp=sharing) for the full code and the [saved report file](https://drive.google.com/file/d/15xeEe_EqY-29V2IlAvDy5yGdJdEPSHOh/view?usp=drive_link) for example output.


## Results

Our query for testing is _“How has The Simpsons changed over time?”_ and the data source is the Wikipedia article for “The Simpsons”. Here is one section of the [generated report](https://drive.google.com/file/d/15xeEe_EqY-29V2IlAvDy5yGdJdEPSHOh/view?usp=sharing):

![](https://assets.zilliz.com/result_query_424beba224.jpg)


## Summary: What we built and what’s next

In just a few hours, we have designed a basic agentic workflow that can reason, plan, and retrieve information from Wikipedia to generate a structured research report. While this prototype is far from OpenAI’s Deep Research, it demonstrates the power of open-source tools like Milvus, DeepSeek, and LangChain in building autonomous research agents. 

Of course, there’s plenty of room for improvement. Future iterations could:

- Expand beyond Wikipedia to search multiple sources dynamically

- Introduce backtracking and reflection to refine responses

- Optimize execution flow based on the agent's own reasoning

Open-source gives us flexibility and control that closed source doesn’t. Whether for academic research, content synthesis, or AI-powered assistance, building our own research agents open up exciting possibilities. Stay tuned for the next post where we explore adding real-time web retrieval, multi-step reasoning, and conditional execution flow!


## Resources

- Notebook: _“_[_Baseline for An Open-Source Deep Research_](https://colab.research.google.com/drive/1W5tW8SqWXve7ZwbSb9pVdbt5R2wq105O?usp=sharing)_”_

- Report: _“_[_The evolution of The Simpsons as a show over time, covering changes in content, humor, character development, animation, and its role in society._](https://drive.google.com/file/d/15xeEe_EqY-29V2IlAvDy5yGdJdEPSHOh/view?usp=drive_link)_”_

- [Milvus vector database documentation](https://milvus.io/docs)

- [Distilled and quantized DeepSeek R1 model page](https://huggingface.co/unsloth/DeepSeek-R1-Distill-Llama-8B-unsloth-bnb-4bit)

- [️🔗 LangChain](https://python.langchain.com/docs/introduction/)

- [Deep Research FAQ | OpenAI Help Center](https://help.openai.com/en/articles/10500283-deep-research-faq)





I Built a Deep Research with Open Source—and So Can You!

Multimodal Semantic Search with Images and Text

How to Get Started with Milvus



By now, everyone knows that hybrid search has improved [RAG](https://zilliz.com/learn/Retrieval-Augmented-Generation) (Retrieval-Augmented Generation) search quality. While [dense embedding](https://zilliz.com/learn/sparse-and-dense-embeddings) search has shown impressive capabilities in capturing deep semantic relationships between queries and documents, it still has notable limitations. These include a lack of explainability and suboptimal performance with long-tail queries and rare terms.

Many RAG applications struggle because pre-trained models often lack domain-specific knowledge. In some scenarios, simple BM25 keyword matching outperforms these sophisticated models. This is where hybrid search bridges the gap, combining the semantic understanding of dense vector retrieval with the precision of keyword matching.


## Why Hybrid Search is Complex in Production

While frameworks like [LangChain](https://zilliz.com/learn/LangChain) or [LlamaIndex](https://zilliz.com/learn/getting-started-with-llamaindex) make building a proof-of-concept hybrid retriever easy, scaling to production with massive datasets is challenging. Traditional architectures require separate vector databases and search engines, leading to several key challenges:

- High infrastructure maintenance costs and operational complexity

- Data redundancy across multiple systems

- Difficult data consistency management

- Complex security and access control across systems

The market needs a unified solution that supports lexical and semantic search while reducing system complexity and cost.

![](https://assets.zilliz.com/elasticsearch_vs_milvus_5be6e2b69e.png)


## The Pain Points of Elasticsearch

Elasticsearch has been one of the past decade's most influential open-source search projects. Built on Apache Lucene, it gained popularity through its high performance, scalability, and distributed architecture. While it added vector ANN search in version 8.0, production deployments face several critical challenges:

**High Update and Indexing Costs:** Elasticsearch's architecture doesn't fully decouple write operations, index building, and querying. This leads to significant CPU and I/O overhead during write operations, especially in bulk updates. The resource contention between indexing and querying impacts performance, creating a major bottleneck for high-frequency update scenarios.

**Poor Real-time Performance:** As a "near real-time" search engine, Elasticsearch introduces noticeable latency in data visibility. This latency becomes particularly problematic for AI applications, such as Agent systems, where high-frequency interactions and dynamic decision-making require immediate data access.

**Difficult Shard Management:** While Elasticsearch uses sharding for distributed architecture, shard management poses significant challenges. The lack of dynamic sharding support creates a dilemma: too many shards in small datasets lead to poor performance, while too few shards in large datasets limit scalability and cause uneven data distribution.

**Non-Cloud-Native Architecture:** Developed before cloud-native architectures became prevalent, Elasticsearch's design tightly couples storage and compute, limiting its integration with modern infrastructure like public clouds and Kubernetes. Resource scaling requires simultaneous increases in both storage and compute, reducing flexibility. In multi-replica scenarios, each shard must build its index independently, increasing computational costs and reducing resource efficiency.

**Poor Vector Search Performance:** Though Elasticsearch 8.0 introduced vector ANN search, its performance significantly lags behind that of dedicated vector engines like Milvus. Based on the Lucene kernel, its index structure proves inefficient for high-dimensional data, struggling with large-scale vector search requirements. Performance becomes particularly unstable in complex scenarios involving scalar filtering and multi-tenancy, making it challenging to support high-load or diverse business needs.

**Excessive Resource Consumption:** Elasticsearch places extreme demands on memory and CPU, especially when processing large-scale data. Its JVM dependency requires frequent heap size adjustments and garbage collection tuning, severely impacting memory efficiency. Vector search operations require intensive SIMD-optimized computations, for which the JVM environment is far from ideal.

These fundamental limitations become increasingly problematic as organizations scale their AI infrastructure, making Elasticsearch particularly challenging for modern AI applications requiring high performance and reliability.


## Introducing Sparse-BM25: Reimagining Lexical Search

[Milvus 2.5](https://milvus.io/blog/introduce-milvus-2-5-full-text-search-powerful-metadata-filtering-and-more.md) introduces native lexical search support through Sparse-BM25, building upon the hybrid search capabilities introduced in version 2.4. This innovative approach includes the following key components:

- Advanced tokenization and preprocessing via Tantivy

- Distributed vocabulary and term frequency management

- Sparse vector generation using corpus TF and query TF-IDF

- Inverted index support with WAND algorithm (Block-Max WAND and graph index support in development)

Compared to Elasticsearch, Milvus offers significant advantages in algorithm flexibility. Its vector distance-based similarity computation enables more sophisticated matching, including implementing TW-BERT (Term Weighting BERT) based on "End-to-End Query Term Weighting" research. This approach has demonstrated superior performance in both in-domain and out-domain testing.

Another crucial advantage is cost efficiency. By leveraging both inverted index and dense embedding compression, Milvus achieves a fivefold performance improvement with less than 1% recall degradation. Through tail-term pruning and vector quantization, memory usage has been reduced by over 50%.

Long query optimization stands out as a particular strength. Where traditional WAND algorithms struggle with longer queries, Milvus excels by combining sparse embeddings with graph indices, delivering a tenfold performance improvement in high-dimensional sparse vector search scenarios.

![](https://assets.zilliz.com/document_in_and_out_b84771bec4.png)


## Milvus: The Ultimate Vector Database for RAG

Milvus is the premier choice for RAG applications through its comprehensive feature set. Key advantages include:

- Rich metadata support with dynamic schema capabilities and powerful filtering options

- Enterprise-grade multi-tenancy with flexible isolation through collections, partitions, and partition keys

- Industry-first disk vector index support with multi-tier storage from memory to S3

- Cloud-native scalability supporting seamless scaling from 10M to 1B+ vectors

- Comprehensive search capabilities, including grouping, range, and hybrid search

- Deep ecosystem integration with LangChain, LlamaIndex, Dify, and other AI tools

The system's diverse search capabilities encompass grouping, range, and hybrid search methodologies. Deep integration with tools like LangChain, LlamaIndex, and Dify, as well as support for numerous AI products, places Milvus at the center of the modern AI infrastructure ecosystem.


## Looking Forward

As AI transitions from POC to production, Milvus continues to evolve. We focus on making vector search more accessible and cost-effective while enhancing search quality. Whether you're a startup or an enterprise, Milvus reduces the technical barriers to AI application development.

This commitment to accessibility and innovation has led us to another major step forward. While our open-source solution continues to serve as the foundation for thousands of applications worldwide, we recognize that many organizations need a fully managed solution that eliminates operational overhead.


## Zilliz Cloud: The Managed Solution

We've built [Zilliz Cloud](https://zilliz.com/cloud), a fully managed vector database service based on Milvus, over the past three years. Through a cloud-native reimplementation of the Milvus protocol, it offers enhanced usability, cost efficiency, and security.

Drawing from our experience maintaining the world's largest vector search clusters and supporting thousands of AI application developers, Zilliz Cloud significantly reduces operational overhead and costs compared to self-hosted solutions.

Ready to experience the future of vector search? Start your free trial today with up to $200 in credits, no credit card required.


Elasticsearch is Dead, Long Live Lexical Search


In this article, we will show you how to quickly get up and running with the new full-text search feature and combine it with the conventional semantic search based on vector embeddings.


<iframe width="100%" height="480" src="https://www.youtube.com/embed/3bftbAjQF7Q" title="Beyond Keywords: Hybrid Search with Milvus 2.5" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe>


## Requirement

First, ensure you have installed Milvus 2.5:


```
pip install -U pymilvus[model]
```

and have a running instance of Milvus Standalone (e.g. on your local machine) using the [installation instructions in the Milvus docs](https://milvus.io/docs/prerequisite-docker.md).

## Building the Data Schema and Search Indices

We import the required classes and functions:

```
from pymilvus import MilvusClient, DataType, Function, FunctionType, model
```

You may have noticed two new entries for Milvus 2.5, `Function` and `FunctionType`, which we will explain shortly.

Next we open the database with Milvus Standalone, that is, locally, and create the data schema. The schema comprises an integer primary key, a text string, a dense vector of dimension 384, and a sparse vector (of unlimited dimensionality).
Note that Milvus Lite does not currently support full-text search, only Milvus Standalone and Milvus Distributed.

```
client = MilvusClient(uri="http://localhost:19530")

schema = client.create_schema()

schema.add_field(field_name="id", datatype=DataType.INT64, is_primary=True, auto_id=True)
schema.add_field(field_name="text", datatype=DataType.VARCHAR, max_length=1000, enable_analyzer=True)
schema.add_field(field_name="dense", datatype=DataType.FLOAT_VECTOR, dim=768),
schema.add_field(field_name="sparse", datatype=DataType.SPARSE_FLOAT_VECTOR)
```

```
{'auto_id': False, 'description': '', 'fields': [{'name': 'id', 'description': '', 'type': <DataType.INT64: 5>, 'is_primary': True, 'auto_id': True}, {'name': 'text', 'description': '', 'type': <DataType.VARCHAR: 21>, 'params': {'max_length': 1000, 'enable_analyzer': True}}, {'name': 'dense', 'description': '', 'type': <DataType.FLOAT_VECTOR: 101>, 'params': {'dim': 768}}, {'name': 'sparse', 'description': '', 'type': <DataType.SPARSE_FLOAT_VECTOR: 104>}], 'enable_dynamic_field': False}
```

You may have noticed the `enable_analyzer=True` parameter. This tells Milvus 2.5 to enable the lexical parser on this field and build a list of tokens and token frequencies, which are required for full-text search. The `sparse` field will hold a vector representation of the documentation as a bag-of-words produced from the parsing `text`.

But how do we connect the `text` and `sparse` fields, and tell Milvus how `sparse` should be calculated from `text`? This is where we need to invoke the `Function` object and add it to the schema:


```
bm25_function = Function(
    name="text_bm25_emb", # Function name
    input_field_names=["text"], # Name of the VARCHAR field containing raw text data
    output_field_names=["sparse"], # Name of the SPARSE_FLOAT_VECTOR field reserved to store generated embeddings
    function_type=FunctionType.BM25,
)

schema.add_function(bm25_function)
```


```
{'auto_id': False, 'description': '', 'fields': [{'name': 'id', 'description': '', 'type': <DataType.INT64: 5>, 'is_primary': True, 'auto_id': True}, {'name': 'text', 'description': '', 'type': <DataType.VARCHAR: 21>, 'params': {'max_length': 1000, 'enable_analyzer': True}}, {'name': 'dense', 'description': '', 'type': <DataType.FLOAT_VECTOR: 101>, 'params': {'dim': 768}}, {'name': 'sparse', 'description': '', 'type': <DataType.SPARSE_FLOAT_VECTOR: 104>, 'is_function_output': True}], 'enable_dynamic_field': False, 'functions': [{'name': 'text_bm25_emb', 'description': '', 'type': <FunctionType.BM25: 1>, 'input_field_names': ['text'], 'output_field_names': ['sparse'], 'params': {}}]}
```

The abstraction of the `Function` object is more general than that of applying full-text search. In the future, it may be used for other cases where one field needs to be a function of another field. In our case, we specify that `sparse` is a function of `text` via the function `FunctionType.BM25`. `BM25` refers to a common metric in information retrieval used for calculating a query's similarity to a document (relative to a collection of documents).

We use the default embedding model in Milvus, which is [paraphrase-albert-small-v2](https://huggingface.co/GPTCache/paraphrase-albert-small-v2):


```
embedding_fn = model.DefaultEmbeddingFunction()
```

The next step is to add our search indices. We have one for the dense vector and a separate one for the sparse vector. The index type is `SPARSE_INVERTED_INDEX` with `BM25` since full-text search requires a different search method than those for standard dense vectors.


```
index_params = client.prepare_index_params()

index_params.add_index(
    field_name="dense",
    index_type="AUTOINDEX", 
    metric_type="COSINE"
)

index_params.add_index(
    field_name="sparse",
    index_type="SPARSE_INVERTED_INDEX", 
    metric_type="BM25"
)
```

Finally, we create our collection:

```
client.drop_collection('demo')
client.list_collections()
```

```
[]
```

```
client.create_collection(
    collection_name='demo', 
    schema=schema, 
    index_params=index_params
)

client.list_collections()
```


```
['demo']
```
And with that, we have an empty database set up to accept text documents and perform semantic and full-text searches!

## Inserting Data and Performing Full-Text Search

Inserting data is no different than previous versions of Milvus:

```
docs = [
    'information retrieval is a field of study.',
    'information retrieval focuses on finding relevant information in large datasets.',
    'data mining and information retrieval overlap in research.'
]

embeddings = embedding_fn(docs)

client.insert('demo', [
    {'text': doc, 'dense': vec} for doc, vec in zip(docs, embeddings)
])
```

```
{'insert_count': 3, 'ids': [454387371651630485, 454387371651630486, 454387371651630487], 'cost': 0}
```

Let's first illustrate a full-text search before we move on to hybrid search:

```
search_params = {
    'params': {'drop_ratio_search': 0.2},
}

results = client.search(
    collection_name='demo', 
    data=['whats the focus of information retrieval?'],
    output_fields=['text'],
    anns_field='sparse',
    limit=3,
    search_params=search_params
)
```

The search parameter `drop_ratio_search` refers to the proportion of lower-scoring documents to drop during the search algorithm.

Let's view the results:

```
for hit in results[0]:
    print(hit)
```

```
{'id': 454387371651630485, 'distance': 1.3352930545806885, 'entity': {'text': 'information retrieval is a field of study.'}}
{'id': 454387371651630486, 'distance': 0.29726022481918335, 'entity': {'text': 'information retrieval focuses on finding relevant information in large datasets.'}}
{'id': 454387371651630487, 'distance': 0.2715056240558624, 'entity': {'text': 'data mining and information retrieval overlap in research.'}}
```

## Performing Hybrid Semantic and Full-Text Search

Let's now combine what we've learned to perform a hybrid search that combines separate semantic and full-text searches with a reranker:

```
from pymilvus import AnnSearchRequest, RRFRanker
query = 'whats the focus of information retrieval?'
query_dense_vector = embedding_fn([query])

search_param_1 = {
    "data": query_dense_vector,
    "anns_field": "dense",
    "param": {
        "metric_type": "COSINE",
    },
    "limit": 3
}
request_1 = AnnSearchRequest(**search_param_1)

search_param_2 = {
    "data": [query],
    "anns_field": "sparse",
    "param": {
        "metric_type": "BM25",
        "params": {"drop_ratio_build": 0.0}
    },
    "limit": 3
}
request_2 = AnnSearchRequest(**search_param_2)

reqs = [request_1, request_2]
```

```
ranker = RRFRanker()

res = client.hybrid_search(
    collection_name="demo",
    output_fields=['text'],
    reqs=reqs,
    ranker=ranker,
    limit=3
)
for hit in res[0]:
    print(hit)
```

```
{'id': 454387371651630485, 'distance': 0.032786883413791656, 'entity': {'text': 'information retrieval is a field of study.'}}
{'id': 454387371651630486, 'distance': 0.032258063554763794, 'entity': {'text': 'information retrieval focuses on finding relevant information in large datasets.'}}
{'id': 454387371651630487, 'distance': 0.0317460335791111, 'entity': {'text': 'data mining and information retrieval overlap in research.'}}
```

As you may have noticed, this is no different than a hybrid search with two separate semantic fields (available since Milvus 2.4). The results are identical to full-text search in this simple example, but for larger databases and keyword specific searches hybrid search typically has higher recall.

## Summary

You're now equipped with all the knowledge needed to perform full-text and hybrid semantic/full-text search with Milvus 2.5. See the following articles for more discussion on how full-text search works and why it is complementary to semantic search:

- [Introducing Milvus 2.5: Full-Text Search, More Powerful Metadata Filtering, and Usability Improvements!](https://milvus.io/blog/introduce-milvus-2-5-full-text-search-powerful-metadata-filtering-and-more.md)
- [Semantic Search v.s. Full-Text Search: Which Do I Choose in Milvus 2.5?](https://milvus.io/blog/semantic-search-vs-full-text-search-which-one-should-i-choose-with-milvus-2-5.md)


Getting Started with Hybrid Semantic / Full-Text Search with Milvus 2.5



## Overview

We are thrilled to present the latest version of Milvus, 2.5, which introduces a powerful new capability: [full-text search](https://milvus.io/docs/full-text-search.md#Full-Text-Search), also known as lexical or keyword search. If you are new to search, full-text search allows you to find documents by searching for specific words or phrases within them, similar to how you search in Google. This complements our existing semantic search capabilities, which understand the meaning behind your search rather than just matching exact words.

We use the industry-standard BM25 metric for document similarity, and our implemention is based on sparse vectors, allowing for more efficient storage and retrieval. For those unfamiliar with the term, sparse vectors are a way to represent text where most values are zero, making them very efficient to store and process—imagine a huge spreadsheet where only a few cells contain numbers, and the rest are empty. This approach fits well into Milvus's product philosophy where the vector is the core search entity.

An additional noteworthy aspect of our implementation is the capability to insert and query text _directly_ rather than having users first manually convert text into sparse vectors. This takes Milvus one step closer towards fully processing unstructured data.

But this is just the beginning. With the release of 2.5, we updated the [Milvus product roadmap](https://milvus.io/docs/roadmap.md). In future product iterations of Milvus, our focus will be on evolving Milvus's capabilities in four key directions:

- Streamlined unstructured data processing;
- Better search quality and efficiency;
- Easier data management;
- Lowering costs through algorithmic and design advances

Our aim is to build data infrastructure that can both efficiently store and effectively retrieve information in the AI era.


## Full-text Search via Sparse-BM25

Although semantic search typically has better contexual awareness and intent understanding, when a user needs to search for specific proper nouns, serial numbers, or a completely-matching phrase, full-text retrieval with keyword matching often produces more accurate results.

To illustrate this with an example:

- Semantic search excels when you ask: "Find documents about renewable energy solutions"
- Full-text search is better when you need: "Find documents mentioning _Tesla Model 3 2024_"

In our previous version (Milvus 2.4), users had to pre-process their text using a separate tool (the BM25EmbeddingFunction module to the PyMilvus) on their own machines before they could search it This approach had several limitations: it couldn't handle growing datasets well, required extra setup steps, and made the whole process more complicated than necessary. For the technically minded, the key limitations were that it could only work on a single machine; the vocabulary and other corpus statistics used for BM25 scoring couldn't be updated as the corpus changed; and converting text to vectors on the client side is less intuitive working withtext directly.

Milvus 2.5 simplifies everything. Now you can work with your text directly:

- Store your original text documents as they are
- Search using natural language queries
- Get results back in readable form

Behind the scenes, Milvus handles all the complex vector conversions automatically making it easier to work with text data. This is what we call our "Doc in, Doc out" approach—you work with readable text, and we handle the rest.


### Techical Implementation

For those interested in the technical details, Milvus 2.5 adds the full-text search capability through its built-in Sparse-BM25 implementation, including:

- **A Tokenizer built on tantivy**: Milvus now integrates with the thriving tantivy ecosystem
- **Capability to ingest and retrieve raw documents**: Support for direct ingestion and query of text data
- **BM25 relevance scoring**: Internalize BM25 scoring, implemented based on sparse vector

We chose to work with the well-developed tantivy ecosystem and build the Milvus text tokenizer on tantivy. In the future, Milvus will support more tokenizers and expose the tokenization process to help users better understand the retrieval quality. We will also explore deep learning-based tokenizers and stemmer strategies to further optimize the performance of full-text search. Below is sample code for using and configuring the tokenizer:

```Python
# Tokenizer configuration
schema.add_field(
    field_name='text',
    datatype=DataType.VARCHAR,
    max_length=65535,
    enable_analyzer=True, # Enable tokenizer on this column
    analyzer_params={"type": "english"},  # Configure tokenizer parameters, here we choose the english template, fine-grained configuration is also supported
    enable_match=True, # Build an inverted index for Text_Match
)
```

After configuring the tokenizer in the collection schema, users can register the text to bm25 function via add_function method. This will run internally in the Milvus server. All subsequent data flows such as additions, deletions, modifications, and queries can be completed by operating on the raw text string, as opposed to the vector representation. See below code example for how to ingest text and conduct full-text search with the new API:

```Python
# Define the mapping relationship between raw text data and vectors on the schema
bm25_function = Function(
    name="text_bm25_emb",
    input_field_names=["text"], # Input text field
    output_field_names=["sparse"], # Internal mapping sparse vector field
    function_type=FunctionType.BM25, # Model for processing mapping relationship
)

schema.add_function(bm25_function)
...
# Support for raw text in/out
MilvusClient.insert('demo', [
    {'text': 'Artificial intelligence was founded as an academic discipline in 1956.'},
    {'text': 'Alan Turing was the first person to conduct substantial research in AI.'},
    {'text': 'Born in Maida Vale, London, Turing was raised in southern England.'},
])

MilvusClient.search(
    collection_name='demo',
    data=['Who started AI research?'],
    anns_field='sparse',
    limit=3
)
```

We have adopted an implementation of BM25 relevance scoring that represents queries and documents as sparse vectors, called **Sparse-BM25**. This unlocks many optimizations based on sparse vector, such as:

Milvus achieves hybrid search capabilities through its cutting-edge **Sparse-BM25 implementation**, which integrates full-text search into the vector database architecture. By representing term frequencies as sparse vectors instead of traditional inverted indexes, Sparse-BM25 enables advanced optimizations, such as **graph indexing**, **product quantization (PQ)**, and **scalar quantization (SQ)**. These optimizations minimize memory usage and accelerate search performance. Similar to inverted index approach, Milvus supports taking raw text as input and generating sparse vectors internally. This make it able to work with any tokenizer and grasp any word shown in the dynamicly changing corpus.

Additionally, heuristic-based pruning discards low-value sparse vectors, further enhancing efficiency without compromising accuracy. Unlike previous approach using sparse vector, it can adapt to a growing corpus, not the accuracy of BM25 scoring.

1. Building graph indexes on the sparse vector, which performs better than inverted index on queries with long text as inverted index needs more steps to finish matching the tokens in the query;
2. Leveraging approximation techniques to speed up search with only minor impact on retrieval quality, such as vector quantization and heuristic based pruning;
3. Unifying the interface and data model for performing semantic search and full-text search, thus enhancing the user experience.

```Python
# Creating an index on the sparse column
index_params.add_index(
    field_name="sparse",
    index_type="AUTOINDEX",  # Default WAND index
    metric_type="BM25" # Configure relevance scoring through metric_type
)

# Configurable parameters at search time to speed up search
search_params = {
    'params': {'drop_ratio_search': 0.6}, # WAND search parameter configuration can speed up search
}
```

In summary, Milvus 2.5 has expanded its search capability beyond semantic search by introducing full-text retrieval, making it easier for users to build high quality AI applications. These are just initial steps in the space of Sparse-BM25 search and we anticipate that there will be further optimization measures to try in the future.


## Text Matching Search Filters

A second text search feature released with Milvus 2.5 is **Text Match**, which allows the user to filter the search to entries containing a specific text string. This feature is also built on the basis of tokenization and is activated with `enable_match=True`.

It is worth noting that with Text Match, the processing of the query text is based on the logic of OR after tokenization. For example, in the example below, the result will return all documents (using the 'text' field) that contain either 'vector' or 'database'.

```Python
filter = "TEXT_MATCH(text, 'vector database')"
```

If your scenario requires matching both 'vector' and 'database', then you need to write two separate Text Matches and overlay them with AND to achieve your goal.

```Python
filter = "TEXT_MATCH(text, 'vector') and TEXT_MATCH(text, 'database')"
```


## Significant Enhancement in Scalar Filtering Performance

Our emphasis on scalar filtering performance originates from our discovery that the combination of vector retrieval and metadata filtering can greatly improve query performance and accuracy in various scenarios. These scenarios range from image search applications such as corner case identification in autonomous driving to complex RAG scenarios in enterprise knowledge bases. Thus, it is highly suitable for enterprise users to implement in large-scale data application scenarios.

In practice, many factors like how much data you're filtering, how your data is organized, and how you're searching can affect performance. To address this, Milvus 2.5 introduces three new types of indexes—BitMap Index, Array Inverted Index, and the Inverted Index after tokenizing the Varchar text field. These new indexes can significantly improve performance in real-world use cases.

Specifically:

1. **BitMap Index** can be used to accelerate tag filtering (common operators include in, array_contains, etc.), and is suitable for scenarios with fewer field category data (data cardinality). The principle is to determine whether a row of data has a certain value on a column, with 1 for yes and 0 for no, and then maintain a BitMap list. The following chart shows the performance test comparison we conducted based on a customer's business scenario. In this scenario, the data volume is 500 million, the data category is 20, different values have different distribution proportions (1%, 5%, 10%, 50%), and the performance under different filtering amounts also varies. With 50% filtering, we can achieve a 6.8-fold performance gain through BitMap Index. It's worth noting that as cardinality increases, compared to BitMap Index, Inverted Index will show more balanced performance. 

![](https://assets.zilliz.com/QPS_comparison_f3f580d697.png)


2. **Text Match** is based on the Inverted Index after the text field is tokenized. Its performance far exceeds the Wildcard Match (i.e., like + %) function we provided in 2.4. According to our internal test results, the advantages of Text Match are very clear, especially in concurrent query scenarios, where it can achieve up to a 400-fold QPS increase. 

![](https://assets.zilliz.com/data_size_and_concurrency_e19dc44c59.png)

In terms of JSON data processing, we plan to introduce in subsequent versions of 2.5.x the building of inverted indices for user-specified keys and default location information recording for all keys to speed up parsing. We expect both of these areas to significantly enhance the query performance of JSON and Dynamic Field. We plan to showcase more information in future release notes and technical blogs, so stay tuned!


## New Management Interface

Managing a database shouldn't require a computer science degree, but we know database administrators need powerful tools. That's why we've introduced the **Cluster Management WebUI**, a new web-based interface accessible at your cluster's address on port 9091/webui. This observability tool provides:

- Real-time monitoring dashboards showing cluster-wide metrics
- Detailed memory and performance analytics per node
- Segment information and slow query tracking
- System health indicators and node status
- Easy-to-use troubleshooting tools for complex system issues

While this interface is still in beta, we're actively developing it based on user feedback from database administrators. Future updates will include AI-assisted diagnostics, more interactive management features, and enhanced cluster observability capabilities.


## Documentation and Developer Experience

We've completely revamped our **documentation** and **SDK/API** experience to make Milvus more accessible while maintaining depth for experienced users. The improvements include:

- A restructured documentation system with clearer progression from basic to advanced concepts
- Interactive tutorials and real-world examples that showcase practical implementations
- Comprehensive API references with practical code samples
- A more user-friendly SDK design that simplifies common operations
- Illustrated guides that make complex concepts easier to understand
- An AI-powered documentation assistant (ASK AI) for quick answers

The updated SDK/API focuses on improving developer experience through more intuitive interfaces and better integration with the documentation. We believe you'll notice these improvements when working with the 2.5.x series.

However, we know documentation and SDK development is an ongoing process. We'll continue optimizing both the content structure and SDK design based on community feedback. Join our Discord channel to share your suggestions and help us improve further.


## **Summary**

Milvus 2.5 contains 13 new features and several system-level optimizations, contributed not just by Zilliz but the open-source community. We have only touched on a few of them in this post and encourage you to visit our [release note](https://milvus.io/docs/release_notes.md) and [official documents](https://milvus.io/docs) for more information!


Introducing Milvus 2.5: Full-Text Search, More Powerful Metadata Filtering, and Usability Improvements!

Semantic Search vs. Full-Text Search: Which Do I Choose in Milvus 2.5?



## Introduction 

Over the past couple of years, [Retrieval-Augmented Generation (RAG)](https://zilliz.com/learn/Retrieval-Augmented-Generation) has emerged as a trusted solution for large organizations to enhance their [LLM](https://zilliz.com/glossary/large-language-models-(llms))-powered applications, especially those with diverse users. As such applications grow, implementing a multi-tenancy framework becomes essential. **Multi-tenancy** provides secure, isolated access to data for different user groups, ensuring user trust, meeting regulatory standards, and improving operational efficiency.

[Milvus](https://zilliz.com/what-is-milvus) is an open-source [vector database](https://zilliz.com/learn/what-is-vector-database) built to handle high-dimensional [vector data](https://zilliz.com/glossary/vector-embeddings). It is an indispensable infrastructure component of RAG, storing and retrieving contextual information for LLMs from external sources. Milvus offers [flexible multi-tenancy strategies](https://milvus.io/docs/multi_tenancy.md) for various needs, including **database-level, collection-level, and partition-level multi-tenancy**. 

In this post, we’ll cover: 

- What is Multi-Tenancy and Why It Matters

- Multi-Tenancy Strategies in Milvus

- Example: Multi-Tenancy Strategy for a RAG-Powered Enterprise Knowledge Base


## What is Multi-Tenancy and Why It Matters

[**Multi-tenancy**](https://milvus.io/docs/multi_tenancy.md) is an architecture where multiple customers or teams, known as "**tenants,**" share a single instance of an application or system. Each tenant’s data and configurations are logically isolated, ensuring privacy and security, while all tenants share the same underlying infrastructure. 

Imagine a SaaS platform that provides knowledge-based solutions to multiple companies. Each company is a tenant. 

- Tenant A is a healthcare organization storing patient-facing FAQs and compliance documents.

- Tenant B is a tech company managing internal IT troubleshooting workflows.

- Tenant C is a retail business with customer service FAQs for product returns.

Each tenant operates in a completely isolated environment, ensuring that no data from Tenant A leaks into Tenant B’s system or vice versa. Furthermore, resource allocation, query performance, and scaling decisions are tenant-specific, ensuring high performance regardless of workload spikes in one tenant.

Multi-tenancy also works for systems serving different teams within the same organization. Imagine a large company using a RAG-powered knowledge base to serve its internal departments, such as HR, Legal, and Marketing. Each **department is a tenant** with isolated data and resources in this setup. 

Multi-tenancy offers significant benefits, including **cost efficiency, scalability, and robust data security**. By sharing a single infrastructure, service providers can reduce overhead costs and ensure more effective resource consumption. This approach also scales effortlessly—onboarding new tenants requires far fewer resources than creating separate instances for each one, as with single-tenancy models. Importantly, multi-tenancy maintains robust data security by ensuring strict data isolation for each tenant, with access controls and encryption protecting sensitive information from unauthorized access. Additionally, updates, patches, and new features can be deployed across all tenants simultaneously, simplifying system maintenance and reducing the burden on administrators while ensuring that security and compliance standards are consistently upheld. 


## Multi-Tenancy Strategies in Milvus

To understand how Milvus supports multi-tenancy, it's important to first look at how it organizes user data. 


### How Milvus Organizes User Data

Milvus structures data across three layers, moving from broad to granular: [**Database**](https://milvus.io/docs/manage_databases.md), [**Collection**](https://milvus.io/docs/manage-collections.md), and [**Partition/Partition Key**](https://milvus.io/docs/manage-partitions.md). 

![Figure- How Milvus organizes user data .png](https://assets.zilliz.com/Figure_How_Milvus_organizes_user_data_4521c4b8f9.png)

_Figure: How Milvus organizes user data_ 

- **Database**: This acts as a logical container, similar to a database in traditional relational systems.

- **Collection**: Comparable to a table within a database, a collection organizes data into manageable groups.

- **Partition/Partition Key**: Within a collection, data can be further segmented by **Partitions**. Using a **Partition Key**, data with the same key is grouped together. For example, if you use a **user ID** as the **Partition Key**, all data for a specific user will be stored in the same logical segment. This makes it straightforward to retrieve data tied to individual users.

As you move from **Database** to **Collection** to **Partition Key**, the granularity of data organization becomes progressively finer. 

To ensure stronger data security and proper access control, Milvus also provides robust [**Role-Based Access Control (RBAC)**](https://zilliz.com/blog/enabling-fine-grained-access-control-with-milvus-row-level-rbac), allowing administrators to define specific permissions for each user. Only authorized users can access certain data. 

Milvus supports [multiple strategies](https://milvus.io/docs/multi_tenancy.md) for implementing multi-tenancy, offering flexibility based on the needs of your application: **database-level, collection-level, and partition-level multi-tenancy**. 


### Database-Level Multi-Tenancy

With the database-level multi-tenancy approach, each tenant is assigned their own database within the same Milvus cluster. This strategy provides strong data isolation and ensures optimal search performance. However, it can lead to inefficient resource utilization if certain tenants remain inactive.


### Collection-Level Multi-Tenancy

Here, in collection-level multi-tenancy, we can organize data for tenants in two ways. 

- **One Collection for All Tenants**: All tenants share a single collection, with tenant-specific fields used for filtering. While simple to implement, this approach can encounter performance bottlenecks as the number of tenants increases. 

- **One Collection per Tenant**: Each tenant can have a dedicated collection, improving isolation and performance but requiring more resources. This setup may face scalability limitations if the number of tenants exceeds Milvus's collection capacity.


### Partition-Level Multi-Tenancy

Partition-Level Multi-Tenancy focuses on organizing tenants within a single collection. Here, we also have two ways to organize tenant data. 

- **One Partition per Tenant**: Tenants share a collection, but their data is stored in separate partitions. We can isolate data by assigning each tenant a dedicated partition, balancing isolation and search performance. However, this approach is constrained by Milvus's maximum partition limit. 

- **Partition-Key-Based Multi-Tenancy**: This is a more scalable option in which a single collection uses partition keys to distinguish tenants. This method simplifies resource management and supports higher scalability but does not support bulk data inserts.

The table below summarizes the key differences between key multi-tenancy approaches. 

| **Granularity**               | **Database-level**                                               | **Collection-level**                                              | **Partition Key-level**                                           |
| ----------------------------- | ---------------------------------------------------------------- | ----------------------------------------------------------------- | ----------------------------------------------------------------- |
| Max Tenants Supported         | ~1,000                                                          | ~10,000                                                          | ~10,000,000                                                      |
| Data Organization Flexibility | High: Users can define multiple collections with custom schemas. | Medium: Users are limited to one collection with a custom schema. | Low: All users share a collection, requiring a consistent schema. |
| Cost per User                 | High                                                             | Medium                                                            | Low                                                               |
| Physical Resource Isolation   | Yes                                                              | Yes                                                               | No                                                                |
| RBAC                          | Yes                                                              | Yes                                                               | No                                                                |
| Search Performance            | Strong                                                           | Medium                                                            | Strong                                                            |


## Example: Multi-Tenancy Strategy for a RAG-Powered Enterprise Knowledge Base

When designing the multi-tenancy strategy for a RAG system, it's essential to align your approach with the specific needs of your business and your tenants. Milvus offers various multi-tenancy strategies, and choosing the right one depends on the number of tenants, their requirements, and the level of data isolation needed. Here's a practical guide for making these decisions, taking a  RAG-powered enterprise knowledge base as an example. 


### Understanding Tenant Structure Before Choosing a Multi-Tenancy Strategy 

A RAG-powered enterprise knowledge base often serves a small number of tenants. These tenants are usually independent business units like IT, Sales, Legal, and Marketing, each requiring distinct knowledge base services. For example, the HR Department manages sensitive employee information like onboarding guides and benefits policies, which should be confidential and accessible only to HR personnel.

In this case, each business unit should be treated as a separate tenant and a **Database-level multi-tenancy strategy** is often the most suitable. By assigning dedicated databases to each tenant, organizations can achieve strong logical isolation, simplifying management and enhancing security. This setup provides tenants with significant flexibility—they can define custom data models within collections, create as many collections as needed, and independently manage access control for their collections.


### Enhancing Security with Physical Resource Isolation

In situations where data security is highly prioritized, logical isolation at the database level may not be enough. For example, some business units might handle critical or highly sensitive data, requiring stronger guarantees against interference from other tenants. In such cases, we can implement a [physical isolation approach](https://milvus.io/docs/resource_group.md) on top of a database-level multi-tenancy structure. 

Milvus enables us to map logical components, such as databases and collections, to physical resources. This method ensures that the activities of other tenants do not impact critical operations. Let's explore how this approach works in practice. 


![Figure- How Milvus manages physical resources.png](https://assets.zilliz.com/Figure_How_Milvus_manages_physical_resources_6269b908d7.png)


Figure: How Milvus manages physical resources

As shown in the diagram above, there are three layers of resource management in Milvus: **Query Node**, **Resource Group**, and **Database**. 

- **Query Node**: The component that processes query tasks. It runs on a physical machine or container (e.g., a pod in Kubernetes).

- **Resource Group**: A collection of Query Nodes that acts as a bridge between logical components (databases and collections) and physical resources. You can allocate one or more databases or collections to a single Resource Group.

In the example shown in the diagram above, there are three logical **Databases**: X, Y, and Z. 

- **Database X**: Contains **Collection A**.

- **Database Y**: Contains **Collections B** and **C**.

- **Database Z**: Contains **Collections D** and **E**.

Let’s say **Database X** holds a critical knowledge base that we don’t want to be affected by the load from **Database Y** or **Database Z**. To ensure data isolation:

- **Database X** is assigned its own **Resource Group** to guarantee that its critical knowledge base is unaffected by workloads from other databases.

- **Collection E** is also allocated to a separate **Resource Group** within its parent database (**Z**). This provides isolation at the collection level for specific critical data within a shared database.

Meanwhile, the remaining collections in **Databases Y** and **Z** share the physical resources of **Resource Group 2**. 

By carefully mapping logical components to physical resources, organizations can achieve a flexible, scalable, and secure multi-tenancy architecture tailored to their specific business needs.


### Designing End User-Level Access 

Now that we've learned the best practices for choosing a multi-tenancy strategy for an enterprise RAG, let's explore how to design user-level access in such systems. 

In these systems, end users usually interact with the knowledge base in a read-only mode through LLMs. However, organizations still need to track such Q&A data generated by users and link it to specific users for various purposes, such as improving the knowledge base’s accuracy or offering personalized services.

Take a hospital's smart consultation service desk as an example. Patients might ask questions like, “Are there any available appointments with the specialist today?” or "Is there any specific preparation needed for my upcoming surgery?" While these questions don’t directly impact the knowledge base, it’s important for the hospital to track such interactions to improve services. These Q&A pairs are usually stored in a separate database (it doesn't necessarily have to be a vector database) dedicated to logging interactions.

![Figure- The multi-tenancy architecture for an enterprise RAG knowledge base .png](https://assets.zilliz.com/Figure_The_multi_tenancy_architecture_for_an_enterprise_RAG_knowledge_base_7c9ad8d4d1.png)

_Figure: The multi-tenancy architecture for an enterprise RAG knowledge base_ 

The diagram above shows the multi-tenancy architecture of an enterprise RAG system. 

- **System Administrators** oversee the RAG system, manage resource allocation, assign databases, map them to resource groups, and ensure scalability. They handle the physical infrastructure, as shown in the diagram, where each resource group (e.g., Resource Group 1, 2, and 3) is mapped to physical servers (query nodes).

- **Tenants (Database owners and developers)** manage the knowledge base, iterating on it based on the user-generated Q&A data, as shown in the diagram. Different databases (Database X, Y, Z) contain collections with different knowledge base content (Collection A, B, etc.).

- **End Users** interact with the system in a read-only manner through the LLM. As they query the system, their questions are logged in the separate Q&A record table (a separate database), continuously feeding valuable data back into the system.

This design ensures that each process layer—from user interaction to system administration—works seamlessly, helping the organization build a robust and continuously improving knowledge base.


## Summary 

In this blog, we’ve explored how [**multi-tenancy**](https://milvus.io/docs/multi_tenancy.md) frameworks play a critical role in the scalability, security, and performance of RAG-powered knowledge bases. By isolating data and resources for different tenants, businesses can ensure privacy, regulatory compliance, and optimized resource allocation across a shared infrastructure. [Milvus](https://milvus.io/docs/overview.md), with its flexible multi-tenancy strategies, allows businesses to choose the right level of data isolation—from database level to partition level—depending on their specific needs. Choosing the right multi-tenancy approach ensures companies can provide tailored services to tenants, even when dealing with diverse data and workloads. 

By following the best practices outlined here, organizations can effectively design and manage multi-tenancy RAG systems that not only deliver superior user experiences but also scale effortlessly as business needs grow. Milvus’ architecture ensures that enterprises can maintain high levels of isolation, security, and performance, making it a crucial component in building enterprise-grade, RAG-powered knowledge bases.


## Stay Tuned for More Insights into Multi-Tenancy RAG

In this blog, we’ve discussed how Milvus’ multi-tenancy strategies are designed to manage tenants, but not end users within those tenants. End-user interactions usually happen at the application layer, while the vector database itself remains unaware of those users.

You might be wondering: _If I want to provide more precise answers based on each end user’s query history, doesn’t Milvus need to maintain a personalized Q&A context for each user?_

That’s a great question, and the answer really depends on the use case. For example, in an on-demand consultation service, queries are random, and the main focus is on the quality of the knowledge base rather than on keeping track of a user's historical context.

However, in other cases, RAG systems must be context-aware. When this is required, Milvus needs to collaborate with the application layer to maintain a personalized memory of each user's context. This design is especially important for applications with massive end users, which we’ll explore in greater detail in my next post. Stay tuned for more insights!


Designing Multi-Tenancy RAG with Milvus: Best Practices for Scalable Enterprise Knowledge Bases




[**Milvus**](https://github.com/milvus-io/milvus) is an open-source [vector database](https://zilliz.com/learn/what-is-vector-database) designed to manage high-dimensional vector data. Whether you're building intelligent search engines, recommendation systems, or next-gen AI solutions such as retrieval augmented generation ([RAG](https://zilliz.com/learn/Retrieval-Augmented-Generation)), Milvus is a powerful tool at your fingertips. 

But what truly drives Milvus forward isn’t just its advanced technology—it’s the vibrant, passionate [developer community](https://zilliz.com/community) behind it. As an open-source project, Milvus thrives and evolves thanks to the contributions of developers like you. Every bug fix, feature addition, and performance enhancement from the community makes Milvus faster, more scalable, and more reliable.

Whether you're passionate about open-source, eager to learn, or want to make a lasting impact in AI, Milvus is the perfect place to contribute. This guide will walk you through the process—from setting up your development environment to submitting your first pull request. We’ll also highlight common challenges you might face and provide solutions to overcome them.

Ready to dive in? Let’s make Milvus even better together!


## Setting Up Your Milvus Development Environment

First thing first: setting up your development environment. You can either install Milvus on your local machine or use Docker—both methods are straightforward, but you’ll also need to install a few third-party dependencies to get everything running.


### Building Milvus Locally

If you like building things from scratch, building Milvus on your local machine is a breeze. Milvus makes it easy by bundling all the dependencies in the `install_deps.sh` script. Here’s the quick setup:


```
# Install third-party dependencies.
$ cd milvus/
$ ./scripts/install_deps.sh

# Compile Milvus.
$ make
```

### Building Milvus with Docker

If you prefer Docker, there are two ways to go about it: you can either run commands in a pre-built container or spin up a dev container for a more hands-on approach.



```
# Option 1: Run commands in a pre-built Docker container  
build/builder.sh make  

# Option 2: Spin up a dev container  
./scripts/devcontainer.sh up  
docker-compose -f docker-compose-devcontainer.yml ps  
docker exec -ti milvus-builder-1 bash  
make milvus  
```



**Platform Notes:** If you’re on Linux, you’re good to go—compilation issues are pretty rare. However, Mac users, especially with M1 chips, might run into some bumps along the way. Don’t sweat it, though—we have a guide to help you work through the most common issues. 

![](https://assets.zilliz.com/Figure_OS_configuration_52092fb1b7.png)

_Figure: OS configuration_ 

For the full setup guide, check out the official [Milvus Development Guide](https://github.com/milvus-io/milvus/blob/master/DEVELOPMENT.md).


### Common Issues and How to Fix Them

Sometimes, setting up your Milvus development environment doesn’t go as smoothly as planned. Don’t worry—here’s a quick rundown of common issues you might hit and how to fix them fast.


#### Homebrew: Unexpected Disconnect While Reading Sideband Packet

If you’re using Homebrew and see an error like this:


```
==> Tapping homebrew/core
remote: Enumerating objects: 1107077, done.
remote: Counting objects: 100% (228/228), done.
remote: Compressing objects: 100% (157/157), done.
error: 545 bytes of body are still expected.44 MiB | 341.00 KiB/s
fetch-pack: unexpected disconnect while reading sideband packet
fatal: early EOF
fatal: index-pack failed
Failed during: git fetch --force origin refs/heads/master:refs/remotes/origin/master
myuser~ %
```



**Fix:** Increase the `http.postBuffer` size:


```
git config --global http.postBuffer 1M
```


If you also run into `Brew: command not found` after installing Homebrew, you might need to set up your Git user configuration:


```
git config --global user.email xxxgit config --global user.name xxx
```



#### Docker: Error Getting Credentials

When working with Docker, you might see this:


```
error getting credentials - err: exit status 1, out: ``  
```


![](https://assets.zilliz.com/Docker_Error_Getting_Credentials_797f3043fb.png)



**Fix:** Open` ~/.docker/config.json` and remove the `credsStore` field.


#### Python: No Module Named 'imp'

If Python throws this error, it’s because Python 3.12 removed the `imp` module, which some older dependencies still use.


 ![](https://assets.zilliz.com/Python_No_Module_Named_imp_65eb2c5c66.png)

**Fix:** Downgrade to Python 3.11:



```
brew install python@3.11  
```


#### Conan: Unrecognized Arguments or Command Not Found

**Issue:** If you see `Unrecognized arguments: --install-folder conan`, you’re likely using an incompatible Conan version.

![](https://assets.zilliz.com/Conan_Unrecognized_Arguments_or_Command_Not_Found_8f2029db72.png)

**Fix:** Downgrade to Conan 1.61:


```
pip install conan==1.61  
```


**Issue:** If you see `Conan command not found`, it means your Python environment isn’t properly set up.

**Fix:** Add Python’s bin directory to your `PATH`:


```
export PATH="/path/to/python/bin:$PATH"
```



#### LLVM: Use of Undeclared Identifier 'kSecFormatOpenSSL'

This error usually means your LLVM dependencies are outdated.

![](https://assets.zilliz.com/LLVM_Use_of_Undeclared_Identifier_k_Sec_Format_Open_SSL_f0ca6f0166.png)

**Fix:** Reinstall LLVM 15 and update your environment variables:


```
brew reinstall llvm@15
export LDFLAGS="-L/opt/homebrew/opt/llvm@15/lib"
export CPPFLAGS="-I/opt/homebrew/opt/llvm@15/include"
```


**Pro Tips**

- Always double-check your tool versions and dependencies.

- If something still doesn’t work, the[ Milvus GitHub Issues page](https://github.com/milvus-io/milvus/issues) is a great place to find answers or ask for help.


### Configuring VS Code for C++ and Go Integration

Getting C++ and Go to work together in VS Code is easier than it sounds. With the right setup, you can streamline your development process for Milvus. Just tweak your `user.settings` file with the configuration below:


```
{
   "go.toolsEnvVars": {
       "PKG_CONFIG_PATH": "/Users/zilliz/milvus/internal/core/output/lib/pkgconfig:/Users/zilliz/workspace/milvus/internal/core/output/lib64/pkgconfig",
       "LD_LIBRARY_PATH": "/Users/zilliz/workspace/milvus/internal/core/output/lib:/Users/zilliz/workspace/milvus/internal/core/output/lib64",
       "RPATH": "/Users/zilliz/workspace/milvus/internal/core/output/lib:/Users/zilliz/workspace/milvus/internal/core/output/lib64"
   },
   "go.testEnvVars": {
       "PKG_CONFIG_PATH": "/Users/zilliz/workspace/milvus/internal/core/output/lib/pkgconfig:/Users/zilliz/workspace/milvus/internal/core/output/lib64/pkgconfig",
       "LD_LIBRARY_PATH": "/Users/zilliz/workspace/milvus/internal/core/output/lib:/Users/zilliz/workspace/milvus/internal/core/output/lib64",
       "RPATH": "/Users/zilliz/workspace/milvus/internal/core/output/lib:/Users/zilliz/workspace/milvus/internal/core/output/lib64"
   },
   "go.buildFlags": [
       "-ldflags=-r /Users/zilliz/workspace/milvus/internal/core/output/lib"
   ],
   "terminal.integrated.env.linux": {
       "PKG_CONFIG_PATH": "/Users/zilliz/workspace/milvus/internal/core/output/lib/pkgconfig:/Users/zilliz/workspace/milvus/internal/core/output/lib64/pkgconfig",
       "LD_LIBRARY_PATH": "/Users/zilliz/workspace/milvus/internal/core/output/lib:/Users/zilliz/workspace/milvus/internal/core/output/lib64",
       "RPATH": "/Users/zilliz/workspace/milvus/internal/core/output/lib:/Users/zilliz/workspace/milvus/internal/core/output/lib64"
   },
   "go.useLanguageServer": true,
   "gopls": {
       "formatting.gofumpt": true
   },
   "go.formatTool": "gofumpt",
   "go.lintTool": "golangci-lint",
   "go.testTags": "dynamic",
   "go.testTimeout": "10m"
}
```



Here’s what this configuration does:

- **Environment Variables:** Sets up paths for `PKG_CONFIG_PATH`, `LD_LIBRARY_PATH`, and `RPATH`, which are critical for locating libraries during builds and tests.

- **Go Tools Integration:** Enables Go’s language server (`gopls`) and configures tools like `gofumpt` for formatting and `golangci-lint` for linting.

- **Testing Setup:** Adds `testTags` and increases the timeout for running tests to 10 minutes.

Once added, this setup ensures a seamless integration between C++ and Go workflows. It’s perfect for building and testing Milvus without constant environment tweaking.


**Pro Tip**

After setting this up, run a quick test build to confirm everything works. If something feels off, double-check the paths and VS Code’s Go extension version.


## Deploying Milvus

Milvus supports [three deployment modes](https://milvus.io/docs/install-overview.md)—**Lite, Standalone,** and **Distributed.**  

- [**Milvus Lite**](https://milvus.io/blog/introducing-milvus-lite.md) is a Python library and an ultra-lightweight version of Milvus. It’s perfect for rapid prototyping in Python or notebook environments and for small-scale local experiments. 

- **Milvus Standalone** is the single-node deployment option for Milvus, using a client-server model. It is the Milvus equivalent of MySQL, while Milvus Lite is like SQLite. 

- **Milvus Distributed** is the distributed mode of Milvus, which is ideal for enterprise users building large-scale vector database systems or vector data platforms. 

All these deployments rely on three core components:

- **Milvus:** The vector database engine that drives all operations.

- **Etcd:** The metadata engine that manages Milvus's internal metadata.

- **MinIO:** The storage engine that ensures data persistence.

When running in **Distributed** mode, Milvus also incorporates **Pulsar** for distributed message processing using a Pub/Sub mechanism, making it scalable for high-throughput environments.


### Milvus Standalone

The Standalone mode is tailored for single-instance setups, making it perfect for testing and small-scale applications. Here’s how to get started:


```
# Deploy Milvus Standalone  
sudo docker-compose -f deployments/docker/dev/docker-compose.yml up -d
# Start the standalone service  
bash ./scripts/start_standalone.sh
```



### Milvus Distributed (previously known as Milvus Cluster) 

For larger datasets and higher traffic, the Distributed mode offers horizontal scalability. It combines multiple Milvus instances into a single cohesive system. Deployment is made easy with the **Milvus Operator**, which runs on Kubernetes and manages the entire Milvus stack for you.

Want step-by-step guidance? Check out the [Milvus Installation Guide](https://milvus.io/docs/install_cluster-milvusoperator.md).


## Running End-to-End (E2E) Tests

Once your Milvus deployment is up and running, testing its functionality is a breeze with E2E tests. These tests cover every part of your setup to ensure everything works as expected. Here’s how to run them:


```
# Navigate to the test directory  
cd tests/python_client  

# Install dependencies  
pip install -r requirements.txt  

# Run E2E tests  
pytest --tags=L0 -n auto  
```



For in-depth instructions and troubleshooting tips, refer to the [Milvus Development Guide](https://github.com/milvus-io/milvus/blob/master/DEVELOPMENT.md#e2e-tests).

**Pro Tip**

If you're new to Milvus, start with Milvus Lite or Standalone mode to get a feel for its capabilities before scaling up to Distributed mode for production-level workloads.


## Submitting Your Code

Congrats! You’ve cleared all unit and E2E tests (or debugged and recompiled as needed). While the first build can take some time, future ones will be much faster—so no need to worry. With everything passing, you’re ready to submit your changes and contribute to Milvus!


### Link Your Pull Request (PR) to an Issue

Every PR to Milvus needs to be tied to a relevant issue. Here's how to handle this:

- **Check for Existing Issues:** Look through the[ Milvus issue tracker](https://github.com/milvus-io/milvus/issues) to see if there’s already an issue related to your changes.

- **Create a New Issue:** If no relevant issue exists, open a new one and explain the problem you’re solving or the feature you’re adding.


### Submitting Your Code

1. **Fork the Repository:** Start by forking the[ Milvus repo](https://github.com/milvus-io/milvus) to your GitHub account.

2. **Create a Branch:** Clone your fork locally and make a new branch for your changes.

3. **Commit with Signed-off-by Signature:** Ensure your commits include a `Signed-off-by` signature to comply with open-source licensing:


```
git commit -m "Commit of your change" -s
```


This step certifies your contribution is in line with the Developer Certificate of Origin (DCO).


#### **Helpful Resources**

For detailed steps and best practices, check out the[ Milvus Contribution Guide](https://github.com/milvus-io/milvus/blob/master/CONTRIBUTING.md).


## Opportunities to Contribute

Congrats—you’ve got Milvus up and running! You’ve explored its deployment modes, run your tests, and maybe even dug into the code. Now it’s time to level up: contribute to [Milvus](https://github.com/milvus-io/milvus) and help shape the future of AI and [unstructured data](https://zilliz.com/learn/introduction-to-unstructured-data).

No matter your skillset, there’s a place for you in the Milvus community! Whether you’re a developer who loves solving complex challenges, a tech writer who loves writing clean documentation or engineering blogs, or a Kubernetes enthusiast looking to improve deployments, there’s a way for you to make an impact.

Take a look at the opportunities below and find your perfect match. Every contribution helps move Milvus forward—and who knows? Your next pull request might just power the next wave of innovation. So, what are you waiting for? Let’s get started! 🚀

| Projects                                                                                                                                                                            | Suitable for                             | Guidelines                                                                                          |
| ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------- | --------------------------------------------------------------------------------------------------- |
| [milvus](https://github.com/milvus-io/milvus), [milvus-sdk-go](https://github.com/milvus-io/milvus-sdk-go)                                                                          | Go developers                            | /                                                                                                   |
| [milvus](https://github.com/milvus-io/milvus), [knowhere](https://github.com/milvus-io/knowhere)                                                                                    | CPP developers                           | /                                                                                                   |
| [pymilvus](https://github.com/milvus-io/pymilvus), [milvus-sdk-node](https://github.com/milvus-io/milvus-sdk-node), [milvus-sdk-java](https://github.com/milvus-io/milvus-sdk-java) | Developers interested in other languages | [Contributing to PyMilvus](https://github.com/milvus-io/pymilvus/blob/master/CONTRIBUTING.md)       |
| [milvus-helm](https://github.com/milvus-io/milvus-helm)                                                                                                                             | Kubernetes enthusiasts                   | /                                                                                                   |
| [Milvus-docs](https://github.com/milvus-io/milvus-docs), [milvus-io/community/blog](https://github.com/milvus-io/community)                                                         | Tech writers                             | [Contributing to milvus docs](https://github.com/milvus-io/milvus-docs/blob/v2.0.0/CONTRIBUTING.md) |
| [milvus-insight](https://github.com/zilliztech/milvus-insight)                                                                                                                      | Web developers                           | /                                                                                                   |


## A Final Word

Milvus offers various SDKs—[Python](https://milvus.io/docs/install-pymilvus.md) (PyMilvus), [Java](https://milvus.io/docs/install-java.md), [Go](https://milvus.io/docs/install-go.md), and [Node.js](https://milvus.io/docs/install-node.md)—that make it simple to start building. Contributing to Milvus isn’t just about code—it’s about joining a vibrant and innovative community. 

🚀Welcome to the Milvus developer community, and happy coding! We can’t wait to see what you’ll create. 


## Further Reading

- [Join the Milvus Community of AI Developers](https://zilliz.com/community)

- [What are Vector Databases and How Do They Work?](https://zilliz.com/learn/what-is-vector-database)

- [Milvus Lite vs. Standalone vs. Distributed: Which Mode is Right for You? ](https://zilliz.com/blog/choose-the-right-milvus-deployment-mode-ai-applications)

- [Build AI Apps with Milvus: Tutorials & Notebooks](https://zilliz.com/learn/milvus-notebooks)

- [Top Performing AI Models for Your GenAI Apps | Zilliz](https://zilliz.com/ai-models)

- [What is RAG?](https://zilliz.com/learn/Retrieval-Augmented-Generation)

- [Generative AI Resource Hub | Zilliz](https://zilliz.com/learn/generative-ai)

How to Contribute to Milvus: A Quick Start for Developers




[Semantic search](https://zilliz.com/glossary/semantic-search) allows machines to understand language and yield better search results, which is essential in AI and data analytics. Once the language is represented as [embeddings](https://zilliz.com/learn/everything-you-should-know-about-vector-embeddings), the search can be performed using exact or approximate methods. Approximate Nearest Neighbor ([ANN](https://zilliz.com/glossary/anns)) search is a method used to quickly find points in a dataset that are closest to a given query point, unlike [exact nearest neighbor search](https://zilliz.com/blog/k-nearest-neighbor-algorithm-for-machine-learning), which can be computationally expensive for high-dimensional data. ANN allows faster retrieval by providing results that are approximately close to the nearest neighbors. 

One of the algorithms for Approximate Nearest Neighbor (ANN) search is [HNSW](https://zilliz.com/learn/hierarchical-navigable-small-worlds-HNSW) (Hierarchical Navigable Small Worlds), implemented under [HNSWlib](https://zilliz.com/learn/learn-hnswlib-graph-based-library-for-fast-anns), which will be the focus of today's discussion. In this blog, we will:

- Understand the HNSW algorithm.

- Explore HNSWlib and its key features.

- Set up HNSWlib, covering index building and search implementation.

- Compare it with Milvus.


## Understanding HNSW

**Hierarchical Navigable Small Worlds (**[**HNSW**](https://zilliz.com/learn/hierarchical-navigable-small-worlds-HNSW)**)** is a graph-based data structure that allows efficient similarity searches, particularly in high-dimensional spaces, by building a multi-layered graph of "small world" networks. Introduced in [2016](https://arxiv.org/abs/1603.09320), HNSW addresses the scalability issues associated with traditional search methods like brute-force and tree-based searches. It is ideal for applications involving large datasets, such as recommendation systems, image recognition, and [retrieval-augmented generation (RAG)](https://zilliz.com/vector-database-use-cases/llm-retrieval-augmented-generation).


### Why HNSW Matters

HNSW significantly enhances the performance of nearest-neighbor search in high-dimensional spaces. Combining the hierarchical structure with small-world navigability avoids the computational inefficiency of older methods, enabling it to perform well even with massive, complex datasets. To understand this better, let’s look at how it works now.


### How HNSW Works

1. **Hierarchical Layers:** HNSW organizes data into a hierarchy of layers, where each layer contains nodes connected by edges. The top layers are sparser, allowing for broad "skips" across the graph, much like zooming out on a map to see only major highways between cities. Lower layers increase in density, providing finer detail and more connections between closer neighbors.

2. **Navigable Small Worlds Concept:** Each layer in HNSW builds on the concept of a "small world" network, where nodes (data points) are only a few "hops" away from each other. The search algorithm begins at the highest, sparsest layer and works downward, moving to progressively denser layers to refine the search. This approach is like moving from a global view down to neighborhood-level details, gradually narrowing the search area.

![](https://assets.zilliz.com/Fig_1_An_Example_of_a_Navigable_Small_World_Graph_afa737ee9f.png)

[Fig 1](https://daniel-at-world.blogspot.com/2019/04/navigable-small-world-graphs-for.html): An Example of a Navigable Small World Graph

3. **Skip List-Like Structure:** The hierarchical aspect of HNSW resembles a skip list, a probabilistic data structure where higher layers have fewer nodes, allowing for faster initial searches. 

![](https://assets.zilliz.com/Fig_2_An_Example_of_Skip_List_Structure_f41b07234d.png)

[Fig 2](https://www.cs.cmu.edu/~ckingsf/bioinfo-lectures/skiplists.pdf): An Example of Skip List Structure

To search for 96 in the given skip list, we begin at the top level on the far left at the header node. Moving to the right, we encounter 31, less than 96, so we continue to the next node. Now, we need to move down a level where we see 31 again; since it’s still less than 96, we descend another level. Finding 31 once more, we then move right and reach 96, our target value. Thus, we locate 96 without needing to descend to the lowest levels of the skip list.

4. **Search Efficiency:** The HNSW algorithm starts from an entry node at the highest layer, progressing to closer neighbors with each step. It descends through the layers, using each one for coarse-to-fine-grained exploration, until it reaches the lowest layer where the most similar nodes are likely found. This layered navigation reduces the number of nodes and edges that need to be explored, making the search fast and accurate.

5. **Insertion and Maintenance**: When adding a new node, the algorithm determines its entry layer based on probability and connects it to nearby nodes using a neighbor selection heuristic. The heuristic aims to optimize connectivity, creating links that improve navigability while balancing graph density. This approach keeps the structure robust and adaptable to new data points.

While we have a foundational understanding of the HNSW algorithm, implementing it from scratch can be overwhelming. Fortunately, the community has developed libraries like [HNSWlib](https://github.com/nmslib/hnswlib) to simplify usage, making it accessible without scratching your head. So, let’s take a closer look at HNSWlib.


## Overview of HNSWlib

HNSWlib, a popular library implementing HNSW, is highly efficient and scalable, performing well even with millions of points. It achieves sublinear time complexity by allowing quick jumps between graph layers and optimizing the search for dense, high-dimensional data. Here are the key features of HNSWlib include:

- **Graph-Based Structure:** A multi-layered graph represents data points, allowing fast, nearest-neighbor searches.

- **High-Dimensional Efficiency:** Optimized for high-dimensional data, providing quick and accurate approximate searches.

- **Sublinear Search Time:** Achieves sublinear complexity by skipping layers, improving speed significantly.

- **Dynamic Updates:** Supports real-time insertion and deletion of nodes without requiring a complete graph rebuild.

- **Memory Efficiency:** Efficient memory usage, suitable for large datasets.

- **Scalability:** Scales well to millions of data points, making it ideal for medium-scale applications like recommendation systems.

**Note:** HNSWlib is excellent for creating simple prototypes for vector search applications. However, due to scalability limitations, there may be better choices such as [purpose-built vector databases](https://zilliz.com/blog/what-is-a-real-vector-database) for more complex scenarios involving hundreds of millions or even billions of data points. Let’s see that in action.


## Getting Started with HNSWlib: A Step-by-Step Guide

This section will demonstrate using HNSWlib as a [vector search library](https://zilliz.com/learn/comparing-vector-database-vector-search-library-and-vector-search-plugin) by creating an HNSW index, inserting data, and performing searches. Let’s start with installation:


### Setup and Imports

To get started with HNSWlib in Python, first install it using pip:

```
pip install hnswlib
```

Then, import the necessary libraries:

```
import hnswlib 
import numpy as np
```

### Preparing Data

In this example, we’ll use `NumPy`to generate a random dataset with 10,000 elements, each with a dimension size 256.

```
dim = 256  # Dimensionality of your vectors
num_elements = 10000  # Number of elements to insert
```

Let’s create the data:

```
data = np.random.rand(num_elements, dim).astype(np.float32)  # Example data
```

Now our data is ready, let’s build an index.


### Building an Index

In building an index, we need to define the dimensionality of the vectors and the space type. Let’s create an index:

```
p = hnswlib.Index(space='l2', dim=dim)
```
- `space='l2'`: This parameter defines the distance metric used for similarity. Setting it to `'l2'` means using the Euclidean distance (L2 norm). If you instead set it to `'ip'`, it would use the inner product, which is helpful for tasks like cosine similarity.

* `dim=dim`: This parameter specifies the dimensionality of the data points you’ll be working with. It must match the dimension of the data you plan to add to the index.

Here’s how to initialize an index:

```
p.init_index(max_elements=num_elements, ef_construction=200, M=16)
```

- `max_elements=num_elements`: This sets the maximum number of elements that can be added to the index. `Num_elements` is the maximum capacity, so we set this to 10,000 as we are working with 10,000 data points.

* `ef_construction=200`: This parameter controls the accuracy vs. construction speed trade-off during index creation. A higher value improves recall (accuracy) but increases memory usage and build time. Common values range from 100 to 200.

- `M=16`: This parameter determines the number of bi-directional links created for each data point, influencing accuracy and search speed. Typical values are between 12 and 48; 16 is often a good balance for moderate accuracy and speed.

```
p.set_ef(50)  # This parameter controls the speed/accuracy trade-off
```

- `ef`: The `ef` parameter, short for “exploration factor,” determines how many neighbors are examined during a search. A higher `ef` value results in more neighbors being explored, which generally increases the accuracy (recall) of the search but also makes it slower. Conversely, a lower `ef` value can search faster but might reduce accuracy.

In this case, Setting `ef` to 50 means the search algorithm will evaluate up to 50 neighbors when finding the most similar data points.

Note: `ef_construction` sets neighbor search effort during index creation, enhancing accuracy but slowing construction. `ef` controls search effort during querying, balancing speed and recall dynamically for each query.


### Performing Searches

To perform a nearest neighbor search using HNSWlib, we first create a random query vector. In this example, the vector's dimensionality matches the indexed data.

```
query_vector = np.random.rand(dim).astype(np.float32)  # Example query

labels, distances = p.knn_query(query_vector, k=5)  # k is the number of nearest neighbors
```

- `query_vector`: This line generates a random vector with the same dimensionality as the indexed data, ensuring compatibility for the nearest neighbor search.
- `knn_query`: The method searches for the `k` nearest neighbors of the `query_vector` within the index `p`. It returns two arrays: `labels`, which contain the indices of the nearest neighbors, and `distances`, which indicate the distances from the query vector to each of these neighbors. Here, `k=5` specifies that we want to find the five closest neighbors.

Here are the results after printing the labels and distances:

```
print("Nearest neighbors' labels:", labels)
print("Distances:", distances)
```

```
> Nearest neighbors' labels: [[4498 1751 5647 4483 2471]]
> Distances: [[33.718    35.484592 35.627766 35.828312 35.91495 ]]
```

Here we have it, a simple guide to get your wheels started with HNSWlib. 

As mentioned, HNSWlib is a great vector search engine for prototyping or experimenting with medium-sized datasets. If you have higher scalability requirements or need other enterprise-level features, you may need to choose a purpose-built vector database like the open-source [Milvus](https://zilliz.com/what-is-milvus) or its fully managed service on [Zilliz Cloud](https://zilliz.com/cloud). So, in the following section, we will compare HNSWlib with Milvus. 


## HNSWlib vs. Purpose-Built Vector Databases Like Milvus

A [vector database](https://zilliz.com/learn/what-is-vector-database) stores data as mathematical representations, enabling [machine learning models](https://zilliz.com/ai-models) to power search, recommendations, and text generation by identifying data through [similarity metrics](https://zilliz.com/blog/similarity-metrics-for-vector-search) for contextual understanding. 

Vector indices libraries like HNSWlib improve v[ector search](https://zilliz.com/learn/vector-similarity-search) and retrieval but lack the management features of a full database. On the other hand, vector databases, like [Milvus](https://milvus.io/), are designed to handle vector embeddings at scale, providing advantages in data management, indexing, and querying capabilities that standalone libraries typically lack. Here are some other benefits of using Milvus:

- **High-Speed Vector Similarity Search**: Milvus provides millisecond-level search performance across billion-scale vector datasets, ideal for applications like image retrieval, recommendation systems, natural language processing ([NLP](https://zilliz.com/learn/A-Beginner-Guide-to-Natural-Language-Processing)), and retrieval augmented generation ([RAG](https://zilliz.com/learn/Retrieval-Augmented-Generation)). 

- **Scalability and High Availability:** Built to handle massive data volumes, Milvus scales horizontally and includes replication and failover mechanisms for reliability.

- **Distributed Architecture:** Milvus uses a distributed, scalable architecture that separates storage and computing across multiple nodes for flexibility and robustness.

- [**Hybrid search**](https://zilliz.com/blog/a-review-of-hybrid-search-in-milvus)**:** Milvus supports multimodal search, hybrid [sparse and dense search](https://zilliz.com/learn/sparse-and-dense-embeddings), and hybrid dense and [full-text search](https://thenewstack.io/elasticsearch-was-great-but-vector-databases-are-the-future/), offering versatile and flexible search functionality.

- **Flexible Data Support**: Milvus supports various data types—vectors, scalars, and structured data—allowing seamless management and analysis within a single system.

- [**Active Community**](https://discord.com/invite/8uyFbECzPX) **and Support**: A thriving community provides regular updates, tutorials, and support, ensuring Milvus remains aligned with user needs and advances in the field. 

- [AI integration](https://milvus.io/docs/integrations_overview.md): Milvus has integrated with various popular AI frameworks and technologies, making it easier for developers to build applications with their familiar tech stacks. 

Milvus also provides a fully managed service on [Ziliz Cloud](https://zilliz.com/cloud), which is hassle-free and 10x faster than Milvus. 


### Comparison: Milvus vs. HNSWlib

| **Feature** |                                      **Milvus**                                     |                   **HNSWlib**                   |
| :---------: | :---------------------------------------------------------------------------------: | :---------------------------------------------: |
| Scalability |                        Handles billions of vectors with ease                        |    Fit for smaller datasets due to RAM usage    |
|  Ideal for  |            Prototyping, experimenting, and enterprise-level applications            | Focuses on prototypes and lightweight ANN tasks |
|   Indexing  | Supports 10+ indexing algorithms, including HNSW, DiskANN, Quantization, and Binary |           Uses a graph-based HNSW only          |
| Integration |                        Offers APIs and cloud-native services                        |   Serves as a lightweight, standalone library   |
| Performance |                    Optimizes for large data, distributed queries                    |    Offers high speed but limited scalability    |

Overall, Milvus is generally preferable for large-scale, production-grade applications with complex indexing needs, while HNSWlib is ideal for prototyping and more straightforward use cases.


## Conclusion

Semantic search can be resource-intensive, so internal data structuring, like that performed by HNSW, is essential for faster data retrieval. Libraries like HNSWlib care about the implementation, so the developers have the recipes ready to prototype vector capabilities. With just a few lines of code, we can build up our own index and perform searches.

HNSWlib is a great way to start. However, if you want to build complex and production-ready AI applications, purpose-built vector databases are the best option. For example, [Milvus](https://milvus.io/) is an open-source vector database with many enterprise-ready features such as high-speed vector search, scalability, availability, and flexibility in terms of data types and programming language.  


## Further Reading

- [What is Faiss (Facebook AI Similarity Search)? ](https://zilliz.com/learn/faiss)

- [What is HNSWlib? A Graph-based Library for Fast ANN Search ](https://zilliz.com/learn/learn-hnswlib-graph-based-library-for-fast-anns)

- [What is ScaNN (Scalable Nearest Neighbors)? ](https://zilliz.com/learn/what-is-scann-scalable-nearest-neighbors-google)

- [VectorDBBench: An Open-Source VectorDB Benchmark Tool](https://zilliz.com/vector-database-benchmark-tool?database=ZillizCloud%2CMilvus%2CElasticCloud%2CPgVector%2CPinecone%2CQdrantCloud%2CWeaviateCloud&dataset=medium&filter=none%2Clow%2Chigh&tab=1)

- [Generative AI Resource Hub | Zilliz](https://zilliz.com/learn/generative-ai)

- [What are Vector Databases and How Do They Work? ](https://zilliz.com/learn/what-is-vector-database)

- [What is RAG? ](https://zilliz.com/learn/Retrieval-Augmented-Generation)

- [Top Performing AI Models for Your GenAI Apps | Zilliz](https://zilliz.com/ai-models)



Getting Started with HNSWlib


Get expert help with your Milvus [vector database](https://zilliz.com/learn/what-is-vector-database) projects. Book a 20-minute one-on-one session to get insights, guidance, and answers to your questions. Whether you’re just getting started or need help with a specific issue, we’re here to help you make the most out of Milvus.

[![](https://assets.zilliz.com/milvus_office_hours_09848e076b.png)](https://meetings.hubspot.com/chloe-williams1/milvus-office-hours)


## Why Book a Session?

Here are a few great reasons to book time during our office hours:

- **Optimizing Milvus Performance:** Get tips on tuning configurations to boost database performance.

- **Data Schema Design Advice:** Learn best practices for data schema and index types to best fit your use case.

- **Scaling Your Milvus Deployment:** Get advice on managing and scaling Milvus for larger data volumes.

- **Troubleshooting Specific Issues:** Stuck on an error or unusual behavior? We can help troubleshoot and resolve it.

- **Integration Assistance:** Need guidance on using Milvus with other integrated frameworks like LangChain, LlamaIndex, Spark, or your data processing pipelines?

- **Learning about New Features:** Stay updated with the latest Milvus capabilities and how they could benefit your project.

Don’t miss this opportunity to level up your Milvus implementation with expert guidance.

If you’re unable to join us live, don’t worry—there are still plenty of ways to get support! The Milvus community is active and ready to help on both [Discord](https://discord.com/invite/8uyFbECzPX) and[ GitHub Discussions](https://github.com/search?q=milvus&type=discussions). Whether you prefer live chat or asynchronous discussions, these platforms are a great place to ask questions, share knowledge, and connect with other Milvus users and contributors.



Join Milvus Office Hours to Get Support from Vector DB Experts!



## What are Matryoshka Embeddings? 

When building efficient [vector search](https://zilliz.com/learn/vector-similarity-search) systems, one key challenge is managing storage costs while maintaining acceptable latency and recall. Modern [embedding models](https://zilliz.com/blog/choosing-the-right-embedding-model-for-your-data) output vectors with hundreds or thousands of dimensions, creating significant storage and computational overhead for the raw vector and index.

Traditionally, the storage requirements are reduced by applying a quantization or dimensionality reduction method just before building the index. For instance, we can save storage by lowering the precision using [Product Quantization](https://zilliz.com/learn/scalar-quantization-and-product-quantization) (PQ) or the number of dimensions using Principal Component Analysis (PCA). These methods analyze the entire vector set to find a more compact one that maintains the semantic relationships between vectors.

While effective, these standard approaches reduce precision or dimensionality only once and at a single scale. But what if we could maintain multiple layers of detail simultaneously, like a pyramid of increasingly precise representations?

Enter [**Matryoshka embeddings**](https://arxiv.org/abs/2205.13147). Named after Russian nesting dolls (see illustration), these clever constructs embed multiple scales of representation within a single vector. Unlike traditional post-processing methods, Matryoshka embeddings learn this multi-scale structure during the initial training process. The result is remarkable: **not only does the full embedding capture input semantics, but each nested subset prefix (first half, first quarter, etc.) provides a coherent, if less detailed, representation.**


![Figure: Visualization of Matryoshka embeddings with multiple layers of detail](https://assets.zilliz.com/Visualization_of_Matryoshka_embeddings_with_multiple_layers_of_detail_274f2c7aba.png)

_Figure: Visualization of Matryoshka embeddings with multiple layers of detail_

This approach contrasts sharply with conventional [embeddings](https://zilliz.com/glossary/vector-embeddings), where using arbitrary subsets of the vector dimensions typically destroys semantic meaning. With Matryoshka embeddings, you can choose the granularity that best balances your specific task's precision and computational cost. 

Need a quick approximate search? Use the smallest "doll." Need maximum accuracy? Use the full embedding. This flexibility makes them particularly valuable for systems adapting to different performance requirements or resource constraints.


## Inference

A valuable application of Matryoshka embeddings is accelerating similarity searches without sacrificing recall. By leveraging smaller subsets of query and database embeddings—such as the first 1/32 of their dimensions—we can build an index over this reduced space that still preserves much of the similarity information. Initial results from this smaller embedding space can be used directly. Still, there’s also a technique to boost recall and account for any minor reduction from the dimensional cutback, making this approach both efficient and effective for similarity search tasks.

![Figure: How the funnel search works with Matryoshka embeddings](https://assets.zilliz.com/How_the_funnel_search_works_with_Matryoshka_embeddings_8fa05a2fe7.png)

_Figure: How the funnel search works with Matryoshka embeddings_ 

To efficiently speed up similarity search while maintaining accuracy, we can use a "funnel search" approach. First, we perform an initial similarity search using only the first 1/32 of the embedding dimensions, generating a broad pool of candidate items. We then rerank these candidates based on their similarity to the query using the first 1/16 of the dimensions, pruning a portion of the list. This process continues iteratively, reranking and pruning with increasingly larger subsets of the embedding dimensions—1/8, 1/4, and so on. Importantly, we only perform one initial similarity search in this lower-dimensional space, and a single pass of the embedding model computes the query embedding. This funneling process narrows down candidates at each step and is faster and more efficient than directly searching in the full-dimensional space. Drawing many matches from the 1/32-dimensional space and refining them through funnel search can significantly accelerate similarity search while preserving strong recall.


## Training

Let’s go into a few of the technical details. The method is very simple to apply. Consider the context of fine-tuning a [BERT model](https://zilliz.com/learn/what-is-bert) for sentence embedding. To convert a BERT model, which has been pre-trained on the masked-token loss, into a sentence embedding model, we form the sentence embedding as the average of the final layer, that is, the average of the per-token contextualized embeddings. 

One choice of training objective is the [Cosine Sentence (CoSENT) loss](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#cosentloss), $L(u, v; s)$. It inputs a pair of sentence embeddings, $u,v$, and their desired similarity score, $s$ (see the link above for the formula). Now, to learn Matryoshka embeddings, we make a small modification to the training objective:

$L_M(u, v) = w_0L(u_{1:d}, v_{1:d}) + w_1L(u_{1:d/2}, v_{1:d/2}) + w_2L(u_{1:d/4}, v_{1:d/4}) + \cdots$

where the sum is continued by calculating the loss on half of the input to the previous term until an information bottleneck is reached. The authors suggest setting

$w_0=w_1=\cdots=1$.

_Simply put, the Matryoshka loss is a weighted sum of the original loss over recursive subsets of the input._

One key takeaway from the equation above is that the Matryoshka loss achieves efficient learning of representations at multiple scales by sharing weights across the embedding models (the same model is used to encode, for example, $u_{1:d}$ and $u_{1:d/2}$) and sharing dimensions across scales ($u_{1:d/2}$ is a subset of $u$).


## Matryoshka Embeddings and Milvus

Milvus seamlessly supports any Matryoshka embedding model that can be loaded via standard libraries such as [pymilvus.model](https://milvus.io/docs/embeddings.md), [sentence-transformers](https://milvus.io/docs/integrate_with_sentencetransformers.md), or other similar tools. From the system’s perspective, there’s no functional difference between a regular embedding model and one specifically trained to generate Matryoshka embeddings. 

Popular Matryoshka embedding models include: 

- OpenAI's [`text-embedding-3-large`](https://zilliz.com/ai-models/text-embedding-3-large)

- Nomic’s [`nomic-embed-text-v1`](https://huggingface.co/nomic-ai/nomic-embed-text-v1)

- Alibaba’s [`gte-multilingual-base`](https://huggingface.co/Alibaba-NLP/gte-multilingual-base)

For a complete guide on using Matryoshka embeddings with Milvus, see the notebook _[Funnel Search with Matryoshka Embeddings](https://github.com/milvus-io/bootcamp/blob/master/bootcamp/tutorials/quickstart/funnel_search_with_matryoshka.ipynb)_.


## Summary

Matryoshka embedding lets developers create shortened embeddings without sacrificing semantic integrity, making them ideal for more efficient search and storage. While you can modify an existing model, pre-trained options, such as those from [OpenAI](https://zilliz.com/ai-models) and [Hugging Face](https://zilliz.com/ai-models), are also available. 

However, a current limitation is the scarcity of open-source Matryoshka embeddings, with few available on the Hugging Face hub. Additionally, these models are often not explicitly labeled as “Matryoshka,” making them harder to locate. Hopefully, with growing interest, broader availability and clearer labeling may soon follow. 

Ready to streamline your search capabilities? Get started with Milvus + Matryoshka embeddings today!


## Resources

- Notebook: [Funnel Search with Matryoshka Embeddings](https://github.com/milvus-io/bootcamp/blob/master/bootcamp/tutorials/quickstart/funnel_search_with_matryoshka.ipynb) 

- Paper: [Matryoshka Representation Learning](https://arxiv.org/abs/2205.13147)

- Paper: [mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrieval](https://arxiv.org/pdf/2407.19669)

- [Introducing PyMilvus Integration with Embedding Models ](https://milvus.io/blog/introducing-pymilvus-integrations-with-embedding-models.md)

- [Exploring BGE-M3: The Future of Information Retrieval with Milvus ](https://zilliz.com/learn/Exploring-BGE-M3-the-future-of-information-retrieval-with-milvus)

- [Nomic Embed: Training a Reproducible Long Context Text Embedder](https://static.nomic.ai/reports/2024_Nomic_Embed_Text_Technical_Report.pdf)

- [Training Matryoshka Embeddings with Sentence Transformers Library](https://sbert.net/examples/training/matryoshka/README.html)

- [Milvus Bootcamp](https://milvus.io/bootcamp)



What are Matryoshka Embeddings?

Matryoshka Embeddings: Detail at Multiple Scales

How to Use the Milvus Backup Tool: A Step-by-Step Guide



[**Milvus**](https://zilliz.com/what-is-milvus) is an open-source [vector database](https://zilliz.com/learn/what-is-vector-database) designed to store, index, and search massive amounts of [unstructured data](https://zilliz.com/learn/introduction-to-unstructured-data) through vector representations, making it perfect for AI-driven applications, such as similarity search, [semantic search](https://zilliz.com/glossary/semantic-search), retrieval augmented generation ([RAG](https://zilliz.com/learn/Retrieval-Augmented-Generation)), recommendation engines, and other machine learning tasks. 

But what makes Milvus even more powerful is its seamless integration with Kubernetes. If you're a Kubernetes aficionado, you know the platform is perfect for orchestrating scalable, distributed systems. Milvus takes full advantage of Kubernetes' capabilities, allowing you to easily deploy, scale, and manage distributed Milvus clusters. This guide will provide a clear, step-by-step walkthrough for setting up Milvus on Kubernetes using the Milvus Operator. 


## Prerequisites

Before we begin, ensure you have the following prerequisites in place:

- A Kubernetes cluster up and running. If you're testing locally, `minikube` is a great choice.

- `kubectl` installed and configured to interact with your Kubernetes cluster.

- Familiarity with basic Kubernetes concepts like pods, services, and deployments.


## Step 1: Installing Minikube (For Local Testing)

If you need to set up a local Kubernetes environment, `minikube` is the tool for you. Official installation instructions are on the [minikube getting started page](https://minikube.sigs.k8s.io/docs/start/).


### 1. Install Minikube

Visit the[ minikube releases page](https://github.com/kubernetes/minikube/releases) and download the appropriate version for your operating system. For macOS/Linux, you can use the following command:

```
$ curl -Lo minikube https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64
$ sudo install minikube-linux-amd64 /usr/local/bin/minikube && rm minikube-linux-amd64
```


### 2. Start Minikube

```
$ minikube start
```


### 3. Interact with Cluster

Now, you can interact with your clusters with the kubectl inside minikube. If you haven’t installed kubectl, minikube will download the appropriate version by default.

```
$ minikube kubectl cluster-info
```

Alternatively, you can create a symbolic link to minikube’s binary named `kubectl` for easier usage.

```
$ sudo ln -s $(which minikube) /usr/local/bin/kubectl
$ kubectl cluster-info
```


## Step 2: Configuring the StorageClass

In Kubernetes, a **StorageClass** defines the types of storage available for your workloads, providing flexibility in managing different storage configurations. Before proceeding, you must ensure a default StorageClass is available in your cluster. Here’s how to check and configure one if necessary.


### 1. Check Installed StorageClasses

To see the available StorageClasses in your Kubernetes cluster, run the following command:

```
$ kubectl get sc
```

This will display the list of storage classes installed in your cluster. If a default StorageClass is already configured, it will be marked with `(default)`.

![](https://assets.zilliz.com/storage_classes_installed_in_your_cluster_21d36d6ac8.png)


### 2. Configure a Default StorageClass (if necessary)

If no default StorageClass is set, you can create one by defining it in a YAML file. Use the following example to create a default StorageClass:

```
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
 name: default-storageclass
provisioner: k8s.io/minikube-hostpath
```

This YAML configuration defines a `StorageClass` called `default-storageclass` that uses the `minikube-hostpath` provisioner, commonly used in local development environments.


### 3. Apply the StorageClass

Once the `default-storageclass.yaml` file is created, apply it to your cluster using the following command:

```
$ kubectl apply -f default-storageclass.yaml
```

This will set up the default StorageClass for your cluster, ensuring that your storage needs are properly managed in the future.


## Step 3: Installing Milvus Using the Milvus Operator

The Milvus Operator simplifies deploying Milvus on Kubernetes, managing the deployment, scaling, and updates. Before installing the Milvus Operator, you'll need to install the **cert-manager**, which provides certificates for the webhook server used by the Milvus Operator.


### 1. Install cert-manager

Milvus Operator requires a [cert-manager](https://cert-manager.io/docs/installation/supported-releases/) to manage certificates for secure communication. Make sure you install **cert-manager version 1.1.3** or later. To install it, run the following command:

```
$ kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.5.3/cert-manager.yaml
```

After the installation, verify that the cert-manager pods are running by executing:

```
$ kubectl get pods -n cert-manager
```


![](https://assets.zilliz.com/verify_that_the_cert_manager_pods_are_running_bb44c2b6d5.png)


### 2. Install the Milvus Operator

Once the cert-manager is up and running, you can install the Milvus Operator. Run the following command to deploy it using `kubectl`:

```
$ kubectl apply -f https://raw.githubusercontent.com/zilliztech/milvus-operator/main/deploy/manifests/deployment.yaml
```

You can check if the Milvus Operator pod is running using the following command: 

```
$ kubectl get pods -n milvus-operator
```


![](https://assets.zilliz.com/check_if_the_Milvus_Operator_pod_is_running_6e7ac41ebf.png)

### 3. Deploy Milvus Cluster

Once the Milvus Operator pod is running, you can deploy a Milvus cluster with the operator. The following command deploys a Milvus cluster with its components and dependencies in separate pods using default configurations:

```
$ kubectl apply -f https://raw.githubusercontent.com/zilliztech/milvus-operator/main/config/samples/milvus_cluster_default.yaml
```



![](https://assets.zilliz.com/Deploy_Milvus_Cluster_8b5d5343af.png)

To customize the Milvus settings, you will need to replace the YAML file with your own configuration YAML file. In addition to manually editing or creating the file, you can use the Milvus Sizing Tool to adjust the configurations and then download the corresponding YAML file.

To customize Milvus settings, you must replace the default YAML file with your own configuration. You can either manually edit or create this file, tailoring it to your specific requirements. 

Alternatively, you can use the [**Milvus Sizing Tool**](https://milvus.io/tools/sizing) for a more streamlined approach. This tool allows you to adjust various settings, such as resource allocation and storage options, and then download the corresponding YAML file with your desired configurations. This ensures that your Milvus deployment is optimized for your specific use case.

![](https://assets.zilliz.com/Figure_Milvus_sizing_tool_024693df9d.png)

Figure: Milvus sizing tool 

It may take some time to finish the deployment. You can check the status of your Milvus cluster via the command:

```
$ kubectl get milvus my-release
```


![](https://assets.zilliz.com/check_the_status_of_your_Milvus_cluster_bcbd85fd70.png)

Once your Milvus cluster is ready, all pods in the Milvus cluster should be running or completed:

```
$ kubectl get pods
```


## Step 4: Accessing Your Milvus Cluster

Once your Milvus cluster is deployed, you need to access it by forwarding a local port to the Milvus service port. Follow these steps to retrieve the service port and set up port forwarding.


#### **1. Get the Service Port**

First, identify the service port by using the following command. Replace `<YOUR_MILVUS_PROXY_POD>` with the name of your Milvus proxy pod, which typically starts with `my-release-milvus-proxy-`:

```
$ kubectl get pod <YOUR_MILVUS_PROXY_POD> --template ='{{(index (index .spec.containers 0).ports 0).containerPort}}{{"\n"}}'
```

This command will return the port number that your Milvus service is using.


#### **2. Forward the Port**

To access your Milvus cluster locally, forward a local port to the service port using the following command. Replace `<YOUR_LOCAL_PORT>` with the local port you want to use and `<YOUR_SERVICE_PORT>` with the service port retrieved in the previous step:

```
$ kubectl port-forward --address 0.0.0.0 service/my-release-milvus <YOUR_LOCAL_PORT>:<YOUR_SERVICE_PORT>
```

This command allows port-forwarding to listen on all IP addresses of the host machine. If you only need the service to listen on `localhost`, you can omit the `--address 0.0.0.0` option.

Once the port-forwarding is set up, you can access your Milvus cluster via the specified local port for further operations or integrations.


## Step 5: Connecting to Milvus Using Python SDK

With your Milvus cluster up and running, you can now interact with it using any Milvus SDK. In this example, we'll use [PyMilvus](https://zilliz.com/blog/what-is-pymilvus), Milvus’s **Python SDK,**  to connect to the cluster and perform basic operations.


### 1. Install PyMilvus

To interact with Milvus via Python, you need to install the `pymilvus` package:

```
$ pip install pymilvus
```


### 2. Connect to Milvus

The following is a sample Python script that connects to your Milvus cluster and demonstrates how to perform basic operations such as creating a collection.

```
from pymilvus import MilvusClient
```

```
# Connect to the Milvus server
client = MilvusClient(uri="http://localhost:<YOUR_LOCAL_PORT>")
```

```
# Create a collection
collection_name = "example_collection"
if client.has_collection(collection_name):
   client.drop_collection(collection_name)
client.create_collection(
   collection_name=collection_name,
   dimension=768,  # The vectors we will use in this demo has 768 dimensions
)
```


#### Explanation:

- Connect to Milvus: The script connects to the Milvus server running on `localhost` using the local port you set up in Step 4.

- Create a Collection: It checks if a collection named `example_collection` already exists, drops it if so, and then creates a new collection with vectors of 768 dimensions.

This script establishes a connection to the Milvus cluster and creates a collection, serving as a starting point for more complex operations like inserting vectors and performing similarity searches.


## Conclusion

Deploying Milvus in a distributed setup on Kubernetes unlocks powerful capabilities for managing large-scale vector data, enabling seamless scalability and high-performance AI-driven applications. Following this guide, you've learned how to set up Milvus using the Milvus Operator, making the process streamlined and efficient.

As you continue to explore Milvus, consider scaling your cluster to meet growing demands or deploying it on cloud platforms such as Amazon EKS, Google Cloud, or Microsoft Azure. For enhanced management and monitoring, tools like [**Milvus Backup**](https://milvus.io/docs/milvus_backup_overview.md), [**Birdwatcher**](https://milvus.io/docs/birdwatcher_overview.md), and [**Attu**](https://github.com/zilliztech/attu) offer valuable support for maintaining the health and performance of your deployments.

You're now ready to harness the full potential of Milvus on Kubernetes—happy deploying! 🚀


## Further Resources

- [Milvus Documentation](https://milvus.io/docs/overview.md)

- [Milvus Lite vs. Standalone vs. Distributed: Which Mode is Right for You? ](https://zilliz.com/blog/choose-the-right-milvus-deployment-mode-ai-applications)

- [Supercharging Vector Search: Milvus on GPUs with NVIDIA RAPIDS cuVS](https://zilliz.com/blog/milvus-on-gpu-with-nvidia-rapids-cuvs)

- [What is RAG? ](https://zilliz.com/learn/Retrieval-Augmented-Generation)

- [Generative AI Resource Hub | Zilliz](https://zilliz.com/learn/generative-ai)

- [Top Performing AI Models for Your GenAI Apps | Zilliz](https://zilliz.com/ai-models)


Deploying Milvus on Kubernetes: A Step-by-Step Guide for Kubernetes Users

How to Deploy the Open-Source Milvus Vector Database on Amazon EKS


[Milvus](https://milvus.io/intro) is an open-source vector database designed specifically for AI applications. Whether you're working on machine learning, deep learning, or any other AI-related project, Milvus offers a robust and efficient way to handle large-scale vector data.

Now, with the [model module integration](https://milvus.io/docs/embeddings.md) in PyMilvus, the Python SDK for Milvus, it's even easier to add Embedding and Reranking models. This integration simplifies transforming your data into searchable vectors or reranking results for more accurate outcomes, such as in [Retrieval Augmented Generation (RAG)](https://zilliz.com/learn/Retrieval-Augmented-Generation).

In this blog, we will review dense embedding models, sparse embedding models, and re-rankers and demonstrate how to use them in practice using [Milvus Lite](https://milvus.io/blog/introducing-milvus-lite.md), a lightweight version of Milvus that can run locally in your Python applications. 


## Dense vs Sparse Embeddings 

Before we walk you through how to use our integrations, let’s look at two main categories of vector embeddings. 

[Vector Embeddings](https://zilliz.com/glossary/vector-embeddings) generally fall into two main categories: [**Dense Embeddings** and **Sparse Embeddings**](https://zilliz.com/learn/sparse-and-dense-embeddings).

- Dense Embeddings are high-dimensional vectors in which most or all elements are non-zero, making them ideal for encoding text semantics or fuzzy meaning.

- Sparse Embeddings are high-dimensional vectors with many zero elements, better suited for encoding exact or adjacent concepts.

Milvus supports both types of embeddings and offers hybrid search. [Hybrid Search](https://zilliz.com/blog/hybrid-search-with-milvus) allows you to conduct searches across various vector fields within the same collection. These vectors can represent different facets of data, use diverse embedding models, or employ distinct data processing methods, combining the results using re-rankers.

## How to Use Our Embedding and Reranking Integrations

In the following sections, we’ll demonstrate three practical examples of using our integrations to generate embeddings and conduct vector searches. 

### Example 1: Use the Default Embedding Function to Generate Dense Vectors 

You must install the `pymilvus` client with the `model` package to use embedding and reranking functions with Milvus. 

```
pip install "pymilvus[model]"
```

This step will install [Milvus Lite](https://milvus.io/docs/quickstart.md), allowing you to run Milvus locally within your Python application. It also includes the model subpackage, which includes all utilities for Embedding and reranking.

The model subpackage supports various embedding models, including those from OpenAI, [Sentence Transformers](https://zilliz.com/learn/Sentence-Transformers-for-Long-Form-Text), [BGE-M3](https://zilliz.com/learn/bge-m3-and-splade-two-machine-learning-models-for-generating-sparse-embeddings), BM25, [SPLADE](https://zilliz.com/learn/bge-m3-and-splade-two-machine-learning-models-for-generating-sparse-embeddings), and Jina AI pre-trained models.

This example uses the `DefaultEmbeddingFunction`, based on the `all-MiniLM-L6-v2` Sentence Transformer model for simplicity. The model is about 70MB and will be downloaded during the first use:

```
from pymilvus import model

# This will download "all-MiniLM-L6-v2", a lightweight model.
ef = model.DefaultEmbeddingFunction()

# Data from which embeddings are to be generated
docs = [
   "Artificial intelligence was founded as an academic discipline in 1956.",
   "Alan Turing was the first person to conduct substantial research in AI.",
   "Born in Maida Vale, London, Turing was raised in southern England.",
]

embeddings = ef.encode_documents(docs)

print("Embeddings:", embeddings)
# Print dimension and shape of embeddings
print("Dim:", ef.dim, embeddings[0].shape)
```

The expected output should be something like the following: 

```
Embeddings: [array([-3.09392996e-02, -1.80662833e-02,  1.34775648e-02,  2.77156215e-02,
      -4.86349640e-03, -3.12581174e-02, -3.55921760e-02,  5.76934684e-03,
       2.80773244e-03,  1.35783911e-01,  3.59678417e-02,  6.17732145e-02,
...
      -4.61330153e-02, -4.85207550e-02,  3.13997865e-02,  7.82178566e-02,
      -4.75336798e-02,  5.21207601e-02,  9.04406682e-02, -5.36676683e-02],
     dtype=float32)]
Dim: 384 (384,)
```


### Example 2: Generate Sparse Vectors Using The BM25 Model

BM25 is a well-known method that uses word occurrence frequencies to determine the relevance between queries and documents. In this example, we’ll show how to use `BM25EmbeddingFunction` to generate sparse embeddings for queries and documents.

In BM25, it's important to calculate the statistics in your documents to obtain the IDF (Inverse Document Frequency), which can represent the patterns in your documents. The IDF measures how much information a word provides, whether it's common or rare across all documents.

```
from pymilvus.model.sparse import BM25EmbeddingFunction

# 1. Prepare a small corpus to search
docs = [
   "Artificial intelligence was founded as an academic discipline in 1956.",
   "Alan Turing was the first person to conduct substantial research in AI.",
   "Born in Maida Vale, London, Turing was raised in southern England.",
]
query = "Where was Turing born?"
bm25_ef = BM25EmbeddingFunction()

# 2. Fit the corpus to get BM25 model parameters on your documents.
bm25_ef.fit(docs)

# 3. Store the fitted parameters to expedite future processing.
bm25_ef.save("bm25_params.json")

# 4. Load the saved params
new_bm25_ef = BM25EmbeddingFunction()
new_bm25_ef.load("bm25_params.json")

docs_embeddings = new_bm25_ef.encode_documents(docs)
query_embeddings = new_bm25_ef.encode_queries([query])
print("Dim:", new_bm25_ef.dim, list(docs_embeddings)[0].shape)
```

### Example 3: Using a ReRanker 

A search system aims to find the most relevant results quickly and efficiently. Traditionally, methods like BM25 or TF-IDF have been used to rank search results based on keyword matching. Recent methods, such as embedding-based cosine similarity, are straightforward but can sometimes miss the subtleties of language and, most importantly, the interaction between documents and a query's intent.

This is where using a [re-ranker](https://zilliz.com/learn/optimize-rag-with-rerankers-the-role-and-tradeoffs) helps. A re-ranker is an advanced AI model that takes the initial set of results from a search—often provided by an embeddings/token-based search—and re-evaluates them to ensure they align more closely with the user's intent. It looks beyond the surface-level matching of terms to consider the deeper interaction between the search query and the content of the documents.

For this example, we’ll use the [Jina AI Reranker](https://milvus.io/docs/integrate_with_jina.md).



```
from pymilvus.model.reranker import JinaRerankFunction

jina_api_key = "<YOUR_JINA_API_KEY>"

rf = JinaRerankFunction("jina-reranker-v1-base-en", jina_api_key)

query = "What event in 1956 marked the official birth of artificial intelligence as a discipline?"

documents = [
   "In 1950, Alan Turing published his seminal paper, 'Computing Machinery and Intelligence,' proposing the Turing Test as a criterion of intelligence, a foundational concept in the philosophy and development of artificial intelligence.",
   "The Dartmouth Conference in 1956 is considered the birthplace of artificial intelligence as a field; here, John McCarthy and others coined the term 'artificial intelligence' and laid out its basic goals.",
   "In 1951, British mathematician and computer scientist Alan Turing also developed the first program designed to play chess, demonstrating an early example of AI in game strategy.",
   "The invention of the Logic Theorist by Allen Newell, Herbert A. Simon, and Cliff Shaw in 1955 marked the creation of the first true AI program, which was capable of solving logic problems, akin to proving mathematical theorems."
]

results = rf(query, documents)

for result in results:
   print(f"Index: {result.index}")
   print(f"Score: {result.score:.6f}")
   print(f"Text: {result.text}\n")
```

The expected output is similar to the following:

```
Index: 1
Score: 0.937096
Text: The Dartmouth Conference in 1956 is considered the birthplace of artificial intelligence as a field; here, John McCarthy and others coined the term 'artificial intelligence' and laid out its basic goals.

Index: 3
Score: 0.354210
Text: The invention of the Logic Theorist by Allen Newell, Herbert A. Simon, and Cliff Shaw in 1955 marked the creation of the first true AI program, which was capable of solving logic problems, akin to proving mathematical theorems.

Index: 0
Score: 0.349866
Text: In 1950, Alan Turing published his seminal paper, 'Computing Machinery and Intelligence,' proposing the Turing Test as a criterion of intelligence, a foundational concept in the philosophy and development of artificial intelligence.

Index: 2
Score: 0.272896
Text: In 1951, British mathematician and computer scientist Alan Turing also developed the first program designed to play chess, demonstrating an early example of AI in game strategy.
```

## Star Us On GitHub and Join Our Discord!

If you liked this blog post, consider starring Milvus on [GitHub](https://github.com/milvus-io/milvus), and feel free to join our [Discord](https://discord.gg/FG6hMJStWu)! 💙


Introducing PyMilvus Integration with Embedding Models


![](https://assets.zilliz.com/2_72e444c8dc.JPG)

We are excited to introduce [Milvus Lite](https://milvus.io/docs/milvus_lite.md), a lightweight vector database that runs locally within your Python application. Based on the popular open-source [Milvus](https://milvus.io/intro) vector database, Milvus Lite reuses the core components for vector indexing and query parsing while removing elements designed for high scalability in distributed systems. This design makes a compact and efficient solution ideal for environments with limited computing resources, such as laptops, Jupyter Notebooks, and mobile or edge devices.

Milvus Lite integrates with various AI development stacks like LangChain and LlamaIndex, enabling its use as a vector store in Retrieval Augmented Generation (RAG) pipelines without the need for server setup. Simply run `pip install pymilvus` (version 2.4.3 or above) to incorporate it into your AI application as a Python library.

Milvus Lite shares the Milvus API, ensuring that your client-side code works for both small-scale local deployments and Milvus servers deployed on Docker or Kubernetes with billions of vectors.

## Why We Built Milvus Lite

Many AI applications require vector similarity search for unstructured data, including text, images, voices, and videos, for applications such as chatbots and shopping assistants. Vector databases are crafted for storing and searching vector embeddings and are a crucial part of the AI development stack, particularly for generative AI use cases like [Retrieval Augmented Generation (RAG)](https://zilliz.com/learn/Retrieval-Augmented-Generation).

Despite the availability of numerous vector search solutions, an easy-to-start option that also works for large-scale production deployments was missing. As the creators of Milvus, we designed Milvus Lite to help AI developers build applications faster while ensuring a consistent experience across various deployment options, including Milvus on Kubernetes, Docker, and managed cloud services.

Milvus Lite is a crucial addition to our suite of offerings within the Milvus ecosystem. It provides developers with a versatile tool that supports every stage of their development journey. From prototyping to production environments and from edge computing to large-scale deployments, Milvus is now the only vector database that covers use cases of any size and all stages of development.

## How Milvus Lite Works 

Milvus Lite supports all the basic operations available in Milvus, such as creating collections and inserting, searching, and deleting vectors. It will soon support advanced features like hybrid search. Milvus Lite loads data into memory for efficient searches and persists it as an SQLite file.

Milvus Lite is included in the [Python SDK of Milvus](https://github.com/milvus-io/pymilvus) and can be deployed with a simple `pip install pymilvus`. The following code snippet demonstrates how to set up a vector database with Milvus Lite by specifying a local file name and then creating a new collection. For those familiar with the Milvus API, the only difference is that the `uri` refers to a local file name instead of a network endpoint, e.g., `"milvus_demo.db"` instead of `"http://localhost:19530"` for a Milvus server. Everything else remains the same. Milvus Lite also supports storing raw text and other labels as metadata, using a dynamic or explicitly defined schema, as shown below.

```
from pymilvus import MilvusClient

client = MilvusClient("milvus_demo.db")
# This collection can take input with mandatory fields named "id", "vector" and
# any other fields as "dynamic schema". You can also define the schema explicitly.
client.create_collection(
    collection_name="demo_collection",
    dimension=384  # Dimension for vectors.
)
```
For scalability, an AI application developed with Milvus Lite can easily transition to using Milvus deployed on Docker or Kubernetes by simply specifying the `uri` with the server endpoint.


## Integration with AI Development Stack

In addition to introducing Milvus Lite to make vector search easy to start with, Milvus also integrates with many frameworks and providers of the AI development stack, including [LangChain](https://python.langchain.com/v0.2/docs/integrations/vectorstores/milvus/), [LlamaIndex](https://docs.llamaindex.ai/en/stable/examples/vector_stores/MilvusIndexDemo/), [Haystack](https://haystack.deepset.ai/integrations/milvus-document-store), [Voyage AI](https://blog.voyageai.com/2024/05/30/semantic-search-with-milvus-lite-and-voyage-ai/), [Ragas](https://milvus.io/docs/integrate_with_ragas.md), [Jina AI](https://jina.ai/news/implementing-a-chat-history-rag-with-jina-ai-and-milvus-lite/), [DSPy](https://dspy-docs.vercel.app/docs/deep-dive/retrieval_models_clients/MilvusRM), [BentoML](https://www.bentoml.com/blog/building-a-rag-app-with-bentocloud-and-milvus-lite), [WhyHow](https://chiajy.medium.com/70873c7576f1), [Relari AI](https://blog.relari.ai/case-study-using-synthetic-data-to-benchmark-rag-systems-be324904ace1), [Airbyte](https://docs.airbyte.com/integrations/destinations/milvus), [HuggingFace](https://milvus.io/docs/integrate_with_hugging-face.md) and [MemGPT](https://memgpt.readme.io/docs/storage#milvus). Thanks to their extensive tooling and services, these integrations simplify the development of AI applications with vector search capability.

And this is just the beginning—many more exciting integrations are coming soon! Stay tuned! 


## More Resources and Examples

Explore [Milvus quickstart documentation](https://milvus.io/docs/quickstart.md) for detailed guides and code examples on using Milvus Lite to build AI applications like Retrieval-Augmented Generation ([RAG](https://github.com/milvus-io/bootcamp/blob/master/bootcamp/tutorials/quickstart/build_RAG_with_milvus.ipynb)) and [image search](https://github.com/milvus-io/bootcamp/blob/master/bootcamp/tutorials/quickstart/image_search_with_milvus.ipynb).

Milvus Lite is an open-source project, and we welcome your contributions. Check out our [Contributing Guide](https://github.com/milvus-io/milvus-lite/blob/main/CONTRIBUTING.md) to get started. You can also report bugs or request features by filing an issue on the [Milvus Lite GitHub](https://github.com/milvus-io/milvus-lite) repository.


Introducing Milvus Lite: Start Building a GenAI Application in Seconds


*This post was originally published on [Intel's Medium Channel](https://medium.com/intel-tech/optimize-vector-databases-enhance-rag-driven-generative-ai-90c10416cb9c) and is reposted here with permission.*


<br>

Two methods to optimize your vector database when using RAG


![](https://miro.medium.com/v2/resize:fit:1400/1*FRWBVwOHPYFDIVTp_ylZNQ.jpeg)


Photo by [Ilya Pavlov](https://unsplash.com/@ilyapavlov?utm_content=creditCopyText\&utm_medium=referral\&utm_source=unsplash) on [Unsplash](https://unsplash.com/photos/monitor-showing-java-programming-OqtafYT5kTw?utm_content=creditCopyText\&utm_medium=referral\&utm_source=unsplash)



By Cathy Zhang and Dr. Malini Bhandaru
Contributors: Lin Yang and Changyan Liu


Generative AI (GenAI) models, which are seeing exponential adoption in our daily lives, are being improved by [retrieval-augmented generation (RAG)](https://www.techtarget.com/searchenterpriseai/definition/retrieval-augmented-generation), a technique used to enhance response accuracy and reliability by fetching facts from external sources. RAG helps a regular [large language model (LLM)](https://www.techtarget.com/whatis/definition/large-language-model-LLM) understand context and reduce [hallucinations](https://en.wikipedia.org/wiki/Hallucination_$artificial_intelligence$) by leveraging a giant database of unstructured data stored as vectors — a mathematical presentation that helps capture context and relationships between data.


RAG helps to retrieve more contextual information and thus generate better responses, but the vector databases they rely on are getting ever larger to provide rich content to draw upon. Just as trillion-parameter LLMs are on the horizon, vector databases of billions of vectors are not far behind. As optimization engineers, we were curious to see if we could make vector databases more performant, load data faster, and create indices faster to ensure retrieval speed even as new data is added. Doing so would not only result in reduced user wait time, but also make RAG-based AI solutions a little more sustainable.


In this article, you’ll learn more about vector databases and their benchmarking frameworks, datasets to tackle different aspects, and the tools used for performance analysis — everything you need to start optimizing vector databases. We will also share our optimization achievements on two popular vector database solutions to inspire you on your optimization journey of performance and sustainability impact.




## Understanding Vector Databases


Unlike traditional relational or non-relational databases where data is stored in a structured manner, a vector database contains a mathematical representation of individual data items, called a vector, constructed using an embedding or transformation function. The vector commonly represents features or semantic meanings and can be short or long. Vector databases do vector retrieval by similarity search using a distance metric (where closer means the results are more similar) such as [Euclidean, dot product, or cosine similarity](https://www.pinecone.io/learn/vector-similarity/).


To accelerate the retrieval process, the vector data is organized using an indexing mechanism. Examples of these organization methods include flat structures, [inverted file (IVF),](https://arxiv.org/abs/2002.09094) [Hierarchical Navigable Small Worlds (HNSW),](https://arxiv.org/abs/1603.09320) and [locality-sensitive hashing (LSH)](https://en.wikipedia.org/wiki/Locality-sensitive_hashing), among others. Each of these methods contributes to the efficiency and effectiveness of retrieving similar vectors when needed.


Let’s examine how you would use a vector database in a GenAI system. Figure 1 illustrates both the loading of data into a vector database and using it in the context of a GenAI application. When you input your prompt, it undergoes a transformation process identical to the one used to generate vectors in the database. This transformed vector prompt is then used to retrieve similar vectors from the vector database. These retrieved items essentially serve as conversational memory, furnishing contextual history for prompts, akin to how LLMs operate. This feature proves particularly advantageous in natural language processing, computer vision, recommendation systems, and other domains requiring semantic comprehension and data matching. Your initial prompt is subsequently “merged” with the retrieved elements, supplying context, and assisting the LLM in formulating responses based on the provided context rather than solely relying on its original training data.


![](https://miro.medium.com/v2/resize:fit:1400/1*zQj_YJdWc2xKB6Vv89lzDQ.jpeg)


Figure 1. A RAG application architecture.


Vectors are stored and indexed for speedy retrieval. Vector databases come in two main flavors, traditional databases that have been extended to store vectors, and purpose-built vector databases. Some examples of traditional databases that provide vector support are [Redis](https://redis.io/), [pgvector](https://github.com/pgvector/pgvector), [Elasticsearch](https://www.elastic.co/elasticsearch), and [OpenSearch](https://opensearch.org/). Examples of purpose-built vector databases include proprietary solutions [Zilliz](https://zilliz.com/) and [Pinecone](https://www.pinecone.io/), and open source projects [Milvus](https://milvus.io/), [Weaviate](https://weaviate.io/), [Qdrant](https://qdrant.tech/), [Faiss](https://github.com/facebookresearch/faiss), and [Chroma](https://www.trychroma.com/). You can learn more about vector databases on GitHub via [LangChain ](https://github.com/langchain-ai/langchain/tree/master/libs/langchain/langchain/vectorstores)and [OpenAI Cookbook](https://github.com/openai/openai-cookbook/tree/main/examples/vector_databases).


We’ll take a closer look at one from each category, Milvus and Redis.




## Improving Performance


Before diving into the optimizations, let’s review how vector databases are evaluated, some evaluation frameworks, and available performance analysis tools.




### Performance Metrics


Let’s look at key metrics that can help you measure vector database performance.


- **Load latency** measures the time required to load data into the vector database’s memory and build an index. An index is a data structure used to efficiently organize and retrieve vector data based on its similarity or distance. Types of [in-memory indices](https://milvus.io/docs/index.md#In-memory-Index) include [flat index](https://thedataquarry.com/posts/vector-db-3/#flat-indexes), [IVF\_FLAT](https://supabase.com/docs/guides/ai/vector-indexes/ivf-indexes), [IVF\_PQ, HNSW](https://towardsdatascience.com/ivfpq-hnsw-for-billion-scale-similarity-search-89ff2f89d90e), [scalable nearest neighbors (ScaNN),](https://github.com/google-research/google-research/tree/master/scann)and [DiskANN](https://milvus.io/docs/disk_index.md).
- **Recall** is the proportion of true matches, or relevant items, found in the [Top K](https://redis.io/docs/data-types/probabilistic/top-k/) results retrieved by the search algorithm. Higher recall values indicate better retrieval of relevant items.
- **Queries per second (QPS)** is the rate at which the vector database can process incoming queries. Higher QPS values imply better query processing capability and system throughput.




### Benchmarking Frameworks


![](https://miro.medium.com/v2/resize:fit:920/1*mssEjZAuXg6nf-pad67rHA.jpeg)


Figure 2. The vector database benchmarking framework.


Benchmarking a vector database requires a vector database server and clients. In our performance tests, we used two popular open source tools.


- [**VectorDBBench**](https://github.com/zilliztech/VectorDBBench/tree/main)**:** Developed and open sourced by Zilliz, VectorDBBench helps test different vector databases with different index types and provides a convenient web interface.
- [**vector-db-benchmark**](https://github.com/qdrant/vector-db-benchmark/tree/master)**:** Developed and open sourced by Qdrant, vector-db-benchmark helps test several typical vector databases for the [HNSW](https://www.datastax.com/guides/hierarchical-navigable-small-worlds) index type. It runs tests through the command line and provides a [Docker Compose](https://docs.docker.com/compose/) __file to simplify starting server components.


![](https://miro.medium.com/v2/resize:fit:1400/1*NpHHEFV0TxRMse83hK6H1A.jpeg)


Figure 3. An example vector-db-benchmark command used to run the benchmark test.


But the benchmark framework is only part of the equation. We need data that exercises different aspects of the vector database solution itself, such as its ability to handle large volumes of data, various vector sizes, and speed of retrieval.With that, let’s look at some available public datasets.




### Open Datasets to Exercise Vector Databases


Large datasets are good candidates to test load latency and resource allocation. Some datasets have high dimensional data and are good for testing speed of computing similarity.


Datasets range from a dimension of 25 to a dimension of 2048. The [LAION](https://laion.ai/) dataset, an open image collection, has been used for training very large visual and language deep-neural models like stable diffusion generative models. OpenAI’s dataset of 5M vectors, each with a dimension of 1536, was created by VectorDBBench by running OpenAI on [raw data](https://huggingface.co/datasets/allenai/c4). Given each vector element is of type FLOAT, to save the vectors alone, approximately 29 GB (5M \* 1536 \* 4) of memory is needed, plus a similar amount extra to hold indices and other metadata for a total of 58 GB of memory for testing. When using the vector-db-benchmark tool, ensure adequate disk storage to save results.


To test for load latency, we needed a large collection of vectors, which [deep-image-96-angular](https://docs.hippo.transwarp.io/docs/performance-dataset) offers. To test performance of index generation and similarity computation, high dimensional vectors provide more stress. To this end we chose the 500K dataset of 1536 dimension vectors.




### Performance Tools


We’ve covered ways to stress the system to identify metrics of interest, but let’s examine what’s happening at a lower level: How busy is the computing unit, memory consumption, waits on locks, and more? These provide clues to databasebehavior, particularly useful in identifying problem areas.


The Linux [top](https://www.redhat.com/sysadmin/interpret-top-output) utility provides system-performance information. However, the [perf](https://perf.wiki.kernel.org/index.php/Main_Page) tool in Linux provides a deeper set of insights. To learn more, we also recommend reading [Linux perf examples](https://www.brendangregg.com/perf.html) and the [Intel top-down microarchitecture analysis method](https://www.intel.com/content/www/us/en/docs/vtune-profiler/cookbook/2023-0/top-down-microarchitecture-analysis-method.html). Yet another tool is the [Intel® vTune™ Profiler](https://www.intel.com/content/www/us/en/developer/tools/oneapi/vtune-profiler.html), which is useful when optimizing not just application but also system performance and configuration for a variety of workloads spanning HPC, cloud, IoT, media, storage, and more.




## Milvus Vector Database Optimizations


Let’s walk through some examples of how we attempted to improve the performance of the Milvus vector database.




### Reducing Memory Movement Overhead in Datanode Buffer Write


Milvus’s write path proxies write data into a log broker via _MsgStream_. The data nodes then consume the data, converting and storing it into segments. Segments will merge the newly inserted data. The merge logic allocates a new buffer to hold/move both the old data and the new data to be inserted and then returns the new buffer as old data for the next data merge. This results in the old data getting successively larger, which in turn makes data movement slower. Perf profiles showed a high overhead for this logic.


![](https://miro.medium.com/v2/resize:fit:1400/1*Az4dMVBcGmdeyKNrwpR19g.jpeg)


Figure 4. Merging and moving data in the vector database generates a high-performance overhead.


We changed the _merge buffer_ logic to directly append the new data to be inserted into the old data, avoiding allocating a new buffer and moving the large old data. Perf profiles confirm that there is no overhead to this logic. The microcode metrics _metric\_CPU operating frequency_ and _metric\_CPU utilization_ indicate an improvement that is consistent with the system not having to wait for the long memory movement anymore. Load latency improved by more than 60 percent. The improvement is captured on [GitHub](https://github.com/milvus-io/milvus/pull/26839).


![](https://miro.medium.com/v2/resize:fit:1400/1*MmaUtBTdqmMvC5MlQ8V0wQ.jpeg)


Figure 5. With less copying we see a performance improvement of more than 50 percent in load latency.




### Inverted Index Building with Reduced Memory Allocation Overhead


The Milvus search engine, [Knowhere](https://milvus.io/docs/knowhere.md), employs the [Elkan k-means algorithm](https://www.vlfeat.org/api/kmeans-fundamentals.html#kmeans-elkan) to train cluster data for creating [inverted file (IVF) indices](https://milvus.io/docs/v1.1.1/index.md). Each round of data training defines an iteration count. The larger the count, the better the training results. However, it also implies that the Elkan algorithm will be called more frequently.


The Elkan algorithm handles memory allocation and deallocation each time it’s executed. Specifically, it allocates memory to store half the size of symmetric matrix data, excluding the diagonal elements. In Knowhere, the symmetric matrix dimension used by the Elkan algorithm is set to 1024, resulting in a memory size of approximately 2 MB. This means for each training round Elkan repeatedly allocates and deallocates 2 MB memory.


Perf profiling data indicated frequent large memory allocation activity. In fact, it triggered [virtual memory area (VMA)](https://www.oreilly.com/library/view/linux-device-drivers/9781785280009/4759692f-43fb-4066-86b2-76a90f0707a2.xhtml)allocation, physical page allocation, page map setup, and updating of memory cgroup statistics in the kernel. This pattern of large memory allocation/deallocation activity can, in some situations, also aggravate memory fragmentation. This is a significant tax.


The _IndexFlatElkan_ structure is specifically designed and constructed to support the Elkan algorithm. Each data training process will have an _IndexFlatElkan_ instance initialized. To mitigate the performance impact resulting from frequent memory allocation and deallocation in the Elkan algorithm, we refactored the code logic, moving the memory management outside of the Elkan algorithm function up into the construction process of _IndexFlatElkan_. This enables memory allocation to occur only once during the initialization phase while serving all subsequent Elkan algorithm function calls from the current data training process and helps to improve load latency by around 3 percent. Find the [Knowhere patch here](https://github.com/zilliztech/knowhere/pull/280).




## Redis Vector Search Acceleration through Software Prefetch


Redis, a popular traditional in-memory key-value data store, recently began supporting vector search. To go beyond a typical key-value store, it offers extensibility modules; the [RediSearch](https://github.com/RediSearch/RediSearch) module facilitates the storage and search of vectors directly within Redis.


For vector similarity search, Redis supports two algorithms, namely brute force and HNSW. The HNSW algorithm is specifically crafted for efficiently locating approximate nearest neighbors in high-dimensional spaces. It uses a priority queue named _candidate\_set_ to manage all vector candidates for distance computing.


Each vector candidate encompasses substantial metadata in addition to the vector data. As a result, when loading a candidate from memory it can cause data cache misses, which incur processing delays. Our optimization introduces software prefetching to proactively load the next candidate while processing the current one. This enhancement has resulted in a 2 to 3 percent throughput improvement for vector similarity searches in a single instance Redis setup. The patch is in the process of being upstreamed.




## GCC Default Behavior Change to Prevent Mixed Assembly Code Penalties


To drive maximum performance, frequently used sections of code are often handwritten in assembly. However, when different segments of code are written either by different people or at different points in time, the instructions used may come from incompatible assembly instruction sets such as [Intel® Advanced Vector Extensions 512 (Intel® AVX-512)](https://www.intel.com/content/www/us/en/architecture-and-technology/avx-512-overview.html) and [Streaming SIMD Extensions (SSE)](https://en.wikipedia.org/wiki/Streaming_SIMD_Extensions). If not compiled appropriately, the mixed code results in a performance penalty. [Learn more about mixing Intel AVX and SSE instructions here](https://www.intel.com/content/dam/develop/external/us/en/documents/11mc12-avoiding-2bavx-sse-2btransition-2bpenalties-2brh-2bfinal-809104.pdf).


You can easily determine if you’re using mixed-mode assembly code and have not compiled the code with _VZEROUPPER_, incurring the performance penalty. It can be observed through a perf command like _sudo perf stat -e ‘assists.sse\_avx\_mix/event/event=0xc1,umask=0x10/’ \<workload>_. If your OS doesn’t have support for the event, use _cpu/event=0xc1,umask=0x10,name=assists\_sse\_avx\_mix/_.


The Clang compiler by default inserts _VZEROUPPER_, avoiding any mixed mode penalty. But the GCC compiler only inserted _VZEROUPPER_ when the -O2 or -O3 compiler flags were specified. We contacted the GCC team and explained the issue and they now, by default, correctly handle mixed mode assembly code.



## Start Optimizing Your Vector Databases


Vector databases are playing an integral role in GenAI, and they are growing ever larger to generate higher-quality responses. With respect to optimization, AI applications are no different from other software applications in that they reveal their secrets when one employs standard performance analysis tools along with benchmark frameworks and stress input.

Using these tools, we uncovered performance traps pertaining to unnecessary memory allocation, failing to prefetch instructions, and using incorrect compiler options. Based on our findings, we upstreamed enhancements to Milvus, Knowhere, Redis, and the GCC compiler to help make AI a little more performant and sustainable. Vector databases are an important class of applications worthy of your optimization efforts. We hope this article helps you get started.


Optimize Vector Databases, Enhance RAG-Driven Generative AI



## Introduction

Milvus is a distributed vector database that aims to store, index and manage massive embedding vectors. Its ability to efficiently index and search through trillions of vectors makes Milvus a go-to choice for AI and machine learning workloads.

Kubernetes (K8s), on the other hand, excels in managing and scaling containerized applications. It provides features like auto-scaling, self-healing, and load balancing, which are crucial for maintaining high availability and performance in production environments.


## Why Use Them Together? 

K8s can automatically scale the Milvus clusters based on the workload. As your data grows or the number of queries increases, K8s can spin up more Milvus instances to handle the load, ensuring your applications remain responsive.

One of the standout features of K8s is its horizontal scaling, which makes expanding your Milvus cluster a breeze. As your dataset grows, K8s effortlessly accommodates this growth, making it a straightforward and efficient solution.

In addition, the ability to handle queries also scales horizontally with K8s. As the query load increases, K8s can deploy more Milvus instances to handle the increased similarity search queries, ensuring low latency responses even under heavy loads.


## Prerequisites & Setting Up K8s

### Prerequisites

- **Docker** - Ensure Docker is installed on your system.

- **Kubernetes** - Have a Kubernetes cluster ready. You can use `minikube` for local development or a cloud provider's Kubernetes service for production environments.

- **Helm** - Install Helm, a package manager for Kubernetes, to help you manage Kubernetes applications, you can check our documentation to see how to do that <https://milvus.io/docs/install_cluster-helm.md>

- **Kubectl** - Install `kubectl`, a command-line tool for interacting with Kubernetes clusters, to deploy applications, inspect and manage cluster resources, and view logs.

### Setting Up K8s

After installing everything needed to run a K8s cluster, and if you used `minikube`, start your cluster with: 

```
minikube start
```

Check the status of your K8s cluster with:

```
kubectl cluster-info
```

### Deploying Milvus on K8s

For this deployment, we're opting for Milvus in cluster-mode to leverage its full distributed capabilities. We'll be using Helm, to streamline the installation process.

**1. Helm Installation Command** 

```
helm install my-milvus milvus/milvus --set pulsar.enabled=false --set kafka.enabled=true
```

This command installs Milvus on your K8s cluster with Kafka enabled and Pulsar disabled. Kafka serves as the messaging system within Milvus, handling data streaming between different components. Disabling Pulsar and enabling Kafka tailors the deployment to our specific messaging preferences and requirements.

**2. Port Forwarding**

To access Milvus from your local machine, create a port forward: `kubectl port-forward svc/my-milvus 27017:19530`.

This command maps port `19530` from the Milvus service `svc/my-milvus` to the same port on your local machine, allowing you to connect to Milvus using local tools. If you leave the local port unspecified (as in `:19530`), K8s will allocate an available port, making it dynamic. Ensure you note the allocated local port if you choose this method.

**3. Verifying the Deployment:**

```
kubectl get pods 

NAME                                    READY   STATUS    RESTARTS   AGE
my-milvus-datacoord-595b996bd4-zprpd    1/1     Running   0          85m
my-milvus-datanode-d9d555785-47nkt      1/1     Running   0          85m
my-milvus-etcd-0                        1/1     Running   0          84m
my-milvus-etcd-1                        1/1     Running   0          85m
my-milvus-etcd-2                        1/1     Running   0          85m
my-milvus-indexcoord-65bc68968c-6jg6q   1/1     Running   0          85m
my-milvus-indexnode-54586f55d-z9vx4     1/1     Running   0          85m
my-milvus-kafka-0                       1/1     Running   0          85m
my-milvus-kafka-1                       1/1     Running   0          85m
my-milvus-kafka-2                       1/1     Running   0          85m
my-milvus-minio-0                       1/1     Running   0          96m
my-milvus-minio-1                       1/1     Running   0          96m
my-milvus-minio-2                       1/1     Running   0          96m
my-milvus-minio-3                       1/1     Running   0          96m
my-milvus-proxy-76bb7d497f-sqwvd        1/1     Running   0          85m
my-milvus-querycoord-6f4c7b7598-b6twj   1/1     Running   0          85m
my-milvus-querynode-677bdf485b-ktc6m    1/1     Running   0          85m
my-milvus-rootcoord-7498fddfd8-v5zw8    1/1     Running   0          85m
my-milvus-zookeeper-0                   1/1     Running   0          85m
my-milvus-zookeeper-1                   1/1     Running   0          85m
my-milvus-zookeeper-2                   1/1     Running   0          85m
```


You should see a list of pods similar to the output above, all in the Running state. This indicates that your Milvus cluster is operational. Specifically, look for the 1/1 under the `READY` column, which signifies that each pod is fully ready and running. If any pods are not in the Running state, you may need to investigate further to ensure a successful deployment.

With your Milvus cluster deployed and all components confirmed running, you're now ready to proceed to data ingestion and indexing. This will involve connecting to your Milvus instance, creating collections, and inserting vectors for search and retrieval.


## Data Ingestion and Indexing

To start ingesting and indexing data in our Milvus cluster, we'll use the pymilvus SDK. There are two installation options:

- Basic SDK: `pip install pymilvus`

- For rich text embeddings and advanced models: `pip install pymilvus[model]`

Time to insert data in our cluster, we’ll be using `pymilvus`, you can either install the SDK only with `pip install pymilvus` or if you want to extract rich text embeddings, you can also use `PyMilvus Models` by installing `pip install pymilvus[model]`. 

### Connecting and Creating a Collection:

First, connect to your Milvus instance using the port you forwarded earlier. Ensure the URI matches the local port assigned by K8s:

```
from pymilvus import MilvusClient

client = MilvusClient(
        uri="http://127.0.0.1:52070",
    )

client.create_collection(collection_name="quick_setup", dimension=5)
```

The `dimension=5` parameter defines the vector size for this collection, essential for the vector search capabilities.


### Insert Data

Here's how to insert an initial set of data, where each vector represents an item, and the color field adds a descriptive attribute:

```
data=[
    {"id": 0, "vector": [0.3580376395471989, -0.6023495712049978, 0.18414012509913835, -0.26286205330961354, 0.9029438446296592], "color": "pink_8682"},
    {"id": 1, "vector": [0.19886812562848388, 0.06023560599112088, 0.6976963061752597, 0.2614474506242501, 0.838729485096104], "color": "red_7025"},
    {"id": 2, "vector": [0.43742130801983836, -0.5597502546264526, 0.6457887650909682, 0.7894058910881185, 0.20785793220625592], "color": "orange_6781"},
    {"id": 3, "vector": [0.3172005263489739, 0.9719044792798428, -0.36981146090600725, -0.4860894583077995, 0.95791889146345], "color": "pink_9298"},
    {"id": 4, "vector": [0.4452349528804562, -0.8757026943054742, 0.8220779437047674, 0.46406290649483184, 0.30337481143159106], "color": "red_4794"},
    {"id": 5, "vector": [0.985825131989184, -0.8144651566660419, 0.6299267002202009, 0.1206906911183383, -0.1446277761879955], "color": "yellow_4222"},
    {"id": 6, "vector": [0.8371977790571115, -0.015764369584852833, -0.31062937026679327, -0.562666951622192, -0.8984947637863987], "color": "red_9392"},
    {"id": 7, "vector": [-0.33445148015177995, -0.2567135004164067, 0.8987539745369246, 0.9402995886420709, 0.5378064918413052], "color": "grey_8510"},
    {"id": 8, "vector": [0.39524717779832685, 0.4000257286739164, -0.5890507376891594, -0.8650502298996872, -0.6140360785406336], "color": "white_9381"},
    {"id": 9, "vector": [0.5718280481994695, 0.24070317428066512, -0.3737913482606834, -0.06726932177492717, -0.6980531615588608], "color": "purple_4976"}
]

res = client.insert(
    collection_name="quick_setup",
    data=data
)

print(res)
```


The provided code assumes that you have created a collection in the Quick Setup manner. As shown in the above code,

The data to insert is organized into a list of dictionaries, where each dictionary represents a data record, termed as an entity.

Each dictionary contains a non-schema-defined field named color.

Each dictionary contains the keys corresponding to both pre-defined and dynamic fields.


### Insert Even More Data

```
colors = ["green", "blue", "yellow", "red", "black", "white", "purple", "pink", "orange", "brown", "grey"]
data = [ {
    "id": i, 
    "vector": [ random.uniform(-1, 1) for _ in range(5) ], 
    "color": f"{random.choice(colors)}_{str(random.randint(1000, 9999))}" 
} for i in range(1000) ]

res = client.insert(
    collection_name="quick_setup",
    data=data[10:]
)

print(res)
```


## Similarity Search

After populating the collection, you can perform a similarity search to find vectors close to a query vector. The value of the query_vectors variable is a list containing a sub-list of floats. The sub-list represents a vector embedding of 5 dimensions.

```
query_vectors = [
    [0.041732933, 0.013779674, -0.027564144, -0.013061441, 0.009748648]
]

res = client.search(
    collection_name="quick_setup",     # target collection
    data=query_vectors,                # query vectors
    limit=3,                           # number of returned entities
)

print(res)
```


This query searches for the top 3 vectors most similar to our query vector, demonstrating Milvus's powerful search capabilities.


## Uninstall Milvus from K8s

Once you are done with this tutorial, feel free to uninstall Milvus from your K8s cluster with:`helm uninstall my-milvus`.

This command will remove all Milvus components deployed in the `my-milvus` release, freeing up cluster resources.

## Conclusion

- Deploying Milvus on a Kubernetes cluster showcases the scalability and flexibility of vector databases in handling AI and machine learning workloads. Through this tutorial, you've learned the basics of setting up Milvus with Helm, creating a collection, and performing data ingestion and similarity searches.

- Installing Milvus on a Kubernetes cluster with Helm should be straightforward. To go deeper into scaling Milvus clusters for larger datasets or more intensive workloads, our documentation offers detailed guidance <https://milvus.io/docs/scaleout.md>

Feel free to check out the code on [Github](https://github.com/stephen37/K8s-tutorial-milvus), check out [Milvus](https://github.com/milvus-io/milvus), experiment with different configurations and use cases, and share your experiences with the community by joining our [Discord](https://discord.gg/FG6hMJStWu).


Getting started with Milvus cluster and K8s


We are happy to announce the launch of Milvus 2.4, a major advancement in enhancing search capabilities for large-scale datasets. This latest release adds new features, such as support for the GPU-based CAGRA index, beta support for [sparse embeddings](https://zilliz.com/learn/sparse-and-dense-embeddings), group search, and various other improvements in search capabilities. These developments reinforce our commitment to the community by offering developers like you a powerful and efficient tool for handling and querying vector data. Let's jump into the key benefits of Milvus 2.4 together.


## Enabled Multi-vector Search for Simplified Multimodal Searches

Milvus 2.4 provides multivector search capability, allowing simultaneous search and reranking of different vector types within the same Milvus system. This feature streamlines multimodal searches, significantly enhancing recall rates and enabling developers to effortlessly manage intricate AI applications with varied data types. Additionally, this functionality simplifies the integration and fine-tuning of custom reranking models, aiding in the creation of advanced search functions like precise [recommender systems](https://zilliz.com/vector-database-use-cases/recommender-system) that utilize insights from multidimensional data.

![How the Milti-Vector Search Feature Works](https://assets.zilliz.com/How_the_multi_vector_search_feature_works_6c85961349.png)

Multivector support in Milvus has two components: 

1. The ability to store/query multiple vectors for a single entity within a collection, which is a more natural way to organize data 

2. The ability to build/optimize a reranking algorithm by leveraging the prebuilt reranking algorithms in Milvus

Besides being a highly [requested feature](https://github.com/milvus-io/milvus/issues/25639), we built this capability because the industry is moving towards multimodal models with the release of GPT-4 and Claude 3. Reranking is a commonly used technique to further improve query performance in search. We aimed to make it easy for developers to build and optimize their rerankers within the Milvus ecosystem.


## Grouping Search Support for Enhanced Compute Efficiency

Grouping Search is another often [requested feature](https://github.com/milvus-io/milvus/issues/25343) we added to Milvus 2.4. It integrates a group-by operation designed for fields of types BOOL, INT, or VARCHAR, filling a crucial efficiency gap in executing large-scale grouping queries. 

Traditionally, developers relied on extensive Top-K searches followed by manual post-processing to distill group-specific results, a compute-intensive and code-heavy method. Grouping Search refines this process by efficiently linking query outcomes to aggregate group identifiers like document or video names, streamlining the handling of segmented entities within larger datasets.

Milvus distinguishes its Grouping Search with an iterator-based implementation, offering a marked improvement in computational efficiency over similar technologies. This choice ensures superior performance scalability, particularly in production environments where compute resource optimization is paramount. By reducing data traversal and computation overhead, Milvus supports more efficient query processing, significantly reducing response times and operational costs compared to other vector databases. 

Grouping Search bolsters Milvus's capability to manage high-volume, complex queries and aligns with high-performance computing practices for robust data management solutions.

## Beta Support for Sparse Vector Embeddings

[Sparse embeddings](https://zilliz.com/learn/sparse-and-dense-embeddings) represent a paradigm shift from traditional dense vector approaches, catering to the nuances of semantic similarity rather than mere keyword frequency. This distinction allows for a more nuanced search capability, aligning closely with the semantic content of the query and the documents. Sparse vector models, particularly useful in information retrieval and natural language processing, offer powerful out-of-domain search capabilities and interpretability compared to their dense counterparts.

In Milvus 2.4, we have expanded the Hybrid Search to include sparse embeddings generated by advanced neural models like SPLADEv2 or statistical models such as BM25. In Milvus, sparse vectors are treated on par with dense vectors, enabling the creation of collections with sparse vector fields, data insertion, index building, and performing similarity searches. Notably, sparse embeddings in Milvus support the [Inner Product](https://zilliz.com/blog/similarity-metrics-for-vector-search#Inner-Product) (IP) distance metric, which is advantageous given their high-dimensional nature, making other metrics less effective. This functionality also supports data types with a dimension as an unsigned 32-bit integer and a 32-bit float for the value, thus facilitating a broad spectrum of applications, from nuanced text searches to elaborate [information retrieval](https://zilliz.com/learn/information-retrieval-metrics) systems.

With this new feature, Milvus allows for hybrid search methodologies that meld keyword and embedding-based techniques, offering a seamless transition for users moving from keyword-centric search frameworks seeking a comprehensive, low-maintenance solution.

We are labeling this feature as “Beta” to continue our performance testing of the feature and gather feedback from the community. The general availability (GA) of sparse vector support is anticipated with the release of Milvus 3.0.


## CAGRA Index Support for Advanced GPU-Accelerated Graph Indexing

Developed by NVIDIA, [CAGRA](https://arxiv.org/abs/2308.15136) (Cuda Anns GRAph-based) is a GPU-based graph indexing technology that significantly surpasses traditional CPU-based methods like the HNSW index in efficiency and performance, especially in high-throughput environments.

With the introduction of the CAGRA Index, Milvus 2.4 provides enhanced GPU-accelerated graph indexing capability. This enhancement is ideal for building similarity search applications requiring minimal latency. Additionally, Milvus 2.4 integrates a brute-force search with the CAGRA index to achieve maximum recall rates in applications. For detailed insights, explore the [introduction blog on CAGRA](https://zilliz.com/blog/Milvus-introduces-GPU-index-CAGRA).

![Milvus Raft CAGRA vs. Milvus HNSW](https://assets.zilliz.com/Milvus_raft_cagra_vs_milvus_hnsw_ffe0415ff5.png)

## Additional Enhancements and Features

Milvus 2.4 also includes other key enhancements, such as Regular Expression support for enhanced substring matching in [metadata filtering](https://zilliz.com/blog/metadata-filtering-with-zilliz-cloud-pipelines), a new scalar inverted index for efficient scalar data type filtering, and a Change Data Capture tool for monitoring and replicating changes in Milvus collections. These updates collectively enhance Milvus's performance and versatility, making it a comprehensive solution for complex data operations.

For more details, see [Milvus 2.4 documentation](https://milvus.io/docs/release_notes.md). 


## Stay Connected!

Excited to learn more about Milvus 2.4? [Join our upcoming webinar](https://zilliz.com/event/unlocking-advanced-search-capabilities-milvus) with James Luan, Zilliz’s VP of Engineering, for an in-depth discussion on the capabilities of this latest release. If you have questions or feedback, join our [Discord channel](https://discord.com/invite/8uyFbECzPX) to engage with our engineers and community members. Don’t forget to follow us on [Twitter](https://twitter.com/milvusio) or [LinkedIn](https://www.linkedin.com/company/the-milvus-project) for the latest news and updates about Milvus.


Milvus 2.4 Supports Multi-vector Search, Sparse Vector, CAGRA, and More!

Unveiling Milvus 2.4: Multi-vector Search, Sparse Vector, CAGRA Index, and More!


[Milvus](https://zilliz.com/what-is-milvus), the highly scalable vector database renowned for its ability to handle vast datasets, takes a significant step forward by introducing Parquet file support in [version 2.3.4](https://zilliz.com/blog/what-is-new-in-milvus-2-3-4). By embracing Apache Parquet, users can streamline their data import processes and enjoy substantial savings in storage and computation costs.

In our latest post, we explore Parquet's advantages and the benefits it brings to Milvus users. We discuss the motivation behind integrating this feature and provide a step-by-step guide on seamlessly importing Parquet files into Milvus, unlocking new possibilities for efficient data management and analysis.

## What Is Apache Parquet? 

[Apache Parquet](https://parquet.apache.org/) is a popular open-source column-oriented data file format designed to enhance the efficiency of storing and processing large-scale datasets. In contrast to traditional row-oriented data formats like CSV or JSON, Parquet stores data by column, offering more efficient data compression and encoding schemes. This approach translates to improved performance, reduced storage requirements, and enhanced processing power, making it ideal for handling complex data in bulk. 

## How Milvus Users Benefit from the Support for Parquet File Imports

Milvus extends support for Parquet file imports, providing users with optimized experiences and various advantages, including lowered storage and computation expenses, streamlined data management, and a simplified importing process.

### Optimized Storage Efficiency and Streamlined Data Management 

Parquet provides flexible compression options and efficient encoding schemes catering to different data types, ensuring optimal storage efficiency. This flexibility is particularly valuable in cloud environments where every ounce of storage savings directly correlates to tangible cost reductions. With this new feature in Milvus, users can effortlessly consolidate all their diverse data into a single file, streamlining data management and enhancing the overall user experience. This feature is particularly beneficial for users working with variable-length Array data types, who can now enjoy a simplified data import process. 

### Improved Query Performance 

Parquet's columnar storage design and advanced compression methods significantly enhance query performance. When conducting queries, users can focus solely on the pertinent data without scanning through the irrelevant data. This selective column reading minimizes CPU usage, resulting in faster query times. 

### Broad Language Compatibility

Parquet is available in multiple languages such as Java, C++, and Python and is compatible with a large number of data processing tools. With the support of Parquet files, Milvus users using different SDKs can seamlessly generate Parquet files for parsing within the database. 

## How to Import Parquet Files into Milvus 

If your data is already in Parquet file format, importing is easy. Upload the Parquet file to an object storage system such as MinIO, and you're ready to import. 

The code snippet below is an example of importing Parquet files into Milvus.


```
remote_files = []
try:
    print("Prepare upload files")
    minio_client = Minio(endpoint=MINIO_ADDRESS, access_key=MINIO_ACCESS_KEY, secret_key=MINIO_SECRET_KEY,
                         secure=False)
    found = minio_client.bucket_exists(bucket_name)
    if not found:
        minio_client.make_bucket(bucket_name)
        print("MinIO bucket '{}' doesn't exist".format(bucket_name))
        return False, []

    # set your remote data path
    remote_data_path = "milvus_bulkinsert"

    def upload_file(f: str):
        file_name = os.path.basename(f)
        minio_file_path = os.path.join(remote_data_path, "parquet", file_name)
        minio_client.fput_object(bucket_name, minio_file_path, f)
        print("Upload file '{}' to '{}'".format(f, minio_file_path))
        remote_files.append(minio_file_path)

    upload_file(data_file)

except S3Error as e:
    print("Failed to connect MinIO server {}, error: {}".format(MINIO_ADDRESS, e))
    return False, []

print("Successfully upload files: {}".format(remote_files))
return True, remote_files
```

If your data is not Parquet files or has dynamic fields, you can leverage BulkWriter, our data format conversion tool, to help you generate Parquet files. BulkWriter has now embraced Parquet as its default output data format, ensuring a more intuitive experience for developers. 

The code snippet below is an example of using BulkWriter to generate Parquet files. 

```
import numpy as np
import json

from pymilvus import (
    RemoteBulkWriter,
    BulkFileType,
)

remote_writer = RemoteBulkWriter(
        schema=your_collection_schema,
        remote_path="your_remote_data_path",
        connect_param=RemoteBulkWriter.ConnectParam(
            endpoint=YOUR_MINIO_ADDRESS,
            access_key=YOUR_MINIO_ACCESS_KEY,
            secret_key=YOUR_MINIO_SECRET_KEY,
            bucket_name="a-bucket",
        ),
        file_type=BulkFileType.PARQUET,
)

# append your data
batch_count = 10000
for i in range(batch_count):
    row = {
        "id": i,
        "bool": True if i % 5 == 0 else False,
        "int8": i % 128,
        "int16": i % 1000,
        "int32": i % 100000,
        "int64": i,
        "float": i / 3,
        "double": i / 7,
        "varchar": f"varchar_{i}",
        "json": {"dummy": i, "ok": f"name_{i}"},
        "vector": gen_binary_vector() if bin_vec else gen_float_vector(),
        f"dynamic_{i}": i,
    }
    remote_writer.append_row(row)

# append rows by numpy type
for i in range(batch_count):
    remote_writer.append_row({
        "id": np.int64(i + batch_count),
        "bool": True if i % 3 == 0 else False,
        "int8": np.int8(i % 128),
        "int16": np.int16(i % 1000),
        "int32": np.int32(i % 100000),
        "int64": np.int64(i),
        "float": np.float32(i / 3),
        "double": np.float64(i / 7),
        "varchar": f"varchar_{i}",
        "json": json.dumps({"dummy": i, "ok": f"name_{i}"}),
        "vector": gen_binary_vector() if bin_vec else gen_float_vector(),
        f"dynamic_{i}": i,
    })

print(f"{remote_writer.total_row_count} rows appends")
print(f"{remote_writer.buffer_row_count} rows in buffer not flushed")
print("Generate data files...")
remote_writer.commit()
print(f"Data files have been uploaded: {remote_writer.batch_files}")
remote_files = remote_writer.batch_files
```

Then, you can start to import your Parquet files into Milvus. 

```
remote_files = [remote_file_path]
task_id = utility.do_bulk_insert(collection_name=collection_name,
                                 files=remote_files)

task_ids = [task_id]         
states = wait_tasks_to_state(task_ids, BulkInsertState.ImportCompleted)
complete_count = 0
for state in states:
    if state.state == BulkInsertState.ImportCompleted:
        complete_count = complete_count + 1
 ```

Now, your data is seamlessly integrated into Milvus.

## What's Next?

As Milvus continues to support ever-growing data volumes, the challenge arises in managing sizable imports, particularly when Parquet files surpass 10GB. To tackle this challenge, we plan to segregate the import data into scalar and vector columns, creating two Parquet files per import to alleviate the I/O pressure. For datasets exceeding several hundred gigabytes, we recommend importing the data multiple times. 



Milvus Supports Imports of Apache Parquet Files

Milvus Supports Imports of Apache Parquet Files for Enhanced Data Processing Efficiency


# Introduction to the Milvus versions

Selecting the appropriate Milvus version is foremost to the success of any project leveraging vector search technology. With different Milvus versions tailored to varying requirements, understanding the importance of selecting the correct version is crucial for achieving the desired results.

The right Milvus version can help a developer to learn and prototype quickly or help optimize resource utilization, streamline development efforts, and ensure compatibility with existing infrastructure and tools. Ultimately, it is about maintaining developer productivity and improving efficiency, reliability, and user satisfaction.


## Available Milvus versions 

Three versions of Milvus are available for developers, and all are open source. The three versions are Milvus Lite, Milvus Standalone, and Milvus Cluster, which differ in features and how users plan to use Milvus in the short and long term. So, let's explore these individually.


## Milvus Lite

As the name suggests, Milvus Lite is a lightweight version that integrates seamlessly with Google Colab and Jupyter Notebook. It is packaged as a single binary with no additional dependencies, making it easy to install and run on your machine or embed in Python applications. Additionally, Milvus Lite includes a CLI-based Milvus standalone server, providing flexibility for running it directly on your machine. Whether you embed it within your Python code or utilize it as a standalone server is entirely up to your preference and specific application requirements.


### Features and Capabilities

Milvus Lite includes all core Milvus vector search features. 

- **Search Capabilities**: Supports top-k, range, and hybrid searches, including metadata filtering, to cater to diverse search requirements.

- **Index Types and Similarity Metrics**: Offers support for 11 index types and five similarity metrics, providing flexibility and customization options for your specific use case.

- **Data Processing**: Enables batch (Apache Parquet, Arrays, JSON) and stream processing, with seamless integration through connectors for Airbyte, Apache Kafka, and Apache Spark.

- **CRUD Operations**: Offers full CRUD support (create, read, update/upsert, delete), empowering users with comprehensive data management capabilities.


### Applications and limitations

Milvus Lite is ideal for rapid prototyping and local development, offering support for quick setup and experimentation with small-scale datasets on your machine. However, its limitations become apparent when transitioning to production environments with larger datasets and more demanding infrastructure requirements. As such, while Milvus Lite is an excellent tool for initial exploration and testing, it may not be suitable for deploying applications in high-volume or production-ready settings.


### Available Resources

- [Documentation](https://milvus.io/docs/milvus_lite.md)

- [Github Repository](https://github.com/milvus-io/milvus-lite/)

- [Google Colab Example](https://github.com/milvus-io/milvus-lite/tree/main/examples)

- [Getting Started Video](https://www.youtube.com/watch?v=IgJdrGiB5ZY)


## Milvus Standalone

Milvus offers two operational modes: Standalone and Cluster. Both modes are identical in core vector database features and differ in data size support and scalability requirements. This distinction allows you to select the mode that best aligns with your dataset size, traffic volume, and other infrastructure requirements for production. 

Milvus Standalone is a mode of operation for the Milvus vector database system where it operates independently as a single instance without any clustering or distributed setup. Milvus runs on a single server or machine in this mode, providing functionalities such as indexing and searching for vectors. It is suitable for situations where the data and traffic volume scale is relatively small and does not require the distributed capabilities provided by a clustered setup.


### Features and Capabilities

- **High Performance**: Conduct vector searches on massive datasets (billions or more) with exceptional speed and efficiency.

- **Search Capabilities**: Supports top-k, range, and hybrid searches, including metadata filtering, to cater to diverse search requirements.

- **Index Types and Similarity Metrics**: Offers support for 11 index types and 5 similarity metrics, providing flexibility and customization options for your specific use case.

- **Data Processing**: Enables both batch (Apache Parquet, Arrays, Json) and stream processing, with seamless integration through connectors for Airbyte, Apache Kafka, and Apache Spark.

- **Scalability**: Achieve dynamic scalability with component-level scaling, allowing for seamless scaling up and down based on demand. Milvus can autoscale at a component level, optimizing resource allocation for enhanced efficiency.

- **Multi-Tenancy**: Supports multi-tenancy with the capability to manage up to 10,000 collections/partitions in a cluster, providing efficient resource utilization and isolation for different users or applications.

- **CRUD Operations**: Offers full CRUD support (create, read, update/upsert, delete), empowering users with comprehensive data management capabilities.


### Essential components:

- Milvus: The core functional component.

- etcd: The metadata engine responsible for accessing and storing metadata from Milvus' internal components, including proxies, index nodes, and more.

- MinIO: The storage engine responsible for data persistence within Milvus.

![](https://assets.zilliz.com/Screenshot_2024_02_19_at_4_16_41_PM_5e635586a7.png)

Figure 1: Milvus Standalone Architecture


### Available Resources

- Documentation

  - [Environment Checklist for Milvus with Docker Compose](https://milvus.io/docs/prerequisite-docker.md)

  - [Install Milvus Standalone with Docker](https://milvus.io/docs/install_standalone-docker.md)

- [Github Repository](https://github.com/milvus-io/milvus)


## Milvus Cluster

Milvus Cluster is a mode of operation for the Milvus vector database system where it operates and is distributed across multiple nodes or servers. In this mode, Milvus instances are clustered together to form a unified system that can handle larger volumes of data and higher traffic loads compared to a standalone setup. Milvus Cluster offers scalability, fault tolerance, and load balancing features, making it suitable for scenarios that need to handle big data and serve many concurrent queries efficiently.


### Features and Capabilities

- Inherits all features available in Milvus Standalone, including high-performance vector search, support for multiple index types and similarity metrics, and seamless integration with batch and stream processing frameworks.

- Offers unparalleled availability, performance, and cost optimization by leveraging distributed computing and load balancing across multiple nodes.

- Enables deploying and scaling secure, enterprise-grade workloads with lower total costs by efficiently utilizing resources across the cluster and optimizing resource allocation based on workload demands.


### Essential components:

Milvus Cluster includes eight microservice components and three third-party dependencies. All microservices can be deployed on Kubernetes independently from each other.


#### Microservice components

- Root coord

- Proxy

- Query coord

- Query node

- Index coord

- Index node

- Data coord

- Data node


#### Third-party dependencies

- etcd: Stores metadata for various components in the cluster.

- MinIO: Responsible for data persistence of large files in the cluster, such as index and binary log files.

- Pulsar: Manages logs of recent mutation operations, outputs streaming log, and provides log publish-subscribe services.

![](https://assets.zilliz.com/Screenshot_2024_02_19_at_4_18_01_PM_88971280ed.png)

Figure 2: Milvus Cluster Architecture


#### Available Resources

- [Documentation](https://milvus.io/docs/install_cluster-milvusoperator.md) | How to get started

  - [Install Milvus Cluster with Milvus Operator](https://milvus.io/docs/install_cluster-milvusoperator.md)

  - [Install Milvus Cluster with Helm](https://milvus.io/docs/install_cluster-helm.md)

  - [How to scale a Milvus Cluster](https://milvus.io/docs/scaleout.md)

- [Github Repository](https://github.com/milvus-io/milvus)


## Making the Decision on which Milvus version to use

When deciding which version of Milvus to use for your project, you must consider factors such as your dataset size, traffic volume, scalability requirements, and production environment constraints. Milvus Lite is perfect for prototyping on your laptop. Milvus Standalone offers high performance and flexibility for conducting vector searches on your datasets, making it suitable for smaller-scale deployments, CI/CD, and offline deployments when you have no Kubernetes support.. And finally, Milvus Cluster provides unparalleled availability, scalability, and cost optimization for enterprise-grade workloads, making it the preferred choice for large-scale, highly available production environments.

There is another version that is a hassle-free version, and that is a managed version of Milvus called [Zilliz Cloud](https://cloud.zilliz.com/signup). 

Ultimately, the Milvus version will depend on your specific use case, infrastructure requirements, and long-term goals. By carefully evaluating these factors and understanding the features and capabilities of each version, you can make an informed decision that aligns with your project's needs and objectives. Whether you choose Milvus Standalone or Milvus Cluster, you can leverage the power of vector databases to enhance the performance and efficiency of your AI applications.


What Milvus version to start with


As we conclude 2023, let's review the Milvus community's remarkable journey: boasting [25,000 GitHub Stars](https://github.com/milvus-io/milvus), the launch of [Milvus 2.3.0](https://milvus.io/blog/unveiling-milvus-2-3-milestone-release-offering-support-for-gpu-arm64-cdc-and-other-features.md), and exceeding 10 million [Docker image](https://hub.docker.com/r/milvusdb/milvus) downloads. This post explores the heart of the community by analyzing chat histories and revealing the top 10 keywords in discussions.


<iframe class="video-player" src="https://www.youtube.com/embed/o5uMdNLioQ0?list=PLPg7_faNDlT5Fb8WN8r1PzzQTNzdechnS" title="Mastering Milvus: Turbocharge Your Vector Database with Optimization Secrets!" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>

## #1 Version — The rise of AIGC drives rapid Milvus iteration

Surprisingly, "Version" emerged as the most discussed keyword in 2023. This revelation is rooted in the year's AI wave, with vector databases as a critical infrastructure to tackle challenges in AIGC applications' hallucination issues.

The enthusiasm around vector databases drives Milvus into a stage of swift iteration. The community witnessed the release of Twenty versions in 2023 alone, accommodating the demands of AIGC developers flooding the community with inquiries about choosing the optimal version of Milvus for various applications. For users navigating these updates, we recommend embracing the latest version for enhanced features and performance. 

If you are interested in Milvus's release planning, refer to the [Milvus Roadmap](https://wiki.lfaidata.foundation/display/MIL/Milvus+Long+Term+Roadmap+and+Time+schedule) page on the official website. 


## #2 Search — beyond Vector Search

"Search" takes second place, reflecting its fundamental role in database operations. Milvus supports various search capabilities, from Top-K ANN search to scalar filtered search and range search. The imminent release of Milvus 3.0 (Beta) promises keyword search (sparse embeddings), which many RAG app developers eagerly await.

Community discussions about searching focus on performance, capabilities, and principles. Users often ask questions about attribute filtering, setting index threshold values, and addressing latency concerns. Resources like [query and search documentation](https://milvus.io/docs/v2.0.x/search.md), [Milvus Enhancement Proposals (MEPs)](https://wiki.lfaidata.foundation/pages/viewpage.action?pageId=43287103), and Discord discussions have become the go-to references for unraveling the intricacies of searching within Milvus.


## #3 Memory — trade-offs between performance and accuracy for minimized memory overhead

“Memory” also took center stage in community discussions over the past year. As a distinctive data type, vectors inherently have high dimensions. Storing vectors in memory is a common practice for optimal performance, but the escalating data volume limits available memory. Milvus optimizes memory usage by adopting techniques like [MMap](https://zilliz.com/blog/milvus-introduced-mmap-for-redefined-data-management-increased-storage-capability) and DiskANN.

However, achieving low memory usage, excellent performance, and high accuracy simultaneously in a database system remains complex, necessitating trade-offs between performance and accuracy to minimize memory overhead.

In the case of Artificial Intelligence Generated Content (AIGC), developers usually prioritize fast responses and result accuracy over stringent performance requirements. Milvus’s addition of MMap and DiskANN minimizes memory usage while maximizing data processing and result accuracy, striking a balance that aligns with the practical needs of AIGC applications. 


## #4 Insert — smooth sailing through data insertion

Efficient data insertion is a crucial concern for developers, sparking frequent discussions on optimizing insertion speed within the Milvus community. Milvus excels in the efficient insertion of streaming data and the building of indexes, thanks to its adept separation of streaming and batch data. This capability sets it apart as a highly performant solution compared to other vector database providers, such as Pinecone.

Here are some valuable insights and recommendations about data insertions: 

- **Batch Insertion:** Opt for batch over single-row insertion for enhanced efficiency. Notably, insertion from files surpasses batch insertion in speed. When handling large datasets exceeding ten million records, consider using the `bulk_insert` interface for a streamlined and accelerated import process.

- **Strategic `flush()` Usage:** Rather than invoking the `flush()` interface after each batch, make a single call after completing all data insertion. Excessive use of the `flush()` interface between batches can lead to the generation of fragmented segment files, placing a considerable compaction burden on the system.

- **Primary Key Deduplication:** Milvus does not perform primary key deduplication when using the `insert` interface for data insertion. If you need to deduplicate primary keys, we recommend you deploy the `upsert` interface. However, the insertion performance of `upsert `is lower than that of `insert`, owing to an additional internal query operation. 


## #5 Configuration — decoding the parameter maze

Milvus is a distributed vector database that integrates many third-party components like object storage, message queues, and Etcd. Users grappled with adjusting parameters and understanding their impact on Milvus's performance, making "Configuration" a frequently discussed topic.

Among all the questions about configurations, "which parameters to adjust" is arguably the most challenging aspect, as the parameters vary in different situations. For instance, optimizing search performance parameters differs from optimizing insertion performance parameters and relies heavily on practical experience.

Once users identify "which parameters to adjust," the subsequent questions of "how to adjust" become more manageable. For specific procedures, refer to our documentation [Configure Milvus](https://milvus.io/docs/configure-helm.md). The great news is that Milvus has supported dynamic parameter adjustments since version 2.3.0, eliminating the need for restarts for changes to take effect. For specific procedures, refer to [Configure Milvus on the Fly](https://milvus.io/docs/dynamic_config.md). 


## #6 Logs — navigating the troubleshooting compass

"Logs" serve as the troubleshooter's compass. Users sought guidance in the community on exporting Milvus logs, adjusting log levels, and integrating with systems like Grafana’s Loki. Here are some suggestions about Milvus logs. 

- **How to view and export Milvus logs:** You can easily export Milvus logs with the one-click script [export-milvus-log.sh](https://github.com/milvus-io/milvus/tree/master/deployments/export-log) which is available on the GitHub repository. 

- **Log level:** Milvus has multiple log levels to accommodate diverse use cases. The info level is enough for most cases, and the debug level is for debugging. An excess of Milvus logs may signal misconfigured log levels.

- **We recommend integrating Milvus logs with a log collection system** like Loki for streamlined log retrieval in future troubleshooting.


## #7 Cluster — scaling for production environments

Given Milvus's identity as a distributed vector database, the term "cluster" is a frequent topic of discussion in the community. Conversations revolve around scaling data in a cluster, data migration, and data backup and synchronization. 

In production environments, robust scalability and high availability are standard requirements for distributed database systems. Milvus's storage-computation separation architecture allows seamless data scalability by expanding resources for computation and storage nodes, accommodating limitless data scales. Milvus also provides high availability with a multi-replica architecture and robust backup and syncing capabilities.  For more information, refer to [Coordinator HA](https://milvus.io/docs/coordinator_ha.md#Coordinator-HA).


## #8 Documentation — the gateway to understanding Milvus

"Documentation" is another frequently raised keyword in community discussions, often tied to questions about whether there is any documentation page for a specific feature and where to find it. 

Serving as the gateway to understanding Milvus, around 80% of community inquiries find answers in the [official documentation](https://milvus.io/docs). We recommend you read our documentation before using Milvus or encountering any problems. In addition, you can explore code examples in various SDK repositories for insights into using Milvus. 


## #9 Deployment — simplifying the Milvus journey

Simple deployment remains the Milvus team’s ongoing goal. To fulfill this commitment, we introduced [Milvus Lite](https://milvus.io/docs/milvus_lite.md#Get-Started-with-Milvus-Lite), a lightweight alternative to Milvus that is fully functional but has no K8s or Docker dependencies. 

We further streamlined deployment by introducing the lighter [NATS](https://zilliz.com/blog/optimizing-data-communication-milvus-embraces-nats-messaging) messaging solution and consolidating node components. Responding to user feedback, we're gearing up to release a standalone version without dependencies, with ongoing efforts to enhance features and simplify deployment operations. The rapid iteration of Milvus showcases the community's continuous commitment to the continued refinement of the deployment process.


## #10 Deletion — unraveling the impact

The prevalent discussions on "deletion" revolve around unchanged data counts after deletion, the continued retrievability of deleted data, and the failure of disk space recovery after deletion.

Milvus 2.3 introduces the `count(*)` expression to address delayed entity count updates. The persistence of deleted data in queries is probably due to the inappropriate use of [data consistency models](https://zilliz.com/blog/understand-consistency-models-for-vector-databases). Disk space recovery failure concerns prompt insights into redesigning Milvus's garbage collection mechanism, which sets a waiting period before the complete deletion of data. This approach allows a time window for potential recovery.


## Conclusion

The top 10 keywords offer a glimpse into the vibrant discussions within the Milvus community. As Milvus continues to evolve, the community remains an invaluable resource for developers seeking solutions, sharing experiences, and contributing to advancing vector databases in the era of AI. 

Join this exciting journey by joining our [Discord channel](https://discord.com/invite/8uyFbECzPX) in 2024. There, you can engage with our brilliant engineers and connect with like-minded Milvus enthusiasts. Also, attend the [Milvus Community Lunch and Learn](https://discord.com/invite/RjNbk8RR4f) every Tuesday from 12:00 to 12:30 PM PST. Share your thoughts, questions, and feedback, as every contribution adds to the collaborative spirit propelling Milvus forward. Your active participation is not just welcomed; it's appreciated. Let's innovate together!


Top 10 Keywords Dominating the Milvus Community in 2023

Unveiling the Top 10 Keywords Dominating the Milvus Community in 2023


![](https://assets.zilliz.com/What_is_new_in_Milvus_2_3_4_1847b0fa8a.png)


We are excited to unveil the latest release of Milvus 2.3.4. This update introduces a suite of features and enhancements meticulously crafted to optimize performance, boost efficiency, and deliver a seamless user experience. In this blog post, we'll delve into the highlights of Milvus 2.3.4. 

## Access logs for improved monitoring

Milvus now supports access logs, offering invaluable insights into interactions with external interfaces. These logs record method names, user requests, response times, error codes, and other interaction information, empowering developers and system administrators to conduct performance analysis, security auditing, and efficient troubleshooting. 

**_Note:_** _Currently, access logs only support gRPC interactions. However, our commitment to improvement continues, and future versions will extend this capability to include RESTful request logs._ 

For more detailed information, refer to [Configure Access Logs](https://milvus.io/docs/configure_access_logs.md).

## Parquet file imports for enhanced data processing efficiency

Milvus 2.3.4 now supports importing Parquet files, a widely embraced columnar storage format designed to enhance the efficiency of storing and processing large-scale datasets. This addition gives users increased flexibility and efficiency in their data processing endeavors. By eliminating the need for laborious data format conversions, users managing substantial datasets in the Parquet format will experience a streamlined data import process, significantly reducing the time from initial data preparation to subsequent vector retrieval.

Furthermore, our data format conversion tool, BulkWriter, has now embraced Parquet as its default output data format, ensuring a more intuitive experience for developers. 

## Binlog index on growing segments for faster searches

Milvus now leverages a binlog index on growing segments, resulting in up to tenfold faster searches in growing segments. This enhancement significantly boosts search efficiency and supports advanced indices like IVF or Fast Scan, improving the overall user experience.

## Support for up to 10,000 collections/partitions

Like tables and partitions in relational databases, collections and partitions are the core units for storing and managing vector data in Milvus. Responding to users' evolving needs for nuanced data organization, Milvus 2.3.4 now supports up to 10,000 collections/partitions in a cluster, a significant jump from the previous limit of 4,096. This enhancement benefits diverse use cases, such as knowledge base management and multi-tenant environments. The expanded support for collections/partitions stems from refinements to the time tick mechanism, goroutine management, and memory usage.

 **_Note:_** _The recommended limit for the number of collections/partitions is 10,000, as exceeding this limit may impact failure recovery and resource usage._

## Other enhancements 

In addition to the features above, Milvus 2.3.4 includes various improvements and bug fixes. These include reduced memory usage during data retrieval and variable-length data handling, refined error messaging, accelerated loading speed, and improved query shard balance. These collective enhancements contribute to a smoother and more efficient overall user experience.

For a comprehensive overview of all the changes introduced in Milvus 2.3.4, refer to our [Release Notes](https://milvus.io/docs/release_notes.md#v234).

## Stay connected!

If you have questions or feedback about Milvus, join our [Discord channel](https://discord.com/invite/8uyFbECzPX) to engage with our engineers and the community directly or join our [Milvus Community Lunch and Learn](https://discord.com/invite/RjNbk8RR4f) Every Tuesday from 12-12:30 PM PST. You’re also welcome to follow us on [Twitter](https://twitter.com/milvusio) or [LinkedIn](https://www.linkedin.com/company/the-milvus-project) for the latest news and updates about Milvus.


Milvus 2.3.4: Faster Searches, Expanded Data Support, Improved Monitoring, and More


![This image is generated by AI. ](https://assets.zilliz.com/image_7_1c3b05e71c.jpg)
# Milvus in 2023: An Unprecedented Vector Database Amidst Tech Buzz


*This post was written by James Luan with the help of ChatGPT. James primarily wrote prompts and reviewed and polished the AI-generated content.*

## 2023: the year of AI 

2023 marks a pivotal turning point in artificial intelligence (AI). [Large Language Models (LLMs)](https://zilliz.com/glossary/large-language-models-$llms$) have taken center stage, garnering widespread recognition for their exceptional natural language processing capabilities. This surge in popularity has substantially expanded the possibilities of machine learning applications, enabling developers to construct more intelligent and interactive applications.


Amidst this revolution, [vector databases](https://zilliz.com/learn/what-is-vector-database) have emerged as a crucial component, acting as the long-term memory for LLMs. The rise of [Retrieval-Augmented Generation (RAG)](https://zilliz.com/use-cases/llm-retrieval-augmented-generation) models, intelligent agents, and multimodal retrieval apps has demonstrated the vast potential of vector databases in enhancing multimodal data retrieval efficiency, reducing hallucinations in LLMs, and supplementing domain knowledge.


The LLM evolution has also catalyzed significant advancements in embedding technologies. According to the [Massive Text Embedding Benchmark (MTEB) Leaderboard](https://huggingface.co/spaces/mteb/leaderboard) on HuggingFace, leading embedding models such as UAE, VoyageAI, CohereV3, and Bge were all released in 2023. These advancements have bolstered the vector retrieval effectiveness of various vector search technologies like Milvus, providing more precise and efficient data processing capabilities for AI applications.


However, with the growing popularity of vector databases, debates arose about the necessity of specialized solutions. Tens of startups have entered the vector database arena. Many traditional relational and NoSQL databases have started treating vectors as a significant data type, and many claim to be capable of substituting specialized vector databases in every situation. 


As we enter 2024, it's an sensible moment to reflect on the entire vector database industry, with a special focus on Milvus—a standout product in this landscape. 




## Milvus in 2023: numbers don't lie


First launched in 2019, [Milvus](https://zilliz.com/what-is-milvus) has pioneered the concept of vector databases and consistently maintained a reputation for high reliability, scalability, search quality, and performance. In 2023, Milvus achieved impressive results and underwent significant shifts, primarily driven by the rapid advancement of LLMs and the boom of AIGC applications. Here are some key figures that best represent Milvus's progress in 2023. 




### ZERO downtime during rolling upgrades


For those new to vector databases, their primary focus centers on functionality rather than operational maintenance. Many application developers also pay less attention to stability in their vector databases than transactional databases since their applications are often in the early stages of exploration. However, stability becomes indispensable if you aim to deploy your AIGC application in a production environment and achieve the best user experience. 


Milvus distinguishes itself by prioritizing not just functionality but also operational stability. We added rolling upgrades to Milvus starting from version 2.2.3. After continuous refinement, this feature can ensure zero downtime during upgrades without interrupting business processes. 




### 3x performance improvement in production environments


Boosting vector search performance needs to be a primary goal for vector databases. Many vector search solutions chose to base their solution on adapting the [HNSW](https://zilliz.com/learn/hierarchical-navigable-small-worlds-HNSW) algorithm to get to market quickly; unfortunately, this leads to them facing significant challenges in real-world production environments, especially with highly filtered searches (over 90%) and frequent data deletions. Milvus considers performance from the get-go and excels in optimizing performance during any phase of development, especially in production environments, achieving a threefold improvement in search performance, especially in filtered search and streaming insert/search situations.


To further assist the vector database community, we introduced [VectorDBBench](https://github.com/zilliztech/VectorDBBench), an open-source benchmarking tool, last year. This tool is vital for early evaluations of vector databases across different conditions. Unlike traditional evaluation methods, VectorDBBench assesses databases using real-world data, including super large datasets or those closely resembling data from actual embedding models, providing users with more insightful information for informed decision-making.




### 5% recall improvement on the Beir dataset


While [dense embeddings](https://zilliz.com/learn/sparse-and-dense-embeddings) have proven effective in vector search, they must catch up when searching for names, objects, abbreviations, and short query contexts. In response to their limitations, Milvus has introduced a hybrid query approach that integrates dense embeddings with [sparse embeddings](https://zilliz.com/learn/sparse-and-dense-embeddings) to enhance the quality of search results. The synergy of this hybrid solution with a reranking model has resulted in a substantial 5% improvement in the recall rate on the Beir dataset, as validated by our tests.


Going beyond improvements in search quality, Milvus has also unveiled a graph-based retrieval solution tailored for sparse embeddings, surpassing the performance of conventional search algorithms like WAND.


At the 2023 NeurIPS BigANN competition, Zihao Wang, a talented engineer at Zilliz, presented [Pyanns](https://big-ann-benchmarks.com/neurips23.html#winners), a search algorithm that demonstrated significant superiority over other entries in the sparse embedding search track. This breakthrough solution is a precursor to our sparse embedding search algorithms for production environments.




### 10x memory saving on large datasets


[Retrieval Augmented Generation](https://zilliz.com/use-cases/llm-retrieval-augmented-generation) (RAG) was the most popular use case for vector databases in 2023. However, the increase in vector data volumes with RAG applications presents a storage challenge for these applications. This challenge is especially true when the volume of transformed vectors exceeds that of the original document chunks, potentially escalating memory usage costs. For example, after dividing documents into chunks, the size of a 1536-dimensional float32 vector (roughly 3kb) transformed from a 500-token chunk (about 1kb) is greater than the 500-token chunk. 


Milvus is the first open-source vector database to support disk-based indexing, bringing about a remarkable 5x memory saving. By the close of 2023, we introduced [Milvus 2.3.4](https://milvus.io/docs/release_notes.md#v234), enabling the capability to load scalar and vector data/indexes onto the disk using memory-mapped files ([MMap](https://zilliz.com/blog/milvus-introduced-mmap-for-redefined-data-management-increased-storage-capability)). This advancement offers more than a 10x reduction in memory usage compared to traditional in-memory indexing. 




### 20 Milvus releases


In 2023, Milvus underwent a transformative journey marked by significant milestones. Over the year, we launched 20 releases, a testament to the dedication of over 300 community developers and the realization of our commitment to a user-driven approach in development.


To illustrate, Milvus 2.2.9 introduced [dynamic schema](https://zilliz.com/blog/what-is-dynamic-schema), marking a crucial shift from prioritizing performance to enhancing usability. Building on this, [Milvus 2.3](https://milvus.io/blog/unveiling-milvus-2-3-milestone-release-offering-support-for-gpu-arm64-cdc-and-other-features.md) introduced critical features such as Upsert, [Range Search](https://zilliz.com/blog/unlock-advanced-recommendation-engines-with-milvus-new-range-search), Cosine metrics, and more, all driven by our user community's specific needs and feedback. This iterative development process underscores our commitment to continually aligning Milvus with the evolving requirements of our users.




### 1,000,000 tenants in a Single Custer 


Implementing multi-tenancy is crucial for developing RAG systems, AI agents, and other LLM applications, meeting the heightened user demands for data isolation. For B2C businesses, tenant numbers can skyrocket into the millions, making physical isolation of user data impractical (as an example, it's unlikely that anyone would create millions of tables in a relational database). Milvus introduced the Partition Key feature, allowing for efficient, logical isolation and data filtering based on partition keys, which is handy at a large scale.


Conversely, B2B enterprises, accustomed to dealing with tens of thousands of tenants, benefit from a more nuanced strategy involving physical resource isolation. The latest Milvus 2.3.4 brings enhanced memory management, coroutine handling, and CPU optimization, making creating tens of thousands of tables within a single cluster easier. This enhancement also accommodates the needs of B2B businesses with enhanced efficiency and control.




### 10,000,000 Docker image pulls


As 2023 drew to a close, Milvus reached an impressive milestone with [10 million Docker pull](https://hub.docker.com/r/milvusdb/milvus) downloads. This accomplishment signals the increasing fascination of the developer community with Milvus and emphasizes its rising significance within the vector database domain. 


As the world's first cloud-native vector database, Milvus boasts seamless integration with Kubernetes and the broader container ecosystem. Gazing into the future, one can't help but ponder the next focal point in the ever-evolving vector database landscape. Could it be the rise of Serverless services?




### 10 billion entities in a single collection


While scalability might not currently steal the spotlight in the AI phenomenon, it certainly plays a pivotal role, far from being a mere sideshow. Milvus vector database can seamlessly scale out to accommodate billions of vector data without breaking a sweat. Take a look at one of our LLM customers, for example. Milvus effortlessly helped this customer store, process, and retrieve an astounding 10 billion data points. But how do you balance the cost and performance when dealing with such a massive volume of data? Rest assured, Mivus has various capabilities to help you address that challenge and elevate your experience. 




## Beyond the numbers: the new insights into vector databases 


Beyond the numerical milestones, 2023 has enriched us with valuable insights. We've delved into the intricacies of the vector database landscape, moving beyond mere statistics to grasp the subtle nuances and evolving dynamics of vector search technology.




### LLM apps are still in the early stages. 


Reflecting on the early days of the mobile internet boom, many developers created simple apps like flashlights or weather forecasts, which eventually were integrated into smartphone operating systems. Last year, most AI Native applications, like AutoGPT, which rapidly hit 100,000 stars on GitHub, didn't deliver practical value but only represented meaningful experiments. For vector database applications, the current use cases may just be the first wave of AI Native transformations, and I eagerly anticipate more killer use cases to emerge.




### Vector databases go toward diversification. 


Similar to the evolution of databases into categories like OLTP, OLAP, and NoSQL, vector databases show a clear trend toward diversification. Departing from the conventional focus on online services, offline analysis has gained significant traction. Another notable instance of this shift is the introduction of [GPTCache](https://zilliz.com/blog/building-llm-apps-100x-faster-responses-drastic-cost-reduction-using-gptcache), an open-sourced semantic cache released in 2023. It enhances the efficiency and speed of GPT-based applications by storing and retrieving responses generated by language models.


We are hopeful and excited to witness even more diversified applications and system designs in vector databases in the coming year.




### Vector operations are becoming more complicated. 


While supporting [Approximate Nearest Neighbor (ANN)](https://zilliz.com/glossary/anns) search is a defining feature of vector databases, it doesn't stand alone. The common belief that merely keeping Nearest Neighbour Search is sufficient to classify a database as a vector or AI native database oversimplifies the intricacies of vector operations. Beyond the basic capabilities of hybrid scalar filtering and vector search, databases tailored for AI native applications should support more sophisticated semantic capabilities like NN Filtering, KNN Join, and cluster querying. 




### Elastic scalability is essential for AI native applications. 


The exponential growth of AI applications, exemplified by ChatGPT amassing over 100 million monthly active users in two months, surpasses any prior business trajectory. Swiftly scaling from 1 million to 1 billion data points becomes paramount once businesses hit their stride in growth. AI application developers benefit from the pay-as-you-go service model set by LLM providers, leading to substantial reductions in operational costs. Similarly, storing data that aligns with this pricing model proves advantageous for developers, allowing them to channel more attention toward core business. 


Unlike Language Models (LLMs) and various other technological systems, vector databases operate in a stateful manner, demanding persistent data storage for their functionality. Consequently, when selecting vector databases, it is crucial to prioritize elasticity and scalability. This prioritization ensures alignment with the dynamic demands of evolving AI applications, highlighting the need for seamless adaptability to changing workloads.




### Leveraging machine learning in vector databases can yield extraordinary results. 


In 2023, our substantial investment in the AI4DB (AI for Database) projects yielded remarkable success. As part of our endeavors, we introduced two pivotal capabilities to [Zilliz Cloud](https://zilliz.com/cloud), the fully managed Milvus solution: 1) AutoIndex, an auto-parameter-tuning index rooted in machine learning, and 2) a data partitioning strategy based on data clustering. Both innovations played a crucial role in significantly enhancing the search performance of Zilliz Cloud.




### Open source vs. closed source


Closed-source LLMs like OpenAI's GPT series and Claude currently take the lead, placing the open-source community disadvantaged due to the absence of comparable computational and data resources.


However, within vector databases, open source will eventually become the favored choice for users. Opting for open source introduces many advantages, including more diverse use cases, expedited iteration, and cultivating a more robust ecosystem. Furthermore, database systems are so intricate that they cannot afford the opacity often associated with LLMs. Users must thoroughly understand the database before choosing the most reasonable approach for its utilization. Moreover, the transparency ingrained in open source empowers users to possess the liberty and the control to customize the database according to their needs.




## Epilogue - And a new beginning!


As 2023 swiftly passes amidst transformative changes, the story of vector databases is just beginning. Our journey with the Milvus vector database is about something other than getting lost in the hype of AIGC. Instead, we focus on meticulously developing our product, identifying and nurturing application use cases that align with our strengths, and unwaveringly serving our users. Our commitment to open source aims to bridge the gap between us and our users, allowing them to sense our dedication and craftsmanship, even from a distance.


2023 also saw many AI startups being founded and getting their first funding rounds. It is exciting to see the innovation from these developers, and it reminds me of why I got into VectorDB development in the first place. 2024 will be a year for all these innovative applications to gain real traction, attracting not just funding but real paying customers. Customer revenue will bring different requirements for these developers, as building a fully scalable solution with little to no downtime is paramount. 


Let's make extraordinary things happen in 2024!


Milvus in 2023: An Unprecedented Vector Database Amidst Tech Buzz

Efficient Vector Similarity Search in Recommender Workflows Using Milvus with NVIDIA Merlin


![](https://assets.zilliz.com/How_to_Get_the_Right_Vector_Embedding_d9ebcacbbb.png)


*This article was originally published in [The New Stack](https://thenewstack.io/how-to-get-the-right-vector-embeddings/) and is reposted here with permission.*

**A comprehensive introduction to vector embeddings and how to generate them with popular open source models.**

![Image by Денис Марчук from Pixabay](https://assets.zilliz.com/how_to_get_right_vector_embeddings_e0838623b7.png)


Vector embeddings are critical when working with  [semantic similarity](https://zilliz.com/blog/vector-similarity-search). However, a vector is simply a series of numbers; a vector embedding is a series of numbers representing input data. Using vector embeddings, we can structure  [unstructured data](https://zilliz.com/blog/introduction-to-unstructured-data)  or work with any type of data by converting it into a series of numbers. This approach allows us to perform mathematical operations on the input data, rather than relying on qualitative comparisons.

Vector embeddings are influential for many tasks, particularly for  [semantic search](https://zilliz.com/glossary/semantic-search). However, it is crucial to obtain the appropriate vector embeddings before using them. For instance, if you use an image model to vectorize text, or vice versa, you will probably get poor results.

In this post, we will learn what vector embeddings mean, how to generate the right vector embeddings for your applications using different models and how to make the best use of vector embeddings with vector databases like  [Milvus](https://milvus.io/)  and  [Zilliz Cloud](https://zilliz.com/).

## How are vector embeddings created?

![](https://assets.zilliz.com/how_vector_embeddings_are_created_03f9b60c68.png)


Now that we understand the importance of vector embeddings, let’s learn how they work. A vector embedding is the internal representation of input data in a deep learning model, also known as embedding models or a deep neural network. So, how do we extract this information?

We obtain vectors by removing the last layer and taking the output from the second-to-last layer. The last layer of a neural network usually outputs the model’s prediction, so we take the output of the second-to-last layer. The vector embedding is the data fed to a neural network’s predictive layer.

The dimensionality of a vector embedding is equivalent to the size of the second-to-last layer in the model and, thus, interchangeable with the vector’s size or length. Common vector dimensionalities include 384 (generated by Sentence Transformers Mini-LM), 768 (by Sentence Transformers MPNet), 1,536 (by OpenAI) and 2,048 (by ResNet-50).


## What does a vector embedding mean?

Someone once asked me about the meaning of each dimension in a vector embedding. The short answer is nothing. A single dimension in a vector embedding does not mean anything, as it is too abstract to determine its meaning. However, when we take all dimensions together, they provide the semantic meaning of the input data.

The dimensions of the vector are high-level, abstract representations of different attributes. The represented attributes depend on the training data and the model itself. Text and image models generate different embeddings because they’re trained for fundamentally different data types. Even different text models generate different embeddings. Sometimes they differ in size; other times, they differ in the attributes they represent. For instance, a model trained on legal data will learn different things than one trained on health-care data. I explored this topic in my post  [comparing vector embeddings](https://zilliz.com/blog/comparing-different-vector-embeddings).

## Generate the right vector embeddings

How do you obtain the proper vector embeddings? It all starts with identifying the type of data you wish to embed. This section covers embedding five different types of data: images, text, audio, videos and multimodal data. All models we introduce here are open source and come from Hugging Face or PyTorch.

### Image embeddings

Image recognition took off in 2012 after AlexNet hit the scene. Since then, the field of computer vision has witnessed numerous advancements. The latest notable image recognition model is ResNet-50, a 50-layer deep residual network based on the former ResNet-34 architecture.

Residual neural networks (ResNet) solve the vanishing gradient problem in deep convolutional neural networks using shortcut connections. These connections allow the output from earlier layers to go to later layers directly without passing through all the intermediate layers, thus avoiding the vanishing gradient problem. This design makes ResNet less complex than VGGNet (Visual Geometry Group), a previously top-performing convolutional neural network.

I recommend two ResNet-50 implementations as examples:  [ResNet 50 on Hugging Face](https://huggingface.co/microsoft/resnet-50)  and  [ResNet 50 on PyTorch Hub](https://pytorch.org/vision/main/models/generated/torchvision.models.resnet50.html). While the networks are the same, the process of obtaining embeddings differs.

The code sample below demonstrates how to use PyTorch to obtain vector embeddings. First, we load the model from PyTorch Hub. Next, we remove the last layer and call  `.eval()`  to instruct the model to behave like it’s running for inference. Then, the  `embed`  function generates the vector embedding.


```
# Load the embedding model with the last layer removed
model = torch.hub.load('pytorch/vision:v0.10.0', 'resnet50', pretrained=True) model = torch.nn.Sequential(*(list(model.children())[:-1]))
model.eval()


def embed(data):
with torch.no_grad():
output = model(torch.stack(data[0])).squeeze()
return output
```


HuggingFace uses a slightly different setup. The code below demonstrates how to obtain a vector embedding from Hugging Face. First, we need a feature extractor and model from the  `transformers`  library. We will use the feature extractor to get inputs for the model and use the model to obtain outputs and extract the last hidden state.


```
# Load model directly
from transformers import AutoFeatureExtractor, AutoModelForImageClassification


extractor = AutoFeatureExtractor.from_pretrained("microsoft/resnet-50")
model = AutoModelForImageClassification.from_pretrained("microsoft/resnet-50")


from PIL import Image


image = Image.open("<image path>")
# image = Resize(size=(256, 256))(image)


inputs = extractor(images=image, return_tensors="pt")
# print(inputs)


outputs = model(**inputs)
vector_embeddings = outputs[1][-1].squeeze()
```


### Text embeddings

Engineers and researchers have been experimenting with natural language and AI since the invention of AI. Some of the earliest experiments include:

-   ELIZA, the first AI therapist chatbot.
-   John Searle’s Chinese Room, a thought experiment that examines whether the ability to translate between Chinese and English requires an understanding of the language.
-   Rule-based translations between English and Russian.

AI’s operation on natural language has evolved significantly from its rule-based embeddings. Starting with primary neural networks, we added recurrence relations through RNNs to keep track of steps in time. From there, we used transformers to solve the sequence transduction problem.

Transformers consist of an encoder, which encodes an input into a matrix representing the state, an attention matrix and a decoder. The decoder decodes the state and attention matrix to predict the correct next token to finish the output sequence. GPT-3, the most popular language model to date, comprises strict decoders. They encode the input and predict the right next token(s).

Here are two models from the  `sentence-transformers`  library by Hugging Face that you can use in addition to OpenAI’s embeddings:

-   [MiniLM-L6-v2](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2): a 384-dimensional model
-   [MPNet-Base-V2](https://huggingface.co/sentence-transformers/all-mpnet-base-v2): a 768-dimensional model

You can access embeddings from both models in the same way.


```
from sentence_transformers import SentenceTransformer


model = SentenceTransformer("<model-name>")
vector_embeddings = model.encode(“<input>”)
```


### Multimodal embeddings

Multimodal models are less well-developed than image or text models. They often relate images to text.

The most useful open source example is [CLIP VIT](https://huggingface.co/openai/clip-vit-large-patch14), an image-to-text model. You can access CLIP VIT’s embeddings in the same way as you would an image model, as shown in the code below.


```
# Load model directly
from transformers import AutoProcessor, AutoModelForZeroShotImageClassification


processor = AutoProcessor.from_pretrained("openai/clip-vit-large-patch14")
model = AutoModelForZeroShotImageClassification.from_pretrained("openai/clip-vit-large-patch14")
from PIL import Image


image = Image.open("<image path>")
# image = Resize(size=(256, 256))(image)


inputs = extractor(images=image, return_tensors="pt")
# print(inputs)


outputs = model(**inputs)
vector_embeddings = outputs[1][-1].squeeze()
```


### Audio embeddings

AI for audio has received less attention than AI for text or images. The most common use case for audio is speech-to-text for industries such as call centers, medical technology and accessibility. One popular open source model for speech-to-text is  [Whisper from OpenAI](https://huggingface.co/openai/whisper-large-v2). The code below shows how to obtain vector embeddings from the speech-to-text model.  


```
import torch
from transformers import AutoFeatureExtractor, WhisperModel
from datasets import load_dataset


model = WhisperModel.from_pretrained("openai/whisper-base")
feature_extractor = AutoFeatureExtractor.from_pretrained("openai/whisper-base")
ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
inputs = feature_extractor(ds[0]["audio"]["array"], return_tensors="pt")
input_features = inputs.input_features
decoder_input_ids = torch.tensor([[1, 1]]) * model.config.decoder_start_token_id
vector_embedding = model(input_features, decoder_input_ids=decoder_input_ids).last_hidden_state
```


### Video embeddings

Video embeddings are more complex than audio or image embeddings. A multimodal approach is necessary when working with videos, as they include synchronized audio and images. One popular video model is the  [multimodal perceiver](https://huggingface.co/deepmind/multimodal-perceiver)  from DeepMind. This  [notebook tutorial](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/Perceiver/Perceiver_for_Multimodal_Autoencoding.ipynb)  shows how to use the model to classify a video.

To get the embeddings of the input, use  `outputs[1][-1].squeeze()`  from the code shown in the notebook instead of deleting the outputs. I highlight this code snippet in the  `autoencode`  function.


```
def autoencode_video(images, audio):
     # only create entire video once as inputs
     inputs = {'image': torch.from_numpy(np.moveaxis(images, -1, 2)).float().to(device),
               'audio': torch.from_numpy(audio).to(device),
               'label': torch.zeros((images.shape[0], 700)).to(device)}
     nchunks = 128
     reconstruction = {}
     for chunk_idx in tqdm(range(nchunks)):
          image_chunk_size = np.prod(images.shape[1:-1]) // nchunks
          audio_chunk_size = audio.shape[1] // SAMPLES_PER_PATCH // nchunks
          subsampling = {
               'image': torch.arange(
                    image_chunk_size * chunk_idx, image_chunk_size * (chunk_idx + 1)),
               'audio': torch.arange(
                    audio_chunk_size * chunk_idx, audio_chunk_size * (chunk_idx + 1)),
               'label': None,
          }
     # forward pass
          with torch.no_grad():
               outputs = model(inputs=inputs, subsampled_output_points=subsampling)


          output = {k:v.cpu() for k,v in outputs.logits.items()}
          reconstruction['label'] = output['label']
          if 'image' not in reconstruction:
               reconstruction['image'] = output['image']
               reconstruction['audio'] = output['audio']
          else:
               reconstruction['image'] = torch.cat(
                    [reconstruction['image'], output['image']], dim=1)
               reconstruction['audio'] = torch.cat(
                    [reconstruction['audio'], output['audio']], dim=1)
          vector_embeddings = outputs[1][-1].squeeze()
# finally, reshape image and audio modalities back to original shape
     reconstruction['image'] = torch.reshape(reconstruction['image'], images.shape)
     reconstruction['audio'] = torch.reshape(reconstruction['audio'], audio.shape)
     return reconstruction


     return None
```


## Storing, indexing, and searching vector embeddings with vector databases

Now that we understand what vector embeddings are and how to generate them using various powerful embedding models, the next question is how to store and take advantage of them. Vector databases are the answer.

Vector databases like  [Milvus](https://zilliz.com/what-is-milvus)  and  [Zilliz Cloud](https://zilliz.com/cloud)  are purposely built for storing, indexing, and searching across massive datasets of unstructured data through vector embeddings. They are also one of the most critical infrastructures for various AI stacks.

Vector databases usually use the  [Approximate Nearest Neighbor (ANN)](https://zilliz.com/glossary/anns)  algorithm to calculate the spatial distance between the query vector and vectors stored in the database. The closer the two vectors are located, the more relevant they are. Then the algorithm finds the top k nearest neighbors and delivers them to the user.

Vector databases are popular in use cases such as  [LLM retrieval augmented generation](https://zilliz.com/use-cases/llm-retrieval-augmented-generation)  (RAG), question and answer systems, recommender systems, semantic searches, and image, video and audio similarity searches.

To learn more about vector embeddings, unstructured data, and vector databases, consider starting with the  [Vector Database 101](https://zilliz.com/blog?tag=39&page=1)  series.

## Summary

Vectors are a powerful tool for working with unstructured data. Using vectors, we can mathematically compare different pieces of unstructured data based on semantic similarity. Choosing the right vector-embedding model is critical for building a vector search engine for any application.

In this post, we learned that vector embeddings are the internal representation of input data in a neural network. As a result, they depend highly on the network architecture and the data used to train the model. Different data types (such as images, text, and audio) require specific models. Fortunately, many pretrained open source models are available for use. In this post, we covered models for the five most common types of data: images, text, multimodal, audio, and video. In addition, if you want to make the best use of vector embeddings, vector databases are the most popular tool.

How to Get the Right Vector Embeddings


![](https://assets.zilliz.com/How_to_Migrate_Your_Data_to_Milvus_with_Ease_485dcb8b22.png)


[Milvus](https://milvus.io/) is a robust open-source vector database for [similarity search](https://zilliz.com/learn/vector-similarity-search) that can store, process, and retrieve billions and even trillions of vector data with minimal latency. It is also highly scalable, reliable, cloud-native, and feature-rich. [The newest release of Milvus](https://milvus.io/blog/unveiling-milvus-2-3-milestone-release-offering-support-for-gpu-arm64-cdc-and-other-features.md) introduces even more exciting features and improvements, including [GPU support](https://zilliz.com/blog/getting-started-with-gpu-powered-milvus-unlocking-10x-higher-performance) for over 10x faster performance and MMap for greater storage capacity on a single machine.

As of September 2023, Milvus has earned almost 23,000 stars on GitHub and has tens of thousands of users from diverse industries with varying needs. It is becoming even more popular as Generative AI technology like [ChatGPT](https://zilliz.com/learn/ChatGPT-Vector-Database-Prompt-as-code) becomes more prevalent. It is an essential component of various AI stacks, especially the [retrieval augmented generation](https://zilliz.com/use-cases/llm-retrieval-augmented-generation) framework, which addresses the hallucination problem of large language models.

To meet the growing demand from new users who want to migrate to Milvus and existing users who wish to upgrade to the latest Milvus versions, we developed  [Milvus Migration](https://github.com/zilliztech/milvus-migration). In this blog, we'll explore the features of Milvus Migration and guide you through quickly transitioning your data to Milvus from Milvus 1.x, [FAISS](https://zilliz.com/blog/set-up-with-facebook-ai-similarity-search-faiss), and [Elasticsearch 7.0](https://zilliz.com/comparison/elastic-vs-milvus) and beyond.

## Milvus Migration, a powerful data migration tool

[Milvus Migration](https://github.com/zilliztech/milvus-migration) is a data migration tool written in Go. It enables users to move their data seamlessly from older versions of Milvus (1.x), FAISS, and Elasticsearch 7.0 and beyond to Milvus 2.x versions.

The diagram below demonstrates how we built Milvus Migration and how it works.

![](https://assets.zilliz.com/milvus_migration_architecture_144e22f499.png)


### How Milvus Migration migrates data

#### From Milvus 1.x and FAISS to Milvus 2.x

The data migration from Milvus 1.x and FAISS involves parsing the content of the original data files, transforming them into the data storage format of Milvus 2.x, and writing the data using Milvus SDK's `bulkInsert`. This entire process is stream-based, theoretically limited only by disk space, and stores data files on your local disk, S3, OSS, GCP, or Minio.

#### From Elasticsearch to Milvus 2.x

In the Elasticsearch data migration, data retrieval is different. Data is not obtained from files but sequentially fetched using Elasticsearch's scroll API. The data is then parsed and transformed into Milvus 2.x storage format, followed by writing it using `bulkInsert`. Besides migrating `dense_vector` type vectors stored in Elasticsearch, Milvus Migration also supports migrating other field types, including long, integer, short, boolean, keyword, text, and double.

### Milvus Migration feature set

Milvus Migration simplifies the migration process through its robust feature set:

- **Supported Data Sources:**

	- Milvus 1.x to Milvus 2.x

	- Elasticsearch 7.0 and beyond to Milvus 2.x

	- FAISS to Milvus 2.x

  

- **Multiple Interaction Modes:**

	- Command-line interface (CLI) using the Cobra framework

	- Restful API with a built-in Swagger UI

	- Integration as a Go module in other tools

  

- **Versatile File Format Support:**

	- Local files

	- Amazon S3

	- Object Storage Service (OSS)

	- Google Cloud Platform (GCP)

  

- **Flexible Elasticsearch Integration:**

	- Migration of `dense_vector` type vectors from Elasticsearch

	- Support for migrating other field types such as long, integer, short, boolean, keyword, text, and double
    

### Interface definitions

Milvus Migration provides the following key interfaces:

-   `/start`: Initiates a migration job (equivalent to a combination of dump and load, currently only supports ES migration).
    
-   `/dump`: Initiates a dump job (writes source data into the target storage medium).
    
-   `/load`: Initiates a load job (writes data from the target storage medium into Milvus 2.x).
    
-   `/get_job`: Allows users to view job execution results. (For more details, refer to [the project's server.go](https://github.com/zilliztech/milvus-migration/blob/main/server/server.go))
    

  

Next, let's use some example data to explore how to use Milvus Migration in this section. You can find these examples [here](https://github.com/zilliztech/milvus-migration#migration-examples-migrationyaml-details) on GitHub.

## Migration from Elasticsearch to Milvus 2.x

1.  Prepare Elasticsearch Data
    

To [migrate Elasticsearch](https://zilliz.com/blog/elasticsearch-cloud-vs-zilliz) data, you should already set up your own Elasticsearch server. You should store vector data in the `dense_vector` field and index them with other fields. The index mappings are as shown below.

![](https://assets.zilliz.com/migrate_elasticsearch_data_milvus_index_mappings_59370f9596.png)



2.  Compile and Build
    

First, download the Milvus Migration’s [source code from GitHub](https://github.com/zilliztech/milvus-migration). Then, run the following commands to compile it.


```
go get
go build
```

This step will generate an executable file named `milvus-migration`.

  

3.  Configure `migration.yaml`
    

Before starting the migration, you must prepare a configuration file named `migration.yaml` that includes information about the data source, target, and other relevant settings. Here's an example configuration:



```
# Configuration for Elasticsearch to Milvus 2.x migration


dumper:
  worker:
    workMode: Elasticsearch
    reader:
      bufferSize: 2500
meta:
  mode: config
  index: test_index
  fields:
    - name: id
      pk: true
      type: long
    - name: other_field
      maxLen: 60
      type: keyword
    - name: data
      type: dense_vector
      dims: 512
  milvus:
      collection: "rename_index_test"
      closeDynamicField: false
      consistencyLevel: Eventually
      shardNum: 1


source:
  es:
    urls:
      - http://localhost:9200
    username: xxx
    password: xxx


target:
  mode: remote
  remote:
    outputDir: outputPath/migration/test1
    cloud: aws
    region: us-west-2
    bucket: xxx
    useIAM: true
    checkBucket: false
  milvus2x:
    endpoint: {yourMilvusAddress}:{port}
    username: ******
    password: ******
```
  

For a more detailed explanation of the configuration file, refer to [this page](https://github.com/zilliztech/milvus-migration/blob/main/README_ES.md#elasticsearch-to-milvus-2x-migrationyaml-example) on GitHub.

  

4.  Execute the migration job
    

Now that you have configured your `migration.yaml` file, you can start the migration task by running the following command:


```
./milvus-migration start --config=/{YourConfigFilePath}/migration.yaml
```

  

Observe the log output. When you see logs similar to the following, it means the migration was successful.

  
```
[task/load_base_task.go:94] ["[LoadTasker] Dec Task Processing-------------->"] [Count=0] [fileName=testfiles/output/zwh/migration/test_mul_field4/data_1_1.json] [taskId=442665677354739304][task/load_base_task.go:76] ["[LoadTasker] Progress Task --------------->"] [fileName=testfiles/output/zwh/migration/test_mul_field4/data_1_1.json] [taskId=442665677354739304][dbclient/cus_field_milvus2x.go:86] ["[Milvus2x] begin to ShowCollectionRows"][loader/cus_milvus2x_loader.go:66] ["[Loader] Static: "] [collection=test_mul_field4_rename1] [beforeCount=50000] [afterCount=100000] [increase=50000][loader/cus_milvus2x_loader.go:66] ["[Loader] Static Total"] ["Total Collections"=1] [beforeTotalCount=50000] [afterTotalCount=100000] [totalIncrease=50000][migration/es_starter.go:25] ["[Starter] migration ES to Milvus finish!!!"] [Cost=80.009174459][starter/starter.go:106] ["[Starter] Migration Success!"] [Cost=80.00928425][cleaner/remote_cleaner.go:27] ["[Remote Cleaner] Begin to clean files"] [bucket=a-bucket] [rootPath=testfiles/output/zwh/migration][cmd/start.go:32] ["[Cleaner] clean file success!"]
```


  

In addition to the command-line approach, Milvus Migration also supports migration using Restful API.

To use the Restful API, start the API server using the following command:


```
./milvus-migration server run -p 8080
```

Once the service runs, you can initiate the migration by calling the API.


```
curl -XPOST http://localhost:8080/api/v1/start
```

  

When the migration is complete, you can use [Attu](https://zilliz.com/attu), an all-in-one vector database administration tool, to view the total number of successful rows migrated and perform other collection-related operations.


![The Attu interface](https://assets.zilliz.com/attu_interface_vector_database_admin_4893a31f6d.png)



## Migration from Milvus 1.x to Milvus 2.x

1.  Prepare Milvus 1.x Data
    

To help you quickly experience the migration process, we’ve put 10,000 Milvus 1.x [test data](https://github.com/zilliztech/milvus-migration/blob/main/README_1X.md) records in the source code of Milvus Migration. However, in real cases, you must export your own `meta.json` file from your Milvus 1.x instance before starting the migration process.

-   You can export the data with the following command.
    

```
./milvus-migration export -m "user:password@tcp(adderss)/milvus?charset=utf8mb4&parseTime=True&loc=Local" -o outputDir
```

  

Make sure to:

-   Replace the placeholders with your actual MySQL credentials.
    
-   Stop the Milvus 1.x server or halt data writes before performing this export.
    
-   Copy the Milvus `tables` folder and the `meta.json` file to the same directory.
    

**Note:** If you use Milvus 2.x on [Zilliz Cloud](https://zilliz.com/cloud) (the fully managed service of Milvus), you can start the migration using Cloud Console.

2.  Compile and Build
    

First, download the Milvus Migration’s [source code from GitHub](https://github.com/zilliztech/milvus-migration). Then, run the following commands to compile it.


```
go get
go build
```


This step will generate an executable file named `milvus-migration`.

3.  Configure `migration.yaml`
    

Prepare a `migration.yaml` configuration file, specifying details about the source, target, and other relevant settings. Here's an example configuration:


```
# Configuration for Milvus 1.x to Milvus 2.x migration


dumper:
  worker:
    limit: 2
    workMode: milvus1x
    reader:
      bufferSize: 1024
    writer:
      bufferSize: 1024
loader:
  worker:
    limit: 16
meta:
  mode: local
  localFile: /outputDir/test/meta.json


source:
  mode: local
  local:
    tablesDir: /db/tables/


target:
  mode: remote
  remote:
    outputDir: "migration/test/xx"
    ak: xxxx
    sk: xxxx
    cloud: aws
    endpoint: 0.0.0.0:9000
    region: ap-southeast-1
    bucket: a-bucket
    useIAM: false
    useSSL: false
    checkBucket: true
  milvus2x:
    endpoint: localhost:19530
    username: xxxxx
    password: xxxxx
```


  

For a more detailed explanation of the configuration file, refer to [this page](https://github.com/zilliztech/milvus-migration/blob/main/README_1X.md) on GitHub.

  

4.  Execute Migration Job
    

You must execute the `dump` and `load` commands separately to finish the migration. These commands convert the data and import it into Milvus 2.x.

**Note:** We’ll simplify this step and enable users to finish migration using just one command shortly. Stay tuned.

  
**Dump Command:**

```
./milvus-migration dump --config=/{YourConfigFilePath}/migration.yaml
```
  

**Load Command:**


```
./milvus-migration load --config=/{YourConfigFilePath}/migration.yaml
```

  

After the migration, the generated collection in Milvus 2.x will contain two fields: `id` and `data`. You can view more details using [Attu](https://zilliz.com/attu), an all-in-one vector database administration tool.

## Migration from FAISS to Milvus 2.x

1.  Prepare FAISS Data
    

To migrate Elasticsearch data, you should have your own FAISS data ready. To help you quickly experience the migration process, we’ve put some [FAISS test data](https://github.com/zilliztech/milvus-migration/blob/main/README_FAISS.md) in the source code of Milvus Migration.

2.  Compile and Build
    

First, download the Milvus Migration’s [source code from GitHub](https://github.com/zilliztech/milvus-migration). Then, run the following commands to compile it.


```
go get
go build
```

This step will generate an executable file named `milvus-migration`.

  

3.  Configure `migration.yaml`
    

Prepare a `migration.yaml` configuration file for FAISS migration, specifying details about the source, target, and other relevant settings. Here's an example configuration:

```
# Configuration for FAISS to Milvus 2.x migration


dumper:
  worker:
    limit: 2
    workMode: FAISS
    reader:
      bufferSize: 1024
    writer:
      bufferSize: 1024
loader:
  worker:
    limit: 2
source:
  mode: local
  local:
    FAISSFile: ./testfiles/FAISS/FAISS_ivf_flat.index


target:
  create:
    collection:
      name: test1w
      shardsNums: 2
      dim: 256
      metricType: L2
  mode: remote
  remote:
    outputDir: testfiles/output/
    cloud: aws
    endpoint: 0.0.0.0:9000
    region: ap-southeast-1
    bucket: a-bucket
    ak: minioadmin
    sk: minioadmin
    useIAM: false
    useSSL: false
    checkBucket: true
  milvus2x:
    endpoint: localhost:19530
    username: xxxxx
    password: xxxxx
```
  

For a more detailed explanation of the configuration file, refer to [this page](https://github.com/zilliztech/milvus-migration/blob/main/README_FAISS.md) on GitHub.

  

4.  Execute Migration Job
    

Like Milvus 1.x to Milvus 2.x migration, FAISS migration requires executing both the `dump` and `load` commands. These commands convert the data and import it into Milvus 2.x.

**Note:** We’ll simplify this step and enable users to finish migration using just one command shortly. Stay tuned.

**Dump Command:**
```
./milvus-migration dump --config=/{YourConfigFilePath}/migration.yaml
```

**Load Command:**

```
./milvus-migration load --config=/{YourConfigFilePath}/migration.yaml
```



  

You can view more details using [Attu](https://zilliz.com/attu), an all-in-one vector database administration tool.

## Stay tuned for future migration plans

In the future, we’ll support migration from more data sources and add more migration features, including:

-   Support migration from Redis to Milvus.
    
-   Support migration from MongoDB to Milvus.
    
-   Support resumable migration.
    
-   Simplify migration commands by merging the dump and load processes into one.
    
-   Support migration from other mainstream data sources to Milvus.
    

## Conclusion

Milvus 2.3, the latest release of Milvus, brings exciting new features and performance improvements that cater to the growing needs of data management. Migrating your data to Milvus 2.x can unlock these benefits, and the Milvus Migration project makes the migration process streamlined and easy. Give it a try, and you won't be disappointed.

***Note:** The information in this blog is based on the state of the Milvus and [Milvus Migration](https://github.com/zilliztech/milvus-migration) projects as of September 2023. Check the official [Milvus documentation](https://milvus.io/docs) for the most up-to-date information and instructions.*

How to Migrate Your Data to Milvus Seamlessly: A Comprehensive Guide


![](https://assets.zilliz.com/Exploring_NATS_878f48c848.png)

In the intricate tapestry of data processing, seamless communication is the thread that binds operations together. [Milvus](https://zilliz.com/what-is-milvus), the trailblazing [open-source vector database](https://zilliz.com/cloud), has embarked on a transformative journey with its latest feature: NATS messaging integration. In this comprehensive blog post, we'll unravel the intricacies of this integration, exploring its core features, setup process, migration benefits, and how it stacks up against its predecessor, RocksMQ.

## Understanding the role of message queues in Milvus

In Milvus’ cloud-native architecture, the message queue, or Log Broker, holds pivotal importance. It’s the backbone ensuring persistent data streams, synchronization, event notifications, and data integrity during system recoveries. Traditionally, RocksMQ was the most straightforward choice in Milvus Standalone mode, especially when compared with Pulsar and Kafka, but its limitations became evident with extensive data and complex scenarios.

Milvus 2.3 introduces NATS, a single-node MQ implementation, redefining how to manage data streams. Unlike its predecessors, NATS liberates Milvus users from performance constraints, delivering a seamless experience in handling substantial data volumes.

## What is NATS?

NATS is a distributed system connectivity technology implemented in Go. It supports various communication modes like Request-Reply and Publish-Subscribe across systems, provides data persistence through JetStream, and offers distributed capabilities through built-in RAFT. You can refer to the [NATS official website](https://nats.io/) for a more detailed understanding of NATS.

In Milvus 2.3 Standalone mode, NATS, JetStream, and PubSub provide Milvus with robust MQ capabilities.

## Enabling NATS

Milvus 2.3 offers a new control option, `mq.type`, which allows users to specify the type of MQ they want to use. To enable NATS, set `mq.type=natsmq`. If you see logs similar to the ones below after you initiate Milvus instances, you have successfully enabled NATS as the message queue.

  
```
[INFO] [dependency/factory.go:83] ["try to init mq"] [standalone=true] [mqType=natsmq]
```
  

## Configuring NATS for Milvus

NATS customization options include specifying the listening port, JetStream storage directory, maximum payload size, and initialization timeout. Fine-tuning these settings ensures optimal performance and reliability.


```
natsmq:
server: # server side configuration for natsmq.
port: 4222 # 4222 by default, Port for nats server listening.
storeDir: /var/lib/milvus/nats # /var/lib/milvus/nats by default, directory to use for JetStream storage of nats.
maxFileStore: 17179869184 # (B) 16GB by default, Maximum size of the 'file' storage.
maxPayload: 8388608 # (B) 8MB by default, Maximum number of bytes in a message payload.
maxPending: 67108864 # (B) 64MB by default, Maximum number of bytes buffered for a connection Applies to client connections.
initializeTimeout: 4000 # (ms) 4s by default, waiting for initialization of natsmq finished.
monitor:
trace: false # false by default, If true enable protocol trace log messages.
debug: false # false by default, If true enable debug log messages.
logTime: true # true by default, If set to false, log without timestamps.
logFile: /tmp/milvus/logs/nats.log # /tmp/milvus/logs/nats.log by default, Log file path relative to .. of milvus binary if use relative path.
logSizeLimit: 536870912 # (B) 512MB by default, Size in bytes after the log file rolls over to a new one.
retention:
maxAge: 4320 # (min) 3 days by default, Maximum age of any message in the P-channel.
maxBytes: # (B) None by default, How many bytes the single P-channel may contain. Removing oldest messages if the P-channel exceeds this size.
maxMsgs: # None by default, How many message the single P-channel may contain. Removing oldest messages if the P-channel exceeds this limit.
```
  

**Note:**

-   You must specify `server.port` for NATS server listening. If there is a port conflict, Milvus cannot start. Set `server.port=-1` to randomly select a port.
    
-   `storeDir` specifies the directory for JetStream storage. We recommend storing the directory in a high-performant solid-state drive (SSD) for better read/write throughput of Milvus.
    
-   `maxFileStore` sets the upper limit of JetStream storage size. Exceeding this limit will prevent further data writing.
    
-   `maxPayload` limits individual message size. You should keep it above 5MB to avoid any write rejections.
    
-   `initializeTimeout`controls NATS server startup timeout.
    
-   `monitor` configures NATS’ independent logs.
    
-   `retention` controls the retention mechanism of NATS messages.
    

For more information, refer to [NATS official documentation](https://docs.nats.io/running-a-nats-service/configuration).

## Migrating from RocksMQ to NATS

Migrating from RocksMQ to NATS is a seamless process involving steps like stopping write operations, flushing data, modifying configurations, and verifying the migration through Milvus logs.

1.  Before initiating the migration, stop all write operations in Milvus.
    
2.  Execute the `FlushALL` operation in Milvus and wait for its completion. This step ensures that all pending data is flushed and the system is ready for shutdown.
    
3.  Modify the Milvus configuration file by setting `mq.type=natsmq` and adjusting relevant options under the `natsmq` section.
    
4.  Start the Milvus 2.3.
    
5.  Back up and clean the original data stored in the `rocksmq.path` directory. (Optional)
    

## NATS vs. RocksMQ: A Performance Showdown

### Pub/Sub Performance Testing

-   **Testing Platform:** M1 Pro Chip / Memory: 16GB
    
-   **Testing Scenario:** Subscribing and publishing random data packets to a topic repeatedly until the last published result is received.
    
-   **Results:**
    
    -   For smaller data packets (< 64kb), RocksMQ outperforms NATS regarding memory, CPU, and response speed.
    
    -   For larger data packets (> 64kb), NATS outshines RocksMQ, offering much faster response times.
    

  

| Test Type           | MQ      | op count | cost per op      | Memory cost | CPU Total Time | Storage cost |
| ------------------- | ------- | -------- | ---------------- | ----------- | -------------- | ------------ |
| 5MB\*100 Pub/Sub    | NATS    | 50       | 1.650328186 s/op | 4.29 GB     | 85.58          | 25G          |
| 5MB\*100 Pub/Sub    | RocksMQ | 50       | 2.475595131 s/op | 1.18 GB     | 81.42          | 19G          |
| 1MB\*500 Pub/Sub    | NATS    | 50       | 2.248722593 s/op | 2.60 GB     | 96.50          | 25G          |
| 1MB\*500 Pub/Sub    | RocksMQ | 50       | 2.554614279 s/op | 614.9 MB    | 80.19          | 19G          |
| 64KB\*10000 Pub/Sub | NATS    | 50       | 2.133345262 s/op | 3.29 GB     | 97.59          | 31G          |
| 64KB\*10000 Pub/Sub | RocksMQ | 50       | 3.253778195 s/op | 331.2 MB    | 134.6          | 24G          |
| 1KB\*50000 Pub/Sub  | NATS    | 50       | 2.629391004 s/op | 635.1 MB    | 179.67         | 2.6G         |
| 1KB\*50000 Pub/Sub  | RocksMQ | 50       | 0.897638581 s/op | 232.3 MB    | 60.42          | 521M         |

  

Table 1: Pub/Sub performance testing results

### Milvus Integration Testing

**Data size:** 100M

**Result:** In extensive testing with a 100 million vectors dataset, NATS showcased lower vector search and query latency.

| Metrics                                 | RocksMQ (ms) | NATS (ms) |
| --------------------------------------- | ------------ | --------- |
| Average vector search latency           | 23.55        | 20.17     |
| Vector search requests per second (RPS) | 2.95         | 3.07      |
| Average query latency                   | 7.2          | 6.74      |
| Query requests per second (RPS)         | 1.47         | 1.54      |

Table 2: Milvus integration testing results with 100m dataset

  

**Dataset: <100M**

**Result:** For datasets smaller than 100M, NATS and RocksMQ show similar performance.

## Conclusion: Empowering Milvus with NATS messaging

The integration of NATS within Milvus marks a significant stride in data processing. Whether delving into real-time analytics, machine learning applications, or any data-intensive venture, NATS empowers your projects with efficiency, reliability, and speed. As the data landscape evolves, having a robust messaging system like NATS within Milvus ensures seamless, reliable, and high-performing data communication.

Optimizing Data Communication: Milvus Embraces NATS Messaging

![](https://assets.zilliz.com/What_s_New_in_Milvus_2_3_2_and_2_3_3_d3d0db03c3.png)

In the ever-evolving landscape of vector search technologies, Milvus remains at the forefront, pushing boundaries and setting new standards. Today, we are thrilled to announce the release of Milvus 2.3.2 and 2.3.3! These updates bring many exciting features, optimizations, and improvements, enhancing system performance, flexibility, and overall user experience.

## Support for Array data types - making search results more accurate and relevant

Adding the Array data type support is a pivotal enhancement for Milvus, particularly in query filtering scenarios like intersection and union. This addition ensures that search results are not only more accurate but also more relevant. In practical terms, for instance, within the e-commerce sector, product tags stored as string Arrays allow consumers to perform advanced searches, filtering out irrelevant results.

Dive into our comprehensive [documentation](https://milvus.io/docs/array_data_type.md) for an in-depth guide on leveraging Array types in Milvus.

## Support for complex delete expressions - improving your data management

In previous versions, Milvus supported primary key deletion expressions, providing a stable and streamlined architecture. With Milvus 2.3.2 or 2.3.3, users can employ complex delete expressions, facilitating sophisticated data management tasks such as rolling cleanup of old data or GDPR compliance-driven data deletion based on user IDs.

Note: Ensure you’ve loaded collections before utilizing complex expressions. Additionally, it's important to note that the deletion process does not guarantee atomicity.

## TiKV integration - scalable metadata storage with stability

Previously relying on Etcd for metadata storage, Milvus faced limited capacity and scalability challenges in metadata storage. To address these problems, Milvus added TiKV, an open-source key-value store, as one more option for metadata storage. TiKV offers enhanced scalability, stability, and efficiency, making it an ideal solution for Milvus's evolving requirements. Starting from Milvus 2.3.2, users can seamlessly transition to TiKV for their metadata storage by modifying the configuration.

## Support for FP16 vector type - embracing machine learning efficiency

Milvus 2.3.2 and later versions now support the FP16 vector type at the interface level. FP16, or 16-bit floating point, is a data format widely used in deep learning and machine learning, providing efficient representation and calculation of numerical values. While full support for FP16 is underway, various indexes in the indexing layer require converting FP16 to FP32 during construction.

We will fully support FP16, BF16, and int8 data types in later versions of Milvus. Stay tuned.

## Significant improvement in the rolling upgrade experience - seamless transition for users

Rolling upgrade is a critical feature for distributed systems, enabling system upgrades without disrupting business services or experiencing downtimes. In the latest Milvus releases, we’ve strengthened Milvus’s rolling upgrade feature, ensuring a more streamlined and efficient transition for users’ upgrading  from version 2.2.15 to 2.3.3 and all later versions. The community has also invested in extensive testing and optimizations, reducing query impact during the upgrade to less than 5 minutes, providing users with a hassle-free experience.

## Performance optimization

In addition to introducing new features, we’ve significantly optimized the performance of Milvus in the latest two releases.

-   Minimized data copy operations for optimized data loading
    
-   Simplified large-capacity inserts using batch varchar reading
    
-   Removed unnecessary offset checks during data padding to improve recall phase performance.
    
-   Addressed high CPU consumption issues in scenarios with substantial data insertions
    

These optimizations collectively contribute to a faster and more efficient Milvus experience. Check out our monitoring dashboard for a quick glance at how Milvus improved its performance.

  

## Incompatible changes

-   Permanently deleted TimeTravel-related code.
    
-   Deprecated support for MySQL as the metadata store.
    

Refer to the [Milvus release notes](https://milvus.io/docs/release_notes.md) for more detailed information about all the new features and enhancements.

## Conclusion

With the latest Milvus 2.3.2 and 2.3.3 releases, we're committed to providing a robust, feature-rich, high-performance database solution. Explore these new features, take advantage of the optimizations, and join us on this exciting journey as we evolve Milvus to meet the demands of modern data management. Download the latest version now and experience the future of data storage with Milvus!

## Let’s keep in touch!

If you have questions or feedback about Milvus, join our [Discord channel](https://discord.com/invite/8uyFbECzPX) to engage with our engineers and the community directly or join our [Milvus Community Lunch and Learn](https://discord.com/invite/RjNbk8RR4f) Every Tuesday from 12-12:30 PM PST. You’re also welcome to follow us on [Twitter](https://twitter.com/milvusio) or [LinkedIn](https://www.linkedin.com/company/the-milvus-project) for the latest news and updates about Milvus.

Revealing Milvus 2.3.2 & 2.3.3: Support for Array Data Types, Complex Delete, TiKV Integration, and More


![](https://assets.zilliz.com/Exploring_M_Map_5086d652bd.png)

[Milvus](https://zilliz.com/what-is-milvus) is the fastest solution in open-source [vector databases](https://zilliz.com/blog/what-is-a-real-vector-database), catering to users with intensive performance requirements. However, the diversity of user needs mirrors the data they work with. Some prioritize budget-friendly solutions and expansive storage over sheer speed. Understanding this spectrum of demands, Milvus introduces the MMap feature, redefining how we handle large data volumes while promising cost efficiency without sacrificing functionality.

## What is MMap?

MMap, short for memory-mapped files, bridges the gap between files and memory within operating systems. This technology allows Milvus to map large files directly into the system's memory space, transforming files into contiguous memory blocks. This integration eliminates the need for explicit read or write operations, fundamentally changing how Milvus manages data. It ensures seamless access and efficient storage for large files or situations where users need to access files randomly.

## Who benefits from MMap?

Vector databases demand substantial memory capacity due to the storage requirements of vector data. With the MMap feature, processing more data within limited memory becomes a reality. However, this increased capability comes at a performance cost. The system intelligently manages memory, evicting some data based on load and usage. This eviction allows Milvus to process more data within the same memory capacity.

During our tests, we observed that with ample memory, all data resides in memory after a warm-up period, preserving system performance. However, as data volume grows, performance gradually decreases. **Therefore, we recommend the MMap feature for users less sensitive to performance fluctuations.**

## Enabling MMap in Milvus: a simple configuration

Enabling MMap in Milvus is remarkably straightforward. All you need to do is modify the `milvus.yaml` file: add the `mmapDirPath` item under the `queryNode` configuration and set a valid path as its value.

![](https://assets.zilliz.com/enabling_mmap_a2df88276b.png)

## Striking the balance: performance, storage, and system limits

Data access patterns significantly impact performance. Milvus's MMap feature optimizes data access based on locality. MMap enables Milvus to write scalar data directly to the disk for sequentially accessed data segments. Variable-length data such as strings undergoes flattening and is indexed using an offsets array in memory. This approach ensures data access locality and eliminates the overhead of storing each variable-length data separately. Optimizations for vector indexes are meticulous. MMap is selectively employed for vector data while retaining adjacency lists in memory, conserving significant memory without compromising performance.

In addition, MMap maximizes data processing by minimizing memory usage. Unlike previous Milvus versions where QueryNode copied entire datasets, MMap adopts a streamlined, copy-free streaming process during development. This optimization drastically reduces memory overhead.

**Our internal testing results show that Milvus can efficiently handle double the data volume when enabling MMap.**

## The road ahead: continuous innovation and user-centric enhancements

While the MMap feature is in its beta phase, Milvus's team is committed to continuous improvement. Future updates will refine the system's memory usage, enabling Milvus to support even more extensive data volumes on a single node. Users can anticipate more granular control over the MMap feature, enabling dynamic changes to collections and advanced field loading modes. These enhancements provide unprecedented flexibility, allowing users to tailor their data processing strategies to specific requirements.

## Conclusion: redefining data processing excellence with Milvus MMap

Milvus 2.3's MMap feature marks a significant leap in data processing technology. By striking a delicate balance between performance, cost, and system limits, Milvus empowers users to handle vast amounts of data efficiently and cost-effectively. As Milvus continues to evolve, it remains at the forefront of innovative solutions, redefining the boundaries of what's achievable in data management.

Stay tuned for more groundbreaking developments as Milvus continues its journey toward unparalleled data processing excellence.


Milvus Introduced MMap for Redefined Data Management and Increased Storage Capability



![](https://assets.zilliz.com/Vector_Databases_vs_Vector_Search_Plugins_vs_Vector_Search_Libraries_74def521ad.png)

Hey there - welcome back to Vector Database 101!

The surge in [ChatGPT](https://zilliz.com/learn/ChatGPT-Vector-Database-Prompt-as-code) and other large language models (LLMs) has driven the growth of vector search technologies, featuring specialized vector databases like [Milvus](https://zilliz.com/what-is-milvus) and [Zilliz Cloud](https://zilliz.com/cloud) alongside libraries such as [FAISS](https://zilliz.com/blog/set-up-with-facebook-ai-similarity-search-faiss) and integrated vector search plugins within conventional databases.


In our [previous series post](https://zilliz.com/learn/what-is-vector-database), we delved into the fundamentals of vector databases. In this post, we’ll continue to explore the intricate realm of vector search, comparing vector databases, vector search plugins, and vector search libraries.


## What is vector search?


[Vector search](https://zilliz.com/learn/vector-similarity-search), also known as vector similarity search, is a technique for retrieving the top-k results that are most similar or semantically related to a given query vector among an extensive collection of dense vector data. Before conducting similarity searches, we leverage neural networks to transform [unstructured data](https://zilliz.com/learn/introduction-to-unstructured-data), such as text, images, videos, and audio, into high-dimensional numerical vectors called embedding vectors. After generating embedding vectors, vector search engines compare the spatial distance between the input query vector and the vectors in the vector stores. The closer they are in space, the more similar they are.


Multiple vector search technologies are available in the market, including machine learning libraries like Python's NumPy, vector search libraries like FAISS, vector search plugins built on traditional databases, and specialized vector databases like Milvus and Zilliz Cloud.




## Vector databases vs. vector search libraries


[Specialized vector databases](https://zilliz.com/blog/what-is-a-real-vector-database) are not the only stack for similarity searches. Before the advent of vector databases, many vector searching libraries, such as FAISS, ScaNN, and HNSW, were used for vector retrieval.


Vector search libraries can help you quickly build a high-performance prototype vector search system. Taking FAISS as an example, it is open-source and developed by Meta for efficient similarity search and dense vector clustering. FAISS can handle vector collections of any size, even those that cannot be fully loaded into memory. Additionally, FAISS offers tools for evaluation and parameter tuning. Even though written in C++, FAISS provides a Python/NumPy interface.


However, vector search libraries are merely lightweight ANN libraries rather than managed solutions, and they have limited functionality. If your dataset is small and limited, these libraries can be sufficient for unstructured data processing, even for systems running in production. However, as dataset sizes increase and more users are onboarded, the scale problem becomes increasingly difficult to solve. Moreover, they don’t allow any modifications to their index data and cannot be queried during data import.


By contrast, vector databases are a more optimal solution for unstructured data storage and retrieval. They can store and query millions or even billions of vectors while providing real-time responses simultaneously; they’re highly scalable to meet users’ growing business needs.


In addition, vector databases like Milvus have much more user-friendly features for structured/semi-structured data:  cloud-nativity, multi-tenancy, scalability, etc. These features will become clear as we dive deeper into this tutorial.


They also operate in a totally different layer of abstraction from vector search libraries - vector databases are full-fledged services, while ANN libraries are meant to be integrated into the application that you’re developing. In this sense, ANN libraries are one of the many components that vector databases are built on top of, similar to how Elasticsearch is built on top of Apache Lucene.


To give an example of why this abstraction is so important, let’s look at inserting a new unstructured data element into a vector database. This is super easy in Milvus:


```
from pymilvus import Collectioncollection = Collection('book')mr = collection.insert(data)
```


It’s really as easy as that - 3 lines of code. With a library such as FAISS or ScaNN, there is, unfortunately, no easy way of doing this without manually re-creating the entire index at certain checkpoints. Even if you could, vector search libraries still lack scalability and multi-tenancy, two of the most important vector database features.


## Vector databases vs. vector search plugins for traditional databases


Great, now that we’ve established the difference between vector search libraries and vector databases, let’s take a look at how vector databases differ from **vector search plugins**.


An increasing number of traditional relational databases, and search systems such as Clickhouse and [Elasticsearch](https://zilliz.com/blog/elasticsearch-cloud-vs-zilliz) are including built-in vector search plugins. Elasticsearch 8.0, for example, includes vector insertion and ANN search functionality that can be called via restful API endpoints. The problem with vector search plugins should be clear as night and day - **these solutions do not take a full-stack approach to embedding management and vector search**. Instead, these plugins are meant to be enhancements on top of existing architectures, thereby making them limited and unoptimized. Developing an unstructured data application atop a traditional database would be like trying to fit lithium batteries and electric motors inside the frame of a gas-powered car - not a great idea!


To illustrate why this is, let’s go back to the list of features that a vector database should implement (from the first section). Vector search plugins are missing two of these features - tunability and user-friendly APIs/SDKs. I’ll continue to use Elasticsearch’s ANN engine as an example; other vector search plugins operate very similarly so I won’t go too much further into detail. Elasticsearch supports vector storage via the `dense_vector` data field type and allows for querying via the `knnsearch endpoint`:


```json
PUT index
{
"mappings": {
  "properties": {
    "image-vector": {
      "type": "dense_vector",
      "dims": 128,
      "index": true,
      "similarity": "l2_norm"
    }
  }
}
}


PUT index/_doc
{
"image-vector": [0.12, 1.34, ...]
}
```


```json
GET index/_knn_search
{
"knn": {
  "field": "image-vector",
  "query_vector": [-0.5, 9.4, ...],
  "k": 10,
  "num_candidates": 100
}
}
```


Elasticsearch's ANN plugin supports only one indexing algorithm: Hierarchical Navigable Small Worlds, also known as HNSW (I like to think that the creator was ahead of Marvel when it came to popularizing the multiverse). On top of that, only L2/Euclidean distance is supported as a distance metric. This is an okay start, but let's compare it to Milvus, a full-fledged vector database. Using `pymilvus`:


```python
>>> field1 = FieldSchema(name='id', dtype=DataType.INT64, description='int64', is_primary=True)
>>> field2 = FieldSchema(name='embedding', dtype=DataType.FLOAT_VECTOR, description='embedding', dim=128, is_primary=False)
>>> schema = CollectionSchema(fields=[field1, field2], description='hello world collection')
>>> collection = Collection(name='my_collection', data=None, schema=schema)
>>> index_params = {
       'index_type': 'IVF_FLAT',
       'params': {'nlist': 1024},
       "metric_type": 'L2'}
>>> collection.create_index('embedding', index_params)
```


```python
>>> search_param = {
       'data': vector,
       'anns_field': 'embedding',
       'param': {'metric_type': 'L2', 'params': {'nprobe': 16}},
       'limit': 10,
       'expr': 'id_field > 0'
   }
>>> results = collection.search(**search_param)
```


While both [Elasticsearch and Milvus](https://zilliz.com/comparison/elastic-vs-milvus) have methods for creating indexes, inserting embedding vectors, and performing nearest neighbor search, it’s clear from these examples that Milvus has a more intuitive vector search API (better user-facing API) and broader vector index + distance metric support (better tunability). Milvus also plans to support more vector indices and allow for querying via SQL-like statements in the future, further improving both tunability and usability.


We just blew through quite a bit of content. This section was admittedly fairly long, so for those of you who skimmed it, here’s a quick tl;dr: Milvus is better than vector search plugins because Milvus was built from the ground-up as a vector database, allowing for a richer set of features and an architecture more suited towards unstructured data.


## How to choose from different vector search technologies?


Not all vector databases are created equal; each possesses unique traits that cater to specific applications. Vector search libraries and plugins are user-friendly and ideal for handling small-scale production environments with millions of vectors. If your data size is small and you just require basic vector search functionality, these technologies are sufficient for your business.


However, a specialized vector database should be your top choice for data-intensive businesses dealing with hundreds of millions of vectors and demanding real-time responses. Milvus, for instance, effortlessly manages billions of vectors, offering lightning-fast query speeds and rich functionality. Moreover, fully managed solutions like Zilliz prove even more advantageous, liberating you from operational challenges and enabling an exclusive focus on your core business activities.


## Take another look at the Vector Database 101 courses


1. [Introduction to Unstructured Data](https://zilliz.com/blog/introduction-to-unstructured-data)
2. [What is a Vector Database?](https://zilliz.com/learn/what-is-vector-database)
3. [Comparing Vector Databases, Vector Search Libraries, and Vector Search Plugins](https://zilliz.com/learn/comparing-vector-database-vector-search-library-and-vector-search-plugin)
4. [Introduction to Milvus](https://zilliz.com/blog/introduction-to-milvus-vector-database)
5. [Milvus Quickstart](https://zilliz.com/blog/milvus-vector-database-quickstart)
6. [Introduction to Vector Similarity Search](https://zilliz.com/blog/vector-similarity-search)
7. [Vector Index Basics and the Inverted File Index](https://zilliz.com/blog/vector-index)
8. [Scalar Quantization and Product Quantization](https://zilliz.com/blog/scalar-quantization-and-product-quantization)
9. [Hierarchical Navigable Small Worlds (HNSW)](https://zilliz.com/blog/hierarchical-navigable-small-worlds-HNSW)
10. [Approximate Nearest Neighbors Oh Yeah (ANNOY)](https://zilliz.com/learn/approximate-nearest-neighbor-oh-yeah-ANNOY)
11. [Choosing the Right Vector Index for Your Project](https://zilliz.com/learn/choosing-right-vector-index-for-your-project)
12. [DiskANN and the Vamana Algorithm](https://zilliz.com/learn/DiskANN-and-the-Vamana-Algorithm)


Comparing Vector Databases, Vector Search Libraries, and Vector Search Plugins




![](https://assets.zilliz.com/Ingesting_Chaos_20231017_110103_54fe2009cb.png)


Data is being generated faster than ever before in every form imaginable. This data is the gasoline that will power a new wave of artificial intelligence applications, but these productivity enhancement engines need help ingesting this fuel. The wide range of scenarios and edge cases surrounding unstructured data makes it challenging to use in production AI systems.

For starters, there are a vast number of data sources. These export data in various file formats, each with its eccentricities. For example, how you process a PDF varies greatly depending on where it comes from. Ingesting a PDF for a securities litigation case will likely focus on textual data. In contrast, a system design specification for a rocket engineer will be full of diagrams that require visual processing. The lack of a defined schema in unstructured data further adds complexity. Even when the challenge of processing the data is overcome, the issue of ingesting it at scale remains. Files can vary significantly in size, which changes how they are processed. You can quickly process a 1MB upload on an API over HTTP, but reading in dozens of GBs from a single file requires streaming and a dedicated worker. 

Overcoming these traditional data engineering challenges is table stakes for connecting raw data to [LLMs](https://zilliz.com/glossary/large-language-models-(llms)) via [vector databases](https://zilliz.com/learn/what-is-vector-database) like [Milvus](https://github.com/milvus-io/milvus). However, emerging use cases such as performing semantic similarity searches with the help of a vector database require new processing steps like chunking the source data, orchestrating metadata for hybrid searches, picking the suitable vector embedding model, and tuning the search parameters to determine what data to feed to the LLM. These workflows are so new that there are no established best practices for developers to follow. Instead, they must experiment to find the correct configuration and use case for their data. To speed up this process, using a vector embedding pipeline to handle the data ingestion into the vector database is invaluable. 

A vector embedding pipeline like [VectorFlow](https://github.com/dgarnitz/vectorflow) will connect your raw data to your vector database, including chunking, metadata orchestration, embedding, and upload. VectorFlow enables engineering teams to focus on the core application logic, experimenting with the different retrieval parameters generated from the embedding model, the chunking strategy, the metadata fields, and aspects of the search to see what performs best. 

In our work helping engineering teams move their [retrieval augmented generation (RAG)](https://zilliz.com/use-cases/llm-retrieval-augmented-generation) systems from prototype to production, we have observed the following approach to be successful in testing the different parameters of a RAG search pipeline:

1. Use a small set of the data you are familiar with for speed of iteration, like a few PDFs where they have relevant chunks for the search queries.
2. Make a standard set of questions and answers about that subset of the data. For example, after reading the PDFs, write a list of questions and have your team agree on the answers.
3. Create an automated evaluation system that scores how the retrieval does on each question. One way to do this is to take the answer from the RAG system and run it back through the LLM with a prompt that asks if this RAG result answers the question given the correct answer. This should be a “yes” or “no” answer. For example, if you have 25 questions on your documents, and the system gets 20 correct, you can use this to benchmark against other approaches.  
4. Ensure you use a different LLM for the evaluation than you used to encode the vector embeddings stored in the database. The evaluation LLM is typically a decoder-type of a model like GPT-4. One thing to remember is the cost of these evaluations when run repeatedly. Open-source models like Llama2 70B or the Deci AI LLM 6B, which can run on a single, smaller GPU, have roughly the same performance at a fraction of the cost.
5. Run each test multiple times and average the score to smooth out the stochasticity of the LLM.

Holding every option constant except one, you can quickly determine which parameters work best for your use case. A vector embedding pipeline like VectorFlow makes this especially easy on the ingestion side, where you can quickly try out different chunking strategies, chunk lengths, chunk overlaps, and open-source embedding models to see what leads to the best results. This is especially useful when your dataset has various file types and data sources that require custom logic. 

Once the team knows what works for its use case, the vector embedding pipeline enables them to quickly move to production without having to redesign the system to consider things like reliability and monitoring. With technologies like VectorFlow and [Milvus](https://zilliz.com/what-is-milvus), which are open-source and platform-agnostic, the team can efficiently test across different environments while complying with privacy and security requirements. 





Ingesting Chaos: The MLOps Behind Handling Unstructured Data Reliably at Scale for RAG 



![](https://assets.zilliz.com/Why_you_need_a_real_vector_database2_20230830_075505_1_4b32582c87.png)

*This article was originally published on [AIAI](https://www.aiacceleratorinstitute.com/why-and-when-do-you-need-a-purpose-built-vector-database/) and is reposted here with permission.*


The increasing popularity of [ChatGPT](https://zilliz.com/learn/ChatGPT-Vector-Database-Prompt-as-code) and other large language models (LLMs) has fueled the rise of vector search technologies, including purpose-built vector databases such as [Milvus](https://milvus.io/docs/overview.md) and [Zilliz Cloud](https://zilliz.com/cloud), vector search libraries such as [FAISS](https://zilliz.com/blog/set-up-with-facebook-ai-similarity-search-faiss), and vector search plugins integrated with traditional databases. However, choosing the best solution for your needs can be challenging. Like choosing between a high-end restaurant and a fast-food chain, selecting the right vector search technology depends on your needs and expectations.

In this post, I will provide an overview of vector search and its functioning, compare different vector search technologies, and explain why opting for a purpose-built vector database is crucial.

## What is vector search, and how does it work?

[Vector search](https://zilliz.com/blog/vector-similarity-search), also known as vector similarity search, is a technique for retrieving the top-k results that are most similar or semantically related to a given query vector among an extensive collection of dense vector data.

Before conducting similarity searches, we leverage neural networks to transform [unstructured data](https://zilliz.com/blog/introduction-to-unstructured-data), such as text, images, videos, and audio, into high-dimensional numerical vectors called embedding vectors. For example, we can use the pre-trained ResNet-50 convolutional neural network to transform a bird image into a collection of embeddings with 2,048 dimensions. Here, we list the first three and last three vector elements: `[0.1392, 0.3572, 0.1988, ..., 0.2888, 0.6611, 0.2909]`.

![A bird image by Patrice Bouchard](https://assets.zilliz.com/bird_image_4a1be18f99.png)

After generating embedding vectors, vector search engines compare the spatial distance between the input query vector and the vectors in the vector stores. The closer they are in space, the more similar they are. 

![Embedding arithmetic](https://assets.zilliz.com/Frame_3732_20230510_073643_25f985523e.png)

## Popular vector search technologies 

Multiple vector search technologies are available in the market, including machine learning libraries like Python's NumPy, vector search libraries like FAISS, vector search plugins built on traditional databases, and specialized vector databases like Milvus and Zilliz Cloud. 

### Machine learning libraries 

Using machine learning libraries is the easiest way to implement vector searches. For instance, we can use Python's NumPy to implement a nearest neighbor algorithm in less than 20 lines of code.

```
import numpy as np

# Function to calculate euclidean distance
def euclidean_distance(a, b):
return np.linalg.norm(a - b)

# Function to perform knn
def knn(data, target, k):
# Calculate distances between target and all points in the data
distances = [euclidean_distance(d, target) for d in data]
# Combine distances with data indices
distances = np.array(list(zip(distances, np.arange(len(data)))))

# Sort by distance
sorted_distances = distances[distances[:, 0].argsort()]

# Get the top k closest indices
closest_k_indices = sorted_distances[:k, 1].astype(int)

# Return the top k closest vectors
return data[closest_k_indices]
```

We can generate 100 two-dimensional vectors and find the nearest neighbor to the vector [0.5, 0.5].

```
# Define some 2D vectors
data = np.random.rand(100, 2)

# Define a target vector
target = np.array([0.5, 0.5])

# Define k
k = 3

# Perform knn
closest_vectors = knn(data, target, k)

# Print the result
print("The closest vectors are:")
print(closest_vectors)
```

Machine learning libraries, such as Python's NumPy, offer great flexibility at a low cost. However, they do have some limitations. For instance, they can only handle a small amount of data and do not ensure data persistence.

I only recommend using NumPy or other machine learning libraries for vector search when:

- You need quick prototyping.
- You don't care about data persistence.
- Your data size is under one million, and you do not require scalar filtering.
- You do not need high performance.

### Vector search libraries

Vector search libraries can help you quickly build a high-performance prototype vector search system. FAISS is a typical example. It is open-source and developed by Meta for efficient similarity search and dense vector clustering. FAISS can handle vector collections of any size, even those that cannot be fully loaded into memory. Additionally, FAISS offers tools for evaluation and parameter tuning. Even though written in C++, FAISS provides a Python/NumPy interface.

Below is the code for an example vector search based on FAISS:

```
import numpy as np
import faiss

# Generate some example data
dimension = 64 # dimension of the vector space
database_size = 10000 # size of the database
query_size = 100 # number of queries to perform
np.random.seed(123) # make the random numbers predictable

# Generating vectors to index in the database (db_vectors)
db_vectors = np.random.random((database_size, dimension)).astype('float32')

# Generating vectors for query (query_vectors)
query_vectors = np.random.random((query_size, dimension)).astype('float32')

# Building the index
index = faiss.IndexFlatL2(dimension) # using the L2 distance metric
print(index.is_trained) # should return True

# Adding vectors to the index
index.add(db_vectors)
print(index.ntotal) # should return database_size (10000)

# Perform a search
k = 4 # we want to see 4 nearest neighbors
distances, indices = index.search(query_vectors, k)

# Print the results
print("Indices of nearest neighbors: \n", indices)
print("\nL2 distances to the nearest neighbors: \n", distances)
```

Vector search libraries such as FAISS are easy to use and fast enough to handle small-scale production environments with millions of vectors. You can enhance their query performance by utilizing quantization and GPUs and reducing data dimensions.

However, these libraries have some limitations when used in production. For example, FAISS does not support real-time data addition and deletion, remote calls, multiple languages, scalar filtering, scalability, or disaster recovery.

### Different types of vector databases 

Vector databases have emerged to address the limitations of the libraries above, providing a more comprehensive and practical solution for production applications.

Four types of vector databases are available on the battlefield:

- Existing relational or columnar databases that incorporate a vector search plugin. PG Vector is an example. 
- Traditional inverted index search engines with support for dense vector indexing. [ElasticSearch](https://zilliz.com/comparison/elastic-vs-milvus) is an example.
- Lightweight vector databases built on vector search libraries. Chroma is an example.
- **Purpose-built vector databases**. This type of database is specifically designed and optimized for vector searching from the bottom up. Purpose-built vector databases typically offer more advanced features, including distributed computing, disaster recovery, and data persistence. [Milvus](https://zilliz.com/what-is-milvus) is a primary example.

Not all vector databases are created equal. Each stack has unique advantages and limitations, making them more or less suitable for different applications. 

I prefer specialized vector databases over other solutions because they are the most efficient and convenient option, offering numerous unique benefits. In the following sections, I will use Milvus as an example to explain the reasons for my preference. 

## Key benefits of purpose-built vector databases

[Milvus](https://milvus.io/) is an open-source, distributed, purpose-built vector database that can store, index, manage, and retrieve billions of embedding vectors. It is also one of the most popular vector databases for [LLM retrieval augmented generation](https://zilliz.com/use-cases/llm-retrieval-augmented-generation). As an exemplary instance of purpose-built vector databases, Milvus shares many unique advantages with its counterparts.

### Data Persistence and Cost-Effective Storage

While preventing data loss is the minimum requirement for a database, many single-machine and lightweight vector databases do not prioritize data reliability. By contrast, purpose-built distributed vector databases like [Milvus](https://zilliz.com/what-is-milvus) prioritize system resilience, scalability, and data persistence by separating storage and computation. 

Moreover, most vector databases that utilize approximate nearest neighbor (ANN) indexes need a lot of memory to perform vector searching, as they load ANN indexes purely into memory. However, Milvus supports disk indexes, making storage over ten times more cost-effective than in-memory indexes.

### Optimal Query Performance 

A specialized vector database provides optimal query performance compared to other vector search options. For example, Milvus is ten times faster at handling queries than vector search plugins. Milvus uses the [ANN algorithm](https://zilliz.com/glossary/anns) instead of the KNN brutal search algorithm for faster vector searching. Additionally, it shards its indexes, reducing the time it takes to construct an index as the data volume increases. This approach enables Milvus to easily handle billions of vectors with real-time data additions and deletions. In contrast, other vector search add-ons are only suitable for scenarios with fewer than tens of millions of data and infrequent additions and deletions.

Milvus also supports GPU acceleration. Internal testing shows that GPU-accelerated vector indexing can achieve 10,000+ QPS when searching tens of millions of data, which is at least ten times faster than traditional CPU indexing for single-machine query performance.

### System Reliability 

Many applications use vector databases for online queries that require low query latency and high throughput. These applications demand single-machine failover at the minute level, and some even require cross-region disaster recovery for critical scenarios. Traditional replication strategies based on Raft/Paxos suffer from serious resource waste and need help to pre-shard the data, leading to poor reliability. In contrast, Milvus has a distributed architecture that leverages K8s message queues for high availability, reducing recovery time and saving resources. 

### Operability and Observability

To better serve enterprise users, vector databases must offer a range of enterprise-level features for better operability and observability. Milvus supports multiple deployment methods, including K8s Operator and Helm chart, docker-compose, and pip install, making it accessible to users with different needs. Milvus also provides a monitoring and alarm system based on Grafana, Prometheus, and Loki, improving its observability. With a distributed cloud-native architecture, Milvus is the industry's first vector database to support multi-tenant isolation, RBAC, quota limiting, and rolling upgrades. All of these approaches make managing and monitoring Milvus much simpler.

## Getting started with Milvus in 3 simple steps within 10 minutes 

Building a vector database is a complex task, but using one is as simple as using Numpy and FAISS. Even students unfamiliar with AI can implement vector search based on Milvus in just ten minutes. To experience highly scalable and high-performance vector search services, follow these three steps:

- Deploy Milvus on your server with the help of the [Milvus deployment document](https://milvus.io/docs/install_standalone-docker.md).
- Implement vector search with just 50 lines of code by referring to the [Hello Milvus document](https://milvus.io/docs/example_code.md).
- Explore the [example documents of Towhee](https://github.com/towhee-io/examples/) to gain insight into popular [use cases of vector databases](https://zilliz.com/use-cases).


Why and When Do You Need a Purpose-Built Vector Database?


![](https://assets.zilliz.com/Milvus_2_3_Milvus_io_2e3b0eb55c.jpeg)

Exciting news! After eight months of concerted effort, we're thrilled to announce the release of Milvus 2.3, a milestone version that brings numerous highly anticipated features, including support for GPU, Arm64, upsert, change data capture, ScaNN index, and MMap technology. Milvus 2.3 also introduces improved query performance, more robust load balancing and scheduling, and better observability and operability. 

Join me to look at these new features and enhancements and learn how you can benefit from this release. 

## Support for GPU index that leads to 3-10 times faster in QPS 

GPU index is a highly anticipated feature in the Milvus community. Thanks to a great collaboration with the Nvidia engineers, Milvus 2.3 has supported GPU indexing with the robust RAFT algorithm added to Knowhere, the Milvus index engine. With GPU support, Milvus 2.3 is more than three times faster in QPS than older versions using the CPU HNSW index and almost ten times faster for specific datasets that require heavy computation. 

## Arm64 support to accommodate growing user demand

Arm CPUs are becoming increasingly popular among cloud providers and developers. To meet this growing demand, Milvus now provides Docker images for the ARM64 architecture. With this new CPU support, MacOS users can build their applications with Milvus more seamlessly. 

## Upsert support for better user experience 

Milvus 2.3 introduces a notable enhancement by supporting the upsert operation. This new functionality allows users to update or insert data seamlessly and empowers them to perform both operations in a single request through the Upsert interface. This feature streamlines data management and brings efficiency to the table.

**Note**:
- The upsert feature does not apply to auto-increment IDs.
- Upsert is implemented as a combination of `delete` and `insert`, which may result in some performance loss. We recommend using `insert` if you use Milvus in write-heavy scenarios.

## Range search for more accurate results 
Milvus 2.3 allows users to specify the distance between the input vector and the vectors stored in Milvus during a query. Milvus then returns all matching results within the set range. Below is an example of specifying the search distance using the range search feature.

```
// add radius and range_filter to params in search_params
search_params = {"params": {"nprobe": 10, "radius": 10, "range_filter" : 20}, "metric_type": "L2"}
res = collection.search(
vectors, "float_vector", search_params, topK,
"int64 > 100", output_fields=["int64", "float"]
)
```

In this example, the user requires Milvus to return vectors within a distance of 10 to 20 units from the input vector.

**Note**: Different distance metrics vary in how they calculate distances, resulting in distinct value ranges and sorting strategies. Therefore, it is essential to understand their characteristics before using the range search feature.

## ScaNN index for faster query speed 

Milvus 2.3 now supports the ScaNN index, an open-source [approximate nearest neighbor (ANN)](https://zilliz.com/glossary/anns) index developed by Google. ScaNN index has demonstrated superior performance in various benchmarks, outperforming HNSW by around 20% and being approximately seven times faster than IVFFlat. With the support for the ScaNN index, Milvus achieves much faster query speed compared to older versions.

## Growing index for stable and better query performance 

Milvus includes two categories of data: indexed data and streaming data. Milvus can use indexes to search indexed data quickly but can only brutely search streaming data row by row, which can impact performance. Milvus 2.3 introduces the Growing Index, which automatically creates real-time indexes for streaming data to improve query performance.

## Iterator for data retrieval in batches 

In Milvus 2.3, Pymilvus has introduced an iterator interface that allows users to retrieve more than 16,384 entities in a search or range search. This feature is handy when users need to export tens of thousands or even more vectors in batches.

## Support for MMap for increased capacity

MMap is a UNIX system call used to map files and other objects into memory. Milvus 2.3 supports MMap, which enables users to load data onto local disks and map it to memory, thereby increasing single-machine capacity.

Our testing results indicate that using MMap technology, Milvus can double its data capacity while limiting performance degradation to within 20%. This approach significantly reduces overall costs, making it particularly beneficial for users on a tight budget who do not mind compromising performance.

## CDC support for higher system availability 

Change Data Capture (CDC) is a commonly used feature in database systems that captures and replicates data changes to a designated destination. With the CDC feature, Milvus 2.3 enables users to synchronize data across data centers, back up incremental data, and seamlessly migrate data, making the system more available. 

In addition to the features above, Milvus 2.3 introduces a count interface to accurately calculate the number of rows of data stored in a collection in real-time, supports the Cosine metric to measure vector distance, and more operations on JSON arrays. For more features and detailed information, refer to [Milvus 2.3 release notes](https://milvus.io/docs/release_notes.md). 

## Enhancements and bug fixes 

In addition to new features, Milvus 2.3 includes many improvements and bug fixes for older versions.

### Improved performance for data filtering

Milvus performs scalar filtering before vector searching in hybrid scalar and vector data queries to achieve more accurate results. However, the indexing performance may decline if the user has filtered out too much data after scalar filtering. In Milvus 2.3, we optimized the filtering strategy of HNSW to address this issue, resulting in improved query performance.

### Increased multi-core CPU usage 

Approximate nearest search (ANN) is a computationally intensive task that requires massive CPU resources. In previous releases, Milvus could only utilize around 70% of the available multi-core CPU resources. However, with the latest release, Milvus has overcome this limitation and can fully utilize all available multi-core CPU resources, resulting in improved query performance and reduced resource waste.

### Refactored QueryNode

QueryNode is a crucial component in Milvus that is responsible for vector searching. However, in older versions, QueryNode had complex states, duplicate message queues, an unorganized code structure, and non-intuitive error messages.

In Milvus 2.3, we've upgraded QueryNode by introducing a stateless code structure and removing the message queue for deleting data. These updates result in less resource waste and faster and more stable vector searching.

### Enhanced message queues based on NATS

We built Milvus on a log-based architecture, and in previous versions, we used Pulsar and Kafka as the core log brokers. However, this combination faced three key challenges:

- It was unstable in multi-topic situations.
- It consumed resources when idle and struggled to deduplicate messages.
- Pulsar and Kafka are closely tied to the Java ecosystem, so their community rarely maintains and updates their Go SDKs.

To address these problems, we have combined NATS and Bookeeper as our new log broker for Milvus, which fits users' needs better.

### Optimized load balancer

Milvus 2.3 has adopted a more flexible load-balancing algorithm based on the system's real loads. This optimized algorithm lets users quickly detect node failures and unbalanced loads and adjust schedulings accordingly. According to our testing results, Milvus 2.3 can detect faults, unbalanced load, abnormal node status, and other events within seconds and make adjustments promptly.

For more information about Milvus 2.3, refer to [Milvus 2.3 release notes](https://milvus.io/docs/release_notes.md). 

## Tool upgrades

We have also upgraded Birdwatcher and Attu, two valuable tools for operating and maintaining Milvus, along with Milvus 2.3.

### Birdwatcher update 

We've upgraded [Birdwatcher](https://github.com/milvus-io/birdwatcher), the debug tool of Milvus, introducing numerous features and enhancements, including: 
- RESTful API for seamless integration with other diagnostic systems.
- PProf command support to facilitate integration with the Go pprof tool.
- Storage usage analysis capabilities.
- Efficient log analysis functionality.
- Support for viewing and modifying configurations in etcd.

### Attu update 

We’ve launched a brand-new interface for [Attu](https://zilliz.com/attu), an all-in-one vector database administration tool. The new interface has a more straightforward design and is easier to understand.

![](https://assets.zilliz.com/Attu_s_new_interface_e24dd0d670.png)

For more details, refer to [Milvus 2.3 release notes](https://milvus.io/docs/release_notes.md). 

## Let’s keep in touch!

If you have questions or feedback about Milvus, please don't hesitate to contact us through [Twitter](https://twitter.com/milvusio) or [LinkedIn](https://www.linkedin.com/company/the-milvus-project). You're also welcome to join our [Slack channel](https://milvus.io/slack/) to chat with our engineers and the community directly or check out our [Tuesday office hours](https://us02web.zoom.us/meeting/register/tZ0pcO6vrzsuEtVAuGTpNdb6lGnsPBzGfQ1T#/registration)!


Unveiling Milvus 2.3: A Milestone Release Offering Support for GPU, Arm64, CDC, and Many Other Highly Anticipated Features


![](https://assets.zilliz.com/What_s_New_in_2_2_12_20230720_143424_7d19280738.png)

We are thrilled to announce the latest release of Milvus 2.2.12. This update includes multiple new features, such as support for RESTful API, `json_contains` function, and vector retrieval during ANN searches in response to user feedback. We have also streamlined the user experience, enhanced the vector searching speeds, and resolved many issues. Let's delve into what we can expect from Milvus 2.2.12.

## Support for RESTful API

Milvus 2.2.12 now supports RESTful API, which enables users to access Milvus without installing a client, making client-server operations effortless. Additionally, deploying Milvus has become more convenient because the Milvus SDK and RESTful API share the same port number. 

**Note**: We still recommend using the SDK to deploy Milvus for advanced operations or if your business is latency sensitive. 

## Vector retrieval during ANN searches

In earlier versions, Milvus did not allow vector retrieval during approximate nearest neighbor (ANN) searches to prioritize performance and memory usage. As a result, retrieving raw vectors had to be split into two steps: performing the ANN search and then querying the raw vectors based on their IDs. This approach increased development costs and made it harder for users to deploy and adopt Milvus.

With Milvus 2.2.12, users can retrieve raw vectors during ANN searches by setting the vector field as an output field and querying in HNSW-, DiskANN-, or IVF-FLAT-indexed collections. In addition, users can expect a much faster vector retrieval speed. 

## Support for operations on JSON arrays

We recently added support for JSON in Milvus 2.2.8. Since then, users have sent numerous requests to support additional JSON arrays operations, such as inclusion, exclusion, intersection, union, difference, and more. In Milvus 2.2.12, we've prioritized supporting the `json_contains` function to enable the inclusion operation. We will continue to add support for other operators in future versions.

## Enhancements and bug fixes 

In addition to introducing new features, Milvus 2.2.12 has improved its vector search performance with reduced overhead, making it easier to handle extensive topk searches. Moreover, it enhances the write performance in partition-key-enabled and multi-partition situations and optimizes CPU usage for large machines. 
This update addresses various issues: excessive disk usage, stuck compaction, infrequent data deletions, and bulk insertion failures. For further information, please refer to the [Milvus 2.2.12 Release Notes](https://milvus.io/docs/release_notes.md#2212).

## Let's keep in touch!

If you have questions or feedback about Milvus, please don't hesitate to contact us through [Twitter](https://twitter.com/milvusio) or [LinkedIn](https://www.linkedin.com/company/the-milvus-project). You're also welcome to join our [Slack channel](https://milvus.io/slack/) to chat with our engineers and the community directly or check out our [Tuesday office hours](https://us02web.zoom.us/meeting/register/tZ0pcO6vrzsuEtVAuGTpNdb6lGnsPBzGfQ1T#/registration)!


Milvus 2.2.12: Easier Access, Faster Vector Search Speeds, and Better User Experience 



![](https://assets.zilliz.com/What_s_New_in_Milvus_2_2_10_and_2_2_11_5018946465.png)


Greetings, Milvus fans! We're excited to announce that we have just released Milvus 2.2.10 and 2.2.11, two minor updates primarily focusing on bug fixes and overall performance improvement. You can expect a more stable system and a better user experience with the two updates. Let’s take a quick look at what is new in these two releases. 

## Milvus 2.2.10

Milvus 2.2.10 has fixed occasional system crashes, accelerated loading and indexing, reduced memory usage in data nodes, and made many other improvements. Below are some notable changes: 

- Replaced the old CGO payload writer with a new one written in pure Go, reducing memory usage in data nodes.
- Added `go-api/v2` to the `milvus-proto` file to prevent confusion with different `milvus-proto` versions. 
- Upgraded Gin from version 1.9.0 to 1.9.1 to fix a bug in the `Context.FileAttachment` function.
- Added role-based access control (RBAC) for the FlushAll and Database APIs.
- Fixed a random crash caused by the AWS S3 SDK.
- Improved the loading and indexing speeds. 

For more details, see [Milvus 2.2.10 Release Notes](https://milvus.io/docs/release_notes.md#2210). 

## Milvus 2.2.11

Milvus 2.2.11 has resolved various issues to improve the system's stability. It has also improved its performance in monitoring, logging, rate limiting, and interception of cross-cluster requests. See below for the highlights of this update. 

- Added an interceptor to the Milvus GRPC server to prevent any issues with Cross-Cluster routing.
- Added error codes to the minio chunk manager to make diagnosing and fixing errors easier. 
- Utilized a singleton coroutine pool to avoid wasting coroutines and maximize the use of resources. 
- Reduced the disk usage for RocksMq to one-tenth of its original level by enabling zstd compression.
- Fixed occasional QueryNode panic during loading.
- Rectified the read request throttling issue caused by miscalculating queue length twice.
- Fixed issues with GetObject returning null values on MacOS.
- Fixed a crash caused by incorrect use of the noexcept modifier.

For more details, see [Milvus 2.2.11 Release Notes](https://milvus.io/docs/release_notes.md#2211). 

## Let’s keep in touch!

If you have questions or feedback about Milvus, please don't hesitate to contact us through [Twitter](https://twitter.com/milvusio) or [LinkedIn](https://www.linkedin.com/company/the-milvus-project). You're also welcome to join our [Slack channel](https://milvus.io/slack/) to chat with our engineers and the community directly or check out our [Tuesday office hours](https://us02web.zoom.us/meeting/register/tZ0pcO6vrzsuEtVAuGTpNdb6lGnsPBzGfQ1T#/registration)!



Milvus 2.2.10 & 2.2.11 Enhanced System Stability and User Experience

Milvus 2.2.10 & 2.2.11: Minor Updates for Enhanced System Stability and User Experience


![](https://assets.zilliz.com/Milvus_hit_20_000_stars_284ed694d2.png)

Let's break out the confetti and celebrate! 

[Milvus](https://github.com/milvus-io/milvus) has achieved a stellar milestone - surpassing 20,000 stars on GitHub! We couldn't have achieved it without the support and feedback of our loyal fans and community members. Thank you so much. 

To share the joy, we've put together a list of critical topics from the broader Milvus community, including:
 
- Six prominent projects that utilize Milvus
- Five integrations with other open-source projects
- Five well-known use cases of Milvus
- Five excellent upcoming features

## Six LLM projects that utilize Milvus

### PaperGPT

[PaperGPT](http://papergpt.bio) is a search tool for biomedical papers powered by an LLM and a vector database. It leverages the PubMed database and technologies like SentenceTransformers and Zilliz Cloud for efficient search. 

### NoticeAI

NoticeAI helps companies automate their customer support process. It utilizes Milvus and LangChain to track customer support questions, update the knowledge base, and decide whether or not to involve a human.

### Search.anything.io 

[Search Anything](http://search.anything.io) leverages LLMs and Milvus to provide a better search experience. It takes a text description from a user and finds the corresponding images on the web.

### IkuStudies

[IkuStudies](https://ikustudies.xyz/) is a project allowing users to search through homophones in 28 languages. It uses Milvus as the vector store to find similarities between the words. 

### AssistLink AI

[AssistLink AI](https://www.linkedin.com/company/assistlink/about/) is a Seattle-based startup that uses Milvus and LLaMa 65bn to store variables for a Government Assistance system.

### OSS Chat

[OSS Chat](http://osschat.io) allows you to chat with open-source software. It uses Milvus to inject domain knowledge about open-source projects and uses [GPTCache](https://zilliz.com/blog/Caching-LLM-Queries-for-performance-improvements) to cache frequently asked questions to reduce LLM costs. 

## Five AI projects Milvus integrates with

### LlamaIndex 

[LlamaIndex](https://github.com/jerryjliu/llama_index) is a data framework for LLM applications that provides data connectors to link external data sources with an LLM. LlamaIndex enables you to inject your private data or domain-specific knowledge directly into your LLM application. 

### LangChain/LangChainJS

[LangChain](https://github.com/hwchase17/langchain) is a library designed to accelerate LLM application development, offering features such as prompt management, chains, data-augmented generation, memory, and more. 

### ChatGPT Retrieval Plugin 

The [ChatGPT Retrieval Plugin](https://github.com/openai/chatgpt-retrieval-plugin) from OpenAI enables semantic search and retrieval of personal or organizational documents using natural language queries, allowing developers to deploy their plugin and register it with ChatGPT. 

### Haystack 

[Haystack](https://github.com/deepset-ai/haystack) is an end-to-end NLP framework for building NLP applications powered by LLMs, transformer models, vector search, and other technologies for tasks such as question answering, semantic document search, and building complex decision-making. 

### GPTCache

LLM API calls can be both expensive and slow. [GPTCache](https://github.com/zilliztech/gptcache) can remedy both problems by providing a semantic cache for LLM queries. 

## Five well-known use cases of Milvus

### LLM augmentation

LLMs or LLM applications have three major problems: lack of domain-specific data, outdated data, and high costs. Milvus can solve all three of these problems by allowing you to inject external data and serve as a cache for frequent queries.

### Anomaly detection

Anomalous data is significantly different from the rest in a given set. Vector databases like Milvus are beneficial for conducting similarity searches, making it easier to identify such anomalous data. 

### Recommender systems

A recommender system recommends people items similar to what they already enjoy. It is a classic example of Milvus accelerating this process by performing similarity searches. 

### Semantic search

Keyword search doesn’t do it when you want to search text to find things with similar meanings. That’s where semantic search comes in and where Milvus can help. Milvus allows you to compare the intentions behind the text, not just the words themselves.

### Automatic data labeling

Got labeled data and need to mark more? By storing the vector representations in Milvus, you can automatically detect the most similar data points in your new data and apply the appropriate labels.

## Five awesome upcoming features

### NVIDIA GPU support

GPU support is coming in Milvus 2.3 and is already available in the most [recent beta release](https://github.com/milvus-io/milvus/releases/tag/v2.3.0-beta)! 

### Delete by Expression

You can already perform scalar/metadata filtering via boolean expressions in Milvus. The Delete by Expression feature is also coming.

### Change Data Capture (CDC)

CDC is a technique to capture and track changes made to your Milvus instance in real-time.

### Range Search

Range search, also known in some circles as epsilon search, will allow you to find all vectors within a certain distance of your query vector.

### Fast Scan

4-bit quantization and FastScan were added to FAISS recently and will also be coming to Milvus.

## Looking forward to Milvus reaching even greater heights!

We would like to give our most enormous thanks to our users, community members, ecosystem partners, and stargazers for your continued support, feedback, and contribution. Looking forward to Milvus reaching even greater heights! 




A Stellar Milestone: Milvus Surpasses 20,000 Stars on GitHub


![](https://assets.zilliz.com/Conversational_Memory_in_Lang_Chain_7c1b4b7ba9.png)


LangChain is a robust framework for building LLM applications. However, with that power comes quite a bit of complexity. LangChain provides many ways to prompt an LLM and essential features like conversational memory. Conversational memory offers context for the LLM to remember your chat.

In this post, we look at how to use conversational memory with LangChain and Milvus. To follow along, you need to `pip` install four libraries and an OpenAI API key. The four libraries you need can be installed by running `pip install langchain milvus pymilvus python-dotenv`. Or executing the first block in the [CoLab Notebook](https://colab.research.google.com/drive/11p-u8nKqrQYePlXR0HiSrUapmKLD0QN9?usp=sharing) for this article.

In this post, we’ll learn about:
- Conversational Memory with LangChain
- Setting Up Conversation Context
- Prompting the Conversational Memory with LangChain
- LangChain Conversational Memory Summary

## Conversational Memory with LangChain

In the default state, you interact with an LLM through single prompts. Adding memory for context, or “conversational memory” means you no longer have to send everything through one prompt. LangChain offers the ability to store the conversation you’ve already had with an LLM to retrieve that information later.

To set up persistent conversational memory with a vector store, we need six modules from LangChain. First, we must get the `OpenAIEmbeddings` and the `OpenAI` LLM. We also need `VectorStoreRetrieverMemory` and the LangChain version of `Milvus` to use a vector store backend. Then we need `ConversationChain` and `PromptTemplate` to save our conversation and query it.

The `os`, `dotenv`, and `openai` libraries are mainly for operational purposes. We use them to load and use the OpenAI API key. The final setup step is to spin up a local [Milvus Lite](https://milvus.io/docs/milvus_lite.md) instance. We do this through using the `default_server` from the Milvus Python package.

```
from langchain.embeddings.openai import OpenAIEmbeddings
from langchain.llms import OpenAI
from langchain.memory import VectorStoreRetrieverMemory
from langchain.chains import ConversationChain
from langchain.prompts import PromptTemplate
from langchain.vectorstores import Milvus
embeddings = OpenAIEmbeddings()


import os
from dotenv import load_dotenv
import openai
load_dotenv()
openai.api_key = os.getenv("OPENAI_API_KEY")


from milvus import default_server
default_server.start()
```

## Setting Up Conversation Context

Now that we have all our prerequisites set up, we can proceed to create our conversational memory. Our first step is to create a connection to the Milvus server using LangChain. Next, we use an empty dictionary to create our LangChain Milvus collection. In addition, we pass in the embeddings we created above and the connection details for the Milvus Lite server.

To use the vector database for conversational memory, we need to instantiate it as a retriever. We only retrieve the top 1 result for this case, setting `k=1`. The last conversational memory setup step is using the `VectorStoreRetrieverMemory` object as our conversational memory through the retriever and vector database connection we just set up. 

To use our conversational memory, it has to have some context in it. So let’s give the memory some context. For this example, we give five pieces of information. Let’s store my favorite snack (chocolate), sport (swimming), beer (Guinness), dessert (cheesecake), and musician (Taylor Swift). Each entry is saved to the memory through the `save_context` function.

```
vectordb = Milvus.from_documents(
   {},
   embeddings,
   connection_args={"host": "127.0.0.1", "port": default_server.listen_port})
retriever = Milvus.as_retriever(vectordb, search_kwargs=dict(k=1))
memory = VectorStoreRetrieverMemory(retriever=retriever)
about_me = [
   {"input": "My favorite snack is chocolate",
    "output": "Nice"},
   {"input": "My favorite sport is swimming",
    "output": "Cool"},
   {"input": "My favorite beer is Guinness",
    "output": "Great"},
   {"input": "My favorite dessert is cheesecake",
    "output": "Good to know"},
   {"input": "My favorite musician is Taylor Swift",
    "output": "Same"}
]
for example in about_me:
   memory.save_context({"input": example["input"]}, {"output": example["output"]})
```

## Prompting the Conversational Memory with LangChain

It’s time to look at how we can use our conversational memory. Let’s start by connecting to the OpenAI LLM through LangChain. We use a temperature of 0 to indicate that we don’t want our LLM to be creative. 

Next, we create a template. We tell the LLM that it is engaged in a friendly conversation with a human and inserts two variables. The `history` variable provides the context from the conversational memory. The `input` variable provides the current input. We use the `PromptTemplate` object to insert these variables.

We use the `ConversationChain` object to combine our prompt, LLM, and memory. Now we are ready to check the memory of our conversation by giving it some prompts. We start by telling the LLM that our name is Gary, the main rival in the Pokemon series (everything else in the conversational memory is a fact about me).

```
llm = OpenAI(temperature=0) # Can be any valid LLM
_DEFAULT_TEMPLATE = """The following is a friendly conversation between a human and an AI. The AI is talkative and provides lots of specific details from its context. If the AI does not know the answer to a question, it truthfully says it does not know.


Relevant pieces of previous conversation:
{history}


(You do not need to use these pieces of information if not relevant)


Current conversation:
Human: {input}
AI:"""
PROMPT = PromptTemplate(
   input_variables=["history", "input"], template=_DEFAULT_TEMPLATE
)
conversation_with_summary = ConversationChain(
   llm=llm,
   prompt=PROMPT,
   memory=memory,
   verbose=True
)
conversation_with_summary.predict(input="Hi, my name is Gary, what's up?")
```

The image below shows what an expected response from the LLM could look like. In this example, it has responded by saying its name is “AI”.

![](https://assets.zilliz.com/Conversational_Memory_in_Lang_Chain_graphics_1_2bf386d22a.png)

Now let’s test out the memory so far. We use the `ConversationChain` object we created earlier and query for my favorite musician.

```
conversation_with_summary.predict(input="who is my favorite musician?")
```

The image below shows an expected response from the Conversation Chain. Since we used the verbose option, it also shows us the relevant conversation. We can see that it returns that my favorite artist is Taylor Swift, as expected.

![](https://assets.zilliz.com/Conversational_Memory_in_Lang_Chain_graphics_2_8355206f3e.png)

Next, let’s check for my favorite dessert - cheesecake.

```
conversation_with_summary.predict(input="Whats my favorite dessert?")
```

When we query for my favorite dessert, we can see that the Conversation Chain once again picks the correct information from Milvus. It finds that my favorite dessert is cheesecake, as I told it earlier.

![](https://assets.zilliz.com/Conversational_Memory_in_Lang_Chain_graphics_3_66a5c9690f.png)

Now that we’ve confirmed that we can query for the information we gave earlier, let’s check for one more thing - the information we provided at the beginning of our conversation. We started our conversation by telling the AI that our name was Gary. 

```
conversation_with_summary.predict(input="What's my name?")
```

Our final check yields that the conversation chain stored the bit about our name in our vector store conversational memory. It returns that we said our name is Gary.

![](https://assets.zilliz.com/Conversational_Memory_in_Lang_Chain_graphics_4_f446f49672.png)


## LangChain Conversational Memory Summary

In this tutorial, we learned how to use conversational memory in LangChain. LangChain offers access to vector store backends like Milvus for persistent conversational memory. We can use conversational memory by injecting history into our prompts and saving historical context in the `ConversationChain` object. 

For this example tutorial, we gave the Conversation Chain five facts about me and pretended to be the main rival in Pokemon, Gary. Then, we pinged the Conversation Chain with questions about the a priori knowledge we stored - my favorite musician and dessert. It answered both of these questions correctly and surfaced the relevant entries. Finally, we asked it about our name as given at the beginning of the conversation, and it correctly returned that we said our name was “Gary.”

Conversational Memory in LangChain



![](https://assets.zilliz.com/What_s_New_in_2_2_9_858e54a2d3.png)

We are thrilled to announce the arrival of Milvus 2.2.9, a highly anticipated release that marks a significant milestone for the team and the community. This release offers many exciting features, including long-awaited support for JSON data types, dynamic schema, and partition keys, ensuring an optimized user experience and streamlined development workflow. Additionally, this release incorporates numerous enhancements and bug fixes. Join us in exploring Milvus 2.2.9 and discovering why this release is so exciting.

## Optimized user experience with JSON support

Milvus has introduced highly anticipated support for the JSON data type, allowing for the seamless storage of JSON data alongside the metadata of vectors within users' collections. With this enhancement, users can efficiently insert JSON data in bulk and perform advanced querying and filtering based on their JSON fields' contents. Furthermore, users can leverage expressions and perform operations tailored to their dataset's JSON fields, construct queries, and apply filters based on the content and structure of their JSON fields, allowing them to extract relevant information and manipulate data better.  

In the future, the Milvus team will add indexes for fields within the JSON type, further optimizing the performance of mixed scalar and vector queries. So stay tuned for exciting developments ahead! 

## Added flexibility with support for dynamic schema

With support for JSON data, Milvus 2.2.9 now provides dynamic schema functionality through a simplified software development kit (SDK).

Starting with Milvus 2.2.9, the Milvus SDK includes a high-level API that automatically fills dynamic fields into the hidden JSON field of the collection, allowing users to concentrate solely on their business fields.

## Better data separation and enhanced search efficiency with Partition Key

Milvus 2.2.9 enhances its partitioning capabilities by introducing the Partition Key feature. It allows user-specific columns as primary keys for partitioning, eliminating the need for additional APIs such as `loadPartition` and `releasePartition`. This new feature also removes the limit on the number of partitions, leading to more efficient resource utilization. 

## Support for Alibaba Cloud OSS

Milvus 2.2.9 now supports Alibaba Cloud Object Storage Service (OSS). Alibaba Cloud users can easily configure the `cloudProvider` to Alibaba Cloud and take advantage of seamless integration for efficient storage and retrieval of vector data in the cloud.

In addition to the previously mentioned features, Milvus 2.2.9 offers database support in Role-Based Access Control (RBAC), introduces connection management, and includes multiple enhancements and bug fixes. For more information, refer to [Milvus 2.2.9 Release Notes](https://milvus.io/docs/release_notes.md).

## Let’s keep in touch!

If you have questions or feedback about Milvus, please don't hesitate to contact us through [Twitter](https://twitter.com/milvusio) or [LinkedIn](https://www.linkedin.com/company/the-milvus-project). You're also welcome to join our [Slack channel](https://milvus.io/slack/) to chat with our engineers and the community directly or check out our [Tuesday office hours](https://us02web.zoom.us/meeting/register/tZ0pcO6vrzsuEtVAuGTpNdb6lGnsPBzGfQ1T#/registration)!

Milvus 2.2.9: A Highly Anticipated Release with Optimal User Experience



***Important Note***

_We upgraded Milvus Lite in June 2024, enabling AI developers to build applications faster while ensuring a consistent experience across various deployment options, including Milvus on Kurbernetes, Docker, and managed cloud services. Milvus Lite also integrates with various AI frameworks and technologies, streamlining the development of AI applications with vector search capabilities. For more information, see the following references:_ 

- _Milvus Lite launch blog: ​[https://milvus.io/blog/introducing-milvus-lite.md](https://milvus.io/blog/introducing-milvus-lite.md)_

- _Milvus Lite documentation: [https://milvus.io/docs/quickstart.md](https://milvus.io/docs/quickstart.md)_

- _Milvus Lite GitHub repository: [https://github.com/milvus-io/milvus-lite](https://github.com/milvus-io/milvus-lite)_



<br>


[Milvus](https://github.com/milvus-io/milvus) is an open-source vector database purpose-built to index, store, and query embedding vectors generated by deep neural networks and other machine learning (ML) models at billions of scales. It has become a popular choice for many companies, researchers, and developers who must perform similarity searches on large-scale datasets. 

However, some users may find the full version of Milvus too heavy or complex. To address this problem, [Bin Ji](https://github.com/matrixji), one of the most active contributors in the Milvus community, built [Milvus Lite](https://github.com/milvus-io/milvus-lite), a lightweight version of Milvus. 

## What is Milvus Lite?

As previously mentioned, [Milvus Lite](https://github.com/milvus-io/milvus-lite) is a simplified alternative to Milvus that offers so many advantages and benefits. 

- You can integrate it into your Python application without adding extra weight. 
- It is self-contained and does not require any other dependencies, thanks to the standalone Milvus' ability to work with embedded Etcd and local storage. 
- You can import it as a Python library and use it as a command-line interface (CLI)-based standalone server. 
- It works smoothly with Google Colab and Jupyter Notebook.
- You can safely migrate your work and write code to other Milvus instances (standalone, clustered, and fully-managed versions) without any risk of losing data. 

## When should you use Milvus Lite?

Specifically, Milvus Lite is most helpful in the following situations:
- When you prefer to use Milvus without container techniques and tools like [Milvus Operator](https://milvus.io/docs/install_standalone-operator.md), [Helm](https://milvus.io/docs/install_standalone-helm.md), or [Docker Compose](https://milvus.io/docs/install_standalone-docker.md).
- When you don't require virtual machines or containers for using Milvus.
- When you want to incorporate Milvus features into your Python applications.
- When you want to spin up a Milvus instance in Colab or Notebook for a quick experiment. 

**Note**: We do not recommend using Milvus Lite in any production environment or if you require high performance, strong availability, or high scalability. Instead, consider using [Milvus clusters](https://github.com/milvus-io/milvus) or [fully-managed Milvus on Zilliz Cloud](https://zilliz.com/cloud) for production.

## How to get started with Milvus Lite?

Now, let’s take a look at how to install, configure, and use Milvus Lite. 
### Prerequisites 

To use Milvus Lite, please ensure you have completed the following requirements:
- Installed Python 3.7 or a later version.
- Using one of the verified operating systems listed below:
  - Ubuntu >= 18.04 (x86_64)
  - CentOS >= 7.0 (x86_64)
  - MacOS >= 11.0 (Apple Silicon)

**Notes**: 

1. Milvus Lite uses `manylinux2014` as the base image, making it compatible with most Linux distributions for Linux users. 
2. Running Milvus Lite on Windows is also possible, although this has yet to be fully verified.

### Install Milvus Lite

Milvus Lite is available on PyPI so you can install it via `pip`.

```
$ python3 -m pip install milvus
```

You can also install it with PyMilvus as follows:

```
$ python3 -m pip install milvus[client]
```

### Use and start Milvus Lite

Download the [example notebook](https://github.com/milvus-io/milvus-lite/tree/main/examples) from our project repository's example folder. You have two options for using Milvus Lite: either import it as a Python library or run it as a standalone server on your machine using the CLI.

- To start Milvus Lite as a Python module, execute the following commands:

```
from milvus import default_server
from pymilvus import connections, utility

# Start your milvus server
default_server.start()

# Now you can connect with localhost and the given port
# Port is defined by default_server.listen_port
connections.connect(host='127.0.0.1', port=default_server.listen_port)

# Check if the server is ready.
print(utility.get_server_version())

# Stop your milvus server
default_server.stop()
```

- To suspend or stop Milvus Lite, use the `with` statement. 

```
from milvus import default_server

with default_server:
  # Milvus Lite has already started, use default_server here.
  connections.connect(host='127.0.0.1', port=default_server.listen_port)
```

- To start Milvus Lite as a CLI-based standalone server, run the following command: 

```
milvus-server
```

After you start Milvus Lite, you can use PyMilvus or other tools you prefer to connect to the standalone server.

### Start Milvus Lite in a debug mode

- To run Milvus Lite in a debug mode as a Python Module, execute the following commands: 

```
from milvus import debug_server, MilvusServer

debug_server.run()

# Or you can create a MilvusServer by yourself
# server = MilvusServer(debug=True)
```

- To run the standalone server in a debug mode, execute the following command:

```
milvus-server --debug
```

### Persist data and logs

- To create a local directory for Milvus Lite that will contain all relevant data and logs, execute the following commands:

```
from milvus import default_server

with default_server:
  default_server.set_base_dir('milvus_data')
```

- To persist all data and logs generated by the standalone server on your local drive, run the following command: 

```
$ milvus-server --data milvus_data
```

### Configure Milvus Lite

Configuring Milvus Lite is similar to setting up Milvus instances using Python APIs or CLI.

- To configure Milvus Lite using Python APIs, use the `config.set` API of a `MilvusServer` instance for both the basic and extra settings: 

```
from milvus import default_server

with default_server:
  default_server.config.set('system_Log_level', 'info')
  default_server.config.set('proxy_port', 19531)
  default_server.config.set('dataCoord.segment.maxSize', 1024)
```

- To configure Milvus Lite using CLI, run the following command for basic settings:

```
$ milvus-server --system-log-level info
$ milvus-server --proxy-port 19531
```

- Or, run the following for extra configurations.

```
$ milvus-server --extra-config dataCoord.segment.maxSize=1024
```

All configurable items are in the `config.yaml` template shipped with the Milvus package. 

For more technical details on how to install and configure Milvus Lite, see our [documentation](https://milvus.io/docs/milvus_lite.md#Prerequisites). 

## Summary 

Milvus Lite is an excellent choice for those seeking the capabilities of Milvus in a compact format. Whether you are a researcher, developer, or data scientist, it's worth exploring this option.

Milvus Lite is also a beautiful addition to the open-source community, showcasing the extraordinary work of its contributors. Thanks to Bin Ji's efforts, Milvus is now available to more users. We cannot wait to see the innovative ideas that Bin Ji and other members of the Milvus community will bring forth in the future.

## Let’s keep in touch!

If you encounter problems installing or using Milvus Lite, you can [file an issue here](https://github.com/milvus-io/milvus-lite/issues/new) or contact us through [Twitter](https://twitter.com/milvusio) or [LinkedIn](https://www.linkedin.com/company/the-milvus-project). You're also welcome to join our [Slack channel](https://milvus.io/slack/) to chat with our engineers and the entire community, or check out [our Tuesday office hours](https://us02web.zoom.us/meeting/register/tZ0pcO6vrzsuEtVAuGTpNdb6lGnsPBzGfQ1T#/registration)!


Introducing Milvus Lite: the Lightweight Version of Milvus



![Milvus 2.2.8](https://assets.zilliz.com/What_s_New_in_2_2_8_f4dd6de0f2.png)


We are excited to announce our latest release of Milvus 2.2.8. This release includes numerous improvements and bug fixes from previous versions, resulting in better querying performance, resource-saving, and higher throughputs. Let's take a look at what's new in this release together. 

## Reduced peak memory consumption during collection loading

To perform queries, Milvus needs to load data and indexes into memory. However, during the loading process, multiple memory copies can cause the peak memory usage to increase up to three to four times higher than during actual runtime. The latest version of Milvus 2.2.8 effectively addresses this issue and optimizes memory usage.

## Expanded querying scenarios with QueryNode supporting plugins

QueryNode now supports plugins in the latest Milvus 2.2.8. You can easily specify the path of the plugin file with the `queryNode.soPath` configuration. Then, Milvus can load the plugin at runtime and expand the available querying scenarios. Refer to the [Go plugin documentation](https://pkg.go.dev/plugin), if you need guidance on developing plugins. 

## Optimized querying performance with enhanced compaction algorithm

The compaction algorithm determines the speed at which the segments can converge, directly affecting the querying performance. With the recent improvements to the compaction algorithm, the convergence efficiency has dramatically improved, resulting in faster queries. 

## Better resource saving and querying performance with reduced collection shards

Milvus is a massively parallel processing (MPP) system, which means that the number of collection shards impacts Milvus’ efficiency in writing and querying. In older versions, a collection had two shards by default, which resulted in excellent writing performance but compromised querying performance and resource cost. With the new Milvus 2.2.8 update, the default collection shards have been reduced to one, allowing users to save more resources and perform better queries. Most users in the community have less than 10 million data volumes, and one shard is sufficient to achieve good writing performance.

**Note**: This upgrade does not affect collections created before this release. 

## 20% throughput increase with an improved query grouping algorithm 
Milvus has an efficient query grouping algorithm that combines multiple query requests in the queue into one for faster execution, significantly improving throughput. In the latest release, we make additional enhancements to this algorithm, increasing Milvus' throughput by at least 20%.

In addition to the mentioned improvements, Milvus 2.2.8 also fixes various bugs. For more details, see [Milvus Release Notes](https://milvus.io/docs/release_notes.md). 

## Let’s keep in touch!
If you have questions or feedback about Milvus, please don't hesitate to contact us through [Twitter](https://twitter.com/milvusio) or [LinkedIn](https://www.linkedin.com/company/the-milvus-project). You're also welcome to join our [Slack channel](https://milvus.io/slack/) to chat with our engineers and the entire community directly or check out our [Tuesday office hours](https://us02web.zoom.us/meeting/register/tZ0pcO6vrzsuEtVAuGTpNdb6lGnsPBzGfQ1T#/registration)!


Milvus 2.2.8: Better Query Performance, 20% Higher Throughput 



![Milvus at Its Best](https://assets.zilliz.com/exploring_milvus_latest_versions_4fa890533e.png)

Welcome back, Milvus followers! We know it's been a while since we last shared our updates on this cutting-edge open-source vector database. But fear not, because we’re here to catch you up on all the exciting developments that have taken place since last August.

In this blog post, we'll take you through the latest Milvus releases, spanning from version 2.2 to version 2.2.6. We have much to cover, including new features, improvements, bug fixes, and optimizations. So, fasten your seatbelts, and let’s dive in!

## Milvus v2.2: a major release with enhanced stability, faster search speed, and flexible scalability

Milvus v2.2 is a significant release that introduces seven brand-new features and numerous breakthrough improvements to previous versions. Let's take a closer look at some of the highlights:

* **Bulk Inserts of Entities from Files**: With this feature, you can upload a batch of entities in one or multiple files directly to Milvus with just a few lines of code, saving you time and effort.
* **Query Result Pagination**: To avoid massive search and query results returning in a single remote procedure call (RPC), Milvus v2.2 allows you to configure offset and filter results with keywords in searches and queries. 
* **Role-Based Access Control (RBAC)**: Milvus v2.2 now supports RBAC, allowing you to control access to your Milvus instance by managing users, roles, and permissions. 
* **Quotas and Limits**: Quotas and limits is a new mechanism in Milvus v2.2 that protects the database system from out-of-memory (OOM) errors and crashes during sudden traffic surges. With this feature, you can control ingestion, search, and memory usage.
* **Time to Live (TTL) at a Collection Level**: In previous releases, Milvus only allowed you to configure TTL for your clusters. However, Milvus v2.2 now supports configuring TTL at the collection level. Configuring TTL for a specific collection and entities in that collection will automatically expire after the TTL ends. This configuration provides more fine-grained control over data retention.
* **Disk-Based Approximate Nearest Neighbor Search (ANNS) Indexes (Beta)**: Milvus v2.2 introduces support for DiskANN, an SSD-resident, and Vamana graph-based ANNS algorithm. This support allows for direct searching on large-scale datasets, which can significantly reduce memory usage, by up to 10 times.
* **Data Backup (Beta)**: Milvus v2.2 provides [a brand new tool](https://github.com/zilliztech/milvus-backup) for backing up and restoring your Milvus data properly, either through a command line or an API server. 

In addition to the new features mentioned above, Milvus v2.2 includes fixes for five bugs and multiple improvements to enhance Milvus' stability, observability, and performance. For more details, see [Milvus v2.2 Release Notes](https://milvus.io/docs/release_notes.md#v220). 

## Milvus v2.2.1 & v2.2.2: minor releases with issues fixed

Milvus v2.2.1 and v2.2.2 are minor releases focusing on fixing critical issues in older versions and introducing new features. Here are some highlights:

### Milvus v2.2.1

* Supports Pulsa tenant and authentication
* Supports transport layer security (TLS) in the etcd config source
* Improves search performance by over 30%
* Optimizes the scheduler and increases the probability of merge tasks
* Fixes multiple bugs, including term filtering failures on indexed scalar fields and IndexNode panic upon failures to create an index

### Milvus v2.2.2

* Fixes the issue that the proxy doesn't update the cache of shard leaders
* Fixes the issue that the loaded info is not cleaned for released collections/partitions
* Fixes the issue that the load count is not cleared on time

For more details, see [Milvus v2.2.1 Release Notes](https://milvus.io/docs/release_notes.md#v221) and [Milvus v2.2.2 Release Notes](https://milvus.io/docs/release_notes.md#v222). 

## Milvus v2.2.3: more secure, stable, and available 

Milvus v2.2.3 is a release that focuses on enhancing the system’s security, stability, and availability. In addition, it introduces two important features:

* **Rolling upgrade**: This feature allows Milvus to respond to incoming requests during the upgrade process, which was impossible in previous releases. Rolling upgrades ensure the system remains available and responsive to user requests even during upgrades.

* **Coordinator high availability (HA)**: This feature enables Milvus coordinators to work in an active-standby mode, reducing the risk of single-point failures. Even in unexpected disasters, the recovery time is reduced to at most 30 seconds. 

In addition to these new features, Milvus v2.2.3 includes numerous improvements and bug fixes, including enhanced bulk insert performance, reduced memory usage, optimized monitoring metrics, and improved meta-storage performance. For more details, see [Milvus v2.2.3 Release Notes](https://milvus.io/docs/release_notes.md#v223). 

## Milvus v2.2.4: faster, more reliable and resource saving

Milvus v2.2.4 is a minor update to Milvus v2.2. It introduces four new features and several enhancements, resulting in faster performance, improved reliability, and reduced resource consumption. The highlights of this release include:

* **Resource grouping**: Milvus now supports grouping QueryNodes into other resource groups, allowing for complete isolation of access to physical resources in different groups. 
* **Collection renaming**: The Collection-renaming API allows users to change the name of a collection, providing more flexibility in managing collections and improving usability.
* **Support for Google Cloud Storage**
* **New option in search and query APIs**: This new feature allows users to skip search on all growing segments, offering better search performance in scenarios where the search is performed concurrently with data insertion. 

For more information, see [Milvus v2.2.4 Release Notes](https://milvus.io/docs/release_notes.md#v224). 

## Milvus v2.2.5: NOT RECOMMENDED

Milvus v2.2.5 has several critical issues, and therefore, we do not recommend using this release.  We sincerely apologize for any inconvenience caused by them. However, these issues have been addressed in Milvus v2.2.6.

## Milvus v2.2.6: resolves critical issues from v2.2.5 

Milvus v2.2.6 has successfully addressed the critical issues discovered in v2.2.5, including problems with recycling dirty binlog data and the DataCoord GC failure. If you currently use v2.2.5, please upgrade it to ensure optimal performance and stability.

Critical issues fixed include: 

* DataCoord GC failure
* Override of passed index parameters 
* System delay caused by RootCoord message backlog
* Inaccuracy of metric RootCoordInsertChannelTimeTick 
* Possible timestamp stop
* Occasional coordinator role self-destruction during the restart process 
* Checkpoints falling behind due to abnormal exit of garbage collection 

For more details, see [Milvus v2.2.6 Release Notes](https://milvus.io/docs/release_notes.md#v226). 

## Summary 

In conclusion, the latest Milvus releases from v2.2 to v2.2.6 have delivered many exciting updates and improvements. From new features to bug fixes and optimizations, Milvus continues to meet its commitments to provide cutting-edge solutions and empower applications in various domains. Stay tuned for more exciting updates and innovations from the Milvus community. 



Milvus at Its Best: Exploring v2.2 to v2.2.6



![Cover image](https://assets.zilliz.com/Dynamically_Change_Log_Levels_in_the_Milvus_Vector_Database_58e31c66cc.png "Dynamically Change Log Levels in the Milvus Vector Database.")

> This article is written by [Enwei Jiao](https://github.com/jiaoew1991) and translated by [Angela Ni](https://www.linkedin.com/in/yiyun-n-2aa713163/).

To prevent an over-output of logs from affecting disk and system performance, Milvus by default outputs logs at the `info` level while running. However, sometimes logs at the `info` level are not sufficient enough to help us efficiently identify bugs and issues. What's worse, in some cases, changing the log level and restarting the service might lead to the failure of reproducing the issues, making troubleshooting all the more difficult. Consequently, the support for changing log levels dynamically in the Milvus vector database is urgently needed.

This article aims to introduce the mechanism behind that enables changing log levels dynamically and provide instructions on how to do so in the Milvus vector database.

**Jump to:**
- [Mechanism](#Mechanism)
- [How to dynamically change log levels](#How-to-dynamically-change-log-levels)

## Mechanism

The Milvus vector database adopts the [zap](https://github.com/uber-go/zap) logger open sourced by Uber. As one the most powerful log components in the Go language ecosystem, zap incorporates an [http_handler.go](https://github.com/uber-go/zap/blob/master/http_handler.go) module so that you can view the current log level and dynamically change the log level via an HTTP interface.

Milvus listens the HTTP service provided by the `9091` port. Therefore, you can access the `9091` port to take advantage such features as performance debugging, metrics, health checks. Similarly, the `9091` port is reused to enable dynamic log level modification and a `/log/level` path is also added to the port. See the[ log interface PR](https://github.com/milvus-io/milvus/pull/18430) for more information.

## How to dynamically change log levels

This section provides instructions on how to dynamically change log levels without the need the restarting the running Milvus service.

### Prerequisite

Ensure that you can access the `9091` port of Milvus components.

### Change the log level

Suppose the IP address of the Milvus proxy is `192.168.48.12`. 

You can first run `$ curl -X GET 192.168.48.12:9091/log/level` to check the current log level of the proxy.

Then you can make adjustments by specifying the log level. Log level options include: 

- `debug`

- `info`

- `warn`

- `error`

- `dpanic`

- `panic`

- `fatal`

The following example code changes the log level from the default log level from `info` to `error`. 

```Python
$ curl -X PUT 192.168.48.12:9091/log/level -d level=error
```













Dynamically Change Log Levels in the Milvus Vector Database


![Cover_image](https://assets.zilliz.com/1280_X1280_0e0d4bc107.png "Understanding Consistency Level in the Milvus Vector Database")

> This article is written by [Jiquan Long](https://github.com/longjiquan) and transcreated by [Angela Ni](https://www.linkedin.com/in/yiyun-n-2aa713163/).

In the [previous blog](https://milvus.io/blog/understanding-consistency-levels-in-the-milvus-vector-database.md) about consistency, we have explained what is the connotation of consistency in a distributed vector database, covered the four levels of consistency - strong, bounded staleness, session, and eventual supported in the Milvus vector database, and explained the best-suited application scenario of each consistency level. 

In this post, we will continue to examine the mechanism behind that enables users of the Milvus vector database to flexibly choose the ideal consistency level for various application scenarios. We will also provide the basic tutorial on how to tune consistency level in the Milvus vector database.

**Jump to:**

- [The underlying time tick mechanism](#The-underlying-time-tick-mechanism)
- [Guarantee timestamp](#Guarantee-timestamp)
- [Consistency levels](#Consistency-levels)
- [How to tune consistency level in Milvus?](#How-to-tune-consistency-level-in-Milvus)

## The underlying time tick mechanism

Milvus uses the time tick mechanism to ensure different levels of consistency when a vector search or query is conducted. Time Tick is the watermark of Milvus which acts like a clock in Milvus and signifies at which point of time is the Milvus system in. Whenever there is a data manipulation language (DML) request sent to the Milvus vector database, it assigns a timestamp to the request. As shown in the figure below, when new data are inserted into the message queue for instance, Milvus not only marks a timestamp on these inserted data, but also inserts time ticks at a regular interval. 

![timetick](https://assets.zilliz.com/timetick_b395df9804.png "Timestamp and time tick.")

Let's take `syncTs1` in the figure above as an example. When downstream consumers like query nodes see `syncTs1`, the consumer components understand that all data which are inserted earlier than `syncTs1` has been consumed. In other words, the data insertion requests whose timestamp values are smaller than `syncTs1` will no longer appear in the message queue.

## Guarantee Timestamp

As mentioned in the previous section, downstream consumer components like query nodes continuously obtains messages of data insertion requests and time tick from the message queue. Every time a time tick is consumed, the query node will mark this consumed time tick as the serviceable time - `ServiceTime` and all data inserted before `ServiceTime` are visible to the query node.

In addition to `ServiceTime`, Milvus also adopts a type of timestamp - guarantee timestamp (`GuaranteeTS`) to satisfy the need for various levels of consistency and availability by different users. This means that users of the Milvus vector datbase can specify `GuaranteeTs` in order to inform query nodes that all the data before `GuaranteeTs` should be visible and involved when a search or query is conducted.

There are usually two scenarios when the query node executes a search request in the Milvus vector database.

### Scenario 1: Execute search request immediately

As shown in the figure below, if `GuaranteeTs` is smaller than `ServiceTime`, query nodes can execute the search request immediately.

![execute_immediately](https://assets.zilliz.com/execute_immediately_dd1913775d.png "Execute search request immediately.")

### Scenario 2: Wait till "ServiceTime > GuaranteeTs"

If `GuaranteeTs` is greater than `ServiceTime`, query nodes must continue to consume time tick from the message queue. Search requests cannot be executed until `ServiceTime` is greater than `GuaranteeTs`.

![wait_search](https://assets.zilliz.com/wait_search_f09a2f6cf9.png "Wait till ServiceTime > GuaranteeTs.")

## Consistency Levels

Therefore, the `GuaranteeTs` is configurable in the search request to achieve the level of consistency specified by you. A `GuaranteeTs` with a large value ensures [strong consistency](https://milvus.io/blog/understanding-consistency-levels-in-the-milvus-vector-database.md#Strong) at the cost of a high search latency. And a `GuaranteeTs` with a small value reduces search latency but the data visibility is compromised.

`GuaranteeTs` in Milvus is a hybrid timestamp format. And the user has no idea of the [TSO](https://github.com/milvus-io/milvus/blob/master/docs/design_docs/20211214-milvus_hybrid_ts.md) inside Milvus. Therefore, Specifying the value of`GuaranteeTs` is a much too complicated task for users. To save the trouble for users and provide an optimal user experience, Milvus only requires the users to choose the specific consistency level, and the Milvus vector database will automatically handle the `GuaranteeTs` value for users. That is to say, the Milvus user only needs to choose from the four consistency levels: `Strong`, `Bounded`, `Session`, and `Eventually`. And each of the consistency level corresponds to a certain `GuaranteeTs` value.

The figure below illustrates the `GuaranteeTs` for each of the four levels of consistency in the Milvus vector database.

![guarantee_ts](https://assets.zilliz.com/guarantee_ts_f4b3e119d3.png "The corresponding GuaranteeTs for the four consistency levels in Milvus.")

The Milvus vector database supports four levels of consistency:

- `CONSISTENCY_STRONG`: `GuaranteeTs` is set to the same value as the latest system timestamp, and query nodes wait until the service time proceeds to the latest system timestamp to process the search or query request. 

- `CONSISTENCY_EVENTUALLY`: `GuaranteeTs` is set to a value insignificantly smaller than the latest system timestamp in order to skip the consistency check. Query nodes search immediately on the existing data view. 

- `CONSISTENCY_BOUNDED`: `GuaranteeTs` is set to a value relatively smaller than the latest system timestamp, and query nodes search on a tolerably less updated data view. 

- `CONSISTENCY_SESSION`: The client uses the timestamp of the last write operation as the `GuaranteeTs` so that each client can at least retrieve the data inserted by itself. 

## How to tune consistency level in Milvus?

Milvus supports tuning the consistency level when [creating a collection](https://milvus.io/docs/v2.1.x/create_collection.md) or conducting a [search](https://milvus.io/docs/v2.1.x/search.md) or [query](https://milvus.io/docs/v2.1.x/query.md).  

### Conduct a vector similarity search

To conduct a vector similarity search with the level of consistency you want, simply set the value for the parameter `consistency_level` as either `Strong`, `Bounded`, `Session`, or `Eventually`. If you do not set the value for the parameter `consistency_level`, the consistency level will be `Bounded` by default. The example conducts a vector similarity search with `Strong` consistency.  

```
results = collection.search(
        data=[[0.1, 0.2]], 
        anns_field="book_intro", 
        param=search_params, 
        limit=10, 
        expr=None,
        consistency_level="Strong"
)
```

### Conduct a vector query

Similar to conducting a vector similarity search, you can specify the value for the parameter `consistency_level` when conducting a vector query. The example conducts a vector query with `Strong` consistency.  

```
res = collection.query(
  expr = "book_id in [2,4,6,8]", 
  output_fields = ["book_id", "book_intro"],
  consistency_level="Strong"
)
```

## What's next

With the official release of Milvus 2.1, we have prepared a series of blogs introducing the new features. Read more in this blog series:

- [How to Use String Data to Empower Your Similarity Search Applications](https://milvus.io/blog/2022-08-08-How-to-use-string-data-to-empower-your-similarity-search-applications.md)
- [Using Embedded Milvus to Instantly Install and Run Milvus with Python](https://milvus.io/blog/embedded-milvus.md)
- [Increase Your Vector Database Read Throughput with In-Memory Replicas](https://milvus.io/blog/in-memory-replicas.md)
- [Understanding Consistency Level in the Milvus Vector Database](https://milvus.io/blog/understanding-consistency-levels-in-the-milvus-vector-database.md)
- [Understanding Consistency Level in the Milvus Vector Database (Part II)](https://milvus.io/blog/understanding-consistency-levels-in-the-milvus-vector-database-2.md)
- [How Does the Milvus Vector Database Ensure Data Security?](https://milvus.io/blog/data-security.md)





Understanding Consistency Level in the Milvus Vector Database - Part II


![Cover image](https://assets.zilliz.com/Security_192e35a790.png "How Does the Milvus Vector Database Ensure Data Security?")

In full consideration of your data security, user authentication and transport layer security (TLS) connection are now officially available in Milvus 2.1. Without user authentication, anyone can access all data in your vector database with SDK. However, starting from Milvus 2.1, only those with a valid username and password can access the Milvus vector database. In addition, in Milvus 2.1 data security is further protected by TLS, which ensures secure communications in a computer network. 

This article aims to analyze how Milvus the vector database ensures data security with user authentication and TLS connection and explain how you can utilize these two features as a user who wants to ensure data security when using the vector database.


**Jump to:**

- [What is database security and why is it important?](#What-is-database-security-and-why-is-it-important)
- [How does the Milvus vector database ensure data security?](#How-does-the-Milvus-vector-database-ensure-data-security)
  - [User authentication](#User-authentication)
  - [TLS connection](#TLS-connection)

## What is database security and why is it important?

Database security refers to the measures taken to ensure that all data in the database are safe and kept confidential. Recent data breach and data leak cases at [Twitter, Marriott, and Texas Department of Insurance, etc](https://firewalltimes.com/recent-data-breaches/) makes us all the more vigilant to the issue of data security. All these cases constantly remind us that companies and businesses can suffer from severe loss if the data are not well protected and the databases they use are secure. 

## How does the Milvus vector database ensure data security?

In the current release of 2.1, the Milvus vector database attempts to ensure database security via authentication and encryption. More specifically, on the access level, Milvus supports basic user authentication to control who can access the database. Meanwhile, on the database level, Milvus adopts the transport layer security (TLS) encryption protocol to protect data communication.

### User authentication

The basic user authentication feature in Milvus supports accessing the vector database using a username and password for the sake of data security. This means clients can only access the Milvus instance upon providing an authenticated username and password. 

#### The authentication workflow in the Milvus vector database

All gRPC requests are handled by the Milvus proxy, hence authentication is completed by the proxy. The workflow of logging in with the credentials to connect to the Milvus instance is as follows. 

1. Create credentials for each Milvus instance and the encrypted passwords are stored in etcd. Milvus uses [bcrypt](https://golang.org/x/crypto/bcrypt) for encryption as it implements Provos and Mazières's [adaptive hashing algorithm](http://www.usenix.org/event/usenix99/provos/provos.pdf).
2. On the client side, SDK sends ciphertext when connecting to the Milvus service. The base64 ciphertext (<username>:<password>) is attached to the metadata with the key `authorization`.
3. The Milvus proxy intercepts the request and verifies the credentials.
4. Credentials are cached locally in the proxy.
  

![authetication_workflow](https://assets.zilliz.com/1280_X1280_021e90e3c8.jpeg "Authentication workflow.")
  
When the credentials are updated, the system workflow in Milvus is as follows
1. Root coord is in charge of the credentials when insert, query, delete APIs are called.
2. When you update the credentials because you forget the password for instance, the new password is persisted in etcd. Then all the old credentials in the proxy's local cache are invalidated.
3. The authentication interceptor looks for the records from local cache first. If the credentials in the cache is not correct, the RPC call to fetch the most updated record from root coord will be triggered. And the credentials in the local cache are updated accordingly. 


![credential_update_workflow](https://assets.zilliz.com/update_5af81a4173.jpeg "Credentials update workflow.")
  
  
#### How to manage user authentication in the Milvus vector database

To enable authentication, you need to first set `common.security.authorizationEnabled` to `true` when configuring Milvus in the `milvus.yaml` file.

Once enabled, a root user will be created for the Milvus instance. This root user can use the initial password of `Milvus` to connect to the Milvus vector database. 

```
from pymilvus import connections
connections.connect(
    alias='default',
    host='localhost',
    port='19530',
    user='root_user',
    password='Milvus',
)
```

We highly recommend changing the password of the root user when starting Milvus for the first time.

Then root user can further create more new users for authenticated access by running the following command to create new users.

```
from pymilvus import utility
utility.create_credential('user', 'password', using='default') 
```

There are two things to remember when creating new users:

1. As for the new username, it can not exceed 32 characters in length and must start with a letter. Only underscores, letters, or numbers are allowed in the username. For example a username of "2abc!" is not accepted.

2. As for the password, its length should be 6-256 characters.

Once the new credential is set up, the new user can connect to the Milvus instance with the username and password.

```
from pymilvus import connections
connections.connect(
    alias='default',
    host='localhost',
    port='19530',
    user='user',
    password='password',
)
```

Like all authentication processes, you do not have to worry if you forget the password. The password for an existing user can be reset with the following command.

```
from pymilvus import utility
utility.reset_password('user', 'new_password', using='default')
```

Read the [Milvus documentation](https://milvus.io/docs/v2.1.x/authenticate.md) to learn more about user authentication.

### TLS connection

Transport layer security (TLS) is a type of authentication protocol to provide communications security in a computer network. TLS uses certificates to provide authentication services between two or more communicating parties.

#### How to enable TLS in the Milvus vector database

To enable TLS in Milvus, you need to first run the following command to perpare two files for generating the certificate: a default OpenSSL configuration file named `openssl.cnf` and a file named `gen.sh` used to generate relevant certificates.

```
mkdir cert && cd cert
touch openssl.cnf gen.sh
```

Then you can simply copy and paste the configuration we provide [here](https://milvus.io/docs/v2.1.x/tls.md#Create-files) to the two files. Or you can also make modifications based on our configuration to better suit your application.

When the two files are ready, you can run the `gen.sh` file to create nine certificate files. Likewise, you can also modify the configurations in the nine certificate files to suit your need.

```
chmod +x gen.sh
./gen.sh
```

There is one final step before you can connect to the Milvus service with TLS. You have to set `tlsEnabled` to `true` and configure the file paths of `server.pem`, `server.key`, and `ca.pem` for the server in `config/milvus.yaml`. The code below is an example.

```
tls:
  serverPemPath: configs/cert/server.pem
  serverKeyPath: configs/cert/server.key
  caPemPath: configs/cert/ca.pem

common:
  security:
    tlsEnabled: true
```

Then you are all set and can connect to the Milvus service with TLS as long as you specify the file paths of `client.pem`, `client.key`, and `ca.pem` for the client when using the Milvus connection SDK. The code below is also an example.

```
from pymilvus import connections

_HOST = '127.0.0.1'
_PORT = '19530'

print(f"\nCreate connection...")
connections.connect(host=_HOST, port=_PORT, secure=True, client_pem_path="cert/client.pem",
                        client_key_path="cert/client.key",
                        ca_pem_path="cert/ca.pem", server_name="localhost")
print(f"\nList connections:")
print(connections.list_connections())
```
  
## What's next

With the official release of Milvus 2.1, we have prepared a series of blogs introducing the new features. Read more in this blog series:

- [How to Use String Data to Empower Your Similarity Search Applications](https://milvus.io/blog/2022-08-08-How-to-use-string-data-to-empower-your-similarity-search-applications.md)
- [Using Embedded Milvus to Instantly Install and Run Milvus with Python](https://milvus.io/blog/embedded-milvus.md)
- [Increase Your Vector Database Read Throughput with In-Memory Replicas](https://milvus.io/blog/in-memory-replicas.md)
- [Understanding Consistency Level in the Milvus Vector Database](https://milvus.io/blog/understanding-consistency-levels-in-the-milvus-vector-database.md)
- [Understanding Consistency Level in the Milvus Vector Database (Part II)](https://milvus.io/blog/understanding-consistency-levels-in-the-milvus-vector-database-2.md)
- [How Does the Milvus Vector Database Ensure Data Security?](https://milvus.io/blog/data-security.md)




How Does the Milvus Vector Database Ensure Data Security?


![Cover_image](https://assets.zilliz.com/1280_X1280_0e0d4bc107.png "Understanding Consistency Level in the Milvus Vector Database")

> This article is written by [Chenglong Li](https://github.com/JackLCL) and transcreated by [Angela Ni](https://www.linkedin.com/in/yiyun-n-2aa713163/).

Have you ever wondered why sometimes the data you have deleted from the Mlivus vector database still appear in the search results? 

A very likely reason is that you have not set the appropriate consistency level for your application. Consistency level in a distributed vector database is critical as it determines at which point a particular data write can be read by the system. 

Therefore, this article aims to demystify the concept of consistency and delve into the levels of consistency supported by the Milvus vector database.


**Jump to:**
- [What is consistency](#What-is-consistency)
- [Four levels of consistency in the Milvus vector database](#Four-levels-of-consistency-in-the-Milvus-vector-database)
  - [Strong](#Strong)
  - [Bounded staleness](#Bounded-staleness)
  - [Session](#Session)
  - [Eventual](#Eventual)


## What is consistency

Before getting started, we need to first clarify the connotation of consistency in this article as the word "consistency" is an overloaded term in the computing industry. Consistency in a distributed database specifically refers to the property that ensures every node or replica has the same view of data when writing or reading data at a given time. Therefore, here we are talking about consistency as in the [CAP theorem](https://en.wikipedia.org/wiki/CAP_theorem). 

For serving massive online businesses in the modern world, multiple replicas are commonly adopted. For instance, online e-commerce giant Amazon replicates its orders or SKU data across multiple data centers, zones, or even countries to ensure high system availability in the event of a system crash or failure. This poses a challenge to the system - data consistency across multiple replicas. Without consistency, it is very likely that the deleted item in your Amazon cart reappears, causing very bad user experience. 

Hence, we need different data consistency levels for different applications. And luckily, Milvus, a database for AI, offers flexibility in consistency level and you can set the consistency level that best suits your application.


### Consistency in the Milvus vector database

The concept of consistency level was first introduced with the release of Milvus 2.0. The 1.0 version of Milvus was not a distributed vector database so we did not involve tunable levels of consistency then. Milvus 1.0 flushes data every second, meaning that new data are almost immediately visible upon their insertion and Milvus reads the most updated data view at the exact time point when a vector similarity search or query request comes. 

However, Milvus was refactored in its 2.0 version and [Milvus 2.0 is a distributed vector database](https://milvus.io/blog/deep-dive-1-milvus-architecture-overview.md) based on a pub-sub mechanism. The [PACELC](https://en.wikipedia.org/wiki/PACELC_theorem) theorem points out that a distributed system must trade off among consistency, availability, and latency. Furthermore, different levels of consistency serve for different scenarios. Therefore, the concept of consistency was introduced in [Milvus 2.0](https://milvus.io/blog/2022-1-25-annoucing-general-availability-of-milvus-2-0.md) and it supports tuning levels of consistency.

## Four levels of consistency in the Milvus vector database

Milvus supports four levels of consistency:  strong, bounded staleness, session, and eventual. And a Milvus user can specify the consistency level when [creating a collection](https://milvus.io/docs/v2.1.x/create_collection.md) or conducting a [vector similarity search](https://milvus.io/docs/v2.1.x/search.md) or [query](https://milvus.io/docs/v2.1.x/query.md). This section will continue to explain how these four levels of consistency are different and which scenario are they best suited for. 

### Strong

Strong is the highest and the most strict level of consistency. It ensures that users can read the latest version of data. 

![Strong](https://assets.zilliz.com/Consistency_Strong_5d791eb8b2.png "An illustration of strong consistency.")

According to the PACELC theorem, if the consistency level is set to strong, the latency will increase. Therefore, we recommend choosing strong consistency during functional testings to ensure the accuracy of the test results. And strong consistency is also best suited for applications that have strict demand for data consistency at the cost of search speed. An example can be an online financial system dealing with order payments and billing.

### Bounded staleness

Bounded staleness, as its name suggests, allows data inconsistency during a certain period of time. However, generally, the data are always globally consistent out of that period of time.

![Bounded_staleness](https://assets.zilliz.com/Consistency_Bounded_c034bc6e51.png "An illustration of bounded staleness consistency.")

Bounded staleness is suitable for scenarios that needs to control search latency and can accept sporadic data invisibility. For instance, in recommender systems like video recommendation engines, data invisibility once in a while has really small impact on the overall recall rate, but can significantly boost the performance of the recommender system. An example can be an app for tracking the status of your online orders.

### Session

Session ensures that all data writes can be immediately perceived in reads during the same session. In other words, when you write data via one client, the newly inserted data instantaneously become searchable. 

![Session](https://assets.zilliz.com/Consistency_Session_6dc4782212.png "An illustration of session consistency.")

We recommend choosing session as the consistency level for those scenarios where the demand of data consistency in the same session is high. An example can be deleting the data of a book entry from the library system, and after confirmation of the deletion and refreshing the page (a different session), the book should no longer be visible in search results.

### Eventual

There is no guaranteed order of reads and writes, and replicas eventually converge to the same state given that no further write operations are done. Under eventual consistency, replicas start working on read requests with the latest updated values. Eventual consistency is the weakest level among the four. 

![Eventual](https://assets.zilliz.com/Consistency_Eventual_7c66dd5b6f.png "An illustration of eventual consistency.")

However, according to the PACELC theorem, search latency can be tremendously shortened upon sacrificing consistency. Therefore, eventual consistency is best suited for scenarios that do not have high demand for data consistency but requires blazing-fast search performance. An example can be retrieving reviews and ratings of Amazon products with eventual consistency. 

## Endnote

So going back to the question raised at the beginning of this article, deleted data are still returned as search results because the user has not chosen the proper level of consistency. The default value for consistency level is bounded staleness (`Bounded`) in the Milvus vector database. Therefore, the data read might lag behind and Milvus might happen to read the data view before you conducted delete operations during a similarity search or query. However, this issue is simple to solve. All you need to do is [tune the consistency level](https://milvus.io/docs/v2.1.x/tune_consistency.md) when creating a collection or conducting vector similarity search or query. Simple!

In the next post, we will unveil the mechanism behind and explain how the Milvus vector database achieves different levels of consistency. Stay tuned!

## What's next

With the official release of Milvus 2.1, we have prepared a series of blogs introducing the new features. Read more in this blog series:

- [How to Use String Data to Empower Your Similarity Search Applications](https://milvus.io/blog/2022-08-08-How-to-use-string-data-to-empower-your-similarity-search-applications.md)
- [Using Embedded Milvus to Instantly Install and Run Milvus with Python](https://milvus.io/blog/embedded-milvus.md)
- [Increase Your Vector Database Read Throughput with In-Memory Replicas](https://milvus.io/blog/in-memory-replicas.md)
- [Understanding Consistency Level in the Milvus Vector Database](https://milvus.io/blog/understanding-consistency-levels-in-the-milvus-vector-database.md)
- [Understanding Consistency Level in the Milvus Vector Database (Part II)](https://milvus.io/blog/understanding-consistency-levels-in-the-milvus-vector-database-2.md)
- [How Does the Milvus Vector Database Ensure Data Security?](https://milvus.io/blog/data-security.md)




Understanding Consistency Level in the Milvus Vector Database


![Cover_image](https://assets.zilliz.com/in_memory_replica_af1fa21d61.png "Increase Your Vector Database Read Throughput with In-Memory Replicas")


> This article is co-authored by [Congqi Xia](https://github.com/congqixia) and [Angela Ni](https://www.linkedin.com/in/yiyun-n-2aa713163/).


With its official release, Milvus 2.1 comes with many new features to provide convenience and a better user experience. Though the concept of in-memory replica is nothing new to the world of distributed databases, it is a critical feature that can help you boost system performance and enhance system availability in an effortless way. Therefore, this post sets out to explain what in-memory replica is and why it is important, and then introduces how to enable this new feature in Milvus, a vector database for AI.

**Jump to:**

- [Concepts related to in-memory replica](#Concepts-related-to-in-memory-replica)

- [What is in-memory replica?](#What-is-in-memory-replica)

- [Why are in-memory replicas important?](#Why-are-in-memory-replicas-important)

- [Enable in-memory replicas in the Milvus vector database](#Enable-in-memory-replicas-in-the-Milvus-vector-database)



## Concepts related to in-memory replica

Before getting to know what in-memory replica is and why it is important, we need to first understand a few relevant concepts including replica group, shard replica, streaming replica, historical replica, and shard leader. The image below is an illustration of these concepts.

![Replica_concepts](https://assets.zilliz.com/diagram_1_525afc706a.jpg "Replica concepts.")

### Replica group

A replica group consists of multiple [query nodes](https://milvus.io/docs/v2.1.x/four_layers.md#Query-node) that are responsible for handling historical data and replicas.

### Shard replica

A shard replica consists of a streaming replica and a historical replica, both belonging to the same [shard](https://milvus.io/blog/deep-dive-1-milvus-architecture-overview.md#Shard) (i.e DML channel). Multiple shard replicas make up a replica group. And the exact number of shard replicas in a replica group is determined by the number of shards in a specified collection.

### Streaming replica

A streaming replica contains all the [growing segments](https://milvus.io/docs/v2.1.x/glossary.md#Segment) from the same DML channel. Technically speaking, a streaming replica should be served by only one query node in one replica.

### Historical replica

A historical replica contains all the sealed segments from the same DML channel. The sealed segments of one historical replica can be distributed on several query nodes within the same replica group.

### Shard leader

A shard leader is the query node serving the streaming replica in a shard replica.



## What is in-memory replica?

Enabling in-memory replicas allows you to load data in a collection on multiple query nodes so that you can leverage extra CPU and memory resources. This feature is very useful if you have a relatively small dataset but want to increase read throughput and enhance the utilization of hardware resources.

The Milvus vector database holds one replica for each segment in memory for now. However, with in-memory replicas, you can have multiple replications of a segment on different query nodes. This means if one query node is conducting a search on a segment, an incoming new search request can be assigned to another idle query node as this query node has a replication of exactly the same segment. 

In addition, if we have multiple in-memory replicas, we can better cope with the situation in which a query node crashes. Before, we have to wait for the segment to be reloaded in order to continue and search on another query node. However, with in-memory replication, the search request can be resent to a new query node immediately without having to reload the data again.

![Replication](https://assets.zilliz.com/replication_3_1_2c25513cb9.jpg "In-memory replicas.")



## Why are in-memory replicas important?

One of the most significant benefits of enabling in-memory replicas is the increase in overall QPS (query per second) and throughput. Furthermore, multiple segment replicas can be maintained and the system is more resilient in the face of a failover.



## Enable in-memory replicas in the Milvus vector database

Enabling the new feature of in-memory replicas is effortless in the Milvus vector database. All you need to do is simply specify the number of replicas you want when loading a collection (ie. calling `collection.load()`). 

In the following example tutorial, we suppose you have already [created a collection](https://milvus.io/docs/v2.1.x/create_collection.md) named "book" and [inserted data](https://milvus.io/docs/v2.1.x/insert_data.md) into it. Then you can run the following command to create two replicas when [loading](https://milvus.io/docs/v2.1.x/load_collection.md) a book collection.

```
from pymilvus import Collection
collection = Collection("book")      # Get an existing collection.
collection.load(replica_number=2) # load collection as 2 replicas
```

You can flexibly modify the number of  replicas in the example code above to best suit your application scenario. Then you can directly conduct a vector similarity [search](https://milvus.io/docs/v2.1.x/search.md) or [query](https://milvus.io/docs/v2.1.x/query.md) on multiple replicas without running any extra commands. However, it should be noted that the maximum number of replicas allowed is limited by the total amount of usable memory to run the query nodes. If the number of replicas you specify exceeds the limitations of usable memory, an error will be returned during data load. 

You can also check the information of the in-memory replicas you created by running `collection.get_replicas()`. The information of replica groups and the corresponding query nodes and shards will be returned. The following is an example of the output. 

```
Replica groups:
- Group: <group_id:435309823872729305>, <group_nodes:(21, 20)>, <shards:[Shard: <channel_name:milvus-zong-rootcoord-dml_27_435367661874184193v0>, <shard_leader:21>, <shard_nodes:[21]>, Shard: <channel_name:milvus-zong-rootcoord-dml_28_435367661874184193v1>, <shard_leader:20>, <shard_nodes:[20, 21]>]>
- Group: <group_id:435309823872729304>, <group_nodes:(25,)>, <shards:[Shard: <channel_name:milvus-zong-rootcoord-dml_28_435367661874184193v1>, <shard_leader:25>, <shard_nodes:[25]>, Shard: <channel_name:milvus-zong-rootcoord-dml_27_435367661874184193v0>, <shard_leader:25>, <shard_nodes:[25]>]>
```

## What's next

With the official release of Milvus 2.1, we have prepared a series of blogs introducing the new features. Read more in this blog series:

- [How to Use String Data to Empower Your Similarity Search Applications](https://milvus.io/blog/2022-08-08-How-to-use-string-data-to-empower-your-similarity-search-applications.md)
- [Using Embedded Milvus to Instantly Install and Run Milvus with Python](https://milvus.io/blog/embedded-milvus.md)
- [Increase Your Vector Database Read Throughput with In-Memory Replicas](https://milvus.io/blog/in-memory-replicas.md)
- [Understanding Consistency Level in the Milvus Vector Database](https://milvus.io/blog/understanding-consistency-levels-in-the-milvus-vector-database.md)
- [Understanding Consistency Level in the Milvus Vector Database (Part II)](https://milvus.io/blog/understanding-consistency-levels-in-the-milvus-vector-database-2.md)
- [How Does the Milvus Vector Database Ensure Data Security?](https://milvus.io/blog/data-security.md)


Increase Your Vector Database Read Throughput with In-Memory Replicas


![Cover](https://assets.zilliz.com/Benchmark_Quick_Setup_58cc8eed5b.png "A Quick Guide to Benchmarking Milvus 2.1 ")

Recently, we have updated the [benchmark report of Milvus 2.1](https://milvus.io/docs/v2.1.x/benchmark.md). Tests with a dataset of 1 million vectors have proved that QPS can be dramatically increased by merging small-[nq](https://milvus.io/docs/v2.1.x/benchmark.md#Terminology) queries.

Here are some simple scripts for you to easily reproduce the tests.

## Procedures

1. Deploy a Milvus standalone or cluster. In this case, the IP address of the Milvus server is 10.100.31.105.

2. Deploy a client. In this case, we use Ubuntu 18.04 and Python 3.8.13 for the deployment. Run the following code to install PyMilvus 2.1.1.

```
pip install pymilvus==2.1.1
```

3. Download and copy the following files to the same working directory as the client. In this case, the working directory is `/go_ben`.

   - [`collection_prepare.py`](https://github.com/milvus-io/milvus-tools/blob/main/benchmark/collection_prepare.py)

   - [`go_benchmark.py`](https://github.com/milvus-io/milvus-tools/blob/main/benchmark/go_benchmark.py)

   - [`benchmark`](https://github.com/milvus-io/milvus-tools/blob/main/benchmark/benchmark) (for Ubuntu) or [`benchmark-mac`](https://github.com/milvus-io/milvus-tools/blob/main/benchmark/benchmark-mac) (for macOS)
   
   **Note:**
   
   - `benchmark` and `benchmark-mac` are executable files developed and compiled using Go SDK 2.1.1. They are only used to conduct a concurrent search. 
   
   - For Ubuntu users, please download `benchmark`; for macOS users, please download `benchmark-mac`.
   
   - Executable permissions are required to access `benchmark` or `benchmark-mac`. 
   
   - Mac users need to trust the `benchmark-mac` file by configuring Security & Privacy in System Preferences.
   
   - Settings on concurrent search can be found and modified in the `go_benchmark.py` source code.
   

4. Create a collection and insert vector data.

```
root@milvus-pytest:/go_ben# python collection_prepare.py 10.100.31.105 
```

5. Open `/tmp/collection_prepare.log` to check the running result.

```
...
08/11/2022 17:33:34 PM - INFO - Build index costs 263.626
08/11/2022 17:33:54 PM - INFO - Collection prepared completed
```

6. Call `benchmark` (or `benchmark-mac` on macOS) to conduct a concurrent search.

```
root@milvus-pytest:/go_ben# python go_benchmark.py 10.100.31.105 ./benchmark
[write_json_file] Remove file(search_vector_file.json).
[write_json_file] Write json file:search_vector_file.json done.
Params of go_benchmark: ['./benchmark', 'locust', '-u', '10.100.31.105:19530', '-q', 'search_vector_file.json', '-s', '{\n  "collection_name": "random_1m",\n  "partition_names": [],\n  "fieldName": "embedding",\n  "index_type": "HNSW",\n  "metric_type": "L2",\n  "params": {\n    "sp_value": 64,\n    "dim": 128\n  },\n  "limit": 1,\n  "expr": null,\n  "output_fields": [],\n  "timeout": 600\n}', '-p', '10', '-f', 'json', '-t', '60', '-i', '20', '-l', 'go_log_file.log']
[2022-08-11 11:37:39.811][    INFO] - Name      #   reqs      # fails  |       Avg       Min       Max    Median  |     req/s  failures/s (benchmark_run.go:212:sample)
[2022-08-11 11:37:39.811][    INFO] - go search     9665     0(0.00%)  |    20.679     6.499    81.761    12.810  |    483.25        0.00 (benchmark_run.go:213:sample)
[2022-08-11 11:37:59.811][    INFO] - Name      #   reqs      # fails  |       Avg       Min       Max    Median  |     req/s  failures/s (benchmark_run.go:212:sample)
[2022-08-11 11:37:59.811][    INFO] - go search    19448     0(0.00%)  |    20.443     6.549    78.121    13.401  |    489.22        0.00 (benchmark_run.go:213:sample)
[2022-08-11 11:38:19.811][    INFO] - Name      #   reqs      # fails  |       Avg       Min       Max    Median  |     req/s  failures/s (benchmark_run.go:212:sample)
[2022-08-11 11:38:19.811][    INFO] - go search    29170     0(0.00%)  |    20.568     6.398    76.887    12.828  |    486.15        0.00 (benchmark_run.go:213:sample)
[2022-08-11 11:38:19.811][   DEBUG] - go search run finished, parallel: 10(benchmark_run.go:95:benchmark)
[2022-08-11 11:38:19.811][    INFO] - Name      #   reqs      # fails  |       Avg       Min       Max    Median  |     req/s  failures/s (benchmark_run.go:159:samplingLoop)
[2022-08-11 11:38:19.811][    INFO] - go search    29180     0(0.00%)  |    20.560     6.398    81.761    13.014  |    486.25        0.00 (benchmark_run.go:160:samplingLoop)
Result of go_benchmark: {'response': True, 'err_code': 0, 'err_message': ''} 
```

7. Open the `go_log_file.log` file under the current directory to check the detailed search log. The following is the search information you can find in the search log.
   - reqs: number of search requests from the moment when concurrency happens to the current moment (the current time-span)
   
   - fails: number of failed requests as a percentage of reqs in the current time-span
   
   - Avg: average request response time in the current time-span (unit: milliseconds)
   
   - Min: minimum request response time in the current time-span (unit: milliseconds)
   
   - Max: maximum request response time in the current time-span (unit: milliseconds)
   
   - Median: median request response time in the current time-span (unit: milliseconds)
   
   - req/s: number of requests per second, i.e. QPS
   
   - failures/s: average number of failed requests per second in the current time-span

## Downloading Scripts and Executable Files

   - [collection_prepare.py](https://github.com/milvus-io/milvus-tools/blob/main/benchmark/collection_prepare.py)

   - [go_benchmark.py](https://github.com/milvus-io/milvus-tools/blob/main/benchmark/go_benchmark.py)

   - [benchmark](https://github.com/milvus-io/milvus-tools/blob/main/benchmark/benchmark) for Ubuntu
   
   - [benchmark-mac](https://github.com/milvus-io/milvus-tools/blob/main/benchmark/benchmark-mac) for macOS
 

## What's next

With the official release of Milvus 2.1, we have prepared a series of blogs introducing the new features. Read more in this blog series:

- [How to Use String Data to Empower Your Similarity Search Applications](https://milvus.io/blog/2022-08-08-How-to-use-string-data-to-empower-your-similarity-search-applications.md)
- [Using Embedded Milvus to Instantly Install and Run Milvus with Python](https://milvus.io/blog/embedded-milvus.md)
- [Increase Your Vector Database Read Throughput with In-Memory Replicas](https://milvus.io/blog/in-memory-replicas.md)
- [Understanding Consistency Level in the Milvus Vector Database](https://milvus.io/blog/understanding-consistency-levels-in-the-milvus-vector-database.md)
- [How Does the Milvus Vector Database Ensure Data Security?](https://milvus.io/blog/data-security.md)



A Quick Guide to Benchmarking Milvus 2.1



![Cover](https://assets.zilliz.com/embeddded_milvus_1_8132468cac.png "Using Embedded Milvus to Instantly Install and Run Milvus with Python ")

> This article is co-authored by [Alex Gao](https://github.com/soothing-rain/) and [Angela Ni](https://www.linkedin.com/in/yiyun-n-2aa713163/).

Milvus is an open-source vector database for AI applications. It provides a variety of installation methods including building from source code, and installing Milvus with Docker Compose/Helm/APT/YUM/Ansible. Users can choose one of the installation methods depending on their operating systems and preferences. However, there are many data scientists and AI engineers in the Milvus community who work with Python and yearn for a much simpler installation method than the currently available ones.

Therefore, we released embedded Milvus, a Python user-friendly version, along with Milvus 2.1 to empower more Python developers in our community. This article introduces what embedded Milvus is and provides instructions on how to install and use it.

**Jump to:**

- [An overview of embedded Milvus](#An-overview-of-embedded-Milvus)
  - [When to use embedded Milvus?](#When-to-use-embedded-Milvus)
  - [A comparison of different modes of Milvus](#A-comparison-of-different-modes-of-Milvus)
- [How to install embedded Milvus](#How-to-install-embedded-Milvus)
- [Start and stop embedded Milvus](#Start-and-stop-embedded-Milvus)

## An overview of embedded Milvus

[Embedded Milvus](https://github.com/milvus-io/embd-milvus) enables you to quickly install and use Milvus with Python. It can quickly bring up a Milvus instance and allows you to start and stop the Milvus service whenever you wish to. All data and logs are persisted even if you stop embedded Milvus. 

Embedded Milvus itself does not have any internal dependencies and do not require pre-installing and running any third-party dependencies like etcd, MinIO, Pulsar, etc.

Everything you do with embedded Milvus, and every piece of code you write for it can be safely migrated to other Milvus modes - standalone, cluster, cloud version, etc. This reflects one of the most distinctive features of embedded Milvus - **"Write once, run anywhere"**.

### When to use embedded Milvus?

Embedded Milvus and [PyMilvus](https://milvus.io/docs/v2.1.x/install-pymilvus.md) are constructed for different purposes. You may consider choosing embedded Milvus in the following scenarios:

- You want to use Milvus without installing Milvus in any of the ways provided [here](https://milvus.io/docs/v2.1.x/install_standalone-docker.md).

- You want to use Milvus without keeping a long-running Milvus process in your machine.

- You want to quickly use Milvus without starting a separate Milvus process and other required components like etcd, MinIO, Pulsar, etc.

It is suggested that you should **NOT** use embedded Milvus:

-  In a production environment. (_To use Milvus for production, consider Milvus cluster or [Zilliz cloud](https://zilliz.com/cloud), a fully managed Milvus service._)

- If you have a high demand for performance. (_Comparatively speaking, embedded Milvus might not provide the best performance._)

### A comparison of different modes of Milvus

The table below compares several modes of Milvus: standalone, cluster, embedded Milvus, and the Zilliz Cloud, a fully managed Milvus service.

![comparison](https://assets.zilliz.com/comparison_ebcd7c5b07.jpeg "Comparing different Milvus modes.")

## How to install embedded Milvus?

Before installing embedded Milvus, you need to first ensure that you have installed Python 3.6 or later. Embedded Milvus supports the following operating systems: 

- Ubuntu 18.04

- Mac x86_64 >= 10.4

- Mac M1 >= 11.0

If the requirements are met, you can run `$ python3 -m pip install milvus` to install embedded Milvus. You can also add the version in the command to install a specific version of embedded Milvus. For instance, if you want to install the 2.1.0 version, run `$ python3 -m pip install milvus==2.1.0`. And later when new version of embedded Milvus is released, you can also run `$ python3 -m pip install --upgrade milvus` to upgrade embedded Milvus to the latest version.

If you are an old user of Milvus who has already installed PyMilvus before and wants to install embedded Milvus, you can run `$ python3 -m pip install --no-deps milvus`.

After running the installation command, you need to create a data folder for embedded Milvus under `/var/bin/e-milvus` by running the following command:

```
sudo mkdir -p /var/bin/e-milvus
sudo chmod -R 777 /var/bin/e-milvus
```

## Start and stop embedded Milvus

When the installation is successful, you can start the service.

If you are running embedded Milvus for the first time you need to import Milvus and set up embedded Milvus first.

```
$ python3
Python 3.9.10 (main, Jan 15 2022, 11:40:53)
[Clang 13.0.0 (clang-1300.0.29.3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import milvus
>>> milvus.before()
please do the following if you have not already done so:
1. install required dependencies: bash /var/bin/e-milvus/lib/install_deps.sh
2. export LD_PRELOAD=/SOME_PATH/embd-milvus.so
3. export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/lib:/usr/local/lib:/var/bin/e-milvus/lib/
>>>
```

If you have successfully started embedded Milvus before and comes back to restart it, you can directly run `milvus.start()` after importing Milvus.

```
$ python3
Python 3.9.10 (main, Jan 15 2022, 11:40:53)
[Clang 13.0.0 (clang-1300.0.29.3)] on darwinType "help", "copyright", "credits" or "license" for more information.
>>> import milvus
>>> milvus.start()
>>>
```

You will see the following output if you have successfully started the embedded Milvus service.

```
---Milvus Proxy successfully initialized and ready to serve!---
```

After the service starts, you can start another terminal window and run the example code of "[Hello Milvus](https://github.com/milvus-io/embd-milvus/blob/main/milvus/examples/hello_milvus.py)" to play around with embedded Milvus!

```
# Download hello_milvus script
$ wget https://raw.githubusercontent.com/milvus-io/pymilvus/v2.1.0/examples/hello_milvus.py
# Run Hello Milvus 
$ python3 hello_milvus.py
```

When you are done with using embedded Milvus, we recommend stopping it gracefully and clean up the environment variables by run the following command or press Ctrl-D.

```
>>> milvus.stop()
if you need to clean up the environment variables, run:
export LD_PRELOAD=
export LD_LIBRARY_PATH=
>>>
>>> exit()
```

## What's next

With the official release of Milvus 2.1, we have prepared a series of blogs introducing the new features. Read more in this blog series:

- [How to Use String Data to Empower Your Similarity Search Applications](https://milvus.io/blog/2022-08-08-How-to-use-string-data-to-empower-your-similarity-search-applications.md)
- [Using Embedded Milvus to Instantly Install and Run Milvus with Python](https://milvus.io/blog/embedded-milvus.md)
- [Increase Your Vector Database Read Throughput with In-Memory Replicas](https://milvus.io/blog/in-memory-replicas.md)
- [Understanding Consistency Level in the Milvus Vector Database](https://milvus.io/blog/understanding-consistency-levels-in-the-milvus-vector-database.md)
- [Understanding Consistency Level in the Milvus Vector Database (Part II)](https://milvus.io/blog/understanding-consistency-levels-in-the-milvus-vector-database-2.md)
- [How Does the Milvus Vector Database Ensure Data Security?](https://milvus.io/blog/data-security.md)


Using Embedded Milvus to Instantly Install and Run Milvus with Python


![Cover](https://assets.zilliz.com/string_6129ce83e6.png "How to use string data to empower your similarity search applications?")

Milvus 2.1 comes with [some significant updates](https://milvus.io/blog/2022-08-05-whats-new-in-milvus-2-1.md) which make working with Milvus a lot easier. One of them is the support of string data type. Right now Milvus [supports data types](https://milvus.io/docs/v2.1.x/schema.md#Supported-data-type) including strings, vectors, Boolean, integers, floating-point numbers, and more. 

This article presents an introduction to the support of string data type. Read and learn what you can do with it and how to use it.

**Jump to:**
- [What can you do with string data?](#What-can-you-do-with-string-data)
- [How to manage string data in Milvus 2.1?](#How-to-manage-string-data-in-Milvus-21)
    - [Create a collection](#Create-a-collection)
    - [Insert and delete data](#Insert-data)
    - [Build an index](#Build-an-index)
    - [Hybrid search](#Hybrid-search)
    - [String expressions](#String-expressions)

# What can you do with string data?

The support of string data type has been one of the functions most expected by users. It both streamlines the process of building an application with the Milvus vector database and accelerates the speed of similarity search and vector query, largely increasing the efficiency and reducing the maintenance cost of whatever application you are working on.

Specifically, Milvus 2.1 supports VARCHAR data type, which stores character strings of varying length. With the support of VARCHAR data type, you can:

1. Directly manage string data without the help of an external relational database.

The support of VARCHAR data type enables you to skip the step of converting strings into other data types when inserting data into Milvus. Let's say you're working on a book search system for your own online bookstore. You are creating a book dataset and want to identify the books with their names. While in previous versions where Milvus does not support the string data type, before inserting data into MIilvus, you may need to first transform the strings (the names of the books) into book IDs with the help of a relational database like MySQL. Right now, as string data type is supported, you can simply create a string field and directly enter the book names instead of their ID numbers.

The convenience also goes to the search and query process. Imagine there is a client whose favourite book is *Hello Milvus*. You want to search in the system for similar books and recommend them to the client. In previous versions of Milvus, the system will only return you book IDs and you need to take an extra step to check the corresponding book information in a relational database. But in Milvus 2.1, you can directly get the book names as you have already created a string field with book names in it.

In a word, the support of string data type saves you the effort to turn to other tools to manage string data, which greatly simplifies the development process.

2. Accelerate the speed of [hybrid search](https://milvus.io/docs/v2.1.x/hybridsearch.md) and [vector query](https://milvus.io/docs/v2.1.x/query.md) through attribute filtering.

Like other scalar data types, VARCHAR can be used for attribute filtering in hybrid search and vector query through Boolean expression. It is particularly worth mentioning that Milvus 2.1 adds the operator `like`, which enables you to perform prefix matching. Also, you can perform exact matching using the operator `==`.

Besides, a MARISA-trie based inverted index is supported to accelerate hybrid search and query. Continue to read and find out all the string expressions you may want to know to perform attribute filtering with string data.

# How to manage string data in Milvus 2.1?

Now we know the string data type is extremely useful, but when exactly do we need to use this data type in building our own applications? In the following, you will see some code examples of scenarios that may involve string data, which will give you a better understanding of how to manage VARCHAR data in Milvus 2.1.

## Create a collection

Let's follow the previous example. You are still working on the book recommender system and want to create a book collection with a primary key field called `book_name`, into which you will insert string data. In this case, you can set the data type as `DataType.VARCHAR`when setting the field schema, as shown in the example below. 

Note that when creating a VARCHAR field, it is necessary to specify the maximum character length via the parameter `max_length` whose value can range from 1 to 65,535.  In this example, we set the maximum length as 200.

```Python
from pymilvus import CollectionSchema, FieldSchema, DataType
book_id = FieldSchema(
  name="book_id", 
  dtype=DataType.INT64, 
)
book_name = FieldSchema( 
  name="book_name", 
  dtype=DataType.VARCHAR, 
  max_length=200, 
  is_primary=True, 
)
word_count = FieldSchema(
  name="word_count", 
  dtype=DataType.INT64,  
)
book_intro = FieldSchema(
  name="book_intro", 
  dtype=DataType.FLOAT_VECTOR, 
  dim=2
)
schema = CollectionSchema(
  fields=[book_id, word_count, book_intro], 
  description="Test book search"
)
collection_name = "book"
```

## Insert data

Now that the collection is created, we can insert data into it. In the following example, we insert 2,000 rows of randomly generated string data.

```Python
import random
data = [
  [i for i in range(2000)],
  ["book_" + str(i) for i in range(2000)],
  [i for i in range(10000, 12000)],
  [[random.random() for _ in range(2)] for _ in range(2000)],
]
```

## Delete data

Suppose two books, named `book_0` and `book_1`, are no longer available in your store, so you want to delete the relevant information from your database. In this case, you can use the term expression `in` to filter the entities to delete, as shown in the example below.

Remember that Milvus only supports deleting entities with clearly specified primary keys, so before running the following code, make sure that you have set the `book_name` field as the primary key field.

```Python
expr = "book_name in [\"book_0\", \"book_1\"]" 
from pymilvus import Collection
collection = Collection("book")     
collection.delete(expr)
```

## Build an Index

Milvus 2.1 supports building scalar indexes, which will greatly accelerate the filtering of string fields. Unlike building a vector index, you don't have to prepare parameters before building a scalar index. Milvus temporarily only supports the dictionary tree (MARISA-trie) index, so the index type of VARCHAR type field is MARISA-trie by default.

You can specify the index name when building it. If not specified, the default value of `index_name` is `"_default_idx_"`. In the example below, we named the index `scalar_index`.

```Python
from pymilvus import Collection
collection = Collection("book")   
collection.create_index(
  field_name="book_name", 
  index_name="scalar_index",
)
```

## Hybrid search

By specifying boolean expressions, you can filter the string fields during a vector similarity search. 

For example, if you are searching for books whose intro are most similar to Hello Milvus but only want to get the books whose names start with 'book_2', you can use the operator `like`to perform a prefix match and get the targeted books, as shown in the example below. 

```Python
search_param = {
  "data": [[0.1, 0.2]],
  "anns_field": "book_intro",
  "param": {"metric_type": "L2", "params": {"nprobe": 10}},
  "limit": 2,
  "expr": "book_name like \"Hello%\"",
}
res = collection.search(**search_param)
```

## String expressions

Apart from the newly added operator `like`, other operators, which are already supported in previous versions of Milvus, can also be used for string field filtering. Below are some examples of commonly used [string expressions](https://milvus.io/docs/v2.1.x/boolean.md), where `A` represents a field of type VARCHAR. Remember that all the string expressions below can be logically combined using logical operators, such as AND, OR, and NOT.

### Set operations

You can use `in` and `not in` to realize set operations, such as `A in ["str1", "str2"]`.

### Compare two string fields

You can use relational operators to compare the values of two string fields. Such relational operators include `==`, `!=`, `>`, `>=`, `<`, `<=`. For more information, see [Relational operators](https://milvus.io/docs/v2.1.x/boolean.md#Relational-operators).

Note that string fields can only be compared with other string fields instead of fields of other data types. For example, a field of type VARCHAR cannot be compared with a field of type Boolean or of type integer.

### Compare a field with a constant value

You can use `==` or `!=` to verify if the value of a field is equal to a constant value. 

### Filter fields with a single range

You can use `>`, `>=`, `<`, `<=` to filter string fields with a single range, such as `A > "str1"`. 

### Prefix matching

As mentioned earlier, Milvus 2.1 adds the operator `like` for prefix matching, such as `A like "prefix%"`. 

## What's next

With the official release of Milvus 2.1, we have prepared a series of blogs introducing the new features. Read more in this blog series:

- [How to Use String Data to Empower Your Similarity Search Applications](https://milvus.io/blog/2022-08-08-How-to-use-string-data-to-empower-your-similarity-search-applications.md)
- [Using Embedded Milvus to Instantly Install and Run Milvus with Python](https://milvus.io/blog/embedded-milvus.md)
- [Increase Your Vector Database Read Throughput with In-Memory Replicas](https://milvus.io/blog/in-memory-replicas.md)
- [Understanding Consistency Level in the Milvus Vector Database](https://milvus.io/blog/understanding-consistency-levels-in-the-milvus-vector-database.md)
- [Understanding Consistency Level in the Milvus Vector Database (Part II)](https://milvus.io/blog/understanding-consistency-levels-in-the-milvus-vector-database-2.md)
- [How Does the Milvus Vector Database Ensure Data Security?](https://milvus.io/blog/data-security.md)


How to Use String Data to Empower Your Similarity Search Applications


![What's new in Milvus 2.1 - Towards simplicity and speed](https://assets.zilliz.com/What_s_New_in_2_1_2_a0660df2a5.png)

We are very glad to announce the
[release](https://milvus.io/docs/v2.1.x/release_notes.md) of Milvus 2.1
is now live after six months of hard work by all of our Milvus community
contributors. This major iteration of the popular vector database
emphasizes **performance** and **usability**, two most important
keywords of our focus. We added support for strings, Kafka message
queue, and embedded Milvus, as well as a number of improvements in
performance, scalability, security, and observability. Milvus 2.1 is an
exciting update that will bridge the "last mile" from the algorithm
engineer's laptop to production-level vector similarity search
services.

# Performance - More than a 3.2x boost

## 5ms-level latency

Milvus already supports approximate nearest neighbor (ANN) search, a
substantial leap from the traditional KNN method. However, problems of
throughput and latency continue to challenge users who need to deal with
billion-scale vector data retrieval scenarios.

In Milvus 2.1, there is a new routing protocol that no longer relies on
message queues in the retrieval link, significantly reducing retrieval
latency for small datasets. Our test results show that Milvus now brings
its latency level down to 5ms, which meets the requirements of critical
online links such as similarity search and recommendation.

## Concurrency control

Milvus 2.1 fine-tunes its concurrency model by introducing a new cost
evaluation model and concurrency scheduler. It now provides concurrency
control, which ensures that there will not be a large number of
concurrent requests competing for CPU and cache resources, nor will the
CPU be under-utilized because there are not enough requests. The new,
intelligent scheduler layer in Milvus 2.1 also merges small-nq queries
that have consistent request parameters, delivering an amazing 3.2x
performance boost in scenarios with small-nq and high query concurrency.

## In-memory replicas

Milvus 2.1 brings in-memory replicas that improve scalability and
availability for small datasets. Similar to the read-only replicas in
traditional databases, in-memory replicas can scale horizontally by
adding machines when the read QPS is high. In vector retrieval for small
datasets, a recommendation system often needs to provide QPS that
exceeds the performance limit of a single machine. Now in these
scenarios, the system's throughput can be significantly improved by
loading multiple replicas in the memory. In the future, we will also
introduce a hedged read mechanism based on in-memory replicas, which
will quickly request other functional copies in case the system needs to
recover from failures and makes full use of memory redundancy to improve
the system's overall availability.

![In-memory replicas allow query services to be based on separate
copies of the same data.](https://assets.zilliz.com/What_s_New_in_Milvus_2_1_Figure_1_excalidraw_1f7fe3c998.png)

## Faster data loading

The last performance boost comes from data loading. Milvus 2.1 now
compresses [binary
logs](https://milvus.io/docs/v2.1.x/glossary.md#Log-snapshot) with
Zstandard (zstd), which significantly reduces data size in the object
and message stores as well as network overhead during data loading. In
addition, goroutine pools are now introduced so that Milvus can load
segments concurrently with memory footprints controlled and minimize the
time required to recover from failures and to load data.

The complete benchmark results of Milvus 2.1 will be released on our
website soon. Stay tuned.

## String and scalar index support

With 2.1, Milvus now supports variable-length string (VARCHAR) as a
scalar data type. VARCHAR can be used as the primary key that can be
returned as output, and can also act as attribute filters. [Attribute
filtering](https://milvus.io/docs/v2.1.x/hybridsearch.md) is one of the
most popular functions Milvus users need. If you often find yourself
wanting to "find products most similar to a user in a $200 - $300
price range", or "find articles that have the keyword 'vector
database' and are related to cloud-native topics", you'll love Milvus
2.1.

Milvus 2.1 also supports scalar inverted index to improve filtering
speed based on
[succinct](https://www.cs.le.ac.uk/people/ond1/XMLcomp/confersWEA06_LOUDS.pdf)
[MARISA-Tries](https://github.com/s-yata/marisa-trie) as the data
structure. All the data can now be loaded into memory with a very low
footprint, which allows much quicker comparison, filtering and prefix
matching on strings. Our test results show that the memory requirement
of MARISA-trie is only 10% of that of Python dictionaries to load all
the data into memory and provide query capabilities.

![Milvus 2.1 combines MARISA-Trie with inverted index to significantly improve filtering speed.](https://assets.zilliz.com/What_s_new_in_Milvus_Figure_2_excalidraw_a1149aca96.png)

In the future, Milvus will continue focusing on scalar query-related
developments, support more scalar index types and query operators, and
provide disk-based scalar query capabilities, all as part of an ongoing
effort to reduce storage and usage cost of scalar data.

# Usability improvements

## Kafka support

Our community has long been requesting support for [Apache
Kafka](https://kafka.apache.org) as the [message
storage](https://milvus.io/docs/v2.1.x/deploy_pulsar.md) in Milvus.
Milvus 2.1 now offers you the option to use
[Pulsar](https://pulsar.apache.org) or Kafka as the message storage
based on user configurations, thanks to the abstraction and
encapsulation design of Milvus and the Go Kafka SDK contributed by
Confluent.

## Production-ready Java SDK

With Milvus 2.1, our [Java
SDK](https://github.com/milvus-io/milvus-sdk-java) is now officially
released. The Java SDK has the exact same capabilities as the Python
SDK, with even better concurrency performance. In the next step, our
community contributors will gradually improve documentation and use
cases for the Java SDK, and help push Go and RESTful SDKs into the
production-ready stage, too.

## Observability and maintainability

Milvus 2.1 adds important monitoring
[metrics](https://milvus.io/docs/v2.1.x/metrics_dashboard.md) such as
vector insertion counts, search latency/throughput, node memory
overhead, and CPU overhead. Plus, the new version also significantly
optimizes log keeping by adjusting log levels and reducing useless log
printing.

## Embedded Milvus

Milvus has greatly simplified the deployment of large-scale massive
vector data retrieval services, but for scientists who want to validate
algorithms on a smaller scale, Docker or K8s is still too unnecessarily
complicated. With the introduction of [embedded
Milvus](https://github.com/milvus-io/embd-milvus), you can now install
Milvus using pip, just like with Pyrocksb and Pysqlite. Embedded Milvus
supports all the functionalities of both the cluster and standalone
versions, allowing you to easily switch from your laptop to a
distributed production environment without changing a single line of
code. Algorithm engineers will have a much better experience when
building a prototype with Milvus.

# Try out-of-the-box vector search now

Moreover, Milvus 2.1 also has some great improvements in stability and
scalability, and we look forward to your use and feedbacks.

## What's next

-   See the detailed [Release
    Notes](https://milvus.io/docs/v2.1.x/release_notes.md) for all the
    changes in Milvus 2.1
-   [Install](https://milvus.io/docs/v2.1.x/install_standalone-docker.md)
    Milvus 2.1 and try out the new features
-   Join our [Slack community](https://slack.milvus.io/) and discuss the
    new features with thousands of Milvus users around the world
-   Follow us on [Twitter](https://twitter.com/milvusio) and
    [LinkedIn](https://www.linkedin.com/company/the-milvus-project) to
    get updates once our blogs on specific new features are out

> Edited by [Songxian Jiang](https://github.com/songxianj)


What's new in Milvus 2.1 - Towards simplicity and speed


![cover image](https://assets.zilliz.com/Frame_1282_edc1fb7d99.png "Building an intelligent wardrobe customization system powered by Milvus vector database")

If you are looking for a wardrobe to fit perfectly into your bedroom or fitting room, I bet most people will think of the made-to-measure ones. However, not everyone's budget can stretch that far. Then what about those ready-made ones? The problem with this type of wardrobe is that they are very likely to fall short of your expectation as they are not flexible enough to cater to your unique storage needs. Plus, when searching online, it is rather difficult to summarize the particular type of wardrobe you are looking for with keywords. Very likely, the keyword you type in the search box (eg. A wardrobe with a jewellery tray) might be very different from how it is defined in the search engine (eg. A wardrobe with [pullout tray with insert](https://www.ikea.com/us/en/p/komplement-pull-out-tray-with-insert-black-brown-s79249366/)).  

But thanks to emerging technologies, there is a solution! IKEA, the furniture retail conglomerate, provides a popular design tool [PAX wardrobe](https://www.ikea.com/us/en/rooms/bedroom/how-to/how-to-design-your-perfect-pax-wardrobe-pub8b76dda0) that allows users to choose from a number of ready-made wardrobes and customize the color, size, and interior design of the wardrobes. Whether you need hanging space, multiple shelves or internal drawers, this intelligent wardrobe customization system can always cater to your needs. 

To find or build your ideal wardrobe using this smart wardrobe design system, you need to:
1. Specify the basic requirements - the shape (normal, L-shaped, or U-shaped), length and depth of the wardrobe.
2. Specify your storage need and the interior organization of the wardrobe (eg. Hanging space, a pullout pants rack, etc is needed).
3. Add or remove parts of the wardrobe like drawers or shelves.

Then your design is completed. Simple and easy!




![pax system](https://assets.zilliz.com/Pax_system_ff4c3fa182.png "An online wardrobe customization system.")

A very critical component that makes such a  wardrobe design system possible is the [vector database](https://zilliz.com/learn/what-is-vector-database). Therefore, this article aims to introduce the workflow and similarity search solutions used to build an intelligent wardrobe customization system powered by vector similarity search.

Jump to:
- [System overview](#System-overview)
- [Data flow](#Data-flow)
- [System demo](#System-demo)


## System Overview
In order to deliver such a smart wardrobe customization tool, we need to first define the business logic and understand item attributes and user journey. Wardrobes along with its components like drawers, trays, racks, are all unstructured data. Therefore, the second step is to leverage AI algorithms and rules, prior knowledge, item description, and more, to convert those unstructured data into a type of data that can be understood by computers - vectors!


![Customization tool overview](https://assets.zilliz.com/Customization_tool_overview_86d62e1730.png "An overview of the wardrobe customization tool.")

With the generated vectors, we need powerful vector databases and search engines to process them.


![tool architecture](https://assets.zilliz.com/tool_architecutre_33fb646954.png "The architecture of the customization tool.")

The customization tool leverages some of the most popular search engines and databases: Elasticsearch, [Milvus](https://milvus.io/), and PostgreSQL.

### Why Milvus?
A wardrobe component contains highly complex information, such as color, shape, and interior organization, etc. However, the traditional way of keeping wardrobe data in a relational database is far from enough. A popular way is to use embedding techniques to convert wardrobes into vectors. Therefore, we need to look for a new type of database specifically designed for vector storage and similarity search. After probing into several popular solutions, the [Milvus](https://github.com/milvus-io/milvus) vector database is selected for its excellent performance, stability, compatibility, and ease-of-use. The chart below is a comparison of several popular vector search solutions.


![solution comparison](https://assets.zilliz.com/Solution_comparison_d96b8f1dd5.png "Comparing several solutions.")

### System workflow

![System workflow](https://assets.zilliz.com/System_workflow_250c275ec1.png "System workflow.")


Elasticsearch is used for a coarse filtering by the wardrobe size, color, etc. Then the filtered results go through Milvus the vector database for a similarity search and the results are ranked based on their distance/similarity to the query vector. Finally, the results are consolidated and further refined based on business insights.

## Data flow 
The wardrobe customization system is very similar to traditional search engines and recommender systems. It contains three parts:  
- Offline data preparation including data definition and generation.
- Online services including recall and ranking.
- Data post-processing based on business logic.


![Data flow](https://assets.zilliz.com/data_flow_d0d9fa0fca.png "The overall data flow in the wardrobe customization system.")

### Offline data flow 
1. Define data using business insight.
2. Use prior knowledge to define how to combine different components and form them into a wardrobe.
3. Recognize feature labels of the wardrobes and encode the features into Elasticsearch data in `.json` file.
4. Prepare recall data by encoding unstructured data into vectors.
5. Use Milvus the vector database to rank the recalled results obtained in the previous step.


![offline data flow](https://assets.zilliz.com/offline_data_flow_f91ac9cf4c.png "How data is processed offline.")

### Online data flow
1. Receive query request from users and collect user profiles.
2. Understand user query by identifying their requirements for the wardrobe.
3. Coarse search using Elasticsearch.
4. Score and rank the results obtained from coarse search based on the calculation of vector similarity in Milvus.
5. Post-process and organize the results on the back-end platform to generate the final results.


![online data flow](https://assets.zilliz.com/online_data_flow_1f2af25cc3.png "How data is processed online.")

### Data post-processing
The business logic varies among each company. You can add a final touch to the results by applying your company's business logic.

## System demo
Now let's see how the system we build actually works.

The user interface (UI) displays the possibility of different combinations of wardrobe components. 

Each component is labelled by its feature (size, color, etc.) and stored in Elasticsearch (ES). When storing the labels in ES, there are four main data fields to be filled out: ID, tags, storage path, and other support fields. ES and the labelled data are used for granular recall and attribute filtering.

![es](https://assets.zilliz.com/es_d5b0639610.png "Using Elasticsearch for labelling.")

Then different AI algorithms are used to encode a wardrobe into a set of vectors. The vector sets are stored in Milvus for similarity search and ranking. This step returns more refined and accurate results.

![Milvus](https://assets.zilliz.com/Milvus_38dd93a439.jpeg "Using Milvus for vector storage and similarity search.")

Elasticsearch, Milvus, and other system components altogether form the customization design platform as a whole. During recall, the domain-specific language (DSL) in Elasticsearch and Milvus is as follows.


![dsl](https://assets.zilliz.com/dsl_df60097d23.png "Recall DSL.")

## Looking for more resources?
Learn how the Milvus vector database can power more AI applications:
- [How Short Video Platform Likee Removes Duplicate Videos with Milvus](https://milvus.io/blog/2022-06-23-How-Short-video-Platform-Likee-Removes-Duplicate-Videos-with-Milvus.md)
- [Zhentu - The Photo Fraud Detector Based on Milvus](https://milvus.io/blog/2022-06-20-Zhentu-the-Photo-Fraud-Detector-Based-on-Milvus.md)


Building an Intelligent Wardrobe Customization System Powered by Milvus Vector Database


![Cover image](https://assets.zilliz.com/How_Short_video_Platform_Likee_Removes_Duplicate_Videos_with_Milvus_07bd75ec82.png "How Short-video Platform Likee Removes Duplicate Videos with Milvus")

> This article is written by Xinyang Guo and Baoyu Han, engineers at BIGO, and translated by [Rosie Zhang](https://www.linkedin.cn/incareer/in/rosie-zhang-694528149).

[BIGO Technology](https://www.bigo.sg/) (BIGO) is one of the fastest-growing Singapore technology companies. Powered by artificial intelligence technology, BIGO's video-based products and services have gained immense popularity worldwide, with over 400 million users in more than 150 countries. These include [Bigo Live](https://www.bigo.tv/bigo_intro/en.html?hk=true) (live streaming) and [Likee](https://likee.video/) (short-form video). 

Likee is a global short video creation platform where users can share their moments, express themselves, and connect with the world. To increase user experience and recommend higher-quality content to users, Likee needs to weed out duplicate videos from a tremendous amount of videos generated by users every day, which poses no straightforward task. 

This blog presents how BIGO uses [Milvus](https://milvus.io), an open-source vector database, to effectively remove duplicate videos.

**Jump to:**
- [Overview](#Overview)
- [Video deduplication workflow](#Video-deduplication-workflow)
- [System Architecture](#System-architecture)
- [Using Milvus to power similarity search](#Using-Milvus-vector-database-to-power-similarity-search)

# Overview
Milvus is an open-source vector database featuring ultra-fast vector search. Powered by Milvus, Likee is able to complete a search within 200ms while ensuring a high recall rate. Meanwhile, by [scaling Milvus horizontally](https://milvus.io/docs/v2.0.x/scaleout.md#Scale-a-Milvus-Cluster), Likee successfully increases the throughput of vector queries, further improving its efficiency.

# Video deduplication workflow
How does Likee identify duplicate videos? Every time a query video is input into Likee's system, it will be cut into 15-20 frames and each frame will be converted to a feature vector. Then Likee searches in a database of 700 million vectors to find the top K most similar vectors. Each one of the top K vectors corresponds to a video in the database. Likee further conducts refined searches to get the final results and determine the videos to be removed.

# System architecture
Let's take a closer look at how Likee's video de-duplication system works using Milvus. As is shown in the diagram below, new videos uploaded to Likee will be written to Kafka, a data storage system, in real-time and consumed by Kafka consumers. The feature vectors of these videos are extracted through deep learning models, where unstructured data (video) is converted into feature vectors. These feature vectors will be packaged by the system and sent to the similarity auditor.

![Architechure of Likee's video de-duplication system](https://assets.zilliz.com/Likee_1_6f7ebcd8fc.png "Architechure of Likee's video de-duplication system")

The feature vectors extracted will be indexed by Milvus and stored in Ceph, before being [loaded by the Milvus query node](https://milvus.io/blog/deep-dive-5-real-time-query.md) for further search. The corresponding video IDs of these feature vectors will also be stored simultaneously in TiDB or Pika according to the actual needs.

### Using Milvus vector database to power similarity search
When searching for similar vectors, billions of existing data, along with large amounts of new data generated every day, pose great challenges to the functionality of the vector search engine. After thorough analysis, Likee eventually chose Milvus, a distributed vector search engine of high performance and high recall rate, to conduct vector similarity search.

As is shown in the diagram below, the procedure of a similarity search goes as follows:

1. First, Milvus performs a batch search to recall the top 100 similar vectors for each of the multiple feature vectors extracted from a new video. Each similar vector is bound to its corresponding video ID. 

2. Second, by comparing the video IDs, Milvus removes the duplicate videos and retrieves the feature vectors of the remaining videos from TiDB or Pika. 

3. Finally, Milvus calculates and scores the similarity between each set of the feature vectors retrieved and the feature vectors of the query video. The video ID with the highest score is returned as the result. Thus the video similarity search is concluded.

![Procedure of a similarity search](https://assets.zilliz.com/02_a24d251c8f.png "Procedure of a similarity search")

As a high-performance vector search engine, Milvus has done an extraordinary job in Likee's video de-duplication system, greatly fueling the growth of BIGO's short-video business. In terms of video businesses, there are many other scenarios where Milvus can be applied to, such as illegal content blocking or personalized video recommendation. Both BIGO and Milvus are looking forward to future cooperation in more areas.

How Short-video Platform Likee Removes Duplicate Videos with Milvus

Scalable and Blazing Fast Similarity Search with Milvus Vector Database


![cover image](https://assets.zilliz.com/zhentu_0ae11c98ee.png "Zhentu - the Photo Fraud Detector Based on Milvus")

> This article is written by Yan Shi and Minwei Tang, senior algorithm engineers at BestPay, and translated by [Rosie Zhang](https://www.linkedin.cn/incareer/in/rosie-zhang-694528149).

In recent years, as e-commerce and online transactions become commonplace throughout the world, e-commerce fraud also flourished. By using computer-generated photos instead of real ones to pass identity verification on online business platforms, fraudsters create massive fake accounts and cash in on businesses' special offers (e.g. membership gifts, coupons, tokens), which brings irretrievable losses to both consumers and businesses.

Traditional risk control methods are no longer effective in the face of a substantial amount of data. To solve the problem, [BestPay](https://www.bestpay.com.cn) created a photo fraud detector, namely Zhentu (meaning detecting images in Chinese), based on deep learning (DL) and digital image processing (DIP) technologies. Zhentu is applicable to various scenarios involving image recognition, with one important offshoot being the identification of fake business licenses. If the business license photo submitted by a user is very similar to another photo already existing in a platform's photo library, it is likely that the user has stolen the photo somewhere or has forged a license for fraudulent purposes.

Traditional algorithms for measuring image similarity, such as [PSNR](https://en.wikipedia.org/wiki/Peak_signal-to-noise_ratio) and ORB, are slow and inaccurate, only applicable to offline tasks. Deep learning, on the other hand, is capable of processing large-scale image data in real-time and is the ultimate method for matching similar images. With the joint efforts of BestPay’s R&D team and [the Milvus community](https://milvus.io/), a photo fraud detection system is developed as part of Zhentu. It functions by converting massive amounts of image data into feature vectors through deep learning models and inserting them into [Milvus](https://milvus.io/), a vector search engine. With Milvus, the detection system is able to index trillions of vectors and efficiently retrieve similar photos among tens of millions of images.

**Jump to:**
- [An overview of Zhentu](#an-overview-of-zhentu)
- [System structure](#system-structure)
- [**Deployment**](#deployment)
- [**Real-world performance**](#real-world-performance)
- [**Reference**](#reference)
- [**About BestPay**](#about-bestpay)

## An overview of Zhentu

Zhentu is BestPay’s self-designed multimedia visual risk control product deeply integrated with machine learning (ML) and neural network image recognition technologies. Its built-in algorithm can accurately identify fraudsters during user authentication and respond at the millisecond level. With its industry-leading technology and innovative solution, Zhentu has won five patents and two software copyrights. It is now being used in a number of banks and financial institutions to help identify potential risks in advance.

## System structure

BestPay currently has over 10 million business license photos, and the actual volume is still growing exponentially as the business grows. In order to quickly retrieve similar photos from such a large database, Zhentu has chosen Milvus as the feature vector similarity calculation engine. The general structure of the photo fraud detection system is shown in the diagram below.

![img](https://assets.zilliz.com/Structure_of_the_photo_fraud_detection_system_cf5d20d431.png "Structure of the photo fraud detection system")

The procedure can be divided into four steps:

1. Image pre-processing. Pre-processing, including noise reduction, noise removal, and contrast enhancement, ensures both the integrity of the original information and the removal of useless information from the image signal.

2. Feature vector extraction. A specially trained deep learning model is used to extract the feature vectors of the image. Converting images into vectors for further similarity search is a routine operation.

3. Normalization. Normalizing the extracted feature vectors helps to improve the efficiency of the subsequent processing.

4. Vector search with Milvus. Inserting the normalized feature vectors into Milvus database for vector similarity search.

## **Deployment**

Here is a brief description of how Zhentu's photo fraud detection system is deployed.

![Milvus system architecture](https://assets.zilliz.com/milvus_architecture_ea45a5ab53.png)

We deployed our [Milvus cluster on Kubernetes](https://milvus.io/docs/v2.0.x/install_cluster-helm.md) to ensure high availability and real-time synchronization of cloud services. The general steps are as follows:

1. View available resources. Run the command `kubectl describe nodes` to see the resources that the Kubernetes cluster can allocate to the created cases.

2. Allocate resources. Run the command `kubect`` -- apply xxx.yaml` to allocate memory and CPU resources for Milvus cluster components using Helm.

3. Apply the new configuration. Run the command `helm upgrade my-release milvus/milvus --reuse-values -fresources.yaml`.

4. Apply the new configuration to the Milvus cluster. The cluster deployed in this way not only allows us to adjust system capacity according to different business needs, but also better meets the high-performance requirements for massive vector data retrieval.

You can [configure Milvus](https://milvus.io/docs/v2.0.x/configure-docker.md) to optimize search performance for different types of data from different business scenarios, as shown in the following two examples.

In [building the vector index](https://milvus.io/docs/v2.0.x/build_index.md), we parameterize the index according to the actual scenario of the system as follows:

```Python
index = {"index_type": "IVF_PQ", "params": {"nlist": 2048}, "metric_type": "IP"}
```

[IVF_PQ](https://milvus.io/docs/v2.0.x/index.md#IVF_PQ) performs IVF index clustering before quantizing the product of vectors. It features high-speed disk query and very low memory consumption, which meets the needs of the real-world application of Zhentu.

Besides, we set the optimal search parameters as follows:

```Python
search_params = {"metric_type": "IP", "params": {"nprobe": 32}}
```

As the vectors are already normalized before input into Milvus, the inner product (IP) is chosen to calculate the distance between two vectors. Experiments have proved that the recall rate is raised by about 15% using IP than using the Euclidean distance (L2). 

The above examples show that we can test and set Milvus' parameters according to different business scenarios and performance requirements. 

In addition, Milvus not only integrates different index libraries, but also supports different index types and similarity calculation methods. Milvus also provides official SDKs in multiple languages and rich APIs for insertion, querying, etc., allowing our front-end business groups to use the SDKs to call on the risk control center.

## **Real-world performance**

So far, the photo fraud detection system has been running steadily, helping businesses to identify potential fraudsters. In 2021, it detected over 20,000 fake licenses throughout the year. In terms of query speed, a single vector query among tens of millions of vectors takes less than 1 second, and the average time of batch query is less than 0.08 seconds. Milvus' high-performance search meets businesses' needs for both accuracy and concurrency.

## **Reference**

Aglave P, Kolkure V S. Implementation of High Performance Feature Extraction Method Using Oriented Fast and Rotated Brief Algorithm[J]. Int. J. Res. Eng. Technol, 2015, 4: 394-397. 

## **About BestPay**

China Telecom BestPay Co., Ltd is a wholly owned subsidiary of China Telecom. It operates the payment and finance businesses. BestPay is committed to using cutting-edge technologies such as big data, artificial intelligence and cloud computing to empower business innovation, providing intelligent products, risk control solutions and other services. Up to January 2016, the app called BestPay has attracted over 200 million users and become the third largest payment platform operator in China, closely following Alipay and WeChat Payment.


Zhentu - the Photo Fraud Detector Based on Milvus


![Cover image](https://assets.zilliz.com/Tech_Modify_5_e18025ffbc.png "Raft or not? The Best Solution to Data Consistency in Cloud-native Databases")

> This article was written by [Xiaofan Luan](https://github.com/xiaofan-luan) and transcreated by [Angela Ni](https://www.linkedin.com/in/yiyun-n-2aa713163/).

Consensus-based replication is a widely-adopted strategy in many cloud-native, distributed databases. However, it has certain shortcomings and is definitely not the silver bullet.

This post aims to first explain the concepts of replication, consistency, and consensus in a cloud-native and distributed database, then clarify why consensus-based algorithms like Paxos and Raft are not the silver bullet, and finally propose a [solution to consensus-based replication](#a-log-replication-strategy-for-cloud-native-and-distributed-database).

**Jump to:**
- [Understanding replication, consistency, and consensus](#Understanding-replication-consistency-and-consensus)
- [Consensus-based replication](#Consensus-based-replication)
- [A log replication strategy for cloud-native and distributed database](#A-log-replication-strategy-for-cloud-native-and-distributed-database)
- [Summary](#Summary)


## Understanding replication, consistency, and consensus

Before going deep into the pros and cons of Paxos and Raft, and proposing a best suited log replication strategy, we need to first demystify concepts of replication, consistency, and consensus.

Note that this article mainly focuses on the synchronization of incremental data/log. Therefore, when talking about data/log replication, only replicating incremental data, not historical data, is referred to.

### Replication

Replication is the process of making multiple copies of data and storing them in different disks, processes, machines, clusters, etc, for the purpose of increasing data reliability and accelerating data queries. Since in replication, data are copied and stored at multiple locations, data are more reliable in the face of recovering from disk failures, physical machine failures, or cluster errors. In addition, multiple replicas of data can boost the performance of a distributed database by greatly speeding up queries.

There are various modes of replication, such as synchronous/asynchronous replication, replication with strong/eventual consistency, leader-follower/decentralized replication. The choice of replication mode has an effect on system availability and consistency. Therefore, as proposed in the famous [CAP theorem](https://medium.com/analytics-vidhya/cap-theorem-in-distributed-system-and-its-tradeoffs-d8d981ecf37e), a system architect needs to trade off between consistency and availability when network partition is inevitable.

### Consistency

In short, consistency in a distributed database refers to the property that ensures every node or replica has the same view of data when writing or reading data at a given time. For a full list of consistency levels, read the doc [here](https://docs.microsoft.com/en-us/azure/cosmos-db/consistency-levels).

To clarify, here we are talking about consistency as in the CAP theorem, not ACID (atomicity, consistency, isolation, durability). Consistency in CAP theorem refers to each node in the system having the same data while consistency in ACID refers to a single node enforcing the same rules on every potential commit.

Generally, OLTP (online transaction processing) databases require strong consistency or linearizability to ensure that:
- Each read can access the latest inserted data.
- If a new value is returned after a read, all following reads, regardless on the same or different clients, must return the new value.

The essence of linearizability is to guarantee the recency of multiple data replicas - once a new value is written or read, all subsequent reads can view the new value until the value is later overwritten. A distributed system providing linearizability can save users the trouble of keeping an eye on multiple replicas, and can guarantee the atomicity and order or each operation.

### Consensus

The concept of consensus is introduced to distributed systems as users are eager to see distributed systems work in the same way as standalone systems.

To put it simple, consensus is a general agreement on value. For instance, Steve and Frank wanted to grab something to eat. Steve suggested having sandwiches. Frank agreed to Steve's suggestion and both of them are had sandwiches. They reached a consensus. More specifically, a value (sandwiches) proposed by one of them is agreed upon by both, and both of them take actions based on the value. Similarly, consensus in a distributed system means when a process propose a value, all the rest processes in the system agree on and act upon this value. 


![Consensus](https://assets.zilliz.com/2bb46e57_9eb5_456e_be7e_e7762aa9eb7e_68dd2e8e65.png "Consensus in a distributed system.")

## Consensus-based replication

The earliest consensus-based algorithms were proposed  along with [viewstamped replication](https://pmg.csail.mit.edu/papers/vr.pdf) in 1988. In 1989, Leslie Lamport proposed [Paxos](https://lamport.azurewebsites.net/pubs/paxos-simple.pdf), a consensus-based algorithm.

In recent years, we witness another prevalent consensus-based algorithm in the industry - [Raft](https://raft.github.io/). It has been adopted by many mainstream NewSQL databases like CockroachDB, TiDB, OceanBase, etc.

Notably, a distributed system does not necessarily support linearizability even if it adopts consensus-based replication. However, linearizability is the prerequisite for building ACID distributed database.

When designing a database system, the commit order of logs and state machines should be taken into consideration. Extra caution is also needed to maintain the leader lease of Paxos or Raft and prevent a split-brain under network partition.


![Raft replication state machine](https://user-images.githubusercontent.com/1500781/165926429-69b5144c-f3ba-4819-87c3-ab7e04a7e22e.png "Structure of Raft replication state machine.")

### Pros and cons

Indeed, Raft, ZAB, and [quorum-based log protocol](https://aws.amazon.com/blogs/database/amazon-aurora-under-the-hood-quorum-and-correlated-failure/) in Aurora are all Paxos variations. Consensus-based replication has the following advantages:

1. Though consensus-based replication focus more on consistency and network partition in the CAP theorem, it provides relatively better availability compared to traditional leader-follower replication. 
2. Raft is a breakthrough that greatly simplified consensus-based algorithms. And there are many open-source Raft libraries on GitHub (Eg. [sofa-jraft](https://github.com/sofastack/sofa-jraft)).
3. The performance of consensus-based replication can satisfy most of the applications and businesses. With the coverage of high-performance SSD and gigabyte NICs (network interface card), the burden of synchronizing multiple replicas is relieved, making Paxos and Raft algorithms the mainstream in the industry.

One misconception is that consensus-based replication is the silver bullet to achieving data consistency in a distributed database. However, this is not the truth. The challenges in availability, complexity, and performance faced by consensus-based algorithm blocks it from being the perfect solution.

1. Compromised availability
Optimized Paxos or Raft algorithm has strong dependency on the leader replica, which comes with a weak ability to fight against grey failure. In consensus-based replication, a new election of leader replica will not take place until the leader node does not respond for a long time. Therefore, consensus-based replication is incapable of handling situations where the leader node is slow or a thrashing occurs.

2. High complexity
Though there are already many extended algorithms based on Paxos and Raft, the emergence of [Multi-Raft](http://www.vldb.org/pvldb/vol13/p3072-huang.pdf) and [Parallel Raft](https://www.vldb.org/pvldb/vol11/p1849-cao.pdf) requires more considerations and tests on synchronization between logs and state machines. 

3. Compromised performance
In a cloud-native era, local storage is replaced by shared storage solutions like EBS and S3 to ensure data reliability and consistency. As a result, consensus-based replication is no longer a must for distributed systems. What's more, consensus-based replication comes with the problem of data redundancy as both the solution and EBS has multiple replicas.

For multi-datacenter and multi-cloud replication, the pursuit for consistency compromises not only availability but also [latency](https://en.wikipedia.org/wiki/PACELC_theorem), resulting in a decline in performance. Therefore, linearizability is not a must for multi-datacenter disaster tolerance in most of the applications. 

## A log replication strategy for cloud-native and distributed database

Undeniably, consensus-based algorithms like Raft and Paxos are still the mainstream algorithms adopted by many OLTP databases. However, by observing the examples of [PacificA](https://www.microsoft.com/en-us/research/publication/pacifica-replication-in-log-based-distributed-storage-systems/) protocol, [Socrates](https://www.microsoft.com/en-us/research/uploads/prod/2019/05/socrates.pdf) and [Rockset](https://rockset.com/), we can see the trend is shifting. 

There are two major principles for a solution that can best serve a cloud-native, distributed database.

### 1. Replication as a service

A separate microservice dedicated to data synchronization is needed. The synchronization module and storage module should no longer be tightly coupled within the same process.

For instance, Socrates decouples storage, log, and computing. There is one dedicated log service (XLog service in the middle of the figure below).


![Socrates architecture](https://assets.zilliz.com/1_0d7822a781.png "The architecture of Socrates.")

XLog service is an individual service. Data persistence is achieved with the help of low-latency storage. The landing zone in Socrates is in charge of keeping three replicas at an accelerated speed.


![Socrates XLog service](https://assets.zilliz.com/2_6d1182b6f1.png "The Socrates XLog service.")

The leader node distributes logs to log broker asynchronously, and flushes data to Xstore. Local SSD cache can accelerate data read. Once data flush is successful, buffers in the landing zone can be cleaned. Obviously, all log data are divided into three layers - landing zone, local SSD, and XStore.

### 2. Russian doll principle

One way to design a system is to follow the Russian doll principle: each layer is complete and perfectly suited to what the layer does so that other layers may be built on-top or around it. 

When designing a cloud-native database, we need to cleverly leverage other third-party services to reduce the complexity of system architecture.

It seems like we cannot get around with Paxos to avoid single point failure. However, we can still greatly simplify log replication by handing leader election over to Raft or Paxos services based on [Chubby](https://research.google.com/archive/chubby-osdi06.pdf), [Zk](https://github.com/bloomreach/zk-replicator), and [etcd](https://etcd.io/).

For instance, [Rockset](https://rockset.com/) architecture follows the Russian doll principle and uses Kafka/Kineses for distributed logs, S3 for storage, and local SSD cache for improving query performance. 


![Rockset architecture](https://user-images.githubusercontent.com/1500781/165926697-c8b380dc-d71a-41a9-a76d-a261b77f0b5d.png "Rockset architecture.")

### The Milvus approach

Tunable consistency in Milvus is in fact similar to follower reads in consensus-based replication. Follower read feature refers to using follower replica to undertake data read tasks under the premise of strong consistency. The purpose is to enhance cluster throughput and reduce the load on leader. The mechanism behind follower read feature is inquiring the commit index of the latest log and providing query service until all data in the commit index are applied to state machines.

However, the design of Milvus did not adopt the follower strategy. In other words, Milvus does not inquire commit index every time it receives a query request. Instead, Milvus adopts a mechanism like the watermark in [Flink](https://flink.apache.org/), which notifies query node the location of commit index at a regular interval. The reason for such a mechanism is that Milvus users usually do not have high demand for data consistency, and they can accept a compromise in data visibility for better system performance. 

In addition, Milvus also adopts multiple microservices and separates storage from computing. In the [Milvus architecture](https://milvus.io/blog/deep-dive-1-milvus-architecture-overview.md#A-bare-bones-skeleton-of-the-Milvus-architecture), S3, MinIo, and Azure Blob are used for storage.


![Milvus architecture](https://assets.zilliz.com/Milvus_architecture_b7743a4a7f.png "Milvus architecture.")

## Summary

Nowadays, an increasing number of cloud-native databases are making log replication an individual service. By doing so, the cost of adding read-only replicas and heterogeneous replication can be reduced. Using multiple microservices enables quick utilization of mature cloud-based infrastructure, which is impossible for traditional databases. An individual log service can rely on consensus-based replication, but it can also follow the Russian doll strategy to adopt various consistency protocols together with Paxos or Raft to achieve linearizability.

## References

- Lamport L. Paxos made simple[J]. ACM SIGACT News (Distributed Computing Column) 32, 4 (Whole Number 121, December 2001), 2001: 51-58.
- Ongaro D, Ousterhout J. In search of an understandable consensus algorithm[C]//2014 USENIX Annual Technical Conference (Usenix ATC 14). 2014: 305-319.
- Oki B M, Liskov B H. Viewstamped replication: A new primary copy method to support highly-available distributed systems[C]//Proceedings of the seventh annual ACM Symposium on Principles of distributed computing. 1988: 8-17.
- Lin W, Yang M, Zhang L, et al. PacificA: Replication in log-based distributed storage systems[J]. 2008.
- Verbitski A, Gupta A, Saha D, et al. Amazon aurora: On avoiding distributed consensus for i/os, commits, and membership changes[C]//Proceedings of the 2018 International Conference on Management of Data. 2018: 789-796.
- Antonopoulos P, Budovski A, Diaconu C, et al. Socrates: The new sql server in the cloud[C]//Proceedings of the 2019 International Conference on Management of Data. 2019: 1743-1756.


Raft or not? The Best Solution to Data Consistency in Cloud-native Databases


![cover image](https://assets.zilliz.com/Deep_Dive_8_6919720d59.png "What Powers Similarity Search in Milvus Vector Database?")

> This article is written by [Yudong Cai](https://github.com/cydrain) and translated by [Angela Ni](https://www.linkedin.com/in/yiyun-n-2aa713163/).

As the core vector execution engine, Knowhere to Milvus is what an engine is to a sports car. This post introduces what Knowhere is, how it is different from Faiss, and how the code of Knowhere is structured.

**Jump to:**
- [The concept of Knowhere](#The-concept-of-Knowhere)
- [Knowhere in the Milvus architecture](#Knowhere-in-the-Milvus-architecture)
- [Knowhere Vs Faiss](#Knowhere-Vs-Faiss)
- [Understanding the Knowhere code](#Understanding-the-Knowhere-code)
- [Adding indexes to Knowhere](#Adding-indexes-to-Knowhere)

## The concept of Knowhere
Narrowly speaking, Knowhere is an operation interface for accessing services in the upper layers of the system and vector similarity search libraries like [Faiss](https://github.com/facebookresearch/faiss), [Hnswlib](https://github.com/nmslib/hnswlib), [Annoy](https://github.com/spotify/annoy) in the lower layers of the system. In addition, Knowhere is also in charge of heterogeneous computing. More specifically, Knowhere controls on which hardware (eg. CPU or GPU) to execute index building and search requests. This is how Knowhere gets its name - knowing where to execute the operations. More types of hardware including DPU and TPU will be supported in future releases.

In a broader sense, Knowhere also incorporates other third-party index libraries like Faiss. Therefore, as a whole, Knowhere is recognized as the core vector computation engine in the Milvus vector database.

From the concept of Knowhere, we can see that it only processes data computing tasks, while those tasks like sharding, load balance, disaster recovery are beyond the work scope of Knowhere.

Starting from Milvus 2.0.1, [Knowhere](https://github.com/milvus-io/knowhere) (in the broader sense) becomes independent from the Milvus project.

## Knowhere in the Milvus architecture


![knowhere architecture](https://assets.zilliz.com/ec63d1e9_86e1_48e3_9d75_8fed305bbcb5_26b842e9f6.png "Knowhere architecture in Milvus.")


Computation in Milvus mainly involves vector and scalar operations. Knowhere only handles the operations on vectors in Milvus. The figure above illustrates the Knowhere architecture in Milvus.

The bottom-most layer is the system hardware. The third-party index libraries are on top of the hardware. Then Knowhere interacts with the index node and query node on the top via CGO. 

This article talks about Knowhere in its broader sense, as marked within the blue frame in the architecture illustration.

## Knowhere Vs Faiss

Knowhere not only further extends the functions of Faiss but also optimizes the performance. More specifically, Knowhere has the following advantages.

### 1. Support for BitsetView

Initially, bitset was introduced in Milvus for the purpose of "soft deletion". A soft-deleted vector still exists in the database but will not be computed during a vector similarity search or query. Each bit in the bitset corresponds to an indexed vector. If a vector is marked as "1" in the bitset, it means this vector is soft-deleted and will not be involved during a vector search.

The bitset parameters are added to all the exposed Faiss index query APIs in Knowhere, including CPU and GPU indexes. 

Learn more about [how bitset enables the versatility of vector search](https://milvus.io/blog/2022-2-14-bitset.md).

### 2. Support for more similarity metrics for indexing binary vectors

Apart from [Hamming](https://milvus.io/docs/v2.0.x/metric.md#Hamming-distance), Knowhere also supports [Jaccard](https://milvus.io/docs/v2.0.x/metric.md#Jaccard-distance), [Tanimoto](https://milvus.io/docs/v2.0.x/metric.md#Tanimoto-distance), [Superstructure](https://milvus.io/docs/v2.0.x/metric.md#Superstructure), [Substructure](https://milvus.io/docs/v2.0.x/metric.md#Substructure). Jaccard and Tanimoto can be used to measure the similarity between two sample sets while Superstructure and Substructure can be used to measure the similarity of chemical structures.

### 3. Support for AVX512 instruction set

Faiss itself supports multiple instruction sets including [AArch64](https://en.wikipedia.org/wiki/AArch64), [SSE4.2](https://en.wikipedia.org/wiki/SSE4#SSE4.2), [AVX2](https://en.wikipedia.org/wiki/Advanced_Vector_Extensions). Knowhere further extends the supported instruction sets by adding [AVX512](https://en.wikipedia.org/wiki/AVX-512), which can [improve the performance of index building and query by 20% to 30%](https://milvus.io/blog/milvus-performance-AVX-512-vs-AVX2.md) compared to AVX2.

### 4. Automatic SIMD-instruction selection

Knowhere is designed to work well on a wide spectrum of CPU processors (both on-premises and cloud platforms) with different SIMD instructions (e.g., SIMD SSE, AVX, AVX2, and AVX512). Thus the challenge is, given a single piece of software binary (i.e., Milvus), how to make it automatically invoke the suitable SIMD instructions on any CPU processor? Faiss does not support automatic SIMD-instruction selection and users need to manually specify the SIMD flag (e.g., “-msse4”) during compilation. However, Knowhere is built by refactoring the codebase of Faiss. Common functions (e.g., similarity computing) that rely on SIMD accelerations are factored out. Then for each function, four versions (i.e., SSE, AVX, AVX2, AVX512) are implemented and each put into a separate source file. Then the source files are further compiled individually with the corresponding SIMD flag. Therefore, at runtime, Knowhere can automatically choose the best suited SIMD instructions based on the current CPU flags and then link the right function pointers using hooking. 

### 5. Other performance optimization
Read [Milvus: A Purpose-Built Vector Data Management System](https://www.cs.purdue.edu/homes/csjgwang/pubs/SIGMOD21_Milvus.pdf) for more about Knowhere's performance optimization.

## Understanding the Knowhere code

As mentioned in the first section, Knowhere only handles vector search operations. Therefore, Knowhere only processes the vector field of an entity (currently, only one vector field for entities in a collection is supported). Index building and vector similarity search are also targeted at the vector field in a segment. To have a better understanding of the data model, read the blog [here](https://milvus.io/blog/deep-dive-1-milvus-architecture-overview.md#Data-Model).


![entity fields](https://assets.zilliz.com/Entity_fields_6aa517cc4c.png "Entity fields.")

### Index

Index is a type of independent data structure from the original vector data. Indexing requires four steps: create an index, train data, insert data and build an index.

For some of the AI applications, dataset training is an individual process from vector search. In this type of application, data from datasets are first trained and then inserted into a vector database like Milvus for similarity search. Open datasets like sift1M and sift1B provides data for training and testing. However, in Knowhere, data for training and searching are mixed together. That is to say, Knowhere trains all the data in a segment and then inserts all the trained data and builds an index for them.

### Knowhere code structure

DataObj is the base class of all data structure in Knowhere. `Size()` is the only virtual method in DataObj. The Index class inherits from DataObj with a field named "size_". The Index class also has two virtual methods - `Serialize()` and `Load()`. The VecIndex class derived from Index is the virtual base class for all vector indexes. VecIndex provides methods including `Train()`, `Query()`, `GetStatistics()`, and `ClearStatistics()`.



![base clase](https://assets.zilliz.com/Knowhere_base_classes_9d610618d9.png "Knowhere base classes.")

Other index types are listed on the right in the figure above. 

- The Faiss index has two sub classes: FaissBaseIndex for all indexes on float point vectors, and FaissBaseBinaryIndex for all indexes on binary vectors.
- GPUIndex is the base class for all Faiss GPU indexes.
- OffsetBaseIndex is the base class for all self-developed indexes. Only vector ID is stored in the index file. As a result, an index file size for 128-dimensional vectors can be reduced by 2 orders of magnitude. We recommend taking the original vectors into consideration as well when using this type of index for vector similarity search.



![IDMAP](https://assets.zilliz.com/IDMAP_8773a4511c.png "IDMAP code structure.")

Technically speaking, [IDMAP](https://github.com/facebookresearch/faiss/wiki/Guidelines-to-choose-an-index#then-flat) is not an index, but rather used for brute-force search. When vectors are inserted to the vector database, no data training and index building is required. Searches will be conducted directly on the inserted vector data.

However, for the sake of code consistency, IDMAP also inherits from the VecIndex class with all its virtual interfaces. The usage of IDMAP is the same as other indexes.



![IVF](https://assets.zilliz.com/IVF_42b0f123d1.png "Code structure of IVF indexes.")

The IVF (inverted file) indexes are the most frequently used. The IVF class is derived from VecIndex and FaissBaseIndex, and further extends to IVFSQ and IVFPQ. GPUIVF is derived from GPUIndex and IVF. Then GPUIVF further extends to GPUIVFSQ and GPUIVFPQ.

IVFSQHybrid is a class for self-developed hybrid index that is executed by coarse quantize on GPU. And search in the bucket is executed on CPU. This type of index can reduce the occurrence of memory copy between CPU and GPU by leveraging the computing power of GPU. IVFSQHybrid has the same recall rate as GPUIVFSQ but comes with a better performance.

The base class structure for binary indexes are relatively simpler. BinaryIDMAP and BinaryIVF are derived from FaissBaseBinaryIndex and VecIndex.



![third-party index](https://assets.zilliz.com/third_party_index_34ad029848.png "Code structure of other third-party indexes.")

Currently, only two types of third-party indexes are supported apart from Faiss: tree-based index Annoy, and graph-based index HNSW. These two common and frequently used third-party indexes are both derived from VecIndex. 

## Adding indexes to Knowhere

If you want to add new indexes to Knowhere, you can refer to existing indexes first:
- To add quantization-based index, refer to IVF_FLAT. 
- To add graph-based index, refer to HNSW. 
- To add tree-based index, refer to Annoy.

After referring to the existing index, you can follow the steps below to add a new index to Knowhere.
1. Add the name of the new index in `IndexEnum`. The data type is string.
2. Add data validation check on the new index in the file `ConfAdapter.cpp`. The validation check is mainly to validate the parameters for data training and query.
3. Create a new file for the new index. The base class of the new index should include `VecIndex`, and the necessary virtual interface of `VecIndex`.
4. Add the index building logic for new index in `VecIndexFactory::CreateVecIndex()`.
5. Add unit test under the `unittest` directory.


## About the Deep Dive Series

With the [official announcement of general availability](https://milvus.io/blog/2022-1-25-annoucing-general-availability-of-milvus-2-0.md) of Milvus 2.0, we orchestrated this Milvus Deep Dive blog series to provide an in-depth interpretation of the Milvus architecture and source code. Topics covered in this blog series include:

- [Milvus architecture overview](https://milvus.io/blog/deep-dive-1-milvus-architecture-overview.md)
- [APIs and Python SDKs](https://milvus.io/blog/deep-dive-2-milvus-sdk-and-api.md)
- [Data processing](https://milvus.io/blog/deep-dive-3-data-processing.md)
- [Data management](https://milvus.io/blog/deep-dive-4-data-insertion-and-data-persistence.md)
- [Real-time query](https://milvus.io/blog/deep-dive-5-real-time-query.md)
- [Scalar execution engine](https://milvus.io/blog/deep-dive-7-query-expression.md)
- [QA system](https://milvus.io/blog/deep-dive-6-oss-qa.md)
- [Vector execution engine](https://milvus.io/blog/deep-dive-8-knowhere.md)


What Powers Similarity Search in Milvus Vector Database?


![Cover image](https://assets.zilliz.com/Deep_Dive_7_baae830823.png "How Does the Database Understand and Execute Your Query?")

> This article is transcreated by [Angela Ni](https://www.linkedin.com/in/yiyun-n-2aa713163/).

A [vector query](https://milvus.io/docs/v2.0.x/query.md) in Milvus is the process of retrieving vectors via scalar filtering based on boolean expression. With scalar filtering, users can limit their query results with certain conditions applied on attributes of data. For instance, if a user queries for films released during 1990-2010 and with scores higher than 8.5, only films whose attributes (release year and score) fulfill the condition.

This post aims to examine how a query is completed in Milvus from the input of a query expression to query plan generation and query execution. 

**Jump to:**
- [Query expression](#Query-expression)
- [Plan AST generation](#Plan-AST-generation)
- [Query execution](#Query-execution)

## Query expression

Expression of query with attribute filtering in Milvus adopts the EBNF(Extended Backus–Naur form) syntax. The image below is the expression rules in Milvus.

![Expression Syntax](https://assets.zilliz.com/Expression_Syntax_966493a5be.png "The EBNF syntax of a logical expression.")

Logical expressions can be created using the combination of binary logical operators, unary logical operators, logical expressions, and single expressions. Since EBNF syntax is itself recursive, a logical expression can be the outcome of the combination or part of a bigger logical expression. A logical expression can contain many sub-logical expressions. The same rule applies in Milvus. If a user needs to filter the attributes of the results with many conditions, the user can create his own set of filtering conditions by combining different logical operators and expressions.

![Boolean expression](https://assets.zilliz.com/Boolean_expression_1_dce12f8483.png "Boolean expression rules in Milvus.")

The image above shows part of the [Boolean expression rules](https://milvus.io/docs/v2.0.x/boolean.md) in Milvus. Unary logical operators can be added to an expression. Currently Milvus only supports the unary logical operator "not", which indicates that the system needs to take the vectors whose scalar field values do not satisfy the calculation results. Binary logical operators include "and" and "or". Single expressions include term expressions and compare expressions.

Basic arithmetic calculation like addition, subtraction, multiplication, and division is also supported during a query in Milvus. The following image demonstrates the precedence of the operations. Operators are listed from top to bottom in descending precedence.

![Precedence](https://assets.zilliz.com/Precedence_b8cfbdf17b.png "The precedence of operations in Milvus.")

### How a query expression on certain films is processed in Milvus?

Suppose there is an abundance of film data stored in Milvus and the user wants to query certain films. As an example, each film data stored in Milvus has the following five fields: film ID, release year, film type, score, and poster. In this example, the data type of film ID and release year is int64, while film scores are float point data. Also, film posters are stored in the format of float-point vectors, and film type in the format of string data. Notably, support for string data type is a new feature in Milvus 2.1.

For instance, if a user want to query the movies with scores higher than 8.5. The films should also be  released during a decade before 2000 to a decade after 2000 or their types should be either comedy or action movie, the user need to input the following predicate expression: `score > 8.5 && (2000 - 10 < release_year < 2000 + 10 || type in [comedy,action])`.

Upon receiving the query expression, the system will execute it in the following precedence:

1. Query for films with scores higher than 8.5. The query results are called "result1".
2. Calculate 2000 - 10 to get "result2" (1990).
3. Calculate 2000 + 10 to get "result3" (2010).
4. Query for films with the value of `release_year` greater than "result2" and smaller than "result3". That is to say, the system needs to query for films released between 1990 and 2010. The query results are called "result4".
5. Query for films that are either comedies or action movies. The query results are called "result5".
6. Combine "result4" and "result5" to obtain films that are either released between 1990 and 2010 or belong to the category of comedy or action movie. The results are called "result6".
7. Take the common part in "result1" and "result6" to obtain the final results satisfying all the conditions.

![Film example](https://assets.zilliz.com/Frame_1_16_00972a6e5d.png "Querying films in the database.")

## Plan AST generation

Milvus leverages the open-source tool [ANTLR](https://www.antlr.org/) (ANother Tool for Language Recognition) for plan AST (abstract syntax tree) generation. ANTLR is a powerful parser generator for reading, processing, executing, or translating structure text or binary files. More specifically, ANTLR can generate a parser for building and walking parse trees based on pre-defined syntax or rules. The following image is an example in which the input expression is "SP=100;". LEXER, the built-in language recognition functionality in ANTLR, generates four tokens for the input expression - "SP", "=", "100", and ";". Then the tool will further parse the four tokens to generate the corresponding parse tree.

![parse tree](https://assets.zilliz.com/parse_tree_b2c3fb0b36.png "Generating a parse tree for the input expression.")

The walker mechanism is a crucial part in the ANTLR tool. It is designed to walk through all the parse trees to examine whether each node obeys the syntax rules, or to detect certain sensitive words. Some of the relevant APIs are listed in the image below. Since ANTLR starts from the root node and goes down all the way through each sub-node to the bottom, there is no need to differentiate the order of how to walk through the parse tree. 

![parse tree walker](https://assets.zilliz.com/parse_tree_walker_9a27942502.png "The parse-tree walker mechanism in ANTLR.")

Milvus generates the PlanAST for query in a similar way to the ANTLR. However, using ANTLR requires redefining rather complicated syntax rules. Therefore, Milvus adopts one of the most prevalent rules - Boolean expression rules, and depends on the [Expr](https://github.com/antonmedv/expr) package open sourced on GitHub to query and parse the syntax of query expressions.

During a query with attribute filtering, Milvus will generate a primitive unsolved plan tree using ant-parser, the parsing method provided by Expr, upon receiving the query expression. The primitive plan tree we will get is a simple binary tree. Then the plan tree is fine-tuned by Expr and the built-in optimizer in Milvus. The optimizer in Milvus is quite similar to the aforementioned walker mechanism. Since the plan tree optimization functionality provided by Expr is pretty sophisticated, the burden of the Milvus built-in optimizer is alleviated to a great extent. Ultimately, the analyzer analyzes the optimized plan tree in a recursive way to generate a plan AST in the structure of [protocol buffers](https://developers.google.com/protocol-buffers) (protobuf).

![plan AST workflow](https://assets.zilliz.com/plan_AST_workflow_3e50b7a0d4.png "The workflow of generating a plan AST in Milvus.")

## Query execution

Query execution is at root the execution of the plan AST generated in the previous steps.

In Milvus, a plan AST is defined in a proto structure. The image below is a message with the protobuf structure. There are six types of expressions, among which binary expression and unary expression can further have binary logical expression and unary logical expression.

![protobuf1](https://assets.zilliz.com/Protobuf1_232132dcf2.png "A query message with the protobuf structure.")

![protobuf2](https://assets.zilliz.com/protobuf2_193f92f033.png "A query message with the protobuf structure.")

The image below is a UML image of the query expression. It demonstrates the basic class and derivative class of each expression. Each class comes with a method to accept visitor parameters. This is a typical visitor design pattern. Milvus uses this pattern to execute the plan AST as its biggest advantage is that users do not have to do anything to the primitive expressions but can directly access one of the methods in the patterns to modify certain query expression class and relevant elements.

![UML](https://assets.zilliz.com/UML_1238bc30e1.png "A UML image of the query expression.")

When executing a plan AST, Milvus first receives a proto-type plan node. Then a segcore-type plan node is obtained via the internal C++ proto parser. Upon obtaining the two types of plan nodes, Milvus accepts a series of class access and then modifies and executes in the internal structure of the plan nodes. Finally, Milvus searches through all the execution plan nodes to obtain the filtered results. The final results are output in the format of a bitmask. A bitmask is an array of bit numbers ("0" and "1"). Those data satisfying filter conditions are marked as "1" in the bitmask, while those do not meet the requirements are marked as "0" in the bitmask.

![execute workflow](https://assets.zilliz.com/execute_workflow_d89f1ee925.png "The workflow of executing a plan AST in Milvus.")

## About the Deep Dive Series

With the [official announcement of general availability](https://milvus.io/blog/2022-1-25-annoucing-general-availability-of-milvus-2-0.md) of Milvus 2.0, we orchestrated this Milvus Deep Dive blog series to provide an in-depth interpretation of the Milvus architecture and source code. Topics covered in this blog series include:

- [Milvus architecture overview](https://milvus.io/blog/deep-dive-1-milvus-architecture-overview.md)
- [APIs and Python SDKs](https://milvus.io/blog/deep-dive-2-milvus-sdk-and-api.md)
- [Data processing](https://milvus.io/blog/deep-dive-3-data-processing.md)
- [Data management](https://milvus.io/blog/deep-dive-4-data-insertion-and-data-persistence.md)
- [Real-time query](https://milvus.io/blog/deep-dive-5-real-time-query.md)
- [Scalar execution engine](https://milvus.io/blog/deep-dive-7-query-expression.md)
- [QA system](https://milvus.io/blog/deep-dive-6-oss-qa.md)
- [Vector execution engine](https://milvus.io/blog/deep-dive-8-knowhere.md)






How Does the Database Understand and Execute Your Query?

Open Source Software (OSS) Quality Assurance - A Milvus Case Study


![Cover image](https://assets.zilliz.com/20220516_163732_d40acac221.png "Milvus reached 10K stars on GitHub.")

We recently hit an incredible milestone in the journey of vector database - over 10K stars on the [GitHub repo of Milvus](https://github.com/milvus-io/milvus). Thanks to each and every one of you who has supported us by giving us a star ⭐️.

It has been a wonderful 2.5 years since Milvus became an open-source project. When we first started, very few people understood exactly what a vector database is, let alone talking about how it would enable and accelerate AI adoption in our daily lives. Milvus vector database carries our vision of defining and building the next-generation database for AI. We combine SOTA algorithms with cloud technologies in the Milvus project so developers could build AI applications at ease with Milvus vector database.

![Gif2](https://assets.zilliz.com/Milvus_0_10000_d50a914332.gif "Milvus repository development visualization - road to 10,000 stargazers.")

<div align="center">Click <a href="https://www.youtube.com/watch?v=zIAX_oPI2Jk&ab_channel=Milvus-VectorDatabase">here</a> to watch the full video of Milvus repository development visualization.</div>

<br/>


There are many [brilliant projects](https://github.com/milvus-io/milvus) that laid the foundation for Milvus, and we are thankful to stand on the shoulders of the giants. Without them Milvus wouldn't have existed. 

We are also incredibly grateful to our awesome community, who has helped open an exciting chapter for this new type of database. Before we look forward, we want to take a moment to reflect on this journey, gaze at our milestones and acknowledge and thank our developers:

- We joined LF AI & Data Foundation at [incubation](https://lfaidata.foundation/blog/2020/04/02/milvus-joins-lf-ai-as-new-incubation-project/) level, finding our home alongside many other great open source AI and data projects. Now Milvus is already a [graduation project](https://lfaidata.foundation/blog/2021/06/23/lf-ai-data-foundation-announces-graduation-of-milvus-project/). 
- We delivered two major releases: [Milvus 1.0](https://milvus.io/docs/v1.0.0/announcement.md) and [Milvus 2.0](https://milvus.io/docs/v2.0.x/comparison.md). 
- A SIGMOD paper was published: [Milvus: A Purpose-Built Vector Data Management System](https://dl.acm.org/doi/abs/10.1145/3448016.3457550).
- Currently, we have more than 10,000 community members and over 220 contributors worldwide. 
- There are over [1000 enterprise users](https://milvus.io/) who adopted Milvus. Among them, there are many industry leaders. 

![Timeline gif](https://assets.zilliz.com/Milvus_10000_star_timeline_3sec_de1a7c906a.gif "Milvus 10000 star timeline.")

All of this is only made possible by every single person who contributes. Thank you for your pull requests, issue reports, emails, meetings, and sharing your knowledge of Milvus. You all are playing a valuable part in Milvus' progress. 

## Special event

For this special milestone, we have prepared a small thank-you gift – [a downloadable desktop wallpaper](https://assets.zilliz.com/10_K_stars_2b2e0e0b36.png). Hope you enjoy it! 

It'd be nice if you help [spread the word](https://twitter.com/milvusio)! ❤️



A Small Step for Milvus - 10,000 Stars


![Cover image](https://assets.zilliz.com/deep_dive_5_5e9175c7f7.png "Real-time query in Milvus.")

> This article is written by [Xi Ge](https://github.com/xige-16) and transcreated by [Angela Ni](https://www.linkedin.com/in/yiyun-n-2aa713163/).

In the previous post, we have talked about [data insertion and data persistence](https://milvus.io/blog/deep-dive-4-data-insertion-and-data-persistence.md) in Milvus. In this article, we will continue to explain how [different components](https://milvus.io/blog/deep-dive-1-milvus-architecture-overview.md) in Milvus interact with each other to complete real-time data query.

*Some useful resources before getting started are listed below. We recommend reading them first to better understand the topic in this post.*
- [Deep dive into the Milvus architecture](https://milvus.io/blog/deep-dive-1-milvus-architecture-overview.md)
- [Milvus data model](https://milvus.io/blog/deep-dive-1-milvus-architecture-overview.md#Data-Model)
- [The role and function of each Milvus component](https://milvus.io/docs/v2.0.x/four_layers.md)
- [Data processing in Milvus](https://milvus.io/blog/deep-dive-3-data-processing.md)
- [Data insertion and data persistence in Milvus](https://milvus.io/blog/deep-dive-4-data-insertion-and-data-persistence.md)

## Load data to query node

Before a query is executed, the data has to be loaded to the query nodes first.

There are two types of data that are loaded to query node: streaming data from [log broker](https://milvus.io/docs/v2.0.x/four_layers.md#Log-broker), and historical data from [object storage](https://milvus.io/docs/v2.0.x/four_layers.md#Object-storage) (also called persistent storage below).

![Flowchart](https://assets.zilliz.com/flowchart_b1c51dfdaa.png "A flowchart of loading data to query node.")

Data coord is in charge of handling streaming data that are continuously inserted into Milvus. When a Milvus user calls `collection.load()` to load a collection, query coord will inquire the data coord to learn which segments have been persisted in storage and their corresponding checkpoints. A checkpoint is a mark to signify that persisted segments before the checkpoints are consumed while those after the checkpoint are not.

Then, the query coord outputs allocation strategy based on the information from the data coord: either by segment or by channel. The segment allocator is responsible for allocating segments in persistent storage  (batch data) to different query nodes. For instance, in the image above, the segment allocator allocates segment 1 and 3 (S1, S3) to query node 1, and segment 2 and 4 (S2, S4) to query node 2. The channel allocator assigns different query nodes to watch multiple data manipulation [channels](https://milvus.io/docs/v2.0.x/data_processing.md#Data-insertion) (DMChannels) in the log broker. For instance, in the image above, the channel allocator assigns query node 1 to watch  channel 1 (Ch1), and query node 2 to watch channel 2 (Ch2).

With the allocation strategy, each query node loads segment data and watch the channels accordingly. In query node 1 in the image, historical data (batch data), are loaded via the allocated S1 and S3 from persistent storage. In the meanwhile, query node 1 loads incremental data (streaming data) by subscribing to channel 1 in log broker. 

## Data management in query node

A query node needs to manage both historical and incremental data. Historical data are stored in [sealed segments](https://milvus.io/blog/deep-dive-4-data-insertion-and-data-persistence.md#Sealed-segment) while incremental data are stored in [growing segments](https://milvus.io/blog/deep-dive-4-data-insertion-and-data-persistence.md#Growing-segment).

### Historical data management

There are mainly two considerations for historical data management: load balance and query node failover.

![Load balance](https://assets.zilliz.com/load_balance_c77e22bb5c.png "Load balance.")

For instance, as shown in the illustration, query node 4 has been allocated more sealed segments than the rest of the query nodes. Very likely, this will make query node 4 the bottleneck that slows down the whole query process. To solve this issue, the system needs to allocate several segments in query node 4 to other query nodes. This is called load balance.

![Query node failover](https://assets.zilliz.com/Query_node_failover_3278c0e307.png "Query node failover.")

Another possible situation is illustrated in the image above. One of the nodes, query node 4, is suddenly down. In this case, the load (segments allocated to query node 4) needs to be transferred to other working query nodes to ensure the accuracy of query results.

### Incremental data management

Query node watches DMChannels to receive incremental data. Flowgraph is introduced in this process. It first filters all the data insertion messages. This is to ensure that only data in a specified partition is loaded. Each collection in Milvus has a corresponding channel, which is shared by all partitions in that collection. Therefore, a flowgraph is needed for filtering inserted data if a Milvus user only needs to load data in a certain partition. Otherwise, data in all partitions in the collection will be loaded to query node. 

After being filtered, the incremental data are inserted into growing segments, and further passed on to server time nodes.

![Flowgraph](https://assets.zilliz.com/flow_graph_dc58651367.png "Flow graph in the process of streaming data insertion.")

During data insertion, each insertion message is assigned a timestamp. In the DMChannel shown in the image above, data are are inserted in order, from left to right. The timestamp for the first insertion message is 1; the second, 2; and the third, 6. The fourth message marked in red is not an insertion message, but rather a timetick message. This is to signify that inserted data whose timestamps are smaller than this timetick are already in log broker. In other words, data inserted after this timetick message should all have timestamps whose values are bigger than this timetick. For instance, in the image above, when query node perceives that the current timetick is 5, it means all insertion messages whose timestamp value is less than 5 are all loaded to query node. 

The server time node provides an updated `tsafe` value every time it receives a timetick from the insert node. `tsafe` means safety time, and all data inserted before this point of time can be queried. Take an example, if `tsafe` = 9, inserted data with timestamps smaller than 9 can all be queried. 

## Real-time query in Milvus

Real-time query in Milvus is enabled by query messages. Query messages are inserted into log broker by proxy. Then query nodes obtain query messages by watching the query channel in log broker.

### Query message

![Query message](https://assets.zilliz.com/query_message_4d57814f47.png "A query message.")

A query message includes the following crucial information about a query:
- `msgID`: Message ID, the ID of the query message assigned by the system.
- `collectionID`: The ID of the collection to query (if specified by user).
- `execPlan`: The execution plan is mainly used for attribute filtering in a query. 
- `service_ts`: Service timestamp will be updated together with `tsafe` mentioned above. Service timestamp signifies at which point is the service in. All data inserted before `service_ts` are available for query.
- `travel_ts`: Travel timestamp specifies a range of time in the past. And the query will be conducted on data existing in the time period specified by `travel_ts`. 
- `guarantee_ts`: Guarantee timestamp specifies a period of time after which the query needs to be conducted. Query will only be conducted when `service_ts` > `guarantee_ts`.

### Real-time query

![Query process](https://assets.zilliz.com/query_process_7f676972d8.png "The process of a query.")

When a query message is received, Milvus first judges if the current service time, `service_ts`, is larger than the guarantee timestamp, `guarantee_ts`, in the query message. If yes, the query will be executed. Query will be conducted in parallel on both historical data and incremental data. Since there can be an overlap of data between streaming data and batch data , an action called "local reduce" is needed to filter out the redundant query results. 

However, if the current service time is smaller than the guarantee timestamp in a newly inserted query message, the query message will become an unsolved message and wait to be processed till the service time becomes bigger than the guarantee timestamp.

Query results are ultimately pushed to the result channel. Proxy obtains the query results from that channel. Likewise, proxy will conduct a "global reduce" as well because it receives results from multiple query nodes and query results might be repetitive.

To ensure that the proxy has received all query results before returning them to the SDK, result message will also keep a record of information including searched sealed segments, searched DMChannels, and global sealed segments (all segments on all query nodes). The system can conclude that the proxy has received all query results only if both of the following conditions are met:

- The union of all searched sealed segments recorded in all result messages is larger than global sealed segments,
- All DMChannels in the collection are queried.

Ultimately, proxy returns the final results after "global reduce" to the Milvus SDK.

## About the Deep Dive Series

With the [official announcement of general availability](https://milvus.io/blog/2022-1-25-annoucing-general-availability-of-milvus-2-0.md) of Milvus 2.0, we orchestrated this Milvus Deep Dive blog series to provide an in-depth interpretation of the Milvus architecture and source code. Topics covered in this blog series include:

- [Milvus architecture overview](https://milvus.io/blog/deep-dive-1-milvus-architecture-overview.md)
- [APIs and Python SDKs](https://milvus.io/blog/deep-dive-2-milvus-sdk-and-api.md)
- [Data processing](https://milvus.io/blog/deep-dive-3-data-processing.md)
- [Data management](https://milvus.io/blog/deep-dive-4-data-insertion-and-data-persistence.md)
- [Real-time query](https://milvus.io/blog/deep-dive-5-real-time-query.md)
- [Scalar execution engine](https://milvus.io/blog/deep-dive-7-query-expression.md)
- [QA system](https://milvus.io/blog/deep-dive-6-oss-qa.md)
- [Vector execution engine](https://milvus.io/blog/deep-dive-8-knowhere.md)


Using the Milvus Vector Database for Real-Time Query


![Cover image](https://assets.zilliz.com/Deep_Dive_4_812021d715.png "Data Insertion and Data Persistence in a Vector Database.")

> This article is written by [Bingyi Sun](https://github.com/sunby) and transcreated by [Angela Ni](https://www.linkedin.com/in/yiyun-n-2aa713163/).

In the previous post in the Deep Dive series, we have introduced [how data is processed in Milvus](https://milvus.io/blog/deep-dive-3-data-processing.md), the world's most advanced vector database. In this article, we will continue to examine the components involved in data insertion, illustrate the data model in detail, and explain how data persistence is achieved in Milvus.

Jump to:
- [Milvus architecture recap](#Milvus-architecture-recap)
- [The portal of data insertion requests](#The-portal-of-data-insertion-requests)
- [Data coord and data node](#Data-coord-and-data-node)
- [Root coord and Time Tick](#Root-coord-and-Time-Tick)
- [Data organization: collection, partition, shard (channel), segment](#Data-organization-collection-partition-shard-channel-segment)
- [Data allocation: when and how](#Data-allocation-when-and-how)
- [Binlog file structure and data persistence](#Binlog-file-structure-and-data-persistence)

## Milvus architecture recap

![Milvus architecture.](https://assets.zilliz.com/Milvus_architecture_c7910cb89d.png "Milvus architecture.")

SDK sends data requests to proxy, the portal, via load balancer. Then the proxy interacts with coordinator service to write DDL (data definition language) and DML (data manipulation language) requests into message storage.

Worker nodes, including query node, data node, and index node, consume the requests from message storage. More specifically, the query node is in charge of data query; the data node is responsible for data insertion and data persistence; and the index node mainly deals with index building and query acceleration. 

The bottom layer is object storage, which mainly leverages MinIO, S3, and AzureBlob for storing logs, delta binlogs, and index files.

## The portal of data insertion requests

![Proxy in Milvus.](https://assets.zilliz.com/Proxy_in_Milvus_aa6b724e0b.jpeg "Proxy in Milvus.")

Proxy serves as a portal of data insertion requests. 

1. Initially, proxy accepts data insertion requests from SDKs, and allocates those requests into several buckets using hash algorithm.
2. Then the proxy requests data coord to assign segments, the smallest unit in Milvus for data storage.  
3. Afterwards, the proxy inserts information of the requested segments into message store so that these information will not be lost. 

## Data coord and data node

The main function of data coord is to manage channel and segment allocation while the main function of data node is to consume and persist inserted data.

![Data coord and data node in Milvus.](https://assets.zilliz.com/Data_coord_and_data_node_in_Milvus_8bcf010f9e.jpeg "Data coord and data node in Milvus.")

### Function

Data coord serves in the following aspects:

- **Allocate segment space**
Data coord allocates space in growing segments to the proxy so that the proxy can use free space in segments to insert data.

- **Record segment allocation and the expiry time of the allocated space in the segment**
The space within each segment allocated by the data coord is not permanent, therefore, the data coord also needs to keep a record of the expiry time of each segment allocation.

- **Automatically flush segment data**
If the segment is full, the data coord automatically triggers data flush.

- **Allocate channels to data nodes**
A collection can have multiple [vchannels](https://milvus.io/docs/v2.0.x/glossary.md#VChannel). Data coord determines which vchannels are consumed by which data nodes.

Data node serves in the following aspects:

- **Consume data**
Data node consumes data from the channels allocated by data coord and creates a sequence for the data.

- **Data persistence**
Cache inserted data in memory and auto-flush those inserted data to disk when data volume reach a certain threshold.

### Workflow

![One vchannel can only be assigned to one data node.](https://assets.zilliz.com/One_vchannel_can_only_be_assigned_to_one_data_node_14aa3bd718.png "One vchannel can only be assigned to one data node.")

As shown in the image above, the collection has four vchannels (V1, V2, V3, and V4)  and there are two data nodes. It is very likely that data coord will assign one data node to consume data from V1 and V2, and the other data node from V3 and V4. One single vchannel cannot be assigned to multiple data nodes and this is to prevent repetition of data consumption, which will otherwise cause the same batch of data being inserted into the same segment repetitively.
  
## Root coord and Time Tick

Root coord manages TSO (timestamp Oracle), and publishes time tick messages globally. Each data insertion request has a timestamp assigned by root coord. Time Tick is the cornerstone of Milvus which acts like a clock in Milvus and signifies at which point of time is the Milvus system in.

When data are written in Milvus, each data insertion request carries a timestamp. During data consumption, each time data node consumes data whose timestamps are within a certain range. 

![An example of data insertion and data consumption based on timestamp.](https://assets.zilliz.com/An_example_of_data_insertion_and_data_consumption_based_on_timestamp_e820f682f9.jpeg "An example of data insertion and data consumption based on timestamp.")

The image above is the process of data insertion. The value of the timestamps are represented by numbers 1,2,6,5,7,8. The data are written into the system by two proxies: p1 and p2. During data consumption, if the current time of the Time Tick is 5, data nodes can only read data 1 and 2. Then during the second read, if the current time of the Time Tick becomes 9, data 6,7,8 can be read by data node.

## Data organization: collection, partition, shard (channel), segment

![Data organization in Milvus.](https://assets.zilliz.com/Data_organization_in_Milvus_75ad710752.jpeg "Data organization in Milvus.")

Read this [article](https://milvus.io/blog/deep-dive-1-milvus-architecture-overview.md#Data-Model) first to understand the data model in Milvus and the concepts of collection, shard, partition, and segment.

In summary, the largest data unit in Milvus is a collection which can be likened to a table in a relational database. A collection can have multiple shards (each corresponding to a channel) and multiple partitions within each shard. As shown in the illustration above, channels (shards) are the vertical bars while the partitions are the horizontal ones. At each intersection is the concept of segment, the smallest unit for data allocation. In Milvus, indexes are built on segments. During a query, the Milvus system also balances query loads in different query nodes and this process is conducted based on the unit of segments. Segments contain several [binlogs](https://milvus.io/docs/v2.0.x/glossary.md#Binlog), and when the segment data are consumed, a binlog file will be generated.

### Segment

There are three types of segments with different status in Milvus: growing, sealed, and flushed segment.

#### Growing segment

A growing segment is a newly created segment that can be allocated to the proxy for data insertion. The internal space of a segment can be used, allocated, or free.

![Three status in a growing segment](https://assets.zilliz.com/Three_status_in_a_growing_segment_bdae45e26f.png "Three status in a growing segment.")

- Used: this part of space of a growing segment has been consumed by data node.
- Allocated: this part of space of a growing segment has been requested by the proxy and allocated by data coord. Allocated space will expire after a certain period time.
- Free: this part of space of a growing segment has not been used. The value of free space equals the overall space of the segment subtracted by the value of used and allocated space. So the free space of a segment increases as the allocated space expires.

#### Sealed segment

A sealed segment is a closed segment that can no longer be allocated to the proxy for data insertion.

![Sealed segment in Milvus](https://assets.zilliz.com/Sealed_segment_in_Milvus_8def5567e1.jpeg "Sealed segment in Milvus.")

A growing segment is sealed in the following circumstances: 

- If the used space in a growing segment reaches 75% of the total space, the segment will be sealed.
- Flush() is manually called by a Milvus user to persist all data in a collection.
- Growing segments that are not sealed after a long period of time will be sealed as too many growing segments cause data nodes to over-consume memory.

#### Flushed segment 

A flushed segment is a segment that has already been written into disk. Flush refers to storing segment data to object storage for the sake of data persistence. A segment can only be flushed when the allocated space in a sealed segment expires. When flushed, the sealed segment turns into a flushed segment.

![Flushed segment in Milvus](https://assets.zilliz.com/Flushed_segment_in_Milvus_0c1f54d432.png "Flushed segment in Milvus.")

### Channel

A channel is allocated :

- When data node starts or shuts down; or
- When segment space allocated is requested by proxy.

Then there are several strategies of channel allocation. Milvus supports 2 of the strategies:

1. Consistent hashing

![Consistency hashing in Milvus](https://assets.zilliz.com/Consistency_hashing_in_Milvus_fb5e5d84ce.jpeg "Consistency hashing in Milvus.")

The default strategy in Milvus. This strategy leverages the hashing technique to assign each channel a position on the ring, then searches in a clock-wise direction to find the nearest data node to a channel. Thus, in the illustration above, channel 1 is allocated to data node 2, while channel 2 is allocated to data node 3.

However, one problem with this strategy is that the increase or decrease in the number of data nodes (eg. A new data node starts or a data node suddenly shuts down) can affect the process of channel allocation. To solve this issue, data coord monitors the status of data nodes via etcd so that data coord can be immediately notified if there is any change in the status of data nodes. Then data coord further determines to which data node to allocate the channels properly.

2. Load balancing 

The second strategy is to allocate channels of the same collection to different data nodes, ensuring the channels are evenly allocated. The purpose of this strategy is to achieve load balance.

## Data allocation: when and how

![The process of data allocation in Milvus](https://assets.zilliz.com/The_process_of_data_allocation_in_Milvus_0ba86b3ad1.jpeg "The process of data allocation in Milvus.")

The process of data allocation starts from the client. It first sends data insertion requests with a timestamp `t1` to proxy. Then the proxy sends a request to data coord for segment allocation. 

Upon receiving the segment allocation request, the data coord checks segment status and allocates segment. If the current space of the created segments is sufficient for the newly inserted rows of data, the data coord allocates those created segments. However, if the space available in current segments is not sufficient, the data coord will allocate a new segment. The data coord can return one or more segments upon each request. In the meantime, the data coord also saves the allocated segment in meta server for data persistence. 

Subsequently, the data coord returns the information of the allocated segment (including segment ID, number of rows, expiry time `t2`, etc.) to the proxy. The proxy sends such information of the allocated segment to message store so that these information are properly recorded. Note that the value of `t1` must be smaller than that of `t2`. The default value of `t2` is 2,000 millisecond and it can be changed by configuring the parameter `segment.assignmentExpiration` in the `data_coord.yaml` file. 

## Binlog file structure and data persistence

![Data node flush](https://assets.zilliz.com/Data_node_flush_86832f46d0.png "Data node flush.")

Data node subscribes to the message store because data insertion requests are kept in the message store and the data nodes can thus consume insert messages. The data nodes first place insert requests in an insert buffer, and as the requests accumulate, they will be flushed to object storage after reaching a threshold. 

### Binlog file structure

![Binlog file structure.](https://assets.zilliz.com/Binlog_file_structure_ca2897a095.png "Binlog file structure.")

The binlog file structure in Milvus is similar to that in MySQL. Binlog is used to serve two functions: data recovery and index building. 

A binlog contains many [events](https://github.com/milvus-io/milvus/blob/master/docs/developer_guides/chap08_binlog.md#event-format). Each event has an event header and event data. 

Metadata including binlog creation time, write node ID, event length, and NextPosition (offset of the next event), etc. are written in the event header. 

Event data can be divided into two parts: fixed and variable.

![File structure of an insert event.](https://assets.zilliz.com/File_structure_of_an_insert_event_829b1f628d.png "File structure of an insert event.")

The fixed part in the event data of an `INSERT_EVENT` contains `StartTimestamp`, `EndTimestamp`, and `reserved`.

The variable part in fact stores inserted data. The insert data are sequenced into the format of parquet and stored in this file.

 ### Data persistence

If there are multiple columns in schema, Milvus will store binlogs in columns. 

![Binlog data persistence.](https://assets.zilliz.com/Binlog_data_persistence_0c028bf26a.png "Binlog data persistence.")

As illustrated in the image above, the first column is primary key binlog. The second one is timestamp column. The rest are the columns defined in schema. The file path of binlogs in MinIO is also indicated in the image above. 

## About the Deep Dive Series

With the [official announcement of general availability](https://milvus.io/blog/2022-1-25-annoucing-general-availability-of-milvus-2-0.md) of Milvus 2.0, we orchestrated this Milvus Deep Dive blog series to provide an in-depth interpretation of the Milvus architecture and source code. Topics covered in this blog series include:

- [Milvus architecture overview](https://milvus.io/blog/deep-dive-1-milvus-architecture-overview.md)
- [APIs and Python SDKs](https://milvus.io/blog/deep-dive-2-milvus-sdk-and-api.md)
- [Data processing](https://milvus.io/blog/deep-dive-3-data-processing.md)
- [Data management](https://milvus.io/blog/deep-dive-4-data-insertion-and-data-persistence.md)
- [Real-time query](https://milvus.io/blog/deep-dive-5-real-time-query.md)
- [Scalar execution engine](https://milvus.io/blog/deep-dive-7-query-expression.md)
- [QA system](https://milvus.io/blog/deep-dive-6-oss-qa.md)
- [Vector execution engine](https://milvus.io/blog/deep-dive-8-knowhere.md)


Data Insertion and Data Persistence in a Vector Database


![Cover image](https://assets.zilliz.com/How_Is_Data_Processed_in_a_Vector_Database_9fb236bc01.png "How Is Data Processed in a Vector Database?")

> This article is written by [Zhenshan Cao](https://github.com/czs007) and transcreated by [Angela Ni](https://www.linkedin.com/in/yiyun-n-2aa713163/).

In the previous two posts in this blog series, we have already covered the [system architecture](https://milvus.io/blog/deep-dive-1-milvus-architecture-overview.md) of Milvus, the world's most advanced vector database, and its [Python SDK and API](https://milvus.io/blog/deep-dive-2-milvus-sdk-and-api.md). 

This post mainly aims to help you understand how data is processed in Milvus by going deep into the Milvus system and examining the interaction between the data processing components.

*Some useful resources before getting started are listed below. We recommend reading them first to better understand the topic in this post.*

- [Deep dive into the Milvus architecture](https://milvus.io/blog/deep-dive-1-milvus-architecture-overview.md)
- [Milvus data model](https://milvus.io/blog/deep-dive-1-milvus-architecture-overview.md#Data-Model)
- [The role and function of each Milvus component](https://milvus.io/docs/v2.0.x/four_layers.md)
- [Data processing in Milvus](https://milvus.io/docs/v2.0.x/data_processing.md)


## MsgStream interface

[MsgStream interface](https://github.com/milvus-io/milvus/blob/ca129d4308cc7221bb900b3722dea9b256e514f9/docs/developer_guides/chap04_message_stream.md) is crucial to data processing in Milvus. When `Start()` is called, the coroutine in the background writes data into the [log broker](https://milvus.io/blog/deep-dive-1-milvus-architecture-overview.md#Log-as-data) or reads data from there. When `Close()` is called, the coroutine stops.

![MsgStream interface](https://assets.zilliz.com/Msg_Stream_interface_66b70309a7.png "MsgStream interface.")

The MsgStream can serve as a producer and a consumer. The `AsProducer(channels []string)` interface defines MsgStream as a producer while the `AsConsumer(channels []string, subNamestring)`defines it as a consumer. The parameter `channels` is shared in both interfaces and it is used to define which (physical) channels to writes data into or read data from.

> The number of shards in a collection can be specified when a collection is created. Each shard corresponds to a [virtual channel (vchannel)](https://milvus.io/docs/v2.0.x/glossary.md#VChannel). Therefore, a collection can have multiple vchannels. Milvus assigns each vchannel in the log broker a [physical channel (pchannel)](https://milvus.io/docs/v2.0.x/glossary.md#PChannel). 


![Each virtual channel/shard corresponds to a physical channel.](https://assets.zilliz.com/Each_virtual_channel_shard_corresponds_to_a_physical_channel_7cd60e4ed1.png "Each virtual channel/shard corresponds to a physical channel.")

`Produce()` in the MsgStream interface in charge of writing data into the pchannels in log broker. Data can be written in two ways:
- Single write: entities are written into different shards (vchannel) by the hash values of primary keys. Then these entities flow into corresponding pchannels in the log broker.
- Broadcast write: entities are written into all of the pchannels specified by the parameter `channels`.

`Consume()` is a type of blocking API. If there is no data available in the specified pchannel, the coroutine will be blocked when `Consume()` is called in MsgStream interface. On the other hand, `Chan()` is a non-blocking API, which means that the coroutine reads and processes data only if there is existing data in the specified pchannel. Otherwise, the coroutine can process other tasks and will not be blocked when there is no data available.

`Seek()` is method for failure recovery. When a new node is started, the data consumption record can be obtained and data consumption can resume from where it was interrupted by calling `Seek()`.

## Write data

The data written  into different vchannels (shards) can either be insert message or delete message. These vchannels can also be called DmChannels (data manipulation channels).

Different collections may share the same pchannels in the log broker. One collection can have multiple shards and hence multiple corresponding vchannels. The entities in the same collection consequently flow into multiple corresponding pchannels in the log broker. As a result, the benefit of sharing pchannels is an increased volume of throughput enabled by high concurrency of the log broker. 

When a collection is created, not only the number of shards is specified, but also the mapping between vchannels and pchannels in the log broker is decided.

 ![Write path in Milvus](https://assets.zilliz.com/Write_path_in_Milvus_00d93fb377.png "Write path in Milvus.")

As shown in the illustration above, in the write path, proxies write data into the log broker via the `AsProducer()` interface of the MsgStream. Then data nodes consume the data, then convert and store the consumed data in object storage. The storage path is a type of meta information which will be recorded in etcd by data coordinators.

### Flowgraph

Since different collections may share the same pchannels in the log broker, when consuming data, data nodes or query nodes need to judge to which collection the data in a pchannel belongs. To solve this problem, we introduced flowgraph in Milvus. It is mainly in charge of filtering data in a shared pchannel by collection IDs. So, we can say that each flowgraph handles the data stream in a corresponding shard (vchannel) in a collection.

 ![Flowgraph in write path](https://assets.zilliz.com/Flowgraph_in_write_path_1b201e1b71.png "Flowgraph in write path.")

### MsgStream creation

When writing data, MsgStream object is created in the following two scenarios:
- When the proxy receives a data insertion request, it first tries to obtain the mapping between vchannels and pchannels via root coordinator (root coord). Then the proxy creates an MsgStream object.

 ![Scenario 1](https://assets.zilliz.com/Creating_Msg_Stream_object_in_write_path_Scenario_1_bdd0f94d8b.png "Creating MsgStream object in write path_Scenario 1.")

- When data node starts and reads the meta information of channels in etcd, the MsgStream object is created. 

 ![Scenario 2](https://assets.zilliz.com/Creating_Msg_Stream_object_in_write_path_Scenario_2_5b3f99a6d1.png "Creating MsgStream object in write path_Scenario 2.")

## Read data

![Read path in Milvus](https://assets.zilliz.com/Read_path_in_Milvus_c2f0ae5109.png "Read path in Milvus.")

The general workflow of reading data is illustrated in the image above. Query requests are broadcast via DqRequestChannel to query nodes. The query nodes execute the query tasks in parallel. The query results from query nodes go through gRPC and proxy aggregate the results and returns them to the client.

To take a closer look at the data reading process, we can see that the proxy writes query requests into DqRequestChannel. Query nodes then consume message by subscribing to DqRequestChannel. Each message in the DqRequestChannel is broadcast so that all subscribed query nodes can receive the message.

When query nodes receive query requests, they conduct a local query on both batch data stored in sealed segments and streaming data that are dynamically inserted into Milvus and stored in growing segments. Afterwards, query nodes need to aggregate the query results in both [sealed and growing segments](https://milvus.io/docs/v2.0.x/glossary.md#Segment). These aggregated results are passed on to proxy via gRPC.

The proxy collects all the results from multiple query nodes and then aggregate them to obtain the final results. Then the proxy returns the final query results to the client. Since each query request and its corresponding query results are labelled by the same unique requestID, proxy can figure out which query results correspond to which query request.

### Flowgraph

![Flowgraph in read path](https://assets.zilliz.com/Flowgraph_in_read_path_8a5faf2d58.png "Flowgraph in read path.")

Similar to the write path, flowgraphs are also introduced in the read path. Milvus implements the unified Lambda architecture, which integrates the processing of the incremental and historical data. Therefore, query nodes need to obtain real-time streaming data as well. Similarly, flowgraphs in read path filters and differentiates data from different collections.

### MsgStream creation

![Creating MsgStream object in read path](https://assets.zilliz.com/Creating_Msg_Stream_object_in_read_path_7f059bde2f.png "Creating MsgStream object in read path.")

When reading data, the MsgStream object is created in the following scenario:
- In Milvus, data cannot be read unless they are loaded. When the proxy receives a data load request, it sends the request to query coordinator which decides the way of assigning shards to different query nodes. The assigning information (i.e. The names of vchannels and the mapping between vchannels and their corresponding pchannels) is sent to query nodes via method call or RPC (remote procedure call). Subsequently, the query nodes create corresponding MsgStream objects to consume data.    

## DDL operations
DDL stands for data definition language. DDL operations on metadata can be categorized into write requests and read requests. However, these two types of requests are treated equally during metadata processing. 

Read requests on metadata include:
- Query collection schema
- Query indexing information
And more

Write requests include:
- Create a collection
- Drop a collection
- Build an index
- Drop an index
And more

DDL requests are sent to the proxy from the client, and the proxy further passes on these requests in the received order to the root coord which assigns a timestamp for each DDL request and conducts dynamic checks on the requests. Proxy handles each request in a serial manner, meaning one DDL request at a time. The proxy will not process the next request until it completes processing the previous request and receive results from the root coord. 

![DDL operations.](https://assets.zilliz.com/DDL_operations_02679a393c.png "DDL operations.")

As shown in the illustration above, there are `K` DDL requests in the Root coord task queue. The DDL requests in the task queue are arranged in the order they are received by the root coord. So, `ddl1` is the first sent to root coord, and `ddlK` is the last one in this batch. The root coord processes the requests one by one in the time order.

In a distributed system, the communication between the proxies and the root coord is enabled by gRPC. The root coord keeps a record of the maximum timestamp value of the executed tasks to ensure that all DDL requests are processed in time order.

Suppose there are two independent proxies, proxy 1 and proxy 2. They both send DDL requests to the same root coord. However, one problem is that earlier requests are not necessarily sent to the root coord before those requests received by another proxy later. For instance, in the image above, when `DDL_K-1` is sent to the root coord from proxy 1, `DDL_K` from proxy 2 has already been accepted and executed by the root coord. As recorded by the root coord, the maximum timestamp value of the executed tasks at this point is `K`. So in order not to interrupt the time order, the request `DDL_K-1` will be rejected by the root coord's task queue . However, if proxy 2 sends the request `DDL_K+5` to the root coord at this point, the request will be accepted to the task queue and will be executed later according to its timestamp value.

## Indexing

### Building an index

Upon receiving index building requests from the client, the proxy first carries out static checks on the requests and sends them to the root coord. Then the root coord persists these index building requests into meta storage (etcd) and sends the requests to the index coordinator (index coord).

![Building an index.](https://assets.zilliz.com/Building_an_index_e130a4e715.png "Building an index.")

As illustrated above, when the index coord receives index building requests from the root coord, it first persists the task in etcd for meta store. The initial status of the index building task is `Unissued`. The index coord maintains a record of the task load of each index node, and sends to inbound tasks to a less loaded index node. Upon completion of the task, the index node writes the status of the task, either `Finished` or `Failed` into meta storage, which is etcd in Milvus. Then the index coord will understand if the index building task succeeds or fails by looking up in etcd. If the task fails due to limited system resources or dropout of the index node, the index coord will re-trigger the whole process and assign the same task to another index node.

### Dropping an index

In addition, the index coord is also in charge of the requests of dropping indexes. 

![Dropping an index.](https://assets.zilliz.com/Dropping_an_index_afdab6a339.png "Dropping an index.")

When the root coord receives a request of dropping an index from the client, it first marks the index as "dropped", and returns the result to the client while notifying the index coord. Then the index  coord filters all indexing tasks with the `IndexID` and those tasks matching the condition are dropped.

The background coroutine of the index coord will gradually delete all indexing tasks marked as "dropped" from object storage (MinIO and S3). This process involves the recycleIndexFiles interface. When all corresponding index files are deleted, the meta information of the deleted indexing tasks are removed from meta storage (etcd).

## About the Deep Dive Series

With the [official announcement of general availability](https://milvus.io/blog/2022-1-25-annoucing-general-availability-of-milvus-2-0.md) of Milvus 2.0, we orchestrated this Milvus Deep Dive blog series to provide an in-depth interpretation of the Milvus architecture and source code. Topics covered in this blog series include:

- [Milvus architecture overview](https://milvus.io/blog/deep-dive-1-milvus-architecture-overview.md)
- [APIs and Python SDKs](https://milvus.io/blog/deep-dive-2-milvus-sdk-and-api.md)
- [Data processing](https://milvus.io/blog/deep-dive-3-data-processing.md)
- [Data management](https://milvus.io/blog/deep-dive-4-data-insertion-and-data-persistence.md)
- [Real-time query](https://milvus.io/blog/deep-dive-5-real-time-query.md)
- [Scalar execution engine](https://milvus.io/blog/deep-dive-7-query-expression.md)
- [QA system](https://milvus.io/blog/deep-dive-6-oss-qa.md)
- [Vector execution engine](https://milvus.io/blog/deep-dive-8-knowhere.md)





How Is Data Processed in a Vector Database?


![Cover image](https://assets.zilliz.com/20220322_175856_e8e7bea7dc.png "An Introduction to Milvus Python SDK and API")

By [Xuan Yang](https://github.com/XuanYang-cn)

## Background

The following illustration depicts the interaction between SDKs and Milvus through gRPC. Imagine that Milvus is a black box. Protocol Buffers are used to define the interfaces of the server, and the structure of the information they carry. Therefore, all operations in the black box Milvus is defined by Protocol API.

![Interaction](https://assets.zilliz.com/SDK_10c9673111.png "The interaction between SDKs and Milvus through gRPC")

## Milvus Protocol API

Milvus Protocol API consists of `milvus.proto`, `common.proto`, and `schema.proto`, which are Protocol Buffers files suffixed with `.proto`. To ensure proper operation, SDKs must interact with Milvus with these Protocol Buffers files.

### milvus.proto

`milvus.proto` is the vital component of Milvus Protocol API because it defines the `MilvusService`, which further defines all RPC interfaces of Milvus.

The following code sample shows the interface `CreatePartitionRequest`. It has two major string-type parameters `collection_name` and `partition_name`, based on which you can start a partition creation request.

![CreatePartitionRequest](https://assets.zilliz.com/code_d5f034d58d.png "CreatePartitionRequest interface")

Check an example of Protocol in [PyMilvus GitHub Repository](https://github.com/milvus-io/milvus-proto/blob/44f59db22b27cc55e4168c8e53b6e781c010a713/proto/milvus.proto) on line 19.

![Example](https://assets.zilliz.com/create_partition_938691f07f.png "An example of Protocol")

You can find the definition of `CreatePartitionRequest` here.

![Definition](https://assets.zilliz.com/20220321_112254_4ec4f35bd3.png "The definition of CreatePartitionRequest")

Contributors who wish to develop a feature of Milvus or an SDK in a different programming language are welcome to find all interfaces Milvus offers via RPC.

### common.proto

`common.proto` defines the common types of information, including `ErrorCode`, and `Status`. 

![common.proto](https://assets.zilliz.com/20220321_112303_eaafc432a8.png "common.proto")

### schema.proto

`schema.proto` defines the schema in the parameters. The following code sample is an example of `CollectionSchema`.

![schema.proto](https://assets.zilliz.com/20220321_112313_df4ebe36e7.png "schema.proto")

`milvus.proto`, `common.proto`, and `schema.proto` together constitutes the API of Milvus, representing all operations that can be called via RPC.

If you dig into the source code and observe carefully, you will find that when interfaces like `create_index` are called, they actually call multiple RPC interfaces such as `describe_collection` and `describe_index`. Many of the outward interface of Milvus is a combination of multiple RPC interfaces.

Having understood the behaviors of RPC, you can then develop new features for Milvus through combination. You are more than welcome to use your imagination and creativeness and contribute to Milvus community.

## PyMilvus 2.0

### Object-relational mapping (ORM)

To put it in a nutshell, Object-relational mapping (ORM) refers to that when you operate on a local object, such operations will affect the corresponding object on server. PyMilvus ORM-style API features the following characteristics:

1. It operates directly on objects.
2. It isolates service logic and data access details.
3. It hides the complexity of implementation, and you can run the same scripts across different Milvus instances regardless of their deployment approaches or implementation.

### ORM-style API

One of the essence of ORM-style API lies in the control of Milvus connection. For example, you can specify aliases for multiple Milvus servers, and connect to or disconnect from them merely with their aliases. You can even delete the local server address, and control certain objects via specific connection precisely.

![Control Connection](https://assets.zilliz.com/20220321_112320_d5ff08a582.png "Control connections to Milvus")

Another feature of ORM-style API is that, after abstraction, all operations can be performed directly on objects, including collection, partition, and index.

You can abstract a collection object by getting an existing one or creating a new one. You can also assign a Milvus connection to specific objects using connection alias, so that you can operate on these objects locally.

To create a partition object, you can either create it with its parent collection object, or you can do it just like when you create a collection object. These methods can be employed on an index object as well.

In the case that these partition or index objects exist, you can get them through their parent collection object.

## About the Deep Dive Series

With the [official announcement of general availability](https://milvus.io/blog/2022-1-25-annoucing-general-availability-of-milvus-2-0.md) of Milvus 2.0, we orchestrated this Milvus Deep Dive blog series to provide an in-depth interpretation of the Milvus architecture and source code. Topics covered in this blog series include:

- [Milvus architecture overview](https://milvus.io/blog/deep-dive-1-milvus-architecture-overview.md)
- [APIs and Python SDKs](https://milvus.io/blog/deep-dive-2-milvus-sdk-and-api.md)
- [Data processing](https://milvus.io/blog/deep-dive-3-data-processing.md)
- [Data management](https://milvus.io/blog/deep-dive-4-data-insertion-and-data-persistence.md)
- [Real-time query](https://milvus.io/blog/deep-dive-5-real-time-query.md)
- [Scalar execution engine](https://milvus.io/blog/deep-dive-7-query-expression.md)
- [QA system](https://milvus.io/blog/deep-dive-6-oss-qa.md)
- [Vector execution engine](https://milvus.io/blog/deep-dive-8-knowhere.md)


An Introduction to Milvus Python SDK and API


![Cover image](https://assets.zilliz.com/20220705_102717_dd4124dee3.png "Building a Vector Database for Scalable Similarity Search")

> This article is written by Xiaofan Luan and transcreated by Angela Ni and Claire Yu.

According to [statistics](https://mitsloan.mit.edu/ideas-made-to-matter/tapping-power-unstructured-data), about 80%-90% of the world's data is unstructured. Fueled by the rapid growth of the Internet, an explosion of unstructured data is expected in the coming years. Consequently, companies are in urgent need of a powerful database that can help them better handle and understand such kind of data. However, developing a database is always easier said than done. This article aims to share the thinking process and design principles of building Milvus, an open-source, cloud-native vector database for scalable similarity search. This article also explains the Milvus architecture in detail.

Jump to:
- [Unstructured data requires a complete basic software stack](#Unstructured-data-requires-a-complete-basic-software-stack)
  - [Vectors and scalars](#Vectors-and-scalars)
  - [From vector search engine to vector database](#From-vector-search-engine-to-vector-database)
  - [A cloud-native first approach](#A-cloud-native-first-approach)
- [The design principles of Milvus 2.0](#The-design-principles-of-Milvus-20)
  - [Log as data](#Log-as-data)
  - [Duality of table and log](#Duality-of-table-and-log)
  - [Log persistency](#Log-persistency)
- [Building a vector database for scalable similarity search](#Building-a-vector-database-for-scalable-similarity-search)
  - [Standalone and cluster](#Standalone-and-cluster)
  - [A bare-bones skeleton of the Milvus architecture](#A-bare-bones-skeleton-of-the-Milvus-architecture)
  - [Data model](#Data-Model)

## Unstructured data requires a complete basic software stack 

As the Internet grew and evolved, unstructured data became more and more common, including emails, papers, IoT sensor data, Facebook photos, protein structures, and much more. In order for computers to understand and process unstructured data, these are converted into vectors using [embedding techniques](https://zilliz.com/learn/embedding-generation). 

Milvus stores and indexes these vectors, and analyzes the correlation between two vectors by calculating their similarity distance. If the two embedding vectors are very similar, it means that the original data sources are similar as well.

![The workflow of processing unstructured data.](https://assets.zilliz.com/Frame_1_6_5e0ab80f2c.png "The workflow of processing unstructured data.")

### Vectors and scalars

A scalar is a quantity that is described only in one measurement - magnitude. A scalar can be represented as a number. For instance, a car is traveling at the speed of 80 km/h. Here, the speed (80km/h) is a scalar. Meanwhile, a vector is a quantity that is described in at least two measurements - magnitude and direction. If a car is traveling towards west at the speed of 80 km/h, here the velocity (80 km/h west) is a vector. The image below is an example of common scalars and vectors.

![Scalars vs. Vectors](https://assets.zilliz.com/Frame_1_7_90a142ab5b.png "Scalars and vectors. Image source: NASA Glenn Research Center.")

Since most of the important data have more than one attribute, we can understand these data better if we convert them into vectors. One common way for us to manipulate vector data is to calculate the  distance between vectors using [metrics](https://milvus.io/docs/v2.0.x/metric.md) such as Euclidean distance, inner product, Tanimoto distance, Hamming distance, etc. The closer the distance, the more similar the vectors are. To query a massive vector dataset efficiently, we can organize vector data by building indexes on them. After the dataset is indexed, queries can be routed to clusters, or subsets of data, that are most likely to contain vectors similar to an input query. 

To learn more about the indexes, refer to [Vector Index](https://milvus.io/docs/v2.0.x/index.md).

### From vector search engine to vector database

From the very beginning, Milvus 2.0 is designed to serve not only as a search engine, but more importantly, as a powerful vector database.

One way to help you understand the difference here is by drawing an analogy between [InnoDB](https://dev.mysql.com/doc/refman/5.7/en/innodb-introduction.html) and [MySQL](https://www.mysql.com/), or [Lucene](https://lucene.apache.org/) and [Elasticsearch](https://www.elastic.co/).

Just like MySQL and Elasticsearch, Milvus is also built on top of open-source libraries such as [Faiss](https://github.com/facebookresearch/faiss), [HNSW](https://github.com/nmslib/hnswlib), [Annoy](https://github.com/spotify/annoy), which focus on providing search functionalities and ensuring search performance. However, it would be unfair to degrade Milvus to merely a layer atop Faiss as it stores, retrieves, analyzes vectors, and, just as with any other database, also provides a standard interface for CRUD operations. In addition, Milvus also boasts features including:
- Sharding and partitioning
- Replication
- Disaster recovery
- Load balance
- Query parser or optimizer

![Vector database](https://assets.zilliz.com/database_d912320ea7.png "More than a search engine, Milvus is a powerful vector database.")

For a more comprehensive understanding of what a vector database is, read the blog [here](https://zilliz.com/learn/what-is-vector-database).

### A cloud-native first approach

![Could-native approach](https://assets.zilliz.com/image_2_be82d762db.png "Cloud-native approach.")

#### From shared nothing, to shared storage, then to shared something

Traditional databases used to adopt a "shared nothing" architecture in which nodes in the distributed systems are independent but connected by a network. No memory or storage are shared among the nodes. However, [Snowflake](https://docs.snowflake.com/en/user-guide/intro-key-concepts.html) revolutionized the industry by introducing a "shared storage" architecture in which compute (query processing) is separated from storage (database storage). With a shared storage architecture, databases can achieve greater availability, scalability, and a reduction of data duplication. Inspired by Snowflake, many companies started to leverage cloud-based infrastructure for data persistence while using local storage for caching. This type of database architecture is called "shared something" and has become the mainstream architecture in most applications today. 

Apart from the "shared something" architecture, Milvus supports flexible scaling of each component by using Kubernetes to manage its execution engine and separating read, write and other services with microservices.

#### Database as a service (DBaaS)

Database as a service is a hot trend as many users not only care about regular database functionalities but also yearn for more varied services. This means that apart from the traditional CRUD operations, our database has to enrich the type of services it can provide, such as database management, data transport, charging, visualization, etc.

#### Synergy with the broader open-source ecosystem

Another trend in database development is leveraging the synergy between the database and other cloud-native infrastructure. In the case of Milvus, it relies on some open-source systems. For instance, Milvus uses [etcd](https://etcd.io/) for storing metadata. It also adopts message queue, a type of asynchronous service-to-service communication used in microservices architecture, which can help export incremental data.

In the future, we hope to build Milvus on top of AI infrastructures such as [Spark](https://spark.apache.org/) or [Tensorflow](https://www.tensorflow.org/), and integrate Milvus with streaming engines so that we can better support unified stream and batch processing to meet the various needs of Milvus users.

## The design principles of Milvus 2.0

As our next-generation cloud-native vector database, Milvus 2.0 is built around the following three principles.

### Log as data

A log in a database serially records all the changes made to data. As shown in the figure below, from left to right are "old data" and "new data". And the logs are in time order. Milvus has a global timer mechanism assigning one globally unique and auto-incremental timestamp.

![Logs](https://assets.zilliz.com/Frame_1_8_6e40211f44.png "Logs in a database.")

In Milvus 2.0, the log broker serves as the system's backbone: all data insert and update operations must go through the log broker, and worker nodes execute CRUD operations by subscribing to and consuming logs.

### Duality of table and log

Both the table and the log are data, and they are but just two different forms. Tables are bounded data while logs are unbounded. Logs can be converted into tables. In the case of Milvus, it aggregates logs using a processing window from TimeTick. Based on log sequence, multiple logs are aggregated into one small file called log snapshot. Then these log snapshots are combined to form a segment, which can be used individually for load balance.

### Log persistency

Log persistency is one of the tricky issues faced by many databases. The storage of logs in a distributed system usually depends on replication algorithms.

Unlike databases such as [Aurora](https://aws.amazon.com/rds/aurora/), [HBase](https://hbase.apache.org/), [Cockroach DB](https://www.cockroachlabs.com/), and [TiDB](https://en.pingcap.com/), Milvus takes a ground-breaking approach and introduces a publish-subscribe (pub/sub) system for log storage and persistency. A pub/sub system is analogous to the message queue in [Kafka](https://kafka.apache.org/) or [Pulsar](https://pulsar.apache.org/). All nodes within the system can consume the logs. In Milvus, this kind of system is called a log broker. Thanks to the log broker, logs are decoupled from the server, ensuring that Milvus is itself stateless and better positioned to quickly recover from system failure. 

![Log broker](https://assets.zilliz.com/log_broker_cafe889835.png "Log broker in Milvus.")

## Building a vector database for scalable similarity search

Built on top of popular vector search libraries including Faiss, ANNOY, HNSW, and more, Milvus was designed for similarity search on dense vector datasets containing millions, billions, or even trillions of vectors. 

### Standalone and cluster

Milvus offers two ways of deployment - standalone or cluster. In Milvus standalone, since all nodes are deployed together, we can see Milvus as one single process. Currently, Milvus standalone relies on MinIO and etcd for data persistence and metadata storage. In future releases, we hope to eliminate these two third-party dependencies to ensure the simplicity of the Milvus system. Milvus cluster includes eight microservice components and three third-party dependencies: MinIO, etcd, and Pulsar. Pulsar serves as the log broker and provides log pub/sub services.

![Standalone and cluster](https://assets.zilliz.com/standalone_and_cluster_7558f56e8c.png "Two ways of deploying Milvus: Milvus standalone and Milvus cluster.")

### A bare-bones skeleton of the Milvus architecture

Milvus separates data flow from control flow, and is divided into four layers that are independent in terms of scalability and disaster recovery.

![Milvus architecture](https://assets.zilliz.com/Milvus_architecture_b7743a4a7f.png "Milvus architecture.")

#### Access layer

The access layer acts as the system's face, exposing the endpoint of the client connection to the outside world. It is responsible for processing client connections, carrying out static verification, basic dynamic checks for user requests, forwarding requests, and gathering and returning results to the client. The proxy itself is stateless and provides unified access addresses and services to the outside world through load balancing components (Nginx, Kubernetess Ingress, NodePort, and LVS). Milvus uses a massively parallel processing (MPP) architecture, where proxies return results gathered from worker nodes after global aggregation and post-processing.

#### Coordinator service 

The coordinator service is the system's brain, responsible for cluster topology node management, load balancing, timestamp generation, data declaration, and data management. For a detailed explanation of the function of each coordinator service, read the [Milvus technical documentation](https://milvus.io/docs/v2.0.x/four_layers.md#Coordinator-service).

#### Worker nodes

The worker, or execution, node acts as the limbs of the system, executing instructions issued by the coordinator service and the data manipulation language (DML) commands initiated by the proxy. A worker node in Milvus is similar to a data node in [Hadoop](https://hadoop.apache.org/), or a region server in HBase. Each type of worker node corresponds to a coord service. For a detailed explanation of the function of each worker node, read the [Milvus technical documentation](https://milvus.io/docs/v2.0.x/four_layers.md#Worker-nodes).

#### Storage

Storage is the cornerstone of Milvus, responsible for data persistence. The storage layer is divided into three parts:

- **Meta store:** Responsible for storing snapshots of meta data such as collection schema, node status, message consumption checkpoints, etc. Milvus relies on etcd for these functions and Etcd also assumes the responsibility of service registration and health checks. 
- **Log broker:** A pub/sub system that supports playback and is responsible for streaming data persistence, reliable asynchronous query execution, event notifications, and returning query results. When nodes are performing downtime recovery, the log broker ensures the integrity of incremental data through log broker playback. Milvus cluster uses Pulsar as its log broker, while the standalone mode uses RocksDB. Streaming storage services such as Kafka and Pravega can also be used as log brokers.
- **Object storage:** Stores snapshot files of logs, scalar/vector index files, and intermediate query processing results. Milvus supports [AWS S3](https://aws.amazon.com/s3/) and [Azure Blob](https://azure.microsoft.com/en-us/services/storage/blobs/), as well as [MinIO](https://min.io/), a lightweight, open-source object storage service. Due to the high access latency and billing per query of object storage services, Milvus will soon support memory/SSD-based cache pools and hot/cold data separation to improve performance and reduce costs.

### Data Model

The data model organizes the data in a database. In Milvus, all data are organized by collection, shard, partition, segment, and entity.

![Data model 1](https://assets.zilliz.com/Data_model_1_5d6bb43673.png "Collection, shard, partition, and segment in Milvus.")

#### Collection

A collection in Milvus can be likened to a table in a relational storage system. Collection is the biggest data unit in Milvus.

#### Shard

To take full advantage of the parallel computing power of clusters when writing data, collections in Milvus must spread data writing operations to different nodes. By default, a single collection contains two shards. Depending on your dataset volume, you can have more shards in a collection. Milvus uses a master key hashing method for sharding.

#### Partition

There are also multiple partitions in a shard. A partition in Milvus refers to a set of data marked with the same label in a collection. Common partitioning methods including partitioning by date, gender, user age, and more. Creating partitions can benefit the query process as tremendous data can be filtered by partition tag.

In comparison, sharding is more of scaling capabilities when writing data, while partitioning is more of enhancing system performance when reading data.

![Data model 2](https://assets.zilliz.com/Data_model_2_044a443751.png "Segment and entity in Milvus.")

#### Segments

Within each partition, there are multiple small segments. A segment is the smallest unit for system scheduling in Milvus. There are two types of segments, growing and sealed. Growing segments are subscribed by query nodes. The Milvus user keeps writing data into growing segments. When the size of a growing segment reaches an upper limit (512 MB by default), the system will not allow writing extra data into this growing segment, hence sealing this segment. Indexes are built on sealed segments.

To access data in real time, the system reads data in both growing segments and sealed segments. 

#### Entity

Each segment contains massive amount of entities. An entity in Milvus is equivalent to a row in a traditional database. Each entity has a unique primary key field, which can also be automatically generated. Entities must also contain timestamp (ts), and vector field - the core of Milvus.

## About the Deep Dive Series

With the [official announcement of general availability](https://milvus.io/blog/2022-1-25-annoucing-general-availability-of-milvus-2-0.md) of Milvus 2.0, we orchestrated this Milvus Deep Dive blog series to provide an in-depth interpretation of the Milvus architecture and source code. Topics covered in this blog series include:

- [Milvus architecture overview](https://milvus.io/blog/deep-dive-1-milvus-architecture-overview.md)
- [APIs and Python SDKs](https://milvus.io/blog/deep-dive-2-milvus-sdk-and-api.md)
- [Data processing](https://milvus.io/blog/deep-dive-3-data-processing.md)
- [Data management](https://milvus.io/blog/deep-dive-4-data-insertion-and-data-persistence.md)
- [Real-time query](https://milvus.io/blog/deep-dive-5-real-time-query.md)
- [Scalar execution engine](https://milvus.io/blog/deep-dive-7-query-expression.md)
- [QA system](https://milvus.io/blog/deep-dive-6-oss-qa.md)
- [Vector execution engine](https://milvus.io/blog/deep-dive-8-knowhere.md)


Building a Vector Database for Scalable Similarity Search


![Binlog Cover Image](https://assets.zilliz.com/Attu_3ff9a76156.png "Attu - a GUI tool for Milvus 2.0")

Draft by [Zhen Chen](https://github.com/czhen-zilliz) and transcreation by [Lichen Wang](https://github.com/LocoRichard).

<p style="font-size: 12px;color: #4c5a67">Click <a href="https://zilliz.com/blog/manage-your-milvus-vector-database-with-one-click-simplicity">here</a> to check the original post.</p> 

In the face of rapidly growing demand for unstructured data processing, Milvus 2.0 stands out. It is an AI-oriented vector database system designed for massive production scenarios. Apart from all these Milvus SDKs and Milvus CLI, a command-line interface for Milvus, is there a tool that allows users to operate Milvus more intuitively? The anwer is YES. Zilliz has announced a graphical user interface - Attu - specifically for Milvus. In this article, we would like to show you step by step how to perform a vector similarity search with Attu.

![Attu island](https://assets.zilliz.com/map_aa1cda30d4.png "Attu (/ ˈætu /) is an island on the west edge of Aleutian Islands. This uninhabited realm symbolizes an adventurous spirit.")

In comparison with Milvus CLI which brings the uttermost simplicity of usage, Attu features more:
- Installers for Windows OS, macOS, and Linux OS;
- Intuitive GUI for easier usage of Milvus;
- Coverage of major functionalities of Milvus;
- Plugins for expansion of customized functionalities;
- Complete system topology information for easier understanding and administration of Milvus instance.

## Installation

You can find the newest release of Attu at [GitHub](https://github.com/zilliztech/attu/releases). Attu offers executable installers for different operating systems. It is an open-source project and welcomes contribution from everyone.

![Installation](https://assets.zilliz.com/installation_bbe62873af.png "Attu installers and source code.")

You can also install Attu via Docker.

```shell
docker run -p 8000:3000 -e HOST_URL=http://{ attu IP }:8000 -e MILVUS_URL={milvus server IP}:19530 zilliz/attu:latest
```

`attu IP` is the IP address of the environment where Attu runs, and `milvus server IP` is IP address of the environment where Milvus runs.

Having installed Attu successfully, you can input the Milvus IP and Port in the interface to start Attu.

![Connect Milvus with Attu](https://assets.zilliz.com/connect_1fde46d9d5.png "Connect Milvus with Attu")

## Feature overview

![Overview page](https://assets.zilliz.com/overview_591e230514.png "Attu Overview page")

Attu interface consists of **Overview** page, **Collection** page, **Vector Search** page, and **System View** page, corresponding to the four icons on the left-side navigation pane respectively.

The **Overview** page shows the loaded collections. While the **Collection** page lists all the collections and indicates if they are loaded or released.

![Collection page](https://assets.zilliz.com/collection_42656fe308.png "Attu Collection page")

The **Vector Search** and **System View** pages are plugins of Attu. The concepts and usage of the plugins will be introduced in the final part of the blog.

You can perform vector similarity search in **Vector Search** page.

![Vector Search page](https://assets.zilliz.com/vector_search_be7365687c.png "Attu Vector Search page")

In **System View** page, you can check the topological structure of Milvus.

![System View page](https://assets.zilliz.com/system_view_e1df15023d.png "Attu System View page")

You can also check the detailed information of each node by clicking the node.

![Node view](https://assets.zilliz.com/node_view_5bbc25f9b2.png "Attu Node view page")

## Demonstration

Let's explore Attu with a test dataset.

Check our [GitHub repo](https://github.com/zilliztech/attu/tree/main/examples) for the dataset used in the following test.

First, create a collection named test with the following four fields:
- Field Name: id, primary key field
- Field Name: vector, vector field, float vector, Dimension: 128
- Field Name: brand, scalar field, Int64
- Field Name: color, scalar field, Int64

![Create a collection](https://assets.zilliz.com/create_collection_95dfa15354.png "Create a collection with Attu")

Load the collection for search after it was successfully created.

![Load the collection](https://assets.zilliz.com/load_collection_fec39171df.png "Load the collection with Attu")

You can now check the newly created collection in the **Overview** page.

![Check the collection](https://assets.zilliz.com/check_collection_163b05477e.png "Check the collection with Attu")

Import the test dataset into Milvus.

![Import data](https://assets.zilliz.com/import_data_1_f73d71be85.png "Import data with Attu")

![Import data](https://assets.zilliz.com/import_data_2_4b3c3c3c25.png "Import data with Attu")

![Import data](https://assets.zilliz.com/import_data_3_0def4e8550.png "Import data with Attu")

Click the collection name in Overview or Collection page to enter query interface to check the imported data.

Add filter, specify the expression `id != 0`, click **Apply Filter**, and click **Query**.

![Query data](https://assets.zilliz.com/query_data_24d9f71ccc.png "Query data with Attu")

You will find all fifty entries of entities are imported successfully.

![Query result](https://assets.zilliz.com/query_result_bcbbd17084.png "Query result")

Let's try vector similarity search.

Copy one vector from the `search_vectors.csv` and paste it in **Vector Value** field. Choose the collection and field. Click **Search**.

![Search data](https://assets.zilliz.com/search_data_5af3a1db53.png "Search data with Attu")

You can then check the search result. Without compiling any scripts, you can search with Milvus easily.

![Search result](https://assets.zilliz.com/search_result_961886efab.png "Search result")

Finally, let's check the **System View** page.

With Metrics API encapsulated in Milvus Node.js SDK, you can check the system status, node relations, and node status.

As an exclusive feature of Attu, System Overview page includes a complete system topological graph. By clicking on each node, you can check its status (refresh every 10 seconds).

![Milvus node topological graph](https://assets.zilliz.com/topological_graph_d0c5c17586.png "Milvus node topological graph in Attu")

Click on each node to enter the **Node List View**. You can check all child nodes of a coord node. By sorting, you can identify the nodes with high CPU or memory usage quickly, and locate the problem with the system.

![Milvus node list](https://assets.zilliz.com/node_list_64fc610a8d.png "Milvus node list")

## What's more

As mentioned earlier, the **Vector Search** and **System View** pages are plugins of Attu. We encourage users to develop their own plugins in Attu to suit their application scenarios. In the source code, there is folder built specifically for plugin codes.

![Plugins](https://assets.zilliz.com/plugins_a2d98e4e5b.png "Vector Search and System View pages are plugins of Attu")

You can refer to any of the plugin to learn how to build a plugin. By setting the following config file, you can add the plugin to Attu.

![Add plugins to Attu](https://assets.zilliz.com/add_plugins_e3ef53cc0d.png "Add plugins to Attu")

You can see [Attu GitHub Repo](https://github.com/zilliztech/attu/tree/main/doc) and [Milvus Technical Document](https://milvus.io/docs/v2.0.x/attu.md) for detailed instruction.

Attu is an open-source project. All contributions are welcome. You can also [file an issue](https://github.com/zilliztech/attu/issues) if you had any problem with Attu.

We sincerely hope that Attu can bring you a better user experience with Milvus. And if you like Attu, or have some feedbacks about the usage, you can complete this [Attu User Survey](https://wenjuan.feishu.cn/m/cfm?t=suw4QnODU1ui-ok7r) to help us optimize Attu for a better user experience.


Manage Your Milvus Vector Database with One-click Simplicity


![Binlog Cover Image](https://assets.zilliz.com/Load_balance_b2f35a5577.png "How Milvus Balances Query Load across Nodes?")

By [Xi Ge](https://github.com/xige-16).

In previous blog articles, we have successively introduced the Deletion, Bitset, and Compaction functions in Milvus 2.0. To culminate this series, we would like to share the design behind Load Balance, a vital function in the distributed cluster of Milvus.


## Implementation

Whereas the number and size of segments buffered in query nodes differ, the search performance across the query nodes may also vary. The worst case could happen when a few query nodes are exhausted searching on a large amount of data, but newly created query nodes remain idle because no segment is distributed to them, causing a massive waste of CPU resources and a huge drop in search performance.

To avoid such circumstances, the query coordinator (query coord) is programmed to distribute segments evenly to each query node according to the RAM usage of the nodes. Therefore, CPU resources are consumed equally across the nodes, thereby significantly improving search performance.

### Trigger automatic load balance

According to the default value of the configuration `queryCoord.balanceIntervalSeconds`, the query coord checks the RAM usage (in percentage) of all query nodes every 60 seconds. If either of the following conditions is satisfied, the query coord starts to balance the query load across the query node:

1. RAM usage of any query node in the cluster is larger than `queryCoord.overloadedMemoryThresholdPercentage` (default: 90);
2. Or the absolute value of any two query nodes' RAM usage difference is larger than `queryCoord.memoryUsageMaxDifferencePercentage` (default: 30).

After the segments are transferred from the source query node to the destination query node, they should also satisfy both the following conditions:

1. RAM usage of the destination query node is no larger than `queryCoord.overloadedMemoryThresholdPercentage` (default: 90);
2. The absolute value of the source and destination query nodes' RAM usage difference after load balancing is less than that before load balancing.

With the above conditions satisfied, the query coord proceeds to balance the query load across the nodes.

## Load balance

When load balance is triggered, the query coord first loads the target segment(s) to the destination query node. Both query nodes return search results from the target segment(s) at any search request at this point to guarantee the completeness of the result.

After the destination query node successfully loads the target segment, the query coord publishes a `sealedSegmentChangeInfo` to the Query Channel. As shown below, `onlineNodeID` and `onlineSegmentIDs` indicate the query node that loads the segment and the segment loaded respectively, and `offlineNodeID` and `offlineSegmentIDs` indicate the query node that needs to release the segment and the segment to release respectively.

![sealedSegmentChangeInfo](https://assets.zilliz.com/20220228_145413_f253cec15b.png "sealedSegmentChangeInfo")

Having received the `sealedSegmentChangeInfo`, the source query node then releases the target segment. 

![Load Balance Workflow](https://assets.zilliz.com/20220228_145436_2604bc57a5.png "Load Balance Workflow")

The whole process succeeds when the source query node releases the target segment. By completing that, the query load is set balanced across the query nodes, meaning the RAM usage of all query nodes is no larger than `queryCoord.overloadedMemoryThresholdPercentage`, and the absolute value of the source and destination query nodes' RAM usage difference after load balancing is less than that before load balancing.

## What's next?

In the 2.0 new feature series blog, we aim to explain the design of the new features. Read more in this blog series!
- [How Milvus Deletes Streaming Data in a Distributed Cluster](https://milvus.io/blog/2022-02-07-how-milvus-deletes-streaming-data-in-distributed-cluster.md)
- [How to Compact Data in Milvus?](https://milvus.io/blog/2022-2-21-compact.md)
- [How Milvus Balances Query Load across Nodes?](https://milvus.io/blog/2022-02-28-how-milvus-balances-query-load-across-nodes.md)
- [How Bitset Enables the Versatility of Vector Similarity Search](https://milvus.io/blog/2022-2-14-bitset.md) 

This is the finale of the Milvus 2.0 new feature blog series. Following this series, we are planning a new series of Milvus [Deep Dive](https://milvus.io/blog/deep-dive-1-milvus-architecture-overview.md), which introduces the basic architecture of Milvus 2.0. Please stay tuned.


How Milvus Balances Query Load across Nodes?


![Binlog Cover Image](https://assets.zilliz.com/Compact_173a08ec1c.png "How to compact data in Milvus?")

By [Bingyi Sun](https://github.com/sunby) and [Angela Ni](https://www.linkedin.com/in/yiyun-n-2aa713163/).

With the [official release](https://milvus.io/blog/2022-1-25-annoucing-general-availability-of-milvus-2-0.md) of Milvus 2.0 GA, a list of new features are supported. Among those, compaction is one of the new features that can help you save storage space.

Compaction refers to the process of merging small segments into large ones and clean logically deleted data. In other words, compaction reduces usage of disk space by purging the deleted or expired entities in binlogs. It is a background task that is triggered by [data coord](https://milvus.io/docs/v2.0.x/four_layers.md#Data-coordinator-data-coord) and executed by [data node](https://milvus.io/docs/v2.0.x/four_layers.md#Data-node) in Milvus.

This article dissects the concept and implementation of compaction in Milvus.

## What is compaction?

Before going deep into the details of how to implement compaction in Milvus 2.0, it is critical to figure out what compaction is in Milvus.

More often than not, as a Milvus user you might have been bothered by the increasing usage of hard disk space. Another issue is that a segment with less than 1,024 rows is not indexed and only supports brute-force search to process queries. Small segments caused by auto-flush or user-invoked flush might hamper query efficiency.

Therefore, to solve the two issues mentioned above and help reduce disk usage and improve query efficiency, Milvus supports compaction.

Databases like [LevelDB](https://github.com/google/leveldb) and [RocksDB](http://rocksdb.org/) append data to sorted strings tables (SSTables). The average disk reads per query increase with the number of SSTables, leading to inefficient queries. To reduce read amplification and release hard drive space, these databases compact SSTables into one. Compaction processes run in the background automatically. 

Similarly, Milvus appends inserted and deleted data to [binlogs](https://github.com/milvus-io/milvus/blob/master/docs/developer_guides/chap08_binlog.md). As the number of binlogs increases, more hard disk space is used. To release hard disk space, Milvus compacts binlogs of deleted and inserted data. If an entity is inserted but later deleted, it no longer exists in the binlogs that records data insertion or deletion once compacted. In addition, Milvus also compacts [segments](https://milvus.io/docs/v2.0.x/glossary.md#Segment) - data files automatically created by Milvus for holding inserted data.

## How to configure compaction?

Configuration of compaction in Milvus mainly involves two parameters: `dataCoord.enableCompaction` and `common.retentionDuration`. 

`dataCoord.enableCompaction` specifies whether to enable compaction. Its default value is `true`. 

`common.retentionDuration` specifies a period when compaction does not run. Its unit is second. When you compact data, all deleted entities will be made unavailable for search with Time Travel. Therefore, if you plan to search with Time Travel, you have to specify a period of time during which compaction does not run and does not affect deleted data. To ensure accurate results of searches with Time Travel, Milvus retains data operated in a period specified by `common.retentionDuration`. That is, data operated in this period will not be compacted. For more details, see [Search with Time Travel](https://milvus.io/docs/v2.0.x/timetravel.md).

Compaction is enabled in Milvus by default. If you disabled compaction but later want to manually enable it, you can follow the steps below:

1. Call the `collection.compact()` method to trigger a global compaction process manually. However, please be noted that this operation might take a long time.
2. After calling the method, a compaction ID is returned. View the compaction status by calling the `collection.get_compaction_state()` method.

After compaction is enabled, it runs in the background automatically. Since the compaction process might take a long time, compaction requests are processed asynchronously to save time.

## How to implement compaction?

In Milvus, you can either implement compaction manually or automatically.

Manual compaction of binlogs or segments does not require meeting any trigger conditions. Therefore, if you manually invoke compaction, the binlogs or segments will be compacted no matter what.

However, if you want to enable automatic compaction, certain compaction trigger conditions need to be met in order for the system to compact your segments or binlogs.

Generally there are two types of objects that can be compacted in Milvus: binglogs and segments.


## Binlog compaction

A binlog is a binary log, or a smaller unit in segment, that records and handles the updates and changes made to data in the Milvus vector database. Data from a segment is persisted in multiple binlogs. Binlog compaction involves two types of binlogs in Milvus: insert binlogs and delta binlogs.

Delta binlogs are generated when data is deleted while insert binlogs are generated under the following three circumstances.

- As inserted data is being appended, the segment reaches the upper limit of size and is automatically flushed to the disk.
- DataCoord automatically flushes segments that stay unsealed for a long time.
- Some APIs like `collection.num_entities`, `collection.load()`, and more automatically invoke flush to write segments to disk.

Therefore, binlog compaction, as its name suggests, refers to compacting binlogs within a segment. More specifically, during binlog compaction, all delta binlogs and insert binlogs that are not retained are compacted.

![Binlog compaction](https://assets.zilliz.com/binlog_compaction_d7f8f910c8.png "Binlog compaction.")

When a segment is flushed to disk, or when Milvus requests global compaction as compaction has not run for a long time, at least one of the following two conditions need to be met to trigger automatic compaction:

1. Rows in delta binlogs are more than 20% of the total rows.
2. The size of delta binlogs exceeds 10 MB.


## Segment compaction

A [segment](https://milvus.io/docs/v2.0.x/glossary.md#Segment) is a data file automatically created by Milvus for holding inserted data. There are two types of segments in Milvus: growing segment and sealed segment. 

A growing segment keeps receiving the newly inserted data until it is sealed. A sealed segment no longer receives any new data, and will be flushed to the object storage, leaving new data to be inserted into a newly created growing segment. 

Therefore, segment compaction refers to compacting multiple sealed segments. More specifically, during segment compaction, small segments are compacted into bigger ones.

![Segment compaction](https://assets.zilliz.com/segment_compaction_92eecc0e55.jpeg "Segment compaction.")

Each segment generated after compaction cannot exceed the upper limit of a segment size, which is 512 MB by default. Read [system configurations](https://milvus.io/docs/v2.0.x/system_configuration.md) to learn how to modify the upper limit of a segment size.

When a segment flushes to disk, or when Milvus requests global compaction as compaction has not run for a long time, the following condition needs to be met to trigger automatic compaction:

- Segments smaller than 0.5 * `MaxSegmentSize` is more than 10.

## What's next?

What's next after learning the basics of compaction in Milvus? Currently, not all parameters for configuring compaction are in the `milvus.yaml` file, and plan generation strategies are relatively basic. Come and contribute to Milvus, [the open-source project](https://github.com/milvus-io) if you are interested!

Also, in the 2.0 new feature series blog, we aim to explain the design of the new features. Read more in this blog series!
- [How Milvus Deletes Streaming Data in a Distributed Cluster](https://milvus.io/blog/2022-02-07-how-milvus-deletes-streaming-data-in-distributed-cluster.md)
- [How to Compact Data in Milvus?](https://milvus.io/blog/2022-2-21-compact.md)
- [How Milvus Balances Query Load across Nodes?](https://milvus.io/blog/2022-02-28-how-milvus-balances-query-load-across-nodes.md)
- [How Bitset Enables the Versatility of Vector Similarity Search](https://milvus.io/blog/2022-2-14-bitset.md) 

## About the author

Bingyi Sun, Senior Software Engineer of the Milvus project, achieved his Master's degree in software engineering at Shanghai Jiao Tong University. He is mainly responsible for developing storage related components in Milvus 2.0. His area of interest is database and distributed systems. He is great fan of open source projects and a gourmet who enjoys playing video games and reading in his spare time.


How to Compact Data in Milvus?


![Bitset Cover Image](https://assets.zilliz.com/Bitset_cd54487e7b.png "Cover image.")

By [Yudong Cai](https://github.com/cydrain) and [Angela Ni](https://www.linkedin.com/in/yiyun-n-2aa713163/).

Various [new essential features](https://milvus.io/blog/2022-1-27-milvus-2-0-a-glimpse-at-new-features.md) of a vector database are provided together with the [release](https://milvus.io/blog/2022-1-25-annoucing-general-availability-of-milvus-2-0.md) of Milvus 2.0. Among the new features, [Time Travel](https://milvus.io/docs/v2.0.x/timetravel_ref.md), attribute filtering, and [delete operations](https://milvus.io/blog/2022-02-07-how-milvus-deletes-streaming-data-in-distributed-cluster.md) are correlated as these three features are achieved by one common mechanism - bitset.

Therefore, this article aims to clarify the concept of bitset in Milvus and explain how it works to support delete operations, Time Travel, and attribute filtering with three examples.

## What is bitset?

A bitset is an array of bit numbers ("0" and "1") that can be used to represent certain data information. With bitsets, you can store certain types of data compactly and efficiently as opposed to store them in Ints, floats, or chars. Bitsets work on [boolean logic](https://milvus.io/docs/v2.0.x/boolean.md), according to which the value of an output is either valid or invalid, usually denoted by "1" and "0" respectively. "1" stands for valid, and "0" for invalid. Since bitsets are highly efficient and can save storage, they can also be used to achieve many features such as attribute filtering, delete operations, Time Travel, and more.

Starting from version 0.7.0, the concept of bitset has been introduced in Milvus to enable the [delete function](https://milvus.io/blog/deleting-data-in-milvus.md). More specifically, bitset is used to mark if each row in the segment is deleted. Deleted entities are marked with "1" in the corresponding bitset, and as a result, the deleted entities will not be computed during a search or query.

In the Milvus 2.0 version, the application of bitset is extended to enable more features, like attribute filtering and Time Travel. The general principle in a bitset remains the same. That is, if an entity is marked with "1" in the corresponding bitset, the entity will be ignored during a search or query. Bitsets are used to enable 3 features in Milvus:

- Attribute filtering
- Data deletion
- Query with Time Travel

## How does bitset work in Milvus?

The examples below are used to illustrate how bitset works in Milvus.

### Prerequisites

Suppose there is a segment with eight entities and a series of data manipulation language (DML) events happens in the order shown in the figure below.

- Four of the entities, whose `primary_keys` are \[1, 2, 3, 4] respectively, are inserted when the timestamp `ts` equals 100. 
- The remaining four entities, whose `primary_keys` are \[5, 6, 7, 8], are inserted when the timestamp `ts` equals 200. 
- Entities whose `primary_keys` are \[7, 8] are deleted when the timestamp `ts` equals 300. 
- Only entities, whose `primary_keys` are \[1, 3, 5, 7], satisfy the conditions of attribute filtering. 

![DML events](https://assets.zilliz.com/UML_1_0a3605808c.jpg "Order of DML events.")

### Case one

Suppose the value a user sets for `time_travel` is 150. In other words, the user conducts a query on the data stored in Milvus when `ts` = 150. The bitset generation process is illustrated by Figure 1. 

During the initial filtering stage, the result of the `filter_bitset` should be \[1, 0, 1, 0, 1, 0, 1, 0] as entities \[1, 3, 5, 7] are valid filtering results and marked as "1" in the bitset. However, entities \[4, 5, 6, 7] were not even inserted to the vector database when `ts` equals 150. Therefore, these four entities should be marked as "0" regardless of the filtering condition. Now the bitset result should be \[1, 0, 1, 0, 0, 0, 0, 0]. Since in Milvus, the general principle of bitset computing is that entities marked with "1" in the bitset are ignored during a search or query, the bitset result after Time Travel and attribute filtering needs to be flipped in order to be combined with the deletion bitmap. The flipped result of `filter_bitset` should be \[0, 1, 0, 1, 1, 1, 1, 1].

As for the deletion bitset `del_bitset`, the initial value should be \[0, 0, 0, 0, 0, 0, 1, 1]. However, entities 7 and 8 are not deleted until `ts` is 300. Therefore, when `ts` is 150, entities 7 and 8 are still valid. As a result, the `del_bitset` value after Time Travel should be \[0, 0, 0, 0, 0, 0, 0, 0]. 

Now we have two bitsets after Time Travel and attribute filtering: `filter_bitset` \[0, 1, 0, 1, 1, 1, 1, 1] and `del_bitset` \[0, 0, 0, 0, 0, 0, 0, 0].  Combine these two bitsets with the "OR" binary logic operator. The ultimate value of `result_bitset` is \[0, 1, 0, 1, 1, 1, 1, 1]. That is to say, only entities 1 and 3 will be computed in the following search or query stage.


![Figure 1](https://assets.zilliz.com/bitset_figure1_1b5852f7a7.jpeg "Figure 1. Search with Time Travel = 150.")

### Case two

Suppose the value the user sets for `time_travel` is 250. In other words, the user conducts a query on the data stored in Milvus when `ts` = 250. The bitset generation process is illustrated by Figure 2. 

Like in case one, the resultant `filter_bitset` of the initial attribute filtering stage should be \[1, 0, 1, 0, 1, 0, 1, 0]. 

All entities \[1, 2, 3, 4, 5, 6, 7, 8] are inserted to the vector database when `ts`= 250. Therefore, the previous result of `filter_bitset` remains the same. Again, we need to flip the result of the `filter_bitset`, and we will get \[0, 1, 0, 1, 0, 1, 0, 1].

As for the deletion bitset `del_bitset`, the initial value should be \[0, 0, 0, 0, 0, 0, 1, 1]. However, entities 7 and 8 were not deleted until `ts` is 300. Therefore, when `ts` is 250, entities 7 and 8 are still valid. As a result, the `del_bitset` value after Time Travel should be \[0, 0, 0, 0, 0, 0, 0, 0]. 

Now we have two bitsets after Time Travel and attribute filtering: `filter_bitset` \[0, 1, 0, 1, 0, 1, 0, 1] and `del_bitset` \[0, 0, 0, 0, 0, 0, 0, 0].  Combine these two bitsets with the "OR" binary logic operator. The ultimate value of `result_bitset` is \[0, 1, 0, 1, 0, 1, 0, 1]. That is to say, only entities \[1, 3, 5, 7] will be computed in the following search or query stage.

![Figure 2](https://assets.zilliz.com/bitset_figure2_7cbaa7c719.jpeg "Figure 2. Search with Time Travel = 250.")

### Case three

Suppose the value the user sets for `time_travel` is 350. In other words, the user conducts a query on the data stored in Milvus when `ts` = 350. The bitset generation process is illustrated by Figure 3. 

Same as case one and two, the resultant `filter_bitset` of the initial attribute filtering stage is \[0, 1, 0, 1, 0, 1, 0, 1]. 

All entities \[1, 2, 3, 4, 5, 6, 7, 8] are inserted to the vector database when `ts`= 350. Therefore, the final flipped result of the `filter_bitset` is \[0, 1, 0, 1, 0, 1, 0, 1], the same as in case two.

As for the deletion bitset `del_bitset`, since entities 7 and 8 are already deleted when `ts`=350, therefore, the result of `del_bitset` should be \[0, 0, 0, 0, 0, 0, 1, 1].

Now we have two bitsets after Time Travel and attribute filtering: `filter_bitset` \[0, 1, 0, 1, 0, 1, 0, 1] and `del_bitset` \[0, 0, 0, 0, 0, 0, 1, 1].  Combine these two bitsets with the "OR" binary logic operator. The ultimate value of `result_bitset` is \[0, 1, 0, 1, 0, 1, 1, 1]. That is to say, only entities \[1, 3, 5] will be computed in the following search or query stage.

![Figure 3](https://assets.zilliz.com/bitset_figure3_dd46a6aecf.jpeg "Figure 3. Search with Time Travel = 350.")

## What's next?

In the 2.0 new feature series blog, we aim to explain the design of the new features. Read more in this blog series!

- [How Milvus Deletes Streaming Data in a Distributed Cluster](https://milvus.io/blog/2022-02-07-how-milvus-deletes-streaming-data-in-distributed-cluster.md)
- [How to Compact Data in Milvus?](https://milvus.io/blog/2022-2-21-compact.md)
- [How Milvus Balances Query Load across Nodes?](https://milvus.io/blog/2022-02-28-how-milvus-balances-query-load-across-nodes.md)
- [How Bitset Enables the Versatility of Vector Similarity Search](https://milvus.io/blog/2022-2-14-bitset.md) 


How Bitset Enables the Versatility of Vector Similarity Search


# How Milvus Deletes Streaming Data in a Distributed Cluster

Featuring unified batch-and-stream processing and cloud-native architecture, Milvus 2.0 poses a greater challenge than its predecessor did during the development of the DELETE function. Thanks to its advanced storage-computation disaggregation design and the flexible publication/subscription mechanism, we are proud to announce that we made it happen. In Milvus 2.0, you can delete an entity in a given collection with its primary key so that the deleted entity will no longer be listed in the result of a search or a query.

Please note that the DELETE operation in Milvus refers to logical deletion, whereas physical data cleanup occurs during the Data Compaction. Logical deletion not only greatly boosts the search performance constrained by the I/O speed, but also facilitates data recovery. Logically deleted data can still be retrieved with the help of the Time Travel function.

## Usage

Let's try out the DELETE function in Milvus 2.0 first. (The following example uses PyMilvus 2.0.0 on Milvus 2.0.0).

```python
from pymilvus import connections, utility, Collection, DataType, FieldSchema, CollectionSchema
# Connect to Milvus
connections.connect(
    alias="default", 
    host='x.x.x.x', 
    port='19530'
)
# Create a collection with Strong Consistency level
pk_field = FieldSchema(
    name="id", 
    dtype=DataType.INT64, 
    is_primary=True, 
)
vector_field = FieldSchema(
    name="vector", 
    dtype=DataType.FLOAT_VECTOR, 
    dim=2
)
schema = CollectionSchema(
    fields=[pk_field, vector_field], 
    description="Test delete"
)
collection_name = "test_delete"
collection = Collection(
    name=collection_name, 
    schema=schema, 
    using='default', 
    shards_num=2,
    consistency_level="Strong"
)
# Insert randomly generated vectors
import random
data = [
    [i for i in range(100)],
    [[random.random() for _ in range(2)] for _ in range(100)],
]
collection.insert(data)
# Query to make sure the entities to delete exist
collection.load()
expr = "id in [2,4,6,8,10]"
pre_del_res = collection.query(
    expr,
    output_fields = ["id", "vector"]
)
print(pre_del_res)
# Delete the entities with the previous expression
collection.delete(expr)
# Query again to check if the deleted entities exist
post_del_res = collection.query(
    expr,
    output_fields = ["id", "vector"]
)
print(post_del_res)
```

## Implementation

In a Milvus instance, a data node is mainly responsible for packing streaming data (logs in log broker) as historical data (log snapshots) and automatically flushing them to object storage. A query node executes search requests on full data, i.e. both streaming data and historical data.

To make the most of the data writing capacity of parallel nodes in a cluster, Milvus adopts a sharding strategy based on primary key hashing to distribute writing operations evenly to different worker nodes. That is to say, proxy will route the Data Manipulation Language (DML) messages (i.e. requests) of an entity to the same data node and query node. These messages are published through the DML-Channel and consumed by the data node and query node separately to provide search and query services together.

### Data node

Having received data INSERT messages, the data node inserts the data in a growing segment, which is a new segment created to receive streaming data in memory. If either the data row count or the duration of the growing segment reaches the threshold, the data node seals it to prevent any incoming data. The data node then flushes the sealed segment, which contains the historical data, to the object storage. Meanwhile, the data node generates a bloom filter based on the primary keys of the new data, and flushed it to the object storage together with the sealed segment, saving the bloom filter as a part of the statistics binary log (binlog), which contains the statistical information of the segment.

> A bloom filter is a probabilistic data structure that consists of a long binary vector and a series of random mapping functions. It can be used to test whether an element is a member of a set, but might return false positive matches.           —— Wikipedia

When data DELETE messages come in, data node buffers all bloom filters in the corresponding shard, and matches them with the primary keys provided in the messages to retrieve all segments (from both growing and sealed ones) that possibly include the entities to delete. Having pinpointed the corresponding segments, data node buffers them in memory to generate the Delta binlogs to record the delete operations, and then flushes those binlogs together with the segments back to the object storage.

![Data Node](https://assets.zilliz.com/data_node_2397ad70c3.png "DELETE workflow in data node")

Since one shard is only assigned with one DML-Channel, extra query nodes added to the cluster will not be able to subscribe to the DML-Channel. To ensure that all query nodes can receive the DELETE messages, data nodes filter the DELETE messages from the DML-Channel, and forward them to Delta-Channel to notify all query nodes of the delete operations.

### Query node

When loading a collection from object storage, the query node first obtains each shard's checkpoint, which marks the DML operations since the last flush operation. Based on the checkpoint, the query node loads all sealed segments together with their Delta binlog and bloom filters. With all data loaded, the query node then subscribes to DML-Channel, Delta-Channel, and Query-Channel.

If more data INSERT messages come after the collection is loaded to memory, query node first pinpoints the growing segments according to the messages, and updates corresponding bloom filters in memory for query purposes only. Those query-dedicated bloom filters will not be flushed to object storage after the query is finished.

![Query Node](https://assets.zilliz.com/query_node_a78b1d664f.png "DELETE workflow in query node")

As mentioned above, only a certain number of query nodes can receive DELETE messages from the DML-Channel, meaning only they can execute the DELETE requests in growing segments. For those query nodes which have subscribed to the DML-Channel, they first filter the DELETE messages in the growing segments, locate the entities by matching the provided primary keys with those query-dedicated bloom filters of the growing segments, and then record the delete operations in the corresponding segments. 

Query nodes that cannot subscribe to the DML-Channel are only allowed to process search or query requests on sealed segments because they can only subscribe to the Delta-Channel, and receive the DELETE messages forwarded by data nodes. Having collected all DELETE messages in the sealed segments from Delta-Channel, the query nodes locate the entities by matching the provided primary keys with the bloom filters of the sealed segments, and then record the delete operations in the corresponding segments.

Eventually, in a search or query, the query nodes generate a bitset based on the delete records to omit the deleted entities, and search among the remaining entities from all segments, regardless of the segment status. Last but not least, the consistency level affects the visibility of the deleted data. Under Strong Consistency Level (as shown in the previous code sample), the deleted entities are immediately invisible after deletion. While Bounded Consistency Level is adopted, there will be several seconds of latency before the deleted entities become invisible.

## What's next?

In the 2.0 new feature series blog, we aim to explain the design of the new features. Read more in this blog series!
- [How Milvus Deletes Streaming Data in a Distributed Cluster](https://milvus.io/blog/2022-02-07-how-milvus-deletes-streaming-data-in-distributed-cluster.md)
- [How to Compact Data in Milvus?](https://milvus.io/blog/2022-2-21-compact.md)
- [How Milvus Balances Query Load across Nodes?](https://milvus.io/blog/2022-02-28-how-milvus-balances-query-load-across-nodes.md)
- [How Bitset Enables the Versatility of Vector Similarity Search](https://milvus.io/blog/2022-2-14-bitset.md) 


How Milvus Deletes Streaming Data in a Distributed Cluster


# Milvus 2.0: A Glimpse at New Features


It has been half a year since the first release candidate of Milvus 2.0. Now we are proud to announce the general availability of the Milvus 2.0. Please follow me step by step to catch a glimpse at some of the new features that Milvus supports.


## Entity deletion

Milvus 2.0 supports entity deletion, allowing users to delete vectors based on the primary keys (IDs) of the vectors. They won't be worried about the expired or invalid data anymore. Let's try it.

1. Connect to Milvus, create a new collection, and insert 300 rows of randomly-generated 128-dimensional vectors.

```python
from pymilvus import connections, utility, Collection, DataType, FieldSchema, CollectionSchema
# connect to milvus
host = 'x.x.x.x'
connections.add_connection(default={"host": host, "port": 19530})
connections.connect(alias='default')
# create a collection with customized primary field: id_field
dim = 128
id_field = FieldSchema(name="cus_id", dtype=DataType.INT64, is_primary=True)
age_field = FieldSchema(name="age", dtype=DataType.INT64, description="age")
embedding_field = FieldSchema(name="embedding", dtype=DataType.FLOAT_VECTOR, dim=dim)
schema = CollectionSchema(fields=[id_field, age_field, embedding_field],
                          auto_id=False, description="hello MilMil")
collection_name = "hello_milmil"
collection = Collection(name=collection_name, schema=schema)
import random
# insert data with customized ids
nb = 300
ids = [i for i in range(nb)]
ages = [random.randint(20, 40) for i in range(nb)]
embeddings = [[random.random() for _ in range(dim)] for _ in range(nb)]
entities = [ids, ages, embeddings]
ins_res = collection.insert(entities)
print(f"insert entities primary keys: {ins_res.primary_keys}")
```
```
insert entities primary keys: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254, 255, 256, 257, 258, 259, 260, 261, 262, 263, 264, 265, 266, 267, 268, 269, 270, 271, 272, 273, 274, 275, 276, 277, 278, 279, 280, 281, 282, 283, 284, 285, 286, 287, 288, 289, 290, 291, 292, 293, 294, 295, 296, 297, 298, 299]
```

2. Before proceeding to deletion, check the entities you want to delete exist by search or query, and do it twice to make sure the result is reliable.

```python
# search
nq = 10
search_vec = [[random.random() for _ in range(dim)] for _ in range(nq)]
search_params = {"metric_type": "L2", "params": {"nprobe": 16}}
limit = 3
# search 2 times to verify the vector persists
for i in range(2):
    results = collection.search(search_vec, embedding_field.name, search_params, limit)
    ids = results[0].ids
    print(f"search result ids: {ids}")
    expr = f"cus_id in {ids}"
    # query to verify the ids exist
    query_res = collection.query(expr)
    print(f"query results: {query_res}")
```
```
search result ids: [76, 2, 246]
query results: [{'cus_id': 246}, {'cus_id': 2}, {'cus_id': 76}]
search result ids: [76, 2, 246]
query results: [{'cus_id': 246}, {'cus_id': 2}, {'cus_id': 76}]
```

3. Delete the entity with the cus_id of 76, and then search and query for this entity.

```python
print(f"trying to delete one vector: id={ids[0]}")
collection.delete(expr=f"cus_id in {[ids[0]]}")
results = collection.search(search_vec, embedding_field.name, search_params, limit)
ids = results[0].ids
print(f"after deleted: search result ids: {ids}")
expr = f"cus_id in {ids}"
# query to verify the id exists
query_res = collection.query(expr)
print(f"after deleted: query res: {query_res}")
print("completed")
```
```
trying to delete one vector: id=76
after deleted: search result ids: [76, 2, 246]
after deleted: query res: [{'cus_id': 246}, {'cus_id': 2}, {'cus_id': 76}]
completed
```


Why is the deleted entity still retrievable? If you have checked the source code of Milvus, you will find that the deletion within Milvus is asynchronous and logical, which means that entities won't be physically deleted. Instead, they will be attached with a "deleted" mark so that no search or query requests will retrieve them. In addition, Milvus searches under Bounded Staleness consistency level by default. Therefore, the deleted entities are still retrievable before the data is synchronized in data node and query node. Try search or query the deleted entity after a few seconds, you will then find it is no longer in the result. 


```python
expr = f"cus_id in {[76, 2, 246]}"
# query to verify the id exists
query_res = collection.query(expr)
print(f"after deleted: query res: {query_res}")
print("completed")
```
```
after deleted: query res: [{'cus_id': 246}, {'cus_id': 2}]
completed
```

## Consistency level

The above experiment shows us how the consistency level influences the immediate visibility of the newly deleted data.
Users can adjust the consistency level for Milvus flexibly to adapt it to various service scenarios. Milvus 2.0 supports four levels of consistency:
- `CONSISTENCY_STRONG`: `GuaranteeTs` is set as identical to the newest system timestamp, and query nodes wait until the service time proceeds to the newest system timestamp, and then process the search or query request.
- `CONSISTENCY_EVENTUALLY`: `GuaranteeTs` is set insignificantly smaller than the newest system timestamp to skip the consistency check. Query nodes search immediately on the existing data view.
- `CONSISTENCY_BOUNDED`: `GuaranteeTs` is set relatively smaller than the newest system timestamp, and query nodes search on a tolerable, less updated data view.
- `CONSISTENCY_SESSION`: The client uses the timestamp of the last write operation as the `GuaranteeTs`, so that each client can at least retrieve the data inserted by itself. 


In the previous RC release, Milvus adopts Strong as the default consistency. However, taking account of the fact that most users are less demanding about consistency than performance, Milvus changes the default consistency as Bounded Staleness, which can balance their requirements to a greater extent. In the future, we will further optimize the configuration of the GuaranteeTs, which can be achieved only during collection creation in current release. For more information about `GuaranteeTs`, see [Guarantee Timestamp in Search Requests](https://github.com/milvus-io/milvus/blob/master/docs/developer_guides/how-guarantee-ts-works.md). 

Will lower consistency lead to better performance? You can never find the answer until you try it.

4. Modify the code above to record the search latency.


```python
for i in range(5):
    start = time.time()
    results = collection.search(search_vec, embedding_field.name, search_params, limit)
    end = time.time()
    print(f"search latency: {round(end-start, 4)}")
    ids = results[0].ids
    print(f"search result ids: {ids}")
```


5. Search with the identical data scale and parameters except that `consistency_level` is set as `CONSISTENCY_STRONG`.


```python
collection_name = "hello_milmil_consist_strong"
collection = Collection(name=collection_name, schema=schema,
                        consistency_level=CONSISTENCY_STRONG)
```
```
search latency: 0.3293
search latency: 0.1949
search latency: 0.1998
search latency: 0.2016
search latency: 0.198
completed
```


6. Search in a collection with `consistency_level` set as `CONSISTENCY_BOUNDED`.


```python
collection_name = "hello_milmil_consist_bounded"
collection = Collection(name=collection_name, schema=schema,
                        consistency_level=CONSISTENCY_BOUNDED)
```
```
search latency: 0.0144
search latency: 0.0104
search latency: 0.0107
search latency: 0.0104
search latency: 0.0102
completed
```

7. Clearly, average search latency in `CONSISTENCY_BOUNDED` collection is 200ms shorter than that in `CONSISTENCY_STRONG` collection.


Are the deleted entities immediately invisible if the consistency level is set as Strong? The answer is Yes. You can still try this on your own.

## Handoff
Working with streaming dataset, many users are used to building an index and loading the collection before inserting data into it. In previous releases of Milvus, users have to load collection manually after the index building to replace the raw data with the index, which is slow and laborious. The handoff feature allows Milvus 2.0 to automatically load indexed segment to replace the streaming data that reaches certain thresholds of indexing, greatly improving the search performance.


8. Build index and load the collection before inserting more entities.

```python
# index
index_params = {"index_type": "IVF_SQ8", "metric_type": "L2", "params": {"nlist": 64}}
collection.create_index(field_name=embedding_field.name, index_params=index_params)
# load
collection.load()
```


9. Insert 50,000 rows of entities 200 times (same batches of vectors are used for the sake of convenience, but this will not affect the result).

```python
import random
# insert data with customized ids
nb = 50000
ids = [i for i in range(nb)]
ages = [random.randint(20, 40) for i in range(nb)]
embeddings = [[random.random() for _ in range(dim)] for _ in range(nb)]
entities = [ids, ages, embeddings]
for i in range(200):
    ins_res = collection.insert(entities)
    print(f"insert entities primary keys: {ins_res.primary_keys}")
```


10. Check the loading segment information in query node during and after the inserting. 

```python
# did this in another python console
utility.get_query_segment_info("hello_milmil_handoff")
```

11. You will find that all sealed segments loaded to query node are indexed.

```
[segmentID: 430640405514551298
collectionID: 430640403705757697
partitionID: 430640403705757698
mem_size: 394463520
num_rows: 747090
index_name: "_default_idx"
indexID: 430640403745079297
nodeID: 7
state: Sealed
, segmentID: 430640405514551297
collectionID: 430640403705757697
partitionID: 430640403705757698
mem_size: 397536480
num_rows: 752910
index_name: "_default_idx"
indexID: 430640403745079297
nodeID: 7
state: Sealed
...
```

## What's more

In addition to the above functionalities, new features such as Data Compaction, Dynamic Load Balance, and more are introduced into Milvus 2.0. Please enjoy your exploratory journey with Milvus!

In the near future, we will share with you a series of blogs introducing the design of the new features in Milvus 2.0.
- [Deletion](https://milvus.io/blog/2022-02-07-how-milvus-deletes-streaming-data-in-distributed-cluster.md)
- [Data Compaction](https://milvus.io/blog/2022-2-21-compact.md)
- [Dynamic Load Balance](https://milvus.io/blog/2022-02-28-how-milvus-balances-query-load-across-nodes.md)
- [Bitset](https://milvus.io/blog/2022-2-14-bitset.md) 

Find us on:
- [GitHub](https://github.com/milvus-io/milvus)
- [Milvus.io](https://milvus.io/)
- [Slack Channel](https://slack.milvus.io/)


Milvus 2.0 - A Glimpse at New Features


Dear Members and Friends of the Milvus Community:

Today, six months after the first Release Candidate (RC) was made public, we are thrilled to announce that Milvus 2.0 is [General Available (GA)](https://milvus.io/docs/v2.0.x/release_notes.md#v200) and production ready! It's been a long journey, and we thank everyone – community contributors, users, and the LF AI & Data Foundation – along the way who helped us make this happen.

The ability to handle billions of high dimensional data is a big deal for AI systems these days, and for good reasons:
  1. Unstructured data occupy dominant volumes compared to traditional structured data.
  2. Data freshness has never been more important. Data scientists are eager for timely data solutions rather than the traditional T+1 compromise.
  3. Cost and performance have become even more critical, and yet there still exists a big gap between current solutions and real world use cases.
Hence, Milvus 2.0. Milvus is a database that helps handle high dimensional data at scale. It is designed for cloud with the ability to run everywhere. If you've been following our RC releases, you know we've spent great effort on making Milvus more stable and easier to deploy and maintain. 

## Milvus 2.0 GA now offers

**Entity deletion**

As a database, Milvus now supports [deleting entities by primary key](https://milvus.io/docs/v2.0.x/delete_data.md) and will support deleting entities by expression later on.

**Automatic load balance**

Milvus now supports plugin load balance policy to balance the load of each query node and data node. Thanks to the disaggregation of computation and storage, the balance will be done in just a couple of minutes.

**Handoff** 

Once growing segments are sealed through flush, handoff tasks replace growing segments with indexed historical segments to improve search performance.

**Data compaction**

Data compaction is a background task to merge small segments into large ones and clean logical deleted data.  

**Support embedded etcd and local data storage**

Under Milvus standalone mode, we can remove etcd/MinIO dependency with just a few configurations. Local data storage can also be used as a local cache to avoid loading all data into main memory.

**Multi language SDKs**

In addition to [PyMilvus](https://github.com/milvus-io/pymilvus), [Node.js](https://github.com/milvus-io/milvus-sdk-node), [Java](https://github.com/milvus-io/milvus-sdk-java) and [Go](https://github.com/milvus-io/milvus-sdk-go) SDKs are now ready-to-use.

**Milvus K8s Operator**

[Milvus Operator](https://milvus.io/docs/v2.0.x/install_cluster-milvusoperator.md) provides an easy solution to deploy and manage a full Milvus service stack, including both Milvus components and its relevant dependencies (e.g. etcd, Pulsar and MinIO), to the target [Kubernetes](https://kubernetes.io/) clusters in a scalable and highly available manner.

**Tools that help to manage Milvus**

We have [Zilliz](https://zilliz.com/) to thank for the fantastic contribution of management tools. We now have [Attu](https://milvus.io/docs/v2.0.x/attu.md), which allows us to interact with Milvus via an intuitive GUI, and [Milvus_CLI](https://milvus.io/docs/v2.0.x/cli_overview.md), a command-line tool for managing Milvus.

Thanks to all 212 contributors, the community finished 6718 commits during the last 6 months, and tons of stability and performance issues have been closed. We'll open our stability and performance benchmark report soon after the 2.0 GA release. 

## What's next?

**Functionality**

String type support will be the next killer features for Milvus 2.1. We will also bring in time to live (TTL) mechanism and basic ACL management to better satisfy user needs.

**Availability**

We are working on refactoring the query coord scheduling mechanism to support multi memory replicas for each segment. With multiple active replicas, Milvus can support faster failover and speculative execution to shorten the downtime to within a couple of seconds.

**Performance**

Performance benchmark results will soon be offered on our websites. The following releases are anticipated to see an impressive performance improvement. Our target is to halve the search latency under smaller datasets and double the system throughput.

**Ease of use**

Milvus is designed to run anywhere. We will support Milvus on MacOS (Both M1 and X86) and on ARM servers in the next few small releases. We will also offer embedded PyMilvus so you can simply `pip install` Milvus without complex environment setup.

**Community governance**

We will refine the membership rules and clarify the requirements and responsibilities of contributor roles. A mentorship program is also under development; for anyone who is interested in cloud-native database, vector search, and/or community governance, feel free to contact us.

We’re really excited about the latest Milvus GA release! As always, we are happy to hear your feedback. If you encounter any problems, don't hesitate to contact us on [GitHub](https://github.com/milvus-io/milvus) or via [Slack](http://milvusio.slack.com/).

<br/>

Best regards,

Xiaofan Luan 

Milvus Project Maintainer

<br/>


> *Edited by [Claire Yu](https://github.com/claireyuw).*


Announcing General Availability of Milvus 2.0


This article is translated by [Angela Ni](https://www.linkedin.com/in/yiyun-n-2aa713163/).


Information is everywhere in our lives. Meta (formerly known as Facebook), Instagram, Twitter, and other social media platforms make information streams all the more ubiquitous. Therefore, engines dealing with such information streams have become a must-have in most system architecture. However, as a user of social media platforms and relevant apps, I bet you must have been bothered by duplicate articles, news, memes, and more. Exposure to duplicate content hampers the process of information retrieval and leads to bad user experience.


For a product dealing with information streams, it is a high priority for the developers to find a flexible data processor that can be integrated seamlessly into the system architecture to deduplicate identical news or advertisements.

[SmartNews](https://www.smartnews.com/en/), valued at [2 billion US dollars](https://techcrunch.com/2021/09/15/news-aggregator-smartnews-raises-230-million-valuing-its-business-at-2-billion/), is the most highly-valued news app company in the US. Noticeably, it used to be a user of Milvus, an open-source vector database, but later transformed into an active contributor to the Milvus project.

This article shares the story of SmartNews and tells why it decided to make contributions to the Milvus project.

## An overview of SmartNews

SmartNews, founded in 2012, is headquartered in Tokyo, Japan. The news app developed by SmartNews has always been [top-rated](https://www.businessinsider.com/guides/smartnews-free-news-app-2018-9) in the Japanese market. SmartNews is the [fastest growing](https://about.smartnews.com/en/2019/06/12/smartnews-builds-global-momentum-with-over-500-us-growth-new-executives-and-three-new-offices/) news app and also boasts [high user viscosity](https://about.smartnews.com/en/2018/07/21/smartnews-reaches-more-than-10-million-monthly-active-users-in-the-united-states-and-japan/) in the US market. According to the statistics from [APP Annie](https://www.appannie.com/en/), the monthly average session duration of SmartNews ranked first among all news apps by the end of July, 2021, greater than the accumulated session duration of AppleNews and Google News.

With the rapid growth of user base and viscosity, SmartNews has to face more challenges in terms of recommendation mechanism and AI algorithm. Such challenges include utilizing massive discrete features in large-scale machine learning (ML), accelerating unstructured data query with vector similarity search, and more.

At the beginning of 2021, the dynamic Ad algorithm team at SmartNews sent a request to AI infrastructure team that the functions of recalling and querying advertisements need to be optimized. After two months of research, AI infrastructure engineer Shu decided to use Milvus, an open-source vector database that supports multiple indexes and similarity metrics and online data updates. Milvus is trusted by more than a thousand organizations worldwide.

## Advertisement recommendation powered by vector similarity search

The open-source vector database Milvus is adopted in the SmartNews Ad system to match and recommend dynamic ads from a 10-milllion-scale dataset to its users. By doing so, SmartNews can create a mapping relationship between two previously unmatchable datasets - user data and advertisement data. In the second quarter of 2021, Shu managed to deploy Milvus 1.0 on Kubernetes. Learn more about how to [deploy Milvus](https://milvus.io/docs).

![img](https://assets.zilliz.com/image1_2a88ed162f.png "Milvus documentation.")

After the successful deployment of Milvus 1.0, the first project to use Milvus was the advertisement recall project initiated by the Ad team at SmartNews. During the initial stage, the advertisement dataset was on a million scale. Meanwhile, the P99 latency was strictly controlled within less than 10 milliseconds.

In June, 2021, Shu and his colleagues in the algorithm team applied Milvus to more business scenarios and attempted data aggregation and online data/index update in real time.

By now, Milvus, the open-source vector database has been used in various business scenarios at SmartNews, including ad recommendation.

## **From a user to an active contributor**

While integrating Milvus into the Smartnews product architecture, Shu and other developers came up with requests of functions such as hot reload, item TTL (time-to-live), item update/replace, and more. These are also functions desired by many users in the Milvus community. Therefore, Dennis Zhao, head of the AI infrastructure team at SmartNews decided to develop and contribute the hot reload function to the community. Dennis believed that "SmartNews team has been benefiting from the Milvus community, therefore, we are more than willing to contribute if we have something to share with the community."

Data reload supports code editing while running the code. With the help of data reload, developers no longer need to stop at a breakpoint or restart the application. Instead, they can edit the code directly and see the result in real time.

In late July, Yusup, engineer at SmartNews proposed an idea of using [collection alias](https://milvus.io/docs/v2.0.x/collection_alias.md#Collection-Alias) to achieve hot reload.

Creating collection alias refers to specifying alias names for a collection. A collection can have multiple aliases. However, an alias corresponds to a maximum of one collection. Simply draw an analogy between a collection and a locker. A locker, like a collection, has its own number and position, which will always remain unchanged. However, you can always put in and draw out different things from the locker. Similarly, the name of the collection is fixed but the data in the collection is dynamic. You can always insert or delete vectors in a collection, as data deletion is supported in the Milvus [pre-GA version](https://milvus.io/docs/v2.0.x/release_notes.md#v200-PreGA).

In the case of SmartNews advertisement business, nearly 100 million vectors are inserted or updated as new dynamic ad vectors are generated. There are several solutions to this:

- Solution 1: delete old data first and insert new ones.
- Solution 2: create a new collection for new data.
- Solution 3: use collection alias.

For solution 1, one of the most straightforward shortcoming is that it is extremely time-consuming, especially when the dataset to be updated is tremendous. It generally takes hours to update a dataset on a 100-million-scale.

As for solution 2, the problem is that the new collection is not immediately available for search. That is to say, a collection is not searchable during load. Plus, Milvus does not allow two collections to use the same collection name. Switching to a new collection would always require users to manually modify the client side code. That is to say, users have to revise the value of the parameter `collection_name` every time they need to switch between collections.

Solution 3 would be the silver bullet. You only need to insert the new data in a new collection and use collection alias. By doing so, you only need to swap the collection alias every time you need to switch the collection to conduct the search. You do not need extra efforts to revise the code. This solution saves you the troubles mentioned in the previous two solutions.

Yusup started from this request and helped the whole SmartNews team understand the Milvus architecture. After one and half months, the Milvus project received a PR about hot reload from Yusup. And later, this function is officially available along with the release of Milvus 2.0.0-RC7.

Currently, the AI infrastructure team is taking the lead to deploy Milvus 2.0 and migrate all data gradually from Milvus 1.0 to 2.0.

![img_collection alias](https://assets.zilliz.com/image2_96c064a627.png "Collection alis doc.")


Support for collection alias can greatly improve user experience, especially for those large Internet companies with great volumes of user requests. Chenglong Li, data engineer from the Milvus community, who helped build the bridge between Milvus and Smartnews, said, "The collection alias function arises from the real business request of SmartNews, a Milvus user. And SmartNews contributed the code to the Milvus community. This act of reciprocity is a great example of the open-source spirit: from the community and for the community. We hope to see more contributors like SmartNews and jointly build a more prosperous Milvus community."

"Currently, part of the ad business is adopting Milvus as the offline vector database. The official release of Mivus 2.0 is approaching, and we hope that we can use Milvus to build more reliable systems and provide real-time services for more business scenarios." said Dennis.

> Update: Milvus 2.0 is now general available! [Learn more](2022-1-25-annoucing-general-availability-of-milvus-2-0.md)


The Story of SmartNews - from a Milvus User to an Active Contributor


2021 was an amazing year for Milvus as an open-source project. I want to take a moment to thank all the contributors and users of Milvus, as well as partners for contributing to such an outstanding year.

**One of the most impressive moments of this year for me is the release of Milvus 2.0. Before we started this project, only a few community members believed that we could deliver the most advanced vector database in the world , but now I am proud to say that Milvus 2.0 GA is production-ready.**

We’re already working on a new and exciting set of challenges for 2022, but I thought it would be fun to celebrate a couple of the big steps we took last year. Here are a few:


## Community Growth

First, here’s a summary of community statistics from GitHub and Slack. By the end of December 2021:

- **Contributors** have increased from 121 in December 2020 to 209 in December 2021 (up 172%)

- **Stars** have increased from 4828 in December 2020 to 9090 in December 2021 (up 188%)

- **Forks** have increased from 756 in December 2020 to 1383 in December 2021 (up 182%)

- **Slack members** have increased from 541  in December 2020 to 1233 in December 2021 (up 227%)


![img](https://assets.zilliz.com/1_1_e94deb087f.png "Milvus contributor badge.")

## Community Governance and Advocacy

When Milvus first went open source in Oct. 2019, we had a relatively small team and a small community, so naturally the project was mainly governed by a few core team members. But since then the community had grown significantly, we realized that we needed a better system to run the project so we could welcome new contributors more efficiently. 

As a result, we have appointed 5 new maintainers in 2021 to keep track of the ongoing work and reported issues to make sure that they get reviewed and merged in a timely manner. The GitHub IDs of the five maintainers are @xiaofan-luan; @congqixia; @scsven; @czs007; @yanliang567. Please feel free to contact these maintainers if you need help with your PRs.

We've also launched the [Milvus Advocate Program](https://milvus.io/community/milvus_advocate.md), and we welcome more people to join us to share your experiences, offer help to community members and gain recognition in return.


![img](https://assets.zilliz.com/1_2_835f379fb0.png "Milvus contributors.")

(Image: Milvus GitHub contributors, made with dynamicwebpaige's [project](https://github.com/dynamicwebpaige/nanowrimo-2021/blob/main/15_VS_Code_contributors.ipynb) )


## Milvus Project Announcements and Milestones

1. **Number of version releases：14**

- [Milvus 1.0 release](https://milvus.io/blog/Whats-Inside-Milvus-1.0.md)
- [Milvus 2.0 release RC](https://milvus.io/blog/milvus2.0-redefining-vector-database.md) 
- [Milvus 2.0 release PreGA](https://milvus.io/docs/v2.0.x/release_notes.md#v200-PreGA)

1. **Milvus v2.0.0 GA supported SDKs**

- PyMilvus (Available)

- Go SDK (Available)

- Java SDK (Available)

- Node.js SDK (Available)

- C++ SDK (Developing)

3. **New Milvus tools launched:**

- [Milvus_CLI](https://github.com/zilliztech/milvus_cli#community) (Milvus Command Line)
- [Attu](https://github.com/zilliztech/attu) (Milvus Management GUI)
- [Milvus K8s Operator](https://github.com/milvus-io/milvus-operator)

4. **[Milvus became a graduation project of LF AI & Data Foundation.](https://lfaidata.foundation/blog/2021/06/23/lf-ai-data-foundation-announces-graduation-of-milvus-project/)**

5. **[Milvus: A Purpose-Built Vector Data Management System published in SIGMOD'2021.](https://www.cs.purdue.edu/homes/csjgwang/pubs/SIGMOD21_Milvus.pdf))**

6. **[Milvus Community Forum launched.](https://discuss.milvus.io/)**

## Community Events

We have hosted and joined many events this year so our global community members can meet (mostly virtually) despite the current Covid-19 situation. In total, we have attended 21 conferences and hosted:

- 6 Technical Meetings
- 7 Milvus Office Hours
- 34 Webinars
- 3 Offline Meetups

We are planning for more events in 2022. If you want to join the events near you, please check the [Events and Meetup](https://discuss.milvus.io/c/events-and-meetups/13) category in our community forum to see the upcoming events and their locations. If you'd like to be our speaker or host for the future events, please contact us at [community@milvus.io](mailto:community@milvus.io).

## Looking Ahead to 2022- Roadmap & Announcement

**Community:**

1. Improve Milvus Project Membership to attract/elect more maintainers and committers to build the community together.
2. Launch Mentorship Program to offer more help to newcomers who want to join the community and contribute.
3. Improve community document governance, including **Technical documents, user guides and community documents**. In 2022, hopefully our community members can complete a Milvus Handbook together so people can learn how to use Milvus better.
4. Strengthen the cooperation and interaction with other open source communities, including upstream AI communities and communities like Kubernetes, MinIO, etcd and Pulsar which Milvus relies on.
5. Become more community-driven by having more regular SIG meetings. Besides the sig-pymilvus that is currently running, our plan is to have more SIGs in 2022.


**Milvus Project:**

1. Performance tuning

Excellent performance has always been an important reason why users choose Milvus. In 2022, we plan to start a performance optimization project to increase throughput and delay by at least twice. We also plan to introduce memory replicas to improve throughput and system stability under small data set, and support GPU to accelerate index building and online serving.

2. Functionality

Milvus 2.0 has already supported functionalities such as vector/scalar hybrid search, entity deletion and time travel. We plan to support the following features in the next two major releases:

- Support for richer data types: String, Blob, Geospatial, etc. 
- Role-based access control
- Primary key deduplication
- Support for range search on vectors (search where distance < 0.8)
- Restful API support, and other language SDKs

3. Ease of use

In the coming year we plan to develop several tools to help better deploy and manage Milvus.

- Milvus up:  A deployment component that helps users to bring up Milvus in an offline environment without K8s cluster. It also helps to deploy monitoring, tracing and other Milvus development.

- Attu — We'll keep improving Attu as our cluster management system. We are planing to add functionalities such as health diagnosis and index optimization.

- Milvus DM: Data migration tool for migrating vectors from other database or files to Milvus. We'll first support FAISS, HNSW, Milvus 1.0/2.0, then other databases such as MySQL and Elasticsearch.

## About the author

Xiaofan Luan, partner and Engineering Director of Zilliz, and Technical Advisory Committee member of LF AI & Data Foundation. He worked successively in the Oracle US headquarters and Hedvig, a software defined storage startup. He joined Alibaba Cloud Database team and was in charge of the development of  NoSQL database HBase and  Lindorm. Luan obtained his master's degree in Electronic Computer Engineering from Cornell University.


Milvus in 2021 - Year in Review


In the age of information explosion, we are producing voice, images, videos, and other unstructured data all the time. How do we efficiently analyze this massive amount of data? The emergence of neural networks enables unstructured data to be embedded as vectors, and the Milvus database is a basic data service software, which helps complete the storage, search, and analysis of vector data.

But how can we use the Milvus vector database quickly?

Some users have complained that APIs are hard to memorize and hope there could be simple command lines to operate the Milvus database.

We're thrilled to introduce Milvus_CLI, a command-line tool dedicated to the Milvus vector database.

Milvus_CLI is a convenient database CLI for Milvus, supporting database connection, data import, data export, and vector calculation using interactive commands in shells. The latest version of Milvus_CLI has the following features.

- All platforms supported, including Windows, Mac, and Linux

- Online and offline installation with pip supported

- Portable, can be used anywhere

- Built on the Milvus SDK for Python

- Help docs included

- Auto-complete supported

## Installation

You can install Milvus_CLI either online or offline.

### Install Milvus_CLI online

Run the following command to install Milvus_CLI online with pip. Python 3.8 or later is required.

```
pip install milvus-cli
```

### Install Milvus_CLI offline

To install Milvus_CLI offline, [download](https://github.com/milvus-io/milvus_cli/releases) the latest tarball from the release page first.

![1.png](https://assets.zilliz.com/1_af0e832119.png "Download tarball.")

After the tarball is downloaded, run the following command to install Milvus_CLI.

```
pip install milvus_cli-<version>.tar.gz
```

After Milvus_CLI is installed, run `milvus_cli`. The `milvus_cli >` prompt that appears indicates that the command line is ready.

![2.png](https://assets.zilliz.com/2_b50f5d2a5a.png "The command line is ready.")

If you're using a Mac with the M1 chip or a PC without a Python environment, you can choose to use a portable application instead. To accomplish this, [download](https://github.com/milvus-io/milvus_cli/releases) a file on the release page corresponding to your OS, run `chmod +x` on the file to make it executable, and run `./` on the file to run it.

#### **Example**

The following example makes `milvus_cli-v0.1.8-fix2-macOS` executable and runs it.

```
sudo chmod +x milvus_cli-v0.1.8-fix2-macOS
./milvus_cli-v0.1.8-fix2-macOS
```

## Usage

### Connect to Milvus

Before connecting to Milvus, ensure that Milvus is installed on your server. See [Install Milvus Standalone](https://milvus.io/docs/v2.0.x/install_standalone-docker.md) or [Install Milvus Cluster](https://milvus.io/docs/v2.0.x/install_cluster-docker.md) for more information.

If Milvus is installed on your localhost with the default port, run `connect`.

![3.png](https://assets.zilliz.com/3_f950d3739a.png "Connect.")

Otherwise, run the following command with the IP address of your Milvus server. The following example uses `172.16.20.3` as the IP address and `19530` as the port number.

```
connect -h 172.16.20.3
```

![4.png](https://assets.zilliz.com/4_9ff2db9855.png "Connect.")

### Create a collection

This section introduces how to create a collection.

A collection consists of entities and is similar to a table in RDBMS. See [Glossary](https://milvus.io/docs/v2.0.x/glossary.md) for more information.

![5.png](https://assets.zilliz.com/5_95a88c1cbf.png "Create a collection.")

#### Example

The following example creates a collection named `car`. The `car` collection has four fields which are `id`, `vector`, `color`, and `brand`. The primary key field is `id`. See [create collection](https://milvus.io/docs/v2.0.x/cli_commands.md#create-collection) for more information.

```
create collection -c car -f id:INT64:primary_field -f vector:FLOAT_VECTOR:128 -f color:INT64:color -f brand:INT64:brand -p id -a -d 'car_collection'
```

### List collections

Run the following command to list all collections in this Milvus instance.

```
list collections
```

![6.png](https://assets.zilliz.com/6_1331f4c8bc.png "List collections.")

Run the following command to check the details of the `car` collection.

```
describe collection -c car 
```

![7.png](https://assets.zilliz.com/7_1d70beee54.png "Details of the car collection.")

### Calculate the distance between two vectors

Run the following command to import data into the `car` collection.

```
import -c car 'https://raw.githubusercontent.com/zilliztech/milvus_cli/main/examples/import_csv/vectors.csv'
```

![8.png](https://assets.zilliz.com/8_7609a4359a.png "Calculate the distance between two vectors.")

Run `query` and enter `car` as the collection name and `id>0` as the query expression when prompted. The IDs of the entities that meet the criteria are returned as shown in the following figure.

![9.png](https://assets.zilliz.com/9_f0755589f6.png "Results.")


Run `calc` and enter appropriate values when prompted to calculate the distances between vector arrays.

### Delete a collection

Run the following command to delete the `car` collection.

```
delete collection -c car
```

![10.png](https://assets.zilliz.com/10_16b2b01935.png "Delete a collection.")

## More

Milvus_CLI is not limited to the preceding functions. Run `help` to view all commands that Milvus_CLI includes and the respective descriptions. Run `<command> --help` to view the details of a specified command.

![11.png](https://assets.zilliz.com/11_5f31ccb1e8.png "Details of commands.")

**See also:**

[Milvus_CLI Command Reference](https://milvus.io/docs/v2.0.x/cli_commands.md) under Milvus Docs

We hope Milvus_CLI could help you easily use the Milvus vector database. We will keep optimizing Milvus_CLI and your contributions are welcome.

If you have any questions, feel free to [file an issue](https://github.com/zilliztech/milvus_cli/issues) on GitHub.


Get started with Milvus_CLI

Evolution of Milvus Cloud-scalable Vector Database


In recent years, the issue of IP protection has come under the limelight as people's awareness of IP infringement is ever-increasing. Most notably, the multi-national technology giant Apple Inc. has been actively [filing lawsuits against various companies for IP infringement](https://en.wikipedia.org/wiki/Apple_Inc._litigation), including trademark, patent, and design infringement. Apart from those most notorious cases, Apple Inc. also [disputed a trademark application by Woolworths Limited](https://www.smh.com.au/business/apple-bites-over-woolworths-logo-20091005-ghzr.html), an Australian supermarket chain, on the grounds of trademark infringement in 2009.  Apple. Inc argued that the logo of the Australian brand, a stylized "w", resembles their own logo of an apple. Therefore, Apple Inc. took objection to the range of products, including electronic devices, that Woolworths applied to sell with the logo. The story ends with Woolworths amending its logo and Apple withdrawing its opposition.

![Logo of Woolworths.png](https://assets.zilliz.com/Logo_of_Woolworths_b04ece5b20.png "Logo of Woolworths.")


![Logo of Apple Inc.png](https://assets.zilliz.com/Logo_of_Apple_Inc_181e5bd5f8.png "Logo of Apple Inc.")



With the ever-increasing awareness of brand culture, companies are keeping a closer eye on any threats that will harm their intellectual properties (IP) rights. IP infringement includes:

- Copyright infringement
- Patent infringement
- Trademark infringement
- Design infringement
- Cybersquatting

The aforementioned dispute between Apple and Woolworths is mainly over trademark infringement, precisely the similarity between the trademark images of the two entities. To refrain from becoming another Woolworths, an exhaustive trademark similarity search is a crucial step for applicants both prior to the filing as well as during the review of trademark applications. The most common resort is through a search on the [United States Patent and Trademark Office (USPTO) database](https://tmsearch.uspto.gov/search/search-information) that contains all of the active and inactive trademark registrations and applications. Despite the not so charming UI, this search process is also deeply flawed by its text-based nature as it relies on words and Trademark Design codes (which are hand annotated labels of design features) to search for images.

![8.png](https://assets.zilliz.com/image_8_b2fff6ca11.png "The text-based trademark search options offered by the Trademark Electronic Search System (TESS).")


This article thereby intends to showcase how to build an efficient image-based trademark similarity search system using [Milvus](https://milvus.io), an open-source vector database.

## A vector similarity search system for trademarks

To build a vector similarity search system for trademarks, you need to go through the following steps:

1. Prepare a massive dataset of logos. Likely, the system can use a dataset like [this](https://developer.uspto.gov/product/trademark-24-hour-box-and-supplemental),).
2. Train an image feature extraction model using the dataset and data-driven models or AI algorithms.
3. Convert logos into vectors using the trained model or algorithm in Step 2.
4. Store the vectors and conduct vector similarity searches in Milvus, the open-source vector database.

![Nike.png](https://assets.zilliz.com/trademark_system_e9700df555.png "A demo of the vector similarity search system for trademarks.")


In the following sections, let's take a closer look at the two major steps in building a vector similarity search system for trademarks: using AI models for image feature extraction, and using Milvus for vector similarity search. In our case, we used VGG16, a convolutional neural network (CNN), to extract image features and convert them into embedding vectors.

### Using VGG16 for image feature extraction

[VGG16](https://medium.com/@mygreatlearning/what-is-vgg16-introduction-to-vgg16-f2d63849f615) is a CNN designed for large-scale image recognition. The model is quick and accurate in image recognition and can be applied to images of all sizes. The following are two illustrations of the VGG16 architecture.

![9.png](https://assets.zilliz.com/vgg16_layers_9e621f62cc.png "VGG16 layers")

![10.png](https://assets.zilliz.com/vgg16_architecture_992614e882.png "VGG16 architecture")

The VGG16 model, as its name suggests, is a CNN with 16 layers. All VGG models, including VGG16 and VGG19, contain 5 VGG blocks, with one or more convolutional layers in each VGG block. And at the end of each block, a max pooling layer is connected to reduce the size of the input image. The number of kernels is equivalent within each convolutional layer but doubles in each VGG block. Therefore, the number of kernels in the model grows from 64 in the first block, to 512 in the fourth and fifth blocks. All the convolutional kernels are 3*3-sized while the pooling kernels are all 2*2-sized. This is conducive to preserving more information about the input image.

Therefore, VGG16 is a suitable model for image recognition of massive datasets in this case. You can use Python, Tensorflow, and Keras to train an image feature extraction model on the basis of VGG16.

### Using Milvus for vector similarity search

After using the VGG16 model to extract image features and convert logo images into embedding vectors, you need to search for similar vectors from a massive dataset. 

Milvus is a cloud-native database featuring high scalability and elasticity. Also, as a database, it can ensure data consistency. For a trademark similarity search system like this, new data like the latest trademark registrations are uploaded to the system in real time. And these newly uploaded data need to be available for search immediately. Therefore, this article adopts Milvus, the open-source vector database, to conduct vector similarity search.

When inserting the logo vectors, you can create collections in Milvus for different types of logo vectors according to the [International (Nice) Classification of Goods and Services](https://en.wikipedia.org/wiki/International_(Nice)_Classification_of_Goods_and_Services), a system of classifying goods and services for registering trademarks. For example, you can insert a group of vectors of clothing brand logos into a collection named "clothing" in Milvus and insert another group of vectors of technological brand logos into a different collection named "technology". By doing so, you can greatly increase the efficiency and speed of your vector similarity search.

Milvus not only supports multiple indexes for vector similarity search, but also provides rich APIs and tools to facilitate DevOps. The following diagram is an illustration of the [Milvus architecture](https://milvus.io/docs/v2.0.x/architecture_overview.md). You can learn more about Milvus by reading its [introduction](https://milvus.io/docs/v2.0.x/overview.md).

![11.png](https://assets.zilliz.com/milvus_architecture_ea45a5ab53.png "The Milvus architecture.")


## Looking for more resources?

- Build more vector similarity search systems for other application scenarios with Milvus:
  - [DNA Sequence Classification based on Milvus](https://milvus.io/blog/dna-sequence-classification-based-on-milvus.md)
  - [Audio Retrieval Based on Milvus](https://milvus.io/blog/audio-retrieval-based-on-milvus.md)
  - [4 Steps to Building a Video Search System](https://milvus.io/blog/building-video-search-system-with-milvus.md)
  - [Building an Intelligent QA System with NLP and Milvus](https://milvus.io/blog/building-intelligent-chatbot-with-nlp-and-milvus.md)
  - [Accelerating New Drug Discovery](https://milvus.io/blog/molecular-structure-similarity-with-milvus.md)

- Engage with our open-source community:
  - Find or contribute to Milvus on [GitHub](https://bit.ly/307b7jC).
  - Interact with the community via [Forum](https://bit.ly/3qiyTEk).
  - Connect with us on [Twitter](https://bit.ly/3ob7kd8).


Milvus in IP Protection：Building a Trademark Similarity Search System with Milvus


To help you quickly get started with Milvus, the open-source vector database, we released another affiliated open-source project, [Milvus Bootcamp](https://github.com/milvus-io/bootcamp) on GitHub. The Milvus Bootcamp not only provides scripts and data for benchmark tests, but also includes projects that use Milvus to build some MVPs (minimum viable products), such as a reverse image search system, a video analysis system, a QA chatbot, or a recommender system. You can learn how to apply vector similarity search in a world full of unstructured data and get some hands-on experience in Milvus Bootcamp.

![2.png](https://assets.zilliz.com/2_5b60157b4d.png)

We provide both front-end and back-end services for the projects in Milvus Bootcamp. However, we have recently made the decision to change the adopted web framework from Flask to FastAPI.

This article aims to explain our motivation behind such a change in the adopted web framework for Milvus Bootcamp by clarifying why we chose FastAPI over Flask.

## Web frameworks for Python

A web framework refers to a collection of packages or modules. It is a set of software architecture for web development that allows you to write web applications or services and saves you the trouble of handling low-level details such as protocols, sockets, or process/thread management. Using web framework can significantly reduce the workload of developing web applications as you can simply "plug in" your code into the framework, with no extra attention needed when dealing with data caching, database access, and data security verification. For more information about what a web framework for Python is, see [Web Frameworks](https://wiki.python.org/moin/WebFrameworks).  

There are various types of Python web frameworks. The mainstream ones include Django, Flask, Tornado, and FastAPI.

### Flask

![3.png](https://assets.zilliz.com/3_1abd170939.png)

[Flask](https://flask.palletsprojects.com/en/2.0.x/) is a lightweight microframework designed for Python, with a simple and easy-to-use core that allows you to develop your own web applications. In addition, the Flask core is also extensible. Therefore, Flask supports on-demand extension of different functions to meet your personalized needs during web application development. This is to say, with a library of various plug-ins in Flask, you can develop powerful websites.

Flask has the following characteristics:

1. Flask is a microframework that does not rely on other specific tools or components of third-party libraries to provide shared functionalities. Flask does not have a database abstraction layer, and does not require form validation. However, Flask is highly extensible and supports adding application functionality in a way similar to implementations within Flask itself. Relevant extensions include object-relational mappers, form validation, upload processing, open authentication technologies, and some common tools designed for web frameworks.
2. Flask is a web application framework based on [WSGI](https://wsgi.readthedocs.io/) (Web Server Gateway Interface). WSGI is a simple interface connecting a web server with a web application or framework defined for the Python language.
3. Flask includes two core function libraries, [Werkzeug](https://www.palletsprojects.com/p/werkzeug) and [Jinja2](https://www.palletsprojects.com/p/jinja). Werkzeug is a WSGI toolkit that implements request, response objects and practical functions, which allows you to build web frameworks on top of it. Jinja2 is a popular full-featured templating engine for Python. It has full support for Unicode, with an optional but widely-adopted integrated sandbox execution environment.

### FastAPI

![4.png](https://assets.zilliz.com/4_05cb0dac4e.png)

[FastAPI](https://fastapi.tiangolo.com/) is a modern Python web application framework that has the same level of high performance as Go and NodeJS. The core of FastAPI is based on [Starlette](https://www.starlette.io/) and [Pydantic](https://pydantic-docs.helpmanual.io/). Starlette is a lightweight [ASGI](https://asgi.readthedocs.io/)(Asynchronous Server Gateway Interface) framework toolkit for building high-performance [Asyncio](https://docs.python.org/3/library/asyncio.html) services. Pydantic is a library that defines data validation, serialization, and documentation based on Python type hints.

FastAPI has the following characteristics:

1. FastAPI is a web application framework based on ASGI, an asynchronous gateway protocol interface connecting network protocol services and Python applications. FastAPI can handle a variety of common protocol types, including HTTP, HTTP2, and WebSocket.
2. FastAPI is based on Pydantic, which provides the function of checking the interface data type. You do not need to additionally verify your interface parameter, or write extra code to verify whether the parameters are empty or whether the data type is correct. Using FastAPI can effectively avoid human errors in code and improve development efficiency.
3. FastAPI supports document in two formats - [OpenAPI](https://swagger.io/specification/) (formerly Swagger) and [Redoc](https://www.redoc.com/). Therefore, as a user you do not need to spend extra time writing additional interface documents. The OpenAPI document provided by FastAPI is shown in the screenshot below.

![5.png](https://assets.zilliz.com/5_d91d34cb0f.png)

### Flask Vs. FastAPI

The table below demonstrates the differences between Flask and FastAPI in several aspects.

|                            | **FastAPI**    | **Flask**     |
| -------------------------- | -------------- | ------------- |
| **Interface gateway**      | ASGI           | WSGI          |
| **Asynchronous framework** | ✅              | ❌             |
| **Performance**            | Faster         | Slower        |
| **Interactive doc**        | OpenAPI, Redoc | None          |
| **Data verification**      | ✅              | ❌             |
| **Development costs**      | Lower          | Higher        |
| **Ease of use**            | Lower          | Higher        |
| **Flexibility**            | Less flexible  | More flexible |
| **Community**              | Smaller        | More active   |

## Why FastAPI?

Before deciding which Python web application framework to choose for the projects in Milvus Bootcamp, we researched into several mainstream frameworks including Django, Flask, FastAPI, Tornado, and more. Since the projects in Milvus Bootcamp serve as references for you, our priority is to adopt an external framework of utmost lightweightness and dexterity. According to this rule, we narrowed down our choices to Flask and FastAPI.

You can see the comparison between the two web frameworks in the previous section. The following is a detailed explanation of our motivation to choose FastAPI over Flask for the projects in Milvus Bootcamp. There are several reasons:

### 1. Performance

Most of the projects in Milvus Bootcamp are built around reverse image search systems, QA chatbots, text search engines, which all have high demands for real-time data processing. Accordingly, we need a framework with outstanding performance, which is exactly a highlight of FastAPI. Therefore, from the perspective of system performance, we decided to choose FastAPI.

### 2. Efficiency

When using Flask, you need to write code for data type verification in each of the interfaces so that the system can determine whether the input data is empty or not. However, by supporting automatic data type verification, FastAPI helps avoid human errors in coding during system development and can greatly boost development efficiency. Bootcamp is positioned as a type of training resource. This means that the code and components we use must be intuitive and highly efficient. In this regard, we chose FastAPI to improve system efficiency and enhance user experience.

### 3. Asynchronous framework

FastAPI is inherently an asynchronous framework. Originally, we released four [demos](https://zilliz.com/milvus-demos?isZilliz=true), reverse image search, video analysis, QA chatbot, and molecular similarity search. In these demos, you can upload datasets and will be immediately prompted "request received". And when the data is uploaded to the demo system, you will receive another prompt "data upload successful". This is an asynchronous process which requires a framework that supports this feature. FastAPI is itself an asynchronous framework. To align all Milvus resources, we decided to adopt a single set of development tools and software for both Milvus Bootcamp and Milvus demos. As a result, we changed the framework from Flask to FastAPI.

### 4. Automatic interactive documents

In a traditional way, when you finish writing the code for the server-side, you need to write an extra document to create an interface, and then use tools like [Postman](https://www.postman.com/) for API testing and debugging. So what if you only want to quickly get started with the web server-side development part of the projects in Milvus Bootcamp without writing additional code to create an interface? FastAPI is the fix. By providing an OpenAPI document, FastAPI can save you the trouble of testing or debugging APIs and collaborating with front-end teams to develop a user interface. With FastAPI, you can still quickly try the built application with an automatic but intuitive interface without extra efforts for coding.

### 5. User-friendliness

FastAPI is easier to use and develop, therefore enabling you to pay more attention to the specific implementation of the project itself. Without spending too much time on developing web frameworks, you can focus more on understanding the projects in Milvus Bootcamp.

## Recap

Flask and FlastAPI have their own pros and cons. As an emerging web application framework, FlastAPI, at its core, is built on mature toolkits and library, Starlette and Pydantic. FastAPI is an asynchronous framework with high performance. Its dexterity, extensibility, and support for automatic data type verification, together with many other powerful features, prompted us to adopt FastAPI as the framework for Milvus Bootcamp projects.

Please note that you should choose the appropriate framework according to your application scenario if you want to build a vector similarity search system in production.

## About the author

Yunmei Li, Zilliz Data Engineer, graduated from Huazhong University of Science and Technology with a degree in computer science. Since joining Zilliz, she has been working on exploring solutions for the open source project Milvus and helping users to apply Milvus in real-world scenarios. Her main focus is on NLP and recommendation systems, and she would like to further deepen her focus in these two areas. She likes to spend time alone and read.

## Looking for more resources?

- Start to build AI system with Milvus and get more hands-on experience by reading our tutorials!
  - [What Is It? Who Is She? Milvus Helps Analyze Videos Intelligently](https://milvus.io/blog/2021-10-10-milvus-helps-analys-vedios.md)
  - [Combine AI Models for Image Search using ONNX and Milvus](https://milvus.io/blog/2021-09-26-onnx.md)
  - [DNA Sequence Classification based on Milvus](https://milvus.io/blog/dna-sequence-classification-based-on-milvus.md)
  - [Audio Retrieval Based on Milvus](https://milvus.io/blog/audio-retrieval-based-on-milvus.md)
  - [4 Steps to Building a Video Search System](https://milvus.io/blog/building-video-search-system-with-milvus.md)
  - [Building an Intelligent QA System with NLP and Milvus](https://milvus.io/blog/building-intelligent-chatbot-with-nlp-and-milvus.md)
  - [Accelerating New Drug Discovery](https://milvus.io/blog/molecular-structure-similarity-with-milvus.md)

- Engage with our open-source community:
  - Find or contribute to Milvus on [GitHub](https://bit.ly/307b7jC).
  - Interact with the community via [Forum](https://bit.ly/3qiyTEk).
  - Connect with us on [Twitter](https://bit.ly/3ob7kd8).


Why to Choose FastAPI over Flask?

Accelerating Candidate Generation in Recommender Systems Using Milvus paired with PaddlePaddle


This tutorial introduces how to run Milvus 2.0 on Windows. If you want to get started with Milvus 2.0 using Windows, or simply have to run Milvus on Windows due to environment limits, this tutorial can be a good reference. This article is originally written by [@matrixji](https://github.com/matrixji).

There are two ways to run Milvus 2.0 on Windows:

- [Compile from source code](#Compile-from-source-code)
- [Install from the binary package](#Install-from-the-binary-package)

## Compile from source code

### Prerequisites

You need to install MinGW64/MSYS toolchain before compiling Milvus 2.0 on Windows. See [MSYS2](https://www.msys2.org/) for the installation guide. The compilation in this tutorial is completed in the shell of MSYS2 MinGW 64-bit. You can run `$ pacman -Su` to update the toolchain.

We also recommend that you run `$ pacman -S git` to install git.

### Step 1. Clone code

Run the following command to clone the code from the **windows-dev** branch in the **matrixji/milvus** repository. Currently, all commits can be traced in **windows-dev**, which rebases commits from the upstream main branch [**milvus-io/milvus**](https://github.com/milvus-io/milvus) on a weekly basis.

```python
$ git clone git@github.com:matrixji/milvus.git -b windows-dev
```

### Step 2. Compile Milvus

#### 1. Install dependencies and toolchain

Run the following command to install the required dependencies and toolchain. After the installation, you need to restart the MinGW/MYSY shell to ensure the configurations of some tools are properly applied.

```python
$ cd milvus
$ scripts/install_deps_msys.sh
```

#### 2. Compile Milvus

Run the following command to compile and package Milvus.

```python
$ make
$ sh scripts/package_windows.sh
```

If successful, you will find a zip file in the subdirectory of `windows_package`. The zip file contains all the files for running `milvus.exe`, including `minio.exe` and `etcd.exe`.

<div class="alert note">
Milvus 2.0 is developed in Go. You might need to configure <a href='https://goproxy.cn/'>Goproxy</a> to install third-party modules.
</div>

## Install from the binary package

You can also download the binary package that has already been compiled from [Releases - matrixji/milvus](https://github.com/matrixji/milvus/releases). Each release has **windows-test-** as the prefix, for instance, **windows-test-v8**. Find the release you want and download the corresponding zip package.

### Step 1. Start Milvus

#### 1. Unzip the package

Unzip the package, and you will find a `milvus` directory, which contains all the files required.

#### 2. Start a MinIO service

Double-click the `run_minio.bat` file to start a MinIO service with default configurations. Data will be stored in the subdirectory `s3data`.

#### 3. Start an etcd service

Double-click the `run_etcd.bat` file to start an etcd service with default configurations.

#### 4. Start Milvus service

Double-click the `run_milvus.bat` file to start the Milvus service.

### Step 2. Run `hello_milvus.py`

After starting the Milvus service, you can test by running `hello_milvus.py`. See [Hello Milvus](https://milvus.io/docs/v2.0.x/example_code.md) for more information.


## Engage with our open-source community:
- Find or contribute to Milvus on [GitHub](https://bit.ly/3khejQB).
- Interact with the community via [Forum](https://bit.ly/307HVsY).
- Connect with us on [Twitter](https://bit.ly/3wn5aek).


Run Milvus 2.0 on Windows


# It's a Wrap - Milvus Hacktoberfest 2021

![img](https://assets.zilliz.com/Blog_cover_a6ce8748d7.jpeg "Cover image.")

Hacktoberfest has come to an end but there is no end to contributing to open-source projects!


Throughout October, we had a total of **44 pull requests** (PRs) across our repos from **36 contributors** (excluding our core team). Though it is the first year for the Milvus community to join Hacktoberfest, the number of participants we saw was beyond our expectation, an indication of the growing awareness of open-source spirit.

We hope everyone who participated in this event gained some hands-on experience or knowledge of open source, community, and helpful technical skills  in the process.️️️

In this post, we’d like to invite you to celebrate our achievements together and how to continue to contribute to Milvus after Hacktoberfest.

## **📣 Shout out to our contributors:**

During Hacktoberfest this year, the Milvus project repositories saw **44 pull requests merged**! This is a huge accomplishment on all sides. Good job, everyone! 🎉

[parthiv11](https://github.com/parthiv11), [joremysh](https://github.com/joremysh), [noviicee](https://github.com/noviicee), [Biki-das](https://github.com/Biki-das), [Nadyamilona](https://github.com/Nadyamilona), [ashish4arora](https://github.com/ashish4arora), [Dhruvacube](https://github.com/Dhruvacube), [iamartyaa](https://github.com/iamartyaa), [RafaelDSS](https://github.com/RafaelDSS), [kartikcho](https://github.com/kartikcho), [GuyKh](https://github.com/GuyKh), [Deep1Shikha](https://github.com/Deep1Shikha), [shreemaan-abhishek](https://github.com/shreemaan-abhishek), [daniel-shuy](https://github.com/daniel-shuy), [Hard-Coder05](https://github.com/Hard-Coder05), [sapora1](https://github.com/sapora1), [Rutam21](https://github.com/Rutam21), [idivyanshbansal](https://github.com/idivyanshbansal), [Mihir501](https://github.com/Mihir501), [YushChaudhary](https://github.com/Ayushchaudhary-Github), [sreyan-ghosh](https://github.com/sreyan-ghosh), [chiaramistro](https://github.com/chiaramistro), [appledora](https://github.com/appledora), [luisAzcuaga](https://github.com/luisAzcuaga), [matteomessmer](https://github.com/matteomessmer), [Nadyamilona](https://github.com/Nadyamilona), [Tititesouris](https://github.com/Tititesouris), [amusfq](https://github.com/amusfq), [matrixji](https://github.com/matrixji) & [generalZman](https://github.com/zamanmub)



![image-20211110180357460](https://assets.zilliz.com/_80b0d87746.png "Hacktoberfest 2021.")



### Here are some extraordinary Milvus Hacktoberfest 2021 contributions:



**⚙️ New features**

[Compile & Run Milvus across multiple platforms](https://github.com/milvus-io/milvus/issues/7706) by [matrixji](https://github.com/matrixji)   

**(🏆  Top Contributor  🏆 )**



**📝  Documentation**

- [Translate Hello Milvus (example_code.md) to any language](https://github.com/milvus-io/bootcamp/issues/720) by [chiaramistro](https://github.com/chiaramistro), [appledora](https://github.com/appledora), [luisAzcuaga](https://github.com/luisAzcuaga), [matteomessmer](https://github.com/matteomessmer), [Nadyamilona](https://github.com/Nadyamilona), [Tititesouris](https://github.com/Tititesouris) & [amusfq](https://github.com/amusfq)
- [Add NodeJS example to example_code.md](https://github.com/milvus-io/bootcamp/issues/720) by [GuyKh](https://github.com/GuyKh) 
- [translated upgrade.md](https://github.com/milvus-io/milvus-docs/pull/921/files) & [add and translate parameters in User Guide to CN](https://github.com/milvus-io/milvus-docs/pull/892) by [joremysh](https://github.com/joremysh)
- [Translate tutorials/dna_sequence_classification.md into CN](https://github.com/milvus-io/milvus-docs/pull/753) & [Translate reference/schema/collection_schema.md into CN](https://github.com/milvus-io/milvus-docs/pull/752) by [daniel-shuy](https://github.com/daniel-shuy)



**🚀  Bootcamp**

- [Enhancement Jupyter Notebook tutorial for image hash search ](https://github.com/milvus-io/bootcamp/pull/858)by [generalZman](https://github.com/zamanmub)
- [fix the TOPK bug of reverse_image_search](https://github.com/milvus-io/bootcamp/pull/792) by [RafaelDSS](https://github.com/RafaelDSS)



**🐍 PyMilvus**

- [Update GitHub issue forms ](https://github.com/milvus-io/pymilvus/issues/741)(multiple repos) by [Hard-Coder05](https://github.com/Hard-Coder05) 



### Share your feedback with us! 

![img](https://assets.zilliz.com/h3_412b0f649b.png "Feedback.")

You are more than welcome to share your experience of Milvus Hacktoberfest 2021 with us! Whether it's a blog post, a tweet (@milvusio) or just a post in our [forum](https://discuss.milvus.io/c/hacktoberfest/9), anything will be greatly appreciated!



## What's Next？



### **👩‍💻** **Code & Documentation**

If you have limited knowledge of Milvus, you can get familiar with how the community works first by contributing to [pymilvus](https://github.com/milvus-io/pymilvus) or [docs](https://github.com/milvus-io/milvus-docs) repos. You can also look for tags such as **#goodfirstissue** or **#helpwanted**. 

![img](https://assets.zilliz.com/h4_f18c9b6c2c.png "Good first issue.")

![img](https://assets.zilliz.com/h5_a4f90c24a8.png "Help wanted.")

If you have any questions about contributing, you can always ask the community under the Forum's **Contributor** category: https://discuss.milvus.io/t/things-you-need-to-know-before-you-get-started/64. 



### 🏆 Be a Milvus Advocate

![image-20211110180730866](https://assets.zilliz.com/advocate_1052d8249a.jpg "Milvus advocate.")



Share your experience and things you've learned with the community；propose ideas on how we can improve; answer questions and help out others on Milvus Forum, etc. There are many ways for you to participate in the community, we've listed a few examples below but we welcome any forms of contribution: 

- **Demos and solutions:** Show Milvus users how to leverage the platform in specific scenarios (e.g. music recommendation system). Examples are available in the [Milvus Bootcamp](https://github.com/milvus-io/bootcamp).

- **Blog articles, user stories, or whitepapers:** Write high-quality content that clearly and accurately explains technical details about Milvus.

- **Tech talks/live broadcasts:** Give talks or host live broadcasts that help raise awareness around Milvus.

- **Others:** Any content that plays a positive role in the development of Milvus and its open-source community will be considered eligible.



For more details, please read: https://milvus.io/community/milvus_advocate.md



Lastly, thank you again for joining this year's Hacktoberfest with us and letting us be your mentors and students. Thank you [Digital Ocean](https://hacktoberfest.digitalocean.com/) for hosting yet another year of this amazing event. Until next time!



Happy Coding!

Milvus Community Team


It's a Wrap! Milvus Hacktoberfest 2021


*Yufen Zong, a Zilliz Test Development Engineer, graduated from Huazhong University of Science and Technology with a master's degree in computer technology. She is currently engaged in the quality assurance of Milvus vector database, including but not limited to interface integration testing, SDK testing, Benchmark testing, etc. Yufen is an enthusiastic problem-shooter in the test and development of Milvus, and a huge fan of chaos engineering theory and fault drill practice.*

## Background


While using Milvus vector database, you will need to modify the default configuration to satisfy the requirements of different scenarios. Previously, a Milvus user shared on [How to Modify the Configuration of Milvus Deployed Using Docker Compose](2021-10-22-apply-configuration-changes-on-milvus-2.md). And in this article, I would like to share with you on how to modify the configuration of Milvus deployed on Kubernetes.

## Modify configuration of Milvus on Kubernetes

You may choose different modification plans according to the configuration parameters you wish to modify. All Milvus configuration files are stored under **milvus/configs**. While installing Milvus on Kubernetes, a Milvus Helm Chart repository will be added locally. By running `helm show values milvus/milvus`, you can check the parameters that can be modified directly with Chart. For the modifiable parameters with Chart, you can pass the parameter using `--values` or `--set`. For more information, see [Milvus Helm Chart](https://artifacthub.io/packages/helm/milvus/milvus) and [Helm](https://helm.sh/docs/).

If the parameters you expect to modify are not on the list, you can follow the instruction below.

In the following steps, the parameter `rootcoord.dmlChannelNum` in **/milvus/configs/advanced/root_coord.yaml** is to be modified for demonstration purposes. Configuration file management of Milvus on Kubernetes is implemented through ConfigMap resource object. To change the parameter, you should first update the ConfigMap object of corresponding Chart release, and then modify the deployment resource files of corresponding pods. 

Beware that this method only applies to parameter modification on deployed Milvus application. To modify the parameters in **/milvus/configs/advanced/\*.yaml** before deployment, you will need to re-develop the Milvus Helm Chart.

### Modify ConfigMap YAML

As shown below, your Milvus release running on Kubernetes corresponds to a ConfigMap object with the same name of the release. The `data` section of the ConfigMap object only includes configurations in **milvus.yaml**. To change the `rootcoord.dmlChannelNum` in **root_coord.yaml**, you must add the parameters in **root_coord.yaml** to the `data` section in the ConfigMap YAML and change the specific parameter. 

```
kind: ConfigMap
apiVersion: v1
metadata:
  name: milvus-chaos
  ...
data:
  milvus.yaml: >
    ......
  root_coord.yaml: |
    rootcoord:
      dmlChannelNum: 128
      maxPartitionNum: 4096
      minSegmentSizeToEnableIndex: 1024
      timeout: 3600 # time out, 5 seconds
      timeTickInterval: 200 # ms
```

### Modify Deployment YAML

The data stored in a ConfigMap can be referenced in a volume of type configMap and then consumed by containerized applications running in a pod. To direct the pods to the new configuration files, you must modify the pod templates that need to load the configurations in **root_coord.yaml**. Specifically, you need to add a mount declaration under the `spec.template.spec.containers.volumeMounts` section in deployment YAML.

Taking the deployment YAML of rootcoord pod as an example, a `configMap` type volume named **milvus-config** is specified in `.spec.volumes` section. And, in `spec.template.spec.containers.volumeMounts` section, the volume is declared to mount **milvus.yaml** of your Milvus release on **/milvus/configs/milvus.yaml**. Similarly, you only need to add a mount declaration specifically for rootcoord container to mount the **root_coord.yaml** on **/milvus/configs/advanced/root_coord.yaml**, and thus the container can access the new configuration file.

```yaml
spec:
  replicas: 1
  selector:
    ......
  template:
    metadata:
      ...
    spec:
      volumes:
        - name: milvus-config
          configMap:
            name: milvus-chaos
            defaultMode: 420
      containers:
        - name: rootcoord
          image: 'milvusdb/milvus-dev:master-20210906-86afde4'
          args:
            ...
          ports:
            ...
          resources: {}
          volumeMounts:
            - name: milvus-config
              readOnly: true
              mountPath: /milvus/configs/milvus.yaml
              subPath: milvus.yaml
            - name: milvus-config
              readOnly: true
              mountPath: /milvus/configs/advanced/`root_coord.yaml
              subPath: root_coord.yaml
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          imagePullPolicy: IfNotPresent
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      dnsPolicy: ClusterFirst
      securityContext: {}
      schedulerName: default-scheduler
```

### Verify the result

The kubelet checks whether the mounted ConfigMap is fresh on every periodic sync. When the ConfigMap consumed in the volume is updated, projected keys are automatically updated as well. When the new pod is running again, you can verify if the modification is successful in the pod. Commands to check the parameter `rootcoord.dmlChannelNum` are shared below.

```bash
$ kctl exec -ti milvus-chaos-rootcoord-6f56794f5b-xp2zs -- sh
# cd configs/advanced
# pwd
/milvus/configs/advanced
# ls
channel.yaml  common.yaml  data_coord.yaml  data_node.yaml  etcd.yaml  proxy.yaml  query_node.yaml  root_coord.yaml
# cat root_coord.yaml
rootcoord:
  dmlChannelNum: 128
  maxPartitionNum: 4096
  minSegmentSizeToEnableIndex: 1024
  timeout: 3600 # time out, 5 seconds
  timeTickInterval: 200 # ms
# exit
```



Above is the method to modify the advanced configurations in Milvus deployed on Kubernetes. Future release of Milvus will integrate all configurations in one file, and will support updating configuration via helm chart. But before that, I hope this article can help you as a temporary solution.



## Engage with our open-source community:

- Find or contribute to Milvus on [GitHub](https://bit.ly/307b7jC).

- Interact with the community via [Forum](https://bit.ly/3qiyTEk).

- Connect with us on [Twitter](https://bit.ly/3ob7kd8).

  


How to Modify Milvus Advanced Configurations

Frustrated with New Data? Our Vector Database can Help


# Technical Sharing: Apply Configuration Changes on Milvus 2.0 using Docker Compose


*Jingjing Jia, Zilliz Data Engineer, graduated from Xi’an Jiaotong University with a degree in Computer Science. After joining Zilliz, she mainly works on data pre-processing, AI model deployment, Milvus related technology research, and helping community users to implement application scenarios. SHe is very patient, likes to communicate with community partners, and enjoys listening to music and watching anime.*


As a frequent user of Milvus, I was very excited about the newly released Milvus 2.0 RC. According to the introduction on the official website, Milvus 2.0 seems to outmatch its predecessors by a large margin. I was so eager to try it out myself.



And I did.  However, when I truly got my hands on Milvus 2.0, I realized that I wasn't able to modify the configuration file in Milvus 2.0 as easily as I did with Milvus 1.1.1. I couldn't change the configuration file inside the docker container of Milvus 2.0 started with Docker Compose, and even force change wouldn't take effect. Later, I learned that Milvus 2.0 RC was unable to detect changes to the configuration file after installation. And future stable release will fix this issue.



Having tried different approaches, I've found a reliable way to apply changes to configuration files for Milvus 2.0 standalone & cluster, and here is how.



Note that all changes to configuration must be made before restarting Milvus using Docker Compose.

## Modify configuration file in Milvus standalone

First, you will need to [download](https://github.com/milvus-io/milvus/blob/master/configs/milvus.yaml) a copy of **milvus.yaml** file to your local device.



Then you can change the configurations in the file. For instance, you can change the log format as `.json`. 

![1.1.png](https://assets.zilliz.com/1_1_ee4a16a3ee.png "Change the log format.")



Once **milvus.yaml** file is modified, you will also need to [download](https://github.com/milvus-io/milvus/blob/master/deployments/docker/standalone/docker-compose.yml) and modify in **docker-compose.yaml** file for standalone by mapping the local path to milvus.yaml onto the corresponding docker container path to configuration file `/milvus/configs/milvus.yaml` under the `volumes` section.

![1.2.png](https://assets.zilliz.com/1_2_5e7c73708c.png "Download and modify the file.")

Lastly, start Milvus standalone using `docker-compose up -d` and check if the modifications are successful. For instance, run `docker logs` to check the log format.

![1.3.png](https://assets.zilliz.com/1_3_a0406df3ab.png "Check the log format.")



## Modify configuration file in Milvus cluster

First, [download](https://github.com/milvus-io/milvus/blob/master/configs/milvus.yaml) and modify the **milvus.yaml** file to suit your needs.

![1.4.png](https://assets.zilliz.com/1_4_758b182846.png "Download and modify the file.")


Then you will need to [download](https://github.com/milvus-io/milvus/blob/master/deployments/docker/cluster/docker-compose.yml) and modify the cluster **docker-compose.yml** file by mapping the local path to **milvus.yaml** onto the corresponding path to configuration files in all components, i.e. root coord, data coord, data node, query coord, query node, index coord, index node, and proxy.

![1.5.png](https://assets.zilliz.com/1_5_80e15811b8.png "Download and modify the file.")


![1.6.png](https://assets.zilliz.com/1_6_b2f3e4e47f.png "Download and modify the file.")
![1.7.png](https://assets.zilliz.com/1_7_4d1eb5e1e5.png "Download and modify the file.")


Finally, you can start Milvus cluster using `docker-compose up -d` and check if the modifications are successful.

## Change log file path in configuration file

First, [download](https://github.com/milvus-io/milvus/blob/master/configs/milvus.yaml) the **milvus.yaml** file, and change the `rootPath` section as the directory where you expect to store the log files in Docker container. 


![1.8.png](https://assets.zilliz.com/1_8_e3bdc4843f.png "Change the rootPath.")




After that, download the corresponding **docker-compose.yml** file for Milvus [standalone](https://github.com/milvus-io/milvus/blob/master/deployments/docker/standalone/docker-compose.yml) or [cluster](https://github.com/milvus-io/milvus/blob/master/deployments/docker/cluster/docker-compose.yml). 

For standalone, you need to map the local path to **milvus.yaml** onto the corresponding docker container path to configuration file `/milvus/configs/milvus.yaml`, and map the local log file directory onto the Docker container directory you created previously. 

For cluster, you will need to map both paths in every component.

![1.9.png](https://assets.zilliz.com/1_9_22d8929d92.png "Map paths in every component.")

Lastly, start Milvus standalone or cluster using `docker-compose up -d` and check the log files to see if the modification is successful.


Technical Sharing:Apply Configuration Changes on Milvus 2.0 using Docker Compose


#  Building a Video Analysis System with Milvus Vector Database

*Shiyu Chen, a data engineer at Zilliz, graduated from Xidian University with a degree in Computer Science. Since joining Zilliz, she has been exploring solutions for Milvus in various fields, such as audio and video analysis, molecule formula retrieval, etc., which has greatly enriched the application scenarios of the community. She is currently exploring more interesting solutions. In her spare time, she loves sports and reading.*

When I was watching *Free Guy* last weekend, I felt that I'd seen the actor who plays Buddy, the security guard, somewhere before, yet couldn't recall any of his works. My head was stuffed with "who's this guy?" I was sure about having seen that face and was trying so hard to remember his name. A similar case is that once I saw the leading actor in a video having a drink I used to like a lot, but I ended up failing to recall the brand name.

The answer was on the tip of my tongue, but my brain felt completely stuck.

The tip of the tongue (TOT) phenomenon drives me crazy when watching movies. If only there was a reverse image search engine for videos that enables me to find videos and analyze video content. Before, I built a [reverse image search engine using Milvus](https://github.com/milvus-io/bootcamp/tree/master/solutions/reverse_image_search/quick_deploy). Considering that video content analysis somehow resembles image analysis, I decided to build a video content analysis engine based on Milvus.

## Object detection

### Overview

Before being analyzed, objects in a video should be detected first. Detecting objects in a video effectively and accurately is the main challenge of the task. It is also an important task for applications such as autopilot, wearable devices, and IoT.

Developed from traditional image processing algorithms to deep neural networks (DNN), today's mainstream models for object detection include R-CNN, FRCNN, SSD, and YOLO. The Milvus-based deep learning video analysis system introduced in this topic can detect objects intelligently and quickly.

### Implementation

To detect and recognize objects in a video, the system should first extract frames from a video and detect objects in the frame images using object detection, secondly, extract feature vectors from detected objects, and lastly, analyze the object based on the feature vectors.

- Frame extraction

Video analysis is converted to image analysis using frame extraction. Currently, frame extraction technology is very mature. Programs such as FFmpeg and OpenCV support extracting frames at specified intervals. This article introduces how to extract frames from a video every second using OpenCV.

- Object detection

Object detection is about finding objects in extracted frames and extracting screenshots of the objects according to their positions. As shown in the following figures, a bike, a dog, and a car were detected. This topic introduces how to detect objects using YOLOv3, which is commonly used for object detection.


- Feature extraction

Feature extraction refers to converting unstructured data, which is difficult for machines to recognize, to feature vectors. For example, images can be converted to multi-dimensional feature vectors using deep learning models. Currently, the most popular image recognition AI models include VGG, GNN, and ResNet. This topic introduces how to extract features from detected objects using ResNet-50.

- Vector analysis

Extracted feature vectors are compared with library vectors, and the corresponding information to the most similar vectors is returned. For large-scale feature vector datasets, calculation is a huge challenge. This topic introduces how to analyze feature vectors using Milvus.

## Key technologies

### OpenCV 

Open Source Computer Vision Library (OpenCV) is a cross-platform computer vision library, which provides many universal algorithms for image processing and computer vision. OpenCV is commonly used in the computer vision field. 

The following example shows how to capture video frames at specified intervals and save them as images using OpenCV with Python.

```python
import cv2 
cap = cv2.VideoCapture(file_path)   
framerate = cap.get(cv2.CAP_PROP_FPS)   
allframes = int(cv2.VideoCapture.get(cap, int(cv2.CAP_PROP_FRAME_COUNT)))  
success, image = cap.read() 
cv2.imwrite(file_name, image)
```

### YOLOv3

You Only Look Once, Version 3 (YOLOv3 [5]) is a one-stage object detection algorithm proposed in recent years. Compared to the traditional object detection algorithms with the same accuracy, YOLOv3 is twice as fast. YOLOv3 mentioned in this topic is the enhanced version from PaddlePaddle [6]. It uses multiple optimization methods with a higher inference speed.

### ResNet-50

ResNet [7] is the winner of ILSVRC 2015 in image classification because of its simplicity and practicality. As the basis of many image analysis methods, ResNet proves to be a popular model specialized in image detection, segmentation, and recognition.

### Milvus  

[Milvus](https://milvus.io/) is a cloud-native, open-source vector database built to manage embedding vectors generated by machine learning models and neural networks. It is widely used in scenarios such as computer vision, natural language processing, computational chemistry, personalized recommender systems, and more.

The following procedures describe how Milvus works.

1. Unstructured data is converted to feature vectors by using deep learning models and is imported to Milvus.
2. Milvus stores and indexes the feature vectors.
3. Milvus returns the most similar vectors to the vector queried by users.


## Deployment

Now you have some understanding of Milvus-based video analysis systems. The system mainly consists of two parts, as shown in the following figure.

- The red arrows indicate the data import process. Use ResNet-50 to extract feature vectors from the image dataset and import the feature vectors to Milvus.

- The black arrows indicate the video analysis process. First, extract frames from a video and save the frames as images. Second, detect and extract objects in the images using YOLOv3. Then, use ResNet-50 to extract feature vectors from the images. Lastly, Milvus searches and returns the information of the objects with the corresponding feature vectors.

For more information, see [Milvus Bootcamp: Video Object Detection System](https://github.com/milvus-io/bootcamp/tree/master/solutions/video_similarity_search/object_detection). 

**Data import**

The data import process is simple. Convert the data into 2,048-dimensional vectors and import the vectors into Milvus. 

```python
vector = image_encoder.execute(filename)
entities = [vector]
collection.insert(data=entities)
```

**Video analysis**

As introduced above, the video analysis process includes capturing video frames, detecting objects in each frame, extracting vectors from the objects, calculating vector similarity with Euclidean distance (L2) metrics, and searching for results using Milvus.

```python
images = extract_frame(filename, 1, prefix)   
detector = Detector()   
run(detector, DATA_PATH)       
vectors = get_object_vector(image_encoder, DATA_PATH)
search_params = {"metric_type": "L2", "params": {"nprobe": 10}}
results = collection.search(vectors, param=search_params, limit=10)
```

## Conclusion

Currently, more than 80% of the data is unstructured. With the rapid development of AI, an increasing number of deep learning models have been developed for analyzing unstructured data. Technologies such as object detection and image processing have achieved great breakthroughs in both academia and industry. Empowered by these technologies, more and more AI platforms have fulfilled practical requirements.

The video analysis system discussed in this topic is built with Milvus, which can quickly analyze video content.

As an open-source vector database, Milvus supports feature vectors extracted using various deep learning models. Integrated with libraries such as Faiss, NMSLIB, and Annoy, Milvus provides a set of intuitive APIs, supporting switching index types according to scenarios. Additionally, Milvus supports scalar filtering, which increases recall rate and search flexibility. Milvus has been applied to many fields such as image processing, computer vision, natural language processing, speech recognition, recommender system, and new drug discovery.

## References

[1] A. D. Bagdanov, L. Ballan, M. Bertini, A. Del Bimbo. “Trademark matching and retrieval in sports video databases.” Proceedings of the international workshop on Workshop on multimedia information retrieval, ACM, 2007. https://www.researchgate.net/publication/210113141_Trademark_matching_and_retrieval_in_sports_video_databases 

[2] J. Kleban, X. Xie, W.-Y. Ma. “Spatial pyramid mining for logo detection in natural scenes.” IEEE International Conference, 2008. https://ieeexplore.ieee.org/document/4607625 

[3] R. Boia, C. Florea, L. Florea, R. Dogaru. “Logo localization and recognition in natural images using homographic class graphs.” Machine Vision and Applications 27 (2), 2016. https://link.springer.com/article/10.1007/s00138-015-0741-7 

[4] R. Boia, C. Florea, L. Florea. “Elliptical asift agglomeration in class prototype for logo detection.” BMVC, 2015. http://citeseerx.ist.psu.edu/viewdoc/download;jsessionid=5C87F52DE38AB0C90F8340DFEBB841F7?doi=10.1.1.707.9371&rep=rep1&type=pdf 

[5] https://arxiv.org/abs/1804.02767 

[6] https://paddlepaddle.org.cn/modelbasedetail/yolov3 

[7] https://arxiv.org/abs/1512.03385 


Building a Video Analysis System with Milvus Vector Database


# Milvus To Participate In Hacktoberfest 2021

![hacktoberfestCover](https://assets.zilliz.com/Hacktoberfest_89c929cace.png "Cover image.")

**Milvus, the vector database empowering AI applications, announced today its participation in the 8th annual Hacktoberfest event sponsored by DigitalOcean!**

In 2014, DigitalOcean initiated [Hacktoberfest](https://hacktoberfest.digitalocean.com/) with the idea of celebrating open source software projects. Through the years, the event has steadily grown from tens to thousands of participants.  This, years the month-long event looks to grow to over 200,000 active participants!  

Starting October 1st, everyone from seasoned developers to students, code newbies and tech writers to UX designers, can contribute to open source communities and develop their skills, with the perks of winning limited edition swag and more.

## Everyone is welcome

There are a lot of ways to contribute to the Milvus project - you don't have to be a code superstar to contribute! In fact, you don't even need an extensive background or long list of skills in order to help out. Are you a great writer or editor, awesome!  Do you like to create graphics or illustrations -  perfect!  Here are a few ways you can make a contribution during the event, along with a list of GitHub "issues" for new contributors to tackle!

Check out the details [here](https://discuss.milvus.io/t/join-hacktoberfest-2021-with-us/72#how-to-participate-1).

## Contribute, get noticed & earn swag

![swag](https://assets.zilliz.com/swag_cae44023e8.png)

Certainly contributing to open source, or the concept of open source, is in esence an embodiment of altruism, but some cool swag doesn't hurt. We have up for grabs exclusive stickers, t-shirts, keyboards and a drone too! Check out the details on our [forum page](https://discuss.milvus.io/t/join-hacktoberfest-2021-with-us/72#prizes-8).

DigitalOcean also provides swag for participants, check out the rules from the official Hacktoberfest website.

## What's next

* We recently hosted a virtual Q&A call about Hacktoberfest.  You can view the recording [here](https://www.youtube.com/watch?v=cHjSTEHoiF8).

* To participate in the event, we recommend starting on our milvus.io Hacktoberfest page and checking out the ["Quick Start" Guide]( https://hacktoberfest.com/)
* Check out our [GitHub repo]( https://github.com/milvus-io) (and give us a star)!
* View our list of curated ["issues"](https://github.com/milvus-io/milvus/issues?q=is%3Aopen+is%3Aissue+label%3AHacktoberfest) for participants that can help you become a rockstar!

## Helpful links

* [**Milvus x Hactoberfest event page**](https://hacktoberfest.com/)

* [**Milvus Discussion Forum**](https://discuss.milvus.io/c/hacktoberfest/9) - Join our Milvus forum to stay up-to-date on Milvus releases, interact with other users, and ask questions during the event.

* **Follow us on [Twitter](https://twitter.com/milvusio)** and if your contribution is highlighted during the event you could earn special swag!


Milvus To Participate In Hacktoberfest 2021


# Combine AI Models for Image Search using ONNX and Milvus

Open Neural Network Exchange (ONNX) is an open format built to represent machine learning models. Since it was open-sourced in 2017, ONNX has developed into a standard for AI, providing building blocks for machine learning and deep learning models. ONNX defines a common file format to enable AI developers to use models with various frameworks, tools, runtimes, and compilers, and helps increase the speed of innovation in the artificial intelligence community.

Milvus is an open-source vector database that is highly flexible, reliable and blazing fast. It supports adding, deleting, updating and near real-time search of vector. Milvus has a comprehensive set of intuitive APIs, and support for multiple widely adopted index libraries (e.g. Faiss, NMSLIB, and Annoy), simplifying the index selection for a given scenario. Milvus is simple to use, and it has been used in hundreds of organizations and institutions worldwide, including image, audio and video search, recommendation, chatbot, new drug search, etc.

This article will introduce you how to use multiple models for image search based on ONNX and Milvus. It takes VGG16 and ResNet50 models as examples, uses ONNX to run different AI models to generate feature vectors, and finally performs feature vector retrieval in Milvus to return similar images. 

## Process Models with ONNX

The ONNX format can be easily exchanged between AI models. For example, the TensorFlow model can be converted to ONNX format and run in the Caffe environment. In this example, we convert the pre-trained ResNet50 model under the Keras framework to the ONNX format, and then call the VGG16 model in ONNX format to analyze different models.

```python
from keras.applications.resnet50 import ResNet50
import tensorflow as tf

# load keras-resnet50 model and save as a floder
model_resnet50 = ResNet50(include_top=False, pooling='max', weights='imagenet')
tf.saved_model.save(model_resnet50, "keras_resnet50_model")

# convert resnet50 model to onnx
! python -m tf2onnx.convert --saved-model "keras_resnet50_model" --output "onnx_resnet50.onnx"
```

Note: When we used the interface `keras2onnx.convert_keras(model, model.name)` to convert the model, it will return the error `AttributeError:'KerasTensor' object has no attribute'graph'`. Then we can use Python's Bash command to convert according to the solution on Stack Overflow.

## Extract Feature Vectors using Models

After converting ResNet50 model into ONNX format, you can extract the feature vector of the picture directly through the inference. Note: Feature vectors need to be normalized after extraction.

```python
# get the image vectors with onnx model
def get_onnx_vectors(onnx_model, img_path):
    img = image.load_img(img_path, target_size=(224, 224))
    x = image.img_to_array(img)
    x = np.expand_dims(x, axis=0)
    x = preprocess_input(x)
    
    sess = onnxruntime.InferenceSession(onnx_model)
    x = x if isinstance(x, list) else [x]
    feed = dict([(input.name, x[n]) for n, input in enumerate(sess.get_inputs())])
    feat = sess.run(None, feed)[0]
    
    norm_feat = feat[0] / LA.norm(feat[0])
    norm_feat = [i.item() for i in norm_feat]
    return norm_feat
```

Use ONNX-formatted VGG16 model to process image data:

```python
# generate vectors with ResNet50 and VGG16 ONNX model
2vec_resnet = get_onnx_vectors("onnx_resnet50.onnx", "./pic/example.jpg")
3vec_vgg = get_onnx_vectors("onnx_vgg16.onnx", "./pic/example.jpg")
```

## Store Vector Data

Unstructured data such as pictures cannot be directly processed by computer, but it can be converted into vectors through AI model and then analyzed by a computer. Milvus vector database is designed power massive unstructured data analysis. It can store vector data and perform near real-time analysis. First, create a collection of the corresponding model in Milvus, and then insert the image vectors.

```python
from milvus import *

# create collections in Milvus
milvus.create_collection(resnet_collection_param)
milvus.create_collection(vgg_collection_param)

# insert data to Milvus and return ids
status, resnet_ids = milvus.insert(resnet_collection_name, resnet_vectors)
status, vgg_ids = milvus.insert(vgg_collection_name, vgg_vectors)
```

After inserting the data successfully, Milvus will return the ID corresponding to the vector, and then we can find the picture by ID. Since Milvus 1.1 used in this case does not support scalar filtering (which Milvus 2.0 now supports), Redis is used to store the vector ID and the key-value of the image path.

```python
import redis
def img_ids_to_redis(img_directory, res_ids):
  for img, ids in zip(images, res_ids):
    redis.set(ids, img)
```

## Search for Similar Images

After storing the data, we can retrieve the vector. Milvus supports multiple distance calculation methods, including Euclidean, inner product and Hamming distance. The image similarity search in this article adopts the Euclidean distance calculation between the vectors in Milvus, returns the similar vector ID, and then finds the image corresponding to the ID in Redis.

```python
# search in Milvus and return the similarly results with ids
def search_in_milvus(collection_name, search_vector):
    status, results = milvus.search(collection_name, TOP_K, [search_vector])
    print(status)
    re_ids = [x.id for x in results[0]]
    re_distance = [x.distance for x in results[0]]
    return re_ids, re_distance
    
# get the images according the result ids
def get_sim_imgs(collection_name, search_vector):
    ids, distance = search_in_milvus(collection_name, search_vector)
    img = [red.get(i).decode("utf-8") for i in ids]
    return ids, distance, img
```

Taking the VGG16 and ResNet50 models as examples, this article shows processing multiple models through ONNX and combining multiple models with Milvus for similar vector retrieval to get similar images. The above two models are based on the Keras framework, which can quickly extract feature vectors. It can be seen from the Notebook that although the results of Milvus's search for pictures on the COCO dataset based on these two models are similar, their Euclidean distances are not the same. You can also try to compare the search results of the two models using other datasets.

Milvus is a high-performance, highly available vector database that can be used to process feature vectors generated from massive unstructured data. For more solutions, you can refer to [Milvus bootcamp](https://github.com/milvus-io/bootcamp).

## References

1. https://github.com/onnx/onnx
2. https://onnx.ai/
3. https://milvus.io/cn/
4. https://github.com/milvus-io/bootcamp



### About author

Shiyu Chen, a data engineer at Zilliz, graduated from Xidian University with a degree in Computer Science. Since joining Zilliz, she has been exploring solutions for Milvus in various fields, such as audio and video analysis, molecule formula retrieval, etc., which has greatly enriched the application scenarios of the community. She is currently exploring more interesting solutions. In her spare time, she loves sports and reading.


Combine AI Models for Image Search using ONNX and Milvus

DiskANN, A Disk-based ANNS Solution with High Recall and High QPS on Billion-scale Dataset

Implement Milvus CLI by Python Click

  
# DNA Sequence Classification based on Milvus
> Author:
Mengjia Gu, a data engineer at Zilliz, graduated from McGill University with a Master degree in Information Studies. Her interests include AI applications and similarity search with vector databases. As a community member of open-source project Milvus, she has provided and improved various solutions, like recommendation system and DNA sequence classification model. She enjoys challenges and never gives up!

# Introduction

DNA sequence is a popular concept in both academic research and practical applications, such as gene traceability, species identification, and disease diagnosis. Whereas all industries starve for a more intelligent and efficient research method, artificial intelligence has attracted much attention especially from biological and medical domain. More and more scientists and researchers are contributing to machine learning and deep learning in bioinformatics. To make experimental results more convincing, one common option is increasing sample size. The collaboration with big data in genomics as well brings more possibilities of use case in reality. However, the traditional sequence alignment has limitations, which make it [unsuitable for large data](https://www.frontiersin.org/articles/10.3389/fbioe.2020.01032/full#h5). In order to make less trade-off in reality, vectorization is a good choice for a large dataset of DNA sequences.

The open source vector database [Milvus](https://milvus.io/docs/v2.0.x/overview.md) is friendly for massive data. It is able to store vectors of nucleic acid sequences and perform high-efficiency retrieval. It can also help reduce the cost of production or research. The DNA sequence classification system based on Milvus only takes milliseconds to do gene classification. Moreover, it shows a higher accuracy than other common classifiers in machine learning.

# Data Processing

A gene that encodes genetic information is made up of a small section of DNA sequences, which consists of 4 nucleotide bases [A, C, G, T]. There are about 30,000 genes in human genome, nearly 3 billion DNA base pairs, and each base pair has 2 corresponding bases. To support diverse uses, DNA sequences can be classified into various categories. In order to reduce the cost and make easier use of data of long DNA sequnces, [k-mer ](https://en.wikipedia.org/wiki/K-mer#:~:text=Usually%2C%20the%20term%20k%2Dmer,total%20possible%20k%2Dmers%2C%20where)is introduced to data preprocessing. Meanwhile, it makes DNA sequence data more similar to plain text. Furthermore, vectorized data can speed up calculation in data analysis or machine learning.

![1.png](https://assets.zilliz.com/1_a7469e9eac.png "DNA sequencing.")

**k-mer**

The k-mer method is commonly used in DNA sequence preprocessing. It extracts a small section of length k starting from each base of the original sequence, thereby converting a long sequence of length s to (s-k+1) short sequences of length k. Adjusting the value of k will improve the model performance. Lists of short sequences are easier for data reading, feature extraction, and vectorization.

**Vectorization**

DNA sequences are vectorized in the form of text. A sequence transformed by k-mer becomes a list of short sequences, which looks like a list of individual words in a sentence. Therefore, most natural language processing models should work for DNA sequence data as well. Similar methodologies can be applied to model training, feature extraction, and encoding. Since each model has its own advantages and drawbacks, the selection of models depends on the feature of data and the purpose of research. For example, CountVectorizer, a bag-of-words model, implements feature extraction through straightforward tokenization. It sets no limit on data length, but the result returned is less obvious in terms of similarity comparison.

# Milvus Demo

Milvus can easily manage unstructured data and recall most similar results among trillions of vectors within an average delay of milliseconds. Its similarity search is based on Approximate Nearest Neighbor (ANN) search algorithm. These highlights make Milvus a great option to manage vectors of DNA sequences, hence promote the development and applications of bioinformatics.

Here is a demo showing how to build a DNA sequence classification system with Milvus. The [experimental dataset ](https://www.kaggle.com/nageshsingh/dna-sequence-dataset)includes 3 organisms and 7 gene families. All data are converted to lists of short sequences by k-mers. With a pre-trained CountVectorizer model, the system then encodes sequence data into vectors. The flow chart below depicts the system structure and the processes of inserting and searching.

![1.png](https://assets.zilliz.com/1_ebd89660f6.png "Workflow.")

Try out this demo at [Milvus bootcamp](https://github.com/milvus-io/bootcamp/tree/master/solutions/dna_sequence_classification).

In Milvus, the system creates collection and inserts corresponding vectors of DNA sequences into the collection (or partition if enabled). When receiving a query request, Milvus will return distances between the vector of input DNA sequence and most similar results in database. The class of input sequence and similarity between DNA sequences can be determined by vector distances in results.

```
# Insert vectors to Milvus collection (partition "human")
DNA_human = collection.insert([human_ids, human_vectors], partition_name='human')
# Search topK results (in partition "human") for test vectors
res = collection.search(test_vectors, "vector_field", search_params, limit=topK, partition_names=['human'])
for results in res:
    res_ids = results.ids # primary keys of topK results
    res_distances = results.distances # distances between topK results & search input
```

**DNA Sequence Classification**
Searching for most similar DNA sequences in Milvus could imply the gene family of an unknown sample, thus learn about its possible functionality.[ If a sequence is classified as GPCRs, then it probably has influence in body functions. ](https://www.nature.com/scitable/topicpage/gpcr-14047471/)In this demo, Milvus has successfully enabled the system to identify the gene families of the human DNA sequences searched with.

![3.png](https://assets.zilliz.com/3_1616da5bb0.png "Results.")
![4.png](https://assets.zilliz.com/4_d719b22fc7.png "Results.")

**Genetic Similarity**

Average DNA sequence similarity between organisms illustrates how close between their genomes. The demo searches in human data for most similar DNA sequences as that of chimpanzee and dog respectively. Then it calculates and compares average inner product distances (0.97 for chimpanzee and 0.70 for dog), which proves that chimpanzee shares more similar genes with human than dog shares. With more complex data and system design, Milvus is able to support genetic research even on a higher level.

```
search_params = {"metric_type": "IP", "params": {"nprobe": 20}}
```

**Performance**

The demo trains the classification model with 80% human sample data (3629 in total) and uses the remaining as test data. It compares performance of the DNA sequence classification model which uses Milvus with the one powered by Mysql and 5 popular machine learning classifiers. The model based on Milvus outperforms its counterparts in accuracy.

```
from sklearn.model_selection import train_test_split
X, y = human_sequence_kmers, human_labels
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
```

![1.png](https://assets.zilliz.com/1_6541a7dec6.png "Accuracy.")


# Further Exploration

With the development of big data technology, vectorization of DNA sequence will play a more important role in genetic research and practice. Combined with professional knowledge in bioinformatics, related studies can further benefit from the involvement of DNA sequence vectorization. Therefore, Milvus can present better results in practice. According to different scenarios and user needs, Milvus-powered similarity search and distance calculation show great potential and many possibilities.

- **Study unknown sequences**: [According to some researchers, vectorization can compress DNA sequence data.](https://iopscience.iop.org/article/10.1088/1742-6596/1453/1/012071/pdf) At the same time, it requires less effort to study structure, function, and evolution of unknown DNA sequences. Milvus can store and retrieve a huge number of DNA sequence vectors without losing accuracy.
- **Adapt devices**: Limited by traditional algorithms of sequence alignment, similarity search can barely benefit from device ([CPU](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7884812/)/[GPU](https://mjeer.journals.ekb.eg/article_146090.html)) improvement. Milvus, which supports both regular CPU computation and GPU acceleration, resolves this problem with approximate nearest neighbor algorithm.
- **Detect virus & trace origins**: [Scientists have compared genome sequences and reported that COVID19 virus of probable bat origin belongs to SARS-COV](https://www.nature.com/articles/s41586-020-2012-7?fbclid=IwAR2hxnXb9nLWgA8xexEoNrCNH8WHqvHhhbN38aSm48AaH6fTzGMB1BLljf4). Based on this conclusion, researchers can expand sample size for more evidence and patterns.
- **Diagnose diseases**: Clinically, doctors could compare DNA sequences between patients and healthy group to identify variant genes that cause diseases. It is possible to extract features and encode these data using proper algorithms. Milvus is able to return distances between vectors, which can be related to disease data. In addition to assisting diagnosis of disease, this application can also help to inspire the study of [targeted therapy](https://www.frontiersin.org/articles/10.3389/fgene.2021.680117/full).


# Learn more about Milvus
Milvus is a powerful tool capable of powering a vast array of artificial intelligence and vector similarity search applications. To learn more about the project, check out the following resources:
- Read our [blog](https://milvus.io/blog).
- Interact with our open-source community on [Slack](https://milvusio.slack.com/join/shared_invite/zt-e0u4qu3k-bI2GDNys3ZqX1YCJ9OM~GQ#/shared-invite/email).
- Use or contribute to the world’s most popular vector database on [GitHub](https://github.com/milvus-io/milvus/).
- Quickly test and deploy AI applications with our new [bootcamp](https://github.com/milvus-io/bootcamp).
  


DNA Sequence Classification based on Milvus


# Paper Reading ｜ HM-ANN: When ANNS Meets Heterogeneous Memory

[HM-ANN: Efficient Billion-Point Nearest Neighbor Search on Heterogenous Memory](https://proceedings.neurips.cc/paper/2020/file/788d986905533aba051261497ecffcbb-Paper.pdf) is a research paper that was accepted at the 2020 Conference on Neural Information Processing Systems ([NeurIPS 2020](https://nips.cc/Conferences/2020)). In this paper, a novel algorithm for graph-based similarity search, called HM-ANN, is proposed. This algorithm considers both memory heterogeneity and data heterogeneity in a modern hardware setting. HM-ANN enables billion-scale similarity search on a single machine without compression technologies. Heterogeneous memory (HM) represents the combination of fast but small dynamic random-access memory (DRAM) and slow but large persistent memory (PMem). HM-ANN achieves low search latency and high search accuracy, especially when the dataset cannot fit into DRAM. The algorithm has a distinct advantage over the state-of-art approximate nearest neighbor (ANN) search solutions.

# Motivation

Since their inception, ANN search algorithms have posed a fundamental tradeoff between query accuracy and query latency due to the limited DRAM capacity. To store indexes in DRAM for fast query access, it is necessary to limit the number of data points or store compressed vectors, both of which hurt search accuracy. Graph-based indexes (e.g. Hierarchical Navigable Small World, HNSW) have superior query runtime performance and query accuracy. However, these indexes can also consume 1-TiB-level DRAM when operating on billion-scale datasets.

There are other workarounds to avoid letting DRAM store billion-scale datasets in raw format. When a dataset is too large to fit into memory on a single machine, compressed approaches such as product quantization of the dataset’s points are used. But the recall of those indexes with the compressed dataset is normally low because of the loss of precision during quantization. Subramanya et al. [1] explore leveraging solid-state drive (SSD) to achieve billion-scale ANN search using a single machine with an approach called Disk-ANN, where the raw dataset is stored on SSD and the compressed representation on DRAM.

# Introduction to Heterogeneous Memory

![1.png](https://assets.zilliz.com/image_32_d26cfa9480.png "Name of the pictureMemory/Storage Hierarchy with HM. Source: [http://nvmw.ucsd.edu/nvmw2021-program/nvmw2021-data/nvmw2021-paper63-presentation_slides.pdf](http://nvmw.ucsd.edu/nvmw2021-program/nvmw2021-data/nvmw2021-paper63-presentation_slides.pdf)")


Heterogeneous memory (HM) represents the combination of fast but small DRAM and slow but large PMem. DRAM is normal hardware that can be found in every modern server, and its access is relatively fast. New PMem technologies, such as Intel® Optane™ DC Persistent Memory Modules, bridge the gap between NAND-based flash (SSD) and DRAM, eliminating the I/O bottleneck. PMem is durable like SSD, and directly addressable by the CPU, like memory. Renen et al. [2] discover that the PMem read bandwidth is 2.6× lower, and the write bandwidth 7.5× lower, than DRAM in the configured experiment environment.

# HM-ANN Design

HM-ANN is an accurate and fast billion-scale ANN search algorithm that runs on a single machine without compression. The design of HM-ANN generalizes the idea of HNSW, whose hierarchical structure naturally fits into HM. HNSW consists of multiple layers—only layer 0 contains the whole dataset, and each remaining layer contains a subset of elements from the layer directly underneath it.

![2.png](https://assets.zilliz.com/2_25a1836e8b.png "An example of HNSW with 3 layers. Source: [https://arxiv.org/pdf/1603.09320.pdf](https://arxiv.org/pdf/1603.09320.pdf)" )


- Elements in the upper layers, which include only subsets of the dataset, consume a small portion of the whole storage. This observation makes them decent candidates to be placed in DRAM. In this way, the majority of searches on HM-ANN are expected to happen in the upper layers, which maximizes the utilization of the fast access characteristic of DRAM. However, in the cases of HNSW, most searches happen in the bottom layer.
- The bottom-most layer carries the whole dataset, which makes it suitable to be placed in PMem. Since accessing layer 0 is slower, it is preferable to have only a small portion accessed by each query and the access frequency reduced.

## Graph Construction Algorithm

![3.png](https://assets.zilliz.com/3_dd9627c753.png "An example of graph construction of HM-ANN. Source: [http://nvmw.ucsd.edu/nvmw2021-program/nvmw2021-data/nvmw2021-paper63-poster.pdf](http://nvmw.ucsd.edu/nvmw2021-program/nvmw2021-data/nvmw2021-paper63-poster.pdf)")



The key idea of HM-ANN’s construction is to create high-quality upper layers, in order to provide better navigation for search at layer 0. Thus most memory access happens in DRAM, and access in PMem is reduced. To make this possible, the construction algorithm of HM-ANN has a top-down insertion phase and a bottom-up promotion phase.

The top-down insertion phase builds a navigable small-world graph as the bottom-most layer is placed on the PMem.

The bottom-up promotion phase promotes pivot points from the bottom layer to form upper layers that are placed on DRAM without losing much accuracy. If a high-quality projection of elements from layer 0 is created in layer 1, search in layer 0 finds the accurate nearest neighbors of the query with only a few hops.

- Instead of using HNSW’s random selection for promotion, HM-ANN uses a high-degree promotion strategy to promote elements with the highest degree in layer 0 into layer 1. For higher layers, HM-ANN promotes high-degree nodes to the upper layer based on a promotion rate.
- HM-ANN promotes more nodes from layer 0 to layer 1 and sets a larger maximum number of neighbors for each element in layer 1. The number of nodes in the upper layers is decided by the available DRAM space. Since layer 0 is not stored in DRAM, making each layer stored in DRAM denser increases the search quality.

## Graph Seach Algorithm

![4.png](https://assets.zilliz.com/4_a5a7f29c93.png "An example of graph seach of HM-ANN. Source: [http://nvmw.ucsd.edu/nvmw2021-program/nvmw2021-data/nvmw2021-paper63-poster.pdf](http://nvmw.ucsd.edu/nvmw2021-program/nvmw2021-data/nvmw2021-paper63-poster.pdf)")


The search algorithm consists of two phases: fast memory search and parallel layer-0 search with prefetching.

### Fast memory search

As the same as in HNSW, the search in DRAM begins at the entry point in the very top layer and then performs 1-greedy search from top to layer 2. To narrow down the search space in layer 0, HM-ANN performs the search in layer 1 with a search budget with `efSearchL1`, which limits the size of the candidate list in layer 1. Those candidates of the list are used as multiple entry points for search in layer 0, to enhance search quality in layer 0. While HNSW using only one entry point, the gap between layer 0 and layer 1 is more specially handled in HM-ANN than gaps between any other two layers.

### Parallel layer-0 search with prefetching

In the bottom layer, HM-ANN evenly partitions the aforementioned candidates from searching layer 1 and sees them as entry points to perform a parallel multi-start 1-greedy search with threads. The top candidates from each search are collected to find the best candidates. As known, going down from layer 1 to layer 0 is exactly going to PMem. Parallel search hides the latency of PMem and makes the best use of memory bandwidth, to improve search quality without increasing search time.

HM-ANN implements a software-managed buffer in DRAM to prefetch data from PMem before the memory access happens. When searching layer 1, HM-ANN asynchronously copies neighbor elements of those candidates in `efSearchL1` and the neighbor elements’ connections in layer 1 from PMem to the buffer. When the search in layer 0 happens, a portion of to-be-accessed data is already prefetched in DRAM, which hides the latency to access PMem and leads to shorter query time. It matches the design goal of HM-ANN, where most memory accesses happen in DRAM and memory accesses in PMem are reduced.

# Evaluation

In this paper, an extensive evaluation is conducted. All experiments are done on a machine with Intel Xeon Gold 6252 CPU@2.3GHz. It uses DDR4 (96GB) as fast memory and Optane DC PMM (1.5TB) as slow memory. Five datasets are evaluated: BIGANN, DEEP1B, SIFT1M, DEEP1M, and GIST1M. For billion-scale tests, the following schemes are included: billion-scale quantization-based methods (IMI+OPQ and L&C), the non-compression-based methods (HNSW and NSG).

## Billion-scale algorithm comparison

![5.png](https://assets.zilliz.com/5_4297db66a9.png "Table 1.")

In table 1, the build time and storage of different graph-based indexes are compared. HNSW takes the shortest build time and HM-ANN needs 8% additional time than HNSW. In terms of whole storage usage, HM-ANN indexes are 5–13% larger than HSNW, because it promotes more nodes from layer 0 to layer 1.

![6.png](https://assets.zilliz.com/6_f363e64d3f.png "Figure 1.")

In Figure 1, the query performance of different indexes is analyzed. Figure 1 (a) and (b) show that HM-ANN achieves the top-1 recall of > 95% within 1ms. Figures 1 (c) and (d) show that HM-ANN obtains top-100 recall of > 90% within 4 ms. HM-ANN provides the best latency-vs-recall performance than all other approaches.

## Million-scale algorithm comparison

![7.png](https://assets.zilliz.com/7_a5c23de240.png "Figure 2.")

In Figure 2, the query performance of different indexes is analyzed in a pure DRAM setting. HNSW, NSG, and HM-ANN are evaluated with the three million-scale datasets fitting in DRAM. HM-ANN still achieves better query performance than HNSW. The reason is that the total number of distance computations from HM-ANN is lower (on average 850/query) than that of HNSW (on average 900/query) to achieve 99% recall target.

![8.png](https://assets.zilliz.com/image_33_f99d31f322.png "Figure 3.")

## Effectiveness of high-degree promotion

In Figure 3, the random promotion and high-degree promotion strategies are compared in the same configuration. The high-degree promotion outperforms the baseline. The high-degree promotion performs 1.8x, 4.3x, and 3.9x faster than the random promotion to reach 95%, 99%, and 99.5% recall targets, respectively.

![10.png](https://assets.zilliz.com/image_34_3af47e0842.png "Figure 5.")

## Performance benefit of memory management techniques

Figure 5 contains a series of steps between HNSW and HM-ANN to show how each optimization of HM-ANN contributes to its improvements. BP stands for the Bottom-up Promotion while building index. PL0 represents for Parallel layer-0 search, while DP for data prefetching from PMem to DRAM. Step by step, HM-ANN’s search performance is pushed further.

# Conclusion

A new graph-based indexing and search algorithm, called HM-ANN, maps the hierarchical design of the graph-based ANNs with memory heterogeneity in HM. Evaluations show that HM-ANN belongs to the new state-of-the-art indexes in billion point datasets.

We notice a trend in academia as well as in industry, where building indexes on persistent storage devices is focused on. To offload the pressure of DRAM, Disk-ANN [1] is an index built on SSD, whose throughput is significantly lower than PMem. However, the building of HM-ANN still takes few days, where no huge differences compared with Disk-ANN is established. We believe it is possible to optimize the build time of HM-ANN, when we utilize PMem’s characteristics more carefully, e.g. to be aware of PMem's granularity (256 Bytes) and to use streaming instruction to bypass cachelines. We also believe there would be more approaches with durable storage devices are proposed in the future.

# Reference

[1]: Suhas Jayaram Subramanya and Devvrit and Rohan Kadekodi and Ravishankar Krishaswamy and Ravishankar Krishaswamy: DiskANN: Fast Accurate Billion-point Nearest Neighbor Search on a Single Node, NIPS, 2019

[DiskANN: Fast Accurate Billion-point Nearest Neighbor Search on a Single Node - Microsoft Research](https://www.microsoft.com/en-us/research/publication/diskann-fast-accurate-billion-point-nearest-neighbor-search-on-a-single-node/)

[DiskANN: Fast Accurate Billion-point Nearest Neighbor Search on a Single Node](https://papers.nips.cc/paper/2019/hash/09853c7fb1d3f8ee67a61b6bf4a7f8e6-Abstract.html)

[2]: Alexander van Renen and Lukas Vogel and Viktor Leis and Thomas Neumann and Alfons Kemper: Persistent Memory I/O Primitives, CoRR & DaMoN, 2019

[https://dl.acm.org/doi/abs/10.1145/3329785.3329930](https://dl.acm.org/doi/abs/10.1145/3329785.3329930)

[Persistent Memory I/O Primitives](https://arxiv.org/abs/1904.01614)


Paper Reading｜HM-ANN When ANNS Meets Heterogeneous Memory


# Milvus 2.0: Redefining Vector Database

It was only like yesterday when we put down the first line of code for Milvus in October 2018. In March 2021, after 19 iterations tested by 1,000+ users around the world, we launched Milvus 1.0, our first offical release with long-term support. As the world's most popular open-source vector database, Milvus 1.0 managed to solve some fundamental issues in vector management, such as CRUD operations and data persistence. However, as new scenarios and requirements emerge, we began to realize that there are so many more issues yet to resolve. This article offers a recap on the observations we made in the past three years, the challenges that Milvus 2.0 is expected to address, and why Milvus 2.0 is deemed a better solution to such challenges.
To learn more about what Milvus 2.0 has to offer, check out the [Milvus 2.0 Release Notes](https://milvus.io/docs/v2.0.x/release_notes.md).

## Challenges Milvus 1.x is faced with

**Data silo:** Milvus 1.0 is only capable of handling vector embeddings generated from unstructured data, and gives little support for scalar query. The disaggregation of data storage in its design results in duplicate data and adds to the complexity of application development, and hybrid search between vector and scalar data is unsatisfactory due to the lack of a unified optimizer.

**Dilemma between timeliness and efficiency:** Milvus 1.0 is a near real-time system, which relies on regular or force flush to ensure data visibility. This approach adds to the complexity and uncertainty in stream data processing at a number of levels. Besides, although this batch insertion approach is said to improve processing efficiency, it still consumes plenty of resources. So bulkload approach is needed.

**Lacking scalability & elasticity:** Milvus 1.0 relies on Mishards, a sharding middleware solution, to achieve scability, and network-attached storage (NAS) for data persistence. This classical architecture built upon shared storage does not contribute much to the overall scalability for the following reasons:

1. Only one write node is supported in Mishards and cannot be scaled.
2. The scaling of the read nodes in Mishards is implemented using consistent hash-based routing. Although consistent hashing is easy to implement and helps solve the issue of data distribution uniformity, it is not flexible enough in data scheduling and falls short of addressing the mismatch between data size and computational power.
3. Milvus 1.0 relies on MySQL to manage metadata, but the queries and dataset size that a standalone MySQL server is capable of handling is fairly limited.

**Lacking high availability:** One observation we've made is that most of Milvus' users tend to favor availability over consistency, whilst Milvus 1.x lacks capacities such as in-memory replicas and disaster recovery and is not quite up to par in terms of high availability. Therefore, we are exploring the possibility of sacrificing a certain degree of accuracy to achieve higher availability.

**Prohibitively high costs:** Milvus 1.0 relies on NAS for data persistence, the cost of which is usually tenfold that of a local or object storage. Since vector search relies heavily on computing resources and memory, the high costs it incurs could well become a hurdle to further exploration in large-scale datasets or complex business scenarios.

**Unintuitive user experience:**

1. Complicated distributed deployment incurs high operational costs.
2. A well-designed graphical user interface (GUI) is not available.
3. Unintuitive APIs have become a drag on the development of applications.

Whether to move on from patch or to start from scratch is a big question. Charles Xie, the father of Milvus, believes that, just as many traditional automakers could never progressively turn Tesla, Milvus has to become a game changer in the field of unstructured data processing and analytics in order to thrive. It is this conviction that spurred us to kick start Milvus 2.0, a refactored cloud-native vector database.

## The Making of Milvus 2.0

### Design principles

As our next-generation cloud-native vector database, Milvus 2.0 is built around the following three principles:

**Cloud-native first:** We believe that only architectures supporting storage and computing separation can scale on demand and take the full advantage of cloud's elasticity. We'd also like to bring your attention to the microservice design of Milvus 2.0, which features read and write separation, incremental and historical data separation, and CPU-intensive, memory-intensive, and IO-intensive task separation. Microservices help optimize allocation of resources for the ever-changing heterogeneous workload.

**Logs as data:** In Milvus 2.0, the log broker serves as the system' backbone: All data insert and update operations must go through the log broker, and worker nodes execute CRUD operations by subscribing to and consuming logs. This design reduces system complexity by moving core functions such as data persistence and flashback down to the storage layer, and log pub-sub make the system even more flexible and better positioned for future scaling.

**Unified batch and stream processing:** Milvus 2.0 implements the unified Lambda architecture, which integrates the processing of the incremental and historical data. Compared with the Kappa architecture, Milvus 2.0 introduces log backfill, which stores log snapshots and indexes in the object storage to improve failure recovery efficiency and query performance. To break unbounded (stream) data down into bounded windows, Milvus embraces a new watermark mechanism, which slices the stream data into multiple message packs according to write time or event time, and maintains a timeline for users to query by time.

![2.0 image 1.png](https://assets.zilliz.com/2_0_image_1_59d7316324.png "Unified Lambda architecture.")



### System architecture

As mentioned above, the design of Milvus 2.0 strictly follows the principles of storage and computing separation and control and data plane separation. The system breaks down into four layers: access layer, coordinator service, worker nodes, and storage.

**Access layer:** The interface: The access layer is the front layer of the system and endpoint to users. It is in charge of forwarding requests and gathering results.

**Coordinator service:** The coordinator service assigns tasks to the worker nodes and functions as the system's brain. There are four coordinator types: root coordinator (root coord), data coordinator (data coord), query coordinator (query coord), and index coordinator (index coord).

**Worker nodes:** The arms and legs. Worker nodes are dumb executors that follow the instructions from the coordinator service and respond to the read/write requests from the access layer. There are three types of worker nodes: data nodes, query nodes, and index nodes.

**Storage:** The bones. Storage has three types: meta storage, log broker, and object storage.

- Implemented by etcd, meta storage is used to store metadata such as collection and checkpoint for the coordinator service.
- Implemented by Pulsar, log broker is used mainly to store incremental logs and implement reliable asynchronous notifications.
- Implemented by MinIO or S3, object storage is used mainly to store log snapshots and index files.

The following is the system architecture diagram of Milvus 2.0:
![2.0 image 2.png](https://assets.zilliz.com/2_0_image_2_2cc7d7fc9c.png "System architecture diagram.")



### Key features

The costs of running a database involve not only runtime resource consumption, but also the potential learning costs and the operational and maintenance costs. Practically speaking, the more user-friendly a database is, the more likely it is going to save such potential costs. From Milvus' calendar day one, ease of use is always put on the top of our list, and the latest Milvus 2.0 has quite a few to offer in the way of reducing such costs.

#### Always online

Data reliability and service sustainability are the basic requirements for a database, and our strategy is "fail cheap, fail small, and fail often".

- "Fail cheap" refers to storage and computing separation, which makes the handling of node failure recovery straightforward and at a low cost.
- "Fail small" refers to the "divide and conquer" strategy, which simplifies the design complexity by having each coordinator service handle only a small portion of read/write/incremental/historical data.
- "Fail often" refers to the introduction of chaos testing, which uses fault injection in a testing environment to simulate situations such as hardware failures and dependency failures and accelerate bug discovery.

#### Hybrid search between scalar and vector data

To leverage synergy between structured and unstructured data, Milvus 2.0 supports both scalar and vector data and enables hybrid search between them. Hybrid search helps users find the approximate nearest neighbors that match a filter criteria. Currently, Milvus supports relational operations such as EQUAL, GREATER THAN, and LESS THAN, and logical operations such as NOT, AND, OR, and IN.

#### Tunable consistency

As a distributed database abiding by the PACELC theorem, Milvus 2.0 has to trade off between consistency and availability & latency. In most scenarios, overemphasizing data consistency in production can overkill because allowing a small portion of data to be invisible has little impact on the overall recall but can significantly improve the query performance. Still, we believe that consistency levels, such as _strong, bounded staleness, and session_, have their own unique application. Therefore, Milvus supports tunable consistency at the request level. Taking testing as an example, users may require _strong_ consistence to ensure test results are absolutely correct.

#### Time travel

Data engineers often need to do data rollback to fix dirty data and code bugs. Traditional databases usually implement data rollback through snapshots or even data retrain. This could bring excessive overhead and maintenance costs. Milvus maintains a timeline for all data insert and delete operations, and users can specify a timestamp in a query to retrieve a data view at a specified point in time. With time travel, Milvus can also implement a lightweight data backup or data clone.

#### ORM Python SDK

Object relational mapping (ORM) allows users to focus more on the upper-level business model than on the underlying data model, making it easier for developers to manage relations between collections, fields, and programs. To close the gap between proof of concept (PoC) for AI algorithms and production deployment, we engineered PyMilvus ORM APIs, which can work with an embedded library, a standalone deployment, a distributed cluster , or even a cloud service. With a unified set of APIs, we provide users with a consistent user experience and reduce code migration or adaptation costs.

![2.0 image 3.png](https://assets.zilliz.com/2_0_image_3_55394d6e80.png "Object relational mapping (ORM).")



#### Supporting tools

- Milvus Insight is Milvus's graphical user interface offering practical functionalities such as cluster state management, meta management, and data query. The source code of Milvus Insight will also be open sourced as an independent project. We are looking for more contributors to join this effort.
- Out-of-box experience (OOBE), faster deployment: Milvus 2.0 can be deployed using helm or docker-compose.
- Milvus 2.0 uses Prometheus, an open-source time-series database, to store performance and monitor data, and Grafana, an open observability platform, for metrics visualization.

## Looking to the future

In retrospect, we believe the system architecture based on big data + AI application is overly complicated. The top priority of the Milvus community has always been to make Milvus easier to use. Moving forward, the Milvus project will focus on the following areas:

**DB for AI:** Besides the basic CRUD functions, Milvus, as a database system, needs to have a smarter query optimizer, more powerful data query capabilities, and more comprehensive data management functions. Our work for the next stage will be focusing on the data manipulation language (DML) functions and data types not yet available in Milvus 2.0, including adding delete and update operations and supporting string data types.

**AI for DB:** Knob tuning of parameters such as index types, system configurations, user workload, and hardware types complicates the use of Milvus and should be avoided as much as possible. We've set about analyzing system load and gathering access frequency of the data, and plan on introducing auto-tuning in the future to reduce learning costs.

**Cost optimization:** The biggest challenge for vector retrieval is the need to process massive-scale datasets within a limited period of time. This is both CPU-intensive and memory-intensive. Introducing GPU and FPGA heterogeneous hardware acceleration at the physical layer can greatly reduce CPU overhead. We are also developing a hybrid on-disk and in-memory ANN indexing algorithms to realize high-performance queries on massive datasets with limited memory. What's more, we are evaluating the performance of existing open-source vector indexing algorithms such as ScaNN and NGT.

**Ease of use:** Milvus keeps improving its usability by providing cluster management tools, SDKs in multiple languages, deployment tools, operational tools, and more.

To learn more about Milvus' release plans, check out the [Milvus Roadmap](https://milvus.io/docs/v2.0.x/roadmap.md).

Kudos to all Milvus community contributors, without whom Milvus 2.0 would not have been possible. Feel free to [submit an issue](https://github.com/milvus-io/milvus) or [contribute your code](https://github.com/milvus-io/milvus) to the Milvus community!

<br/>

_About the author_

_Xiaofan Luan is now working at Zilliz as Director of Engineering managing R&D of the Milvus project. He has 7 years' work experience focusing on building database/storage systems. After graduating from Cornell University, he worked at Oracle, HEDVIG, and Alibaba Cloud consecutively._


Milvus 2.0 Redefining Vector Database

  
# Building a Personalized Product Recommender System with Vipshop and Milvus
With the explosive growth of Internet data scale, the product quantity as well as category in the current mainstream e-commerce platform increase on the one hand, the difficulty for users to find the products they need surges on the other hand.

[Vipshop](https://www.vip.com/) is a leading online discount retailer for brands in China. The Company offers high-quality and popular branded products to consumers throughout China at a significant discount from retail prices. To optimize the shopping experience for their customers, the company decided to build a personalized search recommendation system based on user query keywords and user portraits.

The core function of the e-commerce search recommendation system is to retrieve suitable products from a large number of products and display them to users according to their search intent and preference. In this process, the system needs to calculate the similarity between products and users' search intent & preference, and recommends the TopK products with the highest similarity to users.

Data such as product information, user search intent, and user preferences are all unstructured data. We tried to calculate the similarity of such data using CosineSimilarity(7.x) of the search engine Elasticsearch (ES), but this approach has the following drawbacks.

- Long computational response time - the average latency to retrieve TopK results from millions of items is around 300 ms.

- High maintenance cost of ES indexes - the same set of indexes is used for both commodity feature vectors and other related data, which hardly facilitates the index construction, but produces a massive amount of data.

We tried to develop our own locally sensitive hash plug-in to accelerate the CosineSimilarity calculation of ES. Although the performance and throughput were significantly improved after the acceleration, the latency of 100+ ms was still difficult to meet the actual online product retrieval requirements.

After a thorough research, we decided to use Milvus, an open source vector database, which is advantaged with the support for distributed deployment, multi-language SDKs, read/write separation, etc. compared to the commonly used standalone Faiss.

Using various deep learning models, we convert massive unstructured data into feature vectors, and import the vectors into Milvus. With the excellent performance of Milvus, our e-commerce search recommendation system can efficiently query the TopK vectors that are similar to the target vectors.

## Overall Architecture
![Architecture](https://assets.zilliz.com/1_01551e7b2b.jpg "Architecture.)
As shown in the diagram, the system overall architecture consists of two main parts.

- Write process: the item feature vectors (hereinafter referred to as item vectors) generated by the deep learning model are normalized and written into MySQL. MySQL then reads the processed item feature vectors using the data synchronization tool (ETL) and import them into the vector database Milvus.

- Read process: The search service obtains user preference feature vectors (hereinafter referred to as user vectors) based on user query keywords and user portraits, queries similar vectors in Milvus and recalls TopK item vectors.

Milvus supports both incremental data update and entire data update. Each incremental update has to delete the existing item vector and insert a new item vector, meaning that every newly updated collection will be re-indexed. It better suits the scenario with more reads and fewer writes. Therefore, we choose the entire data update method. Moreover, it takes only a few minutes to write the entire data in batches of multiple partitions, which is equivalent to near real-time updates.

Milvus write nodes perform all write operations, including creating data collections, building indexes, inserting vectors, etc., and provide services to the public with write domain names. Milvus read nodes perform all read operations and provide services to the public with read-only domain names.

Whereas the current version of Milvus does not support switching collection aliases, we introduce Redis to seamlessly switch aliases between multiple entire data collections.

The read node only needs to read existing metadata information and vector data or indexes from MySQL, Milvus, and GlusterFS distributed file system, so the read capability can be horizontally extended by deploying multiple instances.

## Implementation Details

### Data Update

The data update service includes not only writing vector data, but also data volume detection of vectors, index construction, index pre-loading, alias control, etc. The overall process is as follows.
![Process](https://assets.zilliz.com/2_6052b01334.jpg "Process.")
1. Assume that before building the entire data, CollectionA provides data service to the public, and the entire data being used is directed to CollectionA (`redis key1 = CollectionA`). The purpose of constructing entire data is to create a new collection CollectionB.

2. Commodity data check - check the item number of commodity data in the MySQL table, compare the commodity data with the existing data in CollectionA. Alert can be set in accordance with quantity or percentage. If the set quantity (percentage) is not reached, the entire data will not be built, and it will be regarded as the failure of this building operation, triggering the alert; once the set quantity (percentage) is reached, the entire data building process starts.

3. Start building the entire data - initialize the alias of the entire data being built, and update Redis. After updating, the alias of the entire data being built is directed to CollectionB (`redis key2 = CollectionB`).

4. Create a new entire collection - determine if CollectionB exists. If it does, delete it before creating a new one.

5. Data batch write-in - calculate the partition ID of each commodity data with its own ID using modulo operation, and write the data to multiple partitions to the newly created collection in batches.

6. Build and pre-load index - Create index (`createIndex()`) for the new collection. The index file is stored in distributed storage server GlusterFS. The system automatically simulates query on the new collection and pre-load the index for query warm-up. 

7. Collection data check - check the item number of data in the new collection, compare the data with the existing collection, and set alarms based on the quantity and percentage. If the set number (percentage) is not reached, the collection will not be switched and the building process will be regarded as a failure, triggering the alert.

8. Switching collection - Alias control. After updating Redis, the entire data alias being used is directed to CollectionB (`redis key1 = CollectionB`), the original Redis key2 is deleted, and the building process is completed.

### Data Recall

The Milvus partition data is called several times to calculate the similarity between user vectors, obtained based on user query keywords and user portrait, and item vector, and the TopK item vectors are returned after merging. The overall workflow schematic is as follow:
![workflow](https://assets.zilliz.com/3_93518602b1.jpg "Workflow.")
The following table lists the main services involved in this process. It can be seen that the average latency for recalling TopK vectors is about 30 ms.

| **Service**              | **Role**                                                | **Input Parameters**                                         | **Output parameters** | **Response latency** |
| ------------------------ | ------------------------------------------------------- | ------------------------------------------------------------ | --------------------- | -------------------- |
| User vectors acquisition | Obtain user vector                                      | user info + query                                            | user vector           | 10 ms                |
| Milvus Search            | Calculate the vector similarity and return TopK results | user vector                                                  | item vector           | 10 ms                |
| Scheduling Logic         | Concurrent result recalling and merging                 | Multi-channel recalled item vectors and the similarity score | TopK items            | 10 ms                |

**Implementation process:**

1. Based on the user query keywords and user portrait, the user vector is calculated by the deep learning model.
2. Obtain the collection alias of the entire data being used from Redis currentInUseKeyRef and get Milvus CollectionName. This process is data synchronization service, i.e. switching alias to Redis after entire data update.
3. Milvus is called concurrently and asynchronously with the user vector to obtain data from different partitions of the same collection, and Milvus calculates the similarity between the user vector and the item vector, and returns the TopK similar item vectors in each partition.
4. Merge the TopK item vectors returned from each partition, and rank the results in the reverse order of similarity distance, which are calculated using the IP inner product (the greater the distance between the vectors, the more similar they are). The final TopK item vectors are returned.

## Looking Ahead

At present, Milvus-based vector search can be used steadily in the search of recommendation scenarios, and its high performance gives us more room to play in the dimensionality of the model and algorithm selection.

Milvus will play a crucial role as the middleware for more scenarios, including recall of main site search and all-scenario recommendations.

The three most anticipated features of Milvus in the future are as follows.

- Logic for collection alias switching - coordinate the switching across collections without external conponents.
- Filtering mechanism - Milvus v0.11.0 only supports ES DSL filtering mechanism in standalone version. The newly released Milvus 2.0 supports scalar filtering, and read/write separation.
- Storage support for Hadoop Distributed File System (HDFS) - The Milvus v0.10.6 we are using only supports POSIX file interface, and we have deployed GlusterFS with FUSE support as the storage backend. However, HDFS is a better choice in terms of performance and ease of scaling.

## Lessons Learned and Best Practices

1. For applications where read operations are the primary focus, a read-write separation deployment can significantly increase the processing power and improve performance.
2. The Milvus Java client lacks a reconnection mechanism because the Milvus client used by the recall service is resident in memory. We have to build our own connection pool to ensure the availability of the connection between the Java client and the server through heartbeat test.
3. Slow queries occur occasionally on Milvus. This is due to insufficient warm-up of the new collection. By simulating the query on the new collection, the index file is loaded into the memory to achieve the  index warm-up.
4. nlist is the index building parameter and nprobe is the query parameter. You need to get a reasonable threshold value according to your business scenario through pressure testing experiments to balance the retrieval performance and accuracy.
5. For static data scenario, it is more efficient to import all data into the collection first and build indexes later. 
  

Building a Personalized Product Recommender System with Vipshop and Milvus

  
# Audio Retrieval Based on Milvus
Sound is an information dense data type. Although it may feel antiquated in the era of video content, audio remains a primary information source for many people. Despite long-term decline in listeners, 83% of Americans ages 12 or older listened to terrestrial (AM/FM) radio in a given week in 2020 (down from 89% in 2019). Conversely, online audio has seen a steady rise in listeners over the past two decades, with 62% of Americans reportedly listening to some form of it on a weekly basis according to the same [Pew Research Center study](https://www.journalism.org/fact-sheet/audio-and-podcasting/).

As a wave, sound includes four properties: frequency, amplitude, wave form, and duration. In musical terminology, these are called pitch, dynamic, tone, and duration. Sounds also help humans and other animals perceive and understand our environment, providing context clues for the location and movement of objects in our surroundings.

As an information carrier, audio can be classified into three categories:

1. **Speech:** A communication medium composed of words and grammar. With speech recognition algorithms, speech can be converted to text.
2. **Music:** Vocal and/or instrumental sounds combined to produce a composition comprised of melody, harmony, rhythm, and timbre. Music can be represented by a score.
3. **Waveform:** A digital audio signal obtained by digitizing analog sounds. Waveforms can represent speech, music, and natural or synthesized sounds.

Audio retrieval can be used to search and monitor online media in real-time to crack down on infringement of intellectual property rights. It also assumes an important role in the classification and statistical analysis of audio data.

## Processing Technologies

Speech, music, and other generic sounds each have unique characteristics and demand different processing methods. Typically, audio is separated into groups that contain speech and groups that do not:

- Speech audio is processed by automatic speech recognition.
- Non-speech audio, including musical audio, sound effects, and digitized speech signals, are processed using audio retrieval systems. 

This article focuses on how to use an audio retrieval system to process non-speech audio data. Speech recognition is not covered in this article

### Audio feature extraction

Feature extraction is the most important technology in audio retrieval systems as it enables audio similarity search. Methods for extracting audio features are divided into two categories: 

- Traditional audio feature extraction models such as Gaussian mixture models (GMMs) and hidden Markov models (HMMs);
- Deep learning-based audio feature extraction models such as recurrent neural networks (RNNs), long short-term memory (LSTM) networks, encoding-decoding frameworks, attention mechanisms, etc.

Deep learning-based models have an error rate that is an order of magnitude lower than traditional models, and therefore are gaining momentum as core technology in the field of audio signal processing.

Audio data is usually represented by the extracted audio features. The retrieval process searches and compares these features and attributes rather than the audio data itself. Therefore, the effectiveness of audio similarity retrieval largely depends on the feature extraction quality.

In this article, [large-scale pre-trained audio neural networks for audio pattern recognition (PANNs)](https://github.com/qiuqiangkong/audioset_tagging_cnn) are used to extract feature vectors for its mean average accuracy (mAP) of 0.439 (Hershey et al., 2017).

After extracting the feature vectors of the audio data, we can implement high-performance feature vector analysis using Milvus.

### Vector similarity search

[Milvus](https://milvus.io/) is a cloud-native, open-source vector database built to manage embedding vectors generated by machine learning models and neural networks. It is widely used in scenarios such as computer vision, natural language processing, computational chemistry, personalized recommender systems, and more.

The following diagram depicts the general similarity search process using Milvus: 
   
![how-does-milvus-work.png](https://assets.zilliz.com/how_does_milvus_work_6926180543.png "Milvus' vector similarity search process.")

1. Unstructured data are converted to feature vectors by deep learning models and inserted into Milvus.
2. Milvus stores and indexes these feature vectors.
3. Upon request, Milvus searches and returns vectors most similar to the query vector.

## System overview

The audio retrieval system mainly consists of two parts: insert (black line) and search(red line).

![audio-retrieval-system.png](https://assets.zilliz.com/audio_retrieval_system_663a911c95.png "Audio retrieval system powered by Milvus.")


The sample dataset used in this project contains open-source game sounds, and the code is detailed in the [Milvus bootcamp](https://github.com/milvus-io/bootcamp/tree/master/solutions/audio_similarity_search).

### Step 1: Insert data

Below is the example code for generating audio embeddings with the pre-trained PANNs-inference model and inserting them into Milvus, which assigns a unique ID to each vector embedding.

```
1 wav_name, vectors_audio = get_audio_embedding(audio_path)  
2 if vectors_audio:    
3     embeddings.append(vectors_audio)  
4     wav_names.append(wav_name)  
5 ids_milvus = insert_vectors(milvus_client, table_name, embeddings)  
6 
```

Then the returned **ids_milvus** are stored along with other relevant information (e.g. **wav_name**) for the audio data held in a MySQL database for subsequent processing. 

```
1 get_ids_correlation(ids_milvus, wav_name)  
2 load_data_to_mysql(conn, cursor, table_name)    
3  
```

### Step 2: Audio search

Milvus calculates the inner product distance between the pre-stored feature vectors and the input feature vectors, extracted from the query audio data using the PANNs-inference model, and returns the **ids_milvus** of similar feature vectors, which correspond to the audio data searched.

```
1 _, vectors_audio = get_audio_embedding(audio_filename)    
2 results = search_vectors(milvus_client, table_name, [vectors_audio], METRIC_TYPE, TOP_K)  
3 ids_milvus = [x.id for x in results[0]]  
4 audio_name = search_by_milvus_ids(conn, cursor, ids_milvus, table_name)    
5
````

## API reference and demo

### API

This audio retrieval system is built with open-source code. Its main features are audio data insertion and deletion. All APIs can be viewed by typing **127.0.0.1:<port>/docs** in the browser. 





### Demo

We host a [live demo](https://zilliz.com/solutions ) of the Milvus-based audio retrieval system online that you can try out with your own audio data. 

![audio-search-demo.png](https://assets.zilliz.com/audio_search_demo_cae60625db.png "Audio search demo powered by Milvus.")

## Conclusion

Living in the era of big data, people find their lives abound with all sorts of information. To make better sense of it, traditional text retrieval no long cuts it. Today's information retrieval technology is in urgent need of the retrieval of various unstructured data types, such as videos, images, and audio.

Unstructured data, which is difficult for computers to process, can be converted into feature vectors using deep learning models. This converted data can easily be processed by machines, enabling us to analyze unstructured data in ways our predecessors were never able to. Milvus, an open-source vector database, can efficiently process the feature vectors extracted by AI models and provides a variety of common vector similarity calculations.

## References
Hershey, S., Chaudhuri, S., Ellis, D.P., Gemmeke, J.F., Jansen, A., Moore, R.C., Plakal, M., Platt, D., Saurous, R.A., Seybold, B. and Slaney, M., 2017, March. CNN architectures for large-scale audio classification. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 131-135, 2017 

## Don't be a stranger

- Find or contribute to Milvus on [GitHub](https://github.com/milvus-io/milvus/).

- Interact with the community via [Slack](https://join.slack.com/t/milvusio/shared_invite/zt-e0u4qu3k-bI2GDNys3ZqX1YCJ9OM~GQ).

- Connect with us on [Twitter](https://twitter.com/milvusio).
  

Why Manual Sharding is a Bad Idea for Vector Database And How to Fix It