
Building a robust RAG system usually starts with **document** [**chunking**](https://zilliz.com/learn/guide-to-chunking-strategies-for-rag#Chunking)—splitting large texts into manageable pieces for embedding and retrieval. Common strategies include:

*   **Fixed‑size chunks** (e.g., every 512 tokens)
    
*   **Variable‑size chunks** (e.g., paragraph or sentence boundaries)
    
*   **Sliding windows** (overlapping spans)
    
*   **Recursive chunking** (hierarchical splits)
    
*   **Semantic chunking** (grouping by topic)
    

While these methods have their merits, they often fracture long‑range context. To address this challenge, Jina AI creates a Late Chunking approach: embed the entire document first, then carve out your chunks. 

In this article, we'll explore how Late Chunking works and demonstrate how combining it with [Milvus](https://milvus.io/)—a high-performance open-source vector database built for similarity search—can dramatically improve your RAG pipelines. Whether you're building enterprise knowledge bases, AI-driven customer support, or advanced search applications, this walkthrough will show you how to manage embeddings more effectively at scale.

## What Is Late Chunking?

Traditional chunking methods can break important connections when key information spans multiple chunks—resulting in poor retrieval performance. 

Consider these release notes for Milvus 2.4.13, split into two chunks like below: 

![](https://assets.zilliz.com/Figure1_Chunking_Milvus2_4_13_Release_Note_fe7fbdb833.png)


_Figure 1. Chunking Milvus 2.4.13 Release Note_


If you query, “What are the new features in Milvus 2.4.13?”, a standard embedding model may fail to link “Milvus 2.4.13” (in Chunk 1) with its features (in Chunk 2). The result? Weaker vectors and lower retrieval accuracy.

Heuristic fixes—such as sliding windows, overlapping contexts, and repeated scans—provide partial relief but no guarantees.

**Traditional chunking** follows this pipeline:

1.  **Pre‑chunk** text (by sentences, paragraphs, or max token length).
    
2.  **Embed** each chunk separately.
    
3.  **Aggregate** token embeddings (e.g., via average pooling) into a single chunk vector.
    

**Late Chunking** flips the pipeline:

1.  **Embed first**: Run a long‑context transformer over the full document, generating rich token embeddings that capture global context.
    
2.  **Chunk later**: Average‑pool contiguous spans of those token embeddings to form your final chunk vectors.
    
![](https://assets.zilliz.com/Figure2_Naive_Chunkingvs_Late_Chunking_a94d30b6ea.png)


_Figure 2. Naive Chunking vs. Late Chunking (_[_Source_](https://jina.ai/news/late-chunking-in-long-context-embedding-models/)_)_



By preserving full‑document context in every chunk, Late Chunking yields:

*   **Higher retrieval accuracy**—each chunk is contextually aware.
    
*   **Fewer chunks**—you send more focused text to your LLM, cutting costs and latency.
    

Many long‑context models like jina-embeddings-v2-base-en can process up to 8,192 tokens—equivalent to about a 20-minute read (roughly 5,000 words)—making Late Chunking practical for most real‑world documents.

Now that we understand the “what” and “why” behind Late Chunking, let’s dive into the “how”. In the next section, we’ll guide you through a hands‑on implementation of the Late Chunking pipeline, benchmark its performance against traditional chunking, and validate its real‑world impact using Milvus. This practical walkthrough will bridge theory and practice, showing exactly how to integrate Late Chunking into your RAG workflows.

## Testing Late Chunking

### Basic Implementation

Below are the core functions for Late Chunking. We’ve added clear docstrings to guide you through each step. The function `sentence_chunker` splits the original document into paragraph‑based chunks, returning both the chunk contents and the chunk annotation information `span_annotations` (i.e., the start and end indices of each chunk).

```
def sentence_chunker(document, batch_size=10000):
    nlp = spacy.blank("en")
    nlp.add_pipe("sentencizer", config={"punct_chars": None})
    doc = nlp(document)

    docs = []
    for i in range(0, len(document), batch_size):
        batch = document[i : i + batch_size]
        docs.append(nlp(batch))

    doc = Doc.from_docs(docs)

    span_annotations = []
    chunks = []
    for i, sent in enumerate(doc.sents):
        span_annotations.append((sent.start, sent.end))
        chunks.append(sent.text)

    return chunks, span_annotations
```


The function `document_to_token_embeddings` uses the jinaai/jina-embeddings-v2-base-en model and its tokenizer to produce embeddings for the entire document.

```
def document_to_token_embeddings(model, tokenizer, document, batch_size=4096):
    tokenized_document = tokenizer(document, return_tensors="pt")
    tokens = tokenized_document.tokens()

    outputs = []
    for i in range(0, len(tokens), batch_size):
        
        start = i
        end   = min(i + batch_size, len(tokens))

        batch_inputs = {k: v[:, start:end] for k, v in tokenized_document.items()}

        with torch.no_grad():
            model_output = model(**batch_inputs)

        outputs.append(model_output.last_hidden_state)

    model_output = torch.cat(outputs, dim=1)
    return model_output
```


The function `late_chunking` takes the document’s token embeddings and the original chunk annotation information `span_annotations`, and then produces the final chunk embeddings.

```
def late_chunking(token_embeddings, span_annotation, max_length=None):
    outputs = []
    for embeddings, annotations in zip(token_embeddings, span_annotation):
        if (
            max_length is not None
        ):
            annotations = [
                (start, min(end, max_length - 1))
                for (start, end) in annotations
                if start < (max_length - 1)
            ]
        pooled_embeddings = []
        for start, end in annotations:
            if (end - start) >= 1:
                pooled_embeddings.append(
                    embeddings[start:end].sum(dim=0) / (end - start)
                )
                    
        pooled_embeddings = [
            embedding.detach().cpu().numpy() for embedding in pooled_embeddings
        ]
        outputs.append(pooled_embeddings)

    return outputs
```



For example, chunking with jinaai/jina-embeddings-v2-base-en:

```
tokenizer = AutoTokenizer.from_pretrained('jinaai/jina-embeddings-v2-base-en', trust_remote_code=True)
model     = AutoModel.from_pretrained('jinaai/jina-embeddings-v2-base-en', trust_remote_code=True)

# First chunk the text as normal, to obtain the beginning and end points of the chunks.
chunks, span_annotations = sentence_chunker(document)
# Then embed the full document.
token_embeddings = document_to_token_embeddings(model, tokenizer, document)
# Then perform the late chunking
chunk_embeddings = late_chunking(token_embeddings, [span_annotations])[0]
```


_Tip:_ Wrapping your pipeline in functions makes it easy to swap in other long‑context models or chunking strategies.

### Comparison with Traditional Embedding Methods

To further demonstrate the advantages of Late Chunking, we also compared it to traditional embedding approaches, using a set of sample documents and queries.

And let’s revisit our Milvus 2.4.13 release note example:

```
Milvus 2.4.13 introduces dynamic replica load, allowing users to adjust the number of collection replicas without needing to release and reload the collection. This version also addresses several critical bugs related to bulk importing, expression parsing, load balancing, and failure recovery. Additionally, significant improvements have been made to MMAP resource usage and import performance, enhancing overall system efficiency. We highly recommend upgrading to this release for better performance and stability.
```


We measure [cosine similarity](https://zilliz.com/blog/similarity-metrics-for-vector-search#Cosine-Similarity) between the query embedding ("milvus 2.4.13") and each chunk:

```
cos_sim = lambda x, y: np.dot(x, y) / (np.linalg.norm(x) * np.linalg.norm(y))

milvus_embedding = model.encode('milvus 2.4.13')

for chunk, late_chunking_embedding, traditional_embedding in zip(chunks, chunk_embeddings, embeddings_traditional_chunking):
    print(f'similarity_late_chunking("milvus 2.4.13", "{chunk}")')
    print('late_chunking: ', cos_sim(milvus_embedding, late_chunking_embedding))
    print(f'similarity_traditional("milvus 2.4.13", "{chunk}")')
    print('traditional_chunking: ', cos_sim(milvus_embedding, traditional_embeddings))
```


Late Chunking consistently outperformed traditional chunking, yielding higher cosine similarities across every chunk. This confirms that embedding the full document first preserves global context more effectively.

```
similarity_late_chunking("milvus 2.4.13", "Milvus 2.4.13 introduces dynamic replica load, allowing users to adjust the number of collection replicas without needing to release and reload the collection.")
late_chunking: 0.8785206
similarity_traditional("milvus 2.4.13", "Milvus 2.4.13 introduces dynamic replica load, allowing users to adjust the number of collection replicas without needing to release and reload the collection.")
traditional_chunking: 0.8354263

similarity_late_chunking("milvus 2.4.13", "This version also addresses several critical bugs related to bulk importing, expression parsing, load balancing, and failure recovery.")
late_chunking: 0.84828955
similarity_traditional("milvus 2.4.13", "This version also addresses several critical bugs related to bulk importing, expression parsing, load balancing, and failure recovery.")
traditional_chunking: 0.7222632

similarity_late_chunking("milvus 2.4.13", "Additionally, significant improvements have been made to MMAP resource usage and import performance, enhancing overall system efficiency.")
late_chunking: 0.84942204
similarity_traditional("milvus 2.4.13", "Additionally, significant improvements have been made to MMAP resource usage and import performance, enhancing overall system efficiency.")
traditional_chunking: 0.6907381

similarity_late_chunking("milvus 2.4.13", "We highly recommend upgrading to this release for better performance and stability.")
late_chunking: 0.85431844
similarity_traditional("milvus 2.4.13", "We highly recommend upgrading to this release for better performance and stability.")
traditional_chunking: 0.71859795
```


We can see that embedding the full paragraph first ensures each chunk carries the “`Milvus 2.4.13`” context—boosting similarity scores and retrieval quality.

### **Testing Late Chunking in Milvus**

Once chunk embeddings are generated, we can store them in Milvus and perform queries. The following code inserts chunk vectors into the collection.

#### **Importing Embeddings into Milvus**

```
batch_data=[]
for i in range(len(chunks)):
    data = {
            "content": chunks[i],
            "embedding": chunk_embeddings[i].tolist(),
        }

    batch_data.append(data)

res = client.insert(
    collection_name=collection,
    data=batch_data,
)
```



#### Querying and Validation

To validate the accuracy of Milvus queries, we compare its retrieval results to brute-force cosine similarity scores calculated manually. If both methods return consistent top‑k results, we can be confident that Milvus’s search accuracy is reliable.

We compare Milvus’s native search with a brute‑force cosine similarity scan:

```
def late_chunking_query_by_milvus(query, top_k = 3):
    query_vector = model(**tokenizer(query, return_tensors="pt")).last_hidden_state.mean(1).detach().cpu().numpy().flatten()

    res = client.search(
                collection_name=collection,
                data=[query_vector.tolist()],
                limit=top_k,
                output_fields=["id", "content"],
            )

    return [item.get("entity").get("content") for items in res for item in items]

def late_chunking_query_by_cosine_sim(query, k = 3):
    cos_sim = lambda x, y: np.dot(x, y) / (np.linalg.norm(x) * np.linalg.norm(y))
    query_vector = model(**tokenizer(query, return_tensors="pt")).last_hidden_state.mean(1).detach().cpu().numpy().flatten()

    results = np.empty(len(chunk_embeddings))
    for i, (chunk, embedding) in enumerate(zip(chunks, chunk_embeddings)):
        results[i] = cos_sim(query_vector, embedding)

    results_order = results.argsort()[::-1]
    return np.array(chunks)[results_order].tolist()[:k]
```


This confirms Milvus returns the same top‑k chunks as a manual cosine‑sim scan.

```
> late_chunking_query_by_milvus("What are new features in milvus 2.4.13", 3)

['\n\n### Features\n\n- Dynamic replica adjustment for loaded collections ([#36417](https://github.com/milvus-io/milvus/pull/36417))\n- Sparse vector MMAP in growing segment types ([#36565](https://github.com/milvus-io/milvus/pull/36565))...
```

```
> late_chunking_query_by_cosine_sim("What are new features in milvus 2.4.13", 3)

['\n\n### Features\n\n- Dynamic replica adjustment for loaded collections ([#36417](https://github.com/milvus-io/milvus/pull/36417))\n- Sparse vector MMAP in growing segment types (#36565)...
```


So both methods yield the same top-3 chunks, confirming Milvus’s accuracy.

## Conclusion


In this article, we took a deep dive into the mechanics and benefits of Late Chunking. We began by identifying the shortcomings of traditional chunking approaches, particularly when handling long documents where preserving context is crucial. We introduced the concept of Late Chunking—embedding the entire document before slicing it into meaningful chunks—and showed how this preserves global context, leading to improved semantic similarity and retrieval accuracy.

We then walked through a hands-on implementation using Jina AI’s jina-embeddings-v2-base-en model and evaluated its performance compared to traditional methods. Finally, we demonstrated how to integrate the chunk embeddings into Milvus for scalable and accurate vector search.

Late Chunking offers a **context‑first** approach to embedding—perfect for long, complex documents where context matters most. By embedding the entire text upfront and slicing later, you gain:

*   🔍 **Sharper retrieval accuracy**
    
*   ⚡ **Lean, focused LLM prompts**
    
*   🛠️ **Simple integration** with any long‑context model


Smarter Retrieval for RAG: Late Chunking with Jina Embeddings v2 and Milvus


How to Choose the Right Embedding Model for RAG


From Word2Vec to LLM2Vec: How to Choose the Right Embedding Model for RAG



Performance is at the heart of Milvus. Under normal conditions, a search request within Milvus completes in just milliseconds. But what happens when your cluster slows down—when search latency stretches into whole seconds instead? 

Slow searches don’t happen often, but they can surface at scale or under complex workloads. And when they do, they matter: they disrupt user experience, skew application performance, and often expose hidden inefficiencies in your setup.

In this post, we’ll walk through how to triage slow requests in Milvus and share practical steps you can take to keep latency predictable, stable, and consistently low.

## Identifying Slow Searches

Diagnosing a slow request starts with two questions: **how often does it happen, and where is the time going?** Milvus gives you both answers through metrics and logs.

### Milvus Metrics

Milvus exports detailed metrics you can monitor in Grafana dashboards. 

![](https://assets.zilliz.com/image_2_64a5881bf2.png)

![](https://assets.zilliz.com/image_3_b7b8b369ec.png)

Key panels include:

*   **Service Quality → Slow Query**: Flags any request exceeding proxy.slowQuerySpanInSeconds (default: 5s). These are also marked in Prometheus.
    
*   **Service Quality → Search Latency**: Shows overall latency distribution. If this looks normal, but end users still see delays, the problem is likely outside Milvus—in the network or application layer.
    
*   **Query Node → Search Latency by Phase**: Breaks latency into queue, query, and reduce stages. For deeper attribution, panels such as _Scalar_ _Filter Latency_, _Vector Search Latency_, and _Wait tSafe Latency_ reveal which stage dominates.
    

### Milvus Logs

Milvus also logs any request lasting more than one second, tagged with markers like [Search slow]. These logs show _which_ queries are slow, complementing the _where_ insights from metrics. As a rule of thumb:

*   **< 30 ms** → healthy search latency in most scenarios
    
*   **> 100 ms** → worth investigating
    
*   **> 1 s** → definitely slow and requires attention
    

Example log:

```
[2025/08/23 19:22:19.900 +00:00] [INFO] [proxy/impl.go:3141] ["Search slow"] [traceID=9100b3092108604716f1472e4c7d54e4] [role=proxy] [db=default] [collection=my_repos] [partitions="[]"] [dsl="user == \"milvus-io\" && repo == \"proxy.slowQuerySpanInSeconds\""] [len(PlaceholderGroup)=8204] [OutputFields="[user,repo,path,descripion]"] [search_params="[{\"key\":\"topk\",\"value\":\"10\"},{\"key\":\"metric_type\",\"value\":\"COSINE\"},{\"key\":\"anns_field\",\"value\":\"vector\"},{\"key\":\"params\",\"value\":\"{\\\"nprobe\\\":256,\\\"metric_type\\\":\\\"COSINE\\\"}\"}]"] [ConsistencyLevel=Strong] [useDefaultConsistency=true] [guarantee_timestamp=460318735832711168] [nq=1] [duration=5m12.002784545s] [durationPerNq=5m12.002784545s]
```

In short, **metrics tell you where the time is going; logs tell you which queries are hit.**

## Analyzing Root Cause

### Heavy Workload

A common cause of slow requests is an excessive workload. When a request has a very large **NQ** (number of queries per request), it can run for an extended period and monopolize query node resources. Other requests stack up behind it, resulting in rising queue latency. Even if each request has a small NQ, a very high overall throughput (QPS) can still cause the same effect, as Milvus may merge concurrent search requests internally.

![ ](https://assets.zilliz.com/high_workload_cf9c75e24c.png)

**Signals to watch for:**

*   All queries show unexpectedly high latency.
    
*   Query Node metrics report high **in-queue latency**.
    
*   Logs show a request with a large NQ and a long total duration, but a relatively small durationPerNQ—indicating that one oversized request is dominating resources.
    

**How to fix it:**

*   **Batch queries**: Keep NQ modest to avoid overloading a single request.
    
*   **Scale out query nodes**: If high concurrency is a regular part of your workload, add query nodes to spread the load and maintain low latency.
    

### Inefficient Filtering

Another common bottleneck comes from inefficient filters. If filter expressions are poorly conducted or fields lack scalar indexes, Milvus may fall back to a **full scan** instead of scanning a small, targeted subset. JSON filters and strict consistency settings can further increase overhead.

![ ](https://assets.zilliz.com/inefficient_filtering_e524615d63.png)

**Signals to watch for:**

*   High **Scalar Filter Latency** in Query Node metrics.
    
*   Noticeable latency spikes only when filters are applied.
    
*   Long **Wait tSafe Latency** if strict consistency is enabled.
    

**How to fix it:**

*   **Simplify filter expressions**: Reduce query plan complexity by optimizing filters. For example, replace long OR chains with an IN expression:
    

```
# Replace chains of OR conditions with IN
tag = {"tag": ["A", "B", "C", "D"]}
filter_expr = "tag IN {tag}"
```

*   Milvus also introduces a filter expression templating mechanism designed to improve efficiency by reducing the time spent parsing complex expressions. See [this document](https://milvus.io/docs/filtering-templating.md) for more details.

 *   **Add proper indexes**: Avoid full scans by creating scalar indexes on fields used in filters.
     
 *   **Handle JSON efficiently**: Milvus 2.6 introduced path and flat indexes for JSON fields, enabling efficient handling of JSON data. JSON shredding is also on [the roadmap](https://milvus.io/docs/roadmap.md) to further improve performance. Refer to [the JSON field document](https://milvus.io/docs/use-json-fields.md#JSON-Field) for additional information.  
     
 *   **Tune consistency level**: Use *Bounded* or *Eventually* consistent reads when strict guarantees are not required, reducing *tSafe* wait time.
    

### Improper Choice of Vector Index

[Vector indexes](https://milvus.io/docs/index-explained.md) are not one-size-fits-all. Selecting the wrong index can significantly impact latency. In-memory indexes deliver the fastest performance but consume more memory, while on-disk indexes save memory at the cost of speed. Binary vectors also require specialized indexing strategies.

![ ](https://assets.zilliz.com/image_4_25fa1b9c13.png)

**Signals to watch for:**

*   High Vector Search Latency in Query Node metrics.
    
*   Disk I/O saturation when using DiskANN or MMAP.
    
*   Slower queries immediately after restart due to cache cold start.
    

**How to fix it:**

*   **Match index to workload (float vectors):**
    
    *   **HNSW** — best for in-memory use cases with high recall and low latency.
        
    *   **IVF family** — flexible trade-offs between recall and speed.
        
    *   **DiskANN** — supports billion-scale datasets, but requires strong disk bandwidth.
        
*   **For binary vectors:** Use the [MINHASH_LSH index](https://milvus.io/docs/minhash-lsh.md) (introduced in Milvus 2.6) with the MHJACCARD metric to efficiently approximate Jaccard similarity.
    
*   **Enable** [**MMAP**](https://milvus.io/docs/mmap.md): Map index files into memory instead of keeping them fully resident to strike a balance between latency and memory usage.
    
*   **Tune index/search parameters**: Adjust settings to balance recall and latency for your workload.
    
*   **Mitigate cold starts**: Warm up frequently accessed segments after a restart to avoid initial query slowness.
    

### Runtime & Environment Conditions

Not all slow queries are caused by the query itself. Query nodes often share resources with background jobs, such as compaction, data migration, or index building. Frequent upserts can generate many small, unindexed segments, forcing searches to scan raw data. In some cases, version-specific inefficiencies can also introduce latency until patched.

![ ](https://assets.zilliz.com/img_v3_02q5_4dd2e545_93dc_4c58_b609_d76d50c2013g_aad0a89208.png)

**Signals to watch for:**

*   CPU usage spikes during background jobs (compaction, migration, index builds).
    
*   Disk I/O saturation affecting query performance.
    
*   Very slow cache warm-up after a restart.
    
*   Large numbers of small, unindexed segments (from frequent upserts).
    
*   Latency regressions tied to specific Milvus versions.
    

**How to fix it:**

*   **Reschedule background tasks** (e.g., compaction) to off-peak hours.
    
*   **Release unused collections** to free memory.
    
*   **Account for warm-up time** after restarts; pre-warm caches if needed.
    
*   **Batch upserts** to reduce the creation of tiny segments and let compaction keep up.
    
*   **Stay current**: upgrade to newer Milvus versions for bug fixes and optimizations.
    
*   **Provision resources**: dedicate extra CPU/memory to latency-sensitive workloads.
    

By matching each signal with the right action, most slow queries can be resolved quickly and predictably.

## Best Practices to Prevent Slow Searches

The best debugging session is the one you never need to run. In our experience, a few simple habits go a long way toward preventing slow queries in Milvus:

*   **Plan resource allocation** to avoid CPU and disk contention.
    
*   **Set proactive alerts** for both failures and latency spikes.
    
*   **Keep filter expressions** short, simple, and efficient.
    
*   **Batch upserts** and keep NQ/QPS at sustainable levels.
    
*   **Index all fields** that are used in filters.
    

Slow queries in Milvus are rare, and when they do appear, they usually have clear, diagnosable causes. With metrics, logs, and a structured approach, you can quickly identify and resolve issues. This is the same playbook our support team uses every day — and now it’s yours too.

We hope this guide provides not only a troubleshooting framework but also the confidence to keep your Milvus workloads running smoothly and efficiently.

## 💡 Want to dive deeper?

*   Join the [**Milvus Discord**](https://discord.com/invite/8uyFbECzPX) to ask questions, share experiences, and learn from the community.
    
*   Sign up for our [**Milvus Office Hours**](https://milvus.io/blog/join-milvus-office-hours-to-get-support-from-vectordb-experts.md) to speak directly with the team and receive hands-on assistance with your workloads.


How to Debug Slow Search Requests in Milvus



For a long time, many of us — myself included — treated LLM hallucinations as nothing more than defects. An entire toolchain has been built around eliminating them: retrieval systems, guardrails, fine-tuning, and more. These safeguards are still valuable. But the more I’ve studied how models actually generate responses — and how systems like [**Milvus**](https://milvus.io/) fit into broader AI pipelines — the less I believe hallucinations are simply failures. In fact, they can also be the spark of AI creativity.

If we look at human creativity, we find the same pattern. Every breakthrough relies on imaginative leaps. But those leaps never come out of nowhere. Poets first master rhythm and meter before they break the rules. Scientists rely on established theories before venturing into untested territory. Progress depends on these leaps, as long as they are grounded in solid knowledge and understanding.

LLMs operate in much the same way. Their so-called “hallucinations” or “leaps”— analogies, associations, and extrapolations — emerge from the same generative process that allows models to make connections, extend knowledge, and surface ideas beyond what they’ve been explicitly trained on. Not every leap succeeds, but when it does, the results can be compelling.

That’s why I see **Context Engineering** as the critical next step. Rather than trying to eliminate every hallucination, we should focus on _steering_ them. By designing the right context, we can strike a balance — keeping models imaginative enough to explore new ground, while ensuring they remain anchored enough to be trusted.

## What is Context Engineering?


So what exactly do we mean by _context engineering_? The term may be new, but the practice has been evolving for years. Techniques such as RAG, prompting, function calling, and MCP are all early attempts at solving the same problem: providing models with the right environment to produce useful results. Context engineering is about unifying those approaches into a coherent framework.

## The Three Pillars of Context Engineering


Effective context engineering rests on three interconnected layers:

![](https://assets.zilliz.com/context_engineering_1_8f2b39c5e7.png)

### 1\. The Instructions Layer — Defining Direction

This layer includes prompts, few-shot examples, and demonstrations. It’s the model’s navigation system: not just a vague “go north,” but a clear route with waypoints. Well-structured instructions set boundaries, define goals, and reduce ambiguity in model behavior.

### 2\. The Knowledge Layer — Supplying Ground Truth

Here we place the facts, code, documents, and state that the model needs to reason effectively. Without this layer, the system improvises from incomplete memory. With it, the model can ground its outputs in domain-specific data. The more accurate and relevant the knowledge, the more reliable the reasoning.

### 3\. The Tools Layer — Enabling Action and Feedback

This layer covers APIs, function calls, and external integrations. It’s what enables the system to move beyond reasoning to execution—retrieving data, performing calculations, or triggering workflows. Just as importantly, these tools provide real-time feedback that can be looped back into the model’s reasoning. That feedback is what enables correction, adaptation, and continuous improvement. In practice, this is what transforms LLMs from passive responders into active participants in a system.

These layers aren’t silos—they reinforce each other. Instructions set the destination, knowledge provides the information to work with, and tools turn decisions into action and feed results back into the loop. Orchestrated well, they create an environment where models can be both creative and dependable.

## The Long Context Challenges: When More Becomes Less

Many AI models now advertise million-token windows—enough for ~75,000 lines of code or a 750,000-word document. But more context doesn’t automatically yield better results. In practice, very long contexts introduce distinct failure modes that can degrade reasoning and reliability.

### Context Poisoning — When Bad Information Spreads

Once false information enters the working context—whether in goals, summaries, or intermediate state—it can derail the entire reasoning process. [DeepMind’s Gemini 2.5 report](https://arxiv.org/pdf/2507.06261) provides a clear example. An LLM agent playing Pokémon misread the game state and decided its mission was to “catch the uncatchable legendary.” That incorrect goal was recorded as fact, leading the agent to generate elaborate but impossible strategies. 

As shown in the excerpt below, the poisoned context trapped the model in a loop—repeating errors, ignoring common sense, and reinforcing the same mistake until the entire reasoning process collapsed.

![](https://assets.zilliz.com/Figure_1_Excerpt_from_Gemini_2_5_Tech_Paper_e89adf9eed.png)

Figure 1: Excerpt from [Gemini 2.5 Tech Paper](https://arxiv.org/pdf/2507.06261)

### Context Distraction — Lost in the Details

As context windows expand, models can start to overweight the transcript and underuse what they learned during training. DeepMind’s Gemini 2.5 Pro, for example, supports a million-token window but [begins to drift around ~100,000 tokens](https://arxiv.org/pdf/2507.06261)—recycling past actions instead of generating new strategies. [Databricks’ research](https://www.databricks.com/blog/long-context-rag-performance-llms) shows that smaller models, like Llama 3.1-405B, reach that limit far sooner at roughly ~32,000 tokens. It’s a familiar human effect: too much background reading, and you lose the plot.

![](https://assets.zilliz.com/Figure_2_Excerpt_from_Gemini_2_5_Tech_Paper_56d775c59d.png)

Figure 2: Excerpt from [Gemini 2.5 Tech Paper](https://arxiv.org/pdf/2507.06261)



![](https://assets.zilliz.com/Figure_3_Long_context_performance_of_GPT_Claude_Llama_Mistral_and_DBRX_models_on_4_curated_RAG_datasets_Databricks_Docs_QA_Finance_Bench_Hot_Pot_QA_and_Natural_Questions_Source_Databricks_99086246b9.png)

_Figure 3: Long context performance of GPT, Claude, Llama, Mistral and DBRX models on 4 curated RAG datasets (Databricks DocsQA, FinanceBench, HotPotQA and Natural Questions) \[Source:_ [_Databricks_](https://www.databricks.com/blog/long-context-rag-performance-llms)_\]_ 

### Context Confusion — Too Many Tools in the Kitchen

Adding more tools doesn’t always help. The [Berkeley Function-Calling Leaderboard](https://gorilla.cs.berkeley.edu/leaderboard.html) shows that when the context displays extensive tool menus—often with many irrelevant options—model reliability decreases, and tools are invoked even when none are needed. One clear example: a quantized Llama 3.1-8B failed with 46 tools available, but succeeded when the set was reduced to 19. It’s the paradox of choice for AI systems—too many options, worse decisions.

### Context Clash — When Information Conflicts

Multi-turn interactions add a distinct failure mode: early misunderstandings compound as the dialogue branches. In [Microsoft and Salesforce experiments](https://arxiv.org/pdf/2505.06120v1), both open- and closed-weight LLMs performed markedly worse in multi-turn vs. single-turn settings—an average 39% drop across six generation tasks. Once a wrong assumption enters the conversation state, subsequent turns inherit it and amplify the error.

![](https://assets.zilliz.com/Figure_4_LL_Ms_get_lost_in_multi_turn_conversations_in_experiments_21f194b02d.png)

_Figure 4: LLMs get lost in multi-turn conversations in experiments_ 

The effect shows up even in frontier models. When benchmark tasks were distributed across turns, the performance score of OpenAI’s o3 model fell from **98.1** to **64.1**. An initial misread effectively “sets” the world model; each reply builds on it, turning a small contradiction into a hardened blind spot unless explicitly corrected.

![](https://assets.zilliz.com/Figure_4_The_performance_scores_in_LLM_multi_turn_conversation_experiments_414d3a0b3f.png)

_Figure 4: The performance scores in LLM multi-turn conversation experiments_ 

## Six Strategies to Tame Long Context

The answer to long-context challenges isn’t to abandon the capability—it’s to engineer it with discipline. Here are six strategies we’ve seen work in practice:

### Context Isolation 

Break complex workflows into specialized agents with isolated contexts. Each agent focuses on its own domain without interference, reducing the risk of error propagation. This not only improves accuracy but also enables parallel execution, much like a well-structured engineering team.

### Context Pruning 

Regularly audit and trim the context. Remove redundant details, stale information, and irrelevant traces. Think of it as refactoring: clean out dead code and dependencies, leaving only the essentials. Effective pruning requires explicit criteria for what belongs and what doesn’t.

### Context Summarization 

Long histories don’t need to be carried around in full. Instead, condense them into concise summaries that capture only what is essential for the next step. Good summarization retains the critical facts, decisions, and constraints, while eliminating repetition and unnecessary details. It’s like replacing a 200-page spec with a one-page design brief that still gives you everything you need to move forward.

### Context Offloading

Not every detail needs to be part of the live context. Persist non-critical data in external systems—knowledge bases, document stores, or vector databases like Milvus—and fetch it only when needed. This lightens the model’s cognitive load while keeping background information accessible.

### Strategic RAG 

Information retrieval is powerful only if it’s selective. Introduce external knowledge through rigorous filtering and quality controls, ensuring the model consumes relevant and accurate inputs. As with any data pipeline: garbage in, garbage out—but with high-quality retrieval, the context becomes an asset, not a liability.

### Optimized Tool Loading

More tools don’t equal better performance. Studies show reliability drops sharply beyond ~30 available tools. Load only the functions a given task requires, and gate access to the rest. A lean toolbox fosters precision and reduces the noise that can overwhelm decision-making.

## The Infrastructure Challenge of Context Engineering

Context engineering is only as effective as the infrastructure it runs on. And today’s enterprises are hitting a perfect storm of data challenges:

### Scale Explosion — From Terabytes to Petabytes

Today, data growth has redefined the baseline. Workloads that once fit comfortably in a single database now span petabytes, demanding distributed storage and compute. A schema change that used to be a one-line SQL update can cascade into a full orchestration effort across clusters, pipelines, and services. Scaling isn’t simply about adding hardware—it’s about engineering for coordination, resilience, and elasticity at a scale where every assumption gets stress-tested.

### Consumption Revolution — Systems That Speak AI

AI agents don’t just query data; they generate, transform, and consume it continuously at machine speeds. Infrastructure designed just for human-facing applications can’t keep up. To support agents, systems must provide low-latency retrieval, streaming updates, and write-heavy workloads without breaking. In other words, the infrastructure stack must be built to “speak AI” as its native workload, not as an afterthought.

### Multimodal Complexity — Many Data Types, One System

AI workloads blend text, images, audio, video, and high-dimensional embeddings, each with rich metadata attached. Managing this heterogeneity is the crux of practical context engineering. The challenge isn’t just storing diverse objects; it’s indexing them, retrieving them efficiently, and keeping semantic consistency across modalities. A truly AI-ready infrastructure must treat multimodality as a first-class design principle, not a bolt-on feature.

## Milvus + Loon: Purpose-Built Data Infrastructure for AI 

The challenges of scale, consumption, and multimodality can’t be solved with theory alone—they demand infrastructure that is purpose-built for AI. That’s why we at [Zilliz](https://zilliz.com/) designed **Milvus** and **Loon** to work together, addressing both sides of the problem: high-performance retrieval at runtime and large-scale data processing upstream.

*   [**Milvus**](https://milvus.io/): the most widely adopted open-source vector database optimized for high-performance vector retrieval and storage. 
    
*   **Loon**: our upcoming cloud-native multimodal data lake service designed to process and organize massive-scale multimodal data before it ever reaches the database. Stay tuned. 
    
![](https://assets.zilliz.com/loon_milvus_min_76aaa39b4e.png)


### Lightning-Fast Vector Search

**Milvus** is built from the ground up for vector workloads. As the serving layer, it delivers sub-10ms retrieval across hundreds of millions—or even billions—of vectors, whether derived from text, images, audio, or video. For AI applications, retrieval speed isn’t a “nice to have.” It’s what determines whether an agent feels responsive or sluggish, whether a search result feels relevant or out of step. Performance here is directly visible in the end-user experience.

### Multimodal Data Lake Service at Scale

**Loon** is our upcoming multimodal data lake service, designed for massive-scale offline processing and analytics of unstructured data. It complements Milvus on the pipeline side, preparing data before it ever reaches the database. Real-world multimodal datasets—spanning text, images, audio, and video—are often messy, with duplication, noise, and inconsistent formats. Loon takes care of this heavy lifting using distributed frameworks like Ray and Daft, compressing, deduplicating, and clustering the data before streaming it directly into Milvus. The result is simple: no staging bottlenecks, no painful format conversions—just clean, structured data that models can use immediately.

### Cloud-Native Elasticity

Both systems are built cloud-native, with storage and compute scaling independently. That means as workloads grow from gigabytes to petabytes, you can balance resources between real-time serving and offline training, rather than overprovisioning for one or undercutting the other.

### Future-Proof Architecture

Most importantly, this architecture is designed to grow with you. Context engineering is still evolving. Right now, most teams are focused on semantic search and RAG pipelines. But the next wave will demand more—integrating multiple data types, reasoning across them, and powering agent-driven workflows.

With Milvus and Loon, that transition doesn’t require ripping out your foundation. The same stack that supports today’s use cases can extend naturally into tomorrow’s. You add new capabilities without starting over, which means less risk, lower cost, and a smoother path as AI workloads become more complex.

## Your Next Move

Context engineering isn’t just another technical discipline—it’s how we unlock AI’s creative potential while keeping it grounded and reliable. If you’re ready to put these ideas into practice, start where it matters most. 

*   [**Experiment with Milvus**](https://milvus.io/docs/overview.md) to see how vector databases can anchor retrieval in real-world deployments.
    
*   [**Follow Milvus**](https://www.linkedin.com/company/the-milvus-project/) for updates on Loon’s release and insights into managing large-scale multimodal data.
    
*   [**Join the Zilliz community on Discord**](https://discord.com/invite/8uyFbECzPX) to share strategies, compare architectures, and help shape best practices.

The companies that master context engineering today will shape the AI landscape tomorrow. Don’t let infrastructure be the constraint—build the foundation your AI creativity deserves.



If Context Engineering Done Right, Hallucinations Can Spark AI Creativity


When Context Engineering Is Done Right, Hallucinations Can Be the Spark of AI Creativity



AI agents are becoming a core pattern in applied AI. More projects are moving past single prompts and wiring models into decision-making loops. That’s exciting, but it also means managing state, coordinating tools, handling branches, and adding human handoffs—things that aren’t immediately obvious.

[**LangGraph**](https://github.com/langchain-ai/langgraph) is a strong choice for this layer. It is an AI framework that provides loops, conditionals, persistence, human-in-the-loop controls, and streaming—enough structure to turn an idea into a real multi-agent app. However, LangGraph has a steep learning curve. Its documentation moves quickly, the abstractions take time to get used to, and jumping from a simple demo to something that feels like a product can be frustrating. 

Recently, I started using [**langgraph-up-react**](https://github.com/webup/langgraph-up-react)—a ready-to-use LangGraph + ReAct template for ReAct agents. It trims setup, ships with sane defaults, and lets you focus on behavior instead of boilerplate. In this post, I’ll walk through how to get started with LangGraph using this template. 


## Understanding ReAct Agents

Before diving into the template itself, it’s worth looking at the kind of agent we’ll be building. One of the most common patterns today is the **ReAct (Reason + Act)** framework, first introduced in Google’s 2022 paper _“_[_ReAct: Synergizing Reasoning and Acting in Language Models._](https://arxiv.org/abs/2210.03629)_”_

The idea is straightforward: instead of treating reasoning and action as separate, ReAct combines them into a feedback loop that looks a lot like human problem solving. The agent **reasons** about the problem, **acts** by calling a tool or API, and then **observes** the result before deciding what to do next. This simple cycle—reason → act → observe—lets agents adapt dynamically instead of following a fixed script.

Here’s how the pieces fit together:

- **Reason**: The model breaks problems into steps, plans strategies, and can even correct mistakes mid-way.

- **Act**: Based on its reasoning, the agent calls tools—whether that’s a search engine, a calculator, or your own custom API.

- **Observe**: The agent looks at the tool’s output, filters the results, and feeds that back into its next round of reasoning.

This loop has quickly become the backbone of modern AI agents. You’ll see traces of it in ChatGPT plugins, RAG pipelines, intelligent assistants, and even robotics. In our case, it’s the foundation that the `langgraph-up-react` template builds on.


## Understanding LangGraph

Now that we’ve looked at the ReAct pattern, the next question is: how do you actually implement something like that in practice? Out of the box, most language models don’t handle multi-step reasoning very well. Each call is stateless: the model generates an answer and forgets everything as soon as it’s done. That makes it hard to carry intermediate results forward or adjust later steps based on earlier ones.

[**LangGraph**](https://github.com/langchain-ai/langgraph) closes this gap. Instead of treating every prompt as a one-off, it gives you a way to break complex tasks into steps, remember what happened at each point, and decide what to do next based on the current state. In other words, it turns an agent’s reasoning process into something structured and repeatable, rather than a chain of ad-hoc prompts.

You can think of it like a **flowchart for AI reasoning**:

- **Analyze** the user query

- **Select** the right tool for the job

- **Execute** the task by calling the tool

- **Process** the results

- **Check** if the task is complete; if not, loop back and continue reasoning

- **Output** the final answer

Along the way, LangGraph handles **memory storage** so results from earlier steps aren’t lost, and it integrates with an **external tool library** (APIs, databases, search, calculators, file systems, etc.).

That’s why it’s called _LangGraph_: **Lang (Language) + Graph**—a framework for organizing how language models think and act over time.


## Understanding langgraph-up-react

LangGraph is powerful, but it comes with overhead. Setting up state management, designing nodes and edges, handling errors, and wiring in models and tools all take time. Debugging multi-step flows can also be painful—when something breaks, the issue might be in any node or transition. As projects grow, even small changes can ripple through the codebase and slow everything down.

This is where a mature template makes a huge difference. Instead of starting from scratch, a template gives you a proven structure, pre-built tools, and scripts that just work. You skip the boilerplate and focus directly on the agent logic.

[**langgraph-up-react**](https://github.com/webup/langgraph-up-react) is one such template. It’s designed to help you spin up a LangGraph ReAct agent quickly, with:

- 🔧 **Built-in tool ecosystem**: adapters and utilities ready to use out of the box

- ⚡ **Quick start**: simple configuration and a working agent in minutes

- 🧪 **Testing included**: unit tests and integration tests for confidence as you extend

- 📦 **Production-ready setup**: architecture patterns and scripts that save time when deploying

In short, it takes care of the boilerplate so you can focus on building agents that actually solve your business problems.


## Getting Started with the langgraph-up-react Template

Getting the template running is straightforward. Here’s the setup process step by step:

1. Install environment dependencies

```
curl -LsSf https://astral.sh/uv/install.sh | sh
```

2. Clone the project

```
git clone https://github.com/webup/langgraph-up-react.git
cd langgraph-up-react
```

3. Install dependencies

```
uv sync --dev
```

4. Configure environment

Copy the example config and add your keys:

```
cp .env.example .env
```

Edit .env and set at least one model provider plus your Tavily API key:

```
TAVILY_API_KEY=your-tavily-api-key      # Required for web search  
DASHSCOPE_API_KEY=your-dashscope-api-key  # Qwen (default recommended)  
OPENAI_API_KEY=your-openai-api-key        # OpenAI or compatible platforms  
# OPENAI_API_BASE=https://your-api-endpoint  # If using OpenAI-compatible API  
REGION=us                # Optional: region flag  
ENABLE_DEEPWIKI=true      # Optional: enable document tools  
```

5. Start the project

```
# Start development server (without UI)
make dev

# Start development server with LangGraph Studio UI
make dev_ui
```

Your dev server will now be up and ready for testing.

![](https://assets.zilliz.com/template_set_up_a42d1819ed.png)

## What Can You Build with langgraph-up-react?

So what can you actually do once the template is up and running? Here are two concrete examples that show how it can be applied in real projects.


### Enterprise Knowledge Base Q&A (Agentic RAG)

A common use case is an internal Q&A assistant for company knowledge. Think product manuals, technical docs, FAQs—information that’s useful but scattered. With `langgraph-up-react`, you can create an agent that indexes these documents in a [**Milvus**](https://milvus.io/) vector database, retrieves the most relevant passages, and generates accurate answers grounded in context.

For deployment, Milvus offers flexible options: **Lite** for quick prototyping, **Standalone** for mid-sized production workloads, and **Distributed** for enterprise-scale systems. You’ll also want to tune index parameters (e.g., HNSW) to balance speed and accuracy, and set up monitoring for latency and recall to ensure the system remains reliable under load.


### Multi-Agent Collaboration

Another powerful use case is multi-agent collaboration. Instead of one agent trying to do everything, you define several specialized agents that work together. In a software development workflow, for example, a Product Manager Agent breaks down requirements, an Architect Agent drafts the design, a Developer Agent writes code, and a Testing Agent validates the results.

This orchestration highlights LangGraph’s strengths—state management, branching, and coordination across agents. We’ll cover this setup in more detail in a later article, but the key point is that `langgraph-up-react` makes it practical to try these patterns without spending weeks on scaffolding.


## Conclusion

Building reliable agents isn’t just about clever prompts—it’s about structuring reasoning, managing state, and wiring everything into a system you can actually maintain. LangGraph gives you the framework to do that, and `langgraph-up-react` lowers the barrier by handling the boilerplate so you can focus on agent behavior.

With this template, you can spin up projects like knowledge base Q&A systems or multi-agent workflows without getting lost in setup. It’s a starting point that saves time, avoids common pitfalls, and makes experimenting with LangGraph far smoother.

In the next post, I’ll go deeper into a hands-on tutorial—showing step by step how to extend the template and build a working agent for a real use case using LangGraph, `langgraph-up-react`, and Milvus vector database. Stay tuned.


Getting Started with langgraph-up-react: A LangGraph Template


Getting Started with langgraph-up-react: A Practical LangGraph Template



When building with large language models (LLMs), the framework you choose has a huge impact on your development experience. A good framework streamlines workflows, reduces boilerplate, and makes it easier to move from prototype to production. A poor fit can do the opposite, adding friction and technical debt.

Two of the most popular options today are [**LangChain**](https://python.langchain.com/docs/introduction/) and [**LangGraph**](https://langchain-ai.github.io/langgraph/) — both open source and created by the LangChain team. LangChain focuses on component orchestration and workflow automation, making it a good fit for common use cases like retrieval-augmented generation ([RAG](https://zilliz.com/learn/Retrieval-Augmented-Generation)). LangGraph builds on top of LangChain with a graph-based architecture, which is better suited for stateful applications, complex decision-making, and multi-agent coordination.

In this guide, we’ll compare the two frameworks side by side: how they work, their strengths, and the types of projects they’re best suited for. By the end, you’ll have a clearer sense of which one makes the most sense for your needs.


## LangChain: Your Component Library and LCEL Orchestration Powerhouse

[**LangChain**](https://github.com/langchain-ai/langchain) is an open-source framework designed to make building LLM applications more manageable. You can think of it as the middleware that sits between your model (say, OpenAI’s [GPT-5](https://milvus.io/blog/gpt-5-review-accuracy-up-prices-down-code-strong-but-bad-for-creativity.md) or Anthropic’s [Claude](https://milvus.io/blog/claude-code-vs-gemini-cli-which-ones-the-real-dev-co-pilot.md)) and your actual app. Its main job is to help you _chain together_ all the moving parts: prompts, external APIs, [vector databases](https://zilliz.com/learn/what-is-vector-database), and custom business logic.

Take RAG as an example. Instead of wiring everything from scratch, LangChain gives you ready-made abstractions to connect an LLM with a vector store (like [Milvus](https://milvus.io/) or [Zilliz Cloud](https://zilliz.com/cloud)), run semantic search, and feed results back into your prompt. Beyond that, it offers utilities for prompt templates, agents that can call tools, and orchestration layers that keep complex workflows maintainable.

**What makes LangChain stand out?**

- **Rich component library** – Document loaders, text splitters, vector storage connectors, model interfaces, and more.

- **LCEL (LangChain Expression Language) orchestration** – A declarative way to mix and match components with less boilerplate.

- **Easy integration** – Works smoothly with APIs, databases, and third-party tools.

- **Mature ecosystem** – Strong documentation, examples, and an active community.


## LangGraph: Your Go-To for Stateful Agent Systems

[LangGraph](https://github.com/langchain-ai/langgraph) is a specialized extension of LangChain that focuses on stateful applications. Instead of writing workflows as a linear script, you define them as a graph of nodes and edges — essentially a state machine. Each node represents an action (like calling an LLM, querying a database, or checking a condition), while the edges define how the flow moves depending on the results. This structure makes it easier to handle loops, branching, and retries without your code turning into a tangle of if/else statements.

This approach is especially useful for advanced use cases such as copilots and [autonomous agents](https://zilliz.com/blog/what-exactly-are-ai-agents-why-openai-and-langchain-are-fighting-over-their-definition). These systems often need to keep track of memory, handle unexpected results, or make decisions dynamically. By modeling the logic explicitly as a graph, LangGraph makes these behaviors more transparent and maintainable.

**Core features of LangGraph include:**

- **Graph-based architecture** – Native support for loops, backtracking, and complex control flows.

- **State management** – Centralized state ensures context is preserved across steps.

- **Multi-agent support** – Built for scenarios where multiple agents collaborate or coordinate.

- **Debugging tools** – Visualization and debugging via LangSmith Studio to trace graph execution.


## LangChain vs LangGraph: Technical Deep Dive

### Architecture 

LangChain uses **LCEL (LangChain Expression Language)** to wire components together in a linear pipeline. It’s declarative, readable, and great for straightforward workflows like RAG.


```
# LangChain LCEL orchestration example
from langchain.prompts import ChatPromptTemplate
from langchain.chat_models import ChatOpenAI

prompt = ChatPromptTemplate.from_template("Please answer the following question: {question}")
model = ChatOpenAI()

# LCEL chain orchestration
chain = prompt | model

# Run the chain
result = chain.invoke({"question": "What is artificial intelligence?"})
```


LangGraph takes a different approach: workflows are expressed as a **graph of nodes and edges**. Each node defines an action, and the graph engine manages state, branching, and retries.

```
# LangGraph graph structure definition
from langgraph.graph import StateGraph
from typing import TypedDict

class State(TypedDict):
    messages: list
    current_step: str

def node_a(state: State) -> State:
    return {"messages": state["messages"] + ["Processing A"], "current_step": "A"}

def node_b(state: State) -> State:
    return {"messages": state["messages"] + ["Processing B"], "current_step": "B"}

graph = StateGraph(State)
graph.add_node("node_a", node_a)
graph.add_node("node_b", node_b)
graph.add_edge("node_a", "node_b")
```


Where LCEL gives you a clean linear pipeline, LangGraph natively supports loops, branching, and conditional flows. This makes it a stronger fit for **agent-like systems** or multi-step interactions that don’t follow a straight line.


### State Management

- **LangChain**: Uses Memory components for passing context. Works fine for simple multi-turn conversations or linear workflows.

- **LangGraph**: Uses a centralized state system that supports rollbacks, backtracking, and detailed history. Essential for long-running, stateful apps where context continuity matters.


### Execution Models

| **Feature**           | **LangChain**                  | **LangGraph**              |
| --------------------- | ------------------------------ | -------------------------- |
| Execution Mode        | Linear orchestration           | Stateful (Graph) Execution |
| Loop Support          | Limited Support                | Native Support             |
| Conditional Branching | Implemented via RunnableMap    | Native Support             |
| Exception Handling    | Implemented via RunnableBranch | Built-in Support           |
| Error Processing      | Chain-style Transmission       | Node-level Processing      |


## Real-World Use Cases: When to Use Each

Frameworks aren’t just about architecture — they shine in different situations. So the real question is: when should you reach for LangChain, and when does LangGraph make more sense? Let’s look at some practical scenarios.


### When LangChain Is Your Best Choice

#### 1. Straightforward Task Processing

LangChain is a great fit when you need to transform input into output without heavy state tracking or branching logic. For example, a browser extension that translates selected text:

```
# Implementing simple text translation using LCEL
from langchain.prompts import ChatPromptTemplate
from langchain.chat_models import ChatOpenAI

prompt = ChatPromptTemplate.from_template("Translate the following text to English: {text}")
model = ChatOpenAI()
chain = prompt | model

result = chain.invoke({"text": "Hello, World!"})
```


In this case, there’s no need for memory, retries, or multi-step reasoning — just efficient input-to-output transformation. LangChain keeps the code clean and focused.


#### 2. Foundation Components

LangChain provides rich basic components that can serve as building blocks for constructing more complex systems. Even when teams build with LangGraph, they often rely on LangChain’s mature components. The framework offers:

- **Vector store connectors** – Unified interfaces for databases like Milvus and Zilliz Cloud.

- **Document loaders & splitters** – For PDFs, web pages, and other content.

- **Model interfaces** – Standardized wrappers for popular LLMs.

This makes LangChain not only a workflow tool but also a reliable component library for larger systems.


### When LangGraph Is the Clear Winner

#### 1. Sophisticated Agent Development

LangGraph excels when you’re building advanced agent systems that need to loop, branch, and adapt. Here’s a simplified agent pattern:

```
# Simplified Agent system example
def agent(state):
    messages = state["messages"]
    # Agent thinks and decides next action
    action = decide_action(messages)
    return {"action": action, "messages": messages}

def tool_executor(state):
    # Execute tool calls
    action = state["action"]
    result = execute_tool(action)
    return {"result": result, "messages": state["messages"] + [result]}

# Build Agent graph
graph = StateGraph()
graph.add_node("agent", agent)
graph.add_node("tool_executor", tool_executor)
graph.add_edge("agent", "tool_executor")
graph.add_edge("tool_executor", "agent")
```


**Example:** GitHub Copilot X's advanced features perfectly demonstrate LangGraph's agent architecture in action. The system understands developer intent, breaks complex programming tasks into manageable steps, executes multiple operations in sequence, learns from intermediate results, and adapts its approach based on what it discovers along the way.


#### 2. Advanced Multi-Turn Conversation Systems

LangGraph's state management capabilities make it very suitable for building complex multi-turn conversation systems:

- **Customer service systems**: Capable of tracking conversation history, understanding context, and providing coherent responses

- **Educational tutoring systems**: Adjusting teaching strategies based on students' answer history

- **Interview simulation systems**: Adjusting interview questions based on candidates' responses

**Example:** Duolingo's AI tutoring system showcases this perfectly. The system continuously analyzes each learner's response patterns, identifies specific knowledge gaps, tracks learning progress across multiple sessions, and delivers personalized language learning experiences that adapt to individual learning styles, pace preferences, and areas of difficulty.


#### 3. Multi-Agent Collaboration Ecosystems

LangGraph natively supports ecosystems where multiple agents coordinate. Examples include:

- **Team collaboration simulation**: Multiple roles collaboratively completing complex tasks

- **Debate systems**: Multiple roles holding different viewpoints engaging in debate

- **Creative collaboration platforms**: Intelligent agents from different professional domains creating together

This approach has shown promise in research domains like drug discovery, where agents model different areas of expertise and combine results into new insights.


### Making the Right Choice: A Decision Framework

| **Project Characteristics**     | **Recommended Framework** | **Why**                                    |
| ------------------------------- | ------------------------- | ------------------------------------------ |
| Simple One-Time Tasks           | LangChain                 | LCEL orchestration is simple and intuitive |
| Text Translation/Optimization   | LangChain                 | No need for complex state management       |
| Agent Systems                   | LangGraph                 | Powerful state management and control flow |
| Multi-Turn Conversation Systems | LangGraph                 | State tracking and context management      |
| Multi-Agent Collaboration       | LangGraph                 | Native support for multi-node interaction  |
| Systems Requiring Tool Usage    | LangGraph                 | Flexible tool invocation flow control      |


## Conclusion

In most cases, LangChain and LangGraph are complementary, not competitors. LangChain gives you a solid foundation of components and LCEL orchestration — great for quick prototypes, stateless tasks, or projects that just need clean input-to-output flows. LangGraph steps in when your application outgrows that linear model and requires state, branching, or multiple agents working together.

- **Choose LangChain** if your focus is on straightforward tasks like text translation, document processing, or data transformation, where each request stands on its own.

- **Choose LangGraph** if you’re building multi-turn conversations, agent systems, or collaborative agent ecosystems where context and decision-making matter.

- **Mix both** for the best results. Many production systems start with LangChain’s components (document loaders, vector store connectors, model interfaces) and then add LangGraph to manage stateful, graph-driven logic on top.

Ultimately, it’s less about chasing trends and more about aligning the framework with your project’s genuine needs. Both ecosystems are evolving rapidly, driven by active communities and robust documentation. By understanding where each fits, you’ll be better equipped to design applications that scale — whether you’re building your first RAG pipeline with Milvus or orchestrating a complex multi-agent system.


LangChain vs LangGraph: A Developer's Guide to Choosing Your AI Frameworks



Nano Banana is going viral on social media right now and for a good reason! You’ve probably seen the images it generates or even tried it yourself. It’s a latest image generation model that turns plain text into collectible-grade figurine shots with startling precision and speed. 

Type in something like _“swap Elon’s hat and skirt,”_ and in about 16 seconds, you’ll get a photoreal result: shirt tucked, colors blended, accessories in place—no manual edits. No lag.

![](https://assets.zilliz.com/beach_side_668179b830.png)

I couldn’t resist testing it either. My prompt was:

_“Use the Nano Banana model to create a 1/7 scale commercialized figure of the character in the illustration, in a realistic style and environment. Place the figure on a computer desk, using a circular transparent acrylic base without any text. On the computer screen, display the ZBrush modeling process of the figure. Next to the screen, place a Bandai-style toy packaging box printed with the original artwork.”_

![](https://assets.zilliz.com/me_with_a_dress_506a0ebf39.png)

The result blew me away—it looked like a production prototype straight from a convention booth.

It’s no surprise that teams are already finding serious use cases for it. One of our customers, a mobile entertainment platform featuring gacha and dress-up gameplay, is developing a feature that allows players to upload photos and instantly dress up their avatars with in-game accessories. E-commerce brands are experimenting with “shoot once, reuse forever”: capturing a base model image and then generating endless outfit and hairstyle variations with AI, instead of re-shooting 20 times in the studio.

But here’s the catch—image generation alone doesn’t solve the whole problem. These systems also need **smart retrieval**: the ability to instantly find the right outfits, props, and visual elements from massive, unstructured media libraries. Without that, the generative model is guessing in the dark. What companies really need is a **multimodal RAG (retrieval-augmented generation) system**—where Nano Banana handles the creativity, and a powerful vector database handles the context.

That’s where **Milvus** comes in. As an open-source vector database, Milvus can index and search across billions of embeddings—images, text, audio, and beyond. Paired with Nano Banana, it becomes the backbone of a production-ready multimodal RAG pipeline: search, match, and generate at enterprise scale.

In the rest of this blog, we’ll walk through how to combine Nano Banana and Milvus to build an enterprise-ready multimodal RAG system—and why this pairing unlocks the next wave of AI applications.

## Building a Text-to-Image Retrieval Engine

For fast-moving consumer goods brands, game studios, and media companies, the bottleneck in AI image generation isn’t the model—it’s the mess.

Their archives are a swamp of unstructured data, including product shots, character assets, promotional videos, and outfit renders. And when you need to find “the red cape from last season’s Lunar drop,” good luck—traditional keyword-based search can’t handle that.

The solution? Build a **text-to-image retrieval system**.

Here’s the play: use [CLIP](https://openai.com/research/clip?utm_source=chatgpt.com) to embed both text and image data into vectors. Store those vectors in **Milvus**, the open-source vector database purpose-built for similarity search. Then, when a user types a description (“red silk cape with gold trim”), you hit the DB and return the top 3 most semantically similar images.

It’s fast. It’s scalable. And it turns your messy media library into a structured, queryable asset bank.

Here’s how to build it:

Install Dependencies

```
# Install necessary packages
%pip install --upgrade pymilvus pillow matplotlib
%pip install git+https://github.com/openai/CLIP.git
```
Import Necessary Libraries
```
import os
import clip
import torch
from PIL import Image
import matplotlib.pyplot as plt
from pymilvus import MilvusClient
from glob import glob
import math

print("All libraries imported successfully!")
```
Initialize Milvus Client
```
# Initialize Milvus client
milvus_client = MilvusClient(uri="http://localhost:19530",token="root:Miluvs")
print("Milvus client initialized successfully!")
```
Load CLIP Model
```
# Load CLIP model
model_name = "ViT-B/32"
device = "cuda" if torch.cuda.is_available() else "cpu"
model, preprocess = clip.load(model_name, device=device)
model.eval()

print(f"CLIP model '{model_name}' loaded successfully, running on device: {device}")
print(f"Model input resolution: {model.visual.input_resolution}")
print(f"Context length: {model.context_length}")
print(f"Vocabulary size: {model.vocab_size}")
```
Output result:
```
CLIP model `ViT-B/32` loaded successfully, running on: cpu
 Model input resolution: 224
 Context length: 77
 Vocabulary size: 49,408
```
Define Feature Extraction Functions
```
def encode_image(image_path):
    """Encode image into normalized feature vector"""
    try:
        image = preprocess(Image.open(image_path)).unsqueeze(0).to(device)
        
        with torch.no_grad():
            image_features = model.encode_image(image)
            image_features /= image_features.norm(dim=-1, keepdim=True)  # Normalize
        
        return image_features.squeeze().cpu().tolist()
    except Exception as e:
        print(f"Error processing image {image_path}: {e}")
        return None
def encode_text(text):
    """Encode text into normalized feature vector"""
    text_tokens = clip.tokenize([text]).to(device)
    
    with torch.no_grad():
        text_features = model.encode_text(text_tokens)
        text_features /= text_features.norm(dim=-1, keepdim=True)  # Normalize
    
    return text_features.squeeze().cpu().tolist()

print("Feature extraction functions defined successfully!")
```
Create Milvus Collection
```
collection_name = "production_image_collection"
# If collection already exists, delete it
if milvus_client.has_collection(collection_name):
    milvus_client.drop_collection(collection_name)
    print(f"Existing collection deleted: {collection_name}")

# Create new collection
milvus_client.create_collection(
    collection_name=collection_name,
    dimension=512,  # CLIP ViT-B/32 embedding dimension
    auto_id=True,  # Auto-generate ID
    enable_dynamic_field=True,  # Enable dynamic fields
    metric_type="COSINE"  # Use cosine similarity
)

print(f"Collection '{collection_name}' created successfully!")
print(f"Collection info: {milvus_client.describe_collection(collection_name)}")
```
Collection creation success output:

```
Existing collection deleted: production_image_collection
Collection 'production_image_collection' created successfully!
Collection info: {'collection_name': 'production_image_collection', 'auto_id': True, 'num_shards': 1, 'description': '', 'fields': [{'field_id': 100, 'name': 'id', 'description': '', 'type': <DataType.INT64: 5>, 'params': {}, 'auto_id': True, 'is_primary': True}, {'field_id': 101, 'name': 'vector', 'description': '', 'type': <DataType.FLOAT_VECTOR: 101>, 'params': {'dim': 512}}, {'field_id': 102, 'name': 'function': [], 'aliases': [], 'collection_id': 460508990706033544, 'consistency_level': 2, 'properties': {}, 'num_partitions': 1, 'enable_dynamic_field': True, 'created_timestamp': 460511723827494913, 'updated_timestamp': 460511723827494913}
```
Process and Insert Images
```
# Set image directory path
image_dir = "./production_image"
raw_data = []

# Get all supported image formats
image_extensions = ['*.jpg', '*.jpeg', '*.png', '*.JPEG', '*.JPG', '*.PNG']
image_paths = []

for ext in image_extensions:
    image_paths.extend(glob(os.path.join(image_dir, ext)))

print(f"Found {len(image_paths)} images in {image_dir}")

# Process images and generate embeddings
successful_count = 0
for i, image_path in enumerate(image_paths):
    print(f"Processing progress: {i+1}/{len(image_paths)} - {os.path.basename(image_path)}")
    
    image_embedding = encode_image(image_path)
    if image_embedding is not None:
        image_dict = {
            "vector": image_embedding,
            "filepath": image_path,
            "filename": os.path.basename(image_path)
        }
        raw_data.append(image_dict)
        successful_count += 1

print(f"Successfully processed {successful_count} images")
```
Image processing progress output:
```
Found 50 images in ./production_image
Processing progress: 1/50 - download (5).jpeg
Processing progress: 2/50 - images (2).jpeg
Processing progress: 3/50 - download (23).jpeg
Processing progress: 4/50 - download.jpeg
Processing progress: 5/50 - images (14).jpeg
Processing progress: 6/50 - images (16).jpeg
…
Processing progress: 44/50 - download (10).jpeg
Processing progress: 45/50 - images (18).jpeg
Processing progress: 46/50 - download (9).jpeg
Processing progress: 47/50 - download (12).jpeg
Processing progress: 48/50 - images (1).jpeg
Processing progress: 49/50 - download.png
Processing progress: 50/50 - images.png
Successfully processed 50 images
```
Insert Data into Milvus
```
# Insert data into Milvus
if raw_data:
    print("Inserting data into Milvus...")
    insert_result = milvus_client.insert(collection_name=collection_name, data=raw_data)
    
    print(f"Successfully inserted {insert_result['insert_count']} images into Milvus")
    print(f"Sample inserted IDs: {insert_result['ids'][:5]}...")  # Show first 5 IDs
else:
    print("No successfully processed image data to insert")
```

Define Search and Visualization Functions
```
def search_images_by_text(query_text, top_k=3):
    """Search images based on text query"""
    print(f"Search query: '{query_text}'")
    
    # Encode query text
    query_embedding = encode_text(query_text)
    
    # Search in Milvus
    search_results = milvus_client.search(
        collection_name=collection_name,
        data=[query_embedding],
        limit=top_k,
        output_fields=["filepath", "filename"]
    )
    
    return search_results[0]


def visualize_search_results(query_text, results):
    """Visualize search results"""
    num_images = len(results)
    
    if num_images == 0:
        print("No matching images found")
        return
    
    # Create subplots
    fig, axes = plt.subplots(1, num_images, figsize=(5*num_images, 5))
    fig.suptitle(f'Search Results: "{query_text}" (Top {num_images})', fontsize=16, fontweight='bold')
    
    # Handle single image case
    if num_images == 1:
        axes = [axes]
    
    # Display images
    for i, result in enumerate(results):
        try:
            img_path = result['entity']['filepath']
            filename = result['entity']['filename']
            score = result['distance']
            
            # Load and display image
            img = Image.open(img_path)
            axes[i].imshow(img)
            axes[i].set_title(f"{filename}\nSimilarity: {score:.3f}", fontsize=10)
            axes[i].axis('off')
            
            print(f"{i+1}. File: {filename}, Similarity score: {score:.4f}")
            
        except Exception as e:
            axes[i].text(0.5, 0.5, f'Error loading image\n{str(e)}',
                        ha='center', va='center', transform=axes[i].transAxes)
            axes[i].axis('off')
    
    plt.tight_layout()
    plt.show()

print("Search and visualization functions defined successfully!")
```

Execute Text-to-Image Search

```
# Example search 1
query1 = "a golden watch"
results1 = search_images_by_text(query1, top_k=3)
visualize_search_results(query1, results1)
```




Search query execution output:

```
Search query: 'a golden watch'
1. File: images (19).jpeg, Similarity score: 0.2934
2. File: download (26).jpeg, Similarity score: 0.3073
3. File: images (17).jpeg, Similarity score: 0.2717
```


![](https://assets.zilliz.com/watch_067c39ba51.png)

## Using Nano-banana to Create Brand Promotional Images


Now that we have our text-to-image search system working with Milvus, let's integrate Nano-banana to generate new promotional content based on the assets we retrieve.

Install Google SDK
```
%pip install google-generativeai
%pip install requests
print("Google Generative AI SDK installation complete!")
```

Configure Gemini API

```
import google.generativeai as genai
from PIL import Image
from io import BytesIO
genai.configure(api_key="<your_api_key>")
```
Generate New Images


```
prompt = (
    "An European male model wearing a suit, carrying a gold watch."
)

image = Image.open("/path/to/image/watch.jpg")

model = genai.GenerativeModel('gemini-2.5-flash-image-preview')
response = model.generate_content([prompt, image])

for part in response.candidates[0].content.parts:
    if part.text is not None:
        print(part.text)
    elif part.inline_data is not None:
        image = Image.open(BytesIO(part.inline_data.data))
        image.save("generated_image.png")
        image.show()
```

![](https://assets.zilliz.com/suit_976b6f1df2.png)

## What This Means for Your Development Workflow

As a developer, this Milvus + Nano-banana integration fundamentally changes how you can approach content generation projects. Instead of managing static asset libraries or relying on expensive creative teams, you now have a dynamic system that retrieves and generates exactly what your application needs in real-time.

Consider the following recent client scenario: a brand launched several new products but opted to skip the traditional photo shoot pipeline entirely. Using our integrated system, they could instantly generate promotional imagery by combining their existing product database with Nano-banana's generation capabilities.


*Prompt: A model is wearing these products on the beach*

![](https://assets.zilliz.com/model_5a2a042b46.png)

The real power becomes apparent when you need to create complex, multi-variant content that would traditionally require extensive coordination between photographers, models, and set designers. With Milvus handling the asset retrieval and Nano-banana managing the generation, you can programmatically create sophisticated scenes that adapt to your specific requirements:



*Prompt: A model is posing and leaning against a blue convertible sports car. She is wearing a halter top dress and the accompanying accessories. She is adorned with a diamond necklace and a blue watch, wearing high heels on her feet and holding a labubu pendant in her hand.*

![](https://assets.zilliz.com/shoes_98e1e4c70b.png)

For developers working in gaming or collectibles, this system opens up entirely new possibilities for rapid prototyping and concept validation. Instead of investing weeks in 3D modeling before knowing if a concept works, you can now generate photorealistic product visualizations that include packaging, environmental context, and even manufacturing processes:

*Prompt: Use the nano-banana model to create a 1/7 scale commercialized figure of the character in the illustration, in a realistic style and environment. Place the figure on a computer desk, using a circular transparent acrylic base without any text. On the computer screen, display the ZBrush modeling process of the figure. Next to the computer screen, place a BANDAI-style toy packaging box printed with the original artwork.*

![](https://assets.zilliz.com/milvus_3d_5189d53773.png)

## Conclusion

From a technical perspective, Nano Banana is more than a novelty—it’s production-ready in ways that matter to developers. Its biggest strength is consistency and controllability, which means fewer edge cases bleeding into your application logic. Just as importantly, it handles the subtle details that often derail automated pipelines: keeping brand colors consistent, generating physically plausible lighting and reflections, and ensuring visual coherence across multiple output formats.

The real magic happens when you combine it with the Milvus vector database. A vector database doesn’t just store embeddings—it becomes an intelligent asset manager that can surface the most relevant historical content to guide new generations. The result: faster generation times (because the model has better context), higher consistency across your application, and the ability to enforce brand or style guidelines automatically. 

In short, Milvus transforms Nano Banana from a creative toy into a scalable enterprise system.

Of course, no system is flawless. Complex, multi-step instructions can still cause hiccups, and lighting physics sometimes stretches reality more than you’d like. The most reliable solution we’ve seen is to supplement text prompts with reference images stored in Milvus, which gives the model richer grounding, more predictable results, and shorter iteration cycles. With this setup, you’re not just experimenting with multimodal RAG—you’re running it in production with confidence.


Nano Banana + Milvus: Turning Hype into Enterprise-Ready Multimodal RAG


Hybrid Document Retrieval System with LangExtract + Milvus


LangExtract + Milvus: A Practical Guide to Building a Hybrid Document Processing and Search System


Why I’m Against Claude Code’s Grep-Only Retrieval? It Just Burns Too Many Tokens



Remember when working with data meant crafting SQL queries for exact matches? Those days are long gone. We’ve entered the era of AI and semantic search, where AI doesn’t just match keywords—it understands intent. And at the heart of this shift are vector databases: the engines powering today’s most advanced applications, from ChatGPT’s retrieval systems to Netflix’s personalized recommendations to Tesla’s autonomous driving stack.

But here's the plot twist: not all [vector databases ](https://zilliz.com/learn/what-is-vector-database)are created equal. 

Your RAG application needs lightning-fast semantic retrieval across billions of documents. Your recommendation system demands sub-millisecond responses under crushing traffic loads. Your computer vision pipeline requires handling exponentially growing image datasets without breaking the bank.

Meanwhile, the market is flooded with options: Elasticsearch, Milvus, PGVector, Qdrant, and even AWS’s new S3 Vector. Each claims to be the best—but the best for what? Choosing wrong could mean wasted months of engineering, runaway infrastructure costs, and a serious hit to your product’s competitive edge.

That’s where this guide comes in. Instead of vendor hype, we’ll walk through a practical decision framework across three critical dimensions: functionality, performance, and ecosystem. By the end, you’ll have the clarity to choose the database that’s not just “popular,” but the one that’s right for your use case.


## 1. Functionality: Can It Handle Your AI Workload?

When choosing a vector database, functionality is the foundation. It’s not just about storing vectors—it’s about whether the system can support the diverse, large-scale, and often messy requirements of real-world AI workloads. You’ll need to evaluate both core vector capabilities and enterprise-grade features that determine long-term viability.


### Complete Vector Data Type Support

Different AI tasks generate different kinds of vectors—text, images, audio, and user behavior. A production system often needs to handle them all at once. Without full support for multiple vector types, your database won’t even make it past day one.

Take an e-commerce product search as an example:

- Product images → dense vectors for visual similarity and image-to-image search.

- Product descriptions → sparse vectors for keyword matching and full-text retrieval.

- User behavior patterns (clicks, purchases, favorites) → binary vectors for fast matching of interests.

On the surface, it looks like “search,” but under the hood, it’s a multi-vector, multimodal retrieval problem.

![](https://assets.zilliz.com/20250822_192755_c6c0842b05.png)


### Rich Indexing Algorithms with Fine-Grained Control

Every workload forces a trade-off between recall, speed, and cost—the classic “impossible triangle.” A robust vector database should offer multiple indexing algorithms so you can choose the right compromise for your use case:

- Flat → highest accuracy, at the cost of speed.

- IVF → scalable, high-performance retrieval for large datasets.

- HNSW → strong balance between recall and latency.

Enterprise-grade systems also go further with:

- Disk-based indexing for petabyte-scale storage at lower cost.

- GPU acceleration for ultra-low-latency inference.

- Granular parameter tuning so teams can optimize every query path to business requirements.

The best systems also provide granular parameter tuning, letting you squeeze optimal performance from limited resources and fine-tune indexing behavior to match your specific business requirements.


### Comprehensive Retrieval Methods

Top-K similarity search is table stakes. Real applications demand more sophisticated retrieval strategies, such as filtering retrieval (price ranges, stock status, thresholds), grouping retrieval (category diversity, e.g., dresses vs. skirts vs. suits), and hybrid retrieval (combining sparse text with dense image embeddings as well as full-text search).

For example, a simple “show me dresses” request on an e-commerce site may trigger:

1. Similarity retrieval on product vectors (image + text).

2. Scalar filtering for price and stock availability.

3. Diversity optimization to surface varied categories.

4. Hybrid personalization blending user profile embeddings with purchase history.

What looks like a simple recommendation is actually powered by a retrieval engine with layered, complementary capabilities.

![](https://assets.zilliz.com/recsyc_da5d86d6f4.png)


### Enterprise-Grade Architecture

Unstructured data is exploding. According to IDC, by 2027 it will reach 246.9 zettabytes—an astonishing 86.8% of all global data. Once you start processing that volume through AI models, you’re dealing with astronomical amounts of vector data that only grow faster over time.

A vector database built for hobby projects won’t survive this curve. To succeed at enterprise scale, you need a database with cloud-native flexibility and scalability baked in. That means:

- Elastic scaling to handle unpredictable spikes in workload.

- Multi-tenant support so teams and applications can share infrastructure securely.

- Seamless integration with Kubernetes and cloud services for automated deployment and scaling.

And because downtime is never acceptable in production, resilience is just as critical as scalability. Enterprise-ready systems should provide:

- High availability with automatic failover.

- Multi-replica disaster recovery across regions or zones.

- Self-healing infrastructure that detects and corrects failures without human intervention.

In short: handling vectors at scale isn’t just about fast queries—it’s about an architecture that grows with your data, protects against failure, and stays cost-efficient at enterprise volumes.


## 2. Performance: Will It Scale When Your App Goes Viral?

Once functionality is covered, performance becomes the make-or-break factor. The right database must not only handle today’s workloads but also scale gracefully when traffic spikes. Evaluating performance means looking at multiple dimensions—not just raw speed.


### Key Performance Metrics

The Complete Vector Database Evaluation Framework covers:

- Latency (P50, P95, P99) → captures both average and worst-case response times.

- Throughput (QPS) → measures concurrency under real-world loads.

- Accuracy (Recall@K) → ensures approximate search still returns relevant results.

- Data scale adaptability → tests performance at millions, tens of millions, and billions of records.

Beyond Basic Metrics: In production, you’ll also want to measure:

- Filtered query performance across varying ratios (1%–99%).

- Streaming workloads with continuous inserts + real-time queries.

- Resource efficiency (CPU, memory, disk I/O) to ensure cost-effectiveness.


### Benchmarking in Practice

While[ ANN-Benchmark](http://ann-benchmarks.com/) offers widely-recognized algorithm-level evaluation, it focuses on underlying algorithm libraries and misses dynamic scenarios. The datasets feel outdated, and the use cases are too simplified for production environments.

For real-world vector database evaluation, we recommend the open-source[ VDBBench](https://github.com/zilliztech/VectorDBBench), which tackles the complexities of production testing with comprehensive scenario coverage.

A solid VDBBench testing approach follows three essential steps:

- Determine use scenarios by selecting appropriate datasets (like SIFT1M or GIST1M) and business scenarios (TopK retrieval, filtered retrieval, concurrent write-and-read operations)

- Configure database and VDBBench parameters to ensure fair, reproducible testing environments

- Execute and analyze tests through the web interface to automatically collect performance metrics, compare results, and make data-driven selection decisions

For more information about how to benchmark a vector database with real-world workloads, check this tutorial: [How to Evaluate VectorDBs that Match Production via VDBBench ](https://milvus.io/blog/hands-on-with-vdbbench-benchmarking-vector-databases-for-pocs-that-match-production.md)


## 3. Ecosystem: Is It Ready for Production Reality?

A vector database doesn’t live in isolation. Its ecosystem determines how easy it is to adopt, how quickly it scales, and whether it can survive in production over the long run. When evaluating, it helps to look at four key dimensions. 


(1) Fit with the AI Ecosystem

A top-tier and production-ready vector database should plug directly into the AI tools you already use. That means:

- Native support for mainstream LLMs (OpenAI, Claude, Qwen) and embedding services.

- Compatibility with development frameworks like LangChain, LlamaIndex, and Dify, so you can build RAG pipelines, recommendation engines, or Q&A systems without fighting the stack.

- Flexibility in handling vectors from multiple sources—text, images, or custom models.

(2) Tooling That Supports Daily Operations

The best vector database in the world won’t succeed if it’s painful to operate. Look for a vector database that is seamlessly compatible with the surrounding tool ecosystem that covers:

- Visual dashboards for managing data, monitoring performance, and handling permissions.

- Backup & recovery with both full and incremental options.

- Capacity planning tools that help forecast resources and scale clusters efficiently.

- Diagnostics & tuning for log analysis, bottleneck detection, and troubleshooting.

- Monitoring & alerts via standard integrations like Prometheus and Grafana.

These aren’t “nice to haves”—they’re what keep your system stable at 2 a.m. when traffic spikes.

(3) Open Source + Commercial Balance

Vector databases are still evolving. Open source brings speed and community feedback, but large-scale projects also need sustainable commercial backing. The most successful data platforms—think Spark, MongoDB, Kafka—all balance open innovation with strong companies behind them.

Commercial offerings should also be cloud-neutral: elastic, low-maintenance, and flexible enough to meet different business needs across industries and geographies.

(4) Proof in Real Deployments

Marketing slides don’t mean much without real customers. A credible vector database should have case studies across industries—finance, healthcare, manufacturing, internet, legal—and across use cases like search, recommendation, risk control, customer support, and quality inspection.

If your peers are already succeeding with it, that’s the best sign you can. And when in doubt, nothing beats running a proof of concept with your own data.


## Milvus: The Most Popular Open-Source Vector Database

If you’ve applied the evaluation framework—functionality, performance, ecosystem—you’ll find only a few vector databases that consistently deliver across all three dimensions. [Milvus](https://milvus.io/) is one of them.

Born as an open-source project and backed by [Zilliz](https://zilliz.com/), [Milvus](https://milvus.io/) is purpose-built for AI-native workloads. It combines advanced indexing and retrieval with enterprise-grade reliability, while still being approachable for developers building RAG, AI Agents, recommendation engines, or semantic search systems. With [36K+ GitHub](https://github.com/milvus-io/milvus) stars and adoption by more than 10,000 enterprise companies, Milvus has become the most popular open-source vector database in production today.

Milvus also provides multiple [deployment options](https://milvus.io/docs/install-overview.md), all under a single API:

- **Milvus Lite** → lightweight version for rapid experimentation and prototyping.

- **Standalone** → simple production deployments.

- **Cluster** → distributed deployments that scale to billions of vectors.

This deployment flexibility means teams can start small and scale seamlessly—without rewriting a single line of code.

Key capabilities at a glance: 

- 🔎**Comprehensive functionality** → Multimodal vector support (text, image, audio, etc.), multiple indexing methods (IVF, HNSW, disk-based, GPU acceleration), and advanced retrieval (hybrid, filtered, grouped, and full-text search).

- ⚡**Proven performance** → Tuned for billion-scale datasets, with adjustable indexing and benchmarking via tools like VDBBench.

- 🌐**Robust ecosystem** → Tight integrations with LLMs, embeddings, and frameworks like LangChain, LlamaIndex, and Dify. Includes a full operational toolchain for monitoring, backup, recovery, and capacity planning.

- 🛡️**Enterprise ready** → High availability, multi-replica disaster recovery, RBAC, observability, plus **Zilliz Cloud** for fully managed, cloud-neutral deployments.

Milvus gives you the flexibility of open source, the scale and reliability of enterprise systems, and the ecosystem integrations needed to move fast in AI development. It’s no surprise that it has become the go-to vector database for both startups and global enterprises.


### If You Want Zero Hassle—Try Zilliz Cloud (Managed Milvus)

Milvus is open source and always free to use. But if you’d rather focus on innovation instead of infrastructure, consider [Zilliz Cloud](https://zilliz.com/cloud)—the fully managed Milvus service built by the original Milvus team. It gives you everything you love about Milvus, plus advanced enterprise-grade features, without the operational overhead.

Why Teams Choose Zilliz Cloud? Key capabilities at a glance: 

- ⚡ **Deploy in minutes, scale automatically** 

- 💰 **Pay only for what you use** 

- 💬 **Natural language querying** 

- 🔒 **Enterprise-grade security**

- 🌍 **Global scale, local performance** 

- 📈 **99.95% uptime SLA**

For startups and enterprises alike, the value is clear: your technical teams should spend their time building products, not managing databases. Zilliz Cloud takes care of the scaling, security, and reliability—so you can pay 100% of your effort on delivering breakthrough AI applications.


## Choose Wisely: Your Vector Database Will Shape Your AI Future

Vector databases are evolving at breakneck speed, with new features and optimizations emerging almost monthly. The framework we’ve outlined—functionality, performance, and ecosystem—gives you a structured way to cut through the noise and make informed decisions today. But adaptability is just as important, since the landscape will keep shifting.

The winning approach is systematic evaluation backed by hands-on testing. Use the framework to narrow your choices, then validate with a proof-of-concept on your own data and workloads. That combination of rigor and real-world validation is what separates successful deployments from costly mistakes.

As AI applications grow more sophisticated and data volumes surge, the vector database you choose now will likely become a cornerstone of your infrastructure. Investing the time to evaluate thoroughly today will pay off in performance, scalability, and team productivity tomorrow.

In the end, the future belongs to teams that can harness semantic search effectively. Choose your vector database wisely—it may be the competitive advantage that sets your AI applications apart.


Guide | How to Choose the Right VectorDB for Your AI Apps


A Practical Guide for Choosing the Right Vector Database for Your AI Applications



## Introduction

Vibe Coding is transforming how we write software. Tools like Cursor and Windsurf are making development feel effortless and intuitive—ask for a function and get a snippet, need a quick API call, and it's generated before you finish typing. The promise is smooth, seamless development where your AI assistant anticipates your needs and delivers exactly what you want.

But there's a critical flaw breaking this beautiful flow: AI assistants frequently generate outdated code that breaks in production.

Consider this example: I asked Cursor to generate Milvus connection code, and it produced this:

```
connections.connect("default", host="localhost", port="19530")
```

This used to work perfectly, but the current pymilvus SDK recommends using `MilvusClient` for all connections and operations. The old method is no longer considered best practice, yet AI assistants continue to suggest it because their training data is often months or years out of date.

Despite all the progress in Vibe Coding tools, developers still spend significant time bridging the “last mile” between generated code and production-ready solutions. The vibe is there, but the accuracy isn’t.


### What is the Milvus SDK Code Helper?

The **Milvus SDK Code Helper** is a developer-focused solution that solves the _“last mile”_ problem in Vibe Coding—bridging the gap between AI-assisted coding and production-ready Milvus applications.

At its core, it’s a **Model Context Protocol (MCP) server** that connects your AI-powered IDE directly to the latest official Milvus documentation. Combined with Retrieval-Augmented Generation (RAG), it ensures the code your assistant generates is always accurate, up-to-date, and aligned with the Milvus best practices.

Instead of outdated snippets or guesswork, you get context-aware, standards-compliant code suggestions—right inside your development workflow.

**Key Benefits:**

- ⚡ **Configure once, boost efficiency forever**: Set it up once and enjoy consistently updated code generation

- 🎯 **Always current**: Access to the latest official Milvus SDK documentation

- 📈 **Improved code quality**: Generate code that follows current best practices

- 🌊 **Restored flow**: Keep your Vibe Coding experience smooth and uninterrupted

**Three Tools in One**

1. `pymilvus-code-generator` → Quickly write Python code for common Milvus tasks (e.g., creating collections, inserting data, running vector searches).

2. `orm-client-code-converter` → Modernize legacy Python code by replacing outdated ORM patterns with the latest `MilvusClient` syntax.

3. `language-translator` → Seamlessly convert Milvus SDK code between languages (e.g., Python ↔ TypeScript).

Check the resources below for more details: 

- Blog: [Why Your Vibe Coding Generates Outdated Code and How to Fix It with Milvus MCP ](https://milvus.io/blog/why-vibe-coding-generate-outdated-code-and-how-to-fix-it-with-milvus-mcp.md)

- Doc: [Milvus SDK Code Helper Guide | Milvus Documentation](https://milvus.io/docs/milvus-sdk-helper-mcp.md#Quickstart)


### Before You Begin

Before diving into the setup process, let's examine the dramatic difference the Code Helper makes in practice. The comparison below shows how the same request for creating a Milvus collection produces completely different results:

| **MCP Code Helper Enabled:**                                                                                                                                                                                              | **MCP Code Helper Disabled:**                                                                                                                                                                                             |
| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| ![](https://assets.zilliz.com/mcp_enabled_fcb94737fb.png) | ![](https://assets.zilliz.com/mcp_disabled_769db4faee.png) |

This perfectly illustrates the core problem: without the Code Helper, even the most advanced AI assistants generate code using outdated ORM SDK patterns that are no longer recommended. The Code Helper ensures you get the most current, efficient, and officially endorsed implementation every time.

**The Difference in Practice:**

- **Modern approach**: Clean, maintainable code using current best practices

- **Deprecated approach**: Code that works but follows outdated patterns

- **Production impact**: Current code is more efficient, easier to maintain, and future-proof

This guide will walk you through setting up the Milvus SDK Code Helper across multiple AI IDEs and development environments. The setup process is straightforward and typically takes just a few minutes per IDE.


## Setting Up the Milvus SDK Code Helper

The following sections provide detailed setup instructions for each supported IDE and development environment. Choose the section that corresponds to your preferred development setup.


### Cursor IDE Setup

Cursor offers seamless integration with MCP servers through its built-in configuration system.

**Step 1: Access MCP Settings** 

Navigate to: Settings → Cursor Settings → Tools & Integrations → Add new global MCP server

![](https://assets.zilliz.com/cursor_mcp_configuration_interface_9ff0b7dcb7.png) _Cursor MCP configuration interface_

**Step 2: Configure the MCP Server** 

You have two options for configuration:

**Option A: Global Configuration (Recommended)** 

Add the following configuration to your Cursor `~/.cursor/mcp.json` file:

```
{
  "mcpServers": {
    "sdk-code-helper": {
      "url": "https://sdk.milvus.io/mcp/",
      "headers": {
        "Accept": "text/event-stream"
      }
    }
  }
}
```

**Option B: Project-Specific Configuration** 

Create a `.cursor/mcp.json` file in your project folder with the same configuration above.

For additional configuration options and troubleshooting, refer to the[ Cursor MCP documentation](https://docs.cursor.com/context/model-context-protocol).


### Claude Desktop Setup

Claude Desktop provides straightforward MCP integration through its configuration system.

**Step 1: Locate Configuration File** 

Add the following configuration to your Claude Desktop configuration file:

```
{
  "mcpServers": {
    "sdk-code-helper": {
      "url": "https://sdk.milvus.io/mcp/",
      "headers": {
        "Accept": "text/event-stream"
      }
    }
  }
}
```

**Step 2: Restart Claude Desktop** 

After saving the configuration, restart Claude Desktop to activate the new MCP server.


### Claude Code Setup

Claude Code offers command-line configuration for MCP servers, making it ideal for developers who prefer terminal-based setup.

**Step 1: Add MCP Server via Command Line** 

Execute the following command in your terminal:

```
claude mcp add-json sdk-code-helper --json '{
  "url": "https://sdk.milvus.io/mcp/",
  "headers": {
    "Accept": "text/event-stream"
  }
}
```

**Step 2: Verify Installation** 

The MCP server will be automatically configured and ready for use immediately after running the command.


### Windsurf IDE Setup

Windsurf supports MCP configuration through its JSON-based settings system.

**Step 1: Access MCP Settings** 

Add the following configuration to your Windsurf MCP settings file:

```
{
  "mcpServers": {
    "sdk-code-helper": {
      "url": "https://sdk.milvus.io/mcp/",
      "headers": {
        "Accept": "text/event-stream"
      }
    }
  }
}
```

**Step 2: Apply Configuration** 

Save the settings file and restart Windsurf to activate the MCP server.


### VS Code Setup

VS Code integration requires an MCP-compatible extension to function properly.

**Step 1: Install MCP Extension** 

Ensure you have an MCP-compatible extension installed in VS Code.

**Step 2: Configure MCP Server** 

Add the following configuration to your VS Code MCP settings:

```
{
  "mcpServers": {
    "sdk-code-helper": {
      "url": "https://sdk.milvus.io/mcp/",
      "headers": {
        "Accept": "text/event-stream"
      }
    }
  }
}
```


### Cherry Studio Setup

Cherry Studio provides a user-friendly graphical interface for MCP server configuration, making it accessible for developers who prefer visual setup processes.

**Step 1: Access MCP Server Settings** 

Navigate to Settings → MCP Servers → Add Server through the Cherry Studio interface.

**Step 2: Configure Server Details** 

Fill in the server configuration form with the following information:

- **Name**: `sdk code helper`

- **Type**: `Streamable HTTP`

- **URL**: `https://sdk.milvus.io/mcp/`

- **Headers**: `"Accept": "text/event-stream"`

**Step 3: Save and Activate** 

Click Save to activate the server configuration.

![](https://assets.zilliz.com/cherry_studio_mcp_configuration_interface_b7dce8b26d.png) 

_Cherry Studio MCP configuration interface_


### Cline Setup

Cline uses a JSON-based configuration system accessible through its interface.

**Step 1: Access MCP Settings**

1. Open Cline and click on the MCP Servers icon in the top navigation bar

2. Select the Installed tab

3. Click Advanced MCP Settings

**Step 2: Edit Configuration File** In the `cline_mcp_settings.json` file, add the following configuration:

```
{
  "mcpServers": {
    "sdk-code-helper": {
      "url": "https://sdk.milvus.io/mcp/",
      "headers": {
        "Accept": "text/event-stream"
      }
    }
  }
}
```

**Step 3: Save and Restart** 

Save the configuration file and restart Cline to apply the changes.


### Augment Setup

Augment provides access to MCP configuration through its advanced settings panel.

**Step 1: Access Settings**

1. Press Cmd/Ctrl + Shift + P or navigate to the hamburger menu in the Augment panel

2. Select Edit Settings

3. Under Advanced, click Edit in settings.json

**Step 2: Add Server Configuration** 

Add the server configuration to the `mcpServers` array in the `augment.advanced` object:

```
{
  "mcpServers": {
    "sdk-code-helper": {
      "url": "https://sdk.milvus.io/mcp/",
      "headers": {
        "Accept": "text/event-stream"
      }
    }
  }
}
```


### Gemini CLI Setup

Gemini CLI requires manual configuration through a JSON settings file.

**Step 1: Create or Edit Settings File**

 Create or edit the `~/.gemini/settings.json` file on your system.

**Step 2: Add Configuration**

 Insert the following configuration into the settings file:

```
{
  "mcpServers": {
    "sdk-code-helper": {
      "url": "https://sdk.milvus.io/mcp/",
      "headers": {
        "Accept": "text/event-stream"
      }
    }
  }
}
```

**Step 3: Apply Changes** 

Save the file and restart Gemini CLI to apply the configuration changes.


### Roo Code Setup

Roo Code utilizes a centralized JSON configuration file for managing MCP servers.

**Step 1: Access Global Configuration**

1. Open Roo Code

2. Navigate to Settings → MCP Servers → Edit Global Config

**Step 2: Edit Configuration File** 

In the `mcp_settings.json` file, add the following configuration:

```
{
  "mcpServers": {
    "sdk-code-helper": {
      "url": "https://sdk.milvus.io/mcp/",
      "headers": {
        "Accept": "text/event-stream"
      }
    }
  }
}
```

**Step 3: Activate Server** 

Save the file to automatically activate the MCP server.


### Verification and Testing

After completing the setup for your chosen IDE, you can verify that the Milvus SDK Code Helper is working correctly by:

1. **Testing Code Generation**: Ask your AI assistant to generate Milvus-related code and observe whether it uses current best practices

2. **Checking Documentation Access**: Request information about specific Milvus features to ensure the helper is providing up-to-date responses

3. **Comparing Results**: Generate the same code request with and without the helper to see the difference in quality and currentness


## Conclusion

By setting up the Milvus SDK Code Helper, you've taken a crucial step toward the future of development—where AI assistants generate not just fast code, but **accurate, current code**. Instead of relying on static training data that becomes obsolete, we're moving toward dynamic, real-time knowledge systems that evolve with the technologies they support.

As AI coding assistants become more sophisticated, the gap between tools with current knowledge and those without will only widen. The Milvus SDK Code Helper is just the beginning—expect to see similar specialized knowledge servers for other major technologies and frameworks. The future belongs to developers who can harness AI's speed while ensuring accuracy and currentness. You're now equipped with both.


Milvus SDK Code Helper Tutorial: Fix AI Outdated Code


Stop Your AI Assistant from Writing Outdated Code with Milvus SDK Code Helper


Tutorial | How to Evaluate VectorDBs that Match Production via VDBBench


Hands-On with VDBBench: Benchmarking Vector Databases for POCs That Match Production



AI agents are everywhere right now—from coding copilots to customer service bots—and they can be jaw-droppingly good at complex reasoning. Like many of you, I love them. But after building both agents and automation workflows, I’ve learned a simple truth: **agents aren’t the best solution for every problem.** 

For example, when I built a multi-agent system with CrewAI for decoding ML, things got messy fast. Research agents ignored web crawlers 70% of the time. Summary agents dropped citations. Coordination fell apart whenever tasks weren’t crystal clear.

And it’s not just in experiments. Many of us are already bouncing between ChatGPT for brainstorming, Claude for coding, and a half-dozen APIs for data processing—quietly thinking: _there has to be a better way to make all this work together._

Sometimes, the answer is an agent. More often, it’s a **well-designed AI workflow** that stitches your existing tools into something powerful, without the unpredictable complexity. 


## Building Smarter AI Workflows with Refly and Milvus

I know some of you are already shaking your heads: “Workflows? Those are rigid. They’re not smart enough for real AI automation.” Fair point—most workflows are rigid, because they’re modeled after old-school assembly lines: step A → step B → step C, no deviation allowed.

But the real issue isn’t the _idea_ of workflows—it’s the _execution_. We don’t have to settle for brittle, linear pipelines. We can design smarter workflows that adapt to context, flex with creativity, and still deliver predictable results. 

In this guide, we’ll build a complete content creation system using Refly and Milvus to show why AI workflows can outperform complex multi-agent architectures, especially if you care about speed, reliability, and maintainability.


### The Tools We’re Using

[**Refly**](https://refly.ai/): An open-source, AI-native content creation platform built around a “free canvas” concept.

- **Core capabilities:** intelligent canvas, knowledge management, multi-threaded dialogue, and professional creation tools.

- **Why it’s useful:** Drag-and-drop workflow building lets you chain tools together into cohesive automation sequences, without locking you into rigid, single-path execution.

[**Milvus**](https://milvus.io/): An open-source vector database handling the data layer.

- **Why it matters:** Content creation is mostly about finding and recombining existing information. Traditional databases handle structured data well, but most creative work involves unstructured formats—documents, images, videos.

- **What it adds:** Milvus leverages integrated embedding models to encode unstructured data as vectors, enabling semantic search so your workflows can retrieve relevant context with millisecond latency. Through protocols like MCP, it integrates seamlessly with your AI frameworks, letting you query data in natural language instead of wrestling with database syntax.


### Setting Up Your Environment

Let me walk you through setting this workflow up locally. 

**Quick setup checklist:**

- Ubuntu 20.04+ (or similar Linux)

- Docker + Docker Compose

- An API key from any LLM that supports function calling. Here in this guide, I’ll use [Moonshot](https://platform.moonshot.ai/docs/introduction#text-generation-model)’s LLM. 

**System Requirements**

- CPU: 8 cores minimum (16 cores recommended)

- Memory: 16GB minimum (32GB recommended)

- Storage: 100GB SSD minimum (500GB recommended)

- Network: Stable internet connection required

**Software Dependencies**

- Operating System: Linux (Ubuntu 20.04+ recommended)

- Containerization: Docker + Docker Compose

- Python: Version 3.11 or higher

- Language Model: Any model supporting function calls (online services or Ollama offline deployment both work)


### Step 1: Deploy the Milvus Vector Database

**1.1 Download Milvus**

```
wget https://github.com/milvus-io/milvus/releases/download/v2.5.12/milvus-standalone-docker-compose.yml -O docker-compose.yml
```

**1.2 Launch Milvus services**

```
docker-compose up -d
docker-compose ps -a
```

![](https://assets.zilliz.com/1_b93ce78614.png)


### Step 2: Deploy the Refly Platform

**2.1 Clone the repository**

You can use default values for all environment variables unless you have specific requirements:

```
git clone https://github.com/refly-ai/refly.git
```
```
cd deploy/docker
cp ../../apps/api/.env.example .env # copy the example api env file
docker compose up -d
```

**2.2 Verify service status**

```
docker ps -a
```

![](https://assets.zilliz.com/2_cfcde2c570.png)

### Step 3: Set Up MCP Services

**3.1 Download the Milvus MCP server**

```
git clone https://github.com/zilliztech/mcp-server-milvus.git
cd mcp-server-milvus
```

**3.2 Start the MCP service**

This example uses SSE mode. Replace the URI with your available Milvus service endpoint:

```
uv run src/mcp_server_milvus/server.py --sse --milvus-uri http://localhost:19530 --port 8000
```

**3.3 Confirm MCP service is running**

![](https://assets.zilliz.com/3_b755922c41.png)


### Step 4: Configuration and Setup

Now that your infrastructure is running, let's configure everything to work together seamlessly.

**4.1 Access the Refly platform**

Navigate to your local Refly instance:

```
http://192.168.7.148:5700
```

![](https://assets.zilliz.com/4_c1e421fece.png)

**4.2 Create your account**

![](https://assets.zilliz.com/42_4b8af22fe3.png)

![](https://assets.zilliz.com/422_e8cd8439f0.png)

**4.3 Configure your language model**

For this guide, we'll use [Moonshot](https://platform.moonshot.ai/docs/introduction#text-generation-model). First, register and obtain your API key.

**4.4 Add your model provider**

Enter the API key you obtained in the previous step:

![](https://assets.zilliz.com/44_b085f9a263.png)

**4.5 Configure the LLM model**

Make sure to select a model that supports function calling capabilities, as this is essential for the workflow integrations we'll be building:

![](https://assets.zilliz.com/45_a05213d0fa.pngQ)

**4.6 Integrate Milvus-MCP service**

Note that the web version doesn't support stdio-type connections, so we'll use the HTTP endpoint we set up earlier:

![](https://assets.zilliz.com/46_027e21e479.png)

![](https://assets.zilliz.com/462_959ee44a78.png)

Excellent! With everything configured, let's see this system in action through some practical examples.

**4.7 Example: Efficient Vector Retrieval with MCP-Milvus-Server**

This example shows how the **MCP-Milvus-Server** works as middleware between your AI models and Milvus vector database instances. It acts like a translator—accepting natural language requests from your AI model, converting them into the right database queries, and returning the results—so your models can work with vector data without knowing any database syntax.

**4.7.1 Create a new canvas**

![](https://assets.zilliz.com/471_a684e275ed.png)

**4.7.2 Start a conversation**

Open the dialogue interface, select your model, input your question, and send. 

**4.7.3 Review the results**

![](https://assets.zilliz.com/473_7c24a28999.png)

What’s happening here is pretty remarkable: we’ve just shown natural language control of a Milvus vector database using [MCP-Milvus-Server](https://milvus.io/blog/talk-to-vector-db-manage-milvus-via-natural-language.md) as the integration layer. No complex query syntax—just tell the system what you need in plain English, and it handles the database operations for you. 

**4.8 Example 2: Building a Refly Deployment Guide with Workflows**

This second example shows the real power of workflow orchestration. We'll create a complete deployment guide by combining multiple AI tools and data sources into a single, coherent process.

**4.8.1 Gather your source materials**

The power of Refly is its flexibility in handling different input formats. You can import resources in multiple formats, whether they're documents, images, or structured data.

**4.8.2 Create tasks and link resource cards**

Now we'll create our workflow by defining tasks and connecting them to our source materials.

**4.8.3 Set up three processing tasks**

This is where the workflow approach really shines. Instead of trying to handle everything in one complex process, we break the work into three focused tasks that integrate uploaded materials and refine them systematically.

- **Content integration task**: Combines and structures source materials

- **Content refinement task**: Improves clarity and flow

- **Final draft compilation**: Creates publication-ready output

The results speak for themselves. What would have taken hours of manual coordination across multiple tools is now handled automatically, with each step building logically on the previous one.

**Multi-modal workflow capabilities:**

- **Image generation and processing**: Integration with high-quality models including flux-schnell, flux-pro, and SDXL

- **Video generation and understanding**: Support for various stylized video models, including Seedance, Kling, and Veo

- **Audio generation tools**: Music generation through models like Lyria-2 and voice synthesis via models like Chatterbox

- **Integrated processing**: All multi-modal outputs can be referenced, analyzed, and reprocessed within the system


##  Conclusion

The integration of **Refly** and **Milvus** offers a pragmatic approach to automation—one that values reliability and ease of use over unnecessary complexity. By combining workflow orchestration with multi-modal processing, teams can move from concept to publication faster while retaining full control at every stage.

This isn’t about dismissing AI agents. They’re valuable for tackling genuinely complex, unpredictable problems. But for many automation needs—especially in content creation and data processing—a well-designed workflow can deliver better results with less overhead.

As AI tech evolves, the most effective systems will likely blend both strategies:

- **Workflows** where predictability, maintainability, and reproducibility are key.

- **Agents** where real reasoning, adaptability, and open-ended problem-solving are required.

The goal isn’t to build the flashiest AI—it’s to build the most _useful_ one. And often, the most helpful solution is also the most straightforward.


AI Agents or Workflows? Why Skip Agents for 80% of Automation


AI Agents or Workflows? Why You Should Skip Agents for 80% of Automation Tasks




**After months of speculation, OpenAI has finally shipped** [**GPT-5**](https://openai.com/gpt-5/)**.** The model isn’t the creative lightning strike that GPT-4 was, but for developers, especially those building agents and RAG pipelines, this release may quietly be the most useful upgrade yet. 

**TL;DR for builders:** GPT-5 unifies architectures, supercharges multimodal I/O, slashes factual error rates, extends context to 400k tokens, and makes large-scale usage affordable. However, the creativity and literary flair have taken a noticeable step back.


## What’s New Under the Hood? 

- **Unified core** — Merges GPT digital series with o-series reasoning models, delivering long-chain reasoning plus multimodal in a single architecture.

- **Full-spectrum multimodal** — Input/output across text, image, audio, and video, all within the same model.

- **Massive accuracy gains**: 

  - `gpt-5-main`: 44% fewer factual errors vs. GPT-4o.

  - `gpt-5-thinking`: 78% fewer factual errors vs. o3.

- **Domain skill boosts** — Stronger in code generation, mathematical reasoning, health consultation, and structured writing; hallucinations reduced significantly.

Alongside GPT-5, OpenAI also released **three additional variants**, each optimized for different needs:

![](https://assets.zilliz.com/gpt_5_family_99a9bee18a.png)

| **Model**  | **Description**                                       | **Input / $ per 1M tokens** | **Output / $ per 1M tokens** | **Knowledge Update** |
| ---------- | ----------------------------------------------------- | --------------------------- | ---------------------------- | -------------------- |
| gpt-5      | Main model, long-chain reasoning + full multimodal    | $1.25                       | $10.00                       | 2024-10-01           |
| gpt-5-chat | Equivalent to gpt-5, used in ChatGPT conversations    | —                           | —                            | 2024-10-01           |
| gpt-5-mini | 60% cheaper, retains ~90% of programming performance | $0.25                       | $2.00                        | 2024-05-31           |
| gpt-5-nano | Edge/offline, 32K context, latency <40ms              | $0.05                       | $0.40                        | 2024-05-31           |

GPT-5 broke records across 25 benchmark categories — from code repair to multimodal reasoning to medical tasks — with consistent accuracy improvements.

![](https://assets.zilliz.com/benchmark_1_8ebf1bed4b.png)

![](https://assets.zilliz.com/benchmark_2_c43781126f.png)


## Why Developers Should Care — Especially for RAG & Agents

Our hands-on tests suggest this release is a quiet revolution for Retrieval-Augmented Generation and agent-driven workflows.

1. **Price cuts** make experimentation viable — API input cost: **$1.25 per million tokens**; output cost: **$10**.

2. **A 400k context window** (vs. 128k in o3/4o) allows you to maintain state across complex multi-step agent workflows without context chopping.

3. **Fewer hallucinations & better tool use** — Supports multi-step chained tool calls, handles complex non-standard tasks, and improves execution reliability.

![](https://assets.zilliz.com/gpt_tool_call_e6a86fc59c.png)


## Not Without Flaws

Despite its technical advances, GPT-5 still shows clear limits. 

At launch, OpenAI’s keynote featured a slide that bizarrely calculated _52.8 > 69.1 = 30.8_, and in our own tests, the model confidently repeated the textbook-but-wrong “Bernoulli effect” explanation for airplane lift—reminding us **it’s still a pattern learner, not a true domain expert.**

**While STEM performance has sharpened, creative depth has slipped.** Many long-time users note a decline in literary flair: poetry feels flatter, philosophical conversations less nuanced, and long-form narratives more mechanical. The trade-off is clear—higher factual accuracy and stronger reasoning in technical domains, but at the expense of the artful, exploratory tone that once made GPT feel almost human.

With that in mind, let’s see how GPT-5 actually performs in our hands-on tests.


## Coding Tests

I started with a simple task: write an HTML script that allows users to upload an image and move it with the mouse. GPT-5 paused for about nine seconds, then produced working code that handled the interaction well. It felt like a good start.

![](https://assets.zilliz.com/gpt52_7b04c9b41b.gif)

The second task was harder: implement polygon–ball collision detection inside a rotating hexagon, with adjustable rotation speed, elasticity, and ball count. GPT-5 generated the first version in around thirteen seconds. The code included all expected features, but it had bugs and wouldn’t run.

![](https://assets.zilliz.com/image_6_3e6a34572a.png)

I then used the editor’s **Fix bug** option, and GPT-5 corrected the errors so the hexagon rendered. However, the balls never appeared — the spawn logic was missing or incorrect, meaning the core function of the program was absent despite the otherwise complete setup.

![](https://assets.zilliz.com/gpt51_6489df9914.gif)

**In summary,** GPT-5 can produce clean, well-structured interactive code and recover from simple runtime errors. But in complex scenarios, it still risks omitting essential logic, so human review and iteration are necessary before deployment.


## Reasoning Test

I posed a multi-step logic puzzle involving item colors, prices, and positional clues—something that would take most humans several minutes to solve.

![](https://assets.zilliz.com/reasoning_test_7ea15ed25b.png)

**Question:** _What is the blue item and what is its price?_

GPT-5 delivered the correct answer in just 9 seconds, with a clear and logically sound explanation. This test reinforced the model’s strength in structured reasoning and rapid deduction.


## Writing Test

I often turn to ChatGPT for help with blogs, social media posts, and other written content, so text generation is one of the capabilities I care about most. For this test, I asked GPT-5 to create a LinkedIn post based on a blog about Milvus 2.6’s multilingual analyzer.

![](https://assets.zilliz.com/Writing_Test_4fe5fef775.png)

The output was well-organized and hit all the key points from the original blog, but it felt too formal and predictable—more like a corporate press release than something meant to spark interest on a social feed. It lacked the warmth, rhythm, and personality that make a post feel human and inviting.

On the upside, the accompanying illustrations were excellent: clear, on-brand, and perfectly aligned with Zilliz’s tech style. Visually, it was spot-on; the writing just needs a bit more creative energy to match.


## Longer Context Window = Death of RAG and VectorDB?

We tackled this topic last year when [Google launched **Gemini 1.5 Pro**](https://zilliz.com/blog/will-retrieval-augmented-generation-RAG-be-killed-by-long-context-LLMs) with its ultra-long 10M-token context window. At the time, some people were quick to predict the end of RAG—and even the end of databases altogether. Fast-forward to today: not only is RAG still alive, it’s thriving. In practice, it’s become _more_ capable and productive, along with vector databases like [**Milvus**](https://milvus.io/) and [**Zilliz Cloud**](https://zilliz.com/cloud).

Now, with GPT-5’s expanded context length and more advanced tool-calling capabilities, the question has popped up again: _Do we still need vector databases for context ingestion, or even dedicated agents/RAG pipelines?_

**The short answer: absolutely yes. We still need them.**

Longer context is useful, but it’s not a replacement for structured retrieval. Multi-agent systems are still on track to be a long-term architectural trend—and these systems often need virtually unlimited context. Plus, when it comes to managing private, unstructured data securely, a vector database will always be the final gatekeeper.


## Conclusion

After watching OpenAI’s launch event and running my own hands-on tests, GPT-5 feels less like a dramatic leap forward and more like a refined blend of past strengths with a few well-placed upgrades. That’s not a bad thing—it’s a sign of the architectural and data-quality limits large models are starting to encounter.

As the saying goes, _severe criticism comes from high expectations_. Any disappointment around GPT-5 mostly comes from the very high bar OpenAI set for itself. And really—better accuracy, lower prices, and integrated multimodal support are still valuable wins. For developers building agents and RAG pipelines, this may actually be the most useful upgrade so far.

Some friends have been joking about making “online memorials” for GPT-4o, claiming their old chat companion’s personality is gone forever. I don’t mind the change—GPT-5 might be less warm and chatty, but its direct, no-nonsense style feels refreshingly straightforward.

**What about you?** Share your thoughts with us—join our [Discord](https://discord.com/invite/8uyFbECzPX), or join the conversation on [LinkedIn](https://www.linkedin.com/company/the-milvus-project/) and [X](https://x.com/milvusio). 


GPT-5 Review: Accuracy Up, Prices Down, Code Strong, Bad Creativity


GPT-5 Review: Accuracy Up, Prices Down, Code Strong — But Bad for Creativity



The AI world has been running hot. In just a few weeks, Anthropic dropped Claude 4.1 Opus, DeepMind stunned everyone with Genie 3 world simulator—and now, OpenAI steals the spotlight by open-sourcing two reasoning models: [gpt-oss-120b](https://huggingface.co/openai/gpt-oss-120b) and [gpt-oss-20b](https://huggingface.co/openai/gpt-oss-20b), permissively licensed under Apache 2.0.

After launch, these models instantly shot to the #1 trending spot on Hugging Face—and for good reason. This is the first time since 2019 that OpenAI has released open-weight models that are actually production-ready. The move isn’t accidental—after years of pushing API-only access, OpenAI is clearly responding to the pressure from open-source leaders like DeepSeek, Meta’s LLaMA, and Qwen, who’ve been dominating both benchmarks and developer workflows.

In this post, we'll explore what makes GPT-oss different, how it compares to leading open models like DeepSeek R1 and Qwen 3, and why developers should care. We'll also walk through building a reasoning-capable RAG system using GPT-oss and Milvus, the most popular open-source vector database.  


## What Makes GPT-oss Special and Why You Should Care? 

GPT-oss isn't just another weight drop. It delivers in five key areas that matter to developers:


### 1: Built for Edge Deployment

GPT-oss comes in two strategically sized variants:

- gpt-oss-120b: 117B total, 5.1B active per token

- gpt-oss-20b: 21B total, 3.6B active per token

Using Mixture-of-Experts (MoE) architecture, only a subset of parameters is active during inference. This makes both models lightweight to run relative to their size:

- gpt-oss-120b runs on a single 80GB GPU (H100)

- gpt-oss-20b fits in just 16GB VRAM, meaning it runs on high-end laptops or edge devices

According to OpenAI's tests, gpt-oss-20b is the fastest OpenAI model for inference—ideal for low-latency deployments or offline reasoning agents.


### 2: Strong Benchmark Performance

According to OpenAI's evaluations:

- **gpt-oss-120b** performs near-parity with o4-mini on reasoning, tool use, and competition coding (Codeforces, MMLU, TauBench)

- **gpt-oss-20b** competes with o3-mini, and even outperforms it in math and healthcare reasoning


### 3: Cost-Efficient Training

OpenAI claims performance equivalent to o3-mini and o4-mini, but with dramatically lower training costs:

- **gpt-oss-120b**: 2.1 million H100-hours → ~$10M

- **gpt-oss-20b**: 210K H100-hours → ~$1M

Compare that to the multi-hundred-million-dollar budgets behind models like GPT-4. GPT-oss proves that efficient scaling and architecture choices can deliver competitive performance without a massive carbon footprint.


### 4: True Open-Source Freedom

GPT-oss uses Apache 2.0 licensing, which means:

- Commercial use allowed

- Full modification and redistribution rights

- No usage restrictions or copyleft clauses

This is really open source, not a research-only release. You can fine-tune for domain-specific use, deploy in production with full control, and build commercial products around it. Key features include configurable reasoning depth (low/medium/high), full chain-of-thought visibility, and native tool calling with structured output support.


### 5: Potential GPT-5 Preview

OpenAI hasn’t disclosed everything—but architecture details suggest this may preview the direction of **GPT-5**:

- Uses MoE with 4 experts per input

- Follows alternating dense + local sparse attention (GPT-3 pattern)

- Features more attention heads

- Interestingly, bias units from GPT-2 have made a comeback

If you're watching for signals on what comes next, GPT-oss may be the clearest public hint yet.


### Core Specifications

|              |                  |                   |             |                    |              |
| ------------ | ---------------- | ----------------- | ----------- | ------------------ | ------------ |
| **Model**    | **Total Params** | **Active Params** | **Experts** | **Context Length** | **VRAM Req** |
| gpt-oss-120b | 117B             | 5.1B              | 128         | 128k               | 80GB         |
| gpt-oss-20b  | 21B              | 3.6B              | 32          | 128k               | 16GB         |

Both models use the o200k_harmony tokenizer and support 128,000-token context length (roughly 96,000-100,000 words).


## GPT-oss vs. Other Reasoning Models

Here's how GPT-oss stacks up against OpenAI's internal models and top open-source competitors:

|                   |                         |             |                                      |
| ----------------- | ----------------------- | ----------- | ------------------------------------ |
| **Model**         | **Parameters (Active)** | **Memory**  | **Strengths**                        |
| **gpt-oss-120b**  | 117B (5.1B active)      | 80GB        | Single-GPU, open reasoning           |
| **gpt-oss-20b**   | 21B (3.6B active)       | 16GB        | Edge deployment, fast inference      |
| **DeepSeek R1**   | 671B (~37B active)     | Distributed | Benchmark leader, proven performance |
| **o4-mini (API)** | Proprietary             | API only    | Strong reasoning (closed)            |
| **o3-mini (API)** | Proprietary             | API only    | Lightweight reasoning (closed)       |

Based on various benchmarking models, here is what we found: 

- **GPT-oss vs. OpenAI's Own Models:** gpt-oss-120b matches o4-mini on competition math (AIME), coding (Codeforces), and tool use (TauBench). The 20b model performs similarly to o3-mini despite being much smaller.

- **GPT-oss vs. DeepSeek R1:** DeepSeek R1 dominates in pure performance but requires distributed infrastructure. GPT-oss offers simpler deployment—no distributed setup needed for the 120b model.

In summary, GPT-oss offers the best combination of performance, open access, and deployability. DeepSeek R1 wins on pure performance, but GPT-oss strikes the optimal balance for most developers. 


## Hands-on: Building with GPT-oss + Milvus

Now that we’ve seen what GPT-oss brings to the table, it’s time to put it to use. 

In the following sections, we'll walk through a hands-on tutorial for building a reasoning-capable RAG system using gpt-oss-20b and Milvus, all running locally, no API key required.


### Environment Setup

```
! pip install --upgrade "pymilvus[model]" openai requests tqdm
```

### Dataset Preparation

We'll use Milvus documentation as our knowledge base:

```
# Download and prepare Milvus docs
! wget https://github.com/milvus-io/milvus-docs/releases/download/v2.4.6-preview/milvus_docs_2.4.x_en.zip
! unzip -q milvus_docs_2.4.x_en.zip -d milvus_docs

from glob import glob
text_lines = []
for file_path in glob("milvus_docs/en/faq/*.md", recursive=True):
    with open(file_path, "r") as file:
        file_text = file.read()
    text_lines += file_text.split("# ")
```

### Model Setup

Access GPT-oss through [OpenRouter](https://openrouter.ai/openai/gpt-oss-20b:free) (or run locally). [**OpenRouter**](https://openrouter.ai/openai/gpt-oss-20b:free) is a platform that lets developers access and switch between multiple AI models (like GPT-4, Claude, Mistral) through a single, unified API. It's useful for comparing models or building apps that work with different AI providers. Now GPT-oss series have been available on OpenRouter now. 

![](https://assets.zilliz.com/1_46b575811f.png)

```
from openai import OpenAI

# Using OpenRouter for cloud access
openai_client = OpenAI(
    api_key="<OPENROUTER_API_KEY>",
    base_url="https://openrouter.ai/api/v1",
)

# Set up embedding model
from pymilvus import model as milvus_model
embedding_model = milvus_model.DefaultEmbeddingFunction()

# Test embedding dimensions
test_embedding = embedding_model.encode_queries(["This is a test"])[0]
embedding_dim = len(test_embedding)
print(embedding_dim)
print(test_embedding[:10])
```

```
768
[-0.04836066  0.07163023 -0.01130064 -0.03789345 -0.03320649 -0.01318448
 -0.03041712 -0.02269499 -0.02317863 -0.00426028]
```

### Set up Milvus vector database 
```
from pymilvus import MilvusClient

# Initialize Milvus client
milvus_client = MilvusClient(uri="http://localhost:19530", token="root:Milvus")
collection_name = "gpt_oss_rag_collection"

# Clean up existing collection
if milvus_client.has_collection(collection_name):
    milvus_client.drop_collection(collection_name)

# Create new collection
milvus_client.create_collection(
    collection_name=collection_name,
    dimension=embedding_dim,
    metric_type="IP",  # Inner product distance
    consistency_level="Strong",
)
```

About MilvusClient parameter settings:

- Setting the URI to a local file (e.g., `./milvus.db`) is the most convenient method as it automatically uses Milvus Lite to store all data in that file.

- For large-scale data, you can set up a more powerful Milvus server on Docker or Kubernetes. In this case, use the server’s URI (e.g., `http://localhost:19530`) as your URI.

- If you want to use [Zilliz Cloud ](https://zilliz.com/cloud)(the managed service of Milvus), adjust the URI and token, which correspond to the Public Endpoint and API key in Zilliz Cloud.


### Adding Documents to the Collection

Now we’ll create embeddings for our text chunks and add them to Milvus:

```
from tqdm import tqdm

data = []
doc_embeddings = embedding_model.encode_documents(text_lines)

for i, line in enumerate(tqdm(text_lines, desc="Creating embeddings")):
    data.append({"id": i, "vector": doc_embeddings[i], "text": line})

milvus_client.insert(collection_name=collection_name, data=data)
```

Output:

```
Creating embeddings: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 72/72 [00:00<00:00, 1222631.13it/s]
{'insert_count': 72, 'ids': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71], 'cost': 0}
```

### RAG Query Pipeline

Now for the exciting part - let’s set up our RAG system to answer questions.

Let’s specify a common question about Milvus:
```
question = "How is data stored in milvus?"
```

Search for this question in the collection and retrieve the top 3 semantically matching results:

```
search_res = milvus_client.search(
    collection_name=collection_name,
    data=embedding_model.encode_queries(
        [question]
    ),  # Convert the question to an embedding vector
    limit=3,  # Return top 3 results
    search_params={"metric_type": "IP", "params": {}},  # Inner product distance
    output_fields=["text"],  # Return the text field
)
```

Let’s look at the search results for this query:
```
import json

retrieved_lines_with_distances = [
    (res["entity"]["text"], res["distance"]) for res in search_res[0]
]
print(json.dumps(retrieved_lines_with_distances, indent=4))
```

```
[
    [
        " Where does Milvus store data?\n\nMilvus deals with two types of data, inserted data and metadata. \n\nInserted data, including vector data, scalar data, and collection-specific schema, are stored in persistent storage as incremental log. Milvus supports multiple object storage backends, including [MinIO](https://min.io/), [AWS S3](https://aws.amazon.com/s3/?nc1=h_ls), [Google Cloud Storage](https://cloud.google.com/storage?hl=en#object-storage-for-companies-of-all-sizes) (GCS), [Azure Blob Storage](https://azure.microsoft.com/en-us/products/storage/blobs), [Alibaba Cloud OSS](https://www.alibabacloud.com/product/object-storage-service), and [Tencent Cloud Object Storage](https://www.tencentcloud.com/products/cos) (COS).\n\nMetadata are generated within Milvus. Each Milvus module has its own metadata that are stored in etcd.\n\n###",
        0.6572664976119995
    ],
    [
        "How does Milvus flush data?\n\nMilvus returns success when inserted data are loaded to the message queue. However, the data are not yet flushed to the disk. Then Milvus' data node writes the data in the message queue to persistent storage as incremental logs. If `flush()` is called, the data node is forced to write all data in the message queue to persistent storage immediately.\n\n###",
        0.6312144994735718
    ],
    [
        "How does Milvus handle vector data types and precision?\n\nMilvus supports Binary, Float32, Float16, and BFloat16 vector types.\n\n- Binary vectors: Store binary data as sequences of 0s and 1s, used in image processing and information retrieval.\n- Float32 vectors: Default storage with a precision of about 7 decimal digits. Even Float64 values are stored with Float32 precision, leading to potential precision loss upon retrieval.\n- Float16 and BFloat16 vectors: Offer reduced precision and memory usage. Float16 is suitable for applications with limited bandwidth and storage, while BFloat16 balances range and efficiency, commonly used in deep learning to reduce computational requirements without significantly impacting accuracy.\n\n###",
        0.6115782856941223
    ]
]
```

### Using the GPT-oss to Build a RAG Response

Convert the retrieved documents to string format:

```
context = "\n".join(
    [line_with_distance[0] for line_with_distance in retrieved_lines_with_distances]
)
```
Provide system prompt and user prompt for the large language model:
```
SYSTEM_PROMPT = """
Human: You are an AI assistant. You are able to find answers to the questions from the contextual passage snippets provided.
"""
USER_PROMPT = f"""
Use the following pieces of information enclosed in <context> tags to provide an answer to the question enclosed in <question> tags.
<context>
{context}
</context>
<question>
{question}
</question>
"""
```
Use the latest gpt-oss model to generate a response based on the prompt:
```
response = openai_client.chat.completions.create(
    model="openai/gpt-oss-120b",
    messages=[
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": USER_PROMPT},
    ],
)
print(response.choices[0].message.content)
```

```
Milvus stores its data in two distinct layers:

| Type of data | Where it is stored | How it is stored |
|-------------|-------------------|-----------------|
| **Inserted data** (vector data, scalar fields, collection‑specific schema) | In the **persistent object storage** configured for the cluster. The data are written as **incremental logs** (append‑only logs) that are persisted by the DataNode. | The DataNode reads from the message‑queue and writes the incoming data into the storage backend (MinIO, AWS S3, GCS, Azure Blob, Alibaba OSS, Tencent COS, etc.). When a `flush()` call is issued, the DataNode forces all queued data to be written to the persistent storage immediately. |
| **Metadata** (information about collections, partitions, indexes, etc.) | In **etcd**. Each Milvus module (catalog, index, etc.) keeps its own metadata. | The metadata is generated and managed by Milvus and persisted in the distributed key‑value store **etcd**. |

**Summary:**  
- **Inserted data** = incremental logs stored in the chosen object‑storage backend.  
- **Metadata** = stored in the distributed configuration store **etcd**.  


Together, these two storage mechanisms (object storage for the actual data and etcd for metadata) make up Milvus’s data‑storage architecture.
```

## Final Thoughts on GPT-oss

GPT-oss is OpenAI’s quiet admission that open-source can't be ignored anymore. It doesn’t blow DeepSeek R1 or Qwen 3 or many other models out of the water, but it does bring something they don’t: OpenAI’s training pipeline, applied to a model you can actually inspect and run locally.

**Performance? Solid. Not mind-blowing, but dependable.** The 20B model running on consumer hardware—or even mobile with LM Studio—is the kind of practical advantage that actually matters to developers. It’s more “this just works” than “wow, this changes everything.” And honestly, that’s fine.

**Where it falls short is multilingual support.** If you’re working in anything other than English, you’ll hit weird phrasing, spelling issues, and general confusion. The model was clearly trained with an English-first lens. If global coverage matters, you’re probably going to need to fine-tune it with a multilingual dataset. 

What’s most interesting, though, is the timing. OpenAI’s teaser on X—with a “5” dropped into the word “LIVESTREAM”—feels like a setup. GPT-oss might not be the main act, but it could be a preview of what’s coming in GPT-5. Same ingredients, different scale. Let’s wait.

![](https://assets.zilliz.com/2_0fed950b8e.png)

**The real win is having more high-quality choices.** Competition drives innovation, and OpenAI re-entering open-source development benefits everyone. Test GPT-oss against your specific requirements, but choose based on what actually works for your use case, not brand recognition.


GPT-oss vs o4-mini: Edge-Ready, Solid, But Not Disruptive


GPT-oss vs o4-mini: Edge-Ready, On-Par Performance — Dependable, Not Mind-Blowing


Deploy Milvus on K8s Just Got Easier with the Milvus Operator 


Deploying Milvus on Kubernetes Just Got Easier with the Milvus Operator




Ever wished you could just tell your AI assistant, _"Show me all collections in my vector database"_ or _"Find documents similar to this text"_ and have it actually work?

The [**Milvus MCP Server**](http://github.com/zilliztech/mcp-server-milvus) makes this possible by connecting your Milvus vector database directly to AI coding assistants like Claude Desktop and Cursor IDE through Model Context Protocol (MCP). Instead of writing `pymilvus` code, you can manage your entire Milvus through natural language conversations.

- Without Milvus MCP Server: Writing Python scripts with pymilvus SDK to search vectors

- With Milvus MCP Server: "Find documents similar to this text in my collection."

👉 **GitHub Repository:**[ github.com/zilliztech/mcp-server-milvus](https://github.com/zilliztech/mcp-server-milvus)

And if you're using [Zilliz Cloud](https://zilliz.com/cloud) (managed Milvus), we’ve got you covered too. At the end of this blog, we’ll also introduce the **Zilliz MCP Server**, a managed option that works seamlessly with Zilliz Cloud. Let’s dive in.


## What You'll Get with Milvus MCP Server

The Milvus MCP Server gives your AI assistant the following capabilities:

- **List and explore** vector collections

- **Search vectors** using semantic similarity

- **Create new collections** with custom schemas

- **Insert and manage** vector data

- **Run complex queries** without writing code

- And more 

All through natural conversation, as if you're talking to a database expert. Check out [this repo](https://github.com/zilliztech/mcp-server-milvus?tab=readme-ov-file#available-tools) for the complete list of capabilities. 

![](https://assets.zilliz.com/demo_adedb25430.gif)


## Quick Start Guide

### Prerequisites

**Required:**

- Python 3.10 or higher

- A running Milvus instance (local or remote)

- [uv package manager](https://github.com/astral-sh/uv) (recommended)

**Supported AI Applications:**

- Claude Desktop

- Cursor IDE

- Any MCP-compatible application


### Tech Stack We’ll Use 

In this tutorial, we'll use the following tech stack:

- **Language Runtime:** [Python 3.11](https://www.python.org/)

- **Package Manager:** UV

- **IDE:** Cursor

- **MCP Server:** mcp-server-milvus

- **LLM:** Claude 3.7

- **Vector Database:** Milvus


### Step 1: Install Dependencies

First, install the uv package manager:

```
curl -LsSf https://astral.sh/uv/install.sh | sh
```

Or:

```
pip3 install uv -i https://mirrors.aliyun.com/pypi/simple
```

Verify installation:

```
uv --version
uvx --version
```


![](https://assets.zilliz.com/Step_1_Install_Dependencies_3e452c55e3.png)


### Step 2: Set Up Milvus

[Milvus](https://milvus.io/) is an open-source vector database native for AI workloads, created by [Zilliz](https://zilliz.com/). Designed to handle millions to billions of vector records, it has gained over 36,000 stars on GitHub. Building on this foundation, Zilliz also offers [Zilliz Cloud](https://zilliz.com/cloud)—a fully managed service of Milvus engineered for usability, cost-efficiency, and security with a cloud-native architecture.

For Milvus deployment requirements, visit [this guide on the doc site](https://milvus.io/docs/prerequisite-docker.md).  

**Minimum requirements:**

- **Software:** Docker, Docker Compose

- **RAM:** 16GB+

- **Disk:** 100GB+

Download the deployment YAML file:

```
[root@Milvus ~]# wget https://github.com/milvus-io/milvus/releases/download/v2.5.4/milvus-standalone-docker-compose.yml -O docker-compose.yml
```

Start Milvus:

```
[root@Milvus ~]# docker-compose up -d
```

```
[root@Milvus ~]# docker ps -a
```



![](https://assets.zilliz.com/Step_2_Set_Up_Milvus_4826468767.png)

Your Milvus instance will be available at `http://localhost:19530`.


### Step 3: Install the MCP Server

Clone and test the MCP server:

```
git clone https://github.com/zilliztech/mcp-server-milvus.git
cd mcp-server-milvus

# Test the server locally
uv run src/mcp_server_milvus/server.py --milvus-uri http://localhost:19530
```

We recommend installing dependencies and verifying locally before registering the server in Cursor:

```
uv run src/mcp_server_milvus/server.py --milvus-uri http://192.168.4.48:19530
```


If you see the server start successfully, you're ready to configure your AI tool.

![](https://assets.zilliz.com/Step_3_Install_the_MCP_Server_9ce01351e6.png)


### Step 4: Configure Your AI Assistant

**Option A: Claude Desktop**

1. #### Install Claude Desktop from `[claude.ai/download](http://claude.ai/download)`.

2. Open the configuration file:

- macOS: `~/Library/Application Support/Claude/claude_desktop_config.json`
- Windows: `%APPDATA%\Claude\claude_desktop_config.json`

Add this configuration:

```
{
  "mcpServers": {
    "milvus": {
      "command": "/path/to/uv",
      "args": [
        "--directory",
        "/path/to/mcp-server-milvus",
        "run",
        "src/mcp_server_milvus/server.py",
        "--milvus-uri",
        "http://localhost:19530"
      ]
    }
  }
}
```



4. Restart Claude Desktop

**Option B: Cursor IDE**

1. Open Cursor Settings → Features → MCP

2. Add new global MCP server (this creates `.cursor/mcp.json`)

3. Add this configuration:

Note: Adjust paths to your actual file structure.

```
{
  "mcpServers": {
    "milvus": {
      "command": "/PATH/TO/uv",
      "args": [
        "--directory",
        "/path/to/mcp-server-milvus/src/mcp_server_milvus",
        "run",
        "server.py",
        "--milvus-uri",
        "http://127.0.0.1:19530"
      ]
    }
  }
}
```


![](https://assets.zilliz.com/Option_B_Cursor_IDE_cd1321ea25.png)

**Parameters:**

- `/PATH/TO/uv` is the path to the uv executable
- `--directory` is the path to the cloned project
- `--milvus-uri` is your Milvus server endpoint

4. Restart the Cursor or reload the window

**Pro tip:** Find your `uv` path with `which uv` on macOS/Linux or `where uv`  on Windows.


### Step 5: See It in Action

Once configured, try these natural language commands:

- **Explore your database:** "What collections do I have in my Milvus database?"

- **Create a new collection:** "Create a collection called 'articles' with fields for title (string), content (string), and a 768-dimension vector field for embeddings."

- **Search for similar content:** "Find the five most similar articles to 'machine learning applications' in my articles collection."

- **Insert data:** "Add a new article with title 'AI Trends 2024' and content 'Artificial intelligence continues to evolve...' to the articles collection"

**What used to require 30+ minutes of coding now takes seconds of conversation.**

You get real-time control and natural language access to Milvus—without writing boilerplate or learning the API.


## Troubleshooting

If MCP tools don't appear, restart your AI application completely, verify the UV path with `which uv`, and test the server manually with `uv run src/mcp_server_milvus/server.py --milvus-uri http://localhost:19530`.

For connection errors, check that Milvus is running with `docker ps | grep milvus`, try using `127.0.0.1` instead of `localhost`, and verify port 19530 is accessible.

If you encounter authentication issues, set the `MILVUS_TOKEN` environment variable if your Milvus requires authentication, and verify your permissions for the operations you're attempting.


## Managed Alternative: Zilliz MCP Server

The open-source **Milvus MCP Server** is a great solution for local or self-hosted deployments of Milvus. But if you’re using [Zilliz Cloud](https://zilliz.com/cloud)—the fully managed, enterprise-grade service built by the creators of Milvus—there’s a purpose-built alternative: the [**Zilliz MCP Server**](https://zilliz.com/blog/introducing-zilliz-mcp-server).

[Zilliz Cloud](https://zilliz.com/cloud) eliminates the overhead of managing your own Milvus instance by offering a scalable, performant, and secure cloud-native vector database. The **Zilliz MCP Server** integrates directly with Zilliz Cloud and exposes its capabilities as MCP-compatible tools. This means your AI assistant—whether in Claude, Cursor, or another MCP-aware environment—can now query, manage, and orchestrate your Zilliz Cloud workspace using natural language.

![](https://assets.zilliz.com/zilliz_mcp_abe1ca1271.png)

No boilerplate code. No switching tabs. No manually writing REST or SDK calls. Just say your request and let your assistant handle the rest.


### 🚀 Getting Started with Zilliz MCP Server

If you're ready for production-ready vector infrastructure with the ease of natural language, getting started takes just a few steps:

1. [**Sign up for Zilliz Cloud**](https://cloud.zilliz.com/signup) – free tier available.

2. [**Install the Zilliz MCP Server** ](http://github.com/zilliztech/zilliz-mcp-server)from the GitHub repository.

3. **Configure your MCP-compatible assistant** (Claude, Cursor, etc.) to connect to your Zilliz Cloud instance.

This gives you the best of both worlds: powerful vector search with production-grade infrastructure, now accessible through plain English.


## Wrapping Up

And that’s it—you’ve just learned how to turn Milvus into a natural language-friendly vector database you can literally _talk to_. No more digging through SDK docs or writing boilerplate just to spin up a collection or run a search.

Whether you’re running Milvus locally or using Zilliz Cloud, the MCP Server gives your AI assistant a toolbox to manage your vector data like a pro. Just type what you want to do, and let Claude or Cursor handle the rest.

So go ahead—fire up your AI dev tool, ask “what collections do I have?” and see it in action. You’ll never want to go back to writing vector queries by hand.

- Local setup? Use the open-source[ Milvus MCP Server](https://github.com/zilliztech/mcp-server-milvus)

- Prefer a managed service? Sign up for Zilliz Cloud and use the[ Zilliz MCP Server](https://github.com/zilliztech/zilliz-mcp-server)

You’ve got the tools. Now let your AI do the typing. 


Talk to Your Vector Database: Managing Milvus via Natural Language



## Introduction 

Modern AI applications are becoming increasingly complex. You can't just throw one search method at a problem and call it done.

Take recommendation systems, for example—they require **vector search** to understand the meaning of text and images, **metadata filtering** to narrow results by price, category, or location, and **keyword search** for direct queries like "Nike Air Max." Each method solves a different part of the problem, and real-world systems need all of them working together.

The future of search isn’t about choosing between vector and keyword. It’s about combining vector AND keyword AND filtering, along with other search types—all in one place. That’s why we started building [hybrid search](https://milvus.io/docs/hybrid_search_with_milvus.md) into Milvus a year ago, with the release of Milvus 2.5.


## But Full-Text Search Works Differently

Bringing full-text search into a vector-native system isn’t easy. Full-text search has its own set of challenges.

While vector search captures the _semantic_ meaning of text—turning it into high-dimensional vectors—full-text search depends on understanding **the structure of language**: how words are formed, where they begin and end, and how they relate to one another. For instance, when a user searches for "running shoes" in English, the text goes through several processing steps: 

_Split on whitespace → lowercase → remove stopwords → stem "running" to "run"._

To handle this correctly, we need a robust **language analyzer**—one that handles splitting, stemming, filtering, and more.

When we introduced [BM25 full-text search](https://milvus.io/docs/full-text-search.md#Full-Text-Search) in Milvus 2.5, we included a customizable analyzer, and it worked well for what it was designed to do. You could define a pipeline using tokenizers, token filters, and character filters to prepare text for indexing and search.

For English, this setup was relatively straightforward. But things become more complex when you deal with multiple languages.


## The Challenge of Multilingual Full-Text Search

Multilingual full-text search introduces a range of challenges:

- **Complex languages need special treatment**: Languages like Chinese, Japanese, and Korean don’t use spaces between words. They need advanced tokenizers to segment characters into meaningful words. These tools may work well for a single language but rarely support multiple complex languages simultaneously.

- **Even similar languages can conflict**: English and French might both use whitespace to separate words, but once you apply language-specific processing like stemming or lemmatization, one language’s rules can interfere with the other’s. What improves accuracy for English might distort French queries—and vice versa.

In short, **different languages require different analyzers**. Trying to process Chinese text with an English analyzer leads to failure—there are no spaces to split on, and English stemming rules can corrupt Chinese characters.

The bottom line? Relying on a single tokenizer and analyzer for multilingual datasets makes it nearly impossible to ensure consistent, high-quality tokenization across all languages. And that leads directly to degraded search performance.

As teams began adopting full-text search in Milvus 2.5, we started hearing the same feedback:

_"This is perfect for our searches in English, but what about our multilingual customer support tickets?" "We love having both vector and BM25 search, but our dataset includes Chinese, Japanese, and English content." "Can we get the same search precision across all our languages?"_

These questions confirmed what we had already seen in practice: full-text search fundamentally differs from vector search. Semantic similarity works well across languages, but accurate text search requires a deep understanding of each language’s structure.

That’s why [Milvus 2.6](https://milvus.io/blog/introduce-milvus-2-6-built-for-scale-designed-to-reduce-costs.md) introduces a completely overhauled text analysis pipeline with comprehensive multi-language support. This new system automatically applies the correct analyzer for each language, enabling accurate and scalable full-text search across multilingual datasets, without manual configuration or compromise in quality.




## How Milvus 2.6 Enables Robust Multilingual Full-Text Search

After extensive research and development, we've built a suite of features that address different multilingual scenarios. Each approach solves the language-dependency problem in its own way.


### 1. Multi-Language Analyzer: Precision Through Control

The [**Multi-Language Analyzer**](https://milvus.io/docs/multi-language-analyzers.md#Multi-language-Analyzers) allows you to define different text processing rules for different languages within the same collection, instead of forcing all languages through the same analysis pipeline. 

**Here's how it works:** you configure language-specific analyzers and tag each document with its language during insertion. When performing a BM25 search, you specify which language analyzer to use for query processing. This ensures that both your indexed content and search queries are processed with the optimal rules for their respective languages.

**Perfect for:** Applications where you know the language of your content and want maximum search precision. Think multinational knowledge bases, localized product catalogs, or region-specific content management systems.

**The requirement:** You need to provide language metadata for each document. Currently only available for BM25 search operations.


### 2. Language Identifier Tokenizer: Automatic Language Detection

We know that manually tagging every piece of content isn't always practical. The [**Language Identifier Tokenizer**](https://milvus.io/docs/multi-language-analyzers.md#Overview) brings automatic language detection directly into the text analysis pipeline.

**Here's how it works:** This intelligent tokenizer analyzes incoming text, detects its language using sophisticated detection algorithms, and automatically applies the appropriate language-specific processing rules. You configure it with multiple analyzer definitions - one for each language you want to support, plus a default fallback analyzer.

We support two detection engines: `whatlang` for faster processing and `lingua` for higher accuracy. The system supports 71-75 languages, depending on your chosen detector. During both indexing and search, the tokenizer automatically selects the right analyzer based on detected language, falling back to your default configuration when detection is uncertain.

**Perfect for:** Dynamic environments with unpredictable language mixing, user-generated content platforms, or applications where manual language tagging isn't feasible.

**The trade-off:** Automatic detection adds processing latency and may struggle with very short text or mixed-language content. But for most real-world applications, the convenience significantly outweighs these limitations.


### 3. ICU Tokenizer: Universal Foundation

If the first two options feel like overkill, we've got something simpler for you. We've newly integrated the[ ICU (International Components for Unicode) tokenizer](https://milvus.io/docs/icu-tokenizer.md#ICU) into Milvus 2.6. ICU has been around forever - it's a mature, widely-used set of libraries that handles text processing for tons of languages and scripts. The cool thing is that it can handle various complex and simple languages all at once.

The ICU tokenizer is honestly a great default choice. It uses Unicode-standard rules for breaking up words, which makes it reliable for dozens of languages that don't have their own specialized tokenizers. If you just need something powerful and general-purpose that works well across multiple languages, ICU does the job.

**Limitation:** ICU still works within a single analyzer, so all your languages end up sharing the same filters. Want to do language-specific stuff like stemming or lemmatization? You'll run into the same conflicts we talked about earlier.

**Where it really shines:** We built ICU to work as the default analyzer within the multi-language or language identifier setups. It's basically your intelligent safety net for handling languages you haven't explicitly configured.


## See It in Action: Hands-On Demo

Enough theory—let’s dive into some code! Here’s how to use the new multilingual features in **pymilvus** to build a multilingual search collection.

We’ll start by defining some reusable analyzer configurations, then walk through **two complete examples**:

- Using the **Multi-Language Analyzer**

- Using the **Language Identifier Tokenizer**

👉 For the complete demo code, check out [this GitHub page](https://github.com/milvus-io/pymilvus/tree/master/examples/full_text_search).



### Step 1: Set up the Milvus Client

_First, we connect to Milvus, set a collection name, and clean up any existing collections to start fresh._

```
from pymilvus import MilvusClient, DataType, Function, FunctionType

# 1. Setup Milvus Client
client = MilvusClient("http://localhost:19530")
COLLECTION_NAME = "multilingual_test"
if client.has_collection(collection_name=COLLECTION_NAME):
    client.drop_collection(collection_name=COLLECTION_NAME)
```


### Step 2: Define Analyzers for Multiple Languages

Next, we define an `analyzers` dictionary with language-specific configurations. These will be used in both multilingual search methods shown later.

```
# 2. Define analyzers for multiple languages
# These individual analyzer definitions will be reused by both methods.
analyzers = {
    "Japanese": { 
        # Use lindera with japanese dict 'ipadic' 
        # and remove punctuation beacuse lindera tokenizer will remain punctuation
        "tokenizer":{
            "type": "lindera",
            "dict_kind": "ipadic"
        },
        "filter": ["removepunct"]
    },
    "English": {
        # Use build-in english analyzer
        "type": "english",
    },
    "default": {
        # use icu tokenizer as a fallback.
        "tokenizer": "icu",
    }
}
```


### Option A: Using The Multi-Language Analyzer

This approach is best when you **know the language of each document ahead of time**. You'll pass that information through a dedicated ` language` field during data insertion.


#### Create a Collection with Multi-Language Analyzer

We’ll create a collection where the ` "text"` field uses different analyzers depending on the `language` field value.

```
# --- Option A: Using Multi-Language Analyzer ---
print("\n--- Demonstrating Multi-Language Analyzer ---")

# 3A. reate a collection with the Multi Analyzer

mutil_analyzer_params = {
    "by_field": "language",
    "analyzers": analyzers,
}

schema = MilvusClient.create_schema(
    auto_id=True,
    enable_dynamic_field=False,
)
schema.add_field(field_name="id", datatype=DataType.INT64, is_primary=True)# Apply our multi-language analyzer to the 'title' field
schema.add_field(field_name="language", datatype=DataType.VARCHAR, max_length=255, nullable = True)
schema.add_field(field_name="text", datatype=DataType.VARCHAR, max_length=25565, enable_analyzer=True, multi_analyzer_params = mutil_analyzer_params)
schema.add_field(field_name="text_sparse", datatype=DataType.SPARSE_FLOAT_VECTOR) # Bm25 Sparse Vector

# add bm25 function
text_bm25_function = Function(
    name="text_bm25",
    function_type=FunctionType.BM25,
    input_field_names=["text"],
    output_field_names=["text_sparse"],
)
schema.add_function(text_bm25_function)

index_params = client.prepare_index_params()
index_params.add_index(
    field_name="text_sparse",
    index_type="AUTOINDEX", # Use auto index for BM25
    metric_type="BM25",
)

client.create_collection(
    collection_name=COLLECTION_NAME,
    schema=schema,
    index_params=index_params
)
print(f"Collection '{COLLECTION_NAME}' created successfully.")
```


#### Insert Multilingual Data and Load Collection

Now insert documents in English and Japanese. The `language` field tells Milvus which analyzer to use.

```
# 4A. Insert data for Multi-Language Analyzer and load collection# Insert English and Japanese movie titles, explicitly setting the 'language' field
client.insert(
    collection_name=COLLECTION_NAME,
    data=[
        {"text": "The Lord of the Rings", "language": "English"},
        {"text": "Spirited Away", "language": "English"},
        {"text": "千と千尋の神隠し", "language": "Japanese"}, # This is "Spirited Away" in Japanese
        {"text": "君の名は。", "language": "Japanese"}, # This is "Your Name." in Japanese
    ]
)
print(f"Inserted multilingual data into '{COLLECTION_NAME}'.")

# Load the collection into memory before searching
client.load_collection(collection_name=COLLECTION_NAME)
```


#### Run Full-Text Search

To search, specify which analyzer to use for the query based on its language.

```
# 5A. Perform a full-text search with Multi-Language Analyzer# When searching, explicitly specify the analyzer to use for the query string.
print("\n--- Search results for Multi-Language Analyzer ---")
results_multi_jp = client.search(
    collection_name=COLLECTION_NAME,
    data=["神隠し"],
    limit=2,
    output_fields=["text"],
    search_params={"metric_type": "BM25", "analyzer_name": "Japanese"}, # Specify Japanese analyzer for query
    consistency_level = "Strong",
)
print("\nSearch results for '神隠し' (Multi-Language Analyzer):")
for result in results_multi_jp[0]:
    print(result)

results_multi_en = client.search(
    collection_name=COLLECTION_NAME,
    data=["Rings"],
    limit=2,
    output_fields=["text"],
    search_params={"metric_type": "BM25", "analyzer_name": "English"}, # Specify English analyzer for query
    consistency_level = "Strong",
)
print("\nSearch results for 'Rings' (Multi-Language Analyzer):")
for result in results_multi_en[0]:
    print(result)

client.drop_collection(collection_name=COLLECTION_NAME)
print(f"Collection '{COLLECTION_NAME}' dropped.")
```


#### Results: 

![](https://assets.zilliz.com/1_results_561f628de3.png)


### Option B: Using the Language Identifier Tokenizer

This approach takes the manual language handling out of your hands. The **Language Identifier Tokenizer** automatically detects the language of each document and applies the correct analyzer—no need to specify a `language` field.


#### Create a Collection with Language Identifier Tokenizer

Here, we create a collection where the `"text"` field uses automatic language detection to choose the right analyzer.

```
# --- Option B: Using Language Identifier Tokenizer ---
print("\n--- Demonstrating Language Identifier Tokenizer ---")

# 3A. create a collection with language identifier
analyzer_params_langid = {
    "tokenizer": {
        "type": "language_identifier",
        "analyzers": analyzers # Referencing the analyzers defined in Step 2
    },
}

schema_langid = MilvusClient.create_schema(
    auto_id=True,
    enable_dynamic_field=False,
)
schema_langid.add_field(field_name="id", datatype=DataType.INT64, is_primary=True)
# The 'language' field is not strictly needed by the analyzer itself here, as detection is automatic.# However, you might keep it for metadata purposes.
schema_langid.add_field(field_name="text", datatype=DataType.VARCHAR, max_length=25565, enable_analyzer=True, analyzer_params = analyzer_params_langid)
schema_langid.add_field(field_name="text_sparse", datatype=DataType.SPARSE_FLOAT_VECTOR) # BM25 Sparse Vector# add bm25 function
text_bm25_function_langid = Function(
    name="text_bm25",
    function_type=FunctionType.BM25,
    input_field_names=["text"],
    output_field_names=["text_sparse"],
)
schema_langid.add_function(text_bm25_function_langid)

index_params_langid = client.prepare_index_params()
index_params_langid.add_index(
    field_name="text_sparse",
    index_type="AUTOINDEX", # Use auto index for BM25
    metric_type="BM25",
)

client.create_collection(
    collection_name=COLLECTION_NAME,
    schema=schema_langid,
    index_params=index_params_langid
)
print(f"Collection '{COLLECTION_NAME}' created successfully with Language Identifier Tokenizer.")
```


#### Insert Data and Load Collection

Insert text in different languages—no need to label them. Milvus detects and applies the correct analyzer automatically.

```
# 4B. Insert Data for Language Identifier Tokenizer and Load Collection
# Insert English and Japanese movie titles. The language_identifier will detect the language.
client.insert(
    collection_name=COLLECTION_NAME,
    data=[
        {"text": "The Lord of the Rings"},
        {"text": "Spirited Away"},
        {"text": "千と千尋の神隠し"}, 
        {"text": "君の名は。"},
    ]
)
print(f"Inserted multilingual data into '{COLLECTION_NAME}'.")

# Load the collection into memory before searching
client.load_collection(collection_name=COLLECTION_NAME)
```


#### Run Full-Text Search

Here’s the best part: **no need to specify an analyzer** when searching. The tokenizer automatically detects the query language and applies the right logic.

```
# 5B. Perform a full-text search with Language Identifier Tokenizer# No need to specify analyzer_name in search_params; it's detected automatically for the query.
print("\n--- Search results for Language Identifier Tokenizer ---")
results_langid_jp = client.search(
    collection_name=COLLECTION_NAME,
    data=["神隠し"],
    limit=2,
    output_fields=["text"],
    search_params={"metric_type": "BM25"}, # Analyzer automatically determined by language_identifier
    consistency_level = "Strong",
)
print("\nSearch results for '神隠し' (Language Identifier Tokenizer):")
for result in results_langid_jp[0]:
    print(result)

results_langid_en = client.search(
    collection_name=COLLECTION_NAME,
    data=["the Rings"],
    limit=2,
    output_fields=["text"],
    search_params={"metric_type": "BM25"}, # Analyzer automatically determined by language_identifier
    consistency_level = "Strong",
)
print("\nSearch results for 'the Rings' (Language Identifier Tokenizer):")
for result in results_langid_en[0]:
    print(result)

client.drop_collection(collection_name=COLLECTION_NAME)
print(f"Collection '{COLLECTION_NAME}' dropped.")
```


#### Results

![](https://assets.zilliz.com/2_results_486712c3f6.png)


## Conclusion

Milvus 2.6 takes a big step forward in making **hybrid search** more powerful and accessible, combining vector search with keyword search, now across multiple languages. With the enhanced multilingual support, you can build apps that understand _what users mean_ and _what they say_, no matter what language they're using.

But that’s just one part of the update. Milvus 2.6 also brings several other features that make search faster, smarter, and easier to work with:

- **Better Query Matching** – Use `phrase_match` and `multi_match` for more accurate searches

- **Faster JSON Filtering** – Thanks to a new, dedicated index for JSON fields

- **Scalar-Based Sorting** – Sort results by any numeric field

- **Advanced Reranking** – Reorder results using models or custom scoring logic

Want the complete breakdown of Milvus 2.6? Check out our latest post: [**Introducing Milvus 2.6: Affordable Vector Search at Billion Scale**](https://milvus.io/blog/introduce-milvus-2-6-built-for-scale-designed-to-reduce-costs.md)**.** 

Have questions or want a deep dive on any feature? Join our[ Discord channel](https://discord.com/invite/8uyFbECzPX) or file issues on[ GitHub](https://github.com/milvus-io/milvus).


How Milvus How Milvus 2.6 Upgrades Multilingual Full-Text Search at Scale


How Milvus 2.6 Upgrades Multilingual Full-Text Search at Scale



The AI coding assistant battlefield is heating up fast. We've seen [Claude Code](https://milvus.io/blog/claude-code-vs-gemini-cli-which-ones-the-real-dev-co-pilot.md#Claude-Code-or-Gemini-CLI-Choose-Your-Co-Pilot-Wisely) from Anthropic making waves, Google's [Gemini CLI](https://milvus.io/blog/claude-code-vs-gemini-cli-which-ones-the-real-dev-co-pilot.md#Claude-Code-or-Gemini-CLI-Choose-Your-Co-Pilot-Wisely) shaking up terminal workflows, OpenAI's Codex powering GitHub Copilot, Cursor winning over VS Code users, and **now Alibaba Cloud enters with Qwen Code.**

Honestly, this is great news for developers. More players mean better tools, innovative features, and most importantly, **open-source alternatives** to expensive proprietary solutions. Let's learn what this latest player brings to the table.


## Meet Qwen3-Coder and Qwen Code

Alibaba Cloud recently released[ **Qwen3-Coder**](https://github.com/QwenLM/Qwen3-Coder), an open-source agentic coding model achieving state-of-the-art results across multiple benchmarks. They also launched[ **Qwen Code**](https://github.com/QwenLM/qwen-code), an open-source AI coding CLI tool built on Gemini CLI but enhanced with specialized parsers for Qwen3-Coder. 

The flagship model, **Qwen3-Coder-480B-A35B-Instruct**, delivers impressive capabilities: native support for 358 programming languages, 256K token context window (expandable to 1M tokens via YaRN), and seamless integration with Claude Code, Cline, and other coding assistants.


## The Universal Blind Spot in Modern AI Coding Copilots

While Qwen3-Coder is powerful, I am more interested in its coding assistant: **Qwen Code**. Here's what I found interesting. Despite all the innovation, Qwen Code shares the exact same limitation as Claude Code and Gemini CLI: **_they're great at generating fresh code but struggle with understanding existing codebases._**

Take this example: you ask Gemini CLI or Qwen Code to "find where this project handles user authentication." The tool starts hunting for obvious keywords like "login" or "password" but completely misses that critical `verifyCredentials()` function. Unless you're willing to burn through tokens by feeding your entire codebase as context—which is both expensive and time-consuming—these tools hit a wall pretty quickly.

**_This is the real gap in today's AI tooling: intelligent code context understanding._**


## Supercharge Any Coding Copilot with Semantic Code Search

What if you could give any AI coding copilot—whether it's Claude Code, Gemini CLI, or Qwen Code—the ability to truly understand your codebase semantically? What if you could build something as powerful as Cursor for your own projects without the hefty subscription fees, while maintaining complete control over your code and data?

Well, enter[ **Code Context**](https://github.com/zilliztech/code-context)—an open-source, MCP-compatible plugin that transforms any AI coding agent into a context-aware powerhouse. It's like giving your AI assistant the institutional memory of a senior developer who's worked on your codebase for years. Whether you're using Qwen Code, Claude Code, Gemini CLI, working in VSCode, or even coding in Chrome, **Code Context** brings semantic code search to your workflow.

Ready to see how this works? Let's build an enterprise-grade AI coding copilot using **Qwen3-Coder + Qwen Code + Code Context**.


## Hands-On Tutorial: Building Your Own AI Coding Copilot 

### Prerequisites

Before we begin, ensure you have:

- **Node.js 20+** installed

- **OpenAI API key** ([Get one here](https://openai.com/index/openai-api/))

- **Alibaba Cloud account** for Qwen3-Coder access ([get one here](https://www.alibabacloud.com/en)) 

- **Zilliz Cloud account** for vector database ([Register here](https://cloud.zilliz.com/login) for free if you don’t have one yet)

**Notes: 1)** In this tutorial, we’ll use Qwen3-Coder-Plus, the commercial version of Qwen3-Coder, because of its strong coding capabilities and ease of use. If you prefer an open-source option, you can use qwen3-coder-480b-a35b-instruct instead. 2) While Qwen3-Coder-Plus offers excellent performance and usability, it comes with high token consumption. Be sure to factor this into your enterprise budgeting plans.


### Step 1: Environment Setup

Verify your Node.js installation:

```
curl -qL https://www.npmjs.com/install.sh | sh
```



### Step 2: Install Qwen Code

```
npm install -g @qwen-code/qwen-code
qwen --version
```


If you see the version number like below, it means the installation was successful.

![](https://assets.zilliz.com/1_0d5ebc152e.png)


### Step 3: Configure Qwen Code

Navigate to your project directory and initialize Qwen Code.

```
Qwen
```


Then, you’ll see a page like below. 

![](https://assets.zilliz.com/2_e6598ea982.png)

**API Configuration Requirements:**

- API Key: Obtain from[ Alibaba Cloud Model Studio](https://modelstudio.console.alibabacloud.com/)

- Base URL: `https://dashscope.aliyuncs.com/compatible-mode/v1`

- Model Selection:

  - `qwen3-coder-plus` (commercial version, most capable)

  - `qwen3-coder-480b-a35b-instruct` (open-source version)

![](https://assets.zilliz.com/3_5ed0c54084.png)

After configuration, press **Enter** to proceed.


### Step 4: Test Basic Functionality

Let's verify your setup with two practical tests:

**Test 1: Code Understanding** 

Prompt: "Summarize this project's architecture and main components in one sentence."

![](https://assets.zilliz.com/4_41e601fc82.png)

Qwen3-Coder-Plus nailed the summary—describing the project as a technical tutorial built on Milvus, with a focus on RAG systems, retrieval strategies, and more.

**Test 2: Code Generation**

Prompt: "Please create a small game of Tetris"

In under a minute, Qwen3-coder-plus:

- Autonomously installs required libraries

- Structures the game logic

- Creates a complete, playable implementation

- Handles all the complexity you'd normally spend hours researching

![](https://assets.zilliz.com/6_c67e1725eb.png)

![](https://assets.zilliz.com/7_fd91d5a290.gif)

This showcases true autonomous development—not just code completion, but architectural decision-making and complete solution delivery.


### Step 5: Set Up Your Vector Database

We’ll use [Zilliz Cloud](https://zilliz.com/cloud) as our vector database in this tutorial. 

**Create a Zilliz Cluster:**

1. Log into[ Zilliz Cloud Console](https://cloud.zilliz.com/)

2. Create a new cluster

3. Copy the **Public Endpoint** and **Token**

![](https://assets.zilliz.com/8_5e692e6e80.png)

![](https://assets.zilliz.com/9_753f281055.png)


### Step 6: Configure Code Context Integration

Create `~/.qwen/settings.json`:

```
{
  "mcpServers": {
    "code-context": {
      "command": "npx",
      "args": ["@zilliz/code-context-mcp@latest"],
      "env": {
        "OPENAI_API_KEY": "sk-xxxxxxxxxx",
        "MILVUS_ADDRESS": "https://in03-xxxx.cloud.zilliz.com",
        "MILVUS_TOKEN": "4f699xxxxx"
      },
      "cwd": "./server-directory",
      "timeout": 30000,
      "trust": false
    }
  }
}
```



### Step 7: Activate Enhanced Capabilities

Restart Qwen Code:

```
Qwen
```


Press **Ctrl + T** to see three new tools within our MCP server: 

- `index-codebase`: Creates semantic indexes for repository understanding

- `search-code`: Natural language code search across your codebase

- `clear-index`: Resets indexes when needed.

![](https://assets.zilliz.com/10_bebbb44460.png)


### Step 8: Test the Complete Integration

Here's a real example: In a big project, we reviewed code names and found that 'wider window' sounded unprofessional, so we decided to change it.

Prompt: "Find all functions related to 'wider window' that need professional renaming."

![](https://assets.zilliz.com/11_c54398c4f2.png)

As shown in the figure below, qwen3-coder-plus first called the `index_codebase` tool to create an index for the entire project.

![](https://assets.zilliz.com/12_25a7f3a039.png)

Then, the `index_codebase` tool created indexes for 539 files in this project, splitting them into 9,991 chunks. Immediately after building the index, it called the `search_code `tool to perform the query.

![](https://assets.zilliz.com/13_6766663346.png)

Next, it informed us that it found the corresponding files that needed modification.

![](https://assets.zilliz.com/14_7b3c7e9cc0.png)

Finally, it discovered 4 issues using Code Context, including functions, imports, and some naming in documentation, helping us complete this small task.

![](https://assets.zilliz.com/15_a529905b28.png)

With the addition of Code Context, `qwen3-coder-plus` now offers smarter code search and better understanding of coding environments. 


### What You've Built

You now have a complete AI coding copilot that combines:

- **Qwen3-Coder**: Intelligent code generation and autonomous development

- **Code Context**: Semantic understanding of existing codebases

- **Universal compatibility**: Works with Claude Code, Gemini CLI, VSCode, and more

This isn't just faster development—it enables entirely new approaches to legacy modernization, cross-team collaboration, and architectural evolution.


## Conclusion

As a developer, I’ve tried plenty of AI coding tools—from Claude Code to Cursor and Gemini CLI, and to Qwen Code—and while they’re great at generating new code, they usually fall flat when it comes to understanding existing codebases. That’s the real pain point: not writing functions from scratch, but navigating complex, messy, legacy code and figuring out _why_ things were done a certain way.

That’s what makes this setup with **Qwen3-Coder + Qwen Code+ Code Context** so compelling. You get the best of both worlds: a powerful coding model that can generate full-featured implementations _and_ a semantic search layer that actually understands your project history, structure, and naming conventions.

With vector search and the MCP plugin ecosystem, you're no longer stuck pasting random files into the prompt window or scrolling through your repo trying to find the right context. You just ask in plain language, and it finds the relevant files, functions, or decisions for you—like having a senior dev who remembers everything.

To be clear, this approach isn’t just faster—it actually changes how you work. It’s a step toward a new kind of development workflow where AI isn’t just a coding helper, but an architectural assistant, a teammate who gets the whole project context.

_That said... fair warning: Qwen3-Coder-Plus is amazing, but very token-hungry. Just building this prototype burned through 20 million tokens. So yeah—I’m now officially out of credits 😅_

__


Build a Coding Copilot with Qwen3-Coder & Code Context


Hands-on Tutorial: Build Your Own Coding Copilot with Qwen3-Coder, Qwen Code, and Code Context


Build a Production-Grade Chatbot with Kimi K2 and Milvus





## The AI Coding Boom—And Its Blind Spot

AI coding tools are everywhere—and they’re going viral for good reason. From [Claude Code, Gemini CLI](https://milvus.io/blog/claude-code-vs-gemini-cli-which-ones-the-real-dev-co-pilot.md) to open-source Cursor alternatives, these agents can write functions, explain code dependency, and refactor entire files with a single prompt. Developers are racing to integrate them into their workflows, and in many ways, they’re delivering on the hype.

**But when it comes to _understanding your codebase_, most AI tools hit a wall.**

Ask Claude Code to find “where this project handles user authentication,” and it falls back on `grep -r "auth"`—spitting out 87 loosely related matches across comments, variable names, and filenames, likely missing many functions with authentication logic but not called “auth”. Try Gemini CLI, and it’ll look for keywords like “login” or “password,” missing functions like `verifyCredentials()` entirely. These tools are great at generating code, but when it’s time to navigate, debug, or explore unfamiliar systems, they fall apart. Unless they send the entire codebase to the LLM for context—burning through tokens and time—they struggle to provide meaningful answers.

_That’s the real gap in today’s AI tooling:_ **_code context._** 


## Cursor Nailed It—But Not for Everyone

**Cursor** tackles this head-on. Instead of keyword search, it builds a semantic map of your codebase using syntax trees, vector embeddings, and code-aware search. Ask it “where’s the email validation logic?” and it returns `isValidEmailFormat()` —not because the name matches, but because it understands what that code _does_.

While Cursor is powerful, it may not be suitable for everyone. **_Cursor is closed-source, cloud-hosted, and subscription-based._** That puts it out of reach for teams working with sensitive code, security-conscious organizations, indie developers, students, and anyone who prefers open systems.


## What if You Could Build Your Own Cursor?

Here’s the thing: the core technology behind Cursor isn’t proprietary. It’s built on proven open-source foundations—vector databases like [Milvus](https://milvus.io/), [embedding models](https://zilliz.com/ai-models), syntax parsers with Tree-sitter—all available to anyone willing to connect the dots.

_So, we asked:_ **_What if anyone could build their own Cursor?_** Runs on your infrastructure. No subscription fees. Fully customizable. Complete control over your code and data.

That’s why we built [**Code Context**](https://github.com/zilliztech/code-context)—an open-source, MCP-compatible plugin that brings powerful semantic code search to any AI coding agent, such as Claude Code and Gemini CLI, IDEs like VSCode, and even environments like Google Chrome. It also gives you the power to build your own coding agent like Cursor from scratch, unlocking real-time, intelligent navigation of your codebase.

**_No subscriptions. No black boxes. Just code intelligence—on your terms._**

In the rest of this post, we’ll walk through how Code Context works—and how you can start using it today.


## Code Context: Open-Source Alternative to Cursor's Intelligence

[**Code Context**](https://github.com/zilliztech/code-context) is an open-source, MCP-compatible semantic code search engine. Whether you're building a custom AI coding assistant from scratch or adding semantic awareness to AI coding agents like Claude Code and Gemini CLI, Code Context is the engine that makes it possible.

It runs locally, integrates with your favorite tools and environments, such as VS Code and Chrome browsers, and delivers robust code understanding without relying on cloud-only, closed-source platforms.

**Core capabilities include:**

- **Semantic Code Search via Natural Language:** Find code using plain English. Search for concepts like “user login verification” or “payment processing logic,” and Code Context locates the relevant functions—even if they don’t match the keywords exactly.

- **Multi-Language Support:** Search seamlessly across 15+ programming languages, including JavaScript, Python, Java, and Go, with consistent semantic understanding across them all.

- **AST-Based Code Chunking:** Code is automatically split into logical units, such as functions and classes, using AST parsing, ensuring search results are complete, meaningful, and never cut off mid-function.

- **Live, Incremental Indexing:** Code changes are indexed in real time. As you edit files, the search index stays up to date—no need for manual refreshes or re-indexing.

- **Fully Local, Secure Deployment:** Run everything on your own infrastructure. Code Context supports local models via Ollama and indexing via [Milvus](https://milvus.io/), so your code never leaves your environment.

- **First-Class IDE Integration:** The VSCode extension lets you search and jump to results instantly—right from your editor, with zero context switching.

- **MCP Protocol Support:** Code Context speaks MCP, making it easy to integrate with AI coding assistants and bring semantic search directly into their workflows.

- **Browser Plugin Support:** Search repositories directly from GitHub in your browser—no tabs, no copy-pasting, just instant context wherever you’re working.


### How Code Context Works

![](https://assets.zilliz.com/How_Code_Context_Works_3faaa2fff3.png)

Code Context uses a modular architecture with a core orchestrator and specialized components for embedding, parsing, storage, and retrieval. 


### The Core Module: Code Context Core 

At the heart of Code Context is the **Code Context Core**, which coordinates code parsing, embedding, storage, and semantic retrieval:

- **Text Processing Module** splits and parses code using Tree-sitter for language-aware AST analysis.

- **Embedding Interface** supports pluggable backends—currently OpenAI and VoyageAI—converting code chunks into vector embeddings that capture their semantic meaning and contextual relationships. 

- **The Vector Database Interface** stores these embeddings in a self-hosted [Milvus](https://milvus.io/) instance (by default) or in [Zilliz Cloud](https://zilliz.com/cloud), the managed version of Milvus. 

All of this is synchronized with your file system on a scheduled basis, ensuring the index stays up to date without requiring manual intervention.


### Extension Modules on top of Code Context Core

- **VSCode Extension**: Seamless IDE integration for fast in-editor semantic search and jump-to-definition.

- **Chrome Extension**: Inline semantic code search while browsing GitHub repositories—no need to switch tabs.

- **MCP Server**: Exposes Code Context to any AI coding assistants via the MCP protocol, enabling real-time, context-aware assistance.


## Getting Started with Code Context 

Code Context can be plugged into coding tools you already use or to build a custom AI coding assistant from scratch. In this section, we’ll walk through both scenarios:

- How to integrate Code Context with existing tools 

- How to set up the Core Module for standalone semantic code search when building your own AI coding assistant 


### MCP Integration

Code Context supports **Model Context Protocol (MCP)**, allowing AI coding agents like Claude Code to use it as a semantic backend.

To integrate with Claude Code:

```
claude mcp add code-context -e OPENAI_API_KEY=your-openai-api-key -e MILVUS_ADDRESS=your-zilliz-cloud-public-endpoint -e MILVUS_TOKEN=your-zilliz-cloud-api-key -- npx @zilliz/code-context-mcp@latest
```

![](https://assets.zilliz.com/mcp_code_context_processed_3908097481.gif)

Once configured, Claude Code will automatically call Code Context for semantic code search when needed.

To integrate with other tools or environments, check out our[ GitHub repo](https://github.com/zilliztech/code-context) for more examples and adapters.

![](https://assets.zilliz.com/MCP_Integration_2_683c7ef73d.png)


### Building Your Own AI Coding Assistant with Code Context

To build a custom AI assistant using Code Context, you'll set up the core module for semantic code search in just three steps:

1. Configure your embedding model

2. Connect to your vector database

3. Index your project and start searching

Here’s an example using **OpenAI Embeddings** and [**Zilliz Cloud**](https://zilliz.com/cloud) **vector database** as the vector backend:

```
import { CodeContext, MilvusVectorDatabase, OpenAIEmbedding } from '@zilliz/code-context-core';

// Initialize embedding model
const embedding = new OpenAIEmbedding({
    apiKey: 'your-openai-api-key',
    model: 'text-embedding-3-small'
});

// Initialize Zilliz Cloud vector database
// Sign up for free at https://zilliz.com/cloud
const vectorDatabase = new MilvusVectorDatabase({
    address: 'https://xxx-xxxxxxxxxxxx.serverless.gcp-us-west1.cloud.zilliz.com',
    token: 'xxxxxxx'
});

// Create the Code Context indexer
const context = new CodeContext({ embedding, vectorDatabase });

// Index the codebase
await context.indexCodebase('./your-project');

// Perform semantic code search
const results = await context.semanticSearch('./your-project', 'vector database operations', 5);
results.forEach(result => {
    console.log(`${result.relativePath}:${result.startLine}-${result.endLine}`);
    console.log(`score: ${(result.score * 100).toFixed(2)}%`);
});
```



### VSCode Extension

Code Context is available as a VSCode extension named **"Semantic Code Search"**, which brings intelligent, natural language–driven code search directly into your editor.

![](https://assets.zilliz.com/VS_Code_Extension_e358f36464.png)


Once installed:

- Configure your API key

- Index your project

- Use plain English queries (no exact match needed)

- Jump to results instantly with click-to-navigate

![](https://assets.zilliz.com/mcp_code_context_processed_3908097481.gif)


This makes semantic exploration a native part of your coding workflow—no terminal or browser required.


### Chrome Extension (Coming Soon)

Our upcoming **Chrome extension** brings Code Context to GitHub web pages, allowing you to run semantic code search directly inside any public repository—no context switching or tabs required.

![](https://assets.zilliz.com/chrome_4e67b683d7.gif)

You’ll be able to explore unfamiliar codebases with the same deep search capabilities you have locally. Stay tuned—the extension is in development and launching soon.


## Why Use Code Context?

The basic setup gets you running quickly, but where **Code Context** truly shines is in professional, high-performance development environments. Its advanced features are designed to support serious workflows, from enterprise-scale deployments to custom AI tooling.


### Private Deployment for Enterprise-Grade Security

Code Context supports fully offline deployment using the **Ollama** local embedding model and **Milvus** as a self-hosted vector database. This enables an entirely private code search pipeline: no API calls, no internet transmission, and no data ever leaves your local environment.

This architecture is ideal for industries with strict compliance requirements—such as finance, government, and defense—where code confidentiality is non-negotiable.


### Real-Time Indexing with Intelligent File Sync

Keeping your code index up to date shouldn’t be slow or manual. Code Context includes a **Merkle Tree–based file monitoring system** that detects changes instantly and performs incremental updates in real time.

![](https://assets.zilliz.com/workflow_0fd958fe81.jpeg)

By only re-indexing modified files, it reduces update times for large repositories from minutes to seconds. This ensures the code you just wrote is already searchable, without needing to click “refresh.”

In fast-paced dev environments, that kind of immediacy is critical.


### AST Parsing That Understands Code Like You Do

Traditional code search tools split text by line or character count, often breaking logical units and returning confusing results.

![](https://assets.zilliz.com/AST_Parsing_That_Understands_Code_Like_You_Do_3236afc075.png)

Code Context does better. It utilizes Tree-sitter AST parsing to comprehend the actual code structure. It identifies complete functions, classes, interfaces, and modules, providing clean, semantically complete results.

It supports major programming languages, including JavaScript/TypeScript, Python, Java, C/C++, Go, and Rust, with language-specific strategies for accurate chunking. For unsupported languages, it falls back to rule-based parsing, ensuring graceful handling without crashes or empty results.

These structured code units also feed into metadata for more accurate semantic search.


### Open Source and Extensible by Design

Code Context is fully open source under the MIT license. All core modules are publicly available on GitHub.

We believe open infrastructure is the key to building powerful, trustworthy developer tools—and invite developers to extend it for new models, languages, or use cases.


### Solving the Context Window Problem for AI Assistants

Large language models (LLMs) have a hard limit: their context window. This restricts them from seeing an entire codebase, which reduces the accuracy of completions, fixes, and suggestions.

Code Context helps bridge that gap. Its semantic code search retrieves the _right_ pieces of code, giving your AI assistant focused, relevant context to reason with. It improves the quality of AI-generated output by letting the model “zoom in” on what actually matters.

Popular AI coding tools, such as Claude Code and Gemini CLI, lack native semantic code search—they rely on shallow, keyword-based heuristics. Code Context, when integrated via **MCP**, gives them a brain upgrade.


### Built for Developers, by Developers

Code Context is packaged for modular reuse: each component is available as an independent **npm** package. You can mix, match, and extend as needed for your project.

- Need only a semantic code search? Use`@zilliz/code-context-core`

- Want to plug into an AI agent? Add `@zilliz/code-context-mcp`

- Building your own IDE/browser tool? Fork our VSCode and Chrome extension examples

Some example applications of code context:

- **Context-aware autocomplete plugins** that pull relevant snippets for better LLM completions

- **Intelligent bug detectors** that gather surrounding code to improve fix suggestions

- **Safe code refactoring tools** that find semantically related locations automatically

- **Architecture visualizers** that build diagrams from semantic code relationships

- **Smarter code review assistants** that surface historical implementations during PR reviews


## Welcome to Join Our Community

[**Code Context**](https://github.com/zilliztech/code-context) is more than just a tool—it’s a platform for exploring how **AI and vector databases** can work together to truly understand code. As AI-assisted development becomes the norm, we believe semantic code search will be a foundational capability.

We welcome contributions of all kinds:

- Support for new languages

- New embedding model backends

- Innovative AI-assisted workflows

- Feedback, bug reports, and design ideas

Find us here:

- [Code Context on GitHub](https://github.com/zilliztech/code-context) | [**MCP npm package**](https://www.npmjs.com/package/@zilliz/code-context-mcp) | [**VSCode marketplace**](https://marketplace.visualstudio.com/items?itemName=zilliz.semanticcodesearch)

- [Discord](https://discuss.milvus.io/) | [LinkedIn](https://www.linkedin.com/company/the-milvus-project/) | [X](https://x.com/zilliz_universe) | [YouTube](https://www.youtube.com/@MilvusVectorDatabase/featured)

Together, we can build the infrastructure for the next generation of AI development tools—transparent, powerful, and developer-first.

[![](https://assets.zilliz.com/office_hour_83d4623510.png)](https://meetings.hubspot.com/chloe-williams1/milvus-office-hour)


Building an Open-Source Alternative to Cursor with Code Context



Every day on tech “X” (formerly Twitter), you see developers showing off their setups—automated deployment pipelines that handle complex multi-environment releases without a hitch; monitoring systems that intelligently route alerts to the right team members based on service ownership; development workflows that automatically sync GitHub issues with project management tools and notify stakeholders at exactly the right moments.

These seemingly "advanced" operations all share the same secret: **workflow automation tools.**

Think about it. A pull request gets merged, and the system automatically triggers tests, deploys to staging, updates the corresponding Jira ticket, and notifies the product team in Slack. A monitoring alert fires, and instead of spamming everyone, it intelligently routes to the service owner, escalates based on severity, and automatically creates incident documentation. A new team member joins, and their development environment, permissions, and onboarding tasks get provisioned automatically.

These integrations that used to require custom scripts and constant maintenance now run themselves 24/7 once you set them up properly. 

Recently, I discovered [N8N](https://github.com/Zie619/n8n-workflows), a visual workflow automation tool, and more importantly, stumbled upon an open-source repository containing over 2000 ready-to-use workflow templates. This post will walk you through what I learned about workflow automation, why N8N caught my attention, and how you can leverage these pre-built templates to set up sophisticated automation in minutes instead of building everything from scratch.


## Workflow: Let Machines Handle the Grunt Work

### What is workflow?

At its core, workflow is just a set of automated task sequences. Picture this: you take a complex process and break it down into smaller, manageable chunks. Each chunk becomes a "node" that handles one specific job—maybe calling an API, processing some data, or sending a notification. String these nodes together with some logic, add a trigger, and you've got a workflow that runs itself.

Here's where it gets practical. You can set up workflows to automatically save email attachments to Google Drive when they arrive, scrape website data on a schedule and dump it into your database, or route customer tickets to the right team members based on keywords or priority levels.


### Workflow vs AI Agent: Different Tools for Different Jobs

Before we go further, let's clear up some confusion. A lot of developers mix up workflows with AI agents, and while both can automate tasks, they're solving completely different problems.

- **Workflows** follow predefined steps with no surprises. They're triggered by specific events or schedules and are perfect for repetitive tasks with clear steps like data syncing and automated notifications.

- **AI Agents** make decisions on the fly and adapt to situations. They continuously monitor and decide when to act, making them ideal for complex scenarios requiring judgment calls like chatbots or automated trading systems.

| **What We're Comparing** | **Workflows**                          | **AI Agents**                                    |
| ------------------------ | -------------------------------------- | ------------------------------------------------ |
| How It Thinks            | Follows predefined steps, no surprises | Makes decisions on the fly, adapts to situations |
| What Triggers It         | Specific events or schedules           | Continuously monitors and decides when to act    |
| Best Used For            | Repetitive tasks with clear steps      | Complex scenarios requiring judgment calls       |
| Real-World Examples      | Data syncing, automated notifications  | Chatbots, automated trading systems              |

For most of the automation headaches you face daily, workflows will handle about 80% of your needs without the complexity.


## Why N8N Caught My Attention

The workflow tool market is pretty crowded, so why did N8N catch my attention? It all comes down to one key advantage: [**N8N**](https://github.com/Zie619/n8n-workflows) **uses a graph-based architecture that actually makes sense for how developers think about complex automation.**


### Why Visual Representation Actually Matters for Workflows 

N8N lets you build workflows by connecting nodes on a visual canvas. Each node represents a step in your process, and the lines between them show how data flows through your system. This isn't just eye candy—it's a fundamentally better way to handle complex, branching automation logic.

![](https://assets.zilliz.com/n8n1_3bcae91c82.png)

N8N brings enterprise-grade capabilities to the table with integrations for over 400 services, complete local deployment options for when you need to keep data in-house, and robust error handling with real-time monitoring that actually helps you debug issues instead of just telling you something broke.

![](https://assets.zilliz.com/n8n2_248855922d.png)

### N8N Has 2000+ Ready-Made Templates

The biggest barrier to adopting new tools isn't learning the syntax—it's figuring out where to start. Here's where I discovered this open-source project '[n8n-workflows](https://github.com/Zie619/n8n-workflows)' that became invaluable. It contains 2,053 ready-to-use workflow templates that you can deploy and customize immediately.


## Getting Started with N8N

Now let’s walk through how to use N8N. It is pretty easy. 


### Environment Setup

I assume most of you have a basic environment setup. If not, check the official resources:

- Docker website: https://www.docker.com/

- Milvus website: https://milvus.io/docs/prerequisite-docker.md

- N8N website: https://n8n.io/

- Python3 website: https://www.python.org/

- N8n-workflows: https://github.com/Zie619/n8n-workflows


### Clone and Run the Template Browser

```
git clone https://github.com/Zie619/n8n-workflows.git
pip install -r requirements.txt
python run.py
http://localhost:8000
```

![](https://assets.zilliz.com/n8n3_0db8e22872.png)


![](https://assets.zilliz.com/n8n4_b6b9ba6635.png)

### Deploy N8N

```
docker run -d -it --rm --name n8n -p 5678:5678 -v n8n_data:/home/node/.n8n -e N8N_SECURE_COOKIE=false -e N8N_HOST=192.168.4.48 -e N8N_LISTEN_ADDRESS=0.0.0.0  n8nio/n8n:latest
```

**⚠️ Important:** Replace N8N_HOST with your actual IP address

![](https://assets.zilliz.com/n8n5_6384caa548.png)

### Importing Templates 

Once you find a template you want to try, getting it into your N8N instance is straightforward:

![](https://assets.zilliz.com/n8n6_2ea8b14bd9.png)

#### **1. Download the JSON File**

Each template is stored as a JSON file that contains the complete workflow definition.

![](https://assets.zilliz.com/n8n7_d58242d81a.png)

#### **2. Open N8N Editor**

Navigate to Menu → Import Workflow

![](https://assets.zilliz.com/n8n8_9961929091.png)

#### **3. Import the JSON**

Select your downloaded file and click Import

![](https://assets.zilliz.com/n8n9_3882b6ade6.png)

From there, you just need to adjust the parameters to match your specific use case. You'll have a professional-grade automation system running in minutes instead of hours.

With your basic workflow system up and running, you might be wondering how to handle more complex scenarios that involve understanding content rather than just processing structured data. That's where vector databases come into play. 


## Vector Databases: Making Workflows Smart with Memory

Modern workflows need to do more than just shuffle data around. You're dealing with unstructured content—documentation, chat logs, knowledge bases—and you need your automation to actually understand what it's working with, not just match exact keywords.


### Why Your Workflow Needs Vector Search

Traditional workflows are basically pattern matching on steroids. They can find exact matches, but they can't understand context or meaning.

When someone asks a question, you want to surface all the relevant information, not just documents that happen to contain the exact words they used.

This is where[ vector databases](https://zilliz.com/learn/what-is-vector-database) like [**Milvus**](https://milvus.io/) and [Zilliz Cloud](https://zilliz.com/cloud) come in. Milvus gives your workflows the ability to understand semantic similarity, which means they can find related content even when the wording is completely different. 

Here's what Milvus brings to your workflow setup: 

- **Massive scale storage** that can handle billions of vectors for enterprise knowledge bases

- **Millisecond-level search performance** that won't slow down your automation

- **Elastic scaling** that grows with your data without requiring a complete rebuild

The combination transforms your workflows from simple data processing into intelligent knowledge services that can actually solve real problems in information management and retrieval.


## What This Actually Means for Your Development Work

Workflow automation isn't rocket science—it's about making complex processes simple and repetitive tasks automatic. The value is in the time you get back and the errors you avoid.

Compared to enterprise solutions that cost tens of thousands of dollars, open-source N8N offers a practical path forward. The open-source version is free, and the drag-and-drop interface means you don't need to write code to build sophisticated automation.

Together with Milvus for intelligent search capabilities, workflow automation tools like N8N upgrade your workflows from simple data processing to smart knowledge services that solve real problems in information management and retrieval.

The next time you find yourself doing the same task for the third time this week, remember: there's probably a template for that. Start small, automate one process, and watch as your productivity multiplies while your frustration disappears.


Boost Efficiency with N8N Workflow Automation


I Discovered This N8N Repo That Actually 10x'd My Workflow Automation Efficiency





Your IDE is bloated. Your coding assistant is outdated. And you're still stuck right-clicking to refactor? Welcome to the CLI renaissance.

AI code assistants are evolving from gimmicks to go-to tools and developers are taking sides. Beyond the startup sensation Cursor, **Anthropic’s** [**Claude Code**](https://www.anthropic.com/claude-code) brings precision and polish. Google’s [**Gemini CLI**](https://github.com/google-gemini/gemini-cli)? Fast, free, and hungry for context. Both promise to make natural language the new shell scripting. So which one should _you_ trust to refactor your next repo?

From what I’ve seen, Claude Code had the early lead. But the game changed fast. After Gemini CLI launched, developers flocked to it—**racking up 15.1k GitHub stars within 24 hours.** As of now, it’s soared past **55,000 stars** and counting. Amazing!

Here's my quick takeaway on why so many developers are excited about Gemini CLI:

- **It's open source under Apache 2.0 and completely free:** Gemini CLI connects to Google's top-tier Gemini 2.0 Flash model at no cost. Simply log in with your personal Google account to access Gemini Code Assist. During the preview period, you get up to 60 requests per minute and 1,000 daily requests—all free of charge.

- **It's a true multi-task powerhouse:** Beyond programming (its strongest suit), it handles file management, content generation, script control, and even Deep Research capabilities.

- **It's lightweight:** You can seamlessly embed it in terminal scripts or use it as a standalone agent.

- **It offers long context length:** With 1 million tokens of context (roughly 750,000 words), it can ingest entire codebases for smaller projects in a single pass.


## Why Developers Are Ditching IDEs for AI-Powered Terminals

Why is there such enthusiasm for these terminal-based tools? As developers, you've probably felt this pain: Traditional IDEs pack impressive features, but they come with workflow complexity that kills momentum. Want to refactor a single function? You need to select the code, right-click for the context menu, navigate to "Refactor," choose the specific refactoring type, configure options in a dialog box, and finally apply changes.

**Terminal AI tools have changed this workflow by streamlining all operations into natural language commands.** Instead of memorizing command syntax, you simply say: "_Help me refactor this function to improve readability,_" and watch as the tool handles the entire process.

This isn't just convenience—it's a fundamental shift in how we think. Complex technical operations become natural language conversations, freeing us to focus on business logic rather than the mechanics of tools.


## Claude Code or Gemini CLI? Choose Your Co-Pilot Wisely

Since Claude Code is also quite popular and easy to use and has previously dominated adoption, how does it compare to the new Gemini CLI? How should we choose between the two? Let's take a closer look at these AI coding tools.


### **1. Cost: Free vs Paid**

- **Gemini CLI** is completely free with any Google account, providing 1,000 requests per day and 60 requests per minute, with no billing setup required.

- **Claude Code** requires an active Anthropic subscription and follows a pay-per-use model, but includes enterprise-level security and support that's valuable for commercial projects.


### **2. Context Window: How Much Code Can It See?**

- **Gemini CLI:** 1 million tokens (roughly 750,000 words)

- **Claude Code:** Approximately 200,000 tokens (about 150,000 words)

Larger context windows enable models to reference more input content when generating responses. They also help maintain conversation coherence in multi-turn dialogues, giving the model better memory of your entire conversation.

Essentially, Gemini CLI can analyze your entire small-to-medium project in a single session, making it ideal for comprehending large codebases and cross-file relationships. Claude Code works better when you're focusing on specific files or functions.


### **3. Code Quality vs Speed**

|                           |                |                 |                                      |
| ------------------------- | -------------- | --------------- | ------------------------------------ |
| **Feature**               | **Gemini CLI** | **Claude Code** | **Notes**                            |
| **Coding speed**          | 8.5/10         | 7.2/10          | Gemini generates code faster         |
| **Coding quality**        | 7.8/10         | 9.1/10          | Claude generates higher quality code |
| **Error handling**        | 7.5/10         | 8.8/10          | Claude is better at error handling   |
| **Context understanding** | 9.2/10         | 7.9/10          | Gemini has longer memory             |
| **Multilingual support**  | 8.9/10         | 8.5/10          | Both are excellent                   |

- **Gemini CLI** generates code faster and excels at understanding large contexts, making it great for rapid prototyping.

- **Claude Code** nails precision and error handling, making it better suited for production environments where code quality is critical.


### **4. Platform Support: Where Can You Run It?**

- **Gemini CLI** works equally well across Windows, macOS, and Linux from day one.

- **Claude Code** was optimized for macOS first, and while it runs on other platforms, the best experience is still on Mac.


### **5. Authentication and Access**

**Claude Code** requires an active Anthropic subscription (Pro, Max, Team, or Enterprise) or API access through AWS Bedrock/Vertex AI. This means you need to set up billing before you can start using it.

**Gemini CLI** offers a generous free plan for individual Google account holders, including 1,000 free requests per day and 60 requests per minute to the full-featured Gemini 2.0 Flash model. Users requiring higher limits or specific models can upgrade via API keys.


### **6. Feature Comparison Overview**

|                       |                  |                               |
| --------------------- | ---------------- | ----------------------------- |
| **Feature**           | **Claude Code**  | **Gemini CLI**                |
| Context Window Length | 200K tokens      | 1M tokens                     |
| Multimodal Support    | Limited          | Powerful (images, PDFs, etc.) |
| Code Understanding    | Excellent        | Excellent                     |
| Tool Integration      | Basic            | Rich (MCP Servers)            |
| Security              | Enterprise-grade | Standard                      |
| Free Requests         | Limited          | 60/min, 1000/day              |


## When to Choose Claude Code vs Gemini CLI?

Now that we've compared the key features of both tools, here are my takeaways on when to choose each:

**Choose Gemini CLI if:**

- Cost-effectiveness and rapid experimentation are priorities

- You're working on large projects that need massive context windows

- You love cutting-edge, open-source tools

- Cross-platform compatibility is crucial

- You want powerful multimodal capabilities

**Choose Claude Code if:**

- You need high-quality code generation

- You're building mission-critical commercial applications

- Enterprise-level support is non-negotiable

- Code quality trumps cost considerations

- You're primarily working on macOS


## Claude Code vs. Gemini CLI: Setup and Best Practices

Now that we have a basic understanding of the capabilities of these two terminal AI tools, let's take a closer look at how to get started with them and the best practices.


### Claude Code Setup and Best Practices

**Installation:** Claude Code requires npm and Node.js version 18 or higher.

```
# Install Claude Code on your system
npm install -g @anthropic-ai/claude-code

# Set up API key
claude config set api-key YOUR_API_KEY

# Verify installation was successful
claude --version

# Launch Claude Code
Claude
```


****![](https://assets.zilliz.com/1_c413bbf950.png)****

**Best Practices for Claude Code:**

1. **Start with architecture understanding:** When approaching a new project, have Claude Code help you understand the overall structure first using natural language.

```
# Let Claude analyze project architecture
> Analyze the main architectural components of this project

# Understand security mechanisms
> What security measures does this system have?

# Get code overview
> Give me an overview of this codebase
```

2. **Be specific and provide context:** The more context you give, the more accurate Claude Code's suggestions will be.

```
# Implement specific features
> Implement an initial version for GitHub issue #123

# Code migration
> Help me migrate this codebase to the latest Java version, first create a plan

# Code refactoring
> Refactor this function to make it more readable and maintainable
```

3. **Use it for debugging and optimization:**

```
# Error analysis
> What caused this error? How can we fix it?

# Performance optimization
> Analyze the performance bottlenecks in this code

# Code review
> Review this pull request and point out potential issues
```

**Summary:**

- Use progressive learning by starting with simple code explanations, then gradually moving to more complex code generation tasks

- Maintain conversation context since Claude Code remembers earlier discussions

- Provide feedback using the `bug` command to report issues and help improve the tool

- Stay security-conscious by reviewing data collection policies and exercising caution with sensitive code


### Gemini CLI Setup and Best Practices

**Installation:** Like Claude Code, Gemini CLI requires npm and Node.js version 18 or higher.

```
# Install Gemini CLI
npm install -g @google/gemini-cli

# Login to your Google account
gemini auth login

# Verify installation
gemini --version

# Launch Gemini CLI
Gemini
```


![](https://assets.zilliz.com/2_bec1984bb0.png)

If you have a personal account, log in with your Google account for immediate access, with a limit of 60 requests per minute. For higher limits, configure your API key:

```
export GEMINI_API_KEY="YOUR_API_KEY"
```


**Best Practices for Gemini CLI:**

1. **Start with architecture understanding:** Like Claude Code, when approaching a new project, have Gemini CLI help you understand the overall structure first using natural language. Note that Gemini CLI supports a 1 million token context window, making it highly effective for large-scale codebase analysis.

```
# Analyze project architecture
> Analyze the main architectural components of this project

# Understand security mechanisms
> What security measures does this system have?

# Get code overview
> Give me an overview of this codebase
```

2. **Leverage its multimodal capabilities:** This is where Gemini CLI truly shines.

```
# Generate app from PDF
> Create a new app based on this PDF design document

# Generate code from sketch
> Generate frontend code based on this UI sketch

# Image processing tasks
> Convert all images in this directory to PNG format and rename using EXIF data
```

3. **Explore tool integrations:** Gemini CLI can integrate with multiple tools and MCP servers for enhanced functionality.

```
# Connect external tools
> Use MCP server to connect my local system tools

# Media generation
> Use Imagen to generate project logo

# Search integration
> Use Google search tool to find related technical documentation
```

**Summary:**

- Be project-oriented: Always launch Gemini from your project directory for better contextual understanding

- Maximize multimodal features by using images, documents, and other media as inputs, not just text

- Explore tool integrations by connecting external tools with MCP servers

- Enhance search capabilities by using built-in Google search for up-to-date information


## AI Code is Outdated on Arrival. Here’s How to Fix it with Milvus

_AI coding tools like Claude Code and Gemini CLI are powerful—but they have a blind spot:_ **_they don’t know what’s current_**_._

_The reality? Most models generate outdated patterns straight out of the box. They were trained months ago, sometimes years. So while they can generate code quickly, they can’t guarantee that it reflects_ **_your latest APIs_**_, frameworks, or SDK versions._

**Real example:**

Ask Cursor how to connect to Milvus, and you might get this:

```
connections.connect("default", host="localhost", port="19530")
```

Looks fine, except that method’s now deprecated. The recommended approach is to use  `MilvusClient` but most assistants don’t know that yet. 

Or take OpenAI’s own API. Many tools still suggest `gpt-3.5-turbo` via `openai.ChatCompletion`, a method deprecated in March 2024. It’s slower, costs more, and delivers worse results. But the LLM doesn’t know that. 

![](https://assets.zilliz.com/3_8f0d1a42b6.png)

![](https://assets.zilliz.com/3_5_3f4c4a0d4c.png)


### The Fix: Real-Time Intelligence with Milvus MCP + RAG

To solve this, we combined two powerful ideas:

- **Model Context Protocol (MCP)**: A standard for agentic tools to interact with live systems through natural language

- **Retrieval-Augmented Generation (RAG)**: Fetches the freshest, most relevant content—on demand

Together, they make your assistant smarter and current.

![](https://assets.zilliz.com/4_e6bc6cacd6.png)

**Here’s how it works:**

1. Preprocess your documentation, SDK references, and API guides

2. Store them as vector embeddings in [**Milvus**](https://milvus.io/), our open-source vector database

3. When a dev asks a question (e.g. “How do I connect to Milvus?”), the system:

   - Runs a **semantic search**

   - Retrieves the most relevant docs and examples
   
   - Injects them into the assistant’s prompt context



4. Result: code suggestions that reflect **exactly what’s true right now**


### Live Code, Live Docs

With the **Milvus MCP Server**, you can plug this flow directly into your coding environment. Assistants get smarter. Code gets better. Devs stay in flow.

And it’s not just theoretical—we’ve battle-tested this against other setups like Cursor’s Agent Mode, Context7, and DeepWiki. The difference? Milvus + MCP doesn't just summarize your project—it stays in sync with it.

See it in Action: [Why Your Vibe Coding Generates Outdated Code and How to Fix It with Milvus MCP ](https://milvus.io/blog/why-vibe-coding-generate-outdated-code-and-how-to-fix-it-with-milvus-mcp.md)


## The Future of Coding is Conversational—And It's Happening Right Now

The terminal AI revolution is just beginning. As these tools mature, we'll likely see even tighter integration with development workflows, better code quality, and solutions to the currency problem through approaches like MCP+RAG.

Whether you choose Claude Code for its quality or Gemini CLI for its accessibility and power, one thing is clear: **natural language programming is here to stay.** The question isn't whether to adopt these tools, but how to integrate them effectively into your development workflow.

We're witnessing a fundamental shift from memorizing syntax to having conversations with our code. **The future of coding is conversational—and it's happening right now in your terminal.**


## Keep Reading

- [Building a Production-Ready AI Assistant with Spring Boot and Milvus](https://milvus.io/blog/building-a-production-ready-ai-assistant-with-spring-boot-and-milvus.md)

- [Zilliz MCP Server: Natural Language Access to Vector Databases - Zilliz blog](https://zilliz.com/blog/introducing-zilliz-mcp-server)

- [VDBBench 1.0: Real-World Benchmarking for Vector Databases - Milvus Blog](https://milvus.io/blog/vdbbench-1-0-benchmarking-with-your-real-world-production-workloads.md)

- [Why Your Vibe Coding Generates Outdated Code and How to Fix It with Milvus MCP](https://milvus.io/blog/why-vibe-coding-generate-outdated-code-and-how-to-fix-it-with-milvus-mcp.md)

- [Why AI Databases Don't Need SQL ](https://milvus.io/blog/why-ai-databases-do-not-need-sql.md)


Claude Code vs Gemini CLI: Who’s the Real Dev Co-Pilot?


Claude Code vs Gemini CLI: Which One’s the Real Dev Co-Pilot?





Most vector database benchmarks test with static data and pre-built indexes. But production systems don't work that way—data flows continuously while users run queries, filters fragment indexes, and performance characteristics shift dramatically under concurrent read/write loads.

Today we're releasing [**VDBBench 1.0**](https://github.com/zilliztech/VectorDBBench/releases/tag/v1.0.0), an open-source benchmark designed from the ground up to test vector databases under realistic production conditions: streaming data ingestion, metadata filtering with varying selectivity, and concurrent workloads that reveal actual system bottlenecks.

[**Download VDBBench 1.0 →**](https://github.com/zilliztech/VectorDBBench/releases/tag/v1.0.0) |[ **View Leaderboard →**](https://zilliz.com/vdbbench-leaderboard?dataset=vectorSearch)


## Why Current Benchmarks Are Misleading 

Let's be honest—there's a strange phenomenon in our industry. Everyone talks about "not gaming benchmarks," yet many participate in exactly that behavior. Since the vector database market exploded in 2023, we've seen numerous examples of systems that "benchmark beautifully" but "fail miserably" in production, wasting engineering time and damaging project credibility.

We've witnessed this disconnect firsthand. For example, Elasticsearch boasts millisecond-level query speeds, but behind the scenes, it can take over 20 hours just to optimize its index. What production system can tolerate such downtime?

The problem stems from three fundamental flaws:

- **Outdated datasets:** Many benchmarks still rely on legacy datasets like SIFT (128 dimensions) while modern embeddings range from 768-3,072 dimensions. The performance characteristics of systems operating on 128D vs. 1024D+ vectors are fundamentally different—memory access patterns, index efficiency, and computational complexity all change dramatically.

- **Vanity metrics:** Benchmarks focus on average latency or peak QPS, creating a distorted picture. A system with 10ms average latency but 2-second P99 latency creates a terrible user experience. Peak throughput measured over 30 seconds tells you nothing about sustained performance.

- **Oversimplified scenarios:** Most benchmarks test basic "write data, build index, query" workflows—essentially "Hello World" level testing. Real production involves continuous data ingestion while serving queries, complex metadata filtering that fragments indexes, and concurrent read/write operations competing for resources.


## What’s New in VDBBench 1.0?

VDBBench doesn't just iterate on outdated benchmarking philosophies—it rebuilds the concept from first principles with one guiding belief: a benchmark is only valuable if it predicts actual production behavior.

We've engineered VDBBench to faithfully replicate real-world conditions across three critical dimensions: **data authenticity, workload patterns, and performance measurement methodologies.**

Let’s take a closer look at what new features are brought to the table. 


### **🚀 Redesigned Dashboard with Production-Relevant Visualizations**

Most benchmarks focus only on raw data output, but what matters is how engineers interpret and act on those results. We redesigned the UI to prioritize clarity and interactivity—enabling you to spot performance gaps between systems and make fast infrastructure decisions.

The new dashboard visualizes not just performance numbers, but the relationships between them: how QPS degrades under different filter selectivity levels, how recall fluctuates during streaming ingestion, and how latency distributions reveal system stability characteristics.
![](https://assets.zilliz.com/vdb_1_df593dea0b.png)

We've retested major vector database platforms including **Milvus, Zilliz Cloud, Elastic Cloud, Qdrant Cloud, Pinecone, and OpenSearch** with their latest configurations and recommended settings, ensuring all benchmark data reflects current capabilities. All testing results are available at the[ VDBBench Leaderboard](https://zilliz.com/vdbbench-leaderboard?dataset=vectorSearch).


### 🏷️ Tag Filtering: The Hidden Performance Killer

Real-world queries rarely happen in isolation. Applications combine vector similarity with metadata filtering ("find shoes that look like this photo but cost under $100"). This filtered vector search creates unique challenges that most benchmarks completely ignore.

Filtered searches introduce complexity in two critical areas:

- **Filter Complexity**: More scalar fields and complex logical conditions increase computational demands and can cause insufficient recall and graph index fragmentation.

- **Filter Selectivity**: This is the "hidden performance killer" we've repeatedly verified in production. When filtering conditions become highly selective (filtering out 99%+ of data), query speeds can fluctuate by orders of magnitude, and recall can become unstable as index structures struggle with sparse result sets.

VDBBench systematically tests various filtering selectivity levels (from 50% to 99.9%), providing a comprehensive performance profile under this critical production pattern. The results often reveal dramatic performance cliffs that would never show up in traditional benchmarks.

**Example**: In Cohere 1M tests, Milvus maintained consistently high recall across all filter selectivity levels, while OpenSearch exhibited unstable performance with recall fluctuating significantly under different filtering conditions—falling below 0.8 recall in many cases, which is unacceptable for most production environments.

![](https://assets.zilliz.com/vdb_2_0ef89463e5.png)

_Figure: QPS and Recall of Milvus and OpenSearch Across Different Filter Selectivity Levels (Cohere 1M Test)._ 


### 🌊 Streaming Read/Write: Beyond Static Index Testing

Production systems rarely enjoy the luxury of static data. New information continuously flows in while searches execute—a scenario where many otherwise impressive databases collapse under the dual pressure of maintaining search performance while handling continuous writes.

VDBBench's streaming scenarios simulate real parallel operations, helping developers understand system stability in high-concurrency environments, particularly how data writing impacts query performance and how performance evolves as data volume increases.

To ensure fair comparisons across different systems, VDBBench uses a structured approach:

- Configure controlled write rates that mirror target production workloads (e.g., 500 rows/sec distributed across 5 parallel processes)

- Trigger search operations after every 10% of data ingestion, alternating between serial and concurrent modes

- Record comprehensive metrics: latency distributions (including P99), sustained QPS, and recall accuracy

- Track performance evolution over time as data volume and system stress increase

This controlled, incremental load testing reveals how well systems maintain stability and accuracy under ongoing ingestion—something traditional benchmarks rarely capture.

**Example**: In Cohere 10M streaming tests, Pinecone maintained higher QPS and recall throughout the write cycle compared to Elasticsearch. Notably, Pinecone's performance significantly improved after ingestion completion, demonstrating strong stability under sustained load, while Elasticsearch showed more erratic behavior during active ingestion phases.
![](https://assets.zilliz.com/vdb3_9d2a5298b0.png)

Figure: QPS and Recall of Pinecone vs. Elasticsearch in the Cohere 10M Streaming Test (500 rows/s Ingestion Rate). 

VDBBench goes even further by supporting an optional optimization step, allowing users to compare streaming search performance before and after index optimization. It also tracks and reports the actual time spent on each stage, offering deeper insights into system efficiency and behavior under production-like conditions.
![](https://assets.zilliz.com/vdb4_0caee3b201.png)

_Figure: QPS and Recall of Pinecone vs. Elasticsearch in the Cohere 10M Streaming Test After Optimization (500 rows/s Ingestion Rate)_

As shown in our tests, Elasticsearch surpassed Pinecone in QPS—after index optimization. But when the x-axis reflects actual elapsed time, it's clear that Elasticsearch took significantly longer to reach that performance. In production, that delay matters. This comparison reveals a key tradeoff: peak throughput vs. time-to-serve.


### 🔬 Modern Datasets That Reflect Current AI Workloads

We've completely overhauled the datasets used for vector database benchmarking. Instead of legacy test sets like SIFT and GloVe, VDBBench uses vectors generated from state-of-the-art embedding models like OpenAI and Cohere that power today's AI applications.

To ensure relevance, especially for use cases like Retrieval-Augmented Generation (RAG), we selected corpora that reflect real-world enterprise and domain-specific scenarios:

|            |                     |                |                 |                              |
| ---------- | ------------------- | -------------- | --------------- | ---------------------------- |
| **Corpus** | **Embedding Model** | **Dimensions** | **Size**        | **Use Case**                 |
| Wikipedia  | Cohere V2           | 768            | 1M / 10M        | General knowledge base       |
| BioASQ     | Cohere V3           | 1024           | 1M / 10M        | Domain-specific (biomedical) |
| C4         | OpenAI              | 1536           | 500K / 5M       | Web-scale text processing    |
| MSMarco V2 | udever-bloom-1b1    | 1536           | 1M / 10M / 138M | Large-scale search           |

These datasets better simulate today's high-volume, high-dimensional vector data, enabling realistic testing of storage efficiency, query performance, and retrieval accuracy under conditions that match modern AI workloads.


### ⚙️ Custom Dataset Support for Industry-Specific Testing

Every business is unique. The financial industry might need testing focused on transaction embeddings, while social platforms care more about user behavior vectors. VDBBench lets you benchmark with your own data generated from your specific embedding models for your specific workloads.

You can customize:

- Vector dimensions and data types

- Metadata schema and filtering patterns

- Data volume and ingestion patterns

- Query distributions that match your production traffic

After all, no dataset tells a better story than your own production data.


## How VDBBench Measures What Actually Matters in Production

### Production-Focused Metric Design

VDBBench prioritizes metrics that reflect real-world performance, not just lab results. We've redesigned benchmarking around what actually matters in production environments: **reliability under load, tail latency characteristics, sustained throughput, and accuracy preservation.**

- **P95/P99 Latency for Real User Experience**: Average/median latency masks the outliers that frustrate real users and can indicate underlying system instability. VDBBench focuses on tail latency like P95/P99, revealing what performance 95% or 99% of your queries will actually achieve. This is crucial for SLA planning and understanding worst-case user experience.

- **Sustainable Throughput Under Load**: A system that performs well for 5 seconds doesn't cut it in production. VDBBench gradually increases concurrency to find your database's maximum sustainable queries per second (`max_qps`)—not the peak number under short, ideal conditions. This methodology reveals how well your system holds up over time and helps with realistic capacity planning.

- **Recall Balanced with Performance**: Speed without accuracy is meaningless. Every performance number in VDBBench is paired with recall measurements, so you know exactly how much relevance you're trading off for throughput. This enables fair, apples-to-apples comparisons between systems with vastly different internal tradeoffs.


### Test Methodology That Reflects Reality

A key innovation in VDBBench's design is the separation of serial and concurrent testing, which helps capture how systems behave under different types of load and reveals performance characteristics that matter for different use cases.

**Latency Measurement Separation:**

- `serial_latency_p99` measures system performance under minimal load, where only one request is processed at a time. This represents the best-case scenario for latency and helps identify baseline system capabilities.

- `conc_latency_p99` captures system behavior under realistic, high-concurrency conditions, where multiple requests arrive simultaneously and compete for system resources.

**Two-Phase Benchmark Structure**:

1. **Serial Test**: Single-process run of 1,000 queries that establishes baseline performance and accuracy, reporting both `serial_latency_p99` and recall. This phase helps identify the theoretical performance ceiling.

2. **Concurrency Test**: Simulates production environment under sustained load with several key innovations:

   - **Realistic client simulation**: Each test process operates independently with its own connection and query set, avoiding shared-state interference that could distort results

   - **Synchronized start**: All processes begin simultaneously, ensuring measured QPS accurately reflects claimed concurrency levels

   - **Independent query sets**: Prevents unrealistic cache hit rates that don't reflect production query diversity

These carefully structured methods ensure that `max_qps` and `conc_latency_p99` values reported by VDBBench are both accurate and production-relevant, providing meaningful insights for production capacity planning and system design.


## Getting Started with VDBBench 1.0

**VDBBench 1.0** represents a fundamental shift toward production-relevant benchmarking. By covering continuous data writing, metadata filtering with varying selectivity, and streaming loads under concurrent access patterns, it provides the closest approximation to actual production environments available today.

The gap between benchmark results and real-world performance shouldn't be a guessing game. If you're planning to deploy a vector database in production, it's worth understanding how it performs beyond idealized lab tests. VDBBench is open-source, transparent, and designed to support meaningful, apples-to-apples comparisons.

Don't be swayed by impressive numbers that don't translate to production value. **Use VDBBench 1.0 to test scenarios that matter to your business, with your data, under conditions that reflect your actual workload.** The era of misleading benchmarks in vector database evaluation is ending—it's time to make decisions based on production-relevant data.

**Try VDBBench with your own workloads:**[ https://github.com/zilliztech/VectorDBBench](https://github.com/zilliztech/VectorDBBench)

**View testing results of major vector databases:**[ VDBBench Leaderboard](https://zilliz.com/vdbbench-leaderboard?dataset=vectorSearch)

Have questions or want to share your results? Join the conversation on[ GitHub](https://github.com/zilliztech/VectorDBBench) or connect with our community on[ Discord](https://discord.com/invite/FG6hMJStWu).


VDBBench 1.0: Real-World Benchmarking for Vector Databases


Announcing VDBBench 1.0: Open-Source Vector Database Benchmarking with Your Real-World Production Workloads



If you've been keeping an eye on the embedding model space, you've probably noticed Alibaba just dropped their [Qwen3 Embedding series](https://qwenlm.github.io/blog/qwen3-embedding/). They released both embedding and reranking models in three sizes each (0.6B, 4B, 8B), all built on the Qwen3 foundation models and designed specifically for retrieval tasks. 

The Qwen3 series has a few features I found interesting:

- **Multilingual embeddings** - they claim a unified semantic space across 100+ languages

- **Instruction prompting** - you can pass custom instructions to modify embedding behavior

- **Variable dimensions** - supports different embedding sizes via Matryoshka Representation Learning

- **32K context length** - can process longer input sequences

- **Standard dual/cross-encoder setup** - embedding model uses dual-encoder, reranker uses cross-encoder

Looking at the benchmarks, Qwen3-Embedding-8B achieved a score of 70.58 on the MTEB multilingual leaderboard, surpassing BGE, E5, and even Google Gemini. The Qwen3-Reranker-8B hit 69.02 on multilingual ranking tasks. These aren't just "pretty good among open-source models" - they're comprehensively matching or even surpassing mainstream commercial APIs. In RAG retrieval, cross-language search, and code search systems, especially in Chinese contexts, these models already have production-ready capabilities.

![](https://lh7-rt.googleusercontent.com/docsz/AD_4nXdZCKoPqf8mpxwQ_s-gGbdHYvw_HhWn6Ib62v8C_VEZF8AOSnY1yLEEv1ztkINpmwgHAVC5kZw6rWplfx5OkISf_gL4VvoqlXxSfs8s_qd8mdBuA0HBhP9kEdipXy0QVuPmEyOJRg?key=nqzZfIwgkzdlEZQ2MYSMGQ)

![](https://lh7-rt.googleusercontent.com/docsz/AD_4nXdNppvBpn_5M9d6WDb0-pCjgTobVc9eFw_m6m6Vg73wJtB9OvcPFw5089FUui_N2-LbJVjJPe1c8_EnYY4F3Ryw0021kvmJ0jU0Q06qG2ZX2D1vywIyd5aKqO_cx-77U_spMVr8cQ?key=nqzZfIwgkzdlEZQ2MYSMGQ)

As someone who's likely dealt with the usual suspects (OpenAI's embeddings, BGE, E5), you might be wondering if these are worth your time. Spoiler: they are. 


## What We're Building

This tutorial walks through building a complete RAG system using Qwen3-Embedding-0.6B and Qwen3-Reranker-0.6B with Milvus. We'll implement a two-stage retrieval pipeline:

1. **Dense retrieval** with Qwen3 embeddings for fast candidate selection

2. **Reranking** with Qwen3 cross-encoder for precision refinement

3. **Generation** with OpenAI's GPT-4 for final responses

By the end, you'll have a working system that handles multilingual queries, uses instruction prompting for domain tuning, and balances speed with accuracy through intelligent reranking.


## Environment Setup

Let's start with the dependencies. Note the minimum version requirements - they're important for compatibility:

```
pip install --upgrade pymilvus openai requests tqdm sentence-transformers transformers
```


_Requires transformers>=4.51.0 and sentence-transformers>=2.7.0_

For this tutorial, we'll use OpenAI as our generation model. Set up your API key:

```
import os

os.environ["OPENAI_API_KEY"] = "sk-***********"
```



## **Data Preparation**

We'll use Milvus documentation as our knowledge base - it's a good mix of technical content that tests both retrieval and generation quality.

Download and extract the documentation:

```
! wget https://github.com/milvus-io/milvus-docs/releases/download/v2.4.6-preview/milvus_docs_2.4.x_en.zip
! unzip -q milvus_docs_2.4.x_en.zip -d milvus_docs
```


Load and chunk the markdown files. We're using a simple header-based splitting strategy here - for production systems, consider more sophisticated chunking approaches:

```
from glob import glob

text_lines = []

for file_path in glob("milvus_docs/en/faq/*.md", recursive=True):
    with open(file_path, "r") as file:
        file_text = file.read()

    text_lines += file_text.split("# ")
```



## **Model Setup**

Now let's initialize our models. We're using the lightweight 0.6B versions, which offer a good balance of performance and resource requirements:

```
from openai import OpenAI
from sentence_transformers import SentenceTransformer
import torch
from transformers import AutoModel, AutoTokenizer, AutoModelForCausalLM

# Initialize OpenAI client for LLM generation
openai_client = OpenAI()

# Load Qwen3-Embedding-0.6B model for text embeddings
embedding_model = SentenceTransformer("Qwen/Qwen3-Embedding-0.6B")

# Load Qwen3-Reranker-0.6B model for reranking
reranker_tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3-Reranker-0.6B", padding_side='left')
reranker_model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-Reranker-0.6B").eval()

# Reranker configuration
token_false_id = reranker_tokenizer.convert_tokens_to_ids("no")
token_true_id = reranker_tokenizer.convert_tokens_to_ids("yes")
max_reranker_length = 8192

prefix = "<|im_start|>system\nJudge whether the Document meets the requirements based on the Query and the Instruct provided. Note that the answer can only be \"yes\" or \"no\".<|im_end|>\n<|im_start|>user\n"
suffix = "<|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\n"
prefix_tokens = reranker_tokenizer.encode(prefix, add_special_tokens=False)
suffix_tokens = reranker_tokenizer.encode(suffix, add_special_tokens=False)
```


The expected output: 

![](https://lh7-rt.googleusercontent.com/docsz/AD_4nXdaUrXQrAs2W8-rGT9njJHEKnQ8YwREmULO6xYJnpPy7bwsmZImDRt_3EMwJuVM3k3zI7pbNvY1fDsqMKYq-rrNArx_gxOA4ZTi0g1tkRIlUqJfx1z2nZ60ATPW0L5t6I_XLTVf?key=nqzZfIwgkzdlEZQ2MYSMGQ)


## Embedding Function

The key insight with Qwen3 embeddings is the ability to use different prompts for queries versus documents. This seemingly small detail can significantly improve retrieval performance:

```
def emb_text(text, is_query=False):
    """
    Generate text embeddings using Qwen3-Embedding-0.6B model.
    
    Args:
        text: Input text to embed
        is_query: Whether this is a query (True) or document (False)
    
    Returns:
        List of embedding values
    """
    if is_query:
        # For queries, use the "query" prompt for better retrieval performance
        embeddings = embedding_model.encode([text], prompt_name="query")
    else:
        # For documents, use default encoding
        embeddings = embedding_model.encode([text])
    
    return embeddings[0].tolist()
```


Let's test the embedding function and check the output dimensions:

```
test_embedding = emb_text("This is a test")
embedding_dim = len(test_embedding)
print(f"Embedding dimension: {embedding_dim}")
print(f"First 10 values: {test_embedding[:10]}")
```


Expected output:

```
Embedding dimension: 1024
First 10 values: [-0.009923271834850311, -0.030248118564486504, -0.011494234204292297, ...]
```



## Reranking Implementation

The reranker uses a cross-encoder architecture to evaluate query-document pairs. This is more computationally expensive than the dual-encoder embedding model, but provides much more nuanced relevance scoring.

Here's the complete reranking pipeline:

```
def format_instruction(instruction, query, doc):
    """Format instruction for reranker input"""
    if instruction is None:
        instruction = 'Given a web search query, retrieve relevant passages that answer the query'
    output = "<Instruct>: {instruction}\n<Query>: {query}\n<Document>: {doc}".format(
        instruction=instruction, query=query, doc=doc
    )
    return output

def process_inputs(pairs):
    """Process inputs for reranker"""
    inputs = reranker_tokenizer(
        pairs, padding=False, truncation='longest_first',
        return_attention_mask=False, max_length=max_reranker_length - len(prefix_tokens) - len(suffix_tokens)
    )
    for i, ele in enumerate(inputs['input_ids']):
        inputs['input_ids'][i] = prefix_tokens + ele + suffix_tokens
    inputs = reranker_tokenizer.pad(inputs, padding=True, return_tensors="pt", max_length=max_reranker_length)
    for key in inputs:
        inputs[key] = inputs[key].to(reranker_model.device)
    return inputs

@torch.no_grad()
def compute_logits(inputs, **kwargs):
    """Compute relevance scores using reranker"""
    batch_scores = reranker_model(**inputs).logits[:, -1, :]
    true_vector = batch_scores[:, token_true_id]
    false_vector = batch_scores[:, token_false_id]
    batch_scores = torch.stack([false_vector, true_vector], dim=1)
    batch_scores = torch.nn.functional.log_softmax(batch_scores, dim=1)
    scores = batch_scores[:, 1].exp().tolist()
    return scores

def rerank_documents(query, documents, task_instruction=None):
    """
    Rerank documents based on query relevance using Qwen3-Reranker
    
    Args:
        query: Search query
        documents: List of documents to rerank
        task_instruction: Task instruction for reranking
    
    Returns:
        List of (document, score) tuples sorted by relevance score
    """
    if task_instruction is None:
        task_instruction = 'Given a web search query, retrieve relevant passages that answer the query'
    
    # Format inputs for reranker
    pairs = [format_instruction(task_instruction, query, doc) for doc in documents]
    
    # Process inputs and compute scores
    inputs = process_inputs(pairs)
    scores = compute_logits(inputs)
    
    # Combine documents with scores and sort by score (descending)
    doc_scores = list(zip(documents, scores))
    doc_scores.sort(key=lambda x: x[1], reverse=True)
    
    return doc_scores
```




## Setting Up Milvus Vector Database

Now let's set up our vector database. We're using Milvus Lite for simplicity, but the same code works with full Milvus deployments:

```
from pymilvus import MilvusClient

milvus_client = MilvusClient(uri="./milvus_demo.db")

collection_name = "my_rag_collection"
```


**Deployment Options:**

- **Local file** (like `./milvus.db`): Uses Milvus Lite, perfect for development

- **Docker/Kubernetes**: Use server URI like `http://localhost:19530` for production

- **Zilliz Cloud**: Use cloud endpoint and API key for managed service

Clean up any existing collection and create a new one:

```
# Remove existing collection if it exists
if milvus_client.has_collection(collection_name):
    milvus_client.drop_collection(collection_name)

# Create new collection with our embedding dimensions
milvus_client.create_collection(
    collection_name=collection_name,
    dimension=embedding_dim,  # 1024 for Qwen3-Embedding-0.6B
    metric_type="IP",  # Inner product for similarity
    consistency_level="Strong",  # Ensure data consistency
)
```



## Loading Data into Milvus

Now let's process our documents and insert them into the vector database:

```
from tqdm import tqdm

data = []

for i, line in enumerate(tqdm(text_lines, desc="Creating embeddings")):
    data.append({"id": i, "vector": emb_text(line), "text": line})

milvus_client.insert(collection_name=collection_name, data=data)
```


Expected output:

```
Creating embeddings: 100%|████████████| 72/72 [00:08<00:00, 8.68it/s]
Inserted 72 documents
```



## Enhancing RAG with Reranking Technology

Now comes the exciting part - putting it all together into a complete retrieval-augmented generation system.


### **Step 1: Query and Initial Retrieval**

Let's test with a common question about Milvus:

```
question = "How is data stored in milvus?"

# Perform initial dense retrieval to get top candidates
search_res = milvus_client.search(
    collection_name=collection_name,
    data=[emb_text(question, is_query=True)],  # Use query prompt
    limit=10,  # Get top 10 candidates for reranking
    search_params={"metric_type": "IP", "params": {}},
    output_fields=["text"],  # Return the actual text content
)

print(f"Found {len(search_res[0])} initial candidates")
```



### **Step 2: Reranking for Precision**

Extract candidate documents and apply reranking:

```
# Extract candidate documents
candidate_docs = [res["entity"]["text"] for res in search_res[0]]

# Rerank using Qwen3-Reranker
print("Reranking documents...")
reranked_docs = rerank_documents(question, candidate_docs)

# Select top 3 after reranking
top_reranked_docs = reranked_docs[:3]
print(f"Selected top {len(top_reranked_docs)} documents after reranking")
```




### **Step 3: Compare Results**

Let's examine how reranking changes the results:

```
Reranked results (top 3):
[
    [
        " Where does Milvus store data?\n\nMilvus deals with two types of data, inserted data and metadata. \n\nInserted data, including vector data, scalar data, and collection-specific schema, are stored in persistent storage as incremental log. Milvus supports multiple object storage backends, including [MinIO](https://min.io/), [AWS S3](https://aws.amazon.com/s3/?nc1=h_ls), [Google Cloud Storage](https://cloud.google.com/storage?hl=en#object-storage-for-companies-of-all-sizes) (GCS), [Azure Blob Storage](https://azure.microsoft.com/en-us/products/storage/blobs), [Alibaba Cloud OSS](https://www.alibabacloud.com/product/object-storage-service), and [Tencent Cloud Object Storage](https://www.tencentcloud.com/products/cos) (COS).\n\nMetadata are generated within Milvus. Each Milvus module has its own metadata that are stored in etcd.\n\n###",
        0.9997891783714294
    ],
    [
        "How does Milvus flush data?\n\nMilvus returns success when inserted data are loaded to the message queue. However, the data are not yet flushed to the disk. Then Milvus' data node writes the data in the message queue to persistent storage as incremental logs. If `flush()` is called, the data node is forced to write all data in the message queue to persistent storage immediately.\n\n###",
        0.9989748001098633
    ],
    [
        "Does the query perform in memory? What are incremental data and historical data?\n\nYes. When a query request comes, Milvus searches both incremental data and historical data by loading them into memory. Incremental data are in the growing segments, which are buffered in memory before they reach the threshold to be persisted in storage engine, while historical data are from the sealed segments that are stored in the object storage. Incremental data and historical data together constitute the whole dataset to search.\n\n###",
        0.9984032511711121
    ]
]

================================================================================
Original embedding-based results (top 3):
[
    [
        " Where does Milvus store data?\n\nMilvus deals with two types of data, inserted data and metadata. \n\nInserted data, including vector data, scalar data, and collection-specific schema, are stored in persistent storage as incremental log. Milvus supports multiple object storage backends, including [MinIO](https://min.io/), [AWS S3](https://aws.amazon.com/s3/?nc1=h_ls), [Google Cloud Storage](https://cloud.google.com/storage?hl=en#object-storage-for-companies-of-all-sizes) (GCS), [Azure Blob Storage](https://azure.microsoft.com/en-us/products/storage/blobs), [Alibaba Cloud OSS](https://www.alibabacloud.com/product/object-storage-service), and [Tencent Cloud Object Storage](https://www.tencentcloud.com/products/cos) (COS).\n\nMetadata are generated within Milvus. Each Milvus module has its own metadata that are stored in etcd.\n\n###",
        0.8306853175163269
    ],
    [
        "How does Milvus flush data?\n\nMilvus returns success when inserted data are loaded to the message queue. However, the data are not yet flushed to the disk. Then Milvus' data node writes the data in the message queue to persistent storage as incremental logs. If `flush()` is called, the data node is forced to write all data in the message queue to persistent storage immediately.\n\n###",
        0.7302717566490173
    ],
    [
        "How does Milvus handle vector data types and precision?\n\nMilvus supports Binary, Float32, Float16, and BFloat16 vector types.\n\n- Binary vectors: Store binary data as sequences of 0s and 1s, used in image processing and information retrieval.\n- Float32 vectors: Default storage with a precision of about 7 decimal digits. Even Float64 values are stored with Float32 precision, leading to potential precision loss upon retrieval.\n- Float16 and BFloat16 vectors: Offer reduced precision and memory usage. Float16 is suitable for applications with limited bandwidth and storage, while BFloat16 balances range and efficiency, commonly used in deep learning to reduce computational requirements without significantly impacting accuracy.\n\n###",
        0.7003671526908875
    ]
]
```


The reranking typically shows much higher discriminative scores (closer to 1.0 for relevant documents) compared to embedding similarity scores.


### **Step 4: Generate Final Response**

Now let's use the retrieved context to generate a comprehensive answer:

First: Convert the retrieved documents to string format.

```
context = "\n".join(
    [line_with_distance[0] for line_with_distance in retrieved_lines_with_distances]
)
```


Provide system prompt and user prompt for the large language model. This prompt is generated from documents retrieved from Milvus.

```
SYSTEM_PROMPT = """
Human: You are an AI assistant. You are able to find answers to the questions from the contextual passage snippets provided.
"""
USER_PROMPT = f"""
Use the following pieces of information enclosed in <context> tags to provide an answer to the question enclosed in <question> tags.
<context>
{context}
</context>
<question>
{question}
</question>
"""
```


Use GPT-4o to generate a response based on the prompts.

```
response = openai_client.chat.completions.create(
    model="gpt-4o",
    messages=[
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": USER_PROMPT},
    ],
)
print(response.choices[0].message.content)
```


Expected output:

```
In Milvus, data is stored in two main forms: inserted data and metadata. 
Inserted data, which includes vector data, scalar data, and collection-specific 
schema, is stored in persistent storage as incremental logs. Milvus supports 
multiple object storage backends for this purpose, including MinIO, AWS S3, 
Google Cloud Storage, Azure Blob Storage, Alibaba Cloud OSS, and Tencent 
Cloud Object Storage. Metadata for Milvus is generated by its various modules 
and stored in etcd.
```



## **Wrapping Up**

This tutorial demonstrated a complete RAG implementation using Qwen3's embedding and reranking models. The key takeaways:

1. **Two-stage retrieval** (dense + reranking) consistently improves accuracy over embedding-only approaches

2. **Instruction prompting** allows domain-specific tuning without retraining

3. **Multilingual capabilities** work naturally without additional complexity

4. **Local deployment** is feasible with the 0.6B models

The Qwen3 series offers solid performance in a lightweight, open-source package. While not revolutionary, they provide incremental improvements and useful features like instruction prompting that can make a real difference in production systems.

Test these models against your specific data and use cases - what works best always depends on your content, query patterns, and performance requirements.


Hands-on RAG with Qwen3 Embedding and Reranking Models using Milvus


For the past few years, we've been focused on one thing: building an enterprise-ready vector database for the AI era. The hard part isn't building _a_ database—it's building one that's scalable, easy to use, and actually solves real problems in production.

This June, we reached a new milestone: Milvus hit [35,000 stars on GitHub](https://github.com/milvus-io/milvus) (now it has 35.5K+ stars at the time of writing). We're not going to pretend this is just another number—it means a lot to us.

Each star represents a developer who took the time to look at what we've built, found it useful enough to bookmark, and in many cases, decided to use it. Some of you have gone further: filing issues, contributing code, answering questions in our forums, and helping other developers when they get stuck.

We wanted to take a moment to share our story—the real one, with all the messy parts included.


## We Started Building Milvus Because Nothing Else Worked

Back in 2017, we started with a simple question: As AI applications were starting to emerge and unstructured data was exploding, how do you efficiently store and search the vector embeddings that power semantic understanding?

Traditional databases weren't built for this. They're optimized for rows and columns, not high-dimensional vectors. The existing technologies and tools were either impossible or painfully slow for what we needed.

We tried everything available. Hacked together solutions with Elasticsearch. Built custom indexes on top of MySQL. Even experimented with FAISS, but it was designed as a research library, not a production database infrastructure. Nothing provided the complete solution we envisioned for enterprise AI workloads.

**So we started building our own.** Not because we thought it would be easy—databases are notoriously hard to get right—but because we could see where AI was heading and knew it needed purpose-built infrastructure to get there.

By 2018, we were deep into developing what would become [Milvus](https://milvus.io/). The term "**vector database**" didn't even exist yet. We were essentially creating a new category of infrastructure software, which was both exciting and terrifying.


## Open-Sourcing Milvus: Building in Public

In November 2019, we decided to open-source Milvus version 0.10.

Open-sourcing means exposing all your flaws to the world. Every hack, every TODO comment, every design decision you're not entirely sure about. But we believed that if vector databases were going to become critical infrastructure for AI, they needed to be open and accessible to everyone.

The response was overwhelming. Developers didn't just use Milvus—they improved it. They found bugs we'd missed, suggested features we hadn't considered, and asked questions that made us think harder about our design choices.

In 2020, we joined the [LF AI & Data Foundation](https://lfaidata.foundation/). This wasn't just for credibility—it taught us how to maintain a sustainable open-source project. How to handle governance, backward compatibility, and building software that lasts years, not months.

By 2021, we released Milvus 1.0 and [graduated from LF AI & Data Foundation](https://lfaidata.foundation/projects/milvus/). That same year, we won the [BigANN global challenge](https://big-ann-benchmarks.com/neurips21.html) for billion-scale vector search. That win felt good, but more importantly, it validated that we were solving real problems the right way.


## The Hardest Decision: Starting Over

Here's where things get complicated. By 2021, Milvus 1.0 was working well for many use cases, but enterprise customers kept asking for the same things: better cloud-native architecture, easier horizontal scaling, more operational simplicity.

We had a choice: patch our way forward or rebuild from the ground up. We chose to rebuild.

Milvus 2.0 was essentially a complete rewrite. We introduced a fully decoupled storage-compute architecture with dynamic scalability. It took us two years and was honestly one of the most stressful periods in our company's history. We were throwing away a working system that thousands of people were using to build something unproven.

**But when we released Milvus 2.0 in 2022, it transformed Milvus from a powerful vector database into production-ready infrastructure that could scale to enterprise workloads.** That same year, we also completed a [Series B+ funding round](https://zilliz.com/news/vector-database-company-zilliz-series-b-extension)—not to burn money, but to double down on product quality and support for global customers. We knew this path would take time, but every step had to be built on a solid foundation.


## When Everything Accelerated with AI

2023 was the year of [RAG](https://zilliz.com/learn/Retrieval-Augmented-Generation) (retrieval-augmented generation). Suddenly, semantic search went from an interesting AI technique to essential infrastructure for chatbots, document Q&A systems, and AI agents.

The GitHub stars of Milvus spiked. Support requests multiplied. Developers who had never heard of vector databases were suddenly asking sophisticated questions about indexing strategies and query optimization.

This growth was exciting but also overwhelming. We realized we needed to scale not just our technology, but our entire approach to community support. We hired more developer advocates, completely rewrote our documentation, and started creating educational content for developers new to vector databases.

We also launched [Zilliz Cloud](https://zilliz.com/cloud)—our fully managed version of Milvus. Some people asked why we were "commercializing" our open-source project. The honest answer is that maintaining enterprise-grade infrastructure is expensive and complex. Zilliz Cloud allows us to sustain and accelerate Milvus development while keeping the core project completely open source.

Then came 2024. [**Forrester named us a leader**](https://zilliz.com/blog/zilliz-named-a-leader-in-the-forrester-wave-vector-database-report) **in the vector database category.** Milvus passed 30,000 GitHub stars. **And we realized: the road we'd been paving for seven years had finally become the highway.** As more enterprises adopted vector databases as critical infrastructure, our business growth accelerated rapidly—validating that the foundation we'd built could scale both technically and commercially.


## The Team Behind Milvus: Zilliz

Here's something interesting: many people know Milvus but not Zilliz. We're actually fine with that. [**Zilliz**](https://zilliz.com/) **is the team behind Milvus—we build it, maintain it, and support it.**

What we care about most are the unglamorous things that make the difference between a cool demo and production-ready infrastructure: performance optimizations, security patches, documentation that actually helps beginners, and responding thoughtfully to GitHub issues.

We've built a 24/7 global support team across the U.S., Europe, and Asia, because developers need help in their time zones, not ours. We have community contributors we call "[Milvus Ambassadors](https://docs.google.com/forms/d/e/1FAIpQLSfkVTYObayOaND8M1ci9eF_YWvoKDb-xQjLJYZ-LhbCdLAt2Q/viewform)" who organize events, answer forum questions, and often explain concepts better than we do.

We've also welcomed integrations with AWS, GCP, and other cloud providers—even when they offer their own managed versions of Milvus. More deployment options are good for users. Though we've noticed that when teams hit complex technical challenges, they often end up reaching out to us directly because we understand the system at the deepest level.

Many people think open source is just a "toolbox," but it's actually an "evolutionary process"—a collective effort by countless people who love and believe in it. Only those who truly understand the architecture can provide the "why" behind bug fixes, performance bottleneck analysis, data system integration, and architectural adjustments.

**So if you're using open-source Milvus, or considering vector databases as a core component of your AI system, we encourage you to reach out to us directly for the most professional and timely support.**


## Real Impact in Production: The Trust from Users

The use cases for Milvus have grown beyond what we initially imagined. We're powering AI infrastructure for some of the world's most demanding enterprises across every industry.

![zilliz customers.png](https://assets.zilliz.com/zilliz_customers_e7340d5dd4.png)


[**Bosch**](https://zilliz.com/customers/bosch), the global automotive technology leader and pioneer in autonomous driving, revolutionized their data analysis with Milvus achieving 80% reduction in data collection costs and $1.4M annual savings while searching billions of driving scenarios in milliseconds for critical edge cases.

[**Read AI**](https://zilliz.com/customers/read-ai), one of the fastest-growing productivity AI companies serving millions of monthly active users, uses Milvus to achieve sub-20-50ms retrieval latency across billions of records and 5× speedup in agentic search. Their CTO says, "Milvus serves as the central repository and powers our information retrieval among billions of records."

[**A global fintech leader**](https://zilliz.com/customers/global-fintech-leader), one of the world's largest digital payment platforms processing tens of billions of transactions across 200+ countries and 25+ currencies, chose Milvus for 5-10× faster batch ingestion than competitors, completing jobs in under 1 hour that took others 8+ hours.

[**Filevine**](https://zilliz.com/customers/filevine), the leading legal work platform trusted by thousands of law firms across the United States, manages 3 billion vectors across millions of legal documents, saving attorneys 60-80% of time in document analysis and achieving "true consciousness of data" for legal case management.

We're also supporting **NVIDIA, OpenAI, Microsoft, Salesforce, Walmart,** and many others in almost every industry. Over 10,000 organizations have made Milvus or Zilliz Cloud their vector database of choice.

These aren't just technical success stories—they're examples of how vector databases are quietly becoming critical infrastructure that powers the AI applications people use every day.


## Why We Built Zilliz Cloud: Enterprise-Grade Vector Database as a Service

Milvus is open-source and free to use. But running Milvus well at enterprise scale requires deep expertise and significant resources. Index selection, memory management, scaling strategies, security configurations—these aren't trivial decisions. Many teams want the power of Milvus without the operational complexity and with enterprise support, SLA guarantees, etc.

That's why we built [Zilliz Cloud](https://zilliz.com/cloud)—a fully managed version of Milvus deployed across 25 global regions and 5 major clouds, including AWS, GCP, and Azure, designed specifically for enterprise-scale AI workloads that demand performance, security, and reliability.

Here's what makes Zilliz Cloud different:

- **Massive Scale with High Performance:** Our proprietary AI-powered AutoIndex engine delivers 3-5× faster query speeds than open-source Milvus, with zero index tuning required. The cloud-native architecture supports billions of vectors and tens of thousands of concurrent queries while maintaining sub-second response times.

- [**Built-in Security & Compliance**](https://zilliz.com/trust-center)**:** Encryption at rest and in transit, fine-grained RBAC, comprehensive audit logging, SAML/OAuth2.0 integration, and [BYOC](https://zilliz.com/bring-your-own-cloud) (bring your own cloud) deployments. We're compliant with GDPR, HIPAA, and other global standards that enterprises actually need. 

- **Optimized for Cost-Efficiency:** Tiered hot/cold data storage, elastic scaling that responds to real workloads, and pay-as-you-go pricing can reduce total cost of ownership by 50% or more compared to self-managed deployments.

- **Truly Cloud-Agnostic without vendor lock-in:** Deploy on AWS, Azure, GCP, Alibaba Cloud, or Tencent Cloud without vendor lock-in. We ensure global consistency and scalability regardless of where you run.

These capabilities might not sound flashy, but they solve real, daily problems that enterprise teams face when building AI applications at scale. And most importantly: it's still Milvus under the hood, so there's no proprietary lock-in or compatibility issues.


## What's Next: Vector Data Lake

We coined the term "[vector database](https://zilliz.com/learn/what-is-vector-database)" and were the first to build one, but we're not stopping there. We're now building the next evolution: **Vector Data Lake.**

**Here's the problem we're solving: not every vector search needs millisecond latency.** Many enterprises have massive datasets that are queried occasionally, including historical document analysis, batch similarity computations, and long-term trend analysis. For these use cases, a traditional real-time vector database is both overkill and expensive. 

Vector Data Lake uses a storage-compute separated architecture specifically optimized for massive-scale, infrequently accessed vectors while keeping costs dramatically lower than real-time systems.

**Core capabilities include:**

- **Unified Data Stack:** Seamlessly connects online and offline data layers with consistent formats and efficient storage, so you can move data between hot and cold tiers without reformatting or complex migrations.

- **Compatible Compute Ecosystem:** Works natively with frameworks like Spark and Ray, supporting everything from vector search to traditional ETL and analytics. This means your existing data teams can work with vector data using tools they already know.

- **Cost-Optimized Architecture:** Hot data stays on SSD or NVMe for fast access; cold data automatically moves to object storage like S3. Smart indexing and storage strategies keep I/O fast when you need it while making storage costs predictable and affordable.

This isn't about replacing vector databases—it's about giving enterprises the right tool for each workload. Real-time search for user-facing applications, cost-effective vector data lakes for analytics and historical processing.

We still believe in the logic behind Moore's Law and Jevons Paradox: as the unit cost of computing drops, adoption scales. The same applies to vector infrastructure.

By improving indexes, storage structures, caching, and deployment models—day in, day out—we hope to make AI infrastructure more accessible and affordable for everyone, and to help bring unstructured data into the AI-native future.


## A Big Thanks to You All!

Those 35K+ stars represent something we're genuinely proud of: a community of developers who find Milvus useful enough to recommend and contribute to.

But we're not done. Milvus has bugs to fix, performance improvements to make, and features our community has been asking for. Our roadmap is public, and we genuinely want your input on what to prioritize.

The number itself isn't what matters—it's the trust those stars represent. Trust that we'll keep building in the open, keep listening to feedback, and keep making Milvus better.

- **To our contributors:** your PRs, bug reports, and documentation improvements make Milvus better every day. Thank you so much. 

- **To our users:** thank you for trusting us with your production workloads and for the feedback that keeps us honest.

- **To our community:** thank you for answering questions, organizing events, and helping newcomers get started.

If you're new to vector databases, we'd love to help you get started. If you're already using Milvus or Zilliz Cloud, we'd love to [hear about your experience](https://zilliz.com/share-your-story). And if you're just curious about what we're building, our community channels are always open.

Let's keep building the infrastructure that makes AI applications possible—together.

***

Find us here: [Milvus on GitHub](https://github.com/milvus-io/milvus) |[ Zilliz Cloud](https://zilliz.com/) |[ Discord](https://discuss.milvus.io/) | [LinkedIn](https://www.linkedin.com/company/the-milvus-project/) | [X](https://x.com/zilliz_universe) | [YouTube](https://www.youtube.com/@MilvusVectorDatabase/featured)

[![](https://assets.zilliz.com/office_hour_4fb9130a9b.png)](https://meetings.hubspot.com/chloe-williams1/milvus-office-hour?__hstc=175614333.dc4bcf53f6c7d650ea8978dcdb9e7009.1727350436713.1751017913702.1751029841530.667&__hssc=175614333.3.1751029841530&__hsfp=3554976067)







Our Journey to 35K+ GitHub Stars: Building Milvus from Scratch


Our Journey to 35K+ GitHub Stars: The Real Story of Building Milvus from Scratch



Every company has the same problem: valuable knowledge trapped in PDFs, Word docs, and file shares that nobody can find when they need it. Support teams answer the same questions repeatedly, while developers waste hours searching through outdated documentation.

**What if your documents could answer questions directly?**

This tutorial shows you how to build a production-ready AI assistant that:

- Transforms your static documents into an intelligent Q&A system

- Maintains conversation context and memory

- Scales to handle enterprise workloads

- Includes security, monitoring, and observability out of the box


## What We'll Build

By the end of this tutorial, you'll have:

- A document ingestion pipeline that processes PDFs and Word docs

- A vector search system powered by Milvus for semantic search

- A chat API with memory and context awareness

- Enterprise-grade security and monitoring

- A complete working example you can deploy


## Key Components We’ll Use

- [**Spring Boot**](https://github.com/spring-projects/spring-boot) is a widely used Java framework for building backend applications with minimal configuration. It offers strong developer productivity, seamless integration with modern tooling, and built-in support for REST APIs, observability, and security.

- [**Milvus**](https://milvus.io/) is an open-source, high-performance, cloud-native vector database designed for semantic search. It allows you to store and search embeddings with millisecond-scale latency, even across billions of vectors.

- [**RAG**](https://zilliz.com/learn/Retrieval-Augmented-Generation) is an architecture that combines retrieval and generation: it fetches relevant knowledge snippets from a vector database like Milvus, then uses a language model to craft a fluent, contextual response.

- [**Ollama**](https://ollama.com/): Local AI model provider (OpenAI-compatible, completely free)


## Prerequisites

Before starting, ensure you have:

- Java 17+ installed

- Docker, Docker Compose

- Git for cloning the example repository

- Ollama installed and running locally

- Milvus (via Docker)

- Spring Boot 3.5.0 + Spring AI 1.0.0

- Micrometer, Testcontainers


## Environment Setup

Clone the Example Repository/: <https://github.com/topikachu/spring-ai-rag>


```
git clone https://github.com/topikachu/spring-ai-rag
cd spring-ai-rag
```
Verify your environment: 
```
# Verify Docker is running correctly
docker version
docker ps

# Verify Java version
java -version

# Verify Ollama installation
ollama --version
```

Download Ollama Models: 
```
# Pull required models for this project
ollama pull mistral          # Chat model
ollama pull nomic-embed-text # Embedding model

# Verify models are available
ollama list
```

Key Configuration (application.properties)
```
# Ollama Configuration (OpenAI-compatible API)
spring.ai.openai.base-url=http://localhost:11434
spring.ai.openai.chat.options.model=mistral
spring.ai.openai.embedding.options.model=nomic-embed-text
spring.ai.openai.embedding.options.dimensions=768

# Vector Store Configuration - dimensions must match embedding model
spring.ai.vectorstore.milvus.embedding-dimension=768
```


## Document ETL: Structuring Unstructured Text

This section walks through the heart of the system---transforming unstructured files into searchable, intelligent responses using vector embeddings, Milvus indexing, and Spring AI's RAG pipeline.

**Workflow Overview:**

- Use `TikaDocReader` to read PDFs and Word files

- Use token-based splitting to chunk documents while preserving context

- Generate embeddings using the OpenAI-compatible embedding model

- Store the embeddings in Milvus for later semantic search

![](https://assets.zilliz.com/workflow_7e9f990b18.png )




Sample Implementation

```
public Flux<Document> ingestionFlux() {
  return documentReader.getDocuments()
          .flatMap(document -> {
            var processChunks = Mono.fromRunnable(() -> {
              var chunks = textSplitter.apply(List.of(document));
              vectorStore.write(chunks); // expensive operation
            }).subscribeOn(Schedulers.boundedElastic());

            return Flux.concat(
                    Flux.just(document),
                    processChunks.then(Mono.empty())
            );
          })
          .doOnComplete(() -> log.info("RunIngestion() finished"))
          .doOnError(e -> log.error("Error during ingestion", e));
}
```


## Vector Storage: Millisecond-Scale Semantic Search with Milvus

Configuration Example: 

```
spring.ai.vectorstore.milvus.initialize-schema=true
spring.ai.vectorstore.milvus.embedding-dimension=768
```



📌 **Example:** When a user asks "Does Spring Boot support reactive programming with WebFlux?", Milvus returns related documentation segments, and the AI model generates a natural language answer with specific implementation details.


## Building a RAG-Enabled Chat: Contextual Q&A with Memory Integration

Core workflow: 

1. The user submits a question

2. Vector search retrieves the most relevant document chunks

3. The system loads past conversation context (via Redis)

4. The AI model generates a response that includes both new and historical context


![](https://assets.zilliz.com/build_a_rag_chat_workflow_976dcd9aa2.png)








Retrieval + Memory Chat Integration Example: 

```
public ChatClient.ChatClientRequestSpec input(String userInput, String conversationId) {
  return chatClient.prompt()
          .advisors(
                  messageChatMemoryAdvisor,
                  retrievalAugmentationAdvisor
          )
          .advisors(spec -> spec.param(CONVERSATION_ID, conversationId))
          .user(userInput);
}
```


For a smoother frontend experience, use the reactive stream API to return `Flux` content via server-sent events (SSE)---ideal for "typing" effects:

```
public Flux<String> stream(String userInput, String conversationId) {
    return input(userInput, conversationId)
            .stream().content();
}
```


REST API Controller:

```
@PostMapping(path = "/chat", produces = MediaType.TEXT_EVENT_STREAM_VALUE)
public Flux<String> chat(@RequestBody ChatRequest chatRequest, @RequestParam() String conversationId, Principal principal) {
  var conversationKey = String.format("%s:%s", principal.getName(), conversationId);
  return chatService.stream(chatRequest.userInput, conversationKey)
          .doOnError(exp -> log.error("Error in chat", exp));
}
```



## Enterprise-Grade API Security and System Observability

This section ensures your AI assistant doesn't just work---it runs securely, is traceable, and performs under real-world workloads.


### API Security: Role-Based Access Control

**Example: Securing Admin Endpoints**

```
@Override
protected void configure(HttpSecurity http) throws Exception {
    http
        .httpBasic()
        .and()
        .authorizeRequests(authz -> authz
            .antMatchers("/api/v1/index").hasRole("ADMIN")
            .anyRequest().authenticated()
        );
}
```


💡 **Production Tip:** For real-world deployments, use OAuth2 or JWT for scalable authentication.


### Observability: Full Stack Tracing and Metrics

**Tracing:** We'll use OpenTelemetry JavaAgent to trace full request flows from user chat to Milvus search and LLM response---including gRPC spans:

```
-javaagent:<path/to/opentelemetry-javaagent.jar> \
-Dotel.metrics.exporter=none \
-Dotel.logs.exporter=none
```


**Metrics:** Micrometer automatically exposes Prometheus-friendly metrics:

- Model Response Time

```
# HELP gen_ai_client_operation_seconds  
# TYPE gen_ai_client_operation_seconds summary
gen_ai_client_operation_seconds_count{...} 1
```


- Vector Retrieval Time

```
# HELP db_vector_client_operation_seconds
# TYPE db_vector_client_operation_seconds summary
db_vector_client_operation_seconds_count{...} 1
```


Configuration: 

```
management.endpoints.web.exposure.include=prometheus
```


💡 **Technical Note:** Spring Boot 3.2 introduces OTEL starters, but they don't cover gRPC (used by Milvus). To ensure end-to-end visibility, this project uses the JavaAgent approach.


## Running the Project: End-to-End Execution

Start the Complete System


```
export OPENAI_API_KEY=dummy
export SPRING_PROFILES_ACTIVE=ollama-openai
ollama pull mistral            # Pull chat model
ollama pull nomic-embed-text   # Pull embedding model

mvn clean test package
docker compose up -d
java -javaagent:target/otel/opentelemetry-javaagent.jar -Dotel.metrics.exporter=none -Dotel.logs.exporter=none  -Dinput.directory=$PWD/src/test/resources/corpus  -jar target/rag-0.0.1-SNAPSHOT.jar

curl --location 'localhost:8080/api/v1/index' \
--user "admin:password" \
--header 'Content-Type: application/json' \
--data '{}'

curl --location 'localhost:8080/api/v1/chat?conversationId=flat' \
--header 'Content-Type: application/json' \
--user "user:password" \
--data '{
    "userInput": "Does milvus support FLAT type index?"
}'

curl --location 'localhost:8080/api/v1/chat?conversationId=flat' \
--header 'Content-Type: application/json' \
--user "user:password" \
--data '{
    "userInput": "When shall I use this index type?"
}'

curl --location 'localhost:8080/api/v1/chat?conversationId=hnsw' \
--header 'Content-Type: application/json' \
--user "user:password" \
--data '{
    "userInput": "Does milvus support HNSW type index?"
}'

curl --location 'localhost:8080/api/v1/chat?conversationId=hnsw' \
--header 'Content-Type: application/json' \
--user "user:password" \
--data '{
    "userInput": "When shall I use this index type?"
}'

curl "http://localhost:8080/actuator/prometheus"
```


To view tracing UI, open[ http://localhost:16686/](http://localhost:16686/)

![](https://assets.zilliz.com/To_view_tracing_UI_686e8f54b9.png)


## Conclusion

You now have a production-ready AI assistant that transforms static documents into intelligent conversations. The system includes:

✅ **Document Processing**: Automated ingestion and vectorization
✅ **Semantic Search**: Fast, accurate retrieval with Milvus
✅ **Conversation Memory**: Context-aware chat experiences
✅ **Enterprise Security**: Authentication and access control 

✅ **Full Observability**: Monitoring, tracing, and metrics

By combining Spring Boot, Milvus, and Ollama, we turn static enterprise documents into dynamic, context-aware conversations---with full observability, memory, and security built-in.

Whether you're building internal copilots, domain-specific assistants, or customer-facing support bots, this architecture is designed to scale your workload and keep you in control of your data.

Curious about what Milvus can do for your AI stack? Explore the[ Milvus open-source project](https://milvus.io), try[ managed Milvus (Zilliz Cloud](https://zilliz.com)) for a hassle-free experience, or join our [Discord channel](https://discord.com/invite/8uyFbECzPX) for more hands-on guides like this.


Building a Production-Ready AI Assistant with Spring Boot and Milvus


From Docs to Dialogue: Building a Production-Ready AI Assistant with Spring Boot and Milvus




## The One Thing Breaking Your Vibe Coding Flow

Vibe Coding is having its moment. Tools like Cursor and Windsurf are redefining how we write software, making development feel effortless and intuitive. Ask for a function and get a snippet. Need a quick API call? It's generated before you finish typing.

**However, here's the catch that's ruining the vibe: AI assistants often generate outdated code that breaks in production.** This is because LLMs powering these tools often rely on outdated training data. Even the slickest AI copilot can suggest code that’s a year—or three—behind the curve. You might end up with a syntax that no longer works, deprecated API calls, or practices that today’s frameworks actively discourage. 

Consider this example: I asked Cursor to generate Milvus connection code, and it produced this:


```
connections.connect("default", host="localhost", port="19530")
```

This used to work perfectly, but the current pymilvus SDK recommends using `MilvusClient` for all connections and operations. The old method is no longer considered best practice, yet AI assistants continue to suggest it because their training data is often months or years out of date.

Even worse, when I requested OpenAI API code, Cursor generated a snippet using `gpt-3.5-turbo`—a model now marked _Legacy_ by OpenAI, costing triple the price of its successor while delivering inferior results. The code also relied on `openai.ChatCompletion`, an API deprecated as of March 2024.

​​![](https://assets.zilliz.com/gpt_pricing_6bfa92d83b.png)

This isn't just about broken code—it's about **broken flow**. The whole promise of Vibe Coding is that development should feel smooth and intuitive. But when your AI assistant generates deprecated APIs and outdated patterns, the vibe dies. You're back to Stack Overflow, back to documentation hunting, back to the old way of doing things.

Despite all the progress in Vibe Coding tools, developers still spend significant time bridging the "last mile" between generated code and production-ready solutions. The vibe is there, but the accuracy isn't.

**Until now.**


## Meet Milvus MCP: Vibe Coding with Always-Up-to-Date Docs

So, is there a way to combine the powerful codegen of tools like Cursor _with_ fresh documentation, so we can generate accurate code right inside the IDE?

Absolutely. By combining the Model Context Protocol (MCP) with Retrieval-Augmented Generation (RAG), we’ve created an enhanced solution called **Milvus MCP**. It helps developers using the Milvus SDK to automatically access the latest docs, enabling their IDE to produce the correct code. This service will be available soon—here’s a sneak peek at the architecture behind it.


### How It Works

![](https://assets.zilliz.com/The_Architecture_Behind_MCP_c9093162b6.png)

The diagram above shows a hybrid system that combines MCP (Model Context Protocol) and RAG (Retrieval-Augmented Generation) architectures to help developers generate accurate code. 

On the left side, developers working in AI-powered IDEs like Cursor or Windsurf interact through a chat interface, which triggers MCP tool calls. These requests are sent to the MCP Server on the right side, which hosts specialized tools for everyday coding tasks like code generation and refactoring.

The RAG component operates on the MCP server side, where the Milvus documentation has been pre-processed and stored as vectors in a Milvus database. When a tool receives a query, it performs a semantic search to retrieve the most relevant documentation snippets and code examples. This contextual information is then sent back to the client, where an LLM uses it to generate accurate, up-to-date code suggestions. 


### MCP transport mechanism 

MCP supports two transport mechanisms: `stdio` and `SSE`:

- Standard Input/Output (stdio): The `stdio` transport allows communication over standard input/output streams. It’s particularly useful for local tools or command-line integrations.

- Server-Sent Events (SSE): SSE supports server-to-client streaming using HTTP POST requests for client-to-server communication.

Because `stdio` relies on local infrastructure, users must manage document ingestion themselves. In our case, **SSE is a better fit**—the server handles all document processing and updates automatically. For example, docs can be re-indexed daily. Users only need to add this JSON config to their MCP setup:

```
{
  "mcpServers": {
    "milvus-code-generate-helper": {
      "url": "http://<SERVER_ADDRESS>:23333/milvus-code-helper/sse"
    }
  }
}
```


Once this is in place, your IDE (such as Cursor or Windsurf) can start communicating with the server-side tools—automatically retrieving the latest Milvus documentation for smarter, up-to-date code generation.


## Milvus MCP in Action

To show how this system works in practice, we've created three ready-to-use tools on the Milvus MCP Server that you can access directly from your IDE. Each tool solves a common problem developers face when working with Milvus:

- **pymilvus-code-generator**: Writes Python code for you when you need to perform common Milvus operations like creating collections, inserting data, or running searches using the pymilvus SDK.

- **orm-client-code-convertor**: Modernizes your existing Python code by replacing outdated ORM (Object Relational Mapping) patterns with the simpler, newer MilvusClient syntax.

- **language-translator**: Converts your Milvus SDK code between programming languages. For instance, if you have working Python SDK code but need it in TypeScript SDK, this tool translates it for you.

Now, let’s take a look at how they work. 


### pymilvus-code-generator


<div style="padding:66.98% 0 0 0;position:relative;"><iframe src="https://player.vimeo.com/video/1093504910?badge=0&amp;autopause=0&amp;player_id=0&amp;app_id=58479" frameborder="0" allow="autoplay; fullscreen; picture-in-picture; clipboard-write; encrypted-media; web-share" style="position:absolute;top:0;left:0;width:100%;height:100%;" title="pymilvus-code-generator"></iframe></div><script src="https://player.vimeo.com/api/player.js"></script>



In this demo, I asked Cursor to generate full-text search code using `pymilvus`. Cursor successfully invokes the correct MCP tool and outputs spec-compliant code. Most `pymilvus` use cases work seamlessly with this tool.

Here’s a side-by-side comparison with and without this tool. 

**With MCP MCP:**

![](https://assets.zilliz.com/With_Milvus_MCP_f72ad4cfb6.png)

→ Cursor with Milvus MCP uses the latest `MilvusClient` interface to create a collection.

**Without MCP:**

![](https://assets.zilliz.com/Without_Milvus_MCP_3336d956a4.png)

→ The Cursor without the Milvus MCP server uses outdated ORM syntax—no longer advised.


### orm-client-code-convertor


<div style="padding:66.98% 0 0 0;position:relative;"><iframe src="https://player.vimeo.com/video/1093504859?badge=0&amp;autopause=0&amp;player_id=0&amp;app_id=58479" frameborder="0" allow="autoplay; fullscreen; picture-in-picture; clipboard-write; encrypted-media; web-share" style="position:absolute;top:0;left:0;width:100%;height:100%;" title="orm-client-code-convertor"></iframe></div><script src="https://player.vimeo.com/api/player.js"></script>


In this example, the user highlights some ORM-style code and requests a conversion. The tool correctly rewrites the connection and schema logic using a `MilvusClient` instance. The user can accept all changes with one click.


### **language-translator**

<div style="padding:66.98% 0 0 0;position:relative;"><iframe src="https://player.vimeo.com/video/1093504885?badge=0&amp;autopause=0&amp;player_id=0&amp;app_id=58479" frameborder="0" allow="autoplay; fullscreen; picture-in-picture; clipboard-write; encrypted-media; web-share" style="position:absolute;top:0;left:0;width:100%;height:100%;" title="tool3 ts-1"></iframe></div><script src="https://player.vimeo.com/api/player.js"></script>

Here, the user selects a `.py` file and asks for a TypeScript translation. The tool calls the correct MCP endpoint, retrieves the latest TypeScript SDK docs, and outputs an equivalent `.ts` file with the same business logic. This is ideal for cross-language migrations.


## Comparing Milvus MCP with Context7, DeepWiki, and Other Tools

We’ve discussed the “last mile” hallucination problem in Vibe Coding. Beyond our Milvus MCP, many other tools also aim to solve this issue, such as Context7 and DeepWiki. These tools, often powered by MCP or RAG, help inject up-to-date docs and code samples into the model’s context window. 


### Context7

![](https://assets.zilliz.com/Context7_fc32b53a0e.png)

Figure: Context7’s Milvus page lets users search and customize doc snippets (<https://context7.com/milvus-io/milvus>)

Context7 provides up-to-date, version-specific documentation and code examples for LLMs and AI code editors. The core problem it addresses is that LLMs rely on outdated or generic information about the libraries you use, giving you code examples that are outdated and based on year-old training data.

Context7 MCP pulls up-to-date, version-specific documentation and code examples straight from the source and places them directly into your prompt. It supports GitHub repo imports and `llms.txt` files, including formats like `.md`, `.mdx`, `.txt`, `.rst`, and `.ipynb` (not `.py` files).

Users can either manually copy content from the site or use Context7’s MCP integration for automated retrieval.


### **DeepWiki**

![](https://assets.zilliz.com/Deep_Wiki_bebe01aa6f.png)

Figure: DeepWiki provides auto-generated summaries of Milvus, including logic and architecture (<https://deepwiki.com/milvus-io/milvus>)

DeepWiki auto-parses open-source GitHub projects to create readable technical docs, diagrams, and flowcharts. It includes a chat interface for natural language Q&A. However, it prioritizes code files over documentation, so it may overlook key doc insights. It currently lacks MCP integration.


### Cursor Agent Mode

![](https://assets.zilliz.com/Cursor_Agent_Mode_fba8ef66af.png)

Agent mode in Cursor enables web search, MCP calls, and plugin toggles. While powerful, it’s sometimes inconsistent. You can use `@` to manually insert docs, but that requires you to find and attach the content first.


### llms.txt

`llms.txt` isn’t a tool—it’s a proposed standard to provide LLMs with structured website content. Usually, in Markdown, it goes in a site’s root directory and organizes titles, doc trees, tutorials, API links, and more.

It’s not a tool on its own, but it pairs well with those that support it.


### Side-by-Side Feature Comparison: Milvus MCP vs. Context7 vs. DeepWiki vs Cursor Agent Mode vs llms.txt 

|                       |                          |                             |                         |                         |                                 |
| :-------------------: | :----------------------: | :-------------------------: | :---------------------: | :---------------------: | :-----------------------------: |
|      **Feature**      |       **Context7**       |         **DeepWiki**        |  **Cursor Agent Mode**  |       **llms.txt**      |          **Milvus MCP**         |
|    **Doc Handling**   |    Docs only, no code    | Code-focused, may miss docs |      User-selected      |   Structured Markdown   |    Official Milvus docs only    |
| **Context Retrieval** |        Auto-inject       |      Manual copy/paste      |   Mixed, less accurate  | Structured pre-labeling | Auto-retrieve from vector store |
|   **Custom Import**   |    ✅ GitHub, llms.txt    |   ✅ GitHub (incl. private)  | ❌ Manual selection only |   ✅ Manually authored   |       ❌ Server-maintained       |
|   **Manual Effort**   | Partial (MCP vs. manual) |         Manual copy         |       Semi-manual       |        Admin only       |      No user action needed      |
|  **MCP Integration**  |           ✅ Yes          |             ❌ No            |    ✅ Yes (with setup)   |       ❌ Not a tool      |            ✅ Required           |
|     **Advantages**    |  Live updates, IDE-ready | Visual diagrams, QA support |     Custom workflows    |  Structured data for AI |   Maintained by Milvus/Zilliz   |
|    **Limitations**    |   No code file support   |          Skips docs         |  Relies on web accuracy |   Requires other tools  |     Focused solely on Milvus    |

Milvus MCP is built specifically for Milvus database development. It automatically gets the latest official documentation and works seamlessly with your coding environment. If you're working with Milvus, this is your best option.

Other tools like Context7, DeepWiki, and Cursor Agent Mode work with many different technologies, but they're not as specialized or accurate for Milvus-specific work.

Choose based on what you need. The good news is these tools work well together - you can use several at once to get the best results for different parts of your project.


## Milvus MCP is Coming Soon!

The hallucination problem in Vibe Coding isn't just a minor inconvenience—it's a productivity killer that forces developers back into manual verification workflows. Milvus MCP demonstrates how specialized MCP servers can solve this by providing real-time access to current documentation.

For Milvus developers, this means no more debugging deprecated `connections.connect()` calls or wrestling with outdated ORM patterns. The three tools—pymilvus-code-generator, orm-client-code-convertor, and language-translator—handle the most common pain points automatically.

Ready to try it? The service will be available soon for early access testing. Stay tuned. 


Why Your Vibe Coding Generates Outdated Code and How to Fix It with Milvus MCP



As AI-powered search has evolved from experimental projects to mission-critical infrastructure, the demands on [vector databases](https://milvus.io/blog/what-is-a-vector-database.md) have intensified. Organizations need to handle billions of vectors while managing infrastructure costs, supporting real-time data ingestion, and providing sophisticated retrieval beyond basic [similarity search](https://zilliz.com/learn/vector-similarity-search). To tackle these evolving challenges, we've been hard at work developing and refining Milvus. The community response has been incredibly encouraging, with valuable feedback helping shape our direction. 

After months of intensive development, we're excited to announce that **Milvus 2.6 is now available**. This release directly addresses the most pressing challenges in vector search today: **_scaling efficiently while keeping costs under control._**

Milvus 2.6 delivers breakthrough innovations across three critical areas: **cost reduction, advanced search capabilities, and architectural improvements for massive scale**. The results speak for themselves:

- **72% memory reduction** with RaBitQ 1-bit quantization while delivering 4x faster queries

- **50% cost savings** through intelligent tiered storage

- **4x faster full-text search** than Elasticsearch with our enhanced BM25 implementation

- **100x faster** JSON filtering with the newly introduced Path Index

- **Search freshness is achieved economically** with the new zero-disk architecture 

- **Streamlined embedding workflow** with the new “data in and data out” experience 

- **Up to 100K collections in a single cluster** for future-proof multi-tenancy


## Innovations for Cost Reduction: Making Vector Search Affordable

Memory consumption presents one of the biggest challenges when scaling vector search to billions of records. Milvus 2.6 introduces several key optimizations that significantly reduce your infrastructure costs while also improving performance.


### RaBitQ 1-bit Quantization: 72% Memory Reduction with 4× Performance

Traditional quantization methods force you to trade in search quality for memory savings. Milvus 2.6 changes this with [RaBitQ 1-bit quantization](https://milvus.io/blog/bring-vector-compression-to-the-extreme-how-milvus-serves-3%C3%97-more-queries-with-rabitq.md) combined with an intelligent refinement mechanism.

The new IVF_RABITQ index compresses the main index to 1/32 of its original size through 1-bit quantization. When used together with an optional SQ8 refinement, this approach maintains high search quality (95% recall) using only 1/4 of the original memory footprint. 

Our preliminary benchmarks reveal promising results: 

|  **Performance Metric** | **Traditional IVF_FLAT** | **RaBitQ (1-bit) Only** | **RaBitQ (1-bit) + SQ8 Refine** |
| --------------------- | ----------------------- | --------------------- | ----------------------------- |
|     Memory Footprint    |      100% (baseline)      |    3% (97% reduction)   |       28% (72% reduction)       |
|          Recall         |           95.2%           |          76.3%          |              94.9%              |
| Search Throughput (QPS) |            236            |    648 (2.7× faster)    |         946 (4× faster)         |

_Table: VectorDBBench evaluation with 1M vectors of 768 dimensions, tested on AWS m6id.2xlarge_

The real breakthrough here isn't just the 72% memory reduction, but achieving this while simultaneously delivering a 4× throughput improvement. This means you can serve the same workload with 75% fewer servers or handle 4× more traffic on your existing infrastructure, all without sacrificing the recall. 

For enterprise users utilizing fully managed Milvus on[ Zilliz Cloud](https://zilliz.com/cloud), we’re developing an automated strategy that dynamically adjusts RaBitQ parameters based on your specific workload characteristics and precision requirements. You will simply enjoy greater cost-effectiveness across all Zilliz Cloud CU types.


### Hot-Cold Tiered Storage: 50% Cost Reduction Through Intelligent Data Placement

Real-world vector search workloads contain data with vastly different access patterns. Frequently accessed data needs instant availability, while archival data can tolerate slightly higher latency in exchange for dramatically lower storage costs.

Milvus 2.6 introduces a tiered storage architecture that automatically classifies data based on access patterns and places it in appropriate storage tiers:

- **Intelligent data classification**: Milvus automatically identifies hot (frequently accessed) and cold (rarely accessed) data segments based on access patterns

- **Optimized storage placement**: Hot data remains in high-performance memory/SSD, while cold data moves to more economical object storage

- **Dynamic data movement**: As usage patterns change, data automatically migrates between tiers

- **Transparent retrieval**: When queries touch cold data, it's automatically loaded on demand

The result is up to 50% reduction in storage costs while maintaining query performance for active data.


### Additional Cost Optimizations

Milvus 2.6 also introduces Int8 vector support for HNSW indexes, Storage v2 format for optimized structure that reduces IOPS and memory requirements, and easier installation directly through APT/YUM package managers.


## Advanced Search Capabilities: Beyond Basic Vector Similarity

Vector search alone isn't enough for modern AI applications. Users demand the precision of traditional information retrieval combined with the semantic understanding of vector embeddings. Milvus 2.6 introduces a suite of advanced search features that bridge this gap.


### Turbocharged BM25: 400% Faster Full-Text Search Than Elasticsearch

[Full-text search](https://milvus.io/blog/full-text-search-in-milvus-what-is-under-the-hood.md) has become essential for building hybrid retrieval systems in vector databases. In Milvus 2.6, significant performance improvements have been made to full-text search, building on the BM25 implementation introduced since version 2.5. For example, this release introduces new parameters like `drop_ratio_search` and `dim_max_score_ratio`, enhancing precision and speed tuning and offering more fine-grained search controls. 

Our benchmarks against the industry-standard BEIR dataset show Milvus 2.6 achieving 3-4× higher throughput than Elasticsearch with equivalent recall rates. For specific workloads, the improvement reaches 7× higher QPS.

![](https://assets.zilliz.com/milvus_vs_ES_when_QPS_with_top_K1000_cadd1ac921.png)


### JSON Path Index: 100x Faster Filtering

Milvus has supported JSON data type for a long time, but filtering on JSON fields was slow due to the lack of index support. Milvus 2.6 adds support for JSON path index to boost the performance significantly. 

Consider a user profile database where each record contains nested metadata like:

```
{
  "user": {
    "location": {
      "city": "San Francisco",
      "country": "USA"
    },
    "interests": ["AI", "Databases", "Cloud Computing"]
  },
  "subscription": {
    "plan": "enterprise",
    "status": "active"
  }
}
```


For a semantic search “users interested in AI” scoped to only San Francisco, Milvus used to parse and evaluate the entire JSON object for every record, making the query very expensive and slow.

Now, Milvus allows you to create indexes on specific paths within JSON fields to speed up the search:

```
index_params.add_index(
    field_name="metadata",
    index_type="INVERTED",
    index_name="json_index",
    params={
        "json_path": "metadata[\"user\"][\"location\"][\"city\"]",  
        "json_cast_type": "varchar"
    }
```

In our performance testing with 100M+ records, JSON Path Index reduced filter latency from **140ms** (P99: 480ms) to just **1.5ms** (P99: 10ms)—a 99% latency reduction that makes such searches practical in production.

This feature is particularly valuable for: 

- Recommendation systems with complex user attribute filtering

- RAG applications that filter documents by metadata

- Multi-tenant systems where data segmentation is critical


### Enhanced Text Processing and Time-Aware Search

Milvus 2.6 introduces a completely revamped text analysis pipeline with sophisticated language handling, including the Lindera tokenizer for Japanese and Korean, the ICU tokenizer for comprehensive multilingual support, and enhanced Jieba with custom dictionary integration.

**Phrase Match Intelligence** captures semantic nuance in word order, distinguishing between "machine learning techniques" and "learning machine techniques":

```
PHRASE_MATCH(document_text, "artificial intelligence research", slop=1)
```

**Time-Aware Decay Functions** automatically prioritize fresh content by adjusting relevance scores based on document age, with configurable decay rates and function types (exponential, Gaussian, or linear).


### Streamlined Search: Data in, Data Out Experience

The disconnect between raw data and vector embeddings is another pain point for developers using vector databases. Before data reaches Milvus for indexing and vector search, it often undergoes preprocessing using external models that convert raw text, images, or audio into vector representations. After retrieval, additional downstream processing is also required, such as mapping result IDs back to the original content.

Milvus 2.6 simplifies these embedding workflows with the new **Function** interface that integrates third-party embedding models directly into your search pipeline. Instead of pre-computing embeddings, you can now:

1. **Insert raw data directly**: Submit text, images, or other content to Milvus

2. **Configure embedding providers**: Connect to embedding API services from OpenAI, AWS Bedrock, Google Vertex AI, Hugging Face, and more.

3. **Query using natural language**: Search using raw text queries directly

This creates a "Data-In, Data-Out" experience where Milvus streamlines all the behind-the-scenes vector transformations for you.


## Architectural Evolution: Scaling to Tens of Billions of Vectors 

Milvus 2.6 introduces fundamental architectural innovations that enable cost-effective scaling to tens of billions of vectors.


### Replacing Kafka and Pulsar with a New Woodpecker WAL

Previous Milvus deployments relied on external message queues, such as Kafka or Pulsar, as the Write-Ahead Log (WAL) system. While these systems initially worked well, they introduced significant operational complexity and resource overhead.

Milvus 2.6 introduces [**Woodpecker**](https://milvus.io/blog/we-replaced-kafka-pulsar-with-a-woodpecker-for-milvus.md), a purpose-built, cloud-native WAL system that eliminates these external dependencies through a revolutionary zero-disk design:

- **Everything on object storage**: All log data is persisted in object storage like S3, Google Cloud Storage, or MinIO

- **Distributed metadata**: Metadata is still managed by the etcd key-value store

- **No local disk dependencies**: A choice to eliminate complex architecture and operational overhead involved in distributed local permanent state.

We ran comprehensive benchmarks comparing Woodpecker's performance:

| **System** | **Kafka**   | **Pulsar** | **WP MinIO** | **WP Local** | **WP S3** |
| ---------- | ----------- | ---------- | ------------ | ------------ | --------- |
| Throughput | 129.96 MB/s | 107 MB/s   | 71 MB/s      | 450 MB/s     | 750 MB/s  |
| Latency    | 58 ms       | 35 ms      | 184 ms       | 1.8 ms       | 166 ms    |

Woodpecker consistently reaches 60-80% of the theoretical maximum throughput for each storage backend, with local file system mode achieving 450 MB/s—3.5× faster than Kafka—and S3 mode reaching 750 MB/s, 5.8× higher than Kafka. 

For more details about Woodpecker, check out this blog: [We Replaced Kafka/Pulsar with a Woodpecker for Milvus](https://milvus.io/blog/we-replaced-kafka-pulsar-with-a-woodpecker-for-milvus.md).


### Search Freshness Achieved Economically

Mission-critical search usually requires newly ingested data to be instantly searchable. Milvus 2.6 replaces message queue dependency to fundamentally improve the handling of fresh updates and provide search freshness at lower resource overhead. The new architecture adds the new **Streaming Node**, a dedicated component that works in close coordination with other Milvus components like the Query Node and Data Node. Streaming Node is built on top of Woodpecker, our lightweight, cloud-native Write-Ahead Log (WAL) system. 

This new component enables: 

- **Great compatibility**: Works with both the new Woodpecker WAL and is backward compatible with Kafka, Pulsar, and other streaming platforms

- **Incremental indexing**: New data becomes searchable immediately, without batch delays

- **Continuous query serving**: Simultaneous high-throughput ingestion and low-latency querying

By isolating streaming from batch processing, the Streaming Node helps Milvus maintain stable performance and search freshness even during high-volume data ingestion. It's designed with horizontal scalability in mind, dynamically scaling node capacity based on data throughput.


### Enhanced Multi-tenancy Capability: Scaling to 100k Collections Per Cluster

Enterprise deployments often require tenant-level isolation. Milvus 2.6 dramatically increases the multi-tenancy support by allowing up to **100,000 collections** per cluster. This is a crucial improvement for organizations running a monolithic large cluster serving many tenants.

This improvement is made possible by numerous engineering optimizations on metadata management, resource allocation, and query planning. Milvus users can now enjoy stable performance even with tens of thousands of collections.


### Other Improvements

Milvus 2.6 offers more architectural enhancements, such as CDC + BulkInsert for simplified data replication across geographic regions and Coord Merge for better cluster coordination in large-scale deployments.


## Getting Started with Milvus 2.6

Milvus 2.6 represents a massive engineering effort with dozens of new features and performance optimizations, developed collaboratively by Zilliz engineers and our amazing community contributors. While we've covered the headline features here, there's more to discover. We highly recommend diving into our comprehensive [release notes](https://milvus.io/docs/release_notes.md) to explore everything this release has to offer! 

Complete documentation, migration guides, and tutorials are available on the[ Milvus website](https://milvus.io/). For questions and community support, join our[ Discord channel](https://discord.com/invite/8uyFbECzPX) or file issues on[ GitHub](https://github.com/milvus-io/milvus).


Introducing Milvus 2.6: Affordable Vector Search at Billion Scale



## Introduction

When building AI-powered applications with [vector databases](https://milvus.io/blog/what-is-a-vector-database.md), understanding system performance becomes critical as your application scales. A single search request might trigger multiple internal operations—vector indexing, similarity calculations, and data retrieval—across different components. Without proper observability, diagnosing slowdowns or failures becomes like finding a needle in a haystack.

**Distributed tracing** solves this problem by tracking requests as they flow through your system, giving you a complete picture of what's happening under the hood.

[**OpenTelemetry (OTEL)**](https://github.com/open-telemetry) is an open-source observability framework backed by the [Cloud Native Computing Foundation (CNCF)](https://www.cncf.io/) that helps you collect traces, metrics, and logs from your applications. It's vendor-neutral, widely adopted, and works seamlessly with popular monitoring tools.

In this guide, we'll show you how to add end-to-end tracing to [**Milvus**](https://milvus.io/), a high-performance vector database built for AI applications. You'll learn to track everything from client requests to internal database operations, making performance optimization and debugging much easier.

We'll also utilize [**Jaeger**](https://github.com/jaegertracing/jaeger-ui) to visualize the trace data, providing you with powerful insights into your vector database operations. 


## What We'll Build

By the end of this tutorial, you'll have a complete tracing pipeline consisting of:

1. **Milvus vector database** with OpenTelemetry tracing enabled

2. **Jaeger** for trace visualization and analysis

3. **A Python client** that automatically traces all Milvus operations

4. **End-to-end visibility** from client requests to database operations

Estimated setup time: 15-20 minutes


## Quick Start (5 Minutes)

Want to see it working first? Here's the fastest path:

1. Clone the demo repository:

```
git clone https://github.com/topikachu/milvus-py-otel
cd milvus-py-otel
```

2. Start the services:

```
docker-compose up -d
```

3. Wait 30 seconds, then check Jaeger UI at: `http://localhost:16686`

4. Run the Python example:

```
pip install -r requirements.txt
python example.py
```

5. Refresh Jaeger and look for traces from both `standalone` (Milvus) and `milvus-client` services.

If you see traces appearing, everything is working! Now let's understand how it all fits together.


## Environment Setup

Here's what you'll need:

- **Milvus 2.5.11** (vector database)

- **Jaeger 1.46.0** (trace visualization)

- **Python 3.7+** (client development)

- **Docker and Docker Compose** (container orchestration)

These versions have been tested together; however, newer versions should also work fine.


## Setting Up Milvus and Jaeger

We'll use Docker Compose to run both services with proper networking and configuration.


### Docker Compose Configuration

Create a `docker-compose.yaml` file:

```
version: '3.7'
Services:
# Milvus - configured to send traces to Jaeger
  milvus:
    image: milvusdb/milvus:v2.5.11
    command: ["milvus", "run", "standalone"]
    environment:
      - ETCD_USE_EMBED=true
      - ETCD_DATA_DIR=/var/lib/milvus/etcd
      - ETCD_CONFIG_PATH=/milvus/configs/embedEtcd.yaml
      - COMMON_STORAGETYPE=local
    volumes:
      - ./embedEtcd.yaml:/milvus/configs/embedEtcd.yaml
      - ./milvus.yaml:/milvus/configs/milvus.yaml
    ports:
      - "19530:19530"
      - "9091:9091"
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9091/healthz"]
      interval: 30s
      start_period: 90s
      timeout: 20s
      retries: 3
    security_opt:
      - seccomp:unconfined
    depends_on:
      - jaeger

# Jaeger - starts first since Milvus depends on it
  jaeger:
    image: jaegertracing/all-in-one:1.46.0
    ports:
      - "16686:16686"  # Jaeger UI
      - "4317:4317"    # OTLP gRPC receiver
      - "4318:4318"    # OTLP HTTP receiver
      - "5778:5778"    # Jaeger agent configs
      - "9411:9411"    # Zipkin compatible endpoint
    environment:
      - COLLECTOR_OTLP_ENABLED=true
```


**Note:** Example configuration files `embedEtcd.yaml` and `milvus.yaml` can be found at: [https://github.com/topikachu/milvus-py-otel](https://github.com/topikachu/milvus-py-otel).


### Milvus Tracing Configuration

Create `configs/milvus.yaml` with the tracing configuration:

```
# OpenTelemetry tracing configuration
trace:
  exporter: otlp           # Use OpenTelemetry Protocol
  sampleFraction: 1.0      # Trace 100% of requests (reduce for production)
  otlp:
    endpoint: jaeger:4317  # Jaeger's OTLP gRPC endpoint
    method: grpc          # Use gRPC protocol
    secure: false         # No TLS (use true in production)
    initTimeoutSeconds: 10
```

Configuration explained:

- `sampleFraction: 1.0` traces every request (useful for development, but use 0.1 or lower in production)

- `secure: false` disables TLS (enable in production)

- `endpoint: jaeger:4317` uses Docker service name for internal communication


### Starting the Services

```
docker-compose up -d
```


### Verifying Trace Delivery from Milvus to Jaeger

Once the services are running, you can verify if trace data is emitted from the Milvus standalone and received by Jaeger. 

- Open your browser and visit Jaeger UI at: `http://localhost:16686/search`

- In the **Search** panel (top-left), choose the **Service** dropdown and select `standalone`. If you see `standalone` in the service list, it means Milvus's built-in OpenTelemetry configuration is working and has successfully pushed trace data to Jaeger.

- Click **Find Traces** to explore trace chains generated by internal Milvus components (such as gRPC interactions between modules).

![](https://assets.zilliz.com/find_traces_811bf9d8ee.png)


### If Trace Data Is Not Showing:

- Double-check that the `trace` block in `milvus.yaml` is configured correctly and that Jaeger is running without issues.

- Inspect the Milvus container logs to see if there are any errors related to Trace initialization.

- Wait a few seconds and refresh the Jaeger UI; trace reporting may experience a short delay.


## Python Client Setup and Dependencies

Now let's set up the Python client to trace all Milvus operations automatically.

First, create a `requirements.txt` file:

```
# OpenTelemetry core
opentelemetry-api==1.33.1
opentelemetry-sdk==1.33.1
# OTLP exporters
opentelemetry-exporter-otlp==1.33.1
opentelemetry-exporter-otlp-proto-grpc==1.33.1
# Automatic gRPC instrumentation
opentelemetry-instrumentation-grpc==0.54b1
# Milvus client
pymilvus==2.5.9
```


Then install dependencies via:

```
pip install -r requirements.txt
```

This ensures your Python environment is ready for tracing gRPC calls made to the Milvus backend.


## Initializing OpenTelemetry in Python

Now, let’s configure tracing within your Python application. This snippet sets up OTEL with gRPC instrumentation and prepares a tracer. 

```
import os
from opentelemetry import trace
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.grpc import GrpcInstrumentorClient

# Set OTEL environment variables (you can also load them from external configs)
os.environ['OTEL_EXPORTER_OTLP_ENDPOINT'] = 'http://localhost:4317'
os.environ['OTEL_SERVICE_NAME'] = 'milvus-client'

# Define service metadata
resource = Resource.create({
    "service.name": "milvus-client",
    "application": "milvus-otel-test"
})

# Initialize tracer and export processor
trace.set_tracer_provider(
    TracerProvider(resource=resource)
)
otlp_exporter = OTLPSpanExporter()
span_processor = BatchSpanProcessor(otlp_exporter)
trace.get_tracer_provider().add_span_processor(span_processor)

# Enable automatic instrumentation for gRPC clients
grpc_client_instrumentor = GrpcInstrumentorClient()
grpc_client_instrumentor.instrument()

# Acquire tracer
tracer = trace.get_tracer(__name__)
```

Here, `GrpcInstrumentorClient()` hooks into the underlying gRPC stack so you don’t need to manually modify client code for instrumentation. The `OTLPSpanExporter()` is configured to send trace data to your local Jaeger instance.


## Complete Milvus Python Example with Tracing

Now let's create a comprehensive example that demonstrates tracing with realistic Milvus operations:

```
from pymilvus import MilvusClient
from opentelemetry import trace

with tracer.start_as_current_span("test_milvus_otel"):
    milvus_client = MilvusClient(
        uri="http://localhost:19530",
    )
    collection_name = "quick_setup"

    # Drop collection if it exists
    if milvus_client.has_collection(collection_name):
        milvus_client.drop_collection(collection_name)

    # Create collection
    milvus_client.create_collection(
        collection_name=collection_name,
        dimension=5
    )

    # Add additional operations here
    
    milvus_client.close()
```


## Viewing Trace Output

Once your Python client sends trace data, return to Jaeger: [`http://localhost:16686`](http://localhost:16686)

Select the `milvus-client` service to view trace spans that correspond to your Python client’s Milvus operations. This makes it much easier to analyze performance and trace interactions across system boundaries.

![](https://assets.zilliz.com/milvus_client_22aab6ab9f.png)


## Examples in Other Languages

Beyond Python, you can implement Milvus tracing in other languages:

👉[**Java**](https://github.com/topikachu/milvus-java-otel): Use the OpenTelemetry Java Agent for zero-code instrumentation
👉[**Go**](https://github.com/topikachu/milvus-go-otel): Leverage the OpenTelemetry Go SDK for native integration
👉[**Node.js**](https://github.com/topikachu/milvus-nodejs-otel): Auto-instrument gRPC calls with the JavaScript SDK

Each example follows similar patterns but uses language-specific OpenTelemetry libraries.


## Summary

You've successfully implemented end-to-end tracing for Milvus operations! Here's what you've accomplished:

- ✅ **Infrastructure**: Set up Milvus and Jaeger with proper networking

- ✅ **Server-side tracing**: Configured Milvus to export traces automatically

- ✅ **Client-side tracing**: Instrumented Python client with OpenTelemetry

- ✅ **Visualization**: Used Jaeger to analyze system performance

- ✅ **Production readiness**: Learned configuration best practices

All works without any changes to the Milvus SDK source code. Just a few configuration settings and your tracing pipeline is live—simple, effective, and ready for production.

You can take it further by integrating logs and metrics to build a complete monitoring solution for your AI-native vector database deployment.


## Learn More 

- Milvus Documentation: [https://milvus.io/docs](https://milvus.io/docs)

- OpenTelemetry for Python: [https://opentelemetry.io/docs/instrumentation/python/](https://opentelemetry.io/docs/instrumentation/python/)

- Jaeger Documentation: [https://www.jaegertracing.io/docs/](https://www.jaegertracing.io/docs/)

- Milvus OpenTelemetry Integration Demo (Python): [https://github.com/topikachu/milvus-py-otel](https://github.com/topikachu/milvus-py-otel)

How to Set Up OpenTelemetry with Milvus for End-to-End Request Tracing


It has never been easier to build Generative AI applications. A rich ecosystem of tools, AI models, and datasets allows even non-specialized software engineers to build impressive chatbots, image generators, and more. This tooling, for the most part, is made for Python and builds on top of PyTorch. But what about when you don’t have access to Python in production and need to use Java, Golang, Rust, C++, or another language?

We will restrict ourselves to model inference, including both embedding models and foundation models; other tasks, such as model training and fine-tuning, are not typically completed at deployment time. What are our options for model inference without Python? The most obvious solution is to utilize an online service from providers like Anthropic or Mistral. They typically provide an SDK for languages other than Python, and if they didn’t, it would require only simple REST API calls. But what if our solution has to be entirely local due to, for example, compliance or privacy concerns?

Another solution is to run a Python server locally. The original problem was posed as being unable to run Python in production, so that rules out using a local Python server. Related local solutions will likely suffer similar legal, security-based, or technical restrictions. _We need a fully contained solution that allows us to call the model directly from Java or another non-Python language._

![](https://assets.zilliz.com/A_Python_metamorphoses_into_an_Onyx_butterfly_a65c340c47.png)

_Figure 1: A Python metamorphoses into an Onyx butterfly._


## What is ONNX (Open Neural Network Exchange)? 
[ONNX](https://github.com/onnx/onnx) (Open Neural Network Exchange) is a platform-agnostic ecosystem of tools for performing neural network model inference. It was initially developed by the PyTorch team at Meta (then Facebook), with further contributions from Microsoft, IBM, Huawei, Intel, AMD, Arm, and Qualcomm. Currently, it is an open-source project owned by the Linux Foundation for AI and Data. ONNX is the de facto method for distributing platform-agnostic neural network models.

![](https://assets.zilliz.com/A_partial_ONNX_computational_graph_for_a_NN_transformer_11deebefe0.png)

_Figure 2: A (partial) ONNX computational graph for a NN transformer_

**We typically use “ONNX” in a narrower sense to refer to its file format.** An ONNX model file represents a computational graph, often including the weight values of a mathematical function, and the standard defines common operations for neural networks. You can think of it similarly to the computational graph created when you use autodiff with PyTorch. From another perspective, the ONNX file format serves as an _intermediate representation_ (IR) for neural networks, much like native code compilation, which also involves an IR step. See the illustration above visualizing an ONNX computational graph.

![](https://assets.zilliz.com/An_IR_allows_many_combinations_of_front_ends_and_back_ends_a05e259849.png)

_Figure 3: An IR allows many combinations of front-ends and back-ends_

The ONNX file format is just one part of the ONNX ecosystem, which also includes libraries for manipulating computational graphs and libraries for loading and running ONNX model files. These libraries span languages and platforms. Since ONNX is just an IR (Intermediate Representation Language), optimizations specific to a given hardware platform can be applied before running it with native code. See the figure above illustrating combinations of front-ends and back-ends.


## ONNX Workflow
For discussion purposes, we will investigate calling a text embedding model from Java, for example, in preparation for data ingestion to the open-source vector database [Milvus](https://milvus.io/). So, if we are to call our embedding or foundation model from Java, is it as simple as using the ONNX library on the corresponding model file? Yes, but we will need to procure files for both the model and the tokenizer encoder (and decoder for foundation models). We can produce these ourselves using Python offline, that is, before production, which we now explain.


## Exporting NN Models from Python
Let’s open a common text embedding model, `all-MiniLM-L6-v2`, from Python using HuggingFace’s sentence-transformers library. We will use the HF library indirectly via .txtai’s util library since we need a wrapper around sentence-transformers that also exports the pooling and normalization layers after the transformer function. (These layers take the context-dependent token embeddings, that is, the output of the transformer, and transform it into a single text embedding.)

```
from txtai.pipeline import HFOnnx

path = "sentence-transformers/all-MiniLM-L6-v2"
onnx_model = HFOnnx()
model = onnx_model(path, "pooling", "model.onnx", True)
```

We instruct the library to export `sentence-transformers/all-MiniLM-L6-v2` from the HuggingFace model hub as ONNX, specifying the task as text embedding and enabling model quantization. Calling `onnx_model()` will download the model from the model hub if it does not already exist locally, convert the three layers to ONNX, and combine their computational graphs.

Are we ready now to perform inference in Java? Not quite so fast. The model inputs a list of tokens (or a list of lists for more than one sample) corresponding to the tokenization of the text we wish to embed. Therefore, unless we can perform all tokenization before production time, we will need to run the tokenizer from within Java.

There are a few options for this. One involves either implementing or finding an implementation of the tokenizer for the model in question in Java or another language, and calling it from Java as a static or dynamically linked library. An easier solution is to convert the tokenizer to an ONNX file and use it from Java, just as we use the model ONNX file.

Plain ONNX, however, does not contain the necessary operations to implement the computational graph of a tokenizer. For this reason, Microsoft created a library to augment ONNX called ONNXRuntime-Extensions. It defines useful operations for all manner of data pre- and postprocessing, not only text tokenizers.

Here is how we export our tokenizer as an ONNX file:

```
from onnxruntime_extensions import gen_processing_models
from sentence_transformers import SentenceTransformer

embedding_model = SentenceTransformer('all-MiniLM-L6-v2')
tok_encode, _ = gen_processing_models(embedding_model.tokenizer, pre_kwargs={})

onnx_tokenizer_path = "tokenizer.onnx"
with open(tokenizer_path, "wb") as f:
  f.write(tok_encode.SerializeToString())
```

We have discarded the decoder of the tokenizer, since embedding sentences doesn’t require it. Now, we have two files: `tokenizer.onnx` for tokenizing text, and `model.onnx` for embedding strings of tokens.


## Model Inference in Java
Running our model from within Java is now trivial. Here are some of the important lines of code from the full example:

```java
// Imports required for Java/ONNX integration
import ai.onnxruntime.*;
import ai.onnxruntime.extensions.*;

…

// Set up inference sessions for tokenizer and model
var env = OrtEnvironment.getEnvironment();

var sess_opt = new OrtSession.SessionOptions();
sess_opt.registerCustomOpLibrary(OrtxPackage.getLibraryPath());

var tokenizer = env.createSession("app/tokenizer.onnx", sess_opt);
var model = env.createSession("app/model.onnx", sess_opt);

…

// Perform inference and extract text embeddings into native Java
var results = session.run(inputs).get("embeddings");
float[][] embeddings = (float[][]) results.get().getValue();
```

A full working example can be found in the resources section.


## Summary 
We have seen in this post how it is possible to export open-source models from HuggingFace’s model hub and use them directly from languages other than Python. We note, however, some caveats:

First, the ONNX libraries and runtime extensions have varying levels of feature support. It may not be possible to use all models across all languages until a future SDK update is released. The ONNX runtime libraries for Python, C++, Java, and JavaScript are the most comprehensive. 

Second, the HuggingFace hub contains pre-exported ONNX, but these models don’t include the final pooling and normalization layers. You should be aware of how `sentence-transformers` works if you intend to use `torch.onnx` directly.

Nevertheless, ONNX has the backing of major industry leaders and is on a trajectory to become a frictionless means of cross-platform Generative AI.


## Resources
- [Example onnx code in Python and Java](https://github.com/milvus-io/bootcamp/tree/master/tutorials/quickstart/onnx_example)

- <https://onnx.ai/>

- <https://onnxruntime.ai/>

- <https://onnxruntime.ai/docs/extensions/>

- <https://milvus.io/blog> 

- <https://github.com/milvus-io/bootcamp/tree/master> 


No Python, No Problem: Model Inference with ONNX in Java, or Any Other Language





For decades, `SELECT * FROM WHERE` has been the golden rule of database queries. Whether for reporting systems, financial analysis, or user behavior queries, we've grown accustomed to using structured language to precisely manipulate data. Even NoSQL, which once proclaimed an "anti-SQL revolution," eventually caved and introduced SQL support, acknowledging its seemingly irreplaceable position.

_But have you ever wondered: we've spent over 50 years teaching computers to speak human language, so why are we still forcing humans to speak "computer"?_

**Whether you like it or not, here's the truth: SQL is destined for decline in the era of AI.** It may still be used in legacy systems, but it's becoming increasingly irrelevant for modern AI applications. The AI revolution isn't just changing how we build software—it's making SQL obsolete, and most developers are too busy optimizing their JOINs to notice. 


![](https://assets.zilliz.com/why_ai_databases_don_t_need_SQL_2d12f615df.png)


## Natural Language: The New Interface for AI Databases 

The future of database interaction isn't about learning better SQL—it's about **abandoning syntax entirely**. 

Instead of wrestling with complex SQL queries, imagine simply saying:

_"Help me find users whose recent purchasing behavior is most similar to our top customers from last quarter."_

The system understands your intent and automatically decides:

- Should it query structured tables or perform a vector similarity search across user embeddings?

- Should it call external APIs to enrich the data?

- How should it rank and filter the results?

All completed automatically. No syntax. No debugging. No Stack Overflow searches for "how to do a window function with multiple CTEs." You're no longer a database "programmer"—you're having a conversation with an intelligent data system.

This isn't science fiction. According to Gartner predictions, by 2026, most enterprises will prioritize natural language as their primary query interface, with SQL changing from a "must-have" to an "optional" skill.

The transformation is already happening:

**✅ Zero syntax barriers:** Field names, table relationships, and query optimization become the system's problem, not yours

**✅ Unstructured data friendly:** Images, audio, and text become first-class query objects

**✅ Democratized access:** Operations teams, product managers, and analysts can directly query data as easily as your senior engineer


## Natural Language Is Just the Surface; AI Agents Are the Real Brain

Natural language queries are just the tip of the iceberg. The real breakthrough is [AI agents](https://zilliz.com/blog/what-exactly-are-ai-agents-why-openai-and-langchain-are-fighting-over-their-definition) that can reason about data like humans do.

Understanding human speech is step one. Understanding what you want and executing it efficiently—that's where the magic happens.

AI agents serve as the database's "brain," handling:

- **🤔 Intent understanding:** Determining which fields, databases, and indexes you actually need

- **⚙️ Strategy selection:** Choosing between structured filtering, vector similarity, or hybrid approaches

- **📦 Capability orchestration:** Executing APIs, triggering services, coordinating cross-system queries

- **🧾 Intelligent formatting:** Returning results you can immediately understand and act on

Here's what this looks like in practice. In the [Milvus vector database,](https://milvus.io/) a complex similarity search becomes trivial:

```
results = collection.search(query_vector, top_k=10, filter="is_active == true")
```


**One line. No JOINs. No subqueries. No performance tuning.** The [vector database](https://zilliz.com/learn/what-is-vector-database) handles semantic similarity while traditional filters handle exact matches. It's faster, simpler, and actually understands what you want.

This "API-first" approach naturally integrates with large language models' [Function Calling](https://zilliz.com/blog/function-calling-vs-mcp-vs-a2a-developers-guide-to-ai-agent-protocols) capabilities—faster execution, fewer errors, easier integration.


## Why SQL Falls Apart in the AI Era

SQL was designed for a structured world. However, the AI-driven future will be dominated by unstructured data, semantic understanding, and intelligent retrieval—everything SQL was never built to handle.

Modern applications are inundated with unstructured data, including text embeddings from language models, image vectors from computer vision systems, audio fingerprints from speech recognition, and multimodal representations that combine text, images, and metadata.

This data doesn't fit neatly into rows and columns—it exists as vector embeddings in high-dimensional semantic space, and SQL has absolutely no idea what to do with it.


### SQL + Vector: A Beautiful Idea That Executes Poorly

Desperate to stay relevant, traditional databases are bolting vector capabilities onto SQL. PostgreSQL added the `<->` operator for vector similarity search:

```
SELECT *
  FROM items
 ORDER BY embedding <-> query_vector
 LIMIT 10;
```


This looks clever, but it's fundamentally flawed. You're forcing vector operations through SQL parsers, query optimizers, and transaction systems designed for a completely different data model.

The performance penalty is brutal:

📊 **Real benchmark data**: Under identical conditions, purpose-built Milvus delivers 60% lower query latency and 4.5x higher throughput compared to PostgreSQL with pgvector.

Why such poor performance? Traditional databases create unnecessarily complex execution paths:

- **Parser overhead**: Vector queries get forced through SQL syntax validation

- **Optimizer confusion**: Query planners optimized for relational joins struggle with similarity searches

- **Storage inefficiency**: Vectors stored as BLOBs require constant encoding/decoding

- **Index mismatch**: B-trees and LSM structures are completely wrong for high-dimensional similarity search


### Relational vs AI/Vector Databases: Fundamentally Different Philosophies

The incompatibility runs deeper than performance. These are entirely different approaches to data:

| **Aspect**        | **SQL/Relational Databases**                             | **Vector/AI Databases**                                                            |
| ----------------- | -------------------------------------------------------- | ---------------------------------------------------------------------------------- |
| Data Model        | Structured fields (numbers, strings) in rows and columns | High-dimensional vector representations of unstructured data (text, images, audio) |
| Query Logic       | Exact matching + boolean operations                      | Similarity matching + semantic search                                              |
| Interface         | SQL                                                      | Natural language + Python APIs                                                     |
| Philosophy        | ACID compliance, perfect consistency                     | Optimized recall, semantic relevance, real-time performance                        |
| Index Strategy    | B+ trees, hash indexes etc.                              | HNSW, IVF, product quantization etc.                                               |
| Primary Use Cases | Transactions, reporting, analytics                       | Semantic search, multimodal search, recommendations, RAG systems, AI agents        |

Trying to make SQL work for vector operations is like using a screwdriver as a hammer—not technically impossible, but you're using the wrong tool for the job.


## Vector Databases: Purpose-Built for AI 

Vector databases like [Milvus](https://milvus.io/) and [Zilliz Cloud](https://zilliz.com/) aren't "SQL databases with vector features"—they're intelligent data systems designed from the ground up for AI-native applications.


### 1. Native Multimodal Support

Real AI applications don't just store text—they work with images, audio, video, and complex nested documents. Vector databases handle diverse data types and multi-vector structures like [ColBERT](https://zilliz.com/learn/explore-colbert-token-level-embedding-and-ranking-model-for-similarity-search) and [ColPALI](https://zilliz.com/blog/colpali-milvus-redefine-document-retrieval-with-vision-language-models), adapting to rich semantic representations from different AI models.


### 2. Agent-Friendly Architecture

Large language models excel at function calling, not SQL generation. Vector databases offer Python-first APIs that integrate seamlessly with AI agents, enabling the completion of complex operations, such as vector retrieval, filtering, reranking, and semantic highlighting, all within a single function call, without requiring a query language translation layer.


### 3. Semantic Intelligence Built-In

Vector databases don't just execute commands—**they understand intent.** Working with AI agents and other AI applications, they break free from literal keyword matching to achieve true semantic retrieval. They know not just "how to query" but "what you really want to find."


### 4. Optimized for Relevance, Not Just Speed

Like large language models, vector databases strike a balance between performance and recall. Through metadata filtering, [hybrid vector and full-text search](https://milvus.io/blog/get-started-with-hybrid-semantic-full-text-search-with-milvus-2-5.md), and reranking algorithms, they continuously improve result quality and relevancy, finding content that's actually valuable, not just fast to retrieve.


## The Future of Databases is Conversational

Vector databases represent a fundamental shift in how we think about data interaction. They're not replacing relational databases—they're purpose-built for AI workloads and addressing entirely different problems in an AI-first world.

Just as large language models didn't upgrade traditional rule engines but redefined human-machine interaction entirely, vector databases are redefining how we find and work with information.

We're transitioning from "languages written for machines to read" to "systems that understand human intent." Databases are evolving from rigid query executors to intelligent data agents that understand context and proactively surface insights.

The developers building AI applications today don't want to write SQL—they want to describe what they need and let intelligent systems figure out how to get it.

So next time you need to find something in your data, try a different approach. Don't write a query—just say what you're looking for. Your database might surprise you by actually understanding what you mean.

_And if it doesn't? Maybe it's time to upgrade your database, not your SQL skills._


Why AI Databases Don't Need SQL



**Probably every AI developer has built a RAG system that works perfectly... in their local environment.**

You've nailed the retrieval accuracy, optimized your vector database, and your demo runs like butter. Then you deploy to production and suddenly:

- Your 200ms local queries take 3 seconds for actual users

- Colleagues in different regions report completely different performance

- The embedding provider you chose for "best accuracy" becomes your biggest bottleneck

What happened? Here's the performance killer no one benchmarks: **embedding API latency**.

While MTEB rankings obsess over recall scores and model sizes, they ignore the metric your users feel—how long they wait before seeing any response. We tested every major embedding provider across real-world conditions and discovered extreme latency differences they'll make you question your entire provider selection strategy.

**_Spoiler: The most popular embedding APIs aren't the fastest. Geography matters more than model architecture. And sometimes a $20/month CPU beats a $200/month API call._**


## Why Embedding API Latency Is the Hidden Bottleneck in RAG

When building RAG systems, e-commerce search, or recommendation engines, embedding models are the core component that transforms text into vectors, enabling machines to understand semantics and perform efficient similarity searches. While we typically pre-compute embeddings for document libraries, user queries still require real-time embedding API calls to convert questions into vectors before retrieval, and this real-time latency often becomes the performance bottleneck in the entire application chain.

Popular embedding benchmarks like MTEB focus on recall accuracy or model size, often overlooking the crucial performance metric—API latency. Using Milvus's `TextEmbedding` Function, we conducted comprehensive real-world tests on major embedding service providers in North America and Asia. 

Embedding latency manifests at two critical stages:


### Query-Time Impact

In a typical RAG workflow, when a user asks a question, the system must:

- Convert the query to a vector via an embedding API call

- Search for similar vectors in Milvus

- Feed results and the original question to an LLM

- Generate and return the answer

Many developers assume the LLM's answer generation is the slowest part. However, many LLMs’ streaming output capability creates an illusion of speed—you see the first token quickly. In reality, if your embedding API call takes hundreds of milliseconds or even seconds, it becomes the first—and most noticeable—bottleneck in your response chain.


### Data Ingestion Impact

Whether building an index from scratch or performing routine updates, bulk ingestion requires vectorizing thousands or millions of text chunks. If each embedding call experiences high latency, your entire data pipeline slows dramatically, delaying product releases and knowledge base updates.

Both situations make embedding API latency a non-negotiable performance metric for production RAG systems.


## Measuring Real-World Embedding API Latency with Milvus

Milvus is an open-source, high-performance vector database that offers a new `TextEmbedding` Function interface. This feature directly integrates popular embedding models from OpenAI, Cohere, AWS Bedrock, Google Vertex AI, Voyage AI, and many more providers into your data pipeline, streamlining your vector search pipeline with a single call.

Using this new function interface, we tested and benchmarked popular embedding APIs from well-known providers like OpenAI and Cohere, as well as others like AliCloud and SiliconFlow, measuring their end-to-end latency in realistic deployment scenarios.

Our comprehensive test suite covered various model configurations:


| **Provider**     | **Model**                             | **Dimensions** |                  
| ---------------- | ------------------------------------- | -------------- |
| OpenAI           | text-embedding-ada-002                | 1536           |
| OpenAI           | text-embedding-3-small                | 1536           |
| OpenAI           | text-embedding-3-large                | 3072           |
| AWS Bedrock      | amazon.titan-embed-text-v2:0          | 1024           |
| Google Vertex AI | text-embedding-005                    | 768            |
| Google Vertex AI | text-multilingual-embedding-002       | 768            |
| VoyageAI         | voyage-3-large                        | 1024           |
| VoyageAI         | voyage-3                              | 1024           |
| VoyageAI         | voyage-3-lite                         | 512            |
| VoyageAI         | voyage-code-3                         | 1024           |
| Cohere           | embed-english-v3.0                    | 1024           |
| Cohere           | embed-multilingual-v3.0               | 1024           |
| Cohere           | embed-english-light-v3.0              | 384            |
| Cohere           | embed-multilingual-light-v3.0         | 384            |
| Aliyun Dashscope | text-embedding-v1                     | 1536           |
| Aliyun Dashscope | text-embedding-v2                     | 1536           |
| Aliyun Dashscope | text-embedding-v3                     | 1024           |
| Siliconflow      | BAAI/bge-large-zh-v1.5                | 1024           |
| Siliconflow      | BAAI/bge-large-en-v1.5                | 1024           |
| Siliconflow      | netease-youdao/bce-embedding-base_v1  | 768            |
| Siliconflow      | BAAI/bge-m3                           | 1024           |
| Siliconflow      | Pro/BAAI/bge-m3                       | 1024           |
| TEI              | BAAI/bge-base-en-v1.5                 | 768            |


## 7 Key Findings from Our Benchmarking Results 

We tested leading embedding models under varying batch sizes, token lengths, and network conditions, measuring median latency across all scenarios. The results uncover key insights that could reshape how you choose and optimize embedding APIs. Let’s take a look.


### 1. Global Network Effects Are More Significant Than You Think

The network environment is perhaps the most critical factor affecting embedding API performance. The same embedding API service provider can perform dramatically differently across network environments. 

![](https://assets.zilliz.com/latency_in_Asia_vs_in_US_cb4b5a425a.png)

When your application is deployed in Asia and accesses services like OpenAI, Cohere, or VoyageAI deployed in North America, network latency increases significantly. Our real-world tests show API call latency universally increased by **3 to 4 times**!

Conversely, when your application is deployed in North America and accesses Asian services like AliCloud Dashscope or SiliconFlow, performance degradation is even more severe. SiliconFlow, in particular, showed latency increases of **nearly 100 times** in cross-region scenarios!

This means you must always select embedding providers based on your deployment location and user geography—performance claims without network context are meaningless.


### 2. Model Performance Rankings Reveal Surprising Results

Our comprehensive latency testing revealed clear performance hierarchies:

- **North America-based Models (median latency)**: Cohere > Google Vertex AI > VoyageAI > OpenAI > AWS Bedrock

- **Asia-based Models (median latency)**: SiliconFlow > AliCloud Dashscope

These rankings challenge conventional wisdom about provider selection. 

![](https://assets.zilliz.com/median_latency_with_batch_size_1_ef83bec9c8.png)

![](https://assets.zilliz.com/median_latency_with_batch_size_10_0d4e52566f.png)

![](https://assets.zilliz.com/all_model_latency_vs_token_length_when_batch_size_is_10_537516cc1c.png)

![](https://assets.zilliz.com/all_model_latency_vstoken_lengthwhen_batch_size_is_10_4dcf0d549a.png)

Note: Due to the significant impact of network environment and server geographic regions on real-time embedding API latency, we compared North America and Asia-based model latencies separately.


### 3. Model Size Impact Varies Dramatically by Provider

We observed a general trend where larger models have higher latency than standard models, which have higher latency than smaller/lite models. However, this pattern wasn't universal and revealed important insights about backend architecture. For example: 

- **Cohere and OpenAI** showed minimal performance gaps between model sizes

- **VoyageAI** exhibited clear performance differences based on model size

![](https://assets.zilliz.com/Model_Size_Impact_Varies_Dramatically_by_Provider_1_f9eaf2be26.png)


![](https://assets.zilliz.com/Model_Size_Impact_Varies_Dramatically_by_Provider_2_cf4d72d1ad.png)


![](https://assets.zilliz.com/Model_Size_Impact_Varies_Dramatically_by_Provider_3_5e0c8d890b.png)


This indicates that API response time depends on multiple factors beyond model architecture, including backend batching strategies, request handling optimization, and provider-specific infrastructure. The lesson is clear: _don't trust model size or release date as reliable performance indicators—always test in your own deployment environment._


### 4. Token Length and Batch Size Create Complex Trade-offs

Depending on your backend implementation, especially your batching strategy. Token length may have little impact on latency until batch sizes grow. Our testing revealed some clear patterns:

- **OpenAI's latency** remained fairly consistent between small and large batches, suggesting generous backend batching capabilities

- **VoyageAI** showed clear token-length effects, implying minimal backend batching optimization

Larger batch sizes increase absolute latency but improve overall throughput. In our tests, moving from batch=1 to batch=10 increased latency by 2×-5×, while substantially boosting total throughput. This represents a critical optimization opportunity for bulk processing workflows where you can trade individual request latency for dramatically improved overall system throughput.

![](https://assets.zilliz.com/Going_from_batch_1_to_10_latency_increased_2_5_9811536a3c.png)

Going from batch=1 to 10, latency increased 2×–5×


### 5. API Reliability Introduces Production Risk

We observed significant variability in latency, particularly with OpenAI and VoyageAI, introducing unpredictability into production systems. 

![](https://assets.zilliz.com/Latency_variance_when_batch_1_d9cd88fb73.png)

Latency variance when batch=1

![](https://assets.zilliz.com/Latency_variance_when_batch_10_5efc33bf4e.png)

Latency variance when batch=10

While our testing focused primarily on latency, relying on any external API introduces inherent failure risks, including network fluctuations, provider rate limiting, and service outages. Without clear SLAs from providers, developers should implement robust error handling strategies, including retries, timeouts, and circuit breakers to maintain system reliability in production environments.


### 6. Local Inference Can Be Surprisingly Competitive

Our tests also revealed that deploying mid-sized embedding models locally can offer performance comparable to cloud APIs—a crucial finding for budget-conscious or latency-sensitive applications.

For example, deploying the open-source `bge-base-en-v1.5` via TEI (Text Embeddings Inference) on a modest 4c8g CPU matched SiliconFlow's latency performance, providing an affordable local-inference alternative. This finding is particularly significant for individual developers and small teams who may lack enterprise-grade GPU resources but still need high-performance embedding capabilities.

![](https://assets.zilliz.com/TEI_Latency_2f09be1ef0.png)

TEI Latency

### 7. Milvus Overhead Is Negligible

Since we used Milvus to test embedding API latency, we validated that the additional overhead introduced by Milvus's TextEmbedding Function is minimal and virtually negligible. Our measurements show Milvus operations add only 20-40ms in total while embedding API calls take hundreds of milliseconds to several seconds, meaning Milvus adds less than 5% overhead to the total operation time. The performance bottleneck primarily lies in network transmission and the embedding API service providers' processing capabilities, not in the Milvus server layer.

## Tips: How to Optimize Your RAG Embedding Performance

Based on our benchmarks, we recommend the following strategies to optimize your RAG system's embedding performance:


### 1. Always Localize Your Testing

Don't trust any generic benchmark reports (including this one!). You should always test models within your actual deployment environment rather than relying solely on published benchmarks. Network conditions, geographic proximity, and infrastructure differences can dramatically impact real-world performance.


### 2. Geo-Match Your Providers Strategically

- **For North American deployments**: Consider Cohere, VoyageAI, OpenAI/Azure, or GCP Vertex AI—and always conduct your own performance validation

- **For Asian deployments**: Seriously consider Asian model providers such as AliCloud Dashscope or SiliconFlow, which offer better regional performance

- **For global audiences**: Implement multi-region routing or select providers with globally distributed infrastructure to minimize cross-region latency penalties


### 3. Question Default Provider Choices

OpenAI's embedding models are so popular that many enterprises and developers choose them as default options. However, our tests revealed that OpenAI's latency and stability were average at best, despite its market popularity. Challenge assumptions about "best" providers with your own rigorous benchmarks—popularity doesn't always correlate with optimal performance for your specific use case.


### 4. Optimize Batch and Chunk Configurations

One configuration doesn't fit all models or use cases. The optimal batch size and chunk length vary significantly between providers due to different backend architectures and batching strategies. Experiment systematically with different configurations to find your optimal performance point, considering the throughput vs. latency trade-offs for your specific application requirements.


### 5. Implement Strategic Caching

For high-frequency queries, cache both the query text and its generated embeddings (using solutions like Redis). Subsequent identical queries can directly hit the cache, reducing latency to milliseconds. This represents one of the most cost-effective and impactful query latency optimization techniques. 


### 6. Consider Local Inference Deployment

If you have extremely high requirements for data ingestion latency, query latency, and data privacy, or if API call costs are prohibitive, consider deploying embedding models locally for inference. Standard API plans often come with QPS limitations, unstable latency, and lack of SLA guarantees—constraints that can be problematic for production environments.


For many individual developers or small teams, the lack of enterprise-grade GPUs is a barrier to local deployment of high-performance embedding models. However, this doesn't mean abandoning local inference entirely. With high-performance inference engines like [Hugging Face's text-embeddings-inference](https://github.com/huggingface/text-embeddings-inference), even running small to medium-sized embedding models on a CPU can achieve decent performance that may outperform high-latency API calls, especially for large-scale offline embedding generation. 


This approach requires careful consideration of trade-offs between cost, performance, and maintenance complexity.


## How Milvus Simplifies Your Embedding Workflow

As mentioned, Milvus is not just a high-performance vector database—it also offers a convenient embedding function interface that seamlessly integrates with popular embedding models from various providers such as OpenAI, Cohere, AWS Bedrock, Google Vertex AI, Voyage AI, and more across the world into your vector search pipeline.

Milvus goes beyond vector storage and retrieval with features that streamline embedding integration:

- **Efficient Vector Management**: As a high-performance database built for massive vector collections, Milvus offers reliable storage, flexible indexing options (HNSW, IVF, RaBitQ, DiskANN, and more), and fast, accurate retrieval capabilities.

- **Streamlined Provider Switching**: Milvus offers a `TextEmbedding` Function interface, allowing you to configure the function with your API keys, switch providers or models instantly, and measure real-world performance without complex SDK integration.

- **End-to-End Data Pipelines**: Call `insert()` with raw text, and Milvus automatically embeds and stores vectors in one operation, dramatically simplifying your data pipeline code.

- **Text-to-Results in One Call**: Call `search()` with text queries, and Milvus handles embedding, searching, and returning results—all in a single API call.

- **Provider-Agnostic Integration**: Milvus abstracts provider implementation details; just configure your Function and API key once, and you're ready to go.

- **Open-Source Ecosystem Compatibility**: Whether you generate embeddings via our built-in `TextEmbedding` Function, local inference, or another method, Milvus provides unified storage and retrieval capabilities.

This creates a streamlined "Data-In, Insight-Out" experience where Milvus handles vector generation internally, making your application code more straightforward and maintainable.


## Conclusion: The Performance Truth Your RAG System Needs

The silent killer of RAG performance isn't where most developers are looking. While teams pour resources into prompt engineering and LLM optimization, embedding API latency quietly sabotages user experience with delays that can be 100x worse than expected. Our comprehensive benchmarks expose the harsh reality: popular doesn't mean performant, geography matters more than algorithm choice in many cases, and local inference sometimes beats expensive cloud APIs.

These findings highlight a crucial blind spot in RAG optimization. Cross-region latency penalties, unexpected provider performance rankings, and the surprising competitiveness of local inference aren't edge cases—they're production realities affecting real applications. Understanding and measuring embedding API performance is essential for delivering responsive user experiences.

Your embedding provider choice is one critical piece of your RAG performance puzzle. By testing in your actual deployment environment, selecting geographically appropriate providers, and considering alternatives like local inference, you can eliminate a major source of user-facing delays and build truly responsive AI applications.

For more details on how we did this benchmarking, check [this notebook](https://github.com/zhuwenxing/text-embedding-bench).


We Benchmarked 20+ Embedding APIs with Milvus: 7 Insights That Will Surprise You



The key operation of [vector databases](https://milvus.io/blog/what-is-a-vector-database.md) is _similarity search_, which involves finding the nearest neighbors in the database to a query vector, for example, by Euclidean distance. A naive method would calculate the distance from the query vector to every vector stored in the database and take the top-K closest. However, this clearly does not scale as the size of the database grows. In practice, a naive similarity search is practical only for databases with fewer than around 1 million vectors. How are we to scale our search to the 10s and 100s of millions, or even to the billions of vectors?

![](https://assets.zilliz.com/Figure_Descending_a_hierarchy_of_vector_search_indices_cf9fb8060a.png)

_Figure: Descending a hierarchy of vector search indices_

Many algorithms and data structures have been developed to scale similarity search in high-dimensional vector spaces to sub-linear time complexity. In this article, we’ll explain and implement a popular and effective method called Hierarchical Navigable Small Worlds (HNSW), which is frequently the default choice for medium-sized vector datasets. It belongs to the family of search methods that construct a graph over the vectors, where vertices denote vectors and edges denote similarity between them. Search is performed by navigating the graph, in the simplest case, greedily traversing to the neighbor of the current node that is closest to the query and repeating until a local minimum is reached.

We will explain in more detail how the search graph is constructed, how the graph enables search, and at the end, link to an HNSW implementation, by yours truly, in simple Python.


## Navigable Small Worlds

![](https://assets.zilliz.com/Figure_NSW_graph_created_from_100_randomly_located_2_D_points_3ffccbd6a7.jpg)

_Figure: NSW graph created from 100 randomly located 2D points._

As mentioned, HNSW constructs a search graph offline before we can perform a query. The algorithm builds on top of prior work, a method called Navigable Small Worlds (NSW). We will explain NSW first and it will then be trivial to go from there to _Hierarchical_ NSW. The illustration above is of a constructed search graph for NSW over 2-dimensional vectors. In all examples below, we restrict ourselves to 2-dimensional vectors so as to be able to visualize them.


## Constructing the Graph

An NSW is a graph where the vertices represent vectors and the edges are constructed heuristically from the similarity between vectors so that most vectors are reachable from anywhere via a small number of hops. This is the so-called “small world” property that permits quick navigation. See the above figure.

The graph is initialized to be empty. We iterate through the vectors, adding each to the graph in turn. For each vector, starting at a random entry node, we greedily find the closest R nodes reachable from the entry point _in the graph so far constructed_. These R nodes are then connected to a new node representing the vector being inserted, optionally pruning any neighboring nodes that now have more than R neighbors. Repeating this process for all vectors will result in the NSW graph. See the above illustration visualizing the algorithm, and refer to the resources at the end of the article for a theoretical analysis of the properties of a graph constructed like this.


## Searching the Graph

We have already seen the search algorithm from its use in graph construction. In this case, however, the query node is provided by the user, rather than being one for insertion into the graph. Starting from a random entry note, we greedily navigate to its neighbor closest to the query, maintaining a dynamic set of the closest vectors encountered so far. See the illustration above. Note that we can enhance search accuracy by initiating searches from multiple random entry points and aggregating the results, along with considering multiple neighbors at each step. However, these improvements come at the cost of increased latency.


# Adding Hierarchy

![](https://assets.zilliz.com/adding_hierarchy_0101234812.png)

So far, we have described the NSW algorithm and data structure that can help us scale up search in high-dimensional space. Nonetheless, the method suffers serious shortcomings, including failure in low dimensions, slow search convergence, and a tendency to be trapped in local minima.

The authors of HNSW fix these shortcomings with three modifications to NSW:

- Explicit selection of the entry nodes during construction and search;

- Separation of edges by different scales; and, 

- Use of an advanced heuristic to select the neighbors.

The first two are realized with a simple idea: building _a hierarchy of search graphs_. Instead of a single graph, as in NSW, HNSW constructs a hierarchy of graphs. Each graph, or layer, is individually searched in the same way as NSW. The top layer, which is searched first, contains very few nodes, and deeper layers progressively include more and more nodes, with the bottom layer including all nodes. This means that the top layers contain longer hops across the vector space, permitting a sort of course-to-fine search. See above for an illustration.


## Constructing the Graph

The construction algorithm works as follows: we fix a number of layers, _L_, in advance. The value l=1 will correspond to the coarsest layer, where search begins, and l=L will correspond to the densest layer, where search finishes. We iterate through each vector to be inserted and sample an insertion layer following a truncated [geometric distribution](https://en.wikipedia.org/wiki/Geometric_distribution) (either rejecting _l > L_ or setting _l’ =_ min_(l, L)_). Say we sample _1 < l < L_ for the current vector. We perform a greedy search on the top layer, L, until we reach its local minimum. Then, we follow an edge from the local minimum in the _L_th layer to the corresponding vector in the _(L-1)_th layer and use it as the entry point to greedily search the _(L-1)_th layer.

This process is repeated until we reach the _l_th layer. We then begin to create nodes for the vector to be inserted, connecting it to its closest neighbors found by greedy search in the _l_th layer so far constructed, navigating to the _(l-1)_th layer and repeating until we have inserted the vector into the _1_st layer. An animation above makes this clear

We can see that this hierarchical graph construction method uses a clever explicit selection of the insertion node for each vector. We search the layers above the insertion layer constructed so far, searching efficiently from course-to-fine distances. Relatedly, the method separates links by different scales in each layer: the top layer affords long-scale hops across the search space, with the scale diminishing down to the bottom layer. Both of these modifications help avoid being trapped in sub-optimal minima and hasten search convergence at the cost of additional memory.


## Searching the Graph

The search procedure works much like the inner graph construction step. Starting from the top layer, we greedily navigate to the node or nodes closest to the query. Then we follow that node(s) down to the next layer and repeat the process. Our answer is obtained by the list of _R_ closest neighbors in the bottom layer, as illustrated by the animation above this.


## Conclusion

Vector databases like Milvus provide highly optimized and tuned implementations of HNSW, and it is often the best default search index for datasets that fit in memory.

We have sketched a high-level overview of how and why HNSW works, preferring visualizations and intuition over theory and mathematics. Consequently, we have omitted an exact description of the construction and search algorithms[[Malkov and Yashushin, 2016](https://arxiv.org/abs/1603.09320); Alg 1-3], analysis of search and construction complexity [[Malkov and Yashushin, 2016](https://arxiv.org/abs/1603.09320); §4.2], and less essential details like a heuristic for more effectively choosing neighbor nodes during construction [[Malkov and Yashushin, 2016](https://arxiv.org/abs/1603.09320); Alg 5]. Moreover, we have omitted discussion of the algorithm’s hyperparameters, their meaning and how they affect the latency/speed/memory trade-off [[Malkov and Yashushin, 2016](https://arxiv.org/abs/1603.09320); §4.1]. An understanding of this is important for using HNSW in practice.

The resources below contain further reading on these topics and a full Python pedagogical implementation (written by myself) for NSW and HNSW, including code to produce the animations in this article.


# Resources

- GitHub: “[HNSW-Illustrated: A small implementation of Hierarchical Navigable Small Worlds (HNSW), a vector search algorithm, for learning purposes](https://github.com/stefanwebb/hnsw-illustrated)”

- [HNSW | Milvus Documentation](https://milvus.io/docs/hnsw.md#HNSW)

- [Understanding Hierarchical Navigable Small Worlds (HNSW) - Zilliz Learn](https://zilliz.com/learn/hierarchical-navigable-small-worlds-HNSW)

- HNSW paper: “[Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs](https://arxiv.org/abs/1603.09320)”

- NSW paper: “[Approximate nearest neighbor algorithm based on navigable small world graphs](https://publications.hse.ru/pubs/share/folder/x5p6h7thif/128296059.pdf)”


Understand HNSW for Vector Search


Understanding Hierarchical Navigable Small Worlds (HNSW) for Vector Search



## What is DiskANN?

[DiskANN](https://github.com/microsoft/DiskANN) represents a paradigm-shifting approach to [vector similarity search](https://zilliz.com/learn/vector-similarity-search). Before that, most vector index types like HNSW rely heavily on RAM to achieve low latency and high recall. While effective for moderate-sized datasets, this approach becomes prohibitively expensive and less scalable as data volumes grow. DiskANN offers a cost-effective alternative by leveraging SSDs to store the index, significantly reducing memory requirements.

DiskANN employs a flat graph structure optimized for disk access, allowing it to handle billion-scale datasets with a fraction of the memory footprint required by in-memory methods. For instance, DiskANN can index up to a billion vectors while achieving 95% search accuracy with 5ms latencies, whereas RAM-based algorithms peak at 100–200 million points for similar performance.


![](https://assets.zilliz.com/Vector_indexing_and_search_workflow_with_Disk_ANN_41cdf33652.png)

_Figure 1: Vector indexing and search workflow with DiskANN_

Although DiskANN may introduce slightly higher latency compared to RAM-based approaches, the trade-off is often acceptable given the substantial cost savings and scalability benefits. DiskANN is particularly suitable for applications requiring large-scale vector search on commodity hardware.

This article will explain the clever methods DiskANN has to leverage SSD in addition to RAM and reduce costly SSD reads. 


## How Does DiskANN Work?

DiskANN is a graph-based vector search method in the same family of methods as HNSW. We first construct a search graph where the nodes correspond to vectors (or groups of vectors), and edges denote that a pair of vectors is “relatively close” in some sense. A typical search randomly chooses an “entry node”, and navigates to its neighbor closest to the query, repeating in a greedy fashion until a local minimum is reached.

Graph-based indexing frameworks differ primarily in how they construct the search graph and perform search. And in this section, we will do a technical deep dive into the innovations of DiskANN for these steps and how they permit low-latency, low-memory performance. (See the above figure for a summary.)


### An Overview

We assume that the user has generated a set of document vector embeddings. The first step is to cluster the embeddings. A search graph for each cluster is constructed separately using the Vamana algorithm (explained in the next section), and the results are merged into a single graph. _The divide-and-conquer strategy for creating the final search graph significantly reduces memory usage without too greatly affecting search latency or recall._

![](https://assets.zilliz.com/How_Disk_ANN_stores_vector_index_across_RAM_and_SSD_d6564b087f.jpg)

_Figure 2: How DiskANN stores vector index across RAM and SSD_

Having produced the global search graph, it's stored on SSD together with the full-precision vector embeddings. A major challenge is to finish the search within a bounded number of SSD reads, since SSD access is expensive relative to RAM access. So, a few clever tricks are used to restrict the number of reads:

First, the Vamana algorithm incentivizes shorter paths between close nodes while capping the maximum number of neighbors of a node. Second, a fixed-size data structure is used to store each node’s embedding and its neighbors (see the above figure). What this means is that we can address a node’s metadata by simply multiplying the data structure size by the node’s index and using this as an offset while simultaneously fetching the node’s embedding. Third, due to how SSD works, we can fetch multiple nodes per read request - in our case, the neighbor nodes - reducing the number of read requests further.

Separately, we compress the embeddings using product quantization and store them in RAM. In doing so, we can fit billions-scale vector datasets into a memory that is feasible on a single machine for quickly calculating _approximate vector similarities_ without disk reads. This provides guidance for reducing the number of neighbor nodes to access next on the SSD. Importantly, however, the search decisions are made using the _exact vector similarities_, with the full embeddings retrieved from SSD, which ensures higher recall. To emphasize, there is an initial phase of search using quantized embeddings in memory, and a subsequent search on a smaller subset reading from SSD.

In this description, we have glossed over two important albeit involved steps: how to construct the graph, and how to search the graph - the two steps indicated by the red boxes above. Let’s examine each of these in turn.


### “Vamana” Graph Construction

![](https://assets.zilliz.com/Vamana_Graph_Construction_ecb4dab839.jpg)

_Figure: “Vamana” Graph Construction_

The DiskANN authors develop a novel method for constructing the search graph, which they call the Vamana algorithm. It initializes the search graph by randomly adding O(N) edges. This will result in a graph that is “well-connected”, although without any guarantees on greedy search convergence. It then prunes and reconnects the edges in an intelligent way to ensure there are sufficient long-range connections (see above figure). Allow us to elaborate:


#### Initialization

The search graph is initialized to a random directed graph where each node has R out-neighbors. We also calculate the medoid of the graph, that is, the point that has the minimum average distance to all other points. You can think of this as analogous to a centroid that is a member of the set of nodes.


#### Search for Candidates

After initialization, we iterate over the nodes, performing both adding and removing edges at each step. First, we run a search algorithm on the selected node, p, to generate a list of candidates. The search algorithm starts at the medoid and greedily navigates closer and closer to the selected node, adding the out-neighbors of the closest node found so far at each step. The list of L found nodes closest to p is returned. (If you’re not familiar with the concept, the medoid of a graph is the point that has the minimum average distance to all other points and acts as an analog of a centroid for graphs.)


#### Pruning and Adding Edges

The node’s candidate neighbors are sorted by distance, and for each candidate, the algorithm checks whether it is “too close” in direction to an already chosen neighbor. If so, it's pruned. This promotes angular diversity among neighbors, which empirically leads to better navigation properties. In practice, this means that a search starting from a random node can more quickly reach any target node by exploring a sparse set of long-range and local links.

After pruning edges, edges along the greedy search path to p are added. Two passes of pruning are performed, varying the distance threshold for pruning so that long-term edges are added in the second pass.


## What’s Next?

Subsequent work has been built upon DiskANN for additional improvements. One noteworthy example, known as [FreshDiskANN](https://arxiv.org/abs/2105.09613), modifies the method to allow for the easy updating of the index after construction. This search index, which provides an excellent tradeoff between performance criteria, is available in the [Milvus](https://milvus.io/docs/overview.md) vector database as the `DISKANN` index type. 

```python
# Prepare index parameters
index_params = client.prepare_index_params()

# Add DiskANN index
index_params.add_index(
    field_name="vector",
    index_type="DISKANN",
    metric_type="COSINE"
)

# Create collection with index
client.create_collection(
    collection_name="diskann_collection",
    schema=schema,
    index_params=index_params
)
```

You can even tune the DiskANN parameters, such as `MaxDegree` and  `BeamWidthRatio`: see [the documentation page](https://milvus.io/docs/disk_index.md#On-disk-Index) for more details.


## Resources

- [Milvus Documentation on using DiskANN](https://milvus.io/docs/disk_index.md#On-disk-Index)

- [“DiskANN: Fast Accurate Billion-point Nearest Neighbor Search on a Single Node”](https://suhasjs.github.io/files/diskann_neurips19.pdf)

- [“FreshDiskANN: A Fast and Accurate Graph-Based ANN Index for Streaming Similarity Search”](https://arxiv.org/abs/2105.09613)


DiskANN Explained


Throughout this week, we've shared a range of exciting innovations in Milvus that push the boundaries of vector database technology: 

- [Vector Search in the Real World: How to Filter Efficiently Without Killing Recall ](https://milvus.io/blog/how-to-filter-efficiently-without-killing-recall.md)

- [Bring Vector Compression to the Extreme: How Milvus Serves 3× More Queries with RaBitQ](https://milvus.io/blog/bring-vector-compression-to-the-extreme-how-milvus-serves-3%C3%97-more-queries-with-rabitq.md)

- [Benchmarks Lie — Vector DBs Deserve a Real Test ](https://milvus.io/blog/benchmarks-lie-vector-dbs-deserve-a-real-test.md)

- [We Replaced Kafka/Pulsar with a Woodpecker for Milvus ](https://milvus.io/blog/we-replaced-kafka-pulsar-with-a-woodpecker-for-milvus.md)

- [MinHash LSH in Milvus: The Secret Weapon for Fighting Duplicates in LLM Training Data ](https://milvus.io/blog/minhash-lsh-in-milvus-the-secret-weapon-for-fighting-duplicates-in-llm-training-data.md)

Now, as we wrap up our Milvus Week series, I'm excited to give you a sneak peek of what's coming in Milvus 2.6—a crucial milestone in our 2025 product roadmap that's currently in development, and how these improvements will transform AI-powered search. This upcoming release brings together all these innovations and more across three critical fronts: **cost-efficiency optimization**, **advanced search capabilities**, and **a new architecture** that pushes vector search beyond 10 billion vector scale.

Let's dive into some of the key improvements you can expect when Milvus 2.6 arrives this June, starting with what might be the most immediately impactful: dramatic reductions in memory usage and cost, and ultra-fast performance. 

![](https://assets.zilliz.com/Milvus_2_6_Preview_72_Memory_Reduction_Without_Compromising_Recall_and_4x_Faster_Than_Elasticsearch_c607b644f1.png)


## Cost-Reduction: Slash Memory Usage While Boosting Performance

Relying on expensive memory presents one of the biggest obstacles to scaling vector search to billions of records. Milvus 2.6 will introduce several key optimizations that dramatically lower your infrastructure costs while improving performance.


### RaBitQ 1-bit Quantization: 72% Memory Reduction with 4× QPS and No Recall Loss

Memory consumption has long been the Achilles' heel of large-scale vector databases. While vector quantization isn't new, most existing approaches sacrifice too much search quality for memory savings. Milvus 2.6 will tackle this challenge head-on by introducing[ RaBitQ 1-bit quantization](https://milvus.io/blog/bring-vector-compression-to-the-extreme-how-milvus-serves-3%C3%97-more-queries-with-rabitq.md) in production environments.

What makes our implementation special is the adjustable Refine optimization capability we're building. By implementing a primary index with RaBitQ quantization plus SQ4/SQ6/SQ8 Refine options, we've achieved an optimal balance between memory usage and search quality (~95% recall).

Our preliminary benchmarks reveal promising results:

| **Performance** **Metric** | **Traditional IVF_FLAT** | **RaBitQ (1-bit) Only** | **RaBitQ (1-bit) + SQ8 Refine** |
| -------------------------- | ------------------------- | ----------------------- | ------------------------------- |
| Memory Footprint           | 100% (baseline)           | 3% (97% reduction)      | 28% (72% reduction)             |
| Recall Quality             | 95.2%                     | 76.3%                   | 94.9%                           |
| Query Throughput (QPS)     | 236                       | 648 (2.7× faster)       | 946 (4× faster)                 |

_Table: VectorDBBench evaluation with 1M vectors of 768 dimensions, tested on AWS m6id.2xlarge_

The real breakthrough here isn't just the memory reduction, but achieving this while simultaneously delivering a 4× throughput improvement without compromising accuracy. This means you'll be able to serve the same workload with 75% fewer servers or handle 4× more traffic on your existing infrastructure.

For enterprise users using fully managed Milvus on[ Zilliz Cloud](https://zilliz.com/cloud), we're developing automated configuration profiles that will dynamically adjust RaBitQ parameters based on your specific workload characteristics and precision requirements.


### 400% Faster Full-text Search Than Elasticsearch

[Full-text search](https://milvus.io/blog/full-text-search-in-milvus-what-is-under-the-hood.md) capabilities in vector databases have become essential for building hybrid retrieval systems. Since introducing BM25 in [Milvus 2.5](https://milvus.io/blog/introduce-milvus-2-5-full-text-search-powerful-metadata-filtering-and-more.md), we've received enthusiastic feedback—along with requests for better performance at scale.

Milvus 2.6 will deliver substantial performance gains on BM25. Our testing on the BEIR dataset shows 3-4× higher throughput than Elasticsearch with equivalent recall rates. For some workloads, the improvement reaches up to 7× higher QPS.

![](https://assets.zilliz.com/Figure_Milvus_vs_Elasticsearch_on_throughput_140b7c1b06.png)

Figure: Milvus vs. Elasticsearch on throughput 


### JSON Path Index: 99% Lower Latency for Complex Filtering

Modern AI applications rarely rely on vector similarity alone—they almost always combine vector search with metadata filtering. As these filtering conditions become more complex (especially with nested JSON objects), query performance can deteriorate rapidly.

Milvus 2.6 will introduce a targeted indexing mechanism for nested JSON paths that allows you to create indexes on specific paths (e.g., `$meta user_info.location`) within JSON fields. Instead of scanning entire objects, Milvus will directly look up values from pre-built indexes.

In our evaluation with 100 M+ records, JSON Path Index reduced filter latency from **140ms** (P99: 480ms) to just **1.5ms** (P99: 10ms)—a 99% reduction that will transform previously impractical queries into instant responses.

This feature will be particularly valuable for:

- Recommendation systems with complex user attribute filtering

- RAG applications that filter documents by various labels


## Next-Generation Search: From Basic Vector Similarity to Production-Grade Retrieval

Vector search alone isn't enough for modern AI applications. Users demand the precision of traditional information retrieval combined with the semantic understanding of vector embeddings. Milvus 2.6 will introduce several advanced search features that bridge this gap.


### Better Full-text Search with Multi-language Analyzer

Full-text search is highly language-dependent.. Milvus 2.6 will introduce a completely revamped text analysis pipeline with multi-language support:

- `RUN_ANALYZER` syntax support for analyzer/tokenization configuration observability

- Lindera tokenizer for Asian languages like Japanese and Korean

- ICU tokenizer for comprehensive multilingual support

- Granular language configuration for defining language-specific tokenization rules

- Enhanced Jieba with support for custom dictionary integration

- Expanded filter options for more precise text processing

For global applications, this means better multilingual search without specialized per-language indexing or complex workarounds.


### Phrase Match: Capturing Semantic Nuance in Word Order

Word order conveys critical meaning distinctions that keyword search often misses. Try comparing "machine learning techniques" with "learning machine techniques"—same words, totally different meaning.

Milvus 2.6 will add **Phrase Match**, giving users more control over word order and proximity than full-text search or exact string match:

```
PHRASE_MATCH(field_name, phrase, slop)
```


The `slop` parameter will provide flexible control over word proximity—0 requires exact consecutive matches, while higher values allow for minor variations in phrasing.

This feature will be particularly valuable for:

- Legal document search where exact phrasing carries legal significance

- Technical content retrieval where term order distinguishes different concepts

- Patent databases where specific technical phrases must be matched precisely


### Time-Aware Decay Functions: Automatically Prioritize Fresh Content

Information value often diminishes with time. News articles, product releases, and social posts all become less relevant as they age, yet traditional search algorithms treat all content equally, regardless of timestamp.

Milvus 2.6 will introduce **Decay Functions** for time-aware ranking that automatically adjust relevance scores based on document age.

![](https://assets.zilliz.com/decay_function_210e65f9a0.png)

You'll be able to configure:

- **Function type**: Exponential (rapid decay), Gaussian (gradual decay), or Linear (constant decay)

- **Decay rate**: How quickly relevance diminishes over time

- **Origin point**: The reference timestamp for measuring time differences

This time-sensitive re-ranking will ensure that the most up-to-date and contextually relevant results appear first, which is crucial for news recommendation systems, e-commerce platforms, and social media feeds.


### Data in, Data Out: From Raw Text to Vector Search in One Step

One of the biggest developer pain points with vector databases has been the disconnect between raw data and vector embeddings. Milvus 2.6 will dramatically simplify this workflow with a new **Function** interface that integrates third-party embedding models directly into your data pipeline. This streamlines your vector search pipeline with one single call. 

Instead of pre-computing embeddings, you'll be able to:

1. **Insert raw data directly**: Submit text, images, or other content to Milvus

2. **Configure embedding providers for vectorization**: Milvus can connect to embedding model services like OpenAI, AWS Bedrock, Google Vertex AI, and Hugging Face.

3. **Query using natural language**: Search using text queries, not vector embeddings

This will create a streamlined "Data-In, Data-Out" experience where Milvus handles the vector generation internally, making your application code more straightforward.


## Architectural Evolution: Scaling to Hundreds of Billions of Vectors

A good database doesn’t just have great features, it must also deliver those features at scale, battle-tested in production. 

Milvus 2.6 will introduce a fundamental architectural change that enables cost-effective scaling to hundreds of billions of vectors. The highlight is a new hot-cold tiered storage architecture that intelligently manages data placement based on access patterns, automatically moving hot data to high-performance memory/SSD while placing cold data in more economical object storage. This approach can dramatically reduce costs while maintaining query performance where it matters most. 

Additionally, a new [Streaming Node](https://milvus.io/blog/we-replaced-kafka-pulsar-with-a-woodpecker-for-milvus.md#StreamingService-Built-for-Real-Time-Data-Flow) will enable real-time vector processing with direct integration to streaming platforms like Kafka and Pulsar and the newly created [Woodpecker](https://milvus.io/blog/we-replaced-kafka-pulsar-with-a-woodpecker-for-milvus.md), making new data searchable immediately without batch delays.


## Stay tuned for Milvus 2.6 

Milvus 2.6 is currently in active development and will be available this June. We're excited to bring you these breakthrough performance optimizations, advanced search capabilities, and a new architecture to help you build scalable AI applications at lower cost.

In the meantime, we welcome your feedback on these upcoming features. What excites you most? Which capabilities would have the most impact on your applications? Join the conversation in our[ Discord channel](https://discord.com/invite/8uyFbECzPX) or follow our progress on[ GitHub](https://github.com/milvus-io/milvus).

Want to be the first to know when Milvus 2.6 is released? Follow us on[ LinkedIn](https://www.linkedin.com/company/zilliz/) or[ X](https://twitter.com/milvusio) for the latest updates.


Milvus 2.6 Preview: 72% Memory Reduction Without Compromising Recall and 4x Faster Than Elasticsearch



Large Language Models (LLMs) have transformed the AI landscape with their ability to write code, create content, and solve complex problems. However, these powerful models require enormous amounts of high-quality data to fuel their training.

The challenge is that raw training data often contains significant redundancy. It's like teaching a child by repeating the same lessons over and over while skipping other important topics. A large AI company approached us with precisely this problem - they were building an ambitious new language model but struggled with deduplicating tens of billions of documents. Traditional matching methods couldn't scale to this volume, and specialized deduplication tools required massive computational resources, making them economically unviable.

To solve this problem, we introduced MinHash LSH (Locality Sensitive Hashing) indexing in Milvus 2.6. This article will explore how MinHash LSH efficiently solves the data deduplication problem for LLM training.


![](https://assets.zilliz.com/Chat_GPT_Image_May_16_2025_09_46_39_PM_1f3290ce5e.png)


## Data Deduplication: Why It Matters for LLM Training

High-quality, diverse data is essential for training powerful LLMs. When duplicate content appears in training data, it creates several significant problems:

- **Wasted Resources:** Redundant data increases training time, costs, and energy consumption.

- **Reduced Performance:** Models can overfit to repeated content, limiting their ability to generalize to new information.

- **Memorization Effect:** Duplicated content increases the chance of models memorizing and reproducing specific text verbatim. It could also potentially lead to privacy leaks or copyright issues.

- **Misleading Evaluations:** Duplicates between training and test sets can accidentally inflate performance metrics.

There are three main approaches to finding and removing duplicates:

- **Exact Matching:** Identifies identical duplicates through hashing.

- **Approximate Matching:** Finds near-duplicates using algorithms like MinHash LSH and Jaccard similarity.

- **Semantic Matching:** Identifies content with similar meaning using vector embeddings.

With pre-training corpora reaching terabytes or even petabytes, traditional exact matching methods like pairwise comparisons are computationally infeasible. Semantic deduplication adds significant overhead by using embedding models to generate vectors. We need more innovative approximate methods—like **MinHash LSH**—that balance recall and precision while keeping costs manageable, making large-scale deduplication practical.


## MinHash LSH: Efficiently Detecting Near-Duplicates in Massive Datasets

To find near-duplicates in an ocean of training data, we need an approximate matching algorithm that’s both efficient and accurate. MinHash LSH (Locality Sensitive Hashing) is a great tool for this goal. Let’s break down this seemingly complex term step by step. 


### Step 1: Representing Documents with MinHash

First, we need a way to measure document similarity. The standard approach uses Jaccard similarity:

$$J(A,B) = \frac{|A\cap B|}{|A \cup B|}$$

This formula measures the overlap between document A and document B - specifically, the ratio of shared elements to total unique elements. A higher value means the documents are more similar.

However, computing this directly for billions of document pairs is resource-intensive and would take years. MinHash creates compact "fingerprints" (signatures) that preserve similarity relationships while making comparisons much faster. 

1. **Shingling:** Break each document into overlapping sequences of words or characters (k-shingles). For example, the sentence “I love vector search” with k=3 (by word) yields: {“I love vector”, “love vector search”}

![](https://assets.zilliz.com/shingling_858ad58efa.png)

2. **MinHashing:** Apply multiple hash functions to each set of shingles and record the minimum hash value from each function. This results in a signature vector for each document.

![](https://assets.zilliz.com/minhash_041003210a.png)

When calculating similarity, the probability that hash values align at the same positions in the MinHash signatures of two documents (which corresponds to the Jaccard distance of these signatures) provides a close approximation of the Jaccard similarity of their original shingle sets. This allows us to effectively estimate document similarity without directly comparing the larger original texts; instead, we can analyze their compact MinHash signatures.  

The MinHash principle involves using the word with the smallest hash value to represent the entire document, enhancing accuracy by incorporating additional hash functions. Minor word changes are likely to be overlooked as they typically do not affect the minimum hash value. In contrast, more substantial changes tend to alter the hash value and are more easily detected. This method can be seen as a min-pooling of hash values across various words. In addition to MinHash, alternatives like SimHash are available for generating document signatures, but those will not be discussed here.


### Step 2: Identifying Similar Documents via LSH

Even with compact MinHash signatures, comparing every pair across millions or billions of documents remains computationally expensive. That’s where **Locality Sensitive Hashing (LSH)** comes in.

The key idea of LSH is to use hash functions that **intentionally cause collisions**—similar items are more likely to hash into the same bucket, while dissimilar ones are not. This is the opposite of traditional hashing, which aims to avoid collisions.

For MinHash, a popular LSH strategy is the **banding technique**:

1. **Banding**: Split each MinHash signature (a vector of length _N_) into _b_ bands, each with _r_ dims (_N = b × r_).

2. **Hashing Bands:** Hash each band (a sub-vector of _r_ values) into a bucket using a standard hash function.

3. **Candidate Pairs:** If two documents share a bucket in **any** band, they are flagged as potential matches.

By adjusting the number of bands (b) and the number of dimensions per band (r), you can control the trade-off between recall, precision, and search efficiency. 

The key idea is: highly similar documents will have many matching hash values in their MinHash signatures. When these signatures are split into bands, even one band with all matching values is enough to place two documents in the same bucket. The more similar the documents are, the higher the probability that this happens in at least one band, allowing LSH to efficiently surface candidate pairs without exhaustively comparing all signatures.

In short, **MinHash + LSH** enables scalable approximate deduplication: MinHash compresses documents into compact signatures, and LSH efficiently narrows the search space by grouping likely matches. It’s like spotting twins in a crowd: first, take a quick feature snapshot of everyone (MinHash), group lookalikes (LSH), then inspect the smaller groups closely for actual duplicates.


## Integrating MinHash LSH in Milvus 2.6

A real-world need drove the integration of MinHash LSH into Milvus 2.6. As mentioned earlier, a Milvus user—one of the leading LLM companies—approached us with a challenge: efficiently deduplicating massive volumes of text data for LLM pre-training. 

Traditional deduplication pipelines typically rely on external tools decoupled from storage and retrieval systems, requiring costly data transfers between components. This fragmented workflow increases operational overhead and prevents full utilization of distributed computing resources.

Recognizing Milvus’s strengths in handling high-throughput vector data, a natural idea emerged: **_What if MinHash LSH were built into Milvus natively, making approximate deduplication a first-class database feature?_**

This approach enables a complete workflow from deduplication to semantic retrieval within Milvus, simplifying MLOps while leveraging its scalability and unified API. Together with our partner, we optimized MinHash LSH for Milvus’s cloud-native architecture, resulting in a fast and scalable solution for large-scale deduplication.


### Core capabilities in Milvus 2.6 include:

- **Native MinHash LSH Indexing:** Implements the standard banding technique for LSH and supports optional Jaccard re-ranking to improve recall. Provides both in-memory and mmap-based implementations for flexibility across different workloads.

- **Seamless API Integration:** Users can define MinHash vector fields, build `MINHASH_LSH` indexes, insert signature data, and perform approximate similarity searches using Milvus’s standard SDK and declarative APIs.

- **Distributed and Scalable:** Built on Milvus’s cloud-native architecture, the feature supports horizontal scaling for large datasets and high-throughput processing.

This integration delivered impressive results. By running MinHash LSH on fully-managed Milvus ([Zilliz Cloud](https://zilliz.com/cloud)), we helped this user deduplicate **10 billion documents** efficiently. Compared to their previous MapReduce-based approach, the new solution **more than doubled processing speed** and achieved **3-5x cost savings**, thanks to Milvus’s optimized indexing and query execution.


## Hands-On: Deduplicating LLM Datasets Using Milvus

Let's roll up our sleeves and use MinHash LSH in Milvus 2.6 to perform approximate deduplication at scale.


### Prerequisite: Generating MinHash Signatures

Milvus handles the indexing and search of **pre-generated** MinHash signatures. You’ll need to generate these during preprocessing using tools like `datasketch` in Python or a custom implementation. The typical steps are:

1. Read raw documents

2. Shingle (tokenize or chunk) each document

3. Apply multiple hash functions to generate a MinHash signature (e.g., an uint64 array of size 128 )

```
from datasketch import MinHash

text = "example text for minhash signature"
tokens = text.lower().split()  # Step 1 & 2: preprocess + tokenize/shingle
mh = MinHash(num_perm=128)     # Step 3: initialize MinHash
for token in tokens:
    mh.update(token.encode('utf-8'))  # Add shingles to MinHash
signature = mh.hashvalues  # This is your MinHash signature (128-dimensional)
```



### Step 1: Create a Schema in Milvus

We need to create a Milvus collection to store the MinHash signatures and their corresponding document IDs.

```
import numpy as np
from pymilvus import MilvusClient
from pymilvus import DataType
MILVUS_URI = "localhost:19530"
COLLECTION_NAME = "llm_data_dedup_minhash"
MINHASH_DIM = 128
MINHASH_BIT_WIDTH = 64 # Assuming 64-bit hash values

# Load data from NPY file
base = np.load('minhash_vectors.npy')
ids = [str(i) for i in range(base.shape[0])]  # Generate string IDs

client = MilvusClient(uri=MILVUS_URI)
# Check and drop existing collection if needed
if collection_name in client.list_collections():
    print(f"Collection {collection_name} exists, dropping it...")
    client.drop_collection(collection_name)
# Create collection schema
schema = MilvusClient.create_schema(
    auto_id=False,
    enable_dynamic_field=False,
)

schema.add_field(field_name="input_id", datatype=DataType.INT64, is_primary=True)
schema.add_field(field_name="minhash", datatype=DataType.BINARY_VECTOR, dim=MINHASH_DIM * MINHASH_BIT_WIDTH)
schema.add_field(field_name="id", datatype=DataType.VARCHAR, max_length=200)
```


### **Step 2: Create the MINHASH_LSH Index and Collection**

This is the core step. We need to specify JACCARD as the metric type and configure LSH-related parameters. 

```
INDEX_FIELD_NAME = "minhash_signature"
# Metric type, should be JACCARD for MinHash LSH
METRIC_TYPE = "MHJACCARD"
INDEX_TYPE = "MINHASH_LSH"
MINHASH_DIM = 128
MINHASH_BIT_WIDTH = 64 # Assuming 64-bit hash values

index_params = MilvusClient.prepare_index_params()

index_params.add_index(
    field_name=INDEX_FIELD_NAME,
    index_type=INDEX_TYPE,
    metric_type=METRIC_TYPE,
    params={
        # LSH-specific parameters might be configured here, e.g.:
        # "band": 32, # Hypothetical parameter: number of bands
        # "element_bit_width": 64 # Bit width of minhash values
    }
)
# Create collection
client.create_collection(
    collection_name=collection_name,
    schema=schema,
    index_params=index_params
)
```


A Note on Parameter Tuning: The effectiveness of MinHash LSH heavily depends on parameter choices. For instance, the number of hash functions used during MinHash signature generation (i.e., `MINHASH_DIM`) affects the signature's precision and size. In the LSH phase, the number of bands (`num_bands`) and rows per band together determine the sensitivity range of the similarity threshold and the balance between recall and precision. Users need to experiment and fine-tune based on their dataset characteristics and deduplication requirements. This is often an iterative process.


### **Step 3: Insert MinHash Signatures**

Let's say you have a batch of documents and their corresponding MinHash signatures.

```
# Insert data in batches
batch_size = 2000
total_records = base.shape[0]
num_batches = (total_records + batch_size - 1) // batch_size

for batch_idx in range(num_batches):
    start = batch_idx * batch_size
    end = min((batch_idx + 1) * batch_size, total_records)
    
    print(f"Inserting batch {batch_idx + 1}/{num_batches} (records {start}-{end})")
    
    # Prepare batch data
    batch_data = [{
        "input_id": i,
        "minhash": base[i].tobytes(),
        "id": ids[i]
    } for i in range(start, end)]
    
    # Insert batch
    client.insert(collection_name, batch_data)

print("Data insertion complete")
```


### Step 5: Search for Near-Duplicates

Use a document’s MinHash signature to search for similar documents in the collection.

```
# Perform search
search_vectors = [vec.tobytes() for vec in base[:10]]
results = client.search(
    collection_name, 
    data=search_vectors, 
    param={"metric_type": METRIC_TYPE, "params": search_params_lsh},
    limit=1, 
    output_fields=["id"])

print("\nSearch results:")
for i, result in enumerate(results):
    print(f"Query {i}:")
    for hit in result:
        print(f"  - ID: {hit['entity']['id']}, Distance: {hit['distance']}")
```



### Step 6: Post-Processing and Clustering

The returned results are **candidate near-duplicates**. To form complete deduplication groups, you can apply clustering techniques like **Union-Find** on the candidate pairs. Each resulting group represents a set of duplicates; keep one representative document and archive or remove the rest.


## **Conclusion**

MinHash LSH in Milvus 2.6 is a leap forward in AI data processing. What started as a solution for LLM data deduplication now opens doors to broader use cases—web content cleanup, catalog management, plagiarism detection, and more.

## Getting Started with Milvus 2.6

Milvus 2.6 is available now. In addition to MinHash LSH, it introduces dozens of new features and performance optimizations such as tiered storage, RabbitQ quantization method, and enhanced full-text search and multitenancy, directly addressing the most pressing challenges in vector search today: scaling efficiently while keeping costs under control.

Ready to explore everything Milvus offers? Dive into our[ release notes](https://milvus.io/docs/release_notes.md), browse the[ complete documentation](https://milvus.io/docs), or check out our[ feature blogs](https://milvus.io/blog). 

If you have any questions or have a similar use case, feel free to reach out to us through our [Discord community](https://discord.com/invite/8uyFbECzPX) or file an issue on[ GitHub](https://github.com/milvus-io/milvus) — we're here to help you make the most of Milvus 2.6.






MinHash LSH in Milvus: The Secret Weapon for Fighting Duplicates in LLM Training Data



**TL;DR:** We built Woodpecker, a cloud-native Write-Ahead Logging (WAL) system, to replace Kafka and Pulsar in Milvus 2.6. The result? Simplified operations, better performance, and lower costs for our Milvus vector database.

![](https://assets.zilliz.com/We_Replaced_Kafka_Pulsar_with_a_Woodpecker_for_Milvus_Here_s_What_Happened_77e8de27a9.png)

## The Starting Point: When Message Queues No Longer Fit

We loved and used Kafka and Pulsar. They worked until they didn't. As Milvus, the leading open-source vector database, evolved, we found that these powerful message queues no longer met our scalability requirements. So we made a bold move: we rewrote the streaming backbone in Milvus 2.6 and implemented our own WAL — **Woodpecker**.

Let me walk you through our journey and explain why we made this change, which might seem counterintuitive at first glance.


## Cloud-Native From Day One

Milvus has been a cloud-native vector database from its inception. We leverage Kubernetes for elastic scaling and quick failure recovery, alongside object storage solutions like Amazon S3 and MinIO for data persistence.

This cloud-first approach offers tremendous advantages, but it also presents some challenges:

- Cloud object storage services like S3 provide virtually unlimited capability of handling throughputs and availability, but with latencies often exceeding 100ms.

- These services’ pricing models (based on access patterns and frequency) can add unexpected costs to real-time database operations.

- Balancing cloud-native characteristics with the demands of real-time vector search introduces significant architectural challenges.


## The Shared Log Architecture: Our Foundation

Many vector search systems restrict themselves to batch processing because building a streaming system in a cloud-native environment presents even greater challenges. In contrast, Milvus prioritizes real-time data freshness and implements a shared log architecture—think of it as a hard drive for a filesystem.

This shared log architecture provides a critical foundation that separates consensus protocols from core database functionality. By adopting this approach, Milvus eliminates the need to manage complex consensus protocols directly, allowing us to focus on delivering exceptional vector search capabilities.

We're not alone in this architectural pattern—databases such as AWS Aurora, Azure Socrates, and Neon all leverage a similar design. **However, a significant gap remains in the open-source ecosystem: despite the clear advantages of this approach, the community lacks a low-latency, scalable, and cost-effective distributed write-ahead log (WAL) implementation.**

Existing solutions like Bookie proved inadequate for our needs due to their heavyweight client design and the absence of production-ready SDKs for Golang and C++. This technological gap led us to our initial approach with message queues.


## Our Initial Solution: Message Queues as WAL

To bridge this gap, our initial approach utilized message queues (Kafka/Pulsar) as our write-ahead log (WAL). The architecture worked like this: 

- All incoming real-time updates flow through the message queue.

- Writers receive immediate confirmation once it is accepted by the message queue.

- QueryNode and DataNode process this data asynchronously, ensuring high write throughput while maintaining data freshness

![](https://assets.zilliz.com/Figure_Milvus_2_0_Architecture_Overview_465f5ba27a.png)

Figure: Milvus 2.0 Architecture Overview

This system effectively provided immediate write confirmation while enabling asynchronous data processing, which was crucial for maintaining the balance between throughput and data freshness that Milvus users expect.


## Why We Needed Something Different for WAL

With Milvus 2.6, we've decided to phase out external message queues in favor of Woodpecker, our purpose-built, cloud-native WAL implementation. This wasn't a decision we made lightly. After all, we had successfully used Kafka and Pulsar for years.

The issue wasn't with these technologies themselves—both are excellent systems with powerful capabilities. Instead, the challenge came from the increasing complexity and overhead that these external systems introduced as Milvus evolved. As our requirements became more specialized, the gap between what general-purpose message queues offered and what our vector database needed continued to widen.

Three specific factors ultimately drove our decision to build a replacement:


### Operational Complexity

External dependencies like Kafka or Pulsar demand dedicated machines with multiple nodes and careful resource management. This creates several challenges:

- Increased operational complexity

* Steeper learning curves for system administrators

- Higher risks of configuration errors and security vulnerabilities


### Architectural Constraints

Message queues like Kafka have inherent limitations on the number of supported topics. We developed VShard as a workaround for topic sharing across components, but this solution—while effectively addressing scaling needs—introduced significant architectural complexity.

These external dependencies made it harder to implement critical features—such as log garbage collection—and increased integration friction with other system modules. Over time, the architectural mismatch between general-purpose message queues and the specific, high-performance demands of a vector database became increasingly clear, prompting us to reassess our design choices.


### Resource Inefficiency

Ensuring high availability with systems like Kafka and Pulsar typically demands:

- Distributed deployment across multiple nodes

- Substantial resource allocation even for smaller workloads

- Storage for ephemeral signals (like Milvus's Timetick), which don’t actually require long-term retention

However, these systems lack the flexibility to bypass persistence for such transient signals, leading to unnecessary I/O operations and storage usage. This leads to disproportionate resource overhead and increased cost—especially in smaller-scale or resource-constrained environments.


## Introducing Woodpecker - A Cloud-Native, High-Performance WAL Engine

In Milvus 2.6, we’ve replaced Kafka/Pulsar with **Woodpecker**, a purpose-built, cloud-native WAL system. Designed for object storage, Woodpecker simplifies operations while boosting performance and scalability.

Woodpecker is built from the ground up to maximize the potential of cloud-native storage, with a focused goal: to become the highest-throughput WAL solution optimized for cloud environments while delivering the core capabilities needed for an append-only write-ahead log.


### The Zero-Disk Architecture for Woodpecker 

Woodpecker's core innovation is its **Zero-Disk architecture**:

- All log data stored in cloud object storage (such as Amazon S3, Google Cloud Storage, or Alibaba OS)

- Metadata managed through distributed key-value stores like etcd

- No local disk dependencies for core operations

![](https://assets.zilliz.com/Figure_Woodpecker_Architecture_cc31e15ed9.png)

Figure:  Woodpecker Architecture Overview

This approach dramatically reduces operational overhead while maximizing durability and cloud efficiency. By eliminating local disk dependencies, Woodpecker aligns perfectly with cloud-native principles and significantly reduces the operational burden on system administrators.


### Performance Benchmarks: Exceeding Expectations

We ran comprehensive benchmarks to evaluate Woodpecker's performance in a single-node, single-client, single-log-stream setup. The results were impressive when compared to Kafka and Pulsar:

| **System** | **Kafka**   | **Pulsar** | **WP MinIO** | **WP Local** | **WP S3** |
| ---------- | ----------- | ---------- | ------------ | ------------ | --------- |
| Throughput | 129.96 MB/s | 107 MB/s   | 71 MB/s      | 450 MB/s     | 750 MB/s  |
| Latency    | 58 ms       | 35 ms      | 184 ms       | 1.8 ms       | 166 ms    |

For context, we measured the theoretical throughput limits of different storage backends on our test machine:

- **MinIO**: ~110 MB/s

- **Local file system**: 600–750 MB/s

- **Amazon S3 (single EC2 instance)**: up to 1.1 GB/s

Remarkably, Woodpecker consistently achieved 60-80% of the maximum possible throughput for each backend—an exceptional efficiency level for middleware.


#### Key Performance Insights

1. **Local File System Mode**: Woodpecker achieved 450 MB/s—3.5× faster than Kafka and 4.2× faster than Pulsar—with ultra-low latency at just 1.8 ms, making it ideal for high-performance single-node deployments.

2. **Cloud Storage Mode (S3)**: When writing directly to S3, Woodpecker reached 750 MB/s (about 68% of S3's theoretical limit), 5.8× higher than Kafka and 7× higher than Pulsar. While latency is higher (166 ms), this setup provides exceptional throughput for batch-oriented workloads.

3. **Object Storage Mode (MinIO)**: Even with MinIO, Woodpecker achieved 71 MB/s—around 65% of MinIO's capacity. This performance is comparable to Kafka and Pulsar but with significantly lower resource requirements.

Woodpecker is particularly optimized for concurrent, high-volume writes where maintaining order is critical. And these results only reflect the early stages of development—ongoing optimizations in I/O merging, intelligent buffering, and prefetching are expected to push performance even closer to theoretical limits.


### Design Goals

Woodpecker addresses the evolving demands of real-time vector search workloads through these key technical requirements:

- High-throughput data ingestion with durable persistence across availability zone

- Low-latency tail reads for real-time subscriptions and high-throughput catch-up reads for failure recovery

- Pluggable storage backends, including cloud object storage and file systems with NFS protocol support

- Flexible deployment options, supporting both lightweight standalone setups and large-scale clusters for multi-tenant Milvus deployments


### Architecture Components

A standard Woodpecker deployment includes the following components. 

- **Client** – Interface layer for issuing read and write requests

- **LogStore** – Manages high-speed write buffering, asynchronous uploads to storage, and log compaction

- **Storage Backend** – Supports scalable, low-cost storage services such as S3, GCS, and file systems like EFS

- **ETCD** – Stores metadata and coordinates log state across distributed nodes


### Flexible Deployments to Match Your Specific Needs 

Woodpecker offers two deployment modes to match your specific needs:

**MemoryBuffer Mode – Lightweight and Maintenance-Free**

MemoryBuffer Mode provides a simple and lightweight deployment option where Woodpecker temporarily buffers incoming writes in memory and periodically flushes them to a cloud object storage service. Metadata is managed using etcd to ensure consistency and coordination. This mode is best suited for batch-heavy workloads in smaller-scale deployments or production environments that prioritize simplicity over performance, especially when low write latency is not critical.

![](https://assets.zilliz.com/Figure_The_memory_Buffer_Mode_3429d693a1.png)

_Figure: The memoryBuffer Mode_ 

**QuorumBuffer Mode – Optimized for Low-Latency, High-Durability Deployments**

QuorumBuffer Mode is designed for latency-sensitive, high-frequency read/write workloads requiring both real-time responsiveness and strong fault tolerance. In this mode, Woodpecker functions as a high-speed write buffer with three-replica quorum writes, ensuring strong consistency and high availability.

A write is considered successful once it's replicated to at least two of the three nodes, typically completing within single-digit milliseconds, after which the data is asynchronously flushed to cloud object storage for long-term durability. This architecture minimizes on-node state, eliminates the need for large local disk volumes, and avoids complex anti-entropy repairs often required in traditional quorum-based systems.

The result is a streamlined, robust WAL layer ideal for mission-critical production environments where consistency, availability, and fast recovery are essential.

![](https://assets.zilliz.com/Figure_The_Quorum_Buffer_Mode_72573dc666.png)

_Figure: The QuorumBuffer Mode_ 


## StreamingService: Built for Real-Time Data Flow

Beyond Woodpecker, Milvus 2.6 introduces the **StreamingService**—a specialized component designed for log management, log ingestion, and streaming data subscription.

To understand how our new architecture works, it's important to clarify the relationship between these two components:

- **Woodpecker** is the storage layer that handles the actual persistence of write-ahead logs, providing durability and reliability

- **StreamingService** is the service layer that manages log operations and provides real-time data streaming capabilities

Together, they form a complete replacement for external message queues. Woodpecker provides the durable storage foundation, while StreamingService delivers the high-level functionality that applications interact with directly. This separation of concerns allows each component to be optimized for its specific role while working seamlessly together as an integrated system. 


### Adding Streaming Service to Milvus 2.6 

![](https://assets.zilliz.com/Figure_Milvus_2_6_Architecture_Overview_238428c58f.png)

Figure: Streaming Service Added in Milvus 2.6 Architecture

The Streaming Service is composed of three core components: 

**Streaming Coordinator**

- Discovers available Streaming Nodes by monitoring Milvus ETCD sessions

- Manages the status of WALs and collects load balancing metrics through the ManagerService

**Streaming Client**

- Queries the AssignmentService to determine WAL segment distribution across Streaming Nodes

- Performs read/write operations via the HandlerService on the appropriate Streaming Node

**Streaming Node**

- Handles actual WAL operations and provides publish-subscribe capabilities for real-time data streaming

- Includes the **ManagerService** for WAL administration and performance reporting

- Features the **HandlerService** that implements efficient publish-subscribe mechanisms for WAL entries

This layered architecture allows Milvus to maintain clear separation between the streaming functionality (subscription, real-time processing) and the actual storage mechanisms. Woodpecker handles the "how" of log storage, while StreamingService manages the "what" and "when" of log operations.

As a result, the Streaming Service significantly enhances the real-time capabilities of Milvus by introducing native subscription support, eliminating the need for external message queues. It reduces memory consumption by consolidating previously duplicated caches in the query and data paths, lowers latency for strongly consistent reads by removing asynchronous synchronization delays, and improves both scalability and recovery speed across the system.


## Conclusion - Streaming on a Zero-Disk Architecture

Managing state is hard. Stateful systems often sacrifice elasticity and scalability. The increasingly accepted answer in cloud-native design is to decouple state from compute—allowing each to scale independently.

Rather than reinventing the wheel, we delegate the complexity of durable, scalable storage to the world-class engineering teams behind services like AWS S3, Google Cloud Storage, and MinIO. Among them, S3 stands out for its virtually unlimited capacity, eleven nines (99.999999999%) of durability, 99.99% availability, and high-throughput read/write performance.

But even "zero-disk" architectures have trade-offs. Object stores still struggle with high write latency and small-file inefficiencies—limitations that remain unresolved in many real-time workloads.

For vector databases—especially those supporting mission-critical RAG, AI agents, and low-latency search workloads—real-time access and fast writes are non-negotiable. That's why we rearchitected Milvus around Woodpecker and the Streaming Service. This shift simplifies the overall system (let's face it—no one wants to maintain a full Pulsar stack inside a vector database), ensures fresher data, improves cost-efficiency, and speeds up failure recovery.

We believe Woodpecker is more than just a Milvus component—it can serve as a foundational building block for other cloud-native systems. As cloud infrastructure evolves, innovations like S3 Express may bring us even closer to the ideal: cross-AZ durability with single-digit millisecond write latency.

## Getting Started with Milvus 2.6

Milvus 2.6 is available now. In addition to Woodpecker, it introduces dozens of new features and performance optimizations such as tiered storage, RabbitQ quantization method, and enhanced full-text search and multitenancy, directly addressing the most pressing challenges in vector search today: scaling efficiently while keeping costs under control.

Ready to explore everything Milvus offers? Dive into our[ release notes](https://milvus.io/docs/release_notes.md), browse the[ complete documentation](https://milvus.io/docs), or check out our[ feature blogs](https://milvus.io/blog).

Have questions? You're also welcome to join our [Discord community](https://discord.com/invite/8uyFbECzPX) or file an issue on[ GitHub](https://github.com/milvus-io/milvus) — we're here to help you make the most of Milvus 2.6.


 


We Replaced Kafka/Pulsar with a Woodpecker for Milvus


We Replaced Kafka/Pulsar with a Woodpecker for Milvus—Here’s What Happened




## The Vector Database You Chose Based on Benchmarks Might Fail in Production


When selecting a [vector database](https://milvus.io/blog/what-is-a-vector-database.md) for your AI application, conventional benchmarks are like test-driving a sports car on an empty track, only to find it stalls in rush hour traffic. The uncomfortable truth? Most benchmarks only evaluate performance in artificial conditions that never exist in production environments.

![](https://assets.zilliz.com/Benchmarks_Lie_Vector_D_Bs_Deserve_a_Real_Test_9280c66efc.png)

Most benchmarks test vector databases **after** all data has been ingested and the index is fully built. But in production, data never stops flowing. You don’t get to pause your system for hours to rebuild an index.

We’ve seen the disconnect firsthand. For example, Elasticsearch might boast millisecond-level query speeds, but behind the scenes, we’ve watched it take **over 20 hours** just to optimize its index. That’s downtime no production system can afford, especially in AI workloads that demand continuous updates and instant responses.

With Milvus, after running countless Proof of Concept (PoC) evaluations with enterprise clients, we've uncovered a troubling pattern: **vector databases that excel in controlled lab environments frequently struggle under actual production loads.** This critical gap doesn't just frustrate infrastructure engineers—it can derail entire AI initiatives built on these misleading performance promises.

That's why we built [VDBBench](https://github.com/zilliztech/VectorDBBench): an open-source benchmark designed from the ground up to simulate production reality. Unlike synthetic tests that cherry-pick scenarios, VDBBench pushes databases through continuous ingestion, rigorous filtering conditions, and diverse scenarios, just like your actual production workloads. Our mission is simple: give engineers a tool that shows how vector databases actually perform under real-world conditions so you can make infrastructure decisions based on trustworthy numbers.


## The Gap between Benchmarks and Reality

Traditional benchmarking approaches suffer from three critical flaws that render their results practically meaningless for production decision-making:


### 1. Outdated Data

Many benchmarks still rely on outdated datasets like SIFT or[ GloVe](https://zilliz.com/glossary/glove), which bear little resemblance to today's complex, high-dimensional vector embeddings generated by AI models. Consider this: SIFT contains 128-dimensional vectors, while popular embeddings from OpenAI’s embedding models range from 768 to 3072 dimensions. 


### 2. Vanity Metrics

Many benchmarks focus solely on average latency or peak QPS, which creates a distorted picture. These idealized metrics fail to capture the outliers and inconsistencies that actual users experience in production environments. For example, what good is an impressive QPS number if it requires unbounded computational resources that would bankrupt your organization?


### 3. Oversimplified Scenarios

Most benchmarks test only basic, static workloads—essentially the "Hello World" of vector search. For example, they issue search requests only after the entire dataset is ingested and indexed, ignoring the dynamic reality where users search while new data streams in. This simplistic design overlooks the complex patterns defining real production systems such as concurrent queries, filtered searches, and continuous data ingestion.

Recognizing these flaws, we realized the industry needed a **radical shift in benchmarking philosophy**—one grounded in how AI systems actually behave in the wild. That’s why we built [VDBBench](https://github.com/zilliztech/VectorDBBench).


## From Lab to Production: How VDBBench Bridges the Gap

VDBBench doesn't just iterate on outdated benchmarking philosophies—it rebuilds the concept from first principles with one guiding belief: **a benchmark is only valuable if it predicts actual production behavior**.

We've engineered VDBBench to faithfully replicate real-world conditions across three critical dimensions: data authenticity, workload patterns, and performance measurement.


### Modernizing the Dataset

We've completely overhauled the datasets used for vectorDB benchmarking. Instead of legacy test sets like SIFT and GloVe, VDBBench uses vectors generated from state-of-the-art embedding models that power today's AI applications. 

To ensure relevance, especially for use cases like Retrieval-Augmented Generation (RAG), we selected corpora that reflect real-world enterprise and domain-specific scenarios. These range from general-purpose knowledge bases to vertical applications like biomedical question answering and large-scale web search.

|            |                     |                |                 |
| ---------- | ------------------- | -------------- | --------------- |
| **Corpus** | **Embedding Model** | **Dimensions** | **Size**        |
| Wikipedia  | Cohere V2           | 768            | 1M / 10M        |
| BioASQ     | Cohere V3           | 1024           | 1M / 10M        |
| C4         | OpenAI              | 1536           | 500K / 5M       |
| MSMarco V2 | udever-bloom-1b1    | 1536           | 1M / 10M / 138M |

Table: Datasets used in VDBBench

VDBBench also supports custom datasets, letting you benchmark with your own data generated from your specific embedding models for your specific workloads. After all, no dataset tells a better story than your own production data.


### Production-Focused Metric Design

**VDBBench prioritizes metrics that reflect real-world performance, not just lab results.** We’ve redesigned benchmarking around what actually matters in production environments: reliability under load, tail latency, sustained throughput, and accuracy.

- **P95/P99 latency to measure real user experience**: Average/median latency masks the outliers that frustrate real users. That’s why VDBBench focuses on tail latency like P95/P99, revealing what performance 95% or 99% of your queries will actually achieve.

- **Sustainable throughput under load:** A system that performs well for 5 seconds doesn’t cut it in production. VDBBench gradually increases concurrency to find your database’s maximum sustainable queries per second (`max_qps`)—not the peak number under short, ideal conditions. This shows how well your system holds up over time.

- **Recall balanced with performance:** Speed without accuracy is meaningless. Every performance number in VDBBench is paired with recall, so you know exactly how much relevance you’re trading off for throughput. This enables fair, apples-to-apples comparisons between systems with vastly different internal tradeoffs.


### Test Methodology That Reflects Reality

A key innovation in VDBBench’s design is the **separation of serial and concurrent testing**, which helps capture how systems behave under different types of load. For instance, latency metrics are divided as follows:

- `serial_latency_p99` measures system performance under minimal load, where only one request is processed at a time. This represents the _best-case scenario_ for latency.

- `conc_latency_p99` captures system behavior under _realistic, high-concurrency conditions_, where multiple requests arrive simultaneously.


### Two Benchmark Phases

VDBBench separates testing into two crucial phases: 

1. **Serial Test**

This is a single-process run of 1,000 queries. This phase establishes a baseline for ideal performance and accuracy, reporting both `serial_latency_p99` and recall.

2. **Concurrency Test**

This phase simulates a production environment under sustained load.

- **Realistic client simulation**: Each test process operates independently with its own connection and query set. This avoids shared-state (e.g., cache) interference that could distort results.

- **Synchronized start**: All processes begin simultaneously, ensuring that the measured QPS accurately reflects the claimed concurrency level.

These carefully structured methods ensure that the `max_qps` and `conc_latency_p99` values reported by VDBBench are both **accurate and production-relevant**, providing meaningful insights for production capacity planning and system design.

![](https://assets.zilliz.com/Figure_QPS_and_Latency_of_Milvus_16c64g_standalone_at_Varying_Concurrency_Levels_Cohere_1_M_Test_7f2294e87a.png)

_Figure: QPS and Latency of Milvus-16c64g-standalone at Varying Concurrency Levels (Cohere 1M Test). In this test, Milvus is initially underutilized—up to_ **_concurrency level 20_**_, increasing concurrency improves system utilization and results in higher QPS. Beyond_ **_concurrency 20_**_, the system reaches full load: further increases in concurrency no longer improve throughput, and latency rises due to queuing delays._


## Beyond Searching Static Data: The Real Production Scenarios

To the best of our knowledge, VDBBench is the only benchmark tool that tests vector databases across the complete spectrum of production-critical scenarios, including static collection, filtering, and streaming cases.


### Static Collection

Unlike other benchmarks that rush into testing, VDBBench first ensures each database has fully optimized its indexes—a critical production prerequisite that many benchmarks often neglect. This gives you the complete picture:

- Data ingestion time

- Indexing time (the time used to build an optimized index, which dramatically affects search performance)

- Search performance on fully optimized indexes under both serial and concurrent conditions


### Filtering

Vector search in production rarely happens in isolation. Real applications combine vector similarity with metadata filtering ("find shoes that look like this photo but cost under $100"). This filtered vector search creates unique challenges:

- **Filter Complexity**: More scalar columns and logic conditions increase computational demands

- **Filter Selectiveness**: [Our production experience](https://milvus.io/blog/how-to-filter-efficiently-without-killing-recall.md) reveals this as the hidden performance killer—query speeds can fluctuate by orders of magnitude depending on how selective filters are

VDBBench systematically evaluates filter performance across varying selectivity levels (from 50% to 99.9%), delivering a comprehensive profile of how databases handle this critical production pattern.

![](https://assets.zilliz.com/Figure_QPS_and_Recall_of_Milvus_and_Open_Search_Across_Different_Filter_Selectivity_Levels_Cohere_1_M_Test_4b5df2244d.png)

_Figure: QPS and Recall of Milvus and OpenSearch Across Different Filter Selectivity Levels (Cohere 1M Test). The X-axis represents the percentage of data filtered. As shown, Milvus maintains consistently high recall across all filter selectivity levels, while OpenSearch exhibits unstable performance, with recall fluctuating significantly under different filtering conditions._


### Streaming

Production systems rarely enjoy the luxury of static data. New information continuously flows in while searches execute—a scenario where many otherwise impressive databases collapse.

VDBBench's unique streaming test case examines search-while-inserting performance, measuring:

1. **Impact of Growing Data Volume**: How search performance scales with increasing data size.

2. **Impact of Write Load**: How concurrent writes affect search latency and throughput, as write also consumes CPU or memory resources in the system.

Streaming scenarios represent a comprehensive stress test for any vector database. But building a _fair_ benchmark for this isn’t trivial. It’s not enough to describe how one system behaves—we need a consistent evaluation model that enables **apples-to-apples comparisons** across different databases.

Drawing from our experience helping enterprises with real-world deployments, we built a structured, repeatable approach. With VDBBench:

- You **define a fixed insertion rate** that mirrors your target production workload.

- VDBBench then applies **identical load pressure** across all systems, ensuring performance results are directly comparable.

For example, with a Cohere 10M dataset and a 500 rows/second ingestion target:

- VDBBench spins up 5 parallel producer processes, each inserting 100 rows per second.

- After every 10% of data is ingested, VDBBench triggers a round of search testing under both serial and concurrent conditions.

- Metrics such as latency, QPS, and recall are recorded after each stage.

This controlled methodology reveals how each system's performance evolves over time and under real operational stress—giving you the insight you need to make infrastructure decisions that scale.

![](https://assets.zilliz.com/igure_QPS_and_Recall_of_Pinecone_vs_Elasticsearch_in_the_Cohere_10_M_Streaming_Test_500_rows_s_Ingestion_Rate_548fc02f24.png)

_Figure: QPS and Recall of Pinecone vs. Elasticsearch in the Cohere 10M Streaming Test (500 rows/s Ingestion Rate). Pinecone maintained higher QPS and recall, showing a significant QPS improvement after inserting 100% of the data._

But this is not the end of the story. VDBBench goes even further by supporting an optional optimization step, allowing users to compare streaming search performance before and after index optimization. It also tracks and reports the actual time spent on each stage, offering deeper insights into system efficiency and behavior under production-like conditions.


​![](https://assets.zilliz.com/Figure_QPS_and_Recall_of_Pinecone_vs_Elasticsearch_in_the_Cohere_10_M_Streaming_Test_After_Optimization_500_rows_s_Ingestion_Rate_d249d290bb.png)


_Figure: QPS and Recall of Pinecone vs. Elasticsearch in the Cohere 10M Streaming Test After Optimization (500 rows/s Ingestion Rate)_ 

As shown in the diagram, ElasticSearch surpassed Pinecone in QPS—after index optimization. A miracle? Not quite. The right chart tells the full story: once the x-axis reflects actual elapsed time, it’s clear that ElasticSearch took significantly longer to reach that performance. And in production, that delay matters. This comparison reveals a key tradeoff: peak throughput vs. time-to-serve.


## Choose Your Vector Database with Confidence

The gap between benchmark results and real-world performance shouldn't be a guessing game. VDBBench provides a way to evaluate vector databases under realistic, production-like conditions, including continuous data ingestion, metadata filtering, and streaming workloads.

If you're planning to deploy a vector database in production, it's worth understanding how it performs beyond idealized lab tests. VDBBench is open-source, transparent, and designed to support meaningful, apples-to-apples comparisons.

Try VDBBench with your own workloads today and see how different systems hold up in practice: <https://github.com/zilliztech/VectorDBBench>. 

Have questions or want to share your results? Join the conversation on[ GitHub](https://github.com/zilliztech/VectorDBBench) or connect with our community on [Discord](https://discord.com/invite/FG6hMJStWu). We’d love to hear your opinions. 


Benchmarks Lie — Vector DBs Deserve a Real Test




[Milvus](https://milvus.io/docs/overview.md) is an open-source, highly scalable vector database powering semantic search at a billion-vector scale. As users deploy RAG chatbots, AI customer service, and visual search at this magnitude, a common challenge emerges: **infrastructure costs.** In contrast, exponential business growth is exciting; skyrocketing cloud bills are not. Fast vector search typically requires storing vectors in memory, which is expensive. Naturally, you might ask: _Can we compress vectors to save space without sacrificing search quality?_

The answer is **YES**, and in this blog, we'll show you how implementing a novel technique called [**RaBitQ**](https://dl.acm.org/doi/pdf/10.1145/3654970) enables Milvus to serve 3× more traffic with lower memory cost while maintaining comparable accuracy. We’ll also share the practical lessons learned from integrating RaBitQ into open-source Milvus and the fully-managed Milvus service on [Zilliz Cloud](https://zilliz.com/cloud).

![](https://assets.zilliz.com/Bring_Vector_Compression_to_the_Extreme_How_Milvus_Serves_3_More_Queries_with_Ra_Bit_Q_12f5b4d932.jpg)


## Understanding Vector Search and Compression

Before diving into RaBitQ, let's understand the challenge. 

[**Approximate Nearest Neighbor (ANN)**](https://zilliz.com/glossary/anns) search algorithms are at the heart of a vector database, finding the top-k vectors closest to a given query. A vector is a coordinate in high-dimensional space, often comprising hundreds of floating-point numbers. As vector data scales up, so do storage and compute demands. For instance, running [HNSW](https://zilliz.com/learn/hierarchical-navigable-small-worlds-HNSW) (an ANN search algorithm) with one billion 768-dimensional vectors in FP32 requires over 3TB of memory!

Like MP3 compresses audio by discarding frequencies imperceptible to the human ear, vector data can be compressed with minimal impact on search accuracy. Research shows that full-precision FP32 is often unnecessary for ANN.[ Scalar Quantization](https://zilliz.com/learn/scalar-quantization-and-product-quantization) (SQ), a popular compression technique, maps floating-point values into discrete bins and stores only the bin indices using low-bit integers. Quantization methods significantly reduce memory usage by representing the same information with fewer bits. Research in this domain strives to achieve the most savings with the least loss in accuracy.

The most extreme compression technique—1-bit Scalar Quantization, also known as [Binary Quantization](https://zilliz.com/learn/scalar-quantization-and-product-quantization)—represents each float with a single bit. Compared to FP32 (32-bit encoding), this reduces memory usage by 32×. Since memory is often the main bottleneck in vector search, such compression can significantly boost performance. **The challenge, however, lies in preserving search accuracy.** Typically, 1-bit SQ reduces recall to below 70%, making it practically unusable.

This is where **RaBitQ** stands out—an excellent compression technique that achieves 1-bit quantization while preserving high recall. Milvus now supports RaBitQ starting from version 2.6, enabling the vector database to serve 3× the QPS while maintaining a comparable level of accuracy.

## A Brief Intro to RaBitQ

[RaBitQ](https://dl.acm.org/doi/pdf/10.1145/3654970) is a smartly designed binary quantization method that leverages the geometry property of high-dimensional space to achieve efficient and accurate vector compression.

At first glance, reducing each dimension of a vector to a single bit may seem too aggressive, but in high-dimensional space, our intuitions often fail us. As Jianyang Gao, an author of RaBitQ,[ illustrated](https://dev.to/gaoj0017/quantization-in-the-counterintuitive-high-dimensional-space-4feg), high-dimensional vectors exhibit the property that individual coordinates tend to be tightly concentrated around zero, a result of a counterintuitive phenomenon explained in[ Concentration of Measure](https://en.wikipedia.org/wiki/Concentration_of_measure). This makes it possible to discard much of the original precision while still preserving the relative structure needed for an accurate nearest neighbor search.

![](https://assets.zilliz.com/Figure_The_counterintuitive_value_distribution_in_high_dimensional_geometry_fad6143bfd.png)

Figure: The counterintuitive value distribution in high-dimensional geometry. _Consider the value of the first dimension for a random unit vector uniformly sampled from the unit sphere; the values are uniformly spread in 3D space. However, for high-dimensional space (e.g., 1000D), the values concentrate around zero, an unintuitive property of high-dimensional geometry. (Image source: [Quantization in The Counterintuitive High-Dimensional Space](https://dev.to/gaoj0017/quantization-in-the-counterintuitive-high-dimensional-space-4feg))_ 

Inspired by this property of high-dimensional space, **RaBitQ focuses on encoding angular information rather than exact spatial coordinates**. It does this by normalizing each data vector relative to a reference point such as the centroid of the dataset. Each vector is then mapped to its nearest vertex on the hypercube, allowing representation with just 1 bit per dimension. This approach naturally extends to `IVF_RABITQ`, where normalization is done relative to the closest cluster centroid, improving local encoding accuracy.

![](https://assets.zilliz.com/Figure_Compressing_a_vector_by_finding_its_closest_approximation_on_the_hypercube_so_that_each_dimension_can_be_represented_with_just_1_bit_cd0d50bb30.png)

_Figure: Compressing a vector by finding its closest approximation on the hypercube, so that each dimension can be represented with just 1 bit. (Image source:_ [_Quantization in The Counterintuitive High-Dimensional Space_](https://dev.to/gaoj0017/quantization-in-the-counterintuitive-high-dimensional-space-4feg)_)_ 

To ensure search remains reliable even with such compressed representations, RaBitQ introduces a **theoretically grounded, unbiased estimator** for the distance between a query vector and binary-quantized document vectors. This helps minimize reconstruction error and sustain high recall.

RaBitQ is also highly compatible with other optimization techniques, such as[ FastScan](https://www.vldb.org/pvldb/vol9/p288-andre.pdf) and[ random rotation preprocessing](https://github.com/facebookresearch/faiss/wiki/Pre--and-post-processing). Moreover, RaBitQ is **lightweight to train and fast to execute**. Training involves simply determining the sign of each vector component, and search is accelerated via fast bitwise operations supported by modern CPUs. Together, these optimizations enable RaBitQ to deliver high-speed search with minimal loss of accuracy.


## Engineering RaBitQ in Milvus: From Academic Research to Production

While RaBitQ is conceptually straightforward and accompanied by a[ reference implementation](https://github.com/gaoj0017/RaBitQ), adapting it in a distributed, production-grade vector database like Milvus presented several engineering challenges. We've implemented RaBitQ in Knowhere, the core vector search engine behind Milvus, and also contributed an optimized version to the open-source ANN search library[ FAISS](https://github.com/facebookresearch/faiss).

Let's look at how we brought this algorithm to life in Milvus.


### Implementation Tradeoffs 

One important design decision involved handling per-vector auxiliary data. RaBitQ requires two floating-point values per vector pre-computed during indexing time, and a third value that can either be computed on-the-fly or pre-computed. In Knowhere, we pre-computed this value at indexing time and stored it to improve efficiency during search. In contrast, the FAISS implementation conserves memory by calculating it at query time, taking a different tradeoff between memory usage and query speed. 

### Hardware Acceleration

Modern CPUs offer specialized instructions that can significantly accelerate binary operations. We tailored the distance computation kernel to take advantage of modern CPU instructions. Since RaBitQ relies on popcount operations, we created a specialized path in Knowhere that uses the `VPOPCNTDQ` instructions for AVX512 when available. On supported hardware (e.g., Intel IceLake or AMD Zen 4), this can accelerate binary distance computations by several factors compared to default implementations.


### Query Optimization

Both Knowhere (Milvus’s search engine) and our optimized FAISS version support scalar quantization (SQ1–SQ8) on query vectors. This provides additional flexibility: even with 4-bit query quantization, recall remains high while computational demands decrease significantly, which is particularly useful when queries must be processed at high throughput. 

We go a step further in optimizing our proprietary Cardinal engine, which powers the fully managed Milvus on Zilliz Cloud. Beyond the capabilities of the open-source Milvus, we introduce advanced enhancements, including integration with a graph-based vector index, additional layers of optimization, and support for Arm SVE instructions.


## The Performance Gain: 3× More QPS with Comparable Accuracy

Starting with version 2.6, Milvus introduces the new `IVF_RABITQ` index type. This new index combines RaBitQ with IVF clustering, random rotation transformation, and optional refinement to deliver an optimal balance of performance, memory efficiency, and accuracy.


### Using IVF_RABITQ in Your Application

Here's how to implement `IVF_RABITQ` in your Milvus application:


```
from pymilvus import MilvusClient

index_params = MilvusClient.prepare_index_params()

index_params.add_index(
    field_name="your_vector_field_name", # Name of the vector field to be indexed
    index_type="IVF_RABITQ", # Will be introduced in Milvus 2.6
    index_name="vector_index", # Name of the index to create
    metric_type="IP", # IVF_RABITQ supports IP and COSINE
    params={
        "nlist": 1024, # IVF param, specifies the number of clusters
    } # Index building params
)
```

### Benchmarking: Numbers Tell the Story

We benchmarked different configurations using[ vdb-bench](https://github.com/zilliztech/vectordbbench), an open-source benchmarking tool for evaluating vector databases. Both the test and control environments use Milvus Standalone deployed on AWS EC2 `m6id.2xlarge` instances. These machines feature 8 vCPUs, 32 GB of RAM, and an Intel Xeon 8375C CPU based on the Ice Lake architecture, which supports the VPOPCNTDQ AVX-512 instruction set.

We used the Search Performance Test from vdb-bench, with a dataset of 1 million vectors, each with 768 dimensions. Since the default segment size in Milvus is 1 GB, and the raw dataset (768 dimensions × 1M vectors × 4 bytes per float) totals roughly 3 GB, the benchmarking involved multiple segments per database.

![](https://assets.zilliz.com/Figure_Example_test_configuration_in_vdb_bench_000142f634.png)

Figure: Example test configuration in vdb-bench.

Here are some low level details about the configuration knobs for IVF, RaBitQ and refinement process: 

- `nlist` and `nprobe` are standard parameters for all `IVF`-based methods

- `nlist` is a non-negative integer that specifies the total number of IVF buckets for the dataset.

- `nprobe` is a non-negative integer that specifies the number of IVF buckets that are visited for a single data vector during the search process. It is a search-related parameter.

- `rbq_bits_query` specifies the level of quantization of a query vector. Use 1..8 values for the `SQ1`..`SQ8` levels of quantization. Use 0 value to disable quantization. It is a search-related parameter.

- `refine`, `refine_type` and `refine_k` parameters are standard parameters for the refine process

- `refine` is a boolean that enables the refinement strategy.

- `refine_k` is a non-negative fp-value. The refining process uses a higher quality quantization method to pick the needed number of nearest neighbors from a `refine_k` times larger pool of candidates, chosen using `IVFRaBitQ`. It is a search-related parameter.

- `refine_type` is a string that specifies the quantization type for a refining index. Available options are `SQ6`, `SQ8`, `FP16`, `BF16` and `FP32` / `FLAT`.

The results reveal important insights:

![](https://assets.zilliz.com/Figure_Cost_and_performance_comparison_of_baseline_IVF_FLAT_IVF_SQ_8_and_IVF_RABITQ_with_different_refinement_strategies_9f69fa449f.png)

Figure: Cost and performance comparison of baseline (IVF_FLAT), IVF_SQ8 and IVF_RABITQ with different refinement strategies

Compared to the baseline `IVF_FLAT` index, which achieves 236 QPS with 95.2% recall, `IVF_RABITQ` reaches significantly higher throughput—648 QPS with FP32 queries and 898 QPS when paired with SQ8-quantized queries. These numbers demonstrate the performance advantage of RaBitQ, especially when refinement is applied.

However, this performance comes with a noticeable trade-off in recall. When `IVF_RABITQ` is used without refinement, recall levels off at around 76%, which may fall short for applications that require high accuracy. That said, achieving this level of recall using 1-bit vector compression is still impressive.

Refinement is essential for recovering accuracy. When configured with SQ8 query and SQ8 refinement, `IVF_RABITQ` delivers both great performance and recall. It maintains a high recall of 94.7%, nearly matching IVF_FLAT, while achieving 864 QPS, over 3× higher than IVF_FLAT. Even compared with another popular quantization index `IVF_SQ8`, `IVF_RABITQ` with SQ8 refinement achieves more than half of the throughput at similar recall, only with a marginal more cost. This makes it an excellent option for scenarios that demand both speed and accuracy.

In short, `IVF_RABITQ` alone is great for maximizing throughput with acceptable recall, and becomes even more powerful when paired with refinement to close the quality gap, using only a fraction of the memory space compared to `IVF_FLAT`.


## Conclusion

RaBitQ marks a significant advancement in vector quantization technology. Combining binary quantization with smart encoding strategies, it achieves what seemed impossible: extreme compression with minimal accuracy loss.

Starting with version 2.6, Milvus will introduce IVF_RABITQ, integrating this powerful compression technique with IVF clustering and refinement strategies to bring binary quantization to production. This combination creates a practical balance between accuracy, speed, and memory efficiency that can transform your vector search workloads.

We're committed to bringing more innovations like this to both open-source Milvus and its fully managed service on Zilliz Cloud, making vector search more efficient and accessible for everyone.


## Getting Started with Milvus 2.6

Milvus 2.6 is available now. In addition to RabitQ, it introduces dozens of new features and performance optimizations such as tiered storage, Meanhash LSH, and enhanced full-text search and multitenancy, directly addressing the most pressing challenges in vector search today: scaling efficiently while keeping costs under control.

Ready to explore everything Milvus 2.6 offers? Dive into our[ release notes](https://milvus.io/docs/release_notes.md), browse the[ complete documentation](https://milvus.io/docs), or check out our[ feature blogs](https://milvus.io/blog). 

If you have any questions or have a similar use case, feel free to reach out to us through our [Discord community](https://discord.com/invite/8uyFbECzPX) or file an issue on[ GitHub](https://github.com/milvus-io/milvus) — we're here to help you make the most of Milvus 2.6.







Bring Vector Compression to the Extreme: How Milvus Serves 3× More Queries with RaBitQ



Many people think vector search is simply about implementing an ANN (Approximate Nearest Neighbor) algorithm and calling it a day. But if you run vector search in production, you know the truth: it gets complicated fast.

Imagine you’re building a product search engine. A user might ask, “_Show me shoes similar to this photo, but only in red and under $100_.” Serving this query requires applying a metadata filter to the semantic similarity search results. Sounds as simple as applying a filter after your vector search returns? Well, not quite.

What happens when your filtering condition is highly selective? You might not return enough results. And simply increasing the vector search's **topK** parameter may quickly degrade performance and consume significantly more resources to handle the same search volume.

![](https://assets.zilliz.com/Show_me_shoes_similar_to_this_photo_but_only_in_red_and_under_100_0862a41a60.png)

Under the hood, efficient metadata filtering is pretty challenging. Your vector database needs to scan the graph index, apply metadata filters, and still respond within a tight latency budget, say, 20 milliseconds. Serving thousands of such queries per second without going bankrupt requires thoughtful engineering and careful optimization.

This blog explores popular filtering techniques in vector search, along with the innovative optimizations we built into the [Milvus](https://milvus.io/docs/overview.md) vector database and its fully managed cloud service ([Zilliz Cloud](https://zilliz.com/cloud)). We’ll also share a benchmark test demonstrating how much more performance the fully-managed Milvus can achieve with a $1000 cloud budget over the other vector databases.

## Graph Index Optimization

Vector databases need efficient indexing methods to handle large datasets. Without indexes, a database must compare your query against every vector in the dataset (brute-force scanning), which becomes extremely slow as your data grows.

**Milvus** supports various index types to solve this performance challenge. The most popular ones are graph-based index types: HNSW (runs entirely in memory) and DiskANN (efficiently uses both memory and SSD). These indexes organize vectors into a network structure where neighborhoods of vectors are connected on a map, allowing searches to quickly navigate to relevant results while checking only a small fraction of all vectors. **Zilliz Cloud**, the fully-managed Milvus service, takes one step further by introducing Cardinal, an advanced proprietary vector search engine, further enhancing these indexes for even better performance.

However, when we add filtering requirements (like "only show products less than $100"), a new problem emerges. The standard approach is creating a _bitset_ - a list marking which vectors meet the filter criteria. During search, the system only considers vectors marked as valid in this bitset. This approach seems logical, but it creates a serious problem: **broken connectivity**. When many vectors get filtered out, the carefully constructed paths in our graph index get disrupted.

Here's a simple example of the problem: In the diagram below, Point A connects to B, C and D, but B, C, and D don't directly connect to each other. If our filter removes point A (perhaps it’s too expensive), then even if B, C, and D are relevant to our search, the path between them is broken. This creates "islands" of disconnected vectors that become unreachable during search, hurting the quality of results (recall).

![](https://assets.zilliz.com/simple_example_of_the_problem_0f09b36639.png)

There are two common approaches to filtering during graph traversal: exclude all filtered-out points upfront, or include everything and apply the filter afterward. As illustrated in the diagram below, neither approach is ideal. Skipping filtered points entirely can cause recall to collapse as the filtering ratio nears 1 (blue line), while visiting every point regardless of its metadata bloats the search space and slows down performance significantly (red line).

![](https://assets.zilliz.com/Filtering_ratio_911e32783b.png)

Researchers have proposed several approaches to strike a balance between recall and performance:

1. **Alpha Strategy:** This introduces a probabilistic approach: even though a vector doesn't match the filter, we might still visit it during search with some probability. This probability (alpha) depends on the filtering ratio - how strict the filter is. This helps maintain essential connections in the graph without visiting too many irrelevant vectors.

2) **ACORN Method [1]:** In standard HNSW, edge pruning is used during index construction to create a sparse graph and speed up search. The ACORN method deliberately skips this pruning step to retain more edges and strengthen connectivity—crucial when filters might exclude many nodes. In some cases, ACORN also expands each node’s neighbor list by gathering additional approximate nearest neighbors, further reinforcing the graph. Moreover, its traversal algorithm looks two steps ahead (i.e., examines neighbors of neighbors), improving the chances of finding valid paths even under high filtering ratios.

3. **Dynamically Selected Neighbors:** A method improves over Alpha Strategy. Instead of relying on probabilistic skipping, this approach adaptively selects the next nodes during search. It offers more control than Alpha Strategy.

In Milvus, we implemented the Alpha strategy alongside other optimization techniques. For example, it dynamically switches strategies when detecting extremely selective filters: when, say, approximately 99% of the data doesn't match the filtering expression, the "include-all" strategy would cause graph traversal paths to lengthen significantly, resulting in performance degradation and isolated "islands" of data. In such cases, Milvus automatically falls back to a brute-force scan, bypassing the graph index entirely for better efficiency. In Cardinal, the vector search engine powering fully-managed Milvus (Zilliz Cloud), we've taken this further by implementing a dynamic combination of "include-all" and "exclude-all" traversal methods that intelligently adapts based on data statistics to optimize query performance.

Our experiments on the Cohere 1M dataset (dimension = 768) using an AWS r7gd.4xlarge instance demonstrate the effectiveness of this approach. In the chart below, the blue line represents our dynamic combination strategy, while the red line illustrates the baseline approach that traverses all filtered points in the graph.

![](https://assets.zilliz.com/Graph_2_067a13500b.png)

## Metadata-Aware Indexing

Another challenge comes from how metadata and vector embeddings relate to each other. In most applications, an item's metadata properties (e.g., a product’s price) have minimal connection to what the vector actually represents (the semantic meaning or visual features). For example, a $90 dress and a $90 belt share the same price point but exhibit completely different visual characteristics. This disconnect makes combining filtering with vector search inherently inefficient.

To solve this problem, we've developed **metadata-aware vector indexes**. Instead of having just one graph for all vectors, it builds specialized "subgraphs" for different metadata values. For example, if your data has fields for "color" and "shape," it creates separate graph structures for these fields.

When you search with a filter like "color = blue," it uses the color-specific subgraph rather than the main graph. This is much faster because the subgraph is already organized around the metadata you're filtering by.

In the figure below, the main graph index is called the **base graph**, while the specialized graphs built for specific metadata fields are called **column graphs**. To manage memory usage effectively, it limits how many connections each point can have (out-degree). When a search doesn't include any metadata filters, it defaults to the base graph. When filters are applied, it switches to the appropriate column graph, offering a significant speed advantage.

![](https://assets.zilliz.com/Metadata_Aware_Indexing_7c3e0707d9.png)

## Iterative Filtering

Sometimes the filtering itself becomes the bottleneck, not the vector search. This happens especially with complex filters like JSON conditions or detailed string comparisons. The traditional approach (filter first, then search) can be extremely slow because the system has to evaluate these expensive filters on potentially millions of records before even starting the vector search.

You might think: "Why not do vector search first, then filter the top results?" This approach works sometimes, but has a major flaw: if your filter is strict and filters out most results, you might end up with too few (or zero) results after filtering.

To solve this dilemma, we created **Iterative Filtering** in Milvus and Zilliz Cloud, inspired by[ VBase](https://www.usenix.org/system/files/osdi23-zhang-qianxi_1.pdf). Instead of an all-or-nothing approach, Iterative Filtering works in batches:

1. Get a batch of the closest vector matches

2. Apply filters to this batch

3. If we don't have enough filtered results, get another batch

4. Repeat until we have the required number of results

![](https://assets.zilliz.com/Iterative_Filtering_b65a057559.png)

This approach dramatically reduces how many expensive filter operations we need to perform while still ensuring we get enough high-quality results. For more information on enabling iterative filtering, please refer to this [iterative filtering doc page](https://docs.zilliz.com/docs/filtered-search#iterative-filtering).

## External Filtering

Many real-world applications split their data across different systems - vectors in a vector database and metadata in traditional databases. For example, many organizations store product descriptions and user reviews as vectors in Milvus for semantic search, while keeping inventory status, pricing, and other structured data in traditional databases like PostgreSQL or MongoDB.

This separation makes sense architecturally but creates a challenge for filtered searches. The typical workflow becomes:

- Query your relational database for records matching filter criteria (e.g., "in-stock items under $50")

- Get the matching IDs and send them to Milvus to filter the vector search

- Perform semantic search only on vectors that match these IDs

This sounds simple—but when the number of rows grows beyond millions, it becomes a bottleneck. Transferring large lists of IDs consumes network bandwidth, and executing massive filter expressions in Milvus adds overhead.

To address this, we introduced **External Filtering** in Milvus, a lightweight SDK-level solution that uses the search iterator API and reverses the traditional workflow.

- Performs vector search first, retrieving batches of the most semantically relevant candidates

- Applies your custom filter function to each batch on the client side

- Automatically fetches more batches until you have enough filtered results

This batched, iterative approach significantly reduces both network traffic and processing overhead, since you're only working with the most promising candidates from the vector search.

Here's an example of how to use External Filtering in pymilvus:

```
vector_to_search = rng.random((1, DIM), np.float32)
expr = f"10 <= {AGE} <= 25"
valid_ids = [1, 12, 123, 1234]

def external_filter_func(hits: Hits):
    return list(filter(lambda hit: hit.id in valid_ids, hits))

search_iterator = milvus_client.search_iterator(
    collection_name=collection_name,
    data=vector_to_search,
    batch_size=100,
    anns_field=PICTURE,
    filter=expr,
    external_filter_func=external_filter_func,
    output_fields=[USER_ID, AGE]
)

while True:
    res = search_iterator.next()
    if len(res) == 0:
        search_iterator.close()
        break
    for i in range(len(res)):
        print(res[i])
```

Unlike Iterative Filtering, which operates on segment-level iterators, External Filtering works at the global query level. This design minimizes metadata evaluation and avoids executing large filters within Milvus, resulting in leaner and faster end-to-end performance.

## AutoIndex

Vector search always involves a tradeoff between accuracy and speed - the more vectors you check, the better your results but the slower your query. When you add filters, this balance becomes even trickier to get right.

In Zilliz Cloud, we've created **AutoIndex** - an ML-based optimizer that automatically fine-tunes this balance for you. Instead of manually configuring complex parameters, AutoIndex uses machine learning to determine the optimal settings for your specific data and query patterns.

To understand how this works, it helps to know a bit about Milvus's architecture since Zilliz is built on top of Milvus: Queries are distributed across multiple QueryNode instances. Each node handles a portion of your data (a segment), performs its search, and then results are merged together.

AutoIndex analyzes statistics from these segments and makes intelligent adjustments. For low filtering ratio, the index query range is widened to increase recall. For high filtering ratio, the query range is narrowed to avoid wasted effort on unlikely candidates. These decisions are guided by statistical models that predict the most effective search strategy for each specific filtering scenario.

AutoIndex goes beyond indexing parameters. It also helps select the best filter evaluation strategy. By parsing filter expressions and sampling segment data, it can estimate evaluation cost. If it detects high evaluation costs, it automatically switches to more efficient techniques such as Iterative Filtering. This dynamic adjustment ensures you're always using the best-fit strategy for each query.

![](https://assets.zilliz.com/Autoindex_3f37988d5c.png)

## Performance on a $1,000 Budget

While theoretical improvements are important, real-world performance is what matters to most developers. We wanted to test how these optimizations translate to actual application performance under realistic budget constraints.

We benchmarked several vector database solutions with a practical $1,000 monthly budget - a reasonable amount that many companies would allocate to vector search infrastructure. For each solution, we selected the highest-performing instance configuration possible within this budget constraint.

Our testing used:

- The Cohere 1M dataset with 1 million 768-dimensional vectors

- A mix of real-world filtered and unfiltered search workloads

- The open-source vdb-bench benchmark tool for consistent comparisons

The competing solutions (anonymized as "VDB A," "VDB B," and "VDB C") were all configured optimally within the budget. The results showed that fully-managed Milvus (Zilliz Cloud) consistently achieved the highest throughput across both filtered and unfiltered queries. With the same $1000 budge, our optimization techniques deliver the most performance at competitive recall.

![](https://assets.zilliz.com/Performance_on_a_1_000_Budget_5ebefaec48.png)

## Conclusion

Vector search with filtering might look simple on the surface - just add a filter clause to your query and you're done. However, as we've demonstrated in this blog, achieving both high performance and accurate results at scale requires sophisticated engineering solutions. Milvus and Zilliz Cloud address these challenges through several innovative approaches:

- **Graph Index Optimization**: Preserves paths between similar items even when filters remove connecting nodes, preventing the "islands" problem that reduces result quality.

- **Metadata-Aware Indexing**: Creates specialized paths for common filter conditions, making filtered searches significantly faster without sacrificing accuracy.

- **Iterative Filtering**: Processes results in batches, applying complex filters only to the most promising candidates instead of the entire dataset.

- **AutoIndex**: Uses machine learning to automatically tune search parameters based on your data and queries, balancing speed and accuracy without manual configuration.

- **External Filtering**: Bridges vector search with external databases efficiently, eliminating network bottlenecks while maintaining result quality.

Milvus and Zilliz Cloud continue to evolve with new capabilities that further improve filtered search performance. Features like[ Partition Key](https://docs.zilliz.com/docs/use-partition-key) allow for even more efficient data organization based on filtering patterns, and advanced subgraph routing techniques are pushing performance boundaries even further.

The volume and complexity of unstructured data continue to grow exponentially, creating new challenges for search systems everywhere. Our team is constantly pushing the boundaries of what's possible with vector databases to deliver faster, more scalable AI-powered search.

If your applications are hitting performance bottlenecks with filtered vector search, we invite you to join our active developer community at [milvus.io/community](https://milvus.io/community) - where you can share challenges, access expert guidance, and discover emerging best practices.

## References

1. <https://arxiv.org/pdf/2403.04871>

2. <https://www.usenix.org/system/files/osdi23-zhang-qianxi_1.pdf>

---


Vector Search in the Real World: How to Filter Efficiently Without Killing Recall



![](https://assets.zilliz.com/build_RAG_with_qwen_3_and_milvus_64b9f2ad4d.jpeg)


As a developer constantly seeking practical AI tools, I couldn't ignore the buzz surrounding Alibaba Cloud's latest release: the[ Qwen 3](https://qwenlm.github.io/blog/qwen3/) model family, a robust lineup of eight hybrid inference models designed to redefine the balance between intelligence and efficiency. In just 12 hours, the project garnered **over 17,000 GitHub stars** and reached a peak of **23,000 downloads** per hour on Hugging Face.

So what's different this time? In short, Qwen 3 models combine both reasoning (slow, thoughtful responses) and non-reasoning (fast, efficient answers) in a single architecture, include diverse model options, enhanced training and performance, and deliver more enterprise-ready features.

In this post, I'll summarize the key capabilities of Qwen 3 models you should pay attention to and guide you through a process of pairing Qwen 3 with Milvus to build a local, cost-aware retrieval-augmented generation (RAG) system—complete with hands-on code and tips for optimizing performance versus latency.

Let's dive in.


## What's Exciting About Qwen 3?

After testing and digging into Qwen 3, it’s clear that it’s not just about bigger numbers on a spec sheet. It’s about how the model’s design choices actually help developers build better GenAI applications — faster, smarter, and with more control. Here’s what stands out.


### 1. Hybrid Thinking Modes: Smart When You Need Them, Speed When You Don’t

One of the most innovative features in Qwen 3 is its **hybrid inference architecture**. It gives you fine-grained control over how much “thinking” the model does on a task-by-task basis. Not all tasks need complicated reasoning, after all. 

- For complex problems that require deep analysis, you can tap into full reasoning power — even if it’s slower.

- For everyday simple queries, you can switch into a faster, lighter mode.

- You can even set a **“thinking budget”** — capping how much compute or tokens a session burns.

This isn't just a lab feature either. It directly addresses the daily trade-off developers juggle: delivering high-quality responses without blowing up infrastructure costs or user latency.


### 2. A Versatile Lineup: MoE and Dense Models for Different Needs

Qwen 3 provides a wide range of models designed to match different operational needs:

- **Two MoE (Mixture of Experts) models**:

  - **Qwen3-235B-A22B**: 235 billion total parameters, 22 billion active per query

  - **Qwen3-30B-A3B**: 30 billion total, 3 billion active

- **Six Dense models**: ranging from a nimble 0.6B to a hefty 32B parameters

_Quick tech background: Dense models (like GPT-3 or BERT) always activate all parameters, making them heavier but sometimes more predictable._ [_MoE models_](https://zilliz.com/learn/what-is-mixture-of-experts) _activate only a fraction of the network at a time, making them much more efficient at scale._

In practice, this versatile lineup of models means you can:

- Use dense models for tight, predictable workloads (like embedded devices)

- Use MoE models when you need heavyweight capabilities without melting your cloud bill

With this range, you can tailor your deployment — from lightweight, edge-ready setups to powerful cloud-scale deployments — without being locked into a single model type.


### 3. Focused on Efficiency and Real-World Deployment

Instead of focusing solely on scaling model size, Qwen 3 focuses on training efficiency and deployment practicality:

- **Trained on 36 trillion tokens** — double what Qwen 2.5 used

- **Expanded to 235B parameters** — but smartly managed through MoE techniques, balancing capability with resource demands.

- **Optimized for deployment** — dynamic quantization (down from FP4 to INT8) lets you run even the biggest Qwen 3 model on modest infrastructure — for example, deployment on four H20 GPUs.

The goal here is clear: deliver stronger performance without requiring disproportionate infrastructure investment.


### 4. Built for Real Integration: MCP Support and Multilingual Capabilities

Qwen 3 is designed with integration in mind, not just isolated model performance:

- **MCP (Model Context Protocol) compatibility** enables seamless integration with external databases, APIs, and tools, reducing engineering overhead for complex applications.

- **Qwen-Agent** enhances tool calling and workflow orchestration, supporting the building of more dynamic, actionable AI systems.

- **Multilingual support across 119 languages and dialects** makes Qwen 3 a strong choice for applications targeting global and multilingual markets.

These features collectively make it easier for developers to build production-grade systems without needing extensive custom engineering around the model.


## Qwen 3 Now Supported in DeepSearcher

[DeepSearcher](https://github.com/zilliztech/deep-searcher) is an open-source project for deep retrieval and report generation, designed as a local-first alternative to OpenAI’s Deep Research. It helps developers build systems that surface high-quality, context-aware information from private or domain-specific data sources.

DeepSearcher now supports Qwen 3’s hybrid inference architecture, allowing developers to toggle reasoning dynamically — applying deeper inference only when it adds value, and skipping it when speed is more important.

Under the hood, DeepSearcher integrates with[ Milvus](https://milvus.io), a high-performance vector database developed by Zilliz engineers, to provide fast and accurate semantic search over local data. Combined with model flexibility, it gives developers greater control over system behavior, cost, and user experience.


## Hands-on Tutorial: Building a RAG System with Qwen 3 and Milvus

Let's put these Qwen 3 models to work by building a RAG system using the Milvus vector database.


### Set up the environment.

```
# Install required packages
pip install --upgrade pymilvus openai requests tqdm
# Set up API key
import os
os.environ["DASHSCOPE_API_KEY"] = "YOUR_DASHSCOPE_API_KEY" # Get this from Alibaba Cloud DashScope
```

Note: You will need to obtain the API Key from Alibaba Cloud.


### Data Preparation

We will use the Milvus documentation pages as our primary knowledge base.

```
# Download and extract Milvus documentation
!wget https://github.com/milvus-io/milvus-docs/releases/download/v2.4.6-preview/milvus_docs_2.4.x_en.zip
!unzip -q milvus_docs_2.4.x_en.zip -d milvus_docs

# Load and parse the markdown files
from glob import glob

text_lines = []
for file_path in glob("milvus_docs/en/faq/*.md", recursive=True):
    with open(file_path, "r") as file:
        file_text = file.read()
        text_lines += file_text.split("# ")
```


### Setting Up Models

We'll use DashScope's OpenAI-compatible API to access Qwen 3:

```
# Set up OpenAI client to access Qwen 3
from openai import OpenAI

openai_client = OpenAI(
    base_url="https://dashscope.aliyuncs.com/compatible-mode/v1",
    api_key=os.getenv("DASHSCOPE_API_KEY")
)

# Set up embedding model
from pymilvus import model as milvus_model
embedding_model = milvus_model.DefaultEmbeddingFunction()
```


Let's generate a test embedding and print its dimensions and first few elements:

```
test_embedding = embedding_model.encode_queries(["This is a test"])[0]
embedding_dim = len(test_embedding)
print(embedding_dim)
print(test_embedding[:10])
```


Output:

```
768
[-0.04836066 0.07163023 -0.01130064 -0.03789345 -0.03320649 -0.01318448
 -0.03041712 -0.02269499 -0.02317863 -0.00426028]
```


### Creating a Milvus Collection

Let's set up our Milvus vector database:

```
from pymilvus import MilvusClient

# Initialize Milvus client (using local storage for simplicity)
milvus_client = MilvusClient(uri="./milvus_demo.db")
collection_name = "my_rag_collection"

# Create a fresh collection
if milvus_client.has_collection(collection_name):
    milvus_client.drop_collection(collection_name)

# Create a new collection
test_embedding = embedding_model.encode_queries(["This is a test"])[0]
embedding_dim = len(test_embedding)
milvus_client.create_collection(
    collection_name=collection_name,
    dimension=embedding_dim,
    metric_type="IP", # Inner product distance
    consistency_level="Strong", # Strong consistency level
)
```



About MilvusClient parameter settings:

- Setting the URI to a local file (e.g., `./milvus.db`) is the most convenient method as it automatically uses Milvus Lite to store all data in that file.

- For large-scale data, you can set up a more powerful Milvus server on Docker or Kubernetes. In this case, use the server's URI (e.g., `http://localhost:19530`) as your URI.

- If you want to use [Zilliz Cloud ](https://zilliz.com/cloud)(the managed service of Milvus), adjust the URI and token, which correspond to the Public Endpoint and API key in Zilliz Cloud.


### Adding Documents to the Collection

Now we'll create embeddings for our text chunks and add them to Milvus:

```
from tqdm import tqdm

data = []
doc_embeddings = embedding_model.encode_documents(text_lines)

for i, line in enumerate(tqdm(text_lines, desc="Creating embeddings")):
    data.append({"id": i, "vector": doc_embeddings[i], "text": line})

milvus_client.insert(collection_name=collection_name, data=data)
```

Output:

```
Creating embeddings: 100%|██████████████████████████████████████████████████| 72/72 [00:00<00:00, 381300.36it/s]
{'insert_count': 72, 'ids': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71], 'cost': 0}
```



### Building the RAG Query System

Now for the exciting part - let's set up our RAG system to answer questions.

Let's specify a common question about Milvus:

```
question = "How is data stored in milvus?"
```


Search for this question in the collection and retrieve the top 3 semantically matching results:

```
search_res = milvus_client.search(
    collection_name=collection_name,
    data=embedding_model.encode_queries([question]), # Convert the question to an embedding vector
    limit=3, # Return top 3 results
    search_params={"metric_type": "IP", "params": {}}, # Inner product distance
    output_fields=["text"], # Return the text field
)
```


Let's look at the search results for this query:

```
import json
retrieved_lines_with_distances = [
    (res["entity"]["text"], res["distance"]) for res in search_res[0]
]
print(json.dumps(retrieved_lines_with_distances, indent=4))
```


Output:

```
[
    [
        " Where does Milvus store data?\n\nMilvus deals with two types of data, inserted data and metadata. \n\nInserted data, including vector data, scalar data, and collection-specific schema, are stored in persistent storage as incremental log. Milvus supports multiple object storage backends, including [MinIO](https://min.io/), [AWS S3](https://aws.amazon.com/s3/?nc1=h_ls), [Google Cloud Storage](https://cloud.google.com/storage?hl=en#object-storage-for-companies-of-all-sizes) (GCS), [Azure Blob Storage](https://azure.microsoft.com/en-us/products/storage/blobs), [Alibaba Cloud OSS](https://www.alibabacloud.com/product/object-storage-service), and [Tencent Cloud Object Storage](https://www.tencentcloud.com/products/cos) (COS).\n\nMetadata are generated within Milvus. Each Milvus module has its own metadata that are stored in etcd.\n\n###",
        0.6572665572166443
    ],
    [
        "How does Milvus flush data?\n\nMilvus returns success when inserted data are loaded to the message queue. However, the data are not yet flushed to the disk. Then Milvus' data node writes the data in the message queue to persistent storage as incremental logs. If `flush()` is called, the data node is forced to write all data in the message queue to persistent storage immediately.\n\n###",
        0.6312146186828613
    ],
    [
        "How does Milvus handle vector data types and precision?\n\nMilvus supports Binary, Float32, Float16, and BFloat16 vector types.\n\n- Binary vectors: Store binary data as sequences of 0s and 1s, used in image processing and information retrieval.\n- Float32 vectors: Default storage with a precision of about 7 decimal digits. Even Float64 values are stored with Float32 precision, leading to potential precision loss upon retrieval.\n- Float16 and BFloat16 vectors: Offer reduced precision and memory usage. Float16 is suitable for applications with limited bandwidth and storage, while BFloat16 balances range and efficiency, commonly used in deep learning to reduce computational requirements without significantly impacting accuracy.\n\n###",
        0.6115777492523193
    ]
]
```

### Using the LLM to Build a RAG Response

Convert the retrieved documents to string format:

```
context = "\n".join(
    [line_with_distance[0] for line_with_distance in retrieved_lines_with_distances]
)
```

Provide system prompt and user prompt for the large language model:

```
SYSTEM_PROMPT = """
You are an AI assistant. You are able to find answers to the questions from the contextual passage snippets provided.
"""

USER_PROMPT = f"""
Use the following pieces of information enclosed in <context> tags to provide an answer to the question enclosed in <question> tags.

<context>
{context}
</context>

<question>
{question}
</question>
"""
```

Use the latest Qwen model to generate a response based on the prompt:

```
completion = openai_client.chat.completions.create(
    # Model list: https://help.aliyun.com/zh/model-studio/getting-started/models
    model="qwen-plus-2025-04-28",
    messages=[
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": USER_PROMPT},
    ],
    # Control thinking process with enable_thinking parameter (default True for open-source, False for commercial)
    extra_body={"enable_thinking": False},
)

print(completion.choices[0].message.content)

```
Output:

```
In Milvus, data is stored in two main categories: **inserted data** and **metadata**.

- **Inserted Data**: This includes vector data (like Binary, Float32, Float16, and BFloat16 types), scalar data, and collection-specific schema. These are stored in persistent storage as **incremental logs**. Milvus supports various object storage backends for this purpose, including:

  - [MinIO](https://min.io/)
  - [AWS S3](https://aws.amazon.com/s3/?nc1=h_ls)
  - [Google Cloud Storage (GCS)](https://cloud.google.com/storage?hl=en#object-storage-for-companies-of-all-sizes)
  - [Azure Blob Storage](https://azure.microsoft.com/en-us/products/storage/blobs)
  - [Alibaba Cloud OSS](https://www.alibabacloud.com/product/object-storage-service)
  - [Tencent Cloud Object Storage (COS)](https://www.tencentcloud.com/products/cos)

- **Metadata**: Metadata generated within Milvus is stored separately. Each Milvus module maintains its own metadata, which is stored in [etcd](https://etcd.io/), a distributed key-value store.
When data is inserted into Milvus, it is first loaded into a message queue. It is not immediately written to disk. A `flush()` operation ensures that all data in the queue is written to persistent storage immediately.
```


## Comparing Reasoning vs. Non-Reasoning Modes: A Practical Test

I ran a test comparing the two inference modes on a math problem:

**Problem:** Person A and Person B start running from the same location. A leaves first and runs for 2 hours at 5km/h. B follows at 15km/h. How long will it take B to catch up?

![](https://assets.zilliz.com/math_problem_0123815455.png)

```
import os
import time
from openai import OpenAI

os.environ["DASHSCOPE_API_KEY"] = "sk-****************"
client = OpenAI(
   api_key=os.getenv("DASHSCOPE_API_KEY"),
   base_url="https://dashscope.aliyuncs.com/compatible-mode/v1",
)
############################################
# Think
# Record the start time
start_time = time.time()
stream = client.chat.completions.create(
   # model lists：https://help.aliyun.com/zh/model-studio/getting-started/models
   model="qwen3-235b-a22b",
   # model="qwen-plus-2025-04-28",
   messages=[
       {"role": "system", "content": "You are a helpful assistant."},
       {"role": "user", "content": "A and B depart from the same location. A leaves 2 hours earlier at 5 km/h. B follows at 15 km/h. When will B catch up?"},
   ],
   # You can control the thinking mode through the enable_thinking parameter
   extra_body={"enable_thinking": True},
   stream=True,
)
answer_content = ""
for chunk in stream:
   delta = chunk.choices[0].delta
   if delta.content is not None:
       answer_content += delta.content
      
print(answer_content)

# Record the end time and calculate the total runtime
end_time = time.time()
print(f"\n\nTotal runtime：{end_time - start_time:.2f}seconds")
```


**With reasoning mode enabled:**

- Processing time: \~74.83 seconds

- Deep analysis, problem parsing, multiple solution paths

- High-quality markdown output with formulas

(The image below is a screenshot of the visualization of the model's markdown response, for the reader's convenience)

![](https://assets.zilliz.com/screenshot_with_reasoning_1_d231b6ad54.png)

![](https://assets.zilliz.com/screenshot_with_reasoning_2_394d3bff9f.png)

**Non-Reasoning Mode:**

In the code, you only need to set `"enable_thinking": False`

Results of non-reasoning mode on this problem:

- Processing time: ~74.83 seconds
- Deep analysis, problem parsing, multiple solution paths
- High-quality markdown output with formulas

(The image below is a screenshot of the visualization of the model's markdown response, for the reader's convenience)


![](https://assets.zilliz.com/screenshot_without_reasoning_e1f6b82e56.png)

## Conclusion

Qwen 3 introduces a flexible model architecture that aligns well with the real-world needs of GenAI development. With a range of model sizes (including both dense and MoE variants), hybrid inference modes, MCP integration, and multilingual support, it gives developers more options to tune for performance, latency, and cost, depending on the use case.

Rather than emphasizing scale alone, Qwen 3 focuses on adaptability. This makes it a practical choice for building RAG pipelines, [AI agents](https://zilliz.com/blog/what-exactly-are-ai-agents-why-openai-and-langchain-are-fighting-over-their-definition), and production applications that require both reasoning capabilities and cost-efficient operation.

When paired with infrastructure like[ Milvus](https://milvus.io) — a high-performance open-source vector database — Qwen 3’s capabilities become even more useful, enabling fast, semantic search and smooth integration with local data systems. Together, they offer a strong foundation for intelligent, responsive GenAI applications at scale.


Hands-on with Qwen 3 and Milvus: Building RAG with the Latest Hybrid Inference Models



![](https://assets.zilliz.com/build_rag_with_dify_and_milvus_2ab76c6afd.png)

What if you could turn your entire documentation library—thousands of pages of technical specs, internal wikis, and code documentation—into an intelligent AI assistant that instantly answers specific questions?

Even better, what if you could build it in less time than it takes to fix a merge conflict?

That's the promise of [Retrieval Augmented Generation](https://zilliz.com/learn/Retrieval-Augmented-Generation) (RAG) when implemented the right way.

While ChatGPT and other LLMs are impressive, they quickly hit their limits when asked about your company's specific documentation, codebase, or knowledge base. RAG bridges this gap by integrating your proprietary data into the conversation, providing you with AI capabilities that are directly relevant to your work.

The problem? Traditional RAG implementation looks like this:

- Write custom embedding generation pipelines

- Configure and deploy a vector database

- Engineer complex prompt templates

- Build retrieval logic and similarity thresholds

- Create a usable interface

But what if you could skip straight to the results?

In this tutorial, we'll build a simple RAG application using two developer-focused tools:

- [Dify](https://github.com/langgenius/dify): An open-source platform that handles the RAG orchestration with minimal configuration

- [Milvus](https://milvus.io/docs/overview.md): A blazing-fast open-source vector database purpose-built for similarity search and AI searches 

By the end of this 10-minute guide, you'll have a working AI assistant that can answer detailed questions about any document collection you throw at it - no machine learning degree required.


## What You'll Build

In just a few minutes of active work, you'll create:

- A document processing pipeline that converts any PDF into queryable knowledge

- A vector search system that finds exactly the right information

- A chatbot interface that answers technical questions with pinpoint accuracy

- A deployable solution you can integrate with your existing tools

The best part? Most of it is configured through a simple user interface (UI) instead of custom code.


## What You'll Need

- Basic Docker knowledge (just `docker-compose up -d` level)

- An OpenAI API key

- A PDF document to experiment with (we'll use a research paper)

Ready to build something actually useful in record time? Let's get started!


## Building Your RAG Application with Milvus and Dify 

In this section, we will build a simple RAG app with Dify, where we can ask questions about the information contained in a research paper. For the research paper, you can use any paper you want; however, in this case, we will use the famous paper that introduced us to the Transformer architecture, "[Attention is All You Need](https://arxiv.org/abs/1706.03762)."

We will use Milvus as our vector storage, where we will store all the necessary contexts. For the embedding model and the LLM, we'll use models from OpenAI. Therefore, we need to set up an OpenAI API key first. You can learn more about setting it up[ here](https://platform.openai.com/docs/quickstart).


### Step 1: Starting Dify and Milvus Containers

In this example, we’ll self-host Dify with Docker Compose. Therefore, before we begin, ensure that Docker is installed on your local machine. If you haven't, install Docker by referring to[ its installation page](https://docs.docker.com/desktop/).

Once we have Docker installed, we need to clone the Dify source code into our local machine with the following command:

```
git clone <<https://github.com/langgenius/dify.git>>
```

Next, go to the `docker` directory inside of the source code that you’ve just cloned. There, you need to copy the `.env` file with the following command:

```
cd dify/docker
cp .env.example .env
```

In a nutshell, `.env` file contains the configurations needed to set your Dify app up and running, such as the selection of vector databases, the credentials necessary to access your vector database, the address of your Dify app, etc.

Since we’re going to use Milvus as our vector database, then we need to change the value of `VECTOR_STORE` variable inside `.env` file to `milvus`. Also, we need to change the `MILVUS_URI` variable to `http://host.docker.internal:19530` to ensure that there’s no communication issue between Docker containers later on after deployment.

```
VECTOR_STORE=milvus
MILVUS_URI=http://host.docker.internal:19530
```

Now we are ready to start the Docker containers. To do so, all we need to do is to run the `docker compose up -d` command. After it finishes, you’ll see similar output in your terminal as below:

```
docker compose up -d
```

![](https://assets.zilliz.com/docker_compose_179216f113.png)

We can check the status of all containers and see if they’re up and running healthily with `docker compose ps` command. If they’re all healthy, you’ll see an output as below:

```
docker compose ps
```

![](https://assets.zilliz.com/docker_compose_2_1a084ba137.png)

And finally, if we head up to[ ](http://localhost/install)http://localhost/install, you’ll see a Dify landing page where we can sign up and start building our RAG application in no time.

![](https://assets.zilliz.com/dify_login_d2bf4d4468.png)

Once you’ve signed up, then you can just log into Dify with your credentials.

![](https://assets.zilliz.com/difi_login_2_5a8ea9c2d6.png)


### Step 2: Setting Up OpenAI API Key

The first thing we need to do after signing up for Dify is to set up our API keys that we'll use to call the embedding model as well as the LLM. Since we're going to use models from OpenAI, we need to insert our OpenAI API key into our profile. To do so, go to "Settings" by hovering your cursor over your profile on the top right of the UI, as you can see in the screenshot below:

![](https://assets.zilliz.com/dify_settings_8ff08fab97.png)

Next, go to "Model Provider," hover your cursor on OpenAI, and then click "Setup." You'll then see a pop-up screen where you're prompted to enter your OpenAI API key. Once we're done, we're ready to use models from OpenAI as our embedding model and LLM.

![](https://assets.zilliz.com/dify_model_providers_491b313b12.png)


### Step 3: Inserting Documents into Knowledge Base

Now let's store the knowledge base for our RAG app. The knowledge base consists of a collection of internal documents or texts that can be used as relevant contexts to help the LLM generates more accurate responses.

In our use case, our knowledge base is essentially the "Attention is All You Need" paper. However, we can't store the paper as it is due to multiple reasons. First, the paper is too long, and giving an overly long context to the LLM wouldn't help as the context is too broad. Second, we can't perform similarity searches to fetch the most relevant context if our input is raw text.

Therefore, there are at least two steps we need to take before storing our paper into the knowledge base. First, we need to divide the paper into text chunks, and then transform each chunk into an embedding via an embedding model. Finally, we can store these embeddings into Milvus as our vector database.

Dify makes it easy for us to split the texts in the paper into chunks and turn them into embeddings. All we need to do is upload the PDF file of the paper, set the chunk length, and choose the embedding model via a slider. To do all these steps, go to "Knowledge" and then click "Create Knowledge". Next, you'll be prompted to upload the PDF file from your local computer. Therefore, it's better if you download the paper from ArXiv and save it on your computer first.

![](https://assets.zilliz.com/dify_knowledge_cc21a5c430.png)

Once we've uploaded the file, we can set the chunk length, indexing method, the embedding model we want to use, and retrieval settings.

In the "Chunk Setting" area, you can choose any number as the maximum chunk length (in our use case, we'll set it to 100). Next, for "Index Method," we need to choose the "High Quality" option as it'll enable us to perform similarity searches to find relevant contexts. For "Embedding Model," you can choose any embedding model from OpenAI you want, but in this example, we're going to use the text-embedding-3-small model. Lastly, for "Retrieval Setting," we need to choose "Vector Search" as we want to perform similarity searches to find the most relevant contexts.

![](https://assets.zilliz.com/Dify_save_837dbc0cf6.png)

Now if you click on "Save & Process" and everything goes well, you'll see a green tick appear as shown in the following screenshot:

![](https://assets.zilliz.com/dify_knowledge_created_d46b96385f.png)


### Step 4: Creating the RAG App

Up until this point, we have successfully created a knowledge base and stored it inside our Milvus database. Now we're ready to create the RAG app.

Creating the RAG app with Dify is very straightforward. We need to go to "Studio" instead of “Knowledge” like before, and then click on "Create from Blank." Next, choose "Chatbot" as the app type and give your App a name inside the provided field. Once you're done, click "Create." Now you'll see the following page:

![](https://assets.zilliz.com/dify_create_f5691f193d.png)

Under the "Instruction" field, we can write a system prompt such as "Answer the query from the user concisely." Next, as "Context," we need to click on the "Add" symbol, and then add the knowledge base that we've just created. This way, our RAG app will fetch possible contexts from this knowledge base to answer the user's query.

Now that we've added the knowledge base to our RAG app, the last thing we need to do is choose the LLM from OpenAI. To do so, you can click on the model list available in the upper right corner, as you can see in the screenshot below:

![](https://assets.zilliz.com/dify_llm_c3b79ded37.png)

And now we're ready to publish our RAG application! In the upper right-hand corner, click "Publish," and there you can find many ways to publish our RAG app: we can simply run it in a browser, embed it on our website, or access the app via API. In this example, we'll just run our app in a browser, so we can click on "Run App".

And that's it! Now you can ask the LLM anything related to the "Attention is All You Need" paper or any documents included in our knowledge base.

![](https://assets.zilliz.com/rag_attention_is_all_you_need_8582d3a69a.png)


## Conclusion
You've now built a working RAG application using Dify and Milvus, with minimal code and configuration. This approach makes the complex RAG architecture accessible to developers without requiring deep expertise in vector databases or LLM integration.
Key takeaways:
1. **Low setup overhead**: Using Docker Compose simplifies deployment
2. **No-code/low-code orchestration**: Dify handles most of the RAG pipeline
3. **Production-ready vector database**: Milvus provides efficient embedding storage and retrieval
4. **Extensible architecture**: Easy to add documents or adjust parameters
For production deployment, consider:
- Setting up authentication for your application
- Configuring proper scaling for Milvus (especially for larger document collections)
- Implementing monitoring for your Dify and Milvus instances
- Fine-tuning retrieval parameters for optimal performance
The combination of Dify and Milvus enables the rapid development of RAG applications that can effectively leverage your organization's internal knowledge with modern large language models (LLMs).
Happy building!
## Additional Resources
- [Dify Documentation](https://docs.dify.ai/)
- [Milvus Documentation](https://milvus.io/docs)
- [Vector Database Fundamentals](https://zilliz.com/learn/vector-database)
- [RAG Implementation Patterns](https://zilliz.com/learn/Retrieval-Augmented-Generation)








Build a RAG Document Assistant in 10 Minutes | Dify & Milvus Tutorial


Hands-on Tutorial: Build a RAG-Powered Document Assistant in 10 Minutes with Dify and Milvus



## Introduction

As a developer working with Milvus, you've likely encountered the daunting `milvus.yaml` configuration file with its 500+ parameters. Handling this complexity can be challenging when all you want is to optimize your vector database performance.

Good news: you don't need to understand every parameter. This guide cuts through the noise and focuses on the critical settings that actually impact performance, highlighting exactly which values to tweak for your specific use case.

Whether you're building a recommendation system that needs lightning-fast queries or optimizing a vector search application with cost constraints, I'll show you exactly which parameters to modify with practical, tested values. By the end of this guide, you'll know how to tune Milvus configurations for peak performance based on real-world deployment scenarios.


## Configuration Categories

Before diving into specific parameters, let's break down the structure of the configuration file. When working with `milvus.yaml`, you'll be dealing with three parameter categories:

- **Dependency Component Configurations**: External services Milvus connects to (`etcd`, `minio`, `mq`) - critical for cluster setup and data persistence

- **Internal Component Configurations**: Milvus's internal architecture (`proxy`, `queryNode`, etc.) - key for performance tuning

- **Functional Configurations**: Security, logging, and resource limits - important for production deployments

![](https://assets.zilliz.com/Milvus_Configurations_f9a7e45dce.PNG)


## Milvus Dependency Component Configurations

Let's start with the external services Milvus depends on. These configurations are particularly important when moving from development to production.


### `etcd`: Metadata Store

Milvus relies on `etcd` for metadata persistence and service coordination. The following parameters are crucial:

- `Etcd.endpoints`: Specifies the address of the etcd cluster. By default, Milvus launches a bundled instance, but in enterprise environments, it’s best practice to connect to a managed `etcd` service for better availability and operational control.

- `etcd.rootPath`: Defines the key prefix for storing Milvus-related data in etcd. If you’re operating multiple Milvus clusters on the same etcd backend, using different root paths allows clean metadata isolation.

- `etcd.auth`: Controls authentication credentials. Milvus doesn’t enable etcd auth by default, but if your managed etcd instance requires credentials, you must specify them here.

![](https://assets.zilliz.com/etcd_in_milvusyaml_dc600c6974.PNG)


### `minio`: Object Storage

Despite the name, this section governs all S3-compatible object storage service clients. It supports providers such as AWS S3, GCS, and Aliyun OSS via the `cloudProvider` setting.

Pay attention to these four key configurations: 

- `minio.address / minio.port`: Use these to specify the endpoint of your object storage service.

- `minio.bucketName`: Assign separate buckets (or logical prefixes) to avoid data collisions when running multiple Milvus clusters.

- `minio.rootPath`: Enables intra-bucket namespacing for data isolation.

- `minio.cloudProvider`: Identifies the OSS backend. For a full compatibility list, refer to the [Milvus documentation](https://milvus.io/docs/product_faq.md#Where-does-Milvus-store-data). 

![](https://assets.zilliz.com/minio_in_milvusyaml_faa11c9fcd.png)


### `mq`: Message Queue

Milvus uses a message queue for internal event propagation—either Pulsar (default) or Kafka. Pay attention to the following three parameters. 

1. `pulsar.address/pulsar.port`: Set these values to use an external Pulsar cluster.

2. `pulsar.tenant`: Defines the tenant name. When multiple Milvus clusters share a Pulsar instance, this ensures clean channel separation.

3. `msgChannel.chanNamePrefix.cluster`: If you prefer to bypass Pulsar’s tenant model, adjust the channel prefix to prevent collisions.

![](https://assets.zilliz.com/mq_in_milvusyaml1_2214739c84.png)

![](https://assets.zilliz.com/mq_in_milvusyaml2_a44ff64936.png)

Milvus also supports Kafka as the message queue. To use Kafka instead, comment out the Pulsar-specific settings and uncomment the Kafka config block.

![](https://assets.zilliz.com/mq_in_milvusyaml3_d41f44f77a.png)


## Milvus Internal Component Configurations

### `rootCoord`: Metadata + Timestamps

The `rootCoord` node handles metadata changes (DDL/DCL) and timestamp management.

1. `rootCoord.maxPartitionNum`： Sets the upper bound on the number of partitions per collection. While the hard limit is 1024, this parameter primarily serves as a safeguard. For multi-tenant systems, avoid using partitions as isolation boundaries—instead, implement a tenant key strategy that scales to millions of logical tenants.

2. `rootCoord.enableActiveStandby`：Enables high availability by activating a standby node. This is critical since Milvus coordinator nodes don’t scale horizontally by default.

![](https://assets.zilliz.com/root_Coord_in_milvusyaml_9c2417dbaf.png)


### `proxy`: API Gateway + Request Router

The `proxy` handles client-facing requests, request validation, and result aggregation.

- `proxy.maxFieldNum`: Limits the number of fields (scalar + vector) per collection. Keep this under 64 to minimize schema complexity and reduce I/O overhead.

- `proxy.maxVectorFieldNum`: Controls the number of vector fields in a collection. Milvus supports multimodal search, but in practice, 10 vector fields is a safe upper bound.

- `proxy.maxShardNum`:Defines the number of ingestion shards. As a rule of thumb:

  - < 200M records → 1 shard

  - 200–400M records → 2 shards

  - Scale linearly beyond that

- `proxy.accesslog`: When enabled, this logs detailed request info (user, IP, endpoint, SDK). Useful for auditing and debugging.

![](https://assets.zilliz.com/proxy_in_milvusyaml_897b33c759.png)


### `queryNode`: Query Execution

Handles vector search execution and segment loading. Pay attention to the following parameter. 

- `queryNode.mmap`: Toggles memory-mapped I/O for loading scalar fields and segments. Enabling `mmap` helps reduce memory footprint, but may degrade latency if disk I/O becomes a bottleneck.

![](https://assets.zilliz.com/query_Node_in_milvusyaml_36e1bf378a.png)


### `dataCoord`: Segment + Index Management

This parameter controls data segmentation, indexing, compaction, and garbage collection (GC). Key configuration parameters include:

1. `dataCoord.segment.maxSize`: Specifies the maximum size of an in-memory data segment. Larger segments generally mean fewer total segments in the system, which can improve query performance by reducing indexing and search overhead. For example, some users running `queryNode` instances with 128GB of RAM reported that increasing this setting from 1GB to 8GB led to roughly 4× faster query performance.

2. `dataCoord.segment.diskSegmentMaxSize`: Similar to the above, this parameter controls the maximum size for [disk indexes](https://milvus.io/docs/disk_index.md#On-disk-Index) (diskann index) specifically.

3. `dataCoord.segment.sealProportion`: Determines when a growing segment is sealed (i.e., finalized and indexed). The segment is sealed when it reaches `maxSize * sealProportion`. By default, with `maxSize = 1024MB` and `sealProportion = 0.12`, a segment will be sealed at around 123MB.

- Lower values (e.g., 0.12) trigger sealing sooner, which can help with faster index creation—useful in workloads with frequent updates.

- Higher values (e.g., 0.3 to 0.5) delay sealing, reducing indexing overhead—more suitable for offline or batch ingestion scenarios.

4. `dataCoord.segment.expansionRate`:  Sets the allowed expansion factor during compaction. Milvus calculates the maximum allowable segment size during compaction as `maxSize * expansionRate`.

5. `dataCoord.gc.dropTolerance`: After a segment is compacted or a collection is dropped, Milvus does not immediately delete the underlying data. Instead, it marks the segments for deletion and waits for the garbage collection (GC) cycle to complete. This parameter controls the duration of that delay.

![](https://assets.zilliz.com/data_Coord_in_milvusyaml1_100d98a081.png)

![](https://assets.zilliz.com/data_Coord_in_milvusyaml2_7fa8c5f2c0.png)


## Other Functional Configurations

### `log`: Observability and Diagnostics

Robust logging is a cornerstone of any distributed system, and Milvus is no exception. A well-configured logging setup not only helps with debugging issues as they arise but also ensures better visibility into system health and behavior over time.

For production deployments, we recommend integrating Milvus logs with centralized logging and monitoring tools—such as [Loki](https://milvus.io/docs/configure_grafana_loki.md#Deploy-Loki) —to streamline analysis and alerting. Key settings include:

1. `log.level`: Controls the verbosity of log output. For production environments, stick with `info` level to capture essential runtime details without overwhelming the system. During development or troubleshooting, you can switch to `debug` to get more granular insights into internal operations. ⚠️ Be cautious with `debug` level in production—it generates a high volume of logs, which can quickly consume disk space and degrade I/O performance if left unchecked.

2. `log.file`: By default, Milvus writes logs to standard output (stdout), which is suitable for containerized environments where logs are collected via sidecars or node agents. To enable file-based logging instead, you can configure:

- Maximum file size before rotation

- File retention period

- Number of backup log files to keep

This is useful in bare-metal or on-prem environments where stdout log shipping is not available.

![](https://assets.zilliz.com/log_in_milvusyaml_248ead1264.png)


### `security`: Authentication and Access Control

Milvus supports [user authentication](https://milvus.io/docs/authenticate.md?tab=docker) and [role-based access control (RBAC)](https://milvus.io/docs/rbac.md), both of which are configured under the `common` module. These settings are essential for securing multi-tenant environments or any deployment exposed to external clients.

Key parameters include: 

1. `common.security.authorizationEnabled`: This toggle enables or disables authentication and RBAC. It’s turned off by default, meaning all operations are allowed without identity checks. To enforce secure access control, set this parameter to `true`.

2. `common.security.defaultRootPassword`: When authentication is enabled, this setting defines the initial password for the built-in `root` user.

Be sure to change the default password immediately after enabling authentication to avoid security vulnerabilities in production environments.

![](https://assets.zilliz.com/security_in_milvusyaml_a8d0187b5a.png)


### `quotaAndLimits`: Rate Limiting and Write Control

The `quotaAndLimits` section in `milvus.yaml` plays a critical role in controlling how data flows through the system. It governs rate limits for operations like inserts, deletes, flushes, and queries—ensuring cluster stability under heavy workloads and preventing performance degradation due to write amplification or excessive compaction.

Key parameters include:

`quotaAndLimits.flushRate.collection`: Controls how frequently Milvus flushes data from a collection.

- **Default value**: `0.1`, which means the system allows one flush every 10 seconds.

- The flush operation seals a growing segment and persists it from the message queue to object storage.

- Flushing too frequently can generate many small sealed segments, which increases compaction overhead and hurts query performance.

💡 Best practice: In most cases, let Milvus handle this automatically. A growing segment is sealed once it reaches `maxSize * sealProportion`, and sealed segments are flushed every 10 minutes. Manual flushes are only recommended after bulk inserts when you know no more data is coming.

Also keep in mind: **data visibility** is determined by the query’s _consistency level_, not the flush timing—so flushing does not make new data immediately queryable.

![](https://assets.zilliz.com/quota_And_Limits1_be185e571f.png)

`quotaAndLimits.upsertRate`/`quotaAndLimits.deleteRate`: These parameters define the maximum allowed rate for upsert and delete operations.

- Milvus relies on an LSM-Tree storage architecture, which means frequent updates and deletions trigger compaction. This can be resource-intensive and reduce overall throughput if not managed carefully.

- It's recommended to cap both `upsertRate` and `deleteRate` at **0.5 MB/s** to avoid overwhelming the compaction pipeline.

🚀 Need to update a large dataset quickly? Use a collection alias strategy:

- Insert new data into a fresh collection.

- Once the update is complete, repoint the alias to the new collection. This avoids the compaction penalty of in-place updates and allows instant switchover.

![](https://assets.zilliz.com/quota_And_Limits2_32c8640190.png)


## Real-World Configuration Examples

Let’s walk through two common deployment scenarios to illustrate how Milvus configuration settings can be tuned to match different operational goals.


### ⚡ Example 1: High-Performance Configuration

When query latency is mission-critical—think recommendation engines, semantic search platforms, or real-time risk scoring—every millisecond counts. In these use cases, you'll typically lean on graph-based indexes like **HNSW** or **DISKANN**, and optimize both memory usage and segment lifecycle behavior.

Key tuning strategies: 

- Increase `dataCoord.segment.maxSize` and `dataCoord.segment.diskSegmentMaxSize`: Raise these values to 4GB or even 8GB, depending on available RAM. Larger segments reduce the number of index builds and improve query throughput by minimizing segment fanout. However, larger segments do consume more memory at query time—so make sure your `indexNode` and `queryNode` instances have enough headroom.

- Lower `dataCoord.segment.sealProportion` and `dataCoord.segment.expansionRate`: Target a growing segment size around 200MB before sealing. This keeps segment memory usage predictable and reduces the burden on the Delegator (the queryNode leader that coordinates distributed search).

Rule of thumb: Favor fewer, larger segments when memory is abundant and latency is a priority. Be conservative with seal thresholds if index freshness matters.


### 💰 Example 2: Cost-Optimized Configuration

If you're prioritizing cost efficiency over raw performance—common in model training pipelines, low-QPS internal tools, or long-tail image search—you can trade off recall or latency to significantly reduce infrastructure demands.

Recommended strategies:

- **Use index quantization:** Index types like `SCANN`, `IVF_SQ8`, or `HNSW_PQ/PRQ/SQ` (introduced in Milvus 2.5) dramatically reduce index size and memory footprint. These are ideal for workloads where precision is less critical than scale or budget.

- **Adopt a disk-backed indexing strategy:** Set the index type to `DISKANN` to enable pure disk-based search. **Enable** `mmap` for selective memory offloading.

![](https://assets.zilliz.com/query_Node_in_milvusyaml_36e1bf378a.png)

For extreme memory savings, enable `mmap` for the following: `vectorField`, `vectorIndex`, `scalarField`, and `scalarIndex`. This offloads large chunks of data to virtual memory, reducing resident RAM usage significantly.

⚠️ Caveat: If scalar filtering is a major part of your query workload, consider disabling `mmap` for `vectorIndex` and `scalarIndex`. Memory mapping can degrade scalar query performance in I/O-constrained environments.


#### Disk usage tip

- HNSW indexes built with `mmap` can expand total data size by up to **1.8×**.

- A 100GB physical disk might realistically only accommodate ~50GB of effective data when you account for index overhead and caching.

- Always provision extra storage when working with `mmap`, especially if you also cache the original vectors locally.


## Conclusion

Tuning Milvus isn’t about chasing perfect numbers—it’s about shaping the system around your workload’s real-world behavior. The most impactful optimizations often come from understanding how Milvus handles I/O, segment lifecycle, and indexing under pressure. These are the paths where misconfiguration hurts the most—and where thoughtful tuning yields the biggest returns.

If you're new to Milvus, the configuration parameters we've covered will cover 80–90% of your performance and stability needs. Start there. Once you’ve built some intuition, dig deeper into the full `milvus.yaml` spec and the official documentation—you’ll uncover fine-grained controls that can take your deployment from functional to exceptional.

With the right configurations in place, you’ll be ready to build scalable, high-performance vector search systems that align with your operational priorities—whether that means low-latency serving, cost-efficient storage, or high-ingest analytical workloads.


The Developer’s Guide to Milvus Configuration




## TL;DR

You spoke, and we listened! Milvus SDK v2 is a complete reimagining of our developer experience, built directly from your feedback. With a unified API across Python, Java, Go, and Node.js, native async support that you've been asking for, a performance-boosting Schema Cache, and a simplified MilvusClient interface, Milvus SDK v2 makes [vector search](https://zilliz.com/learn/vector-similarity-search) development faster and more intuitive than ever. Whether you're building [RAG](https://zilliz.com/learn/Retrieval-Augmented-Generation) applications, recommendation systems, or [computer vision](https://zilliz.com/learn/what-is-computer-vision) solutions, this community-driven update will transform how you work with Milvus.


## Why We Built It: Addressing Community Pain Points

Over the years, Milvus has become the [vector database](https://milvus.io/blog/what-is-a-vector-database.md) of choice for thousands of AI applications. However, as our community grew, we consistently heard about several limitations with our SDK v1:

**"Handling high concurrency is too complex."** The lack of native async support in some language SDKs forced developers to rely on threads or callbacks, making code harder to manage and debug, especially in scenarios like batch data loading and parallel queries.

**"Performance degrades with scale."** Without a Schema Cache, v1 repeatedly validated schemas during operations, creating bottlenecks for high-volume workloads. In use cases requiring massive vector processing, this problem resulted in increased latency and reduced throughput.

**"Inconsistent interfaces between languages create a steep learning curve."** Different language SDKs implemented interfaces in their own ways, complicating cross-language development.

**"The RESTful API is missing essential features."** Critical functionalities like partition management and index construction were unavailable, forcing developers to switch between different SDKs.

These weren't just feature requests — they were real obstacles in your development workflow. SDK v2 is our promise to remove these barriers and let you focus on what matters: building amazing AI applications.


## The Solution: Milvus SDK v2

Milvus SDK v2 is the result of a complete redesign focused on developer experience, available across multiple languages:

- [Python SDK v2 (pymilvus.MilvusClient)](https://milvus.io/api-reference/pymilvus/v2.5.x/About.md)

- [Java v2](https://github.com/milvus-io/milvus-sdk-java)

- [Go v2](https://github.com/milvus-io/milvus/tree/client/v2.5.1/client)

- [NodeJS](https://github.com/milvus-io/milvus-sdk-node)

- [RESTful v2](https://milvus.io/api-reference/restful/v2.5.x/About.md)


### 1. Native Asynchronous Support: From Complex to Concurrent

The old way of handling concurrency involved cumbersome Future objects and callback patterns. SDK v2 introduces true async/await functionality, particularly in Python with `AsyncMilvusClient` (since v2.5.3). With the same parameters as the synchronous MilvusClient, you can easily run operations like insert, query, and search in parallel.

This simplified approach replaces the old cumbersome Future and callback patterns, leading to cleaner and more efficient code. Complex concurrent logic, like batch vector inserts or parallel multi-queries, can now be effortlessly implemented using tools like `asyncio.gather`.


### 2. Schema Cache: Boosting Performance Where It Counts

SDK v2 introduces a Schema Cache that stores collection schemas locally after the initial fetch, eliminating repeated network requests and CPU overhead during operations.

For high-frequency insert and query scenarios, this update translates to:

- Reduced network traffic between client and server

- Lower latency for operations

- Decreased server-side CPU usage

- Better scaling under high concurrency

This is particularly valuable for applications like real-time recommendation systems or live search features where milliseconds matter.


### 3. A Unified and Streamlined API Experience

Milvus SDK v2 introduces a unified and more complete API experience across all supported programming languages. Particularly, the RESTful API has been significantly enhanced to offer near feature parity with the gRPC interface.

In earlier versions, the RESTful API lagged behind gRPC, limiting what developers could do without switching interfaces. That's no longer the case. Now, developers can use the RESTful API to perform virtually all core operations—such as creating collections, managing partitions, building indexes, and running queries—without needing to fall back on gRPC or other methods.

This unified approach ensures a consistent developer experience across different environments and use cases. It reduces the learning curve, simplifies integration, and improves overall usability.

Note: For most users, the RESTful API offers a faster and easier way to get started with Milvus. However, if your application demands high-performance or advanced features like iterators, the gRPC client remains the go-to option for maximum flexibility and control.

![](https://assets.zilliz.com/RES_Tful_8520a80a8e.png)


### 4. Aligned SDK Design Across All Languages

With Milvus SDK v2, we've standardized the design of our SDKs across all supported programming languages to deliver a more consistent developer experience.

Whether you're building with Python, Java, Go, or Node.js, each SDK now follows a unified structure centered around the MilvusClient class. This redesign brings consistent method naming, parameter formatting, and overall usage patterns to every language we support. (See: [MilvusClient SDK code example update · GitHub Discussion #33979](https://github.com/milvus-io/milvus/discussions/33979))

![](https://assets.zilliz.com/Milvus_Client_9a4a6da9e3.png)

Now, once you're familiar with Milvus in one language, you can easily switch to another without having to relearn how the SDK works. This alignment not only simplifies onboarding but also makes multi-language development much smoother and more intuitive.


### 5. A Simpler, Smarter PyMilvus (Python SDK) with `MilvusClient`

In the previous version, PyMilvus relied on an ORM-style design that introduced a mix of object-oriented and procedural approaches. Developers had to define `FieldSchema` objects, build a `CollectionSchema`, and then instantiate a `Collection` class—all just to create a collection. This process was not only verbose but also introduced a steeper learning curve for new users.

With the new `MilvusClient` interface, things are much simpler. You can now create a collection in a single step using the `create_collection()` method. It allows you to quickly define the schema by passing parameters like `dimension` and `metric_type`, or you can still use a custom schema object if needed.

Even better, `create_collection()` supports index creation as part of the same call. If index parameters are provided, Milvus will automatically build the index and load the data into memory—no need for separate `create_index()` or `load()` calls. One method does it all: _create collection → build index → load collection._

This streamlined approach reduces setup complexity and makes it much easier to get started with Milvus, especially for developers who want a quick and efficient path to prototyping or production.

The new `MilvusClient` module offers clear advantages in usability, consistency, and performance. While the legacy ORM interface remains available for now, we plan to phase it out in the future (see [reference](https://docs.zilliz.com/reference/python/ORM#:~:text=About%20to%20Deprecate)). We strongly recommend upgrading to the new SDK to take full advantage of the improvements.


### 6. Clearer and More Comprehensive Documentation

We have restructured the product documentation to provide a more complete and clearer [API Reference](https://milvus.io/docs). Our User Guides now include multi-language sample code, enabling you to get started quickly and understand Milvus' features with ease. Additionally, the Ask AI assistant available on our documentation site can introduce new features, explain internal mechanisms, and even help generate or modify sample code, making your journey through the documentation smoother and more enjoyable.

![](https://assets.zilliz.com/Ask_AI_Assistant_b044d4621a.png)


### 7. Milvus MCP Server: Designed for the Future of AI Integration

The [MCP Server](https://github.com/zilliztech/mcp-server-milvus), built on top of the Milvus SDK, is our answer to a growing need in the AI ecosystem: seamless integration between large language models ([LLMs](https://zilliz.com/glossary/large-language-models-(llms))), [vector databases](https://milvus.io/blog/what-is-a-vector-database.md), and external tools or data sources. It implements the Model Context Protocol (MCP), providing a unified and intelligent interface for orchestrating Milvus operations and beyond.

As [AI agents](https://zilliz.com/blog/top-10-ai-agents-to-watch-in-2025) become more capable—not just generating code but autonomously managing backend services—the demand for smarter, API-driven infrastructure is rising. The MCP Server was designed with this future in mind. It enables intelligent and automated interactions with Milvus clusters, streamlining tasks like deployment, maintenance, and data management.

More importantly, it lays the groundwork for a new kind of machine-to-machine collaboration. With the MCP Server, AI agents can call APIs to dynamically create collections, run queries, build indexes, and more—all without human intervention.

In short, the MCP Server transforms Milvus into not just a database, but a fully programmable, AI-ready backend—paving the way for intelligent, autonomous, and scalable applications.


## Getting Started with Milvus SDK v2: Sample Code

The examples below show how to use the new PyMilvus (Python SDK v2) interface to create a collection and perform asynchronous operations. Compared to the ORM-style approach in the previous version, this code is cleaner, more consistent, and easier to work with.


### 1. Creating a Collection, Defining Schemas, Building Indexes, and Loading Data with `MilvusClient`

The Python code snippet below demonstrates how to create a collection, define its schema, build indexes, and load data—all in one call:

```
from pymilvus import MilvusClient, DataType

# 1. Connect to Milvus (initialize the client)
client = MilvusClient(uri="http://localhost:19530")

# 2. Define the collection schema
schema = MilvusClient.create_schema(auto_id=False, description="schema for example collection")
schema.add_field("id", DataType.INT64, is_primary=True)  # Primary key field
schema.add_field("embedding", DataType.FLOAT_VECTOR, dim=128)  # Vector field

# 3. Prepare index parameters (optional if indexing at creation time)
index_params = client.prepare_index_params()
index_params.add_index(
    field_name="embedding",
    index_type="AUTOINDEX",
    metric_type="L2"
)

# 4. Create the collection with indexes and load it into memory automatically
client.create_collection(
    collection_name="example_collection",
    schema=schema,
    index_params=index_params
)

print("Collection created and loaded with index!")
```

The `create_collection` method's `index_params` parameter eliminates the need for separate calls for `create_index` and `load_collection`—everything happens automatically.

In addition, `MilvusClient` supports a quick table creation mode. For example, a collection can be created in a single line of code by specifying only the required parameters:

```
client.create_collection(
    collection_name="test_collection",
    dimension=128
)
```

_(Comparison note: In the old ORM approach, you had to create a `Collection(schema)`, then separately call `collection.create_index()` and `collection.load()`; now, MilvusClient streamlines the entire process.)_


### 2. Performing High-Concurrency Asynchronous Inserts with `AsyncMilvusClient`

The following example shows how to use the `AsyncMilvusClient` to perform concurrent insert operations using `async/await`:

```
import asyncio
from pymilvus import AsyncMilvusClient

async def insert_vectors_concurrently():
    client = AsyncMilvusClient(uri="http://localhost:19530")
    
    vectors_to_insert = [[...], [...], ...]  # Assume 100,000 vectors
    batch_size = 1000  # Recommended batch size
    tasks = []

    for i in range(0, len(vectors_to_insert), batch_size):
        batch_vectors = vectors_to_insert[i:i+batch_size]

        # Construct batch data
        data = [
            list(range(i, i + len(batch_vectors))),  # Batch IDs
            batch_vectors  # Batch vectors
        ]

        # Add an asynchronous task for inserting each batch
        tasks.append(client.insert("example_collection", data=data))

    # Concurrently execute batch inserts
    insert_results = await asyncio.gather(*tasks)
    await client.close()

# Execute asynchronous tasks
asyncio.run(insert_vectors_concurrently())
```

In this example, `AsyncMilvusClient` is used to concurrently insert data by scheduling multiple insertion tasks with `asyncio.gather`. This approach takes full advantage of Milvus' backend concurrent processing capabilities. Unlike the synchronous, line-by-line insertions in v1, this native asynchronous support dramatically increases throughput.

Similarly, you can modify the code to perform concurrent queries or searches—for example, by replacing the insert call with `client.search("example_collection", data=[query_vec], limit=5)`. Milvus SDK v2's asynchronous interface ensures that each request is executed in a non-blocking manner, fully leveraging both client and server resources.


## Migration Made Easy

We know you've invested time in SDK v1, so we've designed SDK v2 with your existing applications in mind. SDK v2 includes backward compatibility, so existing v1/ORM-style interfaces will continue to work for a while. But we strongly recommend upgrading to SDK v2 as soon as possible—support for v1 will end with the release of Milvus 3.0 (end of 2025).

Moving to SDK v2 unlocks a more consistent, modern developer experience with simplified syntax, better async support, and improved performance. It's also where all new features and community support are focused going forward. Upgrading now ensures you're ready for what's next and gives you access to the best Milvus has to offer.


## Conclusion

Milvus SDK v2 brings significant improvements over v1: enhanced performance, a unified and consistent interface across multiple programming languages, and native asynchronous support that simplifies high-concurrency operations. With clearer documentation and more intuitive code examples, Milvus SDK v2 is designed to streamline your development process, making it easier and faster to build and deploy AI applications.

For more detailed information, please refer to our latest official [API Reference and User Guides](https://milvus.io/docs/install-pymilvus.md#Install-Milvus-Python-SDK). If you have any questions or suggestions regarding the new SDK, feel free to provide feedback on [GitHub](https://github.com/milvus-io/milvus/discussions) and [Discord](https://discord.com/invite/8uyFbECzPX). We look forward to your input as we continue to enhance Milvus.


Introducing Milvus SDK v2: Native Async Support, Unified APIs, and Superior Performance




## Introduction

While dense embedding retrieval excels at semantic search, full-text search algorithms like BM25 continue to play an important role in retrieval tasks, complementing dense embedding approaches with their predictability and ability to handle specialized terminology.

Since [version 2.5](https://milvus.io/blog/introduce-milvus-2-5-full-text-search-powerful-metadata-filtering-and-more.md), Milvus introduced built-in support for full-text search with the Sparse-BM25 approach, allowing users to easily integrate this capability into their search pipelines without the need for a separate search engine. In this blog, we will unveil how Sparse-BM25 works in Milvus.


## The Famous BM25 Ranking Algorithm

[BM25](https://zilliz.com/learn/mastering-bm25-a-deep-dive-into-the-algorithm-and-application-in-milvus), short for Best Match 25, is a relevance ranking algorithm, designed to improve the TF-IDF (Term Frequency-Inverse Document Frequency) algorithm.

Given a set of documents, this is how we compute the BM25 relevance score between a query $Q$ and an arbitrary document $D$:

$$
\text{score}(D, Q) = \sum_{i=1}^{n} \text{IDF}(q_i) \cdot \frac{\text{TF}(q_i, D) \cdot (k_1 + 1)}{\text{TF}(q_i, D) + k_1 \cdot \left(1 - b + b \cdot \frac{|D|}{\text{avgdl}}\right)}
$$

$$
\text{IDF}(q_i) = \log\left( \frac{N - n(q_i) + 0.5}{n(q_i) + 0.5} + 1 \right)
$$


Here are some key points:

1. **Term Frequency** ( $TF(t, D)$ ): number of occurrences of the term *t* in document *D*. $k_1$ is a saturation argument to **prevent words that occur too frequently from dominating the score**.

2. **Inverse Document Frequency** ( $IDF(t)$ ): measures the importance of a term. The fewer documents it appears in, the more important BM25 thinks it is. $N$ is the total number of documents in the corpus, $n(q_i)$ is the number of documents that contain the word $q_i$.

3. **Document Length Normalization:** TF-IDF tends to rank longer documents higher since they are more likely to contain query terms. BM25 reduces this bias by normalizing with average document length. $b$ can be used to tune normalization strength. $|D|$ is the length of the document (total number of terms), $avgdl$ is the average document length of all documents.

Compared to TF-IDF, BM25 introduces two important optimizations: **TF Saturation** and **Document Length Normalization**. Such mechanisms drastically improved ranking quality, making BM25 the go-to algorithm of full-text search for decades.


## Computing BM25 using Sparse Vector

We introduced support for Sparse Vector in Milvus 2.4. Unlike a dense vector, which has up to thousands of dimensions and most dimensions have a non-zero value, a sparse vector typically has tens of thousands or more dimensions, where the value of most dimensions is zero.

![](https://assets.zilliz.com/whiteboard_exported_image_2c86d45ae4.png)

In ultra-high-dimensional space, the "curse of dimensionality" renders traditional distance metrics like Euclidean and Cosine ineffective. As the number of dimensions increases, the distance between the nearest and farthest vectors becomes nearly identical, making these metrics unreliable (for those who are interested, read [The Curse of Dimensionality in Machine Learning](https://zilliz.com/glossary/curse-of-dimensionality-in-machine-learning)). Due to that, Inner Product, or IP, becomes the only usable metric type for sparse vectors.

![](https://assets.zilliz.com/curse_of_dimensionality_bbb681fe52.png)

To understand how Milvus supports BM25 with a sparse vector approach, let's transform the BM25 formula by replacing the right hand side with $f(TF)$:


$$
f(TF) = \frac{TF \cdot (k_1 + 1)}{TF + k_1 \cdot (1 - b + b \cdot \frac{|D|}{avgdl})}
$$

So the BM25 formula is now:

$$
score(D, Q) = \sum_{i=1}^{N} IDF \cdot f(TF)
$$



For those who know IP metric, does this look familiar to you? This is exactly the equation of IP distance for sparse vectors. Let's see how this works with a concrete example:

Assume we have a set of documents, with one of them being `I love Milvus!`. After tokenization and lowercasing, we have the document's TF value for each term, represented as a sparse vector: `{i: 1, love: 1, milvus: 1}`. We can then apply $f(TF)$ and transform it into：

```
{i: f(1), love: f(1), milvus: f(1)}
```

This will be the **document vector**.

For a query like `Who loves Milvus?`. Again, we first apply tokenization, lowercasing and stemming, and got query token list `[who, love, milvus]`. Apply the $IDF()$ function and we can transform it into

```
{who: IDF(who), love: IDF(love), milvus: IDF(milvus)}
```

This will be our **query vector**.

The Inner Product of the document vector and query vector is:

$$
0 \cdot f(\text{TF}(i)) + \text{IDF}(\text{who}) \cdot 0 + \text{IDF}(\text{love}) \cdot f(\text{TF}(\text{love})) + \text{IDF}(\text{milvus}) \cdot f(\text{TF}(\text{milvus}))
$$

Which is exactly the BM25 score of the query `Who loves Milvus?` and document `I love Milvus!`.

If we can encode the documents into sparse vectors and insert them into Milvus, then we can leverage Milvus' sparse vector index to perform BM25 search. Sounds good? Well, not quite yet:

- The calculation of $f(TF)$ requires $avgdl$, which is a global statistics value and constantly changing overtime as new documents being inserted and old documents being deleted. After we added new documents, the previously inserted document vector values become skewed, leading to inaccurate result!

- There is much burden on the client side: the user of Milvus must have great knowledge of BM25, and must manage the term distribution statistics and perform vector encoding themselves.

These two shortcomings rendered this approach not suitable for serious production use cases. So we developed something better!


## A Better Approach - Dynamic Stats and Search Time Encoding

In Milvus 2.5, we designed a novel approach to tackle those problems.

First, to deal with the ever-changing $avgdl$, instead of pre-computing $f(TF)$, we only store the raw $TF$ as the document vector. We add a new metric `BM25` for the sparse vector field, and compute the BM25 score on the fly during query time. The BM25 score is still calculated with the query vector value $IDF$ and document vector value $TF$.

![](https://assets.zilliz.com/whiteboard_exported_image_1_3eca6fc09d.png)

Secondly, users don't need to pre-compute the sparse vector anymore. Users only need to provide the documents in raw text during ingestion or search. Milvus is responsible for processing the text into tokens with stop-word removal and stemming, and maintain a global term distribution statistics across the corpus that is required to compute both $IDF$ and $avgdl$.

At insertion time, texts are tokenized and converted into sparse vectors by Milvus and stored in a sparse vector field. At search time, Milvus encodes queries as $IDF$ vector based on the global stats and sends it to QueryNode along with a realtime $avgdl$ to perform Sparse-BM25 search.

![](https://assets.zilliz.com/document_insertion_6590983663.png)


## Full-Text Search Demo with PyMilvus

With this new feature, performing full-text search in Milvus is straightforward. Let’s walk through an example using **PyMilvus**, the official Python client for Milvus.


### Step 1: Define a Collection Schema

First, we need to create a collection and define fields for **the raw text data and the encoded document sparse vectors** (we only need to ingest raw text data, the sparse vector is computed by Milvus internally).

First, we define a collection schema with fields for the raw text (document) and the sparse vector (sparse). Note that we also define a `Function` in the schema, which tells Milvus to automatically convert ingested raw text to sparse vector used for BM25 scoring.

```Python
from pymilvus import MilvusClient, DataType, Function, FunctionType

client = MilvusClient(uri="http://localhost:19530")

# Define a schema
schema = client.create_schema()
schema.add_field(field_name="id", datatype=DataType.INT64, is_primary=True, auto_id=True)
schema.add_field(field_name="document", datatype=DataType.VARCHAR, max_length=1000, enable_analyzer=True)
schema.add_field(field_name="sparse", datatype=DataType.SPARSE_FLOAT_VECTOR)

# Add BM25 function mapping text to sparse vector
bm25_function = Function(
    name="text_bm25_emb",
    input_field_names=["document"],
    output_field_names=["sparse"],
    function_type=FunctionType.BM25,
)
schema.add_function(bm25_function)
```

> 💡 Make sure enable_analyzer=True is set for the text field to be searchable using BM25. You can also configure a custom analyzer if needed—see the [Analyzer documentation](https://milvus.io/docs/analyzers.md) for details.


### Step 2: Create an Index

Just like dense vector fields, sparse vector fields also require an index. Here, we create an automatic index using the BM25 metric.

```Python
# Prepare the index
index_params = client.prepare_index_params()
index_params.add_index(
    field_name="sparse",
    index_type="AUTO_INDEX", 
    metric_type="BM25"
)

# Create the collection (Milvus will automatically load it)
client.create_collection(
    collection_name="hello_bm25",
    schema=schema,
    index_params=index_params
)
```


### Step 3: Insert Data (Raw Text Only)

With the schema and index ready, we can now insert raw text into the collection. There’s no need to provide the sparse vector—Milvus generates it automatically using the BM25 function.

```Python
# Insert documents
entities = [
    {"document": "Artificial intelligence was founded ..."},
    {"document": "Alan Turing was the first person to ..."},
    {"document": "Born in Maida Vale, London, Turing ..."},
]
client.insert("hello_bm25", entities)
```


### Step 4: Perform a Full-Text Search

You can now search using natural language queries!

```Python
# Example: Search for "Artificial Intelligence"
client.search(
    collection_name="hello_bm25",
    data=["Artificial Intelligence"],
    anns_field="sparse",
    limit=1,
    output_fields=["document"]
)
```

That's all, simple like that!


## Conclusion

With built-in BM25 support, Milvus brings full-text search and vector search together in one unified engine. By handling tokenization, sparse vector encoding, and dynamic stats internally, it removes the cumbersome manual BM25 sparse vector computation outside of vector database.

We developed this feature to help you build advanced search applications with minimal effort. You can now combine full-text search with dense vector search and reranking function, building your own hybrid search. Check out the [full tutorial](https://milvus.io/docs/full_text_search_with_milvus.md) to get started!


Full-Text Search in Milvus - What's Under the Hood



## Introduction

Selecting the optimal configuration for your Milvus deployment is critical for performance optimization, efficient resource utilization, and cost management. Whether you're building a prototype or planning a production deployment, properly sizing your Milvus instance can mean the difference between a smoothly running vector database and one that struggles with performance or incurs unnecessary costs.

To simplify this process, we've revamped our [Milvus Sizing Tool](https://milvus.io/tools/sizing), a user-friendly calculator that generates recommended resource estimations based on your specific requirements. In this guide, we'll walk you through using the tool and provide deeper insights into the factors that influence the Milvus performance.


## How to Use the Milvus Sizing Tool

It’s super easy to use this sizing tool. Simply follow the following steps. 

1. Visit the[ Milvus Sizing Tool](https://milvus.io/tools/sizing/) page.

2. Enter your key parameters:

   - Number of vectors and dimensions per vector

   - Index type

   - Scalar field data size

   - Segment size

   - Your preferred deployment mode

3. Review the generated resource recommendations

![milvus sizing tool](https://assets.zilliz.com/milvus_sizing_tool_3ca881b3d5.jpeg)


Let's explore how each of these parameters impacts your Milvus deployment.


## Index Selection: Balancing Storage, Cost, Accuracy, and Speed

Milvus offers various index algorithms, including [HNSW](https://zilliz.com/learn/hierarchical-navigable-small-worlds-HNSW), FLAT, IVF_FLAT, IVF_SQ8, [ScaNN](https://zilliz.com/learn/what-is-scann-scalable-nearest-neighbors-google), [DiskANN](https://zilliz.com/learn/DiskANN-and-the-Vamana-Algorithm), and more, each with distinct trade-offs in memory usage, disk space requirements, query speed, and search accuracy. 

Here's what you need to know about the most common options:

![index](https://assets.zilliz.com/index_dde661d579.jpeg)


HNSW (Hierarchical Navigable Small World)

- **Architecture**: Combines skip lists with Navigable Small Worlds (NSWs) graphs in a hierarchical structure

- **Performance**: Very fast querying with excellent recall rates

- **Resource Usage**: Requires the most memory per vector (highest cost)

- **Best For**: Applications where speed and accuracy are critical and memory constraints are less of a concern

- **Technical Note**: The Search begins at the topmost layer with the fewest nodes and traverses downward through increasingly dense layers

FLAT

- **Architecture**: Simple exhaustive search with no approximation

- **Performance**: 100% recall but extremely slow query times (`O(n)` for data size `n`)

- **Resource Usage**: Index size equals the raw vector data size

- **Best For**: Small datasets or applications requiring perfect recall

- **Technical Note**: Performs complete distance calculations between the query vector and every vector in the database

IVF_FLAT

- **Architecture**: Divides vector space into clusters for more efficient searching

- **Performance**: Medium-high recall with moderate query speed (slower than HNSW but faster than FLAT)

- **Resource Usage**: Requires less memory than FLAT but more than HNSW

- **Best For**: Balanced applications where some recall can be traded for better performance

- **Technical Note**: During search, only `nlist`  clusters are examined, significantly reducing computation

IVF_SQ8

- **Architecture**: Applies scalar quantization to IVF_FLAT, compressing vector data

- **Performance**: Medium recall with medium-high query speed

- **Resource Usage**: Reduces disk, compute, and memory consumption by 70-75% compared to IVF_FLAT

- **Best For**: Resource-constrained environments where accuracy can be slightly compromised

- **Technical Note**: Compresses 32-bit floating-point values to 8-bit integer values


### Advanced Index Options: ScaNN, DiskANN, CAGRA, and more 

For developers with specialized requirements, Milvus also offers:

- [**ScaNN**](https://zilliz.com/learn/what-is-scann-scalable-nearest-neighbors-google): 20% faster on CPU than HNSW with similar recall rates

- [**DiskANN**](https://milvus.io/docs/disk_index.md): A hybrid disk/memory index that's ideal when you need to support a large number of vectors with high recall and can accept slightly longer latency (~100ms). It balances memory usage with performance by keeping only part of the index in memory while the rest remains on disk.

- **GPU-based indexes**:

  - [GPU_CAGRA](https://zilliz.com/blog/Milvus-introduces-GPU-index-CAGRA): This is the fastest of the GPU indexes, but it requires an inference card with GDDR memory rather than one with HBM memory

  - GPU_BRUTE_FORCE: Exhaustive search implemented on GPU

  - GPU_IVF_FLAT: GPU-accelerated version of IVF_FLAT

  - GPU_IVF_PQ: GPU-accelerated version of IVF with [Product Quantization](https://zilliz.com/learn/harnessing-product-quantization-for-memory-efficiency-in-vector-databases)

- **HNSW-PQ/SQ/PRQ**: 

  - **HNSW_SQ**: Very high-speed query, limited memory resources; accepts minor compromise in recall rate. 

  - **HNSW_PQ**: Medium speed query; Very limited memory resources; Accepts minor compromise in recall rate

  - **HNSW_PRQ**: Medium speed query; Very limited memory resources; Accepts minor compromise in recall rate

  - **AUTOINDEX**: Defaults to HNSW in open-source Milvus (or uses higher-performing proprietary indexes in [Zilliz Cloud](https://zilliz.com/cloud), the managed Milvus).

- **Binary, Sparse, and other specialized indexes**: For specific data types and use cases. See [this index doc page](https://milvus.io/docs/index.md) for more details. 


## Segment Size and Deployment Configuration

Segments are the fundamental building blocks of Milvus's internal data organization. They function as data chunks that enable distributed search and load balancing across your deployment. This Milvus sizing tool offers three segment size options (512 MB, 1024 MB, 2048 MB), with 1024 MB as the default.

Understanding segments is crucial for performance optimization. As a general guideline:

- 512 MB segments: Best for query nodes with 4-8 GB memory

- 1 GB segments: Optimal for query nodes with 8-16 GB memory

- 2 GB segments: Recommended for query nodes with >16 GB memory

Developer Insight: Fewer, larger segments typically deliver faster search performance. For large-scale deployments, 2 GB segments often provide the best balance between memory efficiency and query speed.


## Message Queue System Selection

When choosing between Pulsar and Kafka as your messaging system:

- **Pulsar**: Recommended for new projects due to lower overhead per topic and better scalability

- **Kafka**: May be preferable if you already have Kafka expertise or infrastructure in your organization


## Enterprise Optimizations in Zilliz Cloud

For production deployments with strict performance requirements, Zilliz Cloud (the fully managed and enterprise version of Milvus on the cloud) offers additional optimizations in indexing and quantization:

- **Out of Memory (OOM) Prevention:** Sophisticated memory management to prevent out-of-memory crashes

- **Compaction Optimization**: Improves search performance and resource utilization

- **Tiered Storage**: Efficiently manage hot and cold data with appropriate compute units

  - Standard compute units (CUs) for frequently accessed data

  - Tiered storage CUs for cost-effective storage of rarely accessed data

For detailed enterprise sizing options, visit the[ Zilliz Cloud service plans documentation](https://docs.zilliz.com/docs/select-zilliz-cloud-service-plans).


## Advanced Configuration Tips for Developers

1. **Multiple Index Types**: The sizing tool focuses on a single index. For complex applications requiring different index algorithms for various collections, create separate collections with custom configurations.

2. **Memory Allocation**: When planning your deployment, account for both vector data and index memory requirements. HNSW typically requires 2-3x the memory of the raw vector data.

3. **Performance Testing**: Before finalizing your configuration, benchmark your specific query patterns on a representative dataset.

4. **Scale Considerations**: Factor in future growth. It's easier to start with slightly more resources than to reconfigure later.


## Conclusion

The[ Milvus Sizing Tool](https://milvus.io/tools/sizing/) provides an excellent starting point for resource planning, but remember that every application has unique requirements. For optimal performance, you'll want to fine-tune your configuration based on your specific workload characteristics, query patterns, and scaling needs.

We're continuously improving our tools and documentation based on user feedback. If you have questions or need further assistance with sizing your Milvus deployment, reach out to our community on[ GitHub](https://github.com/milvus-io/milvus/discussions) or[ Discord](https://discord.com/invite/8uyFbECzPX).


## References

- [📝 Choosing the Right Vector Index For Your Project](https://thesequence.substack.com/p/guest-post-choosing-the-right-vector)

- [In-memory Index | Milvus Documentation](https://milvus.io/docs/index.md?tab=floating)

- [Unveil Milvus CAGRA: Elevating Vector Search with GPU Indexing](https://zilliz.com/blog/Milvus-introduces-GPU-index-CAGRA)

- [Zilliz Cloud Pricing Calculator](https://zilliz.com/pricing#estimate_your_cost)

- [How to Get Started with Milvus ](https://milvus.io/blog/how-to-get-started-with-milvus.md)

- [Zilliz Cloud Resource Planning | Cloud | Zilliz Cloud Developer Hub](https://docs.zilliz.com/docs/resource-planning)


Introducing the Milvus Sizing Tool: Calculating and Optimizing Your Milvus Deployment Resources 




Selecting the right [embedding model](https://zilliz.com/ai-models) is a critical decision when building systems that understand and work with [unstructured data](https://zilliz.com/learn/introduction-to-unstructured-data) like text, images, or audio. These models transform raw input into fixed-size, high-dimensional vectors that capture semantic meaning, enabling powerful applications in similarity search, recommendations, classification, and more.

But not all embedding models are created equal. With so many options available, how do you choose the right one? The wrong choice can lead to suboptimal accuracy, performance bottlenecks, or unnecessary costs. This guide provides a practical framework to help you evaluate and select the best embedding model for your specific requirements.

![](https://assets.zilliz.com/Complete_Workflow_31b4ac825c.gif)


## 1. Define Your Task and Business Requirements

Before choosing an embedding model, start by clarifying your core objectives:

- **Task Type:** Start by identifying the core application you're building—semantic search, a recommender system, a classification pipeline, or something else entirely. Each use case has different requirements for how embeddings should represent and organize information. For example, if you are building a semantic search engine, you need models like Sentence-BERT that capture nuanced semantic meaning between queries and documents, ensuring that similar concepts are close in vector space. For classification tasks, embeddings must reflect category-specific structure, so that inputs from the same class are placed close together in the vector space. This makes it easier for downstream classifiers to distinguish between classes. Models like DistilBERT, and RoBERTa are commonly used. In recommender systems, the goal is to find embeddings that reflect user-item relationships or preferences. For this, you might use models that are specifically trained on implicit feedback data like Neural Collaborative Filtering (NCF). 
- **ROI Assessment:** Balance performance against costs based on your specific business context. Mission-critical applications (like healthcare diagnostics) may justify premium models with higher accuracy since it could be a matter of like and death, while cost-sensitive applications with high volume need careful cost-benefit analysis. The key is determining whether a mere 2-3% performance improvement justifies potentially significant cost increases in your particular scenario.
- **Other Constraints:** Consider your technical requirements when narrowing down options. If you need multilingual support, many general models struggle with non-English content, so specialized multilingual models may be necessary. If you're working in specialized domains (medical/legal), general-purpose embeddings often miss domain-specific jargon—for example, they might not understand that _"stat"_ in a medical context means _"immediately"_, or that _"consideration"_ in legal documents refers to something of value exchanged in a contract. Similarly, hardware limitations and latency requirements will directly impact which models are feasible for your deployment environment.

![](https://assets.zilliz.com/clarify_task_and_business_requirement_b1bce2ccc0.png)


## 2. Evaluate Your Data

The nature of your data significantly influences the choice of embedding model. Key considerations include:

- **Data Modality:** Is your data textual, visual, or multimodal in nature? Match your model to your data type. Use transformer-based models like [BERT](https://zilliz.com/learn/what-is-bert) or [Sentence-BERT](https://zilliz.com/learn/Sentence-Transformers-for-Long-Form-Text)  for text, [CNN architectures](https://zilliz.com/glossary/convolutional-neural-network) or Vision Transformers ([ViT](https://zilliz.com/learn/understanding-vision-transformers-vit)) for images, specialized models for audio, and multimodal models like [CLIP](https://zilliz.com/learn/exploring-openai-clip-the-future-of-multimodal-ai-learning) and MagicLens for multimodal applications.
- **Domain Specificity:** Consider whether general models are sufficient, or if you need domain-specific models that understand specialized knowledge. General models trained on diverse datasets (like [OpenAI text embedding models](https://zilliz.com/ai-models/text-embedding-3-large)) work well for common topics but often miss subtle distinctions in specialized fields. However, in fields like healthcare or legal services, they often miss subtle distinctions—so domain-specific embeddings like [BioBERT](https://arxiv.org/abs/1901.08746) or [LegalBERT](https://arxiv.org/abs/2010.02559) may be more suitable.
- **Embedding Type:** [Sparse embeddings](https://zilliz.com/learn/sparse-and-dense-embeddings) excel at keyword matching, making them ideal for product catalogs or technical documentation. Dense embeddings capture semantic relationships better, making them suitable for natural language queries and intent understanding. Many production systems like e-commerce recommender systems benefit from a hybrid approach that leverages both types—for example, using [BM25](https://zilliz.com/learn/mastering-bm25-a-deep-dive-into-the-algorithm-and-application-in-milvus) (sparse) for keyword matching while adding BERT (dense embeddings) to capture semantic similarity.

![](https://assets.zilliz.com/evaluate_your_data_6caeeb813e.png)


## 3. Research Available Models

After understanding your task and data, it's time to research available embedding models. Here’s how you can approach this:

- **Popularity:** Prioritize models with active communities and widespread adoption. These models usually benefit from better documentation, broader community support, and regular updates. This can significantly reduce implementation difficulties. Familiarize yourself with leading models in your domain. For example:

  - For Text: consider OpenAI embeddings, Sentence-BERT variants, or E5/BGE models.
  - For image: look at ViT and ResNet, or CLIP and SigLIP for text-image alignment.
  - For Audio: check PNNs, CLAP or [other popular models](https://zilliz.com/learn/top-10-most-used-embedding-models-for-audio-data).

- **Copyright and Licensing**: Carefully evaluate the licensing implications as they directly affect both short and long-term costs. Open-source models (like MIT, Apache 2.0, or similar licenses) provide flexibility for modification and commercial use, giving you full control over deployment but requiring infrastructure expertise. Proprietary models accessed via APIs offer convenience and simplicity but come with ongoing costs and potential data privacy concerns. This decision is especially critical for applications in regulated industries where data sovereignty or compliance requirements may make self-hosting necessary despite the higher initial investment.

![](https://assets.zilliz.com/model_research2_b0df75cb55.png)


## 4. Evaluate Candidate Models

Once you've shortlisted a few models, it's time to test them with your sample data. Here are key factors you should consider:

- **Evaluation:** When evaluating embedding quality—especially in retrieval augmented generation (RAG) or search application—it's important to measure _how accurate, relevant, and complete_ the returned results are. Key metrics include faithfulness, answer relevancy, context precision, and recall. Frameworks like Ragas, DeepEval, Phoenix, and TruLens-Eval streamline this evaluation process by providing structured methodologies for assessing different aspects of embedding quality. Datasets are equally important for meaningful evaluation. They can be hand-crafted to represent real use cases, synthetically generated by LLMs to test specific capabilities, or created using tools like Ragas and FiddleCube to target particular testing aspects. The right combination of dataset and framework depends on your specific application and the level of evaluation granularity you need to make confident decisions.
- **Benchmark Performance:** Evaluate models on task-specific benchmarks (e.g., MTEB for retrieval). Remember that rankings vary significantly by scenario (search vs. classification), datasets (general vs. domain-specific like BioASQ), and metrics (accuracy, speed). While benchmark performance provide valuable insights, it doesn't always translate perfectly to real-world applications. Cross-check top performers that align with your data type and goals, but always validate with your own custom test cases to identify models that might overfit to benchmarks but underperform in real-world conditions with your specific data patterns.
- **Load Testing:** For self-hosted models, simulate realistic production loads to evaluate performance under real-world conditions. Measure throughput as well as GPU utilization and memory consumption during inference to identify potential bottlenecks. A model that performs well in isolation may become problematic when handling concurrent requests or complex inputs. If the model is too resource-intensive, it may not be suitable for large-scale or real-time applications regardless of its accuracy on benchmark metrics.

![](https://assets.zilliz.com/evaluate_candidate_models_3a7edd9cd7.png)


## 5. Model Integration

After selecting a model, now it is time to plan your integration approach.

- **Weights Selection:** Decide between using pre-trained weights for quick deployment or fine-tuning on domain-specific data for improved performance. Remember fine-tuning can improve performance but is resource heavy. Consider whether performance gains justify the additional complexity.
- **Self-Hosting vs. Third-party Inference Service:** Choose your deployment approach based on your infrastructure capabilities and requirements. Self-hosting gives you complete control over the model and data flow, potentially reducing per-request costs at scale and ensuring data privacy. However, it requires infrastructure expertise and ongoing maintenance. Third-party inference services offer rapid deployment with minimal setup but introduce network latency, potential usage caps, and continuous costs that can become significant at scale.
- **Integration Design:** Plan your API design, caching strategies, batch processing approach, and [vector database](https://milvus.io/blog/what-is-a-vector-database.md) selection for storing and querying embeddings efficiently.

![](https://assets.zilliz.com/model_integration_8c8f0410c7.png)


## 6. End-to-End Testing

Before full deployment, run end-to-end tests to ensure the model works as expected:

- **Performance**: Always evaluate the model on your own dataset to ensure they perform well in your specific use case. Consider metrics like MRR, MAP and NDCG for retrieval quality, precision, recall and F1 for accuracy, and throughput and latency percentiles for operational performance.
- **Robustness**: Test the model under different conditions, including edge cases and diverse data inputs, to verify that it performs consistently and accurately.

![](https://assets.zilliz.com/end_to_end_testing_7ae244a73b.png)


## Summary

As we've seen throughout this guide, selecting the right embedding model requires following these six critical steps:

1. Define your business requirements and task type
2. Analyze your data characteristics and domain specificity
3. Research available models and their licensing terms
4. Rigorously evaluate candidates against relevant benchmarks and test datasets
5. Plan your integration approach considering deployment options
6. Conduct comprehensive end-to-end testing before production deployment

By following this framework, you can make an informed decision that balances performance, cost, and technical constraints for your specific use case. Remember that the "best" model isn't necessarily the one with the highest benchmark scores—it's the one that best meets your particular requirements within your operational constraints.

With embedding models evolving rapidly, it's also worth periodically reassessing your choice as new options emerge that might offer significant improvements for your application.


How to Choose the Right Embedding Model?




Let's face it: parsing documents is hard—really hard. PDFs, images, reports, tables, messy handwriting; they're packed with valuable information that your users want to search for, but extracting that information and express that accurately in your search index is like solving a puzzle where the pieces keep changing shape: you thought you've solved it with an extra line of code but tomorrow a new doc gets ingested and you find another corner case to deal with.

In this post, we'll tackle this challenge head-on using the powerful combo of Mistral OCR and Milvus Vector DB, turning your doc parsing nightmares into a calm dream with searchable, semantically meaningful vector embeddings. 


## Why Rule-based Parsing Just Won't Cut It

If you've ever struggled with standard OCR tools, you probably know that they have all sorts of issues:

- **Complex layouts**: Tables, lists, multi-column formats -- they can break or pose issues to most parsers.
- **Semantic ambiguity**: Keywords alone don't tell you if "apple" means fruit or company.
- The chanllege of scale and cost: Processing thousands of documents becomes painfully slow.

We need a smarter, more systematic approach that doesn’t just extract text—it _understands_ the content. And that’s exactly where Mistral OCR and Milvus come in.


## Meet Your Dream Team

### Mistral OCR: More than just text extraction

Mistral OCR isn’t your average OCR tool. It's designed to tackle a wide range of documents.

- **Deep Understanding of Complex Documents**: Whether it's embedded images, mathematical equations, or tables, it can understand it all with a very high accuracy.
- **Keeps original layouts:** Not only does it understand the different layouts in the documents, it also keeps the original layouts and structure intact. On top of that, it's also capable of parsing multi-page documents.
- **Multilingual and Multimodal Mastery**: From English to Hindi to Arabic, Mistral OCR can comprehend documents across thousands of languages and scripts, making it invaluable for applications targeting a global user base.


### Milvus: Your Vector Database Built for Scale

- **Billion+ Scale**: [Milvus](https://milvus.io/) can scale to billions of vectors, making it perfect for storing large-scale documents.
- **Full-Text Search: In addition to supporting dense vector embeddings**, Milvus also supports Full Text Search. Making it easy to run queries using text and get better results for your RAG system.


## Examples:

Let's take this handwritten note in English, for example. Using a regular OCR tool to extract this text would be a very hard task.

![A handwritten note in English ](https://assets.zilliz.com/A_handwritten_note_in_English_3bbc40dee7.png)

We process it with Mistral OCR

```python
api_key = os.getenv("MISTRAL_API_KEY")
client = Mistral(api_key=api_key)

url = "https://preview.redd.it/ocr-for-handwritten-documents-v0-os036yiv9xod1.png?width=640&format=png&auto=webp&s=29461b68383534a3c1bf76cc9e36a2ba4de13c86"
result = client.ocr.process(
                model=ocr_model, document={"type": "image_url", "image_url": url}
            )
print(f"Result: {result.pages[0].markdown}")
```

And we get the following output. It can recognize handwritten text well. We can see that it even keeps the capitalized format of the words "FORCED AND UNNATURAL"!

```Markdown
Today is Thursday, October 20th - But it definitely feels like a Friday. I'm already considering making a second cup of coffee - and I haven't even finished my first. Do I have a problem?
Sometimes I'll fly through older notes I've taken, and my handwriting is unrecamptable. Perhaps it depends on the type of pen I use. I've tried writing in all cups but it looks so FORCED AND UNNATURAL.
Often times, I'll just take notes on my lapten, but I still seem to ermittelt forward pen and paper. Any advice on what to
improve? I already feel stressed at looking back at what I've just written - it looks like I different people wrote this!
```

Now we can then insert the text into Milvus for semantic search.

```
from pymilvus import MilvusClient 

COLLECTION_NAME = "document_ocr"

milvus_client = MilvusClient(uri='http://localhost:19530')
"""
This is where you would define the index, create a collection etc. For the sake of this example. I am skipping it. 

schema = CollectionSchema(...)

milvus_client.create_collection(
    collection_name=COLLECTION_NAME,
    schema=schema,
    )

"""

milvus_client.insert(collection_name=COLLECTION_NAME, data=[result.pages[0].markdown])
```

But Mistral can also understand documents in different languages or in more complex format, for example let's try this invoice in German that combines some item names in English.

![An Invoice in German](https://assets.zilliz.com/An_Invoice_in_German_994e204d49.png)

Mistral OCR is still capable of extracting all the information you have and it even creates the table structure in Markdown that represents the table from the scanned image.

```
Rechnungsadresse:

Jähn Jessel GmbH a. Co. KG Marianne Scheibe Karla-Löffler-Weg 2 66522 Wismar

Lieferadresse:

Jähn Jessel GmbH a. Co. KG Marianne Scheibe Karla-Löffler-Weg 2 66522 Wismar

Rechnungsinformationen:

Bestelldatum: 2004-10-20
Bezahit: Ja
Expressversand: Nein
Rechnungsnummer: 4652

Rechnungsübersicht

| Pos. | Produkt | Preis <br> (Netto) | Menge | Steuersatz | Summe <br> Brutto |
| :--: | :--: | :--: | :--: | :--: | :--: |
| 1 | Grundig CH 7280w Multi-Zerkleinerer (Gourmet, 400 Watt, 11 Glasbehälter), weiß | 183.49 C | 2 | $0 \%$ | 366.98 C |
| 2 | Planet K | 349.9 C | 2 | $19.0 \%$ | 832.76 C |
| 3 | The Cabin in the Woods (Blu-ray) | 159.1 C | 2 | $7.0 \%$ | 340.47 C |
| 4 | Schenkung auf Italienisch Taschenbuch - 30. | 274.33 C | 4 | $19.0 \%$ | 1305.81 C |
| 5 | Xbox 360 - Razer 0N2A Controller Tournament Edition | 227.6 C | 2 | $7.0 \%$ | 487.06 C |
| 6 | Philips LED-Lampe ersetzt 25Watt E27 2700 Kelvin - warm-weiß, 2.7 Watt, 250 Lumen IEnergieklasse A++I | 347.57 C | 3 | $7.0 \%$ | 1115.7 C |
| 7 | Spannende Abenteuer Die verschollene Grabkammer | 242.8 C | 6 | $0 \%$ | 1456.8 C |
| Zw. summe |  | 1784.79 C |  |  |  |
| Zzgl. Mwst. 7\% |  | 51.4 C |  |  |  |
| Zzgl. Mwst. 19\% |  | 118.6 C |  |  |  |
| Gesamtbetrag C inkl. MwSt. |  | 1954.79 C |  |  |  |
```


## Real-World Usage: A Case Study

Now that we've seen that Mistral OCR can work on different documents, we could imagine how a legal firm that is drowning in case files and contracts leverage this tool. By implementing a RAG system with Mistral OCR and Milvus, what once took a paralegal countless hours, like manually scanning for specific clauses or comparing past cases, now is done by AI in only a couple of minutes.


### Next Steps

Ready to extract all your content? Head over to the [notebook on GitHub](https://github.com/milvus-io/bootcamp/blob/master/bootcamp/tutorials/integration/mistral_ocr_with_milvus.ipynb) for the full example, join our [Discord](http://zilliz.com/discord) to chat with the community, and start building today! You can also check out [Mistral documentation](https://docs.mistral.ai/capabilities/document/) about their OCR model&#x20;

Say goodbye to parsing chaos, and hello to intelligent, scalable document understanding.


Parsing is Hard: Solving Semantic Understanding with Mistral OCR and Milvus



## Everyone Became an Artist Overnight with GPT-4o

![](https://assets.zilliz.com/four_panel_1788f825e3.png)

_Believe it or not, the picture you just saw was AI-generated—specifically, by the newly released GPT-4o!_

When OpenAI launched GPT-4o's native image generation feature on March 26th, no one could have predicted the creative tsunami that followed. Overnight, the internet exploded with AI-generated Ghibli-style portraits—celebrities, politicians, pets, and even users themselves were transformed into charming Studio Ghibli characters with just a few simple prompts. The demand was so overwhelming that Sam Altman himself had to "plead" with users to slow down, tweeting that the OpenAI's "GPUs are melting."

![](https://assets.zilliz.com/Ghibli_32e739c2ac.png)

Example of GPT-4o generated images (credit X\@Jason Reid)

## Why GPT-4o Changes Everything 

For creative industries, this represents a paradigm shift. Tasks that once required an entire design team a whole day can now be completed in mere minutes. What makes GPT-4o different from previous image generators is **its remarkable visual consistency and intuitive interface**. It supports multi-turn conversations that let you refine images by adding elements, adjusting proportions, changing styles, or even transforming 2D into 3D—essentially putting a professional designer in your pocket.

The secret behind GPT-4o's superior performance? It’s autoregressive architecture. Unlike diffusion models (like Stable Diffusion) that degrade images into noise before reconstructing them, GPT-4o generates images sequentially—one token at a time—maintaining contextual awareness throughout the process. This fundamental architectural difference explains why GPT-4o produces more coherent results with more straightforward, more natural prompts.

But here's where things get interesting for developers: **An increasing number of signs point to a major trend—AI models themselves are becoming products. Simply put, most products that simply wrap large AI models around public domain data are at risk of being left behind.**

The true power of these advancements comes from combining general-purpose large models with **private, domain-specific data**. This combination may well be the optimal survival strategy for most companies in the era of large language models. As base models continue to evolve, the lasting competitive advantage will belong to those who can effectively integrate their proprietary datasets with these powerful AI systems.

Let's explore how to connect your private data with GPT-4o using Milvus, an open-source and high-performance vector database.

## Connecting Your Private Data with GPT-4o Using Milvus for More Curated Image Outputs 

Vector databases are the key technology bridging your private data with AI models. They work by converting your content—whether images, text, or audio—into mathematical representations (vectors) that capture their meaning and characteristics. This allows for a semantic search based on similarity rather than just keywords.

Milvus, as a leading open-source vector database, is particularly well-suited for connecting with generative AI tools like GPT-4o. Here's how I used it to solve a personal challenge.

### Background

One day, I had this brilliant idea—turn all the mischief of my dog Cola, into a comic strip. But there was a catch: How could I sift through tens of thousands of photos from work, travels, and food adventures to find Cola's mischievous moments?

The answer? Import all my photos into Milvus and do an image search.

Let's walk through the implementation step by step.

#### Dependencies and Environment

First, you need to get your environment ready with the right packages:

```
pip install pymilvus --upgrade
pip install torch numpy scikit-learn pillow
```

#### Prepare the Data

I'll use my photo library, which has about 30,000 photos, as the dataset in this guide. If you don't have any dataset at hand, download a sample dataset from Milvus and unzip it:

```
!wget https://github.com/milvus-io/pymilvus-assets/releases/download/imagedata/reverse_image_search.zip
!unzip -q -o reverse_image_search.zip
```

#### Define the Feature Extractor

We'll use the ResNet-50 mode from the `timm` library to extract embedding vectors from our images. This model has been trained on millions of images and can extract meaningful features that represent the visual content.

```
    import torch
    from PIL import Image
    import timm
    from sklearn.preprocessing import normalize
    from timm.data import resolve_data_config
    from timm.data.transforms_factory import create_transform
    class FeatureExtractor:
        def __init__(self, modelname):
            # Load the pre-trained model
            self.model = timm.create_model(
                modelname, pretrained=True, num_classes=0, global_pool="avg"
            )
            self.model.eval()
            # Get the input size required by the model
            self.input_size = self.model.default_cfg["input_size"]
            config = resolve_data_config({}, model=modelname)
            # Get the preprocessing function provided by TIMM for the model
            self.preprocess = create_transform(**config)
        def __call__(self, imagepath):
            # Preprocess the input image
            input_image = Image.open(imagepath).convert("RGB")  # Convert to RGB if needed
            input_image = self.preprocess(input_image)
            # Convert the image to a PyTorch tensor and add a batch dimension
            input_tensor = input_image.unsqueeze(0)
            # Perform inference
            with torch.no_grad():
                output = self.model(input_tensor)
            # Extract the feature vector
            feature_vector = output.squeeze().numpy()
            return normalize(feature_vector.reshape(1, -1), norm="l2").flatten()
```

#### Create a Milvus Collection

Next, we'll create a Milvus collection to store our image embeddings. Think of this as a specialized database explicitly designed for vector similarity search:

```
    from pymilvus import MilvusClient
    client = MilvusClient(uri="example.db")
    if client.has_collection(collection_name="image_embeddings"):
        client.drop_collection(collection_name="image_embeddings")

    client.create_collection(
        collection_name="image_embeddings",
        vector_field_name="vector",
        dimension=2048,
        auto_id=True,
        enable_dynamic_field=True,
        metric_type="COSINE",
    )
```

**Notes on MilvusClient Parameters:**

- **Local Setup:** Using a local file (e.g., `./milvus.db`) is the easiest way to get started—Milvus Lite will handle all your data.

- **Scale Up:** For large datasets, set up a robust Milvus server using Docker or Kubernetes and use its URI (e.g., `http://localhost:19530`).

- **Cloud Option:** If you’re into Zilliz Cloud (the fully managed service of Milvus), adjust your URI and token to match the public endpoint and API key.

#### Insert Image Embeddings into Milvus

Now comes the process of analyzing each image and storing its vector representation. This step might take some time depending on your dataset size, but it's a one-time process:

```
    import os
    from some_module import FeatureExtractor  # Replace with your feature extraction module
    extractor = FeatureExtractor("resnet50")
    root = "./train"  # Path to your dataset
    insert = True
    if insert:
        for dirpath, _, filenames in os.walk(root):
            for filename in filenames:
                if filename.endswith(".jpeg"):
                    filepath = os.path.join(dirpath, filename)
                    image_embedding = extractor(filepath)
                    client.insert(
                        "image_embeddings",
                        {"vector": image_embedding, "filename": filepath},
                    )
```

#### Conduct an Image Search

With our database populated, we can now search for similar images. This is where the magic happens—we can find visually similar photos using vector similarity:

```
    from IPython.display import display
    from PIL import Image
    query_image = "./search-image.jpeg"  # The image you want to search with
    results = client.search(
        "image_embeddings",
        data=[extractor(query_image)],
        output_fields=["filename"],
        search_params={"metric_type": "COSINE"},
        limit=6,  # Top-k results
    )
    images = []
    for result in results:
        for hit in result[:10]:
            filename = hit["entity"]["filename"]
            img = Image.open(filename)
            img = img.resize((150, 150))
            images.append(img)
    width = 150 * 3
    height = 150 * 2
    concatenated_image = Image.new("RGB", (width, height))
    for idx, img in enumerate(images):
        x = idx % 5
        y = idx // 5
        concatenated_image.paste(img, (x * 150, y * 150))

    display("query")
    display(Image.open(query_image).resize((150, 150)))
    display("results")
    display(concatenated_image)
```

**The returned images are shown as below:** 

![](https://assets.zilliz.com/640_1_8d4e88c6dd.png)

### Combine Vector Search with GPT-4o: Generating Ghibli-Style Images with Images Returned by Milvus 

Now comes the exciting part: using our image search results as input for GPT-4o to generate creative content. In my case, I wanted to create comic strips featuring my dog Cola based on photos I've taken.

The workflow is simple but powerful:

1. Use vector search to find relevant images of Cola from my collection

2. Feed these images to GPT-4o with creative prompts

3. Generate unique comics based on visual inspiration

Here are some examples of what this combination can produce:

**The prompts I use:** 

- _"Generate a four-panel, full-color, hilarious comic strip featuring a Border Collie caught gnawing on a mouse—with an awkward moment when the owner finds out."\
  _![](https://assets.zilliz.com/Screenshot_2025_04_02_at_11_34_43_1d7141eef3.png)__

- _"Draw a comic where this dog rocks a cute outfit."\
  _![](https://assets.zilliz.com/cutedog_6fdb1e9c79.png)__

- _"Using this dog as the model, create a comic strip of it attending Hogwarts School of Witchcraft and Wizardry."\
  _![](https://assets.zilliz.com/Screenshot_2025_04_02_at_11_44_00_ce932cd035.png)__

### A Few Quick Tips from My Experience of Image Generation:

1. **Keep it simple**: Unlike those finicky diffusion models, GPT-4o works best with straightforward prompts. I found myself writing shorter and shorter prompts as I went along, and getting better results.

2. **English works best**: I tried prompting in Chinese for some comics, but the results weren't great. I ended up writing my prompts in English and then translating the finished comics when needed.

3. **Not good for Video Generation**: Don’t get your hopes too high with Sora yet—AI-generated videos still have a way to go when it comes to fluid movement and coherent storylines.

## What's Next? My Perspective and Open for Discussion 

With AI-generated images leading the charge, a quick look at OpenAI's major releases over the past six months shows a clear pattern: whether it's GPTs for app marketplaces, DeepResearch for report generation, GPT-4o for conversational image creation, or Sora for video magic - large AI models are stepping from behind the curtain into the spotlight. What was once experimental tech is now maturing into real, usable products.

As GPT-4o and similar models become widely accepted, most workflows and intelligent agents based on Stable Diffusion are heading toward obsolescence. However, the irreplaceable value of private data and human insight remains strong. For example, while AI won't completely replace creative agencies, integrating a Milvus vector database with GPT models enables agencies to quickly generate fresh, creative ideas inspired by their past successes. E-commerce platforms can design personalized clothing based on shopping trends, and academic institutions can instantly create visuals for research papers.

The era of products powered by AI models is here, and the race to mine the data goldmine is just getting started. For developers and businesses alike, the message is clear: combine your unique data with these powerful models or risk being left behind.


Generating More Creative and Curated Ghibli-Style Images with GPT-4o and Milvus


Hey, Milvus users and contributors!

We're excited to share our [**Milvus 2025 roadmap**](https://milvus.io/docs/roadmap.md) with you. 🚀 This technical plan highlights the key features and improvements we’re building to make Milvus even more powerful for your vector search needs.

But this is just the beginning—we want your insights! Your feedback helps shape Milvus, ensuring it evolves to meet real-world challenges. Let us know what you think and help us refine the roadmap as we move forward.


## The Current Landscape

Over the past year, we've seen many of you build impressive RAG and agent applications with Milvus, leveraging many of our popular features, such as our model integration, full-text search, and hybrid search. Your implementations have provided valuable insights into real-world vector search requirements.

As AI technologies evolve, your use cases are becoming more sophisticated - from basic vector search to complex multimodal applications spanning intelligent agents, autonomous systems, and embodied AI. These technical challenges are informing our roadmap as we continue to develop Milvus to meet your needs.


## Two Major Releases in 2025: Milvus 2.6 and Milvus 3.0 

In 2025, we’re rolling out two major versions: Milvus 2.6 (Middle of CY25) and Milvus 3.0 (end of 2025). 

**Milvus 2.6** focuses on core architecture improvements you've been asking for:

- Simpler deployment with fewer dependencies (goodbye, deployment headaches!)

- Faster data ingestion pipelines

- Lower storage costs (we hear your production cost concerns)

- Better handling of large-scale data operations (delete/modify)

- More efficient scalar and full-text search

- Support for the latest embedding models you're working with

**Milvus 3.0** is our bigger architectural evolution, introducing a vector data lake system for:

- Seamless AI service integration

- Next-level search capabilities

- More robust data management

- Better handling of those massive offline datasets you're working with


## Technical Features We're Planning - We Need Your Feedback

Below are key technical features we are planning to add to Milvus. 


| **Key Feature Area**                   | **Technical Features**                                                                                                                                                                |
|----------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| **AI-Driven Unstructured Data Processing** | - Data-In/Out: Native integration with major model services for raw text ingestion<br>- Original Data Handling: Text/URL reference support for raw data processing<br>- Tensor Support: Vector list implementation (for ColBERT/CoPali/Video scenarios)<br>- Extended Data Types: DateTime, Map, GIS support based on requirements<br>- Iterative Search: Query vector refinement through user‘s feedback |
| **Search Quality & Performance Improvements** | - Advanced Matching: phrase_match & multi_match capabilities<br>- Analyzer Upgrade: Enhance Analyzer with expanded tokenizer support and improved observability<br>- JSON Optimization: Faster filtering through improved indexing<br>- Execution Sorting: Scalar field-based result ordering<br>- Advanced Reranker: Model-based reranking & custom scoring functions<br>- Iterative Search: Query vector refinement through user‘s feedback |
| **Data Management Flexibility** | - Schema Change: Add/delete field, modify varchar length<br>- Scalar Aggregations: count/distinct/min/max operations<br>- Support UDF: Support user-defined function<br>- Data Versioning: Snapshot-based rollback system<br>- Data Clustering: Co-location through configuration<br>- Data Sampling: Fast get results based on sampling data |
| **Architectural Improvements** | - Stream Node: Simplified incremental data ingestion<br>- MixCoord: Unified coordinator architecture<br>- Logstore Independence: Reduced external dependencies like pulsar<br>- PK Deduplication: Global primary key deduplication |
| **Cost Efficiency & Architecture Improvements** | - Tiered Storage: Hot/cold data separation for lower storage cost<br>- Data Evict Policy: Users can define their own data evict policy<br>- Bulk Updates: Support field-specific value modifications, ETL, etc<br>- Large TopK: Returns massive datasets<br>- VTS GA: Connect to different sources of data<br>- Advanced Quantization: Optimize memory consumption and performance based on quantization techniques<br>- Resource Elasticity: Dynamically scale resources to accommodate varying write loads, read loads, and background task loads |



As we implement this roadmap, we'd appreciate your thoughts and feedback on the following:

1. **Feature priorities:** Which features in our roadmap would have the most impact on your work?

2. **Implementation ideas:** Any specific approaches you think would work well for these features?

3. **Use case alignment:** How do these planned features align with your current and future use cases?

4. **Performance considerations:** Any performance aspects we should focus on for your specific needs?

**Your insights help us make Milvus better for everyone. Feel free to share your thoughts on our[ Milvus Discussion Forum](https://github.com/milvus-io/milvus/discussions/40263) or our [Discord Channel](https://discord.com/invite/8uyFbECzPX).**


## Welcome to Contribute to Milvus

As an open-source project, Milvus always welcomes your contributions:

- **Share feedback:** Report issues or suggest features through our [GitHub issue page](https://github.com/milvus-io/milvus/issues)

- **Code contributions:** Submit pull requests (see our [Contributor's Guide](https://github.com/milvus-io/milvus/blob/82915a9630ab0ff40d7891b97c367ede5726ff7c/CONTRIBUTING.md))

- **Spread the word:** Share your Milvus experiences and [star our GitHub repository](https://github.com/milvus-io/milvus)

We're excited to build this next chapter of Milvus with you. Your code, ideas, and feedback drive this project forward!

-- The Milvus Team 


Milvus 2025 Roadmap - Tell Us What You Think



DeepSeek quietly dropped a bombshell last night. Their latest release,[ DeepSeek v3-0324](https://huggingface.co/deepseek-ai/DeepSeek-V3-0324), was downplayed in the official announcement as just a **"minor upgrade"** with no API changes. But our extensive testing at [Zilliz](https://zilliz.com/) has revealed something more significant: this update represents a quantum leap in performance, particularly in logic reasoning, programming, and mathematical problem-solving.

What we're seeing isn't just incremental improvement – it's a fundamental shift that positions DeepSeek v3-0324 among the elite tier of language models. And it is open source. 

**This release deserves your immediate attention for developers and enterprises building AI-powered applications.**


## What's New in DeepSeek v3-0324 and How Good Is It Really?

DeepSeek v3-0324 introduces three major improvements over its predecessor, [DeepSeek v3](https://zilliz.com/blog/why-deepseek-v3-is-taking-the-ai-world-by-storm):

- **Larger Model, More Power:** The parameter count has increased from 671 billion to 685 billion, allowing the model to handle more complex reasoning and generate more nuanced responses.

- **A Massive Context Window:** With an upgraded 128K token context length, DeepSeek v3-0324 can retain and process significantly more information in a single query, making it ideal for long-form conversations, document analysis, and retrieval-based AI applications.

- **Enhanced Reasoning, Coding, and Math:** This update brings a noticeable boost in logic, programming, and mathematical capabilities, making it a strong contender for AI-assisted coding, scientific research, and enterprise-grade problem-solving.

But the raw numbers don't tell the whole story. What's truly impressive is how DeepSeek has managed to simultaneously enhance reasoning capacity and generation efficiency—something that typically involves engineering tradeoffs.


### The Secret Sauce: Architectural Innovation

Under the hood, DeepSeek v3-0324 retains its [Multi-head Latent Attention (MLA) ](https://arxiv.org/abs/2502.07864)architecture—an efficient mechanism that compresses Key-Value (KV) caches using latent vectors to reduce memory usage and computational overhead during inference. Additionally, it replaces traditional [Feed-Forward Networks (FFN)](https://zilliz.com/glossary/feedforward-neural-networks-(fnn)) with Mixture of Experts ([MoE](https://zilliz.com/learn/what-is-mixture-of-experts)) layers, optimizing compute efficiency by dynamically activating the best-performing experts for each token.

However, the most exciting upgrade is **multi-token prediction (MTP),** which allows each token to predict multiple future tokens simultaneously. This overcomes a significant bottleneck in traditional autoregressive models, improving both accuracy and inference speed. 

Together, these innovations create a model that doesn't just scale well – it scales intelligently, bringing professional-grade AI capabilities within reach of more development teams.


## Build a RAG System with Milvus and DeepSeek v3-0324 in 5 Minutes

DeepSeek v3-0324's powerful reasoning capabilities make it an ideal candidate for Retrieval-Augmented Generation (RAG) systems. In this tutorial, we'll show you how to build a complete RAG pipeline using DeepSeek v3-0324 and the [Milvus](https://zilliz.com/what-is-milvus) vector database in just five minutes. You'll learn how to retrieve and synthesize knowledge efficiently with minimal setup.


### Setting Up Your Environment

First, let's install the necessary dependencies:

```
! pip install --upgrade pymilvus[model] openai requests tqdm
```

**Note:** If you're using Google Colab, you'll need to restart the runtime after installing these packages. Click on the "Runtime" menu at the top of the screen and select "Restart session" from the dropdown menu.

Since DeepSeek provides an OpenAI-compatible API, you'll need an API key. You can get one by signing up on the[ DeepSeek platform](https://platform.deepseek.com/api_keys):

```
import os

os.environ["DEEPSEEK_API_KEY"] = "***********"
```
### Preparing Your Data

For this tutorial, we'll use the FAQ pages from the [Milvus Documentation 2.4.x](https://github.com/milvus-io/milvus-docs/releases/download/v2.4.6-preview/milvus_docs_2.4.x_en.zip) as our knowledge source: 

```
! wget https://github.com/milvus-io/milvus-docs/releases/download/v2.4.6-preview/milvus_docs_2.4.x_en.zip
! unzip -q milvus_docs_2.4.x_en.zip -d milvus_docs
```
Now, let's load and prepare the FAQ content from the markdown files:

```
from glob import glob

# Load all markdown files from the FAQ directory
text_lines = []
for file_path in glob("milvus_docs/en/faq/*.md", recursive=True):
    with open(file_path, "r") as file:
        file_text = file.read()
        # Split on headings to separate content sections
        text_lines += file_text.split("# ")
```
### Setting Up the Language and Embedding Models

We'll use [OpenRouter](https://openrouter.ai/) to access DeepSeek v3-0324. OpenRouter provides a unified API for multiple AI models, such as DeepSeek and Claude. By creating a free DeepSeek V3 API key on OpenRouter, you can easily try out DeepSeek V3 0324. 

https://assets.zilliz.com/Setting_Up_the_Language_and_Embedding_Models_8b00595a6b.png

```
from openai import OpenAI

deepseek_client = OpenAI(
   api_key="<OPENROUTER_API_KEY>",
   base_url="https://openrouter.ai/api/v1",
)
```
For text embeddings, we'll use Milvus' [built-in embedding model](https://milvus.io/docs/embeddings.md), which is lightweight and effective:

```
from pymilvus import model as milvus_model

# Initialize the embedding model
embedding_model = milvus_model.DefaultEmbeddingFunction()

# Test the embedding model
test_embedding = embedding_model.encode_queries(["This is a test"])[0]
embedding_dim = len(test_embedding)
print(f"Embedding dimension: {embedding_dim}")
print(f"First 10 values: {test_embedding[:10]}")
```
### Creating a Milvus Collection

Now let's set up our vector database using Milvus:

```
from pymilvus import MilvusClient

# Initialize Milvus client (using Milvus Lite for simplicity)
milvus_client = MilvusClient(uri="./milvus_demo.db")
collection_name = "my_rag_collection"

# Remove existing collection if it exists
if milvus_client.has_collection(collection_name):
    milvus_client.drop_collection(collection_name)

# Create a new collection
milvus_client.create_collection(
    collection_name=collection_name,
    dimension=embedding_dim,
    metric_type="IP",  # Inner product distance
    consistency_level="Strong",  # See https://milvus.io/docs/consistency.md for details
)
```

**Pro Tip**: For different deployment scenarios, you can adjust your Milvus setup:

- For local development: Use `uri="./milvus.db"` with [Milvus Lite](https://milvus.io/docs/milvus_lite.md)

- For larger datasets: Set up a Milvus server via [Docker/Kubernetes](https://milvus.io/docs/quickstart.md) and use `uri="http://localhost:19530"`

- For production: Use[ Zilliz Cloud](https://zilliz.com/cloud) with your cloud endpoint and API key.


### Loading Data into Milvus

Let's convert our text data into embeddings and store them in Milvus:

```
from tqdm import tqdm

# Create embeddings for all text chunks
data = []
doc_embeddings = embedding_model.encode_documents(text_lines)

# Create records with IDs, vectors, and text
for i, line in enumerate(tqdm(text_lines, desc="Creating embeddings")):
    data.append({"id": i, "vector": doc_embeddings[i], "text": line})

# Insert data into Milvus
milvus_client.insert(collection_name=collection_name, data=data)
```

```
Creating embeddings:   0%|          | 0/72 [00:00<?, ?it/s]huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
    - Avoid using `tokenizers` before the fork if possible
    - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
Creating embeddings: 100%|██████████| 72/72 [00:00<00:00, 246522.36it/s]





{'insert_count': 72, 'ids': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71], 'cost': 0}
```

### Building the RAG Pipeline

#### Step 1: Retrieve Relevant Information

Let's test our RAG system with a common question:

```
question = "How is data stored in milvus?"

# Search for relevant information
search_res = milvus_client.search(
    collection_name=collection_name,
    data=embedding_model.encode_queries([question]),  # Convert question to embedding
    limit=3,  # Return top 3 results
    search_params={"metric_type": "IP", "params": {}},  # Inner product distance
    output_fields=["text"],  # Return the text field
)

# Examine search results
import json
retrieved_lines_with_distances = [
    (res["entity"]["text"], res["distance"]) for res in search_res[0]
]
print(json.dumps(retrieved_lines_with_distances, indent=4))
```

```
[
    [
        " Where does Milvus store data?\n\nMilvus deals with two types of data, inserted data and metadata. \n\nInserted data, including vector data, scalar data, and collection-specific schema, are stored in persistent storage as incremental log. Milvus supports multiple object storage backends, including [MinIO](https://min.io/), [AWS S3](https://aws.amazon.com/s3/?nc1=h_ls), [Google Cloud Storage](https://cloud.google.com/storage?hl=en#object-storage-for-companies-of-all-sizes) (GCS), [Azure Blob Storage](https://azure.microsoft.com/en-us/products/storage/blobs), [Alibaba Cloud OSS](https://www.alibabacloud.com/product/object-storage-service), and [Tencent Cloud Object Storage](https://www.tencentcloud.com/products/cos) (COS).\n\nMetadata are generated within Milvus. Each Milvus module has its own metadata that are stored in etcd.\n\n###",
        0.6572665572166443
    ],
    [
        "How does Milvus flush data?\n\nMilvus returns success when inserted data are loaded to the message queue. However, the data are not yet flushed to the disk. Then Milvus' data node writes the data in the message queue to persistent storage as incremental logs. If `flush()` is called, the data node is forced to write all data in the message queue to persistent storage immediately.\n\n###",
        0.6312146186828613
    ],
    [
        "How does Milvus handle vector data types and precision?\n\nMilvus supports Binary, Float32, Float16, and BFloat16 vector types.\n\n- Binary vectors: Store binary data as sequences of 0s and 1s, used in image processing and information retrieval.\n- Float32 vectors: Default storage with a precision of about 7 decimal digits. Even Float64 values are stored with Float32 precision, leading to potential precision loss upon retrieval.\n- Float16 and BFloat16 vectors: Offer reduced precision and memory usage. Float16 is suitable for applications with limited bandwidth and storage, while BFloat16 balances range and efficiency, commonly used in deep learning to reduce computational requirements without significantly impacting accuracy.\n\n###",
        0.6115777492523193
    ]
]
```

#### Step 2: Generate a Response with DeepSeek

Now let's use DeepSeek to generate a response based on the retrieved information:

```
# Combine retrieved text chunks
context = "\n".join(
    [line_with_distance[0] for line_with_distance in retrieved_lines_with_distances]
)

# Define prompts for the language model
SYSTEM_PROMPT = """
You are an AI assistant. You are able to find answers to the questions from the contextual passage snippets provided.
"""

USER_PROMPT = f"""
Use the following pieces of information enclosed in <context> tags to provide an answer to the question enclosed in <question> tags.

<context>
{context}
</context>

<question>
{question}
</question>
"""

# Generate response with DeepSeek
response = deepseek_client.chat.completions.create(
    model="deepseek-chat",
    messages=[
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": USER_PROMPT},
    ],
)

print(response.choices[0].message.content)
```


```
In Milvus, data is stored in two main categories: inserted data and metadata.

1. **Inserted Data**: This includes vector data, scalar data, and collection-specific schema. The inserted data is stored in persistent storage as incremental logs. Milvus supports various object storage backends for this purpose, such as MinIO, AWS S3, Google Cloud Storage (GCS), Azure Blob Storage, Alibaba Cloud OSS, and Tencent Cloud Object Storage (COS).

2. **Metadata**: Metadata is generated within Milvus and is specific to each Milvus module. This metadata is stored in etcd, a distributed key-value store.

Additionally, when data is inserted, it is first loaded into a message queue, and Milvus returns success at this stage. The data is then written to persistent storage as incremental logs by the data node. If the `flush()` function is called, the data node is forced to write all data in the message queue to persistent storage immediately.
```

And there you have it! You've successfully built a complete RAG pipeline with DeepSeek v3-0324 and Milvus. This system can now answer questions based on the Milvus documentation with high accuracy and contextual awareness.

## Comparing DeepSeek-V3-0324: Original vs. RAG-Enhanced Version

Theory is one thing, but real-world performance is what matters. We tested both the standard DeepSeek v3-0324 (with "Deep Thinking" disabled) and our RAG-enhanced version with the same prompt: *Write HTML code to create a fancy website about Milvus.*


### Website Built with The Standard Model's Output Code 

Here's what the website looks like: 

![](https://assets.zilliz.com/Website_Built_with_The_Standard_Model_s_Output_Code_695902b18a.png)

While visually appealing, the content relies heavily on generic descriptions and misses many of Milvus' core technical features.


### Website Built with Code Generated by the RAG-Enhanced Version

When we integrated Milvus as the knowledge base, the results were dramatically different:

![](https://assets.zilliz.com/Website_2_01341c647c.png)

The latter website doesn't just look better – it demonstrates genuine understanding of Milvus' architecture, use cases, and technical advantages. 


## Can DeepSeek v3-0324 Replace Dedicated Reasoning Models?

Our most surprising discovery came when comparing DeepSeek v3-0324 against specialized reasoning models like Claude 3.7 Sonnet and GPT-4 Turbo across mathematical, logical, and code reasoning tasks.

While dedicated reasoning models excel at multi-step problem solving, they often do so at the cost of efficiency. Our benchmarks showed that reasoning-heavy models frequently overanalyze simple prompts, generating 2-3x more tokens than necessary and significantly increasing latency and API costs.

DeepSeek v3-0324 takes a different approach. It demonstrates comparable logical consistency but with remarkably greater conciseness – often producing correct solutions with 40-60% fewer tokens. This efficiency doesn't come at the expense of accuracy; in our code generation tests, DeepSeek's solutions matched or exceeded the functionality of those from reasoning-focused competitors.

For developers balancing performance with budget constraints, this efficiency advantage translates directly to lower API costs and faster response times – crucial factors for production applications where user experience hinges on perceived speed.


## The Future of AI Models: Blurring the Reasoning Divide

DeepSeek v3-0324's performance challenges a core assumption in the AI industry: that reasoning and efficiency represent an unavoidable tradeoff. This suggests we may be approaching an inflection point where the distinction between reasoning and non-reasoning models begins to blur.

Leading AI providers may eventually eliminate this distinction entirely, developing models that dynamically adjust their reasoning depth based on task complexity. Such adaptive reasoning would optimize both computational efficiency and response quality, potentially revolutionizing how we build and deploy AI applications.

For developers building RAG systems, this evolution promises more cost-effective solutions that deliver the reasoning depth of premium models without their computational overhead – expanding what's possible with open-source AI.


DeepSeek V3-0324: The "Minor Update" That's Crushing Top AI Models


What Exactly is a Vector Database and How Does It Work

Stop Building Vanilla RAG: Embrace Agentic RAG with DeepSearcher



_"We initially built our semantic search on pgvector instead of Milvus because all our relational data was already in PostgreSQL,"_ recalls Alex, CTO of an enterprise AI SaaS startup. _"But as soon as we hit product-market fit, our growth ran into serious hurdles on the engineering side. It quickly became clear that pgvector wasn’t designed for scalability. Simple tasks such as rolling out schema updates across multiple shards turned into tedious, error-prone processes that consumed days of engineering effort. When we reached 100 million vector embeddings, query latency spiked to over a second, something far beyond what our customers would tolerate. After moving to Milvus, sharding manually felt like stepping into the stone age. It’s no fun juggling shard servers as if they were fragile artifacts. No company should have to endure that."_


## A Common Challenge for AI Companies

Alex's experience isn't unique to pgvector users. Whether you're using pgvector, Qdrant, Weaviate, or any other vector database that relies on manual sharding, the scaling challenges remain the same. What starts as a manageable solution quickly turns into a tech debt as data volumes grow.

For startups today, **scalability isn't optional—it's mission-critical**. This is especially true for AI products powered by Large Language Models(LLM) and vector databases, where the leap from early adoption to exponential growth can happen overnight. Achieving product-market fit often triggers a surge in user growth, overwhelming data inflows, and skyrocketing query demands. But if the database infrastructure can't keep up, slow queries and operational inefficiencies can stall momentum and hinder business success.

A short-term technical decision could lead to long-term bottleneck, forcing engineering teams to constantly address urgent performance issues, database crashes, and system failures instead of focusing on innovation. The worst-case scenario? A costly, time-consuming database re-architecture—precisely when a company should be scaling.


## Isn’t Sharding a Natural Solution to Scalability?

Scalability can be addressed in multiple ways. The most straightforward approach, **Scaling Up**, involves enhancing a single machine’s resources by adding more CPU, memory, or storage to accommodate growing data volumes. While simple, this method has clear limitations. In a Kubernetes environment, for example, large pods are inefficient, and relying on a single node increases the risk of failure, potentially leading to significant downtime.

When Scaling Up is no longer viable, businesses naturally turn to **Scaling Out**, distributing data across multiple servers. At first glance, **sharding** appears to be a simple solution—splitting a database into smaller, independent databases to increase capacity and enable multiple writable primary nodes.

However, while conceptually straightforward, sharding quickly becomes a complex challenge in practice. Most applications are initially designed to work with a single, unified database. The moment a vector database is divided into multiple shards, every part of the application that interacts with data must be modified or entirely rewritten, introducing significant development overhead. Designing an effective sharding strategy becomes crucial, as does implementing routing logic to ensure data is directed to the correct shard. Managing atomic transactions across multiple shards often requires restructuring applications to avoid cross-shard operations. Additionally, failure scenarios must be handled gracefully to prevent disruptions when certain shards become unavailable.


## Why Manual Sharding Becomes a Burden

_"We originally estimated implementing manual sharding for our pgvector database would take two engineers about six months,"_ Alex remembers. _"What we didn't realize was that those engineers would_ **_always_** _be needed. Every schema change, data rebalancing operation, or scaling decision required their specialized expertise. We were essentially committing to a permanent 'sharding team' just to keep our database running."_

Real-world challenges with sharded vector databases include:

1. **Data Distribution Imbalance (Hotspots)**: In multi-tenant use cases, data distribution can range from hundreds to billions of vectors per tenant. This imbalance creates hotspots where certain shards become overloaded while others sit idle.

2. **The Resharding Headache**: Choosing the right number of shards is nearly impossible. Too few leads to frequent and costly resharding operations. Too many creates unnecessary metadata overhead, increasing complexity and reducing performance.

3. **Schema Change Complexity**: Many vector databases implement sharding by managing multiple underlying databases. This makes synchronizing schema changes across shards cumbersome and error-prone, slowing development cycles.

4. **Resource Waste**: In storage-compute coupled databases, you must meticulously allocate resources across every node while anticipating future growth. Typically, when resource utilization reaches 60-70%, you need to start planning for resharding.

Simply put, **managing shards manually is bad for your business**. Instead of locking your engineering team into constant shard management, consider investing in a vector database designed to scale automatically—without the operational burden.


## How Milvus Solves the Scalability Problem

Many developers—from startups to enterprises—have recognized the significant overhead associated with manual database sharding. Milvus takes a fundamentally different approach, enabling seamless scaling from millions to billions of vectors without the complexity.


### Automated Scaling Without the Tech Debt

Milvus leverages Kubernetes and a disaggregated storage-compute architecture to support seamless expansion. This design enables:

- Rapid scaling in response to changing demands

- Automatic load balancing across all available nodes

- Independent resource allocation, letting you adjust compute, memory, and storage separately

- Consistent high performance, even during periods of rapid growth


### Distributed Architecture Designed from the Ground Up

Milvus achieves its scaling capabilities through two key innovations:

**Segment-Based Architecture:** At its core, Milvus organizes data into "segments"—the smallest units of data management:

- Growing Segments reside on StreamNodes, optimizing data freshness for real-time queries

- Sealed Segments are managed by QueryNodes, utilizing powerful indexes to accelerate search

- These segments are evenly distributed across nodes to optimize parallel processing

**Two-Layer Routing**: Unlike traditional databases where each shard lives on a single machine, Milvus distributes data in one shard dynamically across multiple nodes:

- Each shard can store over 1 billion data points

- Segments within each shard are automatically balanced across machines

- Expanding collections is as simple as increasing the number of shards

- The upcoming Milvus 3.0 will introduce dynamic shard splitting, eliminating even this minimal manual step


### Query Processing at Scale

When executing a query, Milvus follows an efficient process:

1. The Proxy identifies relevant shards for the requested collection

2. The Proxy gathers data from both StreamNodes and QueryNodes

3. StreamNodes handle real-time data while QueryNodes process historical data concurrently

4. Results are aggregated and returned to the user


![](https://assets.zilliz.com/Query_Processing_at_Scale_5792dc9e37.png)

## A Different Engineering Experience

_"When scalability is built into the database itself, all those headaches just... disappear,"_ says Alex, reflecting on his team's transition to Milvus. _"My engineers are back to building features customers love instead of babysitting database shards."_

If you're grappling with the engineering burden of manual sharding, performance bottlenecks at scale, or the daunting prospect of database migrations, it's time to rethink your approach. Visit our [docs page](https://milvus.io/docs/overview.md#What-Makes-Milvus-so-Scalable) to learn more about Milvus architecture, or experience effortless scalability firsthand with fully-managed Milvus at [zilliz.com/cloud](https://zilliz.com/cloud).

With the right vector database foundation, your innovation knows no limits.



Why Manual Sharding is a Bad Idea for Vector Database And How to Fix It



Building a performant [vector database](https://zilliz.com/learn/what-is-vector-database) like Milvus that scales to billions of vectors and handles web-scale traffic is no simple feat. It requires the careful, intelligent design of a distributed system. Necessarily, there will be a trade-off between performance vs simplicity in the internals of a system like this.

While we have tried to well balance this trade-off, some aspects of the internals have remained opaque. This article aims to dispel any mystery around how Milvus breaks down data insertion, indexing, and serving across nodes. Understanding these processes at a high level is essential for effectively optimizing query performance, system stability, and debugging-related issues.

So, let’s take a stroll in a day in the life of Dave, the Milvus datum. Imagine you insert Dave into your collection in a [Milvus Distributed deployment](https://milvus.io/docs/install-overview.md#Milvus-Distributed) (see the diagram below). As far as you are concerned, he goes directly into the collection. Behind the scenes, however, many steps occur across independent sub-systems.

![](https://assets.zilliz.com/a_day_in_the_life_of_a_milvus_datum_ca279f7f59.png)


## Proxy Nodes and the Message Queue

![](https://assets.zilliz.com/Proxy_Nodes_and_the_Message_Queue_03a0fde0c5.png)

Initially, you call the MilvusClient object, for example, via the PyMilvus library, and send an `_insert()`_ request to a _proxy node_. Proxy nodes are the gateway between the user and the database system, performing operations like load balancing on incoming traffic and collating multiple outputs before they are returned to the user.

A hash function is applied to the item’s primary key to determine which _channel_ to send it to. Channels, implemented with either Pulsar or Kafka topics, represent a holding ground for streaming data, which can then be sent onwards to subscribers of the channel.


## Data Nodes, Segments, and Chunks

![](https://assets.zilliz.com/Data_Nodes_Segments_and_Chunks_ae122dd1ac.png)

After the data has been sent to the appropriate channel, the channel then sends it to the corresponding segment in the _data node_. Data nodes are responsible for storing and managing data buffers called _growing segments_. There is one growing segment per shard.

As data is inserted into a segment, the segment grows towards a maximum size, defaulting to 122MB. During this time, smaller parts of the segment, by default 16MB and known as _chunks_, are pushed to persistent storage, for example, using AWS’s S3 or other compatible storage like MinIO. Each chunk is a physical file on the object storage and there is a separate file per field. See the figure above illustrating the file hierarchy on object storage.

So to summarize, a collection’s data is split across data nodes, within which it is split into segments for buffering, which are further split into per-field chunks for persistent storage. The two diagrams above make this clearer. By dividing the incoming data in this way, we fully exploit the cluster’s parallelism of network bandwidth, compute, and storage.


## Sealing, Merging, and Compacting Segments

![](https://assets.zilliz.com/Sealing_Merging_and_Compacting_Segments_d5a6a37261.png)

Thus far we have told the story of how our friendly datum Dave makes his way from an `_insert()`_ query into persistent storage. Of course, his story does not end there. There are further steps to make the search and indexing process more efficient. By managing the size and number of segments, the system fully exploits the cluster’s parallelism.

Once a segment reaches its maximum size on a data node, by default 122MB, it is said to be _sealed_. What this means is that the buffer on the data node is cleared to make way for a new segment, and the corresponding chunks in persistent storage are marked as belonging to a closed segment.

The data nodes periodically look for smaller sealed segments and merge them into larger ones until they have reached a maximum size of 1GB (by default) per segment. Recall that when an item is deleted in Milvus, it is simply marked with a deletion flag - think of it as Death Row for Dave. When the number of deleted items in a segment passes a given threshold, by default 20%, the segment is reduced in size, an operation we call _compaction_.


Indexing and Searching through Segments

![](https://assets.zilliz.com/Indexing_and_Searching_through_Segments_478c0067be.png)

![](https://assets.zilliz.com/Indexing_and_Searching_through_Segments_1_0c31b5a340.png)

There is an additional node type, the _index node_, that is responsible for building indexes for sealed segments. When the segment is sealed, the data node sends a request for an index node to construct an index. The index node then sends the completed index to object storage. Each sealed segment has its own index stored in a separate file. You can examine this file manually by accessing the bucket - see the figure above for the file hierarchy.

Query nodes - not only data nodes - subscribe to the message queue topics for the corresponding shards. The growing segments are replicated on the query nodes, and the node loads into memory sealed segments belonging to the collection as required. It builds an index for each growing segment as data comes in, and loads the finalized indexes for sealed segments from the data store.

Imagine now that you call the MilvusClient object with a _search()_ request that encompasses Dave. After being routed to all query nodes via the proxy node, each query node performs a vector similarity search (or another one of the search methods like query, range search, or grouping search), iterating over the segments one by one. The results are collated across nodes in a MapReduce-like fashion and sent back to the user, Dave being happy to find himself reunited with you at last.


## Discussion

We have covered a day in the life of Dave the datum, both for `_insert()`_ and `_search()`_ operations. Other operations like `_delete()`_ and `_upsert()`_ work similarly. Inevitably, we have had to simplify our discussion and omit finer details. On the whole, though, you should now have a sufficient picture of how Milvus is designed for parallelism across nodes in a distributed system to be robust and efficient, and how you can use this for optimization and debugging. 

_An important takeaway from this article: Milvus is designed with a separation of concerns across node types. Each node type has a specific, mutually exclusive function, and there is a separation of storage and compute._ The result is that each component can be scaled independently with parameters tweakable according to the use case and traffic patterns. For example, you can scale the number of query nodes to serve increased traffic without scaling data and index nodes. With that flexibility, there are Milvus users that handle billions of vectors and serve web-scale traffic, with sub-100ms query latency.

You can also enjoy the benefits of Milvus’ distributed design without even deploying a distributed cluster through [Zilliz Cloud](https://zilliz.com/cloud), a fully managed service of Milvus. [Sign up today for the free-tier of Zilliz Cloud and put Dave into action!](https://cloud.zilliz.com/signup)


A Day in the Life of a Milvus Datum

AI for Smarter Browsing: Filtering Web Content with Pixtral, Milvus, and Browser Use




![DeepSearcher](https://assets.zilliz.com/deep_researcher_a0170dadd0.gif)

In the previous post, [_“I Built a Deep Research with Open Source—and So Can You!”_](https://milvus.io/blog/i-built-a-deep-research-with-open-source-so-can-you.md), we explained some of the principles underlying research agents and constructed a simple prototype that generates detailed reports on a given topic or question. The article and corresponding notebook demonstrated the fundamental concepts of _tool use_, _query decomposition_, _reasoning_, and _reflection_. The example in our previous post, in contrast to OpenAI’s Deep Research, ran locally, using only open-source models and tools like [Milvus](https://milvus.io/docs) and LangChain. (I encourage you to read the [above article](https://milvus.io/blog/i-built-a-deep-research-with-open-source-so-can-you.md) before continuing.) 

In the following weeks, there was an explosion of interest in understanding and reproducing OpenAI’s Deep Research. See, for example, [Perplexity Deep Research](https://www.perplexity.ai/hub/blog/introducing-perplexity-deep-research) and [Hugging Face's Open DeepResearch](https://huggingface.co/blog/open-deep-research). These tools differ in architecture and methodology although sharing an objective: iteratively research a topic or question by surfing the web or internal documents and output a detailed, informed, and well-structured report. Importantly, the underlying agent automates reasoning about what action to take at each intermediate step.

In this post, we build upon our previous post and present Zilliz’s [DeepSearcher](https://github.com/zilliztech/deep-searcher) open-source project. Our agent demonstrates additional concepts: _query routing, conditional execution flow_, and _web crawling as a tool_. It is presented as a Python library and command-line tool rather than a Jupyter notebook and is more fully-featured than our previous post. For example, it can input multiple source documents and can set the embedding model and vector database used via a configuration file. While still relatively simple, DeepSearcher is a great showcase of agentic RAG and is a further step towards a state-of-the-art AI applications.

Additionally, we explore the need for faster and more efficient inference services. Reasoning models make use of “inference scaling”, that is, extra computation, to improve their output, and that combined with the fact that a single report may require hundreds or thousands of LLM calls results in inference bandwidth being the primary bottleneck. We use the [DeepSeek-R1 reasoning model on SambaNova’s custom-built hardware](https://sambanova.ai/press/fastest-deepseek-r1-671b-with-highest-efficiency), which is twice as fast in output tokens-per-second as the nearest competitor (see figure below).

SambaNova Cloud also provides inference-as-a-service for other open-source models including Llama 3.x, Qwen2.5, and QwQ. The inference service runs on SambaNova’s custom chip called the reconfigurable dataflow unit (RDU), which is specially designed for efficient inference on Generative AI models, lowering cost and increasing inference speed. [Find out more on their website.](https://sambanova.ai/technology/sn40l-rdu-ai-chip)

![Output Speed- DeepSeek R1](https://assets.zilliz.com/Output_speed_deepseek_r1_d820329f0a.png)


## DeepSearcher Architecture



The architecture of [DeepSearcher](https://github.com/zilliztech/deep-searcher) follows our previous post by breaking the problem up into four steps - _define/refine the question_, _research_, _analyze_, _synthesize_ - although this time with some overlap. We go through each step, highlighting [DeepSearcher](https://github.com/zilliztech/deep-searcher)’s improvements.

![DeepSearcher Architecture](https://assets.zilliz.com/deepsearcher_architecture_088c7066d1.png)

### Define and Refine the Question

```txt
Break down the original query into new sub queries: [
  'How has the cultural impact and societal relevance of The Simpsons evolved from its debut to the present?',
  'What changes in character development, humor, and storytelling styles have occurred across different seasons of The Simpsons?', 
  'How has the animation style and production technology of The Simpsons changed over time?',
  'How have audience demographics, reception, and ratings of The Simpsons shifted throughout its run?']
```


In the design of DeepSearcher, the boundaries between researching and refining the question are blurred. The initial user query is decomposed into sub-queries, much like the previous post. See above for initial subqueries produced from the query “How has The Simpsons changed over time?”. However, the following research step will continue to refine the question as needed.


### Research and Analyze

Having broken down the query into sub-queries, the research portion of the agent begins. It has, roughly speaking, four steps: _routing_, _search_, _reflection, and conditional repeat_.


#### Routing

Our database contains multiple tables or collections from different sources. It would be more efficient if we could restrict our semantic search to only those sources that are relevant to the query at hand. A query router prompts an LLM to decide from which collections information should be retrieved.

Here is the method to form the query routing prompt:

```python
def get_vector_db_search_prompt(
    question: str,
    collection_names: List[str],
    collection_descriptions: List[str],
    context: List[str] = None,
):
    sections = []
    # common prompt
    common_prompt = f"""You are an advanced AI problem analyst. Use your reasoning ability and historical conversation information, based on all the existing data sets, to get absolutely accurate answers to the following questions, and generate a suitable question for each data set according to the data set description that may be related to the question.

Question: {question}
"""
    sections.append(common_prompt)
    
    # data set prompt
    data_set = []
    for i, collection_name in enumerate(collection_names):
        data_set.append(f"{collection_name}: {collection_descriptions[i]}")
    data_set_prompt = f"""The following is all the data set information. The format of data set information is data set name: data set description.

Data Sets And Descriptions:
"""
    sections.append(data_set_prompt + "\n".join(data_set))
    
    # context prompt
    if context:
        context_prompt = f"""The following is a condensed version of the historical conversation. This information needs to be combined in this analysis to generate questions that are closer to the answer. You must not generate the same or similar questions for the same data set, nor can you regenerate questions for data sets that have been determined to be unrelated.

Historical Conversation:
"""
        sections.append(context_prompt + "\n".join(context))
    
    # response prompt
    response_prompt = f"""Based on the above, you can only select a few datasets from the following dataset list to generate appropriate related questions for the selected datasets in order to solve the above problems. The output format is json, where the key is the name of the dataset and the value is the corresponding generated question.

Data Sets:
"""
    sections.append(response_prompt + "\n".join(collection_names))
    
    footer = """Respond exclusively in valid JSON format matching exact JSON schema.

Critical Requirements:
- Include ONLY ONE action type
- Never add unsupported keys
- Exclude all non-JSON text, markdown, or explanations
- Maintain strict JSON syntax"""
    sections.append(footer)
    return "\n\n".join(sections)
```



We make the LLM return structured output as JSON in order to easily convert its output to a decision on what to do next.


#### Search

Having selected various database collections via the previous step, the search step performs a similarity search with [Milvus](https://milvus.io/docs). Much like the previous post, the source data has been specified in advance, chunked, embedded, and stored in the vector database. For DeepSearcher, the data sources, both local and online, must be manually specified. We leave online search for future work.


#### Reflection

Unlike the previous post, DeepSearcher illustrates a true form of agentic reflection, inputting the prior outputs as context into a prompt that “reflects” on whether the questions asked so far and the relevant retrieved chunks contain any informational gaps. This can be seen as an analysis step.

Here is the method to create the prompt:


```python
def get_reflect_prompt(
   question: str,
   mini_questions: List[str],
   mini_chuncks: List[str],
):
    mini_chunk_str = ""
    for i, chunk in enumerate(mini_chuncks):
        mini_chunk_str += f"""<chunk_{i}>\n{chunk}\n</chunk_{i}>\n"""
    reflect_prompt = f"""Determine whether additional search queries are needed based on the original query, previous sub queries, and all retrieved document chunks. If further research is required, provide a Python list of up to 3 search queries. If no further research is required, return an empty list.

If the original query is to write a report, then you prefer to generate some further queries, instead return an empty list.

    Original Query: {question}
    Previous Sub Queries: {mini_questions}
    Related Chunks: 
    {mini_chunk_str}
    """
   
    
    footer = """Respond exclusively in valid List of str format without any other text."""
    return reflect_prompt + footer
```



Once more, we make the LLM return structured output, this time as Python-interpretable data.

Here is an example of new sub-queries “discovered” by reflection after answering the initial sub-queries above:

```
New search queries for next iteration: [
  "How have changes in The Simpsons' voice cast and production team influenced the show's evolution over different seasons?",
  "What role has The Simpsons' satire and social commentary played in its adaptation to contemporary issues across decades?",
  'How has The Simpsons addressed and incorporated shifts in media consumption, such as streaming services, into its distribution and content strategies?']
```



#### Conditional Repeat

Unlike our previous post, DeepSearcher illustrates conditional execution flow. After reflecting on whether the questions and answers so far are complete, if there are additional questions to be asked the agent repeats the above steps. Importantly, the execution flow (a while loop) is a function of the LLM output rather than being hard-coded. In this case there is only a binary choice: _repeat research_ or _generate a report_. In more complex agents there may be several such as: _follow hyperlink_, _retrieve chunks, store in memory, reflect_ etc. In this way, the question continues to be refined as the agent sees fit until it decides to exit the loop and generate the report. In our Simpsons example, DeepSearcher performs two more rounds of filling the gaps with extra sub-queries.


### Synthesize

Finally, the fully decomposed question and retrieved chunks are synthesized into a report with a single prompt. Here is the code to create the prompt:

```python
def get_final_answer_prompt(
   question: str, 
   mini_questions: List[str],
   mini_chuncks: List[str],
):
    mini_chunk_str = ""
    for i, chunk in enumerate(mini_chuncks):
        mini_chunk_str += f"""<chunk_{i}>\n{chunk}\n</chunk_{i}>\n"""
    summary_prompt = f"""You are an AI content analysis expert, good at summarizing content. Please summarize a specific and detailed answer or report based on the previous queries and the retrieved document chunks.

    Original Query: {question}
    Previous Sub Queries: {mini_questions}
    Related Chunks: 
    {mini_chunk_str}
    """
    return summary_prompt
```


This approach has the advantage over our prototype, which analyzed each question separately and simply concatenated the output, of producing a report where all sections are consistent with each other, i.e., containing no repeated or contradictory information. A more complex system could combine aspects of both, using a conditional execution flow to structure the report, summarize, rewrite, reflect and pivot, and so on, which we leave for future work.


## Results

Here is a sample from the report generated by the query “How has The Simpsons changed over time?” with DeepSeek-R1 passing the Wikipedia page on The Simpsons as source material:

```txt
Report: The Evolution of The Simpsons (1989–Present)
1. Cultural Impact and Societal Relevance
The Simpsons debuted as a subversive critique of American middle-class life, gaining notoriety for its bold satire in the 1990s. Initially a countercultural phenomenon, it challenged norms with episodes tackling religion, politics, and consumerism. Over time, its cultural dominance waned as competitors like South Park and Family Guy pushed boundaries further. By the 2010s, the show transitioned from trendsetter to nostalgic institution, balancing legacy appeal with attempts to address modern issues like climate change and LGBTQ+ rights, albeit with less societal resonance.
…
Conclusion
The Simpsons evolved from a radical satire to a television institution, navigating shifts in technology, politics, and audience expectations. While its golden-age brilliance remains unmatched, its adaptability—through streaming, updated humor, and global outreach—secures its place as a cultural touchstone. The show’s longevity reflects both nostalgia and a pragmatic embrace of change, even as it grapples with the challenges of relevance in a fragmented media landscape.
```


Find [the full report here](https://drive.google.com/file/d/1GE3rvxFFTKqro67ctTkknryUf-ojhduN/view?usp=sharing), and [a report produced by DeepSearcher with GPT-4o mini](https://drive.google.com/file/d/1EGd16sJDNFnssk9yTd5o9jzbizrY_NS_/view?usp=sharing) for comparison.


## Discussion

We presented [DeepSearcher](https://github.com/zilliztech/deep-searcher), an agent for performing research and writing reports. Our system is built upon the idea in our previous article, adding features like conditional execution flow, query routing, and an improved interface. We switched from local inference with a small 4-bit quantized reasoning model to an online inference service for the massive DeepSeek-R1 model, qualitatively improving our output report. DeepSearcher works with most inference services like OpenAI, Gemini, DeepSeek and Grok 3 (coming soon!).

Reasoning models, especially as used in research agents, are inference-heavy, and we were fortunate to be able to use the fastest offering of DeepSeek-R1 from SambaNova running on their custom hardware. For our demonstration query, we made sixty-five calls to SambaNova’s DeepSeek-R1 inference service, inputting around 25k tokens, outputting 22k tokens, and costing $0.30. We were impressed with the speed of inference given that the model contains 671-billion parameters and is 3/4 of a terabyte large. [Find out more details here!](https://sambanova.ai/press/fastest-deepseek-r1-671b-with-highest-efficiency)

We will continue to iterate on this work in future posts, examining additional agentic concepts and the design space of research agents. In the meanwhile, we invite everyone to try out [DeepSearcher](https://github.com/zilliztech/deep-searcher), [star us on GitHub](https://github.com/zilliztech/deep-searcher), and share your feedback!


## Resources

- [**Zilliz’s DeepSearcher**](https://github.com/zilliztech/deep-searcher)

- Background reading: [**_“I Built a Deep Research with Open Source—and So Can You!”_**](https://milvus.io/blog/i-built-a-deep-research-with-open-source-so-can-you.md)

- _“_[__SambaNova Launches the Fastest DeepSeek-R1 671B with the Highest Efficiency__](https://sambanova.ai/press/fastest-deepseek-r1-671b-with-highest-efficiency)_”_

- DeepSearcher: [DeepSeek-R1 report on The Simpsons](https://drive.google.com/file/d/1GE3rvxFFTKqro67ctTkknryUf-ojhduN/view?usp=sharing)

- DeepSearcher: [GPT-4o mini report on The Simpsons](https://drive.google.com/file/d/1EGd16sJDNFnssk9yTd5o9jzbizrY_NS_/view?usp=sharing)

- [Milvus Open-Source Vector Database](https://milvus.io/docs)


Introducing DeepSearcher: A Local Open Source Deep Research


## Overview

As Milvus flourished in 2024 with major releases and a thriving open-source ecosystem, a hidden treasure trove of user insights was quietly forming in our community on [Discord](https://discord.gg/xwqmFDURcz). This compilation of community discussions presented a unique opportunity to understand our users' challenges firsthand. Intrigued by this untapped resource, I embarked on a comprehensive analysis of every discussion thread from the year, searching for patterns that could help us compile a frequently asked questions resource for Milvus users.

![](https://assets.zilliz.com/top_image_6bbdbe8caa.png)

My analysis revealed three primary areas where users consistently sought guidance: **Performance Optimization**, **Deployment Strategies**, and **Data Management**. Users frequently discussed how to fine-tune Milvus for production environments and track performance metrics effectively. When it came to deployment, the community grappled with selecting appropriate deployments, choosing optimal search indices, and resolving issues in distributed setups. The data management conversations centered around service-to-service data migration strategies and the selection of embedding models.

Let’s examine each of these areas in more detail.

## Deployment

![](https://assets.zilliz.com/deployment_c951c46339.png)

Milvus provides flexible deployment modes to fit various use cases. However, some users do find it challenging to find the right choice, and want to feel comfortable that they are doing so “correctly.”


### Which deployment type should I choose?

A very frequent question is which deployment to choose out of Milvus [Lite](https://milvus.io/docs/milvus_lite.md), [Standalone](https://milvus.io/docs/prerequisite-docker.md), and [Distributed](https://milvus.io/docs/prerequisite-helm.md). The answer primarily depends on how large your vector database needs to be and how much traffic it will serve: 


#### Milvus Lite

When prototyping on your local system with up to a few million vectors, or looking for an embedded vector db for unit testing and CI/CD, you can use Milvus Lite. Note that some more advanced features like full-text search are not yet available within Milvus Lite but coming soon.


#### Milvus Standalone

If your system needs to serve production traffic and / or you need to store between a few million and a hundred-million vectors, you should use Milvus Standalone, which packs all components of Milvus into a single Docker image. There is a variation which just takes its persistent storage (minio) and metadata store (etcd) dependencies out as separate images. 


#### Milvus Distributed

For any larger scale deployments serving production traffic, like serving billions of vectors at thousands of QPS, you should use Milvus Distributed. Some users may want to perform offline batch processing at scale, for example, for data deduplication or record linkage, and the future version of Milvus 3.0 will provide a more efficient way of doing this by what we term a vector lake.


#### Fully Managed Service

For developers who want to focus on the application development without worrying about DevOps, [Zilliz Cloud](https://cloud.zilliz.com/signup) is the fully managed Milvus that offers a free tier.

See [“Overview of Milvus Deployments”](https://milvus.io/docs/install-overview.md#Choose-the-Right-Deployment-for-Your-Use-Case) for more information.


### How much memory, storage, and compute will I require?

This question comes up a lot, not only for existing Milvus users but also those who are considering whether Milvus is appropriate for their application. The exact combination of how much memory, storage, and compute a deployment will require depends on complex interaction of factors.

Vector embeddings differ in dimensionality due to the model that is used. And some vector search indexes are stored entirely in memory, whereas others store data to disk. Also, many search indexes are able to store a compressed (quantized) copy of the embeddings and require additional memory for graph data structures. These are just a few factors that affect the memory and storage.


#### Milvus Resource Sizing Tool

Luckily, Zilliz (the team that maintains Milvus) has built [a resource sizing tool](https://milvus.io/tools/sizing) that does a fantastic job of answering this question. Input your vector dimensionality, index type, deployment options, and so on and the tool estimates CPU, memory, and storage needed across the various types of Milvus nodes and its dependencies. Your mileage may vary so a real load testing with your data and sample traffic is always a good idea. 


### Which vector index or distance metric should I choose?

Many users are uncertain which index they should choose and how to set the hyperparameters. First, it is always possible to defer the choice of index type to Milvus by selecting AUTOINDEX. If you wish to select a specific index type, however, a few rules of thumb provide a starting point.


#### In-Memory Indexes

Would you like to pay the cost to fit your index entirely into memory? An in-memory index is typically the fastest but also expensive. See [“In-memory indexes”](https://milvus.io/docs/index.md?tab=floating) for a list of the ones supported by Milvus and the tradeoffs they make in terms of latency, memory, and recall.

Keep in mind that your index size is not simply the number of vectors times their dimensionality and floating point size. Most indexes quantize the vectors to reduce memory usage, but require memory for additional data structures. Other non-vector data (scalar) and their index also takes up memory space.


#### On-Disk Indexes

When your index does not fit in memory, you can use one of the [“On-disk indexes”](https://milvus.io/docs/disk_index.md) provided by Milvus. Two choices with very different latency/resource tradeoffs are [DiskANN](https://milvus.io/docs/disk_index.md) and [MMap](https://milvus.io/docs/mmap.md#MMap-enabled-Data-Storage). 

DiskANN stores a highly compressed copy of the vectors in memory, and the uncompressed vectors and graph search structures on disk. It uses some clever ideas to search the vector space while minimizing disk reads and takes advantage of the fast random access speed of SSDs. For minimum latency, the SSD must be connected via NVMe rather than SATA for best I/O performance. 

Technically speaking, MMap is not an index type, but refers to the use of virtual memory with an in-memory index. With virtual memory, pages can be swapped between disk and RAM as required, which allows a much larger index to be used efficiently if the access patterns are such that only a small portion of the data is used at a time.

DiskANN has excellent and consistent latency. MMap has even better latency when it is accessing a page in-memory, but frequent page-swapping will cause latency spikes. Thus MMap can have a higher variability in latency, depending on the memory access patterns.


#### GPU Indexes

A third option is to construct [an index using GPU memory and compute](https://milvus.io/docs/gpu_index.md). Milvus’ GPU support is contributed by the Nvidia [RAPIDS](https://rapids.ai/) team. GPU vector search may have lower latency than a corresponding CPU search, although it usually takes hundreds or thousands of search QPS to fully exploit the parallelism of GPU. Also, GPUs typically have less memory than the CPU RAM and are more costly to run.


#### Distance Metrics

An easier question to answer is which distance metric should you choose to measure similarity between vectors. It is recommended to choose the same distance metric that your embedding model was trained with, which is typically COSINE (or IP when inputs are normalized). The source of your model (e.g. the model page on HuggingFace) will provide clarification on which distance metric was used. Zilliz also put together a convenient [table](https://zilliz.com/ai-models) to look that up.

To summarize, I think a lot of the uncertainty around index choice revolves around uncertainty about how these choices affect the latency/resource usage/recall tradeoff of your deployment. I recommend using the rules of thumb above to decide between in-memory, on-disk, or GPU indexes, and then using the tradeoff guidelines given in the Milvus documentation to pick a particular one.


### Can you fix my broken Milvus Distributed deployment?

Many questions revolve around issues getting a Milvus Distributed deployment up and running, with questions relating to configuration, tooling, and debugging logs. It’s hard to give a single fix as each question seems different from the last, although luckily Milvus has [a vibrant Discord](https://milvus.io/discord) where you can seek help, and we also offer [1-on-1 office hours with an expert](https://milvus.io/blog/join-milvus-office-hours-to-get-support-from-vectordb-experts.md).


### How do I deploy Milvus on Windows?

A question that has come up several times is how to deploy Milvus on Windows machines. Based on your feedback, we have rewritten the documentation for this: see [Run Milvus in Docker (Windows)](https://milvus.io/docs/install_standalone-windows.md) for how to do this, using [Windows Subsystem for Linux 2 (WSL2)](https://learn.microsoft.com/en-us/windows/wsl/about#what-is-wsl-2).


## Performance and Profiling

![](https://assets.zilliz.com/Performance_and_Profiling_481975ea1d.png)

Having chosen a deployment type and got it running, users want to feel comfortable that they have made optimal decisions and would like to profile their deployment’s performance and state. Many questions related to how to profile performance, observe state, and get an insight into what and why. 


### How do I measure performance?

Users want to check metrics related to the performance of their deployment so they can understand and remedy bottlenecks. Metrics mentioned include average query latency, distribution of latencies, query volume, memory usage, disk storage, and so on. These metrics can be observed from the [monitoring system](https://milvus.io/docs/monitor_overview.md). In addition, Milvus 2.5 introduces a new tool called [WebUI](https://milvus.io/docs/milvus-webui.md#Milvus-WebUI) (feedback welcome!), which allows you to access more system internal information like segment compaction status, from a user-friendly web interface.


### What’s happening inside Milvus right now (i.e. observe state)?

Relatedly, users want to observe the internal state of their deployment. Issues raised include understanding why a search index is taking so long to build, how to determine if the cluster is healthy, and understanding how a query is executed across nodes. Many of these questions can be answered with the new [WebUI](https://milvus.io/docs/milvus-webui.md#Milvus-WebUI) that gives transparency to what the system is doing internally.


### How does some (complex) aspect of the internals work?

Advanced users often want some understanding of Milvus internals, for example, having an understanding of the sealing of segments or memory management. The underlying goal is typically to improve performance and sometimes to debug issues. The documentation, particularly under the sections “Concepts” and “Administration Guide" is helpful here, for instance see the pages [“Milvus Architecture Overview”](https://milvus.io/docs/architecture_overview.md) and [“Clustering Compaction”](https://milvus.io/docs/clustering-compaction.md). We will continue to improve the documentation on Milvus internals, make it easier to understand, and welcome any feedback or requests via [Discord](https://milvus.io/discord).


### Which embedding model should I choose?

A question related to performance that has come up multiple times in meetups, office hours, and on Discord is how to choose an embedding model. This is a difficult question to give a definitive answer although we recommend starting with default models like [all-MiniLM-L6-v2](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2). 

Similar to the choice of search index, there are tradeoffs between compute, storage, and recall. An embedding model with larger output dimension will require more storage, all else held equal, although probably result in higher recall of relevant items. Larger embedding models, for a fixed dimension, typically outperform smaller ones in terms of recall, although at the cost of increased compute and time. Leaderboards that rank embedding model performance such as [MTEB](https://huggingface.co/spaces/mteb/leaderboard) are based on benchmarks that may not align with your specific data and task.

So, it does not make sense to think of a “best” embedding model. Start with one that has acceptable recall and meets your compute and time budget for calculating embeddings. Further optimizations like fine-tuning on your data or exploring the compute/recall tradeoff empirically can be deferred to after you have a working system in production.


## Data Management

![](https://assets.zilliz.com/Data_Management_aa2d1159bd.png)

How to move data into and out of a Milvus deployment is another main theme in the Discord discussions, which is no surprise given how central this task is to putting an application into production.


### How do I migrate data from X to Milvus? How do I migrate data from Standalone to Distributed? How do I migrate from 2.4.x to 2.5.x?

A new user commonly wants to get existing data into Milvus from another platform, including traditional search engines like [Elasticsearch](https://docs.zilliz.com/docs/migrate-from-elasticsearch) and other vector databases like [Pinecone](https://docs.zilliz.com/docs/migrate-from-pinecone) or [Qdrant](https://docs.zilliz.com/docs/migrate-from-qdrant). Existing users may also want to migrate their data from one Milvus deployment to another, or [from self-hosted Milvus to fully managed Zilliz Cloud](https://docs.zilliz.com/docs/migrate-from-milvus).

The [Vector Transport Service (VTS)](https://github.com/zilliztech/vts) and the managed [Migration](https://docs.zilliz.com/docs/migrations) service on Zilliz Cloud are designed for this purpose.


### How do I save and load data backups? How do I export data from Milvus?

Milvus has a dedicated tool, [milvus-backup](https://github.com/zilliztech/milvus-backup), to take snapshots on permanent storage and restore them.


## Next Steps

I hope this has given you some pointers on how to tackle common challenges faced when building with a vector database. This definitely helped us to take another look at our documentation and feature roadmap to keep working on things that can help our community best succeed with Milvus. A key takeaway that I would like to emphasize, is that your choices put you within different points of a tradeoff space between compute, storage, latency, and recall. _You cannot maximize all of these performance criteria simultaneously - there is no “optimal” deployment. Yet, by understanding more about how vector search and distributed database systems work you can make an informed decision._

After trawling through the large number of posts from 2024, it got me thinking: why should a human do this? Has not Generative AI promised to solve such a task of crunching large amounts of text and extracting insight? Join me in the second part of this blog post (coming soon), where I investigate the design and implementation of _a multi-agent system for extracting insight from discussion forums._

Thanks again and hope to see you in the community [Discord](https://milvus.io/discord) and our next [Unstructured Data](https://lu.ma/unstructured-data-meetup) meetups. For more hands-on assistance, we welcome you to book a [1-on-1 office hour](https://milvus.io/blog/join-milvus-office-hours-to-get-support-from-vectordb-experts.md). _Your feedback is essential to improving Milvus!_


What Milvus Users Taught Us in 2024


![](https://assets.zilliz.com/deep_research_blog_image_95225226eb.png)

Well actually, a minimally scoped agent that can reason, plan, use tools, etc. to perform research using Wikipedia. Still, not bad for a few hours of work…

Unless you reside under a rock, in a cave, or in a remote mountain monastery, you will have heard about OpenAI’s release of _Deep Research_ on Feb 2, 2025. This new product promises to revolutionize how we answer questions requiring the synthesis of large amounts of diverse information.

You type in your query, select the Deep Research option, and the platform autonomously searches the web, performs reasoning on what it discovers, and synthesizes multiple sources into a coherent, fully-cited report. It takes several orders of magnitude longer to produce its output relative to a standard chatbot, but the result is more detailed, more informed, and more nuanced.


## How does it work?

But how does this technology work, and why is Deep Research a noticeable improvement over previous attempts (like Google’s _Deep Research_ - incoming trademark dispute alert)? We’ll leave the latter for a future post. As for the former, there is no doubt much “secret sauce” underlying Deep Research. We can glean a few details from OpenAI’s release post, which I summarize.

**Deep Research exploits recent advances in foundation models specialized for reasoning tasks:**

- “...fine-tuned on the upcoming OpenAI o3 reasoning model…”

- “...leverages reasoning to search, interpret, and analyze massive amounts of text…”

**Deep Research makes use of a sophisticated agentic workflow with planning, reflection, and memory:**

- “...learned to plan and execute a multi-step trajectory…”

- “...backtracking and reacting to real-time information…”

- “...pivoting as needed in reaction to information it encounters…”

**Deep Research is trained on proprietary data, using several types of fine-tuning, which is likely a key component in its performance:**

- “...trained using end-to-end reinforcement learning on hard browsing and reasoning tasks across a range of domains…”

- “...optimized for web browsing and data analysis…”

The exact design of the agentic workflow is a secret, however, we can build something ourselves based on well-established ideas about how to structure agents.

**One note before we begin**: It is easy to be swept away by Generative AI fever, especially when a new product that seems a step-improvement is released. However, Deep Research, as OpenAI acknowledges, has limitations common to Generative AI technology. We should remember to think critically about the output in that it may contain false facts (“hallucinations”), incorrect formatting and citations, and vary significantly in quality based on the random seed.


## Can I build my own?

Why certainly! Let’s build our own “Deep Research”, running locally and with open-source tools. We’ll be armed with just a basic knowledge of Generative AI, common sense, a couple of spare hours, a GPU, and the open-source [Milvus](https://milvus.io/docs), [DeepSeek R1](https://huggingface.co/unsloth/DeepSeek-R1-Distill-Llama-8B-unsloth-bnb-4bit), and [LangChain](https://python.langchain.com/docs/introduction/).

We cannot hope to replicate OpenAI’s performance of course, but our prototype will minimally demonstrate some of the key ideas likely underlying their technology, combining advances in reasoning models with advances in agentic workflows. Importantly, and unlike OpenAI, we will be using only open-source tools, and be able to deploy our system locally - open-source certainly provides us great flexibility!

We will make a few simplifying assumptions to reduce the scope of our project:

- We will use an open-source reasoning mode distilled then [quantized](https://zilliz.com/learn/unlock-power-of-vector-quantization-techniques-for-efficient-data-compression-and-retrieval) for 4-bits that can be run locally.

- We will not perform additional fine-tuning on our reasoning model ourselves.

- The only tool our agent has is the ability to download and read a Wikipedia page and perform separate RAG queries (we will not have access to the entire web).

- Our agent will only process text data, not images, PDFs, etc.

- Our agent will not backtrack or consider pivots.

- Our agent will (not yet) control its execution flow based on its output.

- Wikipedia contains the truth, the whole truth and nothing but the truth.

We will use [Milvus](https://milvus.io/docs) for our vector database, [DeepSeek R1](https://huggingface.co/unsloth/DeepSeek-R1-Distill-Llama-8B-unsloth-bnb-4bit) as our reasoning model, and [LangChain](https://python.langchain.com/docs/introduction/) to implement RAG. Let’s get started! 


# A Minimal Agent for Online Research

We will use our mental model of how humans conduct research to design the agentic workflow:


### Define/Refine Question

Research starts by defining a question. We take the question to be the user’s query, however, we use our reasoning model to ensure the question is expressed in a way that is specific, clear, and focused. That is, our first step is to rewrite the prompt and extract any subqueries or subquestions. We make effective use of our foundation models specialization for reasoning, and a simple method for JSON structured output.

Here is an example reasoning trace as DeepSeek refines the question “How has the cast changed over time?”:

```text
<think>

Alright, so I need to break down the question "How has the cast changed over time?" related to the evolution of The Simpsons. Let me think about what aspects are involved here.

First, I know that the cast has changed a lot, but I need to figure out the intermediate questions to approach this. The main question is about changes in the cast over time, so I should consider different areas that contribute to this change.

I guess the first sub-question would be about the original cast members. Who were the main voices and how did they evolve? Then, there might be new cast additions over the years, so another sub-question about that.

Also, some original voice actors have left, so I should include a sub-question about departures. Then, new voice actors joining would be another point.

The show has been popular for a long time, so recurring roles changing might be another aspect. Additionally, the role of the show in society might have influenced casting choices, so a sub-question about that.

Lastly, the overall impact on the cast's careers could be another angle. So, I should list these as sub-questions to cover all aspects.

</think>
```

### Search

Next, we conduct a “literature review” of Wikipedia articles. For now, we read a single article and leave navigating links to a future iteration. We discovered during prototyping that link exploration can become very expensive if each link requires a call to the reasoning model. We parse the article, and store its data in our vector database, Milvus, akin to taking notes.

Here is a code snippet showing how we store our Wikipedia page in Milvus using its LangChain integration:

```python
wiki_wiki = wikipediaapi.Wikipedia(user_agent='MilvusDeepResearchBot (<insert your email>)', language='en')
page_py = wiki_wiki.page(page_title)

text_splitter = RecursiveCharacterTextSplitter(chunk_size=2000, chunk_overlap=200)
docs = text_splitter.create_documents([page_py.text])

vectorstore = Milvus.from_documents(  # or Zilliz.from_documents
    documents=docs,
    embedding=embeddings,
    connection_args={
        "uri": "./milvus_demo.db",
    },
    drop_old=True, 
    index_params={
        "metric_type": "COSINE",
        "index_type": "FLAT",  
        "params": {},
    },
)
```

### Analyze

The agent returns to its questions and answers them based on the relevant information in the document. We will leave a multi-step analysis/reflection workflow for future work, as well as any critical thinking on the credibility and bias of our sources.

Here is a code snippet illustrating constructing a RAG with LangChain and answering our subquestions separately.

```python
# Define the RAG chain for response generation
rag_chain = (
    {"context": retriever | format_docs, "question": RunnablePassthrough()}
    | prompt
    | llm
    | StrOutputParser()
)

# Prompt the RAG for each question
answers = {}
total = len(leaves(breakdown))

pbar = tqdm(total=total)
for k, v in breakdown.items():
    if v == []:
        print(k)
        answers[k] = rag_chain.invoke(k).split('</think>')[-1].strip()
        pbar.update(1)
    else:
        for q in v:
            print(q)
            answers[q] = rag_chain.invoke(q).split('</think>')[-1].strip()
            pbar.update(1)
```

### Synthesize

After the agent has performed its research, it creates a structured outline, or rather, a skeleton, of its findings to summarize in a report. It then completes each section, filling it in with a section title and the corresponding content. We leave a more sophisticated workflow with reflection, reordering, and rewriting for a future iteration. This part of the agent involves planning, tool usage, and memory.

See [accompanying notebook](https://drive.google.com/file/d/1waKX_NTgiY-47bYE0cI6qD8Cjn3zjrL6/view?usp=sharing) for the full code and the [saved report file](https://drive.google.com/file/d/15xeEe_EqY-29V2IlAvDy5yGdJdEPSHOh/view?usp=drive_link) for example output.


## Results

Our query for testing is _“How has The Simpsons changed over time?”_ and the data source is the Wikipedia article for “The Simpsons”. Here is one section of the [generated report](https://drive.google.com/file/d/15xeEe_EqY-29V2IlAvDy5yGdJdEPSHOh/view?usp=sharing):

![](https://assets.zilliz.com/result_query_424beba224.jpg)


## Summary: What we built and what’s next

In just a few hours, we have designed a basic agentic workflow that can reason, plan, and retrieve information from Wikipedia to generate a structured research report. While this prototype is far from OpenAI’s Deep Research, it demonstrates the power of open-source tools like Milvus, DeepSeek, and LangChain in building autonomous research agents. 

Of course, there’s plenty of room for improvement. Future iterations could:

- Expand beyond Wikipedia to search multiple sources dynamically

- Introduce backtracking and reflection to refine responses

- Optimize execution flow based on the agent's own reasoning

Open-source gives us flexibility and control that closed source doesn’t. Whether for academic research, content synthesis, or AI-powered assistance, building our own research agents open up exciting possibilities. Stay tuned for the next post where we explore adding real-time web retrieval, multi-step reasoning, and conditional execution flow!


## Resources

- Notebook: _“_[_Baseline for An Open-Source Deep Research_](https://colab.research.google.com/drive/1W5tW8SqWXve7ZwbSb9pVdbt5R2wq105O?usp=sharing)_”_

- Report: _“_[_The evolution of The Simpsons as a show over time, covering changes in content, humor, character development, animation, and its role in society._](https://drive.google.com/file/d/15xeEe_EqY-29V2IlAvDy5yGdJdEPSHOh/view?usp=drive_link)_”_

- [Milvus vector database documentation](https://milvus.io/docs)

- [Distilled and quantized DeepSeek R1 model page](https://huggingface.co/unsloth/DeepSeek-R1-Distill-Llama-8B-unsloth-bnb-4bit)

- [️🔗 LangChain](https://python.langchain.com/docs/introduction/)

- [Deep Research FAQ | OpenAI Help Center](https://help.openai.com/en/articles/10500283-deep-research-faq)





I Built a Deep Research with Open Source—and So Can You!

Multimodal Semantic Search with Images and Text

How to Get Started with Milvus



By now, everyone knows that hybrid search has improved [RAG](https://zilliz.com/learn/Retrieval-Augmented-Generation) (Retrieval-Augmented Generation) search quality. While [dense embedding](https://zilliz.com/learn/sparse-and-dense-embeddings) search has shown impressive capabilities in capturing deep semantic relationships between queries and documents, it still has notable limitations. These include a lack of explainability and suboptimal performance with long-tail queries and rare terms.

Many RAG applications struggle because pre-trained models often lack domain-specific knowledge. In some scenarios, simple BM25 keyword matching outperforms these sophisticated models. This is where hybrid search bridges the gap, combining the semantic understanding of dense vector retrieval with the precision of keyword matching.


## Why Hybrid Search is Complex in Production

While frameworks like [LangChain](https://zilliz.com/learn/LangChain) or [LlamaIndex](https://zilliz.com/learn/getting-started-with-llamaindex) make building a proof-of-concept hybrid retriever easy, scaling to production with massive datasets is challenging. Traditional architectures require separate vector databases and search engines, leading to several key challenges:

- High infrastructure maintenance costs and operational complexity

- Data redundancy across multiple systems

- Difficult data consistency management

- Complex security and access control across systems

The market needs a unified solution that supports lexical and semantic search while reducing system complexity and cost.

![](https://assets.zilliz.com/elasticsearch_vs_milvus_5be6e2b69e.png)


## The Pain Points of Elasticsearch

Elasticsearch has been one of the past decade's most influential open-source search projects. Built on Apache Lucene, it gained popularity through its high performance, scalability, and distributed architecture. While it added vector ANN search in version 8.0, production deployments face several critical challenges:

**High Update and Indexing Costs:** Elasticsearch's architecture doesn't fully decouple write operations, index building, and querying. This leads to significant CPU and I/O overhead during write operations, especially in bulk updates. The resource contention between indexing and querying impacts performance, creating a major bottleneck for high-frequency update scenarios.

**Poor Real-time Performance:** As a "near real-time" search engine, Elasticsearch introduces noticeable latency in data visibility. This latency becomes particularly problematic for AI applications, such as Agent systems, where high-frequency interactions and dynamic decision-making require immediate data access.

**Difficult Shard Management:** While Elasticsearch uses sharding for distributed architecture, shard management poses significant challenges. The lack of dynamic sharding support creates a dilemma: too many shards in small datasets lead to poor performance, while too few shards in large datasets limit scalability and cause uneven data distribution.

**Non-Cloud-Native Architecture:** Developed before cloud-native architectures became prevalent, Elasticsearch's design tightly couples storage and compute, limiting its integration with modern infrastructure like public clouds and Kubernetes. Resource scaling requires simultaneous increases in both storage and compute, reducing flexibility. In multi-replica scenarios, each shard must build its index independently, increasing computational costs and reducing resource efficiency.

**Poor Vector Search Performance:** Though Elasticsearch 8.0 introduced vector ANN search, its performance significantly lags behind that of dedicated vector engines like Milvus. Based on the Lucene kernel, its index structure proves inefficient for high-dimensional data, struggling with large-scale vector search requirements. Performance becomes particularly unstable in complex scenarios involving scalar filtering and multi-tenancy, making it challenging to support high-load or diverse business needs.

**Excessive Resource Consumption:** Elasticsearch places extreme demands on memory and CPU, especially when processing large-scale data. Its JVM dependency requires frequent heap size adjustments and garbage collection tuning, severely impacting memory efficiency. Vector search operations require intensive SIMD-optimized computations, for which the JVM environment is far from ideal.

These fundamental limitations become increasingly problematic as organizations scale their AI infrastructure, making Elasticsearch particularly challenging for modern AI applications requiring high performance and reliability.


## Introducing Sparse-BM25: Reimagining Lexical Search

[Milvus 2.5](https://milvus.io/blog/introduce-milvus-2-5-full-text-search-powerful-metadata-filtering-and-more.md) introduces native lexical search support through Sparse-BM25, building upon the hybrid search capabilities introduced in version 2.4. This innovative approach includes the following key components:

- Advanced tokenization and preprocessing via Tantivy

- Distributed vocabulary and term frequency management

- Sparse vector generation using corpus TF and query TF-IDF

- Inverted index support with WAND algorithm (Block-Max WAND and graph index support in development)

Compared to Elasticsearch, Milvus offers significant advantages in algorithm flexibility. Its vector distance-based similarity computation enables more sophisticated matching, including implementing TW-BERT (Term Weighting BERT) based on "End-to-End Query Term Weighting" research. This approach has demonstrated superior performance in both in-domain and out-domain testing.

Another crucial advantage is cost efficiency. By leveraging both inverted index and dense embedding compression, Milvus achieves a fivefold performance improvement with less than 1% recall degradation. Through tail-term pruning and vector quantization, memory usage has been reduced by over 50%.

Long query optimization stands out as a particular strength. Where traditional WAND algorithms struggle with longer queries, Milvus excels by combining sparse embeddings with graph indices, delivering a tenfold performance improvement in high-dimensional sparse vector search scenarios.

![](https://assets.zilliz.com/document_in_and_out_b84771bec4.png)


## Milvus: The Ultimate Vector Database for RAG

Milvus is the premier choice for RAG applications through its comprehensive feature set. Key advantages include:

- Rich metadata support with dynamic schema capabilities and powerful filtering options

- Enterprise-grade multi-tenancy with flexible isolation through collections, partitions, and partition keys

- Industry-first disk vector index support with multi-tier storage from memory to S3

- Cloud-native scalability supporting seamless scaling from 10M to 1B+ vectors

- Comprehensive search capabilities, including grouping, range, and hybrid search

- Deep ecosystem integration with LangChain, LlamaIndex, Dify, and other AI tools

The system's diverse search capabilities encompass grouping, range, and hybrid search methodologies. Deep integration with tools like LangChain, LlamaIndex, and Dify, as well as support for numerous AI products, places Milvus at the center of the modern AI infrastructure ecosystem.


## Looking Forward

As AI transitions from POC to production, Milvus continues to evolve. We focus on making vector search more accessible and cost-effective while enhancing search quality. Whether you're a startup or an enterprise, Milvus reduces the technical barriers to AI application development.

This commitment to accessibility and innovation has led us to another major step forward. While our open-source solution continues to serve as the foundation for thousands of applications worldwide, we recognize that many organizations need a fully managed solution that eliminates operational overhead.


## Zilliz Cloud: The Managed Solution

We've built [Zilliz Cloud](https://zilliz.com/cloud), a fully managed vector database service based on Milvus, over the past three years. Through a cloud-native reimplementation of the Milvus protocol, it offers enhanced usability, cost efficiency, and security.

Drawing from our experience maintaining the world's largest vector search clusters and supporting thousands of AI application developers, Zilliz Cloud significantly reduces operational overhead and costs compared to self-hosted solutions.

Ready to experience the future of vector search? Start your free trial today with up to $200 in credits, no credit card required.


Elasticsearch is Dead, Long Live Lexical Search


In this article, we will show you how to quickly get up and running with the new full-text search feature and combine it with the conventional semantic search based on vector embeddings.


<iframe width="100%" height="480" src="https://www.youtube.com/embed/3bftbAjQF7Q" title="Beyond Keywords: Hybrid Search with Milvus 2.5" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe>


## Requirement

First, ensure you have installed Milvus 2.5:


```
pip install -U pymilvus[model]
```

and have a running instance of Milvus Standalone (e.g. on your local machine) using the [installation instructions in the Milvus docs](https://milvus.io/docs/prerequisite-docker.md).

## Building the Data Schema and Search Indices

We import the required classes and functions:

```
from pymilvus import MilvusClient, DataType, Function, FunctionType, model
```

You may have noticed two new entries for Milvus 2.5, `Function` and `FunctionType`, which we will explain shortly.

Next we open the database with Milvus Standalone, that is, locally, and create the data schema. The schema comprises an integer primary key, a text string, a dense vector of dimension 384, and a sparse vector (of unlimited dimensionality).
Note that Milvus Lite does not currently support full-text search, only Milvus Standalone and Milvus Distributed.

```
client = MilvusClient(uri="http://localhost:19530")

schema = client.create_schema()

schema.add_field(field_name="id", datatype=DataType.INT64, is_primary=True, auto_id=True)
schema.add_field(field_name="text", datatype=DataType.VARCHAR, max_length=1000, enable_analyzer=True)
schema.add_field(field_name="dense", datatype=DataType.FLOAT_VECTOR, dim=768),
schema.add_field(field_name="sparse", datatype=DataType.SPARSE_FLOAT_VECTOR)
```

```
{'auto_id': False, 'description': '', 'fields': [{'name': 'id', 'description': '', 'type': <DataType.INT64: 5>, 'is_primary': True, 'auto_id': True}, {'name': 'text', 'description': '', 'type': <DataType.VARCHAR: 21>, 'params': {'max_length': 1000, 'enable_analyzer': True}}, {'name': 'dense', 'description': '', 'type': <DataType.FLOAT_VECTOR: 101>, 'params': {'dim': 768}}, {'name': 'sparse', 'description': '', 'type': <DataType.SPARSE_FLOAT_VECTOR: 104>}], 'enable_dynamic_field': False}
```

You may have noticed the `enable_analyzer=True` parameter. This tells Milvus 2.5 to enable the lexical parser on this field and build a list of tokens and token frequencies, which are required for full-text search. The `sparse` field will hold a vector representation of the documentation as a bag-of-words produced from the parsing `text`.

But how do we connect the `text` and `sparse` fields, and tell Milvus how `sparse` should be calculated from `text`? This is where we need to invoke the `Function` object and add it to the schema:


```
bm25_function = Function(
    name="text_bm25_emb", # Function name
    input_field_names=["text"], # Name of the VARCHAR field containing raw text data
    output_field_names=["sparse"], # Name of the SPARSE_FLOAT_VECTOR field reserved to store generated embeddings
    function_type=FunctionType.BM25,
)

schema.add_function(bm25_function)
```


```
{'auto_id': False, 'description': '', 'fields': [{'name': 'id', 'description': '', 'type': <DataType.INT64: 5>, 'is_primary': True, 'auto_id': True}, {'name': 'text', 'description': '', 'type': <DataType.VARCHAR: 21>, 'params': {'max_length': 1000, 'enable_analyzer': True}}, {'name': 'dense', 'description': '', 'type': <DataType.FLOAT_VECTOR: 101>, 'params': {'dim': 768}}, {'name': 'sparse', 'description': '', 'type': <DataType.SPARSE_FLOAT_VECTOR: 104>, 'is_function_output': True}], 'enable_dynamic_field': False, 'functions': [{'name': 'text_bm25_emb', 'description': '', 'type': <FunctionType.BM25: 1>, 'input_field_names': ['text'], 'output_field_names': ['sparse'], 'params': {}}]}
```

The abstraction of the `Function` object is more general than that of applying full-text search. In the future, it may be used for other cases where one field needs to be a function of another field. In our case, we specify that `sparse` is a function of `text` via the function `FunctionType.BM25`. `BM25` refers to a common metric in information retrieval used for calculating a query's similarity to a document (relative to a collection of documents).

We use the default embedding model in Milvus, which is [paraphrase-albert-small-v2](https://huggingface.co/GPTCache/paraphrase-albert-small-v2):


```
embedding_fn = model.DefaultEmbeddingFunction()
```

The next step is to add our search indices. We have one for the dense vector and a separate one for the sparse vector. The index type is `SPARSE_INVERTED_INDEX` with `BM25` since full-text search requires a different search method than those for standard dense vectors.


```
index_params = client.prepare_index_params()

index_params.add_index(
    field_name="dense",
    index_type="AUTOINDEX", 
    metric_type="COSINE"
)

index_params.add_index(
    field_name="sparse",
    index_type="SPARSE_INVERTED_INDEX", 
    metric_type="BM25"
)
```

Finally, we create our collection:

```
client.drop_collection('demo')
client.list_collections()
```

```
[]
```

```
client.create_collection(
    collection_name='demo', 
    schema=schema, 
    index_params=index_params
)

client.list_collections()
```


```
['demo']
```
And with that, we have an empty database set up to accept text documents and perform semantic and full-text searches!

## Inserting Data and Performing Full-Text Search

Inserting data is no different than previous versions of Milvus:

```
docs = [
    'information retrieval is a field of study.',
    'information retrieval focuses on finding relevant information in large datasets.',
    'data mining and information retrieval overlap in research.'
]

embeddings = embedding_fn(docs)

client.insert('demo', [
    {'text': doc, 'dense': vec} for doc, vec in zip(docs, embeddings)
])
```

```
{'insert_count': 3, 'ids': [454387371651630485, 454387371651630486, 454387371651630487], 'cost': 0}
```

Let's first illustrate a full-text search before we move on to hybrid search:

```
search_params = {
    'params': {'drop_ratio_search': 0.2},
}

results = client.search(
    collection_name='demo', 
    data=['whats the focus of information retrieval?'],
    output_fields=['text'],
    anns_field='sparse',
    limit=3,
    search_params=search_params
)
```

The search parameter `drop_ratio_search` refers to the proportion of lower-scoring documents to drop during the search algorithm.

Let's view the results:

```
for hit in results[0]:
    print(hit)
```

```
{'id': 454387371651630485, 'distance': 1.3352930545806885, 'entity': {'text': 'information retrieval is a field of study.'}}
{'id': 454387371651630486, 'distance': 0.29726022481918335, 'entity': {'text': 'information retrieval focuses on finding relevant information in large datasets.'}}
{'id': 454387371651630487, 'distance': 0.2715056240558624, 'entity': {'text': 'data mining and information retrieval overlap in research.'}}
```

## Performing Hybrid Semantic and Full-Text Search

Let's now combine what we've learned to perform a hybrid search that combines separate semantic and full-text searches with a reranker:

```
from pymilvus import AnnSearchRequest, RRFRanker
query = 'whats the focus of information retrieval?'
query_dense_vector = embedding_fn([query])

search_param_1 = {
    "data": query_dense_vector,
    "anns_field": "dense",
    "param": {
        "metric_type": "COSINE",
    },
    "limit": 3
}
request_1 = AnnSearchRequest(**search_param_1)

search_param_2 = {
    "data": [query],
    "anns_field": "sparse",
    "param": {
        "metric_type": "BM25",
        "params": {"drop_ratio_build": 0.0}
    },
    "limit": 3
}
request_2 = AnnSearchRequest(**search_param_2)

reqs = [request_1, request_2]
```

```
ranker = RRFRanker()

res = client.hybrid_search(
    collection_name="demo",
    output_fields=['text'],
    reqs=reqs,
    ranker=ranker,
    limit=3
)
for hit in res[0]:
    print(hit)
```

```
{'id': 454387371651630485, 'distance': 0.032786883413791656, 'entity': {'text': 'information retrieval is a field of study.'}}
{'id': 454387371651630486, 'distance': 0.032258063554763794, 'entity': {'text': 'information retrieval focuses on finding relevant information in large datasets.'}}
{'id': 454387371651630487, 'distance': 0.0317460335791111, 'entity': {'text': 'data mining and information retrieval overlap in research.'}}
```

As you may have noticed, this is no different than a hybrid search with two separate semantic fields (available since Milvus 2.4). The results are identical to full-text search in this simple example, but for larger databases and keyword specific searches hybrid search typically has higher recall.

## Summary

You're now equipped with all the knowledge needed to perform full-text and hybrid semantic/full-text search with Milvus 2.5. See the following articles for more discussion on how full-text search works and why it is complementary to semantic search:

- [Introducing Milvus 2.5: Full-Text Search, More Powerful Metadata Filtering, and Usability Improvements!](https://milvus.io/blog/introduce-milvus-2-5-full-text-search-powerful-metadata-filtering-and-more.md)
- [Semantic Search v.s. Full-Text Search: Which Do I Choose in Milvus 2.5?](https://milvus.io/blog/semantic-search-vs-full-text-search-which-one-should-i-choose-with-milvus-2-5.md)


Getting Started with Hybrid Semantic / Full-Text Search with Milvus 2.5



## Overview

We are thrilled to present the latest version of Milvus, 2.5, which introduces a powerful new capability: [full-text search](https://milvus.io/docs/full-text-search.md#Full-Text-Search), also known as lexical or keyword search. If you are new to search, full-text search allows you to find documents by searching for specific words or phrases within them, similar to how you search in Google. This complements our existing semantic search capabilities, which understand the meaning behind your search rather than just matching exact words.

We use the industry-standard BM25 metric for document similarity, and our implemention is based on sparse vectors, allowing for more efficient storage and retrieval. For those unfamiliar with the term, sparse vectors are a way to represent text where most values are zero, making them very efficient to store and process—imagine a huge spreadsheet where only a few cells contain numbers, and the rest are empty. This approach fits well into Milvus's product philosophy where the vector is the core search entity.

An additional noteworthy aspect of our implementation is the capability to insert and query text _directly_ rather than having users first manually convert text into sparse vectors. This takes Milvus one step closer towards fully processing unstructured data.

But this is just the beginning. With the release of 2.5, we updated the [Milvus product roadmap](https://milvus.io/docs/roadmap.md). In future product iterations of Milvus, our focus will be on evolving Milvus's capabilities in four key directions:

- Streamlined unstructured data processing;
- Better search quality and efficiency;
- Easier data management;
- Lowering costs through algorithmic and design advances

Our aim is to build data infrastructure that can both efficiently store and effectively retrieve information in the AI era.


## Full-text Search via Sparse-BM25

Although semantic search typically has better contexual awareness and intent understanding, when a user needs to search for specific proper nouns, serial numbers, or a completely-matching phrase, full-text retrieval with keyword matching often produces more accurate results.

To illustrate this with an example:

- Semantic search excels when you ask: "Find documents about renewable energy solutions"
- Full-text search is better when you need: "Find documents mentioning _Tesla Model 3 2024_"

In our previous version (Milvus 2.4), users had to pre-process their text using a separate tool (the BM25EmbeddingFunction module to the PyMilvus) on their own machines before they could search it This approach had several limitations: it couldn't handle growing datasets well, required extra setup steps, and made the whole process more complicated than necessary. For the technically minded, the key limitations were that it could only work on a single machine; the vocabulary and other corpus statistics used for BM25 scoring couldn't be updated as the corpus changed; and converting text to vectors on the client side is less intuitive working withtext directly.

Milvus 2.5 simplifies everything. Now you can work with your text directly:

- Store your original text documents as they are
- Search using natural language queries
- Get results back in readable form

Behind the scenes, Milvus handles all the complex vector conversions automatically making it easier to work with text data. This is what we call our "Doc in, Doc out" approach—you work with readable text, and we handle the rest.


### Techical Implementation

For those interested in the technical details, Milvus 2.5 adds the full-text search capability through its built-in Sparse-BM25 implementation, including:

- **A Tokenizer built on tantivy**: Milvus now integrates with the thriving tantivy ecosystem
- **Capability to ingest and retrieve raw documents**: Support for direct ingestion and query of text data
- **BM25 relevance scoring**: Internalize BM25 scoring, implemented based on sparse vector

We chose to work with the well-developed tantivy ecosystem and build the Milvus text tokenizer on tantivy. In the future, Milvus will support more tokenizers and expose the tokenization process to help users better understand the retrieval quality. We will also explore deep learning-based tokenizers and stemmer strategies to further optimize the performance of full-text search. Below is sample code for using and configuring the tokenizer:

```Python
# Tokenizer configuration
schema.add_field(
    field_name='text',
    datatype=DataType.VARCHAR,
    max_length=65535,
    enable_analyzer=True, # Enable tokenizer on this column
    analyzer_params={"type": "english"},  # Configure tokenizer parameters, here we choose the english template, fine-grained configuration is also supported
    enable_match=True, # Build an inverted index for Text_Match
)
```

After configuring the tokenizer in the collection schema, users can register the text to bm25 function via add_function method. This will run internally in the Milvus server. All subsequent data flows such as additions, deletions, modifications, and queries can be completed by operating on the raw text string, as opposed to the vector representation. See below code example for how to ingest text and conduct full-text search with the new API:

```Python
# Define the mapping relationship between raw text data and vectors on the schema
bm25_function = Function(
    name="text_bm25_emb",
    input_field_names=["text"], # Input text field
    output_field_names=["sparse"], # Internal mapping sparse vector field
    function_type=FunctionType.BM25, # Model for processing mapping relationship
)

schema.add_function(bm25_function)
...
# Support for raw text in/out
MilvusClient.insert('demo', [
    {'text': 'Artificial intelligence was founded as an academic discipline in 1956.'},
    {'text': 'Alan Turing was the first person to conduct substantial research in AI.'},
    {'text': 'Born in Maida Vale, London, Turing was raised in southern England.'},
])

MilvusClient.search(
    collection_name='demo',
    data=['Who started AI research?'],
    anns_field='sparse',
    limit=3
)
```

We have adopted an implementation of BM25 relevance scoring that represents queries and documents as sparse vectors, called **Sparse-BM25**. This unlocks many optimizations based on sparse vector, such as:

Milvus achieves hybrid search capabilities through its cutting-edge **Sparse-BM25 implementation**, which integrates full-text search into the vector database architecture. By representing term frequencies as sparse vectors instead of traditional inverted indexes, Sparse-BM25 enables advanced optimizations, such as **graph indexing**, **product quantization (PQ)**, and **scalar quantization (SQ)**. These optimizations minimize memory usage and accelerate search performance. Similar to inverted index approach, Milvus supports taking raw text as input and generating sparse vectors internally. This make it able to work with any tokenizer and grasp any word shown in the dynamicly changing corpus.

Additionally, heuristic-based pruning discards low-value sparse vectors, further enhancing efficiency without compromising accuracy. Unlike previous approach using sparse vector, it can adapt to a growing corpus, not the accuracy of BM25 scoring.

1. Building graph indexes on the sparse vector, which performs better than inverted index on queries with long text as inverted index needs more steps to finish matching the tokens in the query;
2. Leveraging approximation techniques to speed up search with only minor impact on retrieval quality, such as vector quantization and heuristic based pruning;
3. Unifying the interface and data model for performing semantic search and full-text search, thus enhancing the user experience.

```Python
# Creating an index on the sparse column
index_params.add_index(
    field_name="sparse",
    index_type="AUTOINDEX",  # Default WAND index
    metric_type="BM25" # Configure relevance scoring through metric_type
)

# Configurable parameters at search time to speed up search
search_params = {
    'params': {'drop_ratio_search': 0.6}, # WAND search parameter configuration can speed up search
}
```

In summary, Milvus 2.5 has expanded its search capability beyond semantic search by introducing full-text retrieval, making it easier for users to build high quality AI applications. These are just initial steps in the space of Sparse-BM25 search and we anticipate that there will be further optimization measures to try in the future.


## Text Matching Search Filters

A second text search feature released with Milvus 2.5 is **Text Match**, which allows the user to filter the search to entries containing a specific text string. This feature is also built on the basis of tokenization and is activated with `enable_match=True`.

It is worth noting that with Text Match, the processing of the query text is based on the logic of OR after tokenization. For example, in the example below, the result will return all documents (using the 'text' field) that contain either 'vector' or 'database'.

```Python
filter = "TEXT_MATCH(text, 'vector database')"
```

If your scenario requires matching both 'vector' and 'database', then you need to write two separate Text Matches and overlay them with AND to achieve your goal.

```Python
filter = "TEXT_MATCH(text, 'vector') and TEXT_MATCH(text, 'database')"
```


## Significant Enhancement in Scalar Filtering Performance

Our emphasis on scalar filtering performance originates from our discovery that the combination of vector retrieval and metadata filtering can greatly improve query performance and accuracy in various scenarios. These scenarios range from image search applications such as corner case identification in autonomous driving to complex RAG scenarios in enterprise knowledge bases. Thus, it is highly suitable for enterprise users to implement in large-scale data application scenarios.

In practice, many factors like how much data you're filtering, how your data is organized, and how you're searching can affect performance. To address this, Milvus 2.5 introduces three new types of indexes—BitMap Index, Array Inverted Index, and the Inverted Index after tokenizing the Varchar text field. These new indexes can significantly improve performance in real-world use cases.

Specifically:

1. **BitMap Index** can be used to accelerate tag filtering (common operators include in, array_contains, etc.), and is suitable for scenarios with fewer field category data (data cardinality). The principle is to determine whether a row of data has a certain value on a column, with 1 for yes and 0 for no, and then maintain a BitMap list. The following chart shows the performance test comparison we conducted based on a customer's business scenario. In this scenario, the data volume is 500 million, the data category is 20, different values have different distribution proportions (1%, 5%, 10%, 50%), and the performance under different filtering amounts also varies. With 50% filtering, we can achieve a 6.8-fold performance gain through BitMap Index. It's worth noting that as cardinality increases, compared to BitMap Index, Inverted Index will show more balanced performance. 

![](https://assets.zilliz.com/QPS_comparison_f3f580d697.png)


2. **Text Match** is based on the Inverted Index after the text field is tokenized. Its performance far exceeds the Wildcard Match (i.e., like + %) function we provided in 2.4. According to our internal test results, the advantages of Text Match are very clear, especially in concurrent query scenarios, where it can achieve up to a 400-fold QPS increase. 

![](https://assets.zilliz.com/data_size_and_concurrency_e19dc44c59.png)

In terms of JSON data processing, we plan to introduce in subsequent versions of 2.5.x the building of inverted indices for user-specified keys and default location information recording for all keys to speed up parsing. We expect both of these areas to significantly enhance the query performance of JSON and Dynamic Field. We plan to showcase more information in future release notes and technical blogs, so stay tuned!


## New Management Interface

Managing a database shouldn't require a computer science degree, but we know database administrators need powerful tools. That's why we've introduced the **Cluster Management WebUI**, a new web-based interface accessible at your cluster's address on port 9091/webui. This observability tool provides:

- Real-time monitoring dashboards showing cluster-wide metrics
- Detailed memory and performance analytics per node
- Segment information and slow query tracking
- System health indicators and node status
- Easy-to-use troubleshooting tools for complex system issues

While this interface is still in beta, we're actively developing it based on user feedback from database administrators. Future updates will include AI-assisted diagnostics, more interactive management features, and enhanced cluster observability capabilities.


## Documentation and Developer Experience

We've completely revamped our **documentation** and **SDK/API** experience to make Milvus more accessible while maintaining depth for experienced users. The improvements include:

- A restructured documentation system with clearer progression from basic to advanced concepts
- Interactive tutorials and real-world examples that showcase practical implementations
- Comprehensive API references with practical code samples
- A more user-friendly SDK design that simplifies common operations
- Illustrated guides that make complex concepts easier to understand
- An AI-powered documentation assistant (ASK AI) for quick answers

The updated SDK/API focuses on improving developer experience through more intuitive interfaces and better integration with the documentation. We believe you'll notice these improvements when working with the 2.5.x series.

However, we know documentation and SDK development is an ongoing process. We'll continue optimizing both the content structure and SDK design based on community feedback. Join our Discord channel to share your suggestions and help us improve further.


## **Summary**

Milvus 2.5 contains 13 new features and several system-level optimizations, contributed not just by Zilliz but the open-source community. We have only touched on a few of them in this post and encourage you to visit our [release note](https://milvus.io/docs/release_notes.md) and [official documents](https://milvus.io/docs) for more information!


Introducing Milvus 2.5: Full-Text Search, More Powerful Metadata Filtering, and Usability Improvements!

Semantic Search vs. Full-Text Search: Which Do I Choose in Milvus 2.5?



## Introduction 

Over the past couple of years, [Retrieval-Augmented Generation (RAG)](https://zilliz.com/learn/Retrieval-Augmented-Generation) has emerged as a trusted solution for large organizations to enhance their [LLM](https://zilliz.com/glossary/large-language-models-(llms))-powered applications, especially those with diverse users. As such applications grow, implementing a multi-tenancy framework becomes essential. **Multi-tenancy** provides secure, isolated access to data for different user groups, ensuring user trust, meeting regulatory standards, and improving operational efficiency.

[Milvus](https://zilliz.com/what-is-milvus) is an open-source [vector database](https://zilliz.com/learn/what-is-vector-database) built to handle high-dimensional [vector data](https://zilliz.com/glossary/vector-embeddings). It is an indispensable infrastructure component of RAG, storing and retrieving contextual information for LLMs from external sources. Milvus offers [flexible multi-tenancy strategies](https://milvus.io/docs/multi_tenancy.md) for various needs, including **database-level, collection-level, and partition-level multi-tenancy**. 

In this post, we’ll cover: 

- What is Multi-Tenancy and Why It Matters

- Multi-Tenancy Strategies in Milvus

- Example: Multi-Tenancy Strategy for a RAG-Powered Enterprise Knowledge Base


## What is Multi-Tenancy and Why It Matters

[**Multi-tenancy**](https://milvus.io/docs/multi_tenancy.md) is an architecture where multiple customers or teams, known as "**tenants,**" share a single instance of an application or system. Each tenant’s data and configurations are logically isolated, ensuring privacy and security, while all tenants share the same underlying infrastructure. 

Imagine a SaaS platform that provides knowledge-based solutions to multiple companies. Each company is a tenant. 

- Tenant A is a healthcare organization storing patient-facing FAQs and compliance documents.

- Tenant B is a tech company managing internal IT troubleshooting workflows.

- Tenant C is a retail business with customer service FAQs for product returns.

Each tenant operates in a completely isolated environment, ensuring that no data from Tenant A leaks into Tenant B’s system or vice versa. Furthermore, resource allocation, query performance, and scaling decisions are tenant-specific, ensuring high performance regardless of workload spikes in one tenant.

Multi-tenancy also works for systems serving different teams within the same organization. Imagine a large company using a RAG-powered knowledge base to serve its internal departments, such as HR, Legal, and Marketing. Each **department is a tenant** with isolated data and resources in this setup. 

Multi-tenancy offers significant benefits, including **cost efficiency, scalability, and robust data security**. By sharing a single infrastructure, service providers can reduce overhead costs and ensure more effective resource consumption. This approach also scales effortlessly—onboarding new tenants requires far fewer resources than creating separate instances for each one, as with single-tenancy models. Importantly, multi-tenancy maintains robust data security by ensuring strict data isolation for each tenant, with access controls and encryption protecting sensitive information from unauthorized access. Additionally, updates, patches, and new features can be deployed across all tenants simultaneously, simplifying system maintenance and reducing the burden on administrators while ensuring that security and compliance standards are consistently upheld. 


## Multi-Tenancy Strategies in Milvus

To understand how Milvus supports multi-tenancy, it's important to first look at how it organizes user data. 


### How Milvus Organizes User Data

Milvus structures data across three layers, moving from broad to granular: [**Database**](https://milvus.io/docs/manage_databases.md), [**Collection**](https://milvus.io/docs/manage-collections.md), and [**Partition/Partition Key**](https://milvus.io/docs/manage-partitions.md). 

![Figure- How Milvus organizes user data .png](https://assets.zilliz.com/Figure_How_Milvus_organizes_user_data_4521c4b8f9.png)

_Figure: How Milvus organizes user data_ 

- **Database**: This acts as a logical container, similar to a database in traditional relational systems.

- **Collection**: Comparable to a table within a database, a collection organizes data into manageable groups.

- **Partition/Partition Key**: Within a collection, data can be further segmented by **Partitions**. Using a **Partition Key**, data with the same key is grouped together. For example, if you use a **user ID** as the **Partition Key**, all data for a specific user will be stored in the same logical segment. This makes it straightforward to retrieve data tied to individual users.

As you move from **Database** to **Collection** to **Partition Key**, the granularity of data organization becomes progressively finer. 

To ensure stronger data security and proper access control, Milvus also provides robust [**Role-Based Access Control (RBAC)**](https://zilliz.com/blog/enabling-fine-grained-access-control-with-milvus-row-level-rbac), allowing administrators to define specific permissions for each user. Only authorized users can access certain data. 

Milvus supports [multiple strategies](https://milvus.io/docs/multi_tenancy.md) for implementing multi-tenancy, offering flexibility based on the needs of your application: **database-level, collection-level, and partition-level multi-tenancy**. 


### Database-Level Multi-Tenancy

With the database-level multi-tenancy approach, each tenant is assigned their own database within the same Milvus cluster. This strategy provides strong data isolation and ensures optimal search performance. However, it can lead to inefficient resource utilization if certain tenants remain inactive.


### Collection-Level Multi-Tenancy

Here, in collection-level multi-tenancy, we can organize data for tenants in two ways. 

- **One Collection for All Tenants**: All tenants share a single collection, with tenant-specific fields used for filtering. While simple to implement, this approach can encounter performance bottlenecks as the number of tenants increases. 

- **One Collection per Tenant**: Each tenant can have a dedicated collection, improving isolation and performance but requiring more resources. This setup may face scalability limitations if the number of tenants exceeds Milvus's collection capacity.


### Partition-Level Multi-Tenancy

Partition-Level Multi-Tenancy focuses on organizing tenants within a single collection. Here, we also have two ways to organize tenant data. 

- **One Partition per Tenant**: Tenants share a collection, but their data is stored in separate partitions. We can isolate data by assigning each tenant a dedicated partition, balancing isolation and search performance. However, this approach is constrained by Milvus's maximum partition limit. 

- **Partition-Key-Based Multi-Tenancy**: This is a more scalable option in which a single collection uses partition keys to distinguish tenants. This method simplifies resource management and supports higher scalability but does not support bulk data inserts.

The table below summarizes the key differences between key multi-tenancy approaches. 

| **Granularity**               | **Database-level**                                               | **Collection-level**                                              | **Partition Key-level**                                           |
| ----------------------------- | ---------------------------------------------------------------- | ----------------------------------------------------------------- | ----------------------------------------------------------------- |
| Max Tenants Supported         | ~1,000                                                          | ~10,000                                                          | ~10,000,000                                                      |
| Data Organization Flexibility | High: Users can define multiple collections with custom schemas. | Medium: Users are limited to one collection with a custom schema. | Low: All users share a collection, requiring a consistent schema. |
| Cost per User                 | High                                                             | Medium                                                            | Low                                                               |
| Physical Resource Isolation   | Yes                                                              | Yes                                                               | No                                                                |
| RBAC                          | Yes                                                              | Yes                                                               | No                                                                |
| Search Performance            | Strong                                                           | Medium                                                            | Strong                                                            |


## Example: Multi-Tenancy Strategy for a RAG-Powered Enterprise Knowledge Base

When designing the multi-tenancy strategy for a RAG system, it's essential to align your approach with the specific needs of your business and your tenants. Milvus offers various multi-tenancy strategies, and choosing the right one depends on the number of tenants, their requirements, and the level of data isolation needed. Here's a practical guide for making these decisions, taking a  RAG-powered enterprise knowledge base as an example. 


### Understanding Tenant Structure Before Choosing a Multi-Tenancy Strategy 

A RAG-powered enterprise knowledge base often serves a small number of tenants. These tenants are usually independent business units like IT, Sales, Legal, and Marketing, each requiring distinct knowledge base services. For example, the HR Department manages sensitive employee information like onboarding guides and benefits policies, which should be confidential and accessible only to HR personnel.

In this case, each business unit should be treated as a separate tenant and a **Database-level multi-tenancy strategy** is often the most suitable. By assigning dedicated databases to each tenant, organizations can achieve strong logical isolation, simplifying management and enhancing security. This setup provides tenants with significant flexibility—they can define custom data models within collections, create as many collections as needed, and independently manage access control for their collections.


### Enhancing Security with Physical Resource Isolation

In situations where data security is highly prioritized, logical isolation at the database level may not be enough. For example, some business units might handle critical or highly sensitive data, requiring stronger guarantees against interference from other tenants. In such cases, we can implement a [physical isolation approach](https://milvus.io/docs/resource_group.md) on top of a database-level multi-tenancy structure. 

Milvus enables us to map logical components, such as databases and collections, to physical resources. This method ensures that the activities of other tenants do not impact critical operations. Let's explore how this approach works in practice. 


![Figure- How Milvus manages physical resources.png](https://assets.zilliz.com/Figure_How_Milvus_manages_physical_resources_6269b908d7.png)


Figure: How Milvus manages physical resources

As shown in the diagram above, there are three layers of resource management in Milvus: **Query Node**, **Resource Group**, and **Database**. 

- **Query Node**: The component that processes query tasks. It runs on a physical machine or container (e.g., a pod in Kubernetes).

- **Resource Group**: A collection of Query Nodes that acts as a bridge between logical components (databases and collections) and physical resources. You can allocate one or more databases or collections to a single Resource Group.

In the example shown in the diagram above, there are three logical **Databases**: X, Y, and Z. 

- **Database X**: Contains **Collection A**.

- **Database Y**: Contains **Collections B** and **C**.

- **Database Z**: Contains **Collections D** and **E**.

Let’s say **Database X** holds a critical knowledge base that we don’t want to be affected by the load from **Database Y** or **Database Z**. To ensure data isolation:

- **Database X** is assigned its own **Resource Group** to guarantee that its critical knowledge base is unaffected by workloads from other databases.

- **Collection E** is also allocated to a separate **Resource Group** within its parent database (**Z**). This provides isolation at the collection level for specific critical data within a shared database.

Meanwhile, the remaining collections in **Databases Y** and **Z** share the physical resources of **Resource Group 2**. 

By carefully mapping logical components to physical resources, organizations can achieve a flexible, scalable, and secure multi-tenancy architecture tailored to their specific business needs.


### Designing End User-Level Access 

Now that we've learned the best practices for choosing a multi-tenancy strategy for an enterprise RAG, let's explore how to design user-level access in such systems. 

In these systems, end users usually interact with the knowledge base in a read-only mode through LLMs. However, organizations still need to track such Q&A data generated by users and link it to specific users for various purposes, such as improving the knowledge base’s accuracy or offering personalized services.

Take a hospital's smart consultation service desk as an example. Patients might ask questions like, “Are there any available appointments with the specialist today?” or "Is there any specific preparation needed for my upcoming surgery?" While these questions don’t directly impact the knowledge base, it’s important for the hospital to track such interactions to improve services. These Q&A pairs are usually stored in a separate database (it doesn't necessarily have to be a vector database) dedicated to logging interactions.

![Figure- The multi-tenancy architecture for an enterprise RAG knowledge base .png](https://assets.zilliz.com/Figure_The_multi_tenancy_architecture_for_an_enterprise_RAG_knowledge_base_7c9ad8d4d1.png)

_Figure: The multi-tenancy architecture for an enterprise RAG knowledge base_ 

The diagram above shows the multi-tenancy architecture of an enterprise RAG system. 

- **System Administrators** oversee the RAG system, manage resource allocation, assign databases, map them to resource groups, and ensure scalability. They handle the physical infrastructure, as shown in the diagram, where each resource group (e.g., Resource Group 1, 2, and 3) is mapped to physical servers (query nodes).

- **Tenants (Database owners and developers)** manage the knowledge base, iterating on it based on the user-generated Q&A data, as shown in the diagram. Different databases (Database X, Y, Z) contain collections with different knowledge base content (Collection A, B, etc.).

- **End Users** interact with the system in a read-only manner through the LLM. As they query the system, their questions are logged in the separate Q&A record table (a separate database), continuously feeding valuable data back into the system.

This design ensures that each process layer—from user interaction to system administration—works seamlessly, helping the organization build a robust and continuously improving knowledge base.


## Summary 

In this blog, we’ve explored how [**multi-tenancy**](https://milvus.io/docs/multi_tenancy.md) frameworks play a critical role in the scalability, security, and performance of RAG-powered knowledge bases. By isolating data and resources for different tenants, businesses can ensure privacy, regulatory compliance, and optimized resource allocation across a shared infrastructure. [Milvus](https://milvus.io/docs/overview.md), with its flexible multi-tenancy strategies, allows businesses to choose the right level of data isolation—from database level to partition level—depending on their specific needs. Choosing the right multi-tenancy approach ensures companies can provide tailored services to tenants, even when dealing with diverse data and workloads. 

By following the best practices outlined here, organizations can effectively design and manage multi-tenancy RAG systems that not only deliver superior user experiences but also scale effortlessly as business needs grow. Milvus’ architecture ensures that enterprises can maintain high levels of isolation, security, and performance, making it a crucial component in building enterprise-grade, RAG-powered knowledge bases.


## Stay Tuned for More Insights into Multi-Tenancy RAG

In this blog, we’ve discussed how Milvus’ multi-tenancy strategies are designed to manage tenants, but not end users within those tenants. End-user interactions usually happen at the application layer, while the vector database itself remains unaware of those users.

You might be wondering: _If I want to provide more precise answers based on each end user’s query history, doesn’t Milvus need to maintain a personalized Q&A context for each user?_

That’s a great question, and the answer really depends on the use case. For example, in an on-demand consultation service, queries are random, and the main focus is on the quality of the knowledge base rather than on keeping track of a user's historical context.

However, in other cases, RAG systems must be context-aware. When this is required, Milvus needs to collaborate with the application layer to maintain a personalized memory of each user's context. This design is especially important for applications with massive end users, which we’ll explore in greater detail in my next post. Stay tuned for more insights!


Designing Multi-Tenancy RAG with Milvus: Best Practices for Scalable Enterprise Knowledge Bases




[**Milvus**](https://github.com/milvus-io/milvus) is an open-source [vector database](https://zilliz.com/learn/what-is-vector-database) designed to manage high-dimensional vector data. Whether you're building intelligent search engines, recommendation systems, or next-gen AI solutions such as retrieval augmented generation ([RAG](https://zilliz.com/learn/Retrieval-Augmented-Generation)), Milvus is a powerful tool at your fingertips. 

But what truly drives Milvus forward isn’t just its advanced technology—it’s the vibrant, passionate [developer community](https://zilliz.com/community) behind it. As an open-source project, Milvus thrives and evolves thanks to the contributions of developers like you. Every bug fix, feature addition, and performance enhancement from the community makes Milvus faster, more scalable, and more reliable.

Whether you're passionate about open-source, eager to learn, or want to make a lasting impact in AI, Milvus is the perfect place to contribute. This guide will walk you through the process—from setting up your development environment to submitting your first pull request. We’ll also highlight common challenges you might face and provide solutions to overcome them.

Ready to dive in? Let’s make Milvus even better together!


## Setting Up Your Milvus Development Environment

First thing first: setting up your development environment. You can either install Milvus on your local machine or use Docker—both methods are straightforward, but you’ll also need to install a few third-party dependencies to get everything running.


### Building Milvus Locally

If you like building things from scratch, building Milvus on your local machine is a breeze. Milvus makes it easy by bundling all the dependencies in the `install_deps.sh` script. Here’s the quick setup:


```
# Install third-party dependencies.
$ cd milvus/
$ ./scripts/install_deps.sh

# Compile Milvus.
$ make
```

### Building Milvus with Docker

If you prefer Docker, there are two ways to go about it: you can either run commands in a pre-built container or spin up a dev container for a more hands-on approach.



```
# Option 1: Run commands in a pre-built Docker container  
build/builder.sh make  

# Option 2: Spin up a dev container  
./scripts/devcontainer.sh up  
docker-compose -f docker-compose-devcontainer.yml ps  
docker exec -ti milvus-builder-1 bash  
make milvus  
```



**Platform Notes:** If you’re on Linux, you’re good to go—compilation issues are pretty rare. However, Mac users, especially with M1 chips, might run into some bumps along the way. Don’t sweat it, though—we have a guide to help you work through the most common issues. 

![](https://assets.zilliz.com/Figure_OS_configuration_52092fb1b7.png)

_Figure: OS configuration_ 

For the full setup guide, check out the official [Milvus Development Guide](https://github.com/milvus-io/milvus/blob/master/DEVELOPMENT.md).


### Common Issues and How to Fix Them

Sometimes, setting up your Milvus development environment doesn’t go as smoothly as planned. Don’t worry—here’s a quick rundown of common issues you might hit and how to fix them fast.


#### Homebrew: Unexpected Disconnect While Reading Sideband Packet

If you’re using Homebrew and see an error like this:


```
==> Tapping homebrew/core
remote: Enumerating objects: 1107077, done.
remote: Counting objects: 100% (228/228), done.
remote: Compressing objects: 100% (157/157), done.
error: 545 bytes of body are still expected.44 MiB | 341.00 KiB/s
fetch-pack: unexpected disconnect while reading sideband packet
fatal: early EOF
fatal: index-pack failed
Failed during: git fetch --force origin refs/heads/master:refs/remotes/origin/master
myuser~ %
```



**Fix:** Increase the `http.postBuffer` size:


```
git config --global http.postBuffer 1M
```


If you also run into `Brew: command not found` after installing Homebrew, you might need to set up your Git user configuration:


```
git config --global user.email xxxgit config --global user.name xxx
```



#### Docker: Error Getting Credentials

When working with Docker, you might see this:


```
error getting credentials - err: exit status 1, out: ``  
```


![](https://assets.zilliz.com/Docker_Error_Getting_Credentials_797f3043fb.png)



**Fix:** Open` ~/.docker/config.json` and remove the `credsStore` field.


#### Python: No Module Named 'imp'

If Python throws this error, it’s because Python 3.12 removed the `imp` module, which some older dependencies still use.


 ![](https://assets.zilliz.com/Python_No_Module_Named_imp_65eb2c5c66.png)

**Fix:** Downgrade to Python 3.11:



```
brew install python@3.11  
```


#### Conan: Unrecognized Arguments or Command Not Found

**Issue:** If you see `Unrecognized arguments: --install-folder conan`, you’re likely using an incompatible Conan version.

![](https://assets.zilliz.com/Conan_Unrecognized_Arguments_or_Command_Not_Found_8f2029db72.png)

**Fix:** Downgrade to Conan 1.61:


```
pip install conan==1.61  
```


**Issue:** If you see `Conan command not found`, it means your Python environment isn’t properly set up.

**Fix:** Add Python’s bin directory to your `PATH`:


```
export PATH="/path/to/python/bin:$PATH"
```



#### LLVM: Use of Undeclared Identifier 'kSecFormatOpenSSL'

This error usually means your LLVM dependencies are outdated.

![](https://assets.zilliz.com/LLVM_Use_of_Undeclared_Identifier_k_Sec_Format_Open_SSL_f0ca6f0166.png)

**Fix:** Reinstall LLVM 15 and update your environment variables:


```
brew reinstall llvm@15
export LDFLAGS="-L/opt/homebrew/opt/llvm@15/lib"
export CPPFLAGS="-I/opt/homebrew/opt/llvm@15/include"
```


**Pro Tips**

- Always double-check your tool versions and dependencies.

- If something still doesn’t work, the[ Milvus GitHub Issues page](https://github.com/milvus-io/milvus/issues) is a great place to find answers or ask for help.


### Configuring VS Code for C++ and Go Integration

Getting C++ and Go to work together in VS Code is easier than it sounds. With the right setup, you can streamline your development process for Milvus. Just tweak your `user.settings` file with the configuration below:


```
{
   "go.toolsEnvVars": {
       "PKG_CONFIG_PATH": "/Users/zilliz/milvus/internal/core/output/lib/pkgconfig:/Users/zilliz/workspace/milvus/internal/core/output/lib64/pkgconfig",
       "LD_LIBRARY_PATH": "/Users/zilliz/workspace/milvus/internal/core/output/lib:/Users/zilliz/workspace/milvus/internal/core/output/lib64",
       "RPATH": "/Users/zilliz/workspace/milvus/internal/core/output/lib:/Users/zilliz/workspace/milvus/internal/core/output/lib64"
   },
   "go.testEnvVars": {
       "PKG_CONFIG_PATH": "/Users/zilliz/workspace/milvus/internal/core/output/lib/pkgconfig:/Users/zilliz/workspace/milvus/internal/core/output/lib64/pkgconfig",
       "LD_LIBRARY_PATH": "/Users/zilliz/workspace/milvus/internal/core/output/lib:/Users/zilliz/workspace/milvus/internal/core/output/lib64",
       "RPATH": "/Users/zilliz/workspace/milvus/internal/core/output/lib:/Users/zilliz/workspace/milvus/internal/core/output/lib64"
   },
   "go.buildFlags": [
       "-ldflags=-r /Users/zilliz/workspace/milvus/internal/core/output/lib"
   ],
   "terminal.integrated.env.linux": {
       "PKG_CONFIG_PATH": "/Users/zilliz/workspace/milvus/internal/core/output/lib/pkgconfig:/Users/zilliz/workspace/milvus/internal/core/output/lib64/pkgconfig",
       "LD_LIBRARY_PATH": "/Users/zilliz/workspace/milvus/internal/core/output/lib:/Users/zilliz/workspace/milvus/internal/core/output/lib64",
       "RPATH": "/Users/zilliz/workspace/milvus/internal/core/output/lib:/Users/zilliz/workspace/milvus/internal/core/output/lib64"
   },
   "go.useLanguageServer": true,
   "gopls": {
       "formatting.gofumpt": true
   },
   "go.formatTool": "gofumpt",
   "go.lintTool": "golangci-lint",
   "go.testTags": "dynamic",
   "go.testTimeout": "10m"
}
```



Here’s what this configuration does:

- **Environment Variables:** Sets up paths for `PKG_CONFIG_PATH`, `LD_LIBRARY_PATH`, and `RPATH`, which are critical for locating libraries during builds and tests.

- **Go Tools Integration:** Enables Go’s language server (`gopls`) and configures tools like `gofumpt` for formatting and `golangci-lint` for linting.

- **Testing Setup:** Adds `testTags` and increases the timeout for running tests to 10 minutes.

Once added, this setup ensures a seamless integration between C++ and Go workflows. It’s perfect for building and testing Milvus without constant environment tweaking.


**Pro Tip**

After setting this up, run a quick test build to confirm everything works. If something feels off, double-check the paths and VS Code’s Go extension version.


## Deploying Milvus

Milvus supports [three deployment modes](https://milvus.io/docs/install-overview.md)—**Lite, Standalone,** and **Distributed.**  

- [**Milvus Lite**](https://milvus.io/blog/introducing-milvus-lite.md) is a Python library and an ultra-lightweight version of Milvus. It’s perfect for rapid prototyping in Python or notebook environments and for small-scale local experiments. 

- **Milvus Standalone** is the single-node deployment option for Milvus, using a client-server model. It is the Milvus equivalent of MySQL, while Milvus Lite is like SQLite. 

- **Milvus Distributed** is the distributed mode of Milvus, which is ideal for enterprise users building large-scale vector database systems or vector data platforms. 

All these deployments rely on three core components:

- **Milvus:** The vector database engine that drives all operations.

- **Etcd:** The metadata engine that manages Milvus's internal metadata.

- **MinIO:** The storage engine that ensures data persistence.

When running in **Distributed** mode, Milvus also incorporates **Pulsar** for distributed message processing using a Pub/Sub mechanism, making it scalable for high-throughput environments.


### Milvus Standalone

The Standalone mode is tailored for single-instance setups, making it perfect for testing and small-scale applications. Here’s how to get started:


```
# Deploy Milvus Standalone  
sudo docker-compose -f deployments/docker/dev/docker-compose.yml up -d
# Start the standalone service  
bash ./scripts/start_standalone.sh
```



### Milvus Distributed (previously known as Milvus Cluster) 

For larger datasets and higher traffic, the Distributed mode offers horizontal scalability. It combines multiple Milvus instances into a single cohesive system. Deployment is made easy with the **Milvus Operator**, which runs on Kubernetes and manages the entire Milvus stack for you.

Want step-by-step guidance? Check out the [Milvus Installation Guide](https://milvus.io/docs/install_cluster-milvusoperator.md).


## Running End-to-End (E2E) Tests

Once your Milvus deployment is up and running, testing its functionality is a breeze with E2E tests. These tests cover every part of your setup to ensure everything works as expected. Here’s how to run them:


```
# Navigate to the test directory  
cd tests/python_client  

# Install dependencies  
pip install -r requirements.txt  

# Run E2E tests  
pytest --tags=L0 -n auto  
```



For in-depth instructions and troubleshooting tips, refer to the [Milvus Development Guide](https://github.com/milvus-io/milvus/blob/master/DEVELOPMENT.md#e2e-tests).

**Pro Tip**

If you're new to Milvus, start with Milvus Lite or Standalone mode to get a feel for its capabilities before scaling up to Distributed mode for production-level workloads.


## Submitting Your Code

Congrats! You’ve cleared all unit and E2E tests (or debugged and recompiled as needed). While the first build can take some time, future ones will be much faster—so no need to worry. With everything passing, you’re ready to submit your changes and contribute to Milvus!


### Link Your Pull Request (PR) to an Issue

Every PR to Milvus needs to be tied to a relevant issue. Here's how to handle this:

- **Check for Existing Issues:** Look through the[ Milvus issue tracker](https://github.com/milvus-io/milvus/issues) to see if there’s already an issue related to your changes.

- **Create a New Issue:** If no relevant issue exists, open a new one and explain the problem you’re solving or the feature you’re adding.


### Submitting Your Code

1. **Fork the Repository:** Start by forking the[ Milvus repo](https://github.com/milvus-io/milvus) to your GitHub account.

2. **Create a Branch:** Clone your fork locally and make a new branch for your changes.

3. **Commit with Signed-off-by Signature:** Ensure your commits include a `Signed-off-by` signature to comply with open-source licensing:


```
git commit -m "Commit of your change" -s
```


This step certifies your contribution is in line with the Developer Certificate of Origin (DCO).


#### **Helpful Resources**

For detailed steps and best practices, check out the[ Milvus Contribution Guide](https://github.com/milvus-io/milvus/blob/master/CONTRIBUTING.md).


## Opportunities to Contribute

Congrats—you’ve got Milvus up and running! You’ve explored its deployment modes, run your tests, and maybe even dug into the code. Now it’s time to level up: contribute to [Milvus](https://github.com/milvus-io/milvus) and help shape the future of AI and [unstructured data](https://zilliz.com/learn/introduction-to-unstructured-data).

No matter your skillset, there’s a place for you in the Milvus community! Whether you’re a developer who loves solving complex challenges, a tech writer who loves writing clean documentation or engineering blogs, or a Kubernetes enthusiast looking to improve deployments, there’s a way for you to make an impact.

Take a look at the opportunities below and find your perfect match. Every contribution helps move Milvus forward—and who knows? Your next pull request might just power the next wave of innovation. So, what are you waiting for? Let’s get started! 🚀

| Projects                                                                                                                                                                            | Suitable for                             | Guidelines                                                                                          |
| ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------- | --------------------------------------------------------------------------------------------------- |
| [milvus](https://github.com/milvus-io/milvus), [milvus-sdk-go](https://github.com/milvus-io/milvus-sdk-go)                                                                          | Go developers                            | /                                                                                                   |
| [milvus](https://github.com/milvus-io/milvus), [knowhere](https://github.com/milvus-io/knowhere)                                                                                    | CPP developers                           | /                                                                                                   |
| [pymilvus](https://github.com/milvus-io/pymilvus), [milvus-sdk-node](https://github.com/milvus-io/milvus-sdk-node), [milvus-sdk-java](https://github.com/milvus-io/milvus-sdk-java) | Developers interested in other languages | [Contributing to PyMilvus](https://github.com/milvus-io/pymilvus/blob/master/CONTRIBUTING.md)       |
| [milvus-helm](https://github.com/milvus-io/milvus-helm)                                                                                                                             | Kubernetes enthusiasts                   | /                                                                                                   |
| [Milvus-docs](https://github.com/milvus-io/milvus-docs), [milvus-io/community/blog](https://github.com/milvus-io/community)                                                         | Tech writers                             | [Contributing to milvus docs](https://github.com/milvus-io/milvus-docs/blob/v2.0.0/CONTRIBUTING.md) |
| [milvus-insight](https://github.com/zilliztech/milvus-insight)                                                                                                                      | Web developers                           | /                                                                                                   |


## A Final Word

Milvus offers various SDKs—[Python](https://milvus.io/docs/install-pymilvus.md) (PyMilvus), [Java](https://milvus.io/docs/install-java.md), [Go](https://milvus.io/docs/install-go.md), and [Node.js](https://milvus.io/docs/install-node.md)—that make it simple to start building. Contributing to Milvus isn’t just about code—it’s about joining a vibrant and innovative community. 

🚀Welcome to the Milvus developer community, and happy coding! We can’t wait to see what you’ll create. 


## Further Reading

- [Join the Milvus Community of AI Developers](https://zilliz.com/community)

- [What are Vector Databases and How Do They Work?](https://zilliz.com/learn/what-is-vector-database)

- [Milvus Lite vs. Standalone vs. Distributed: Which Mode is Right for You? ](https://zilliz.com/blog/choose-the-right-milvus-deployment-mode-ai-applications)

- [Build AI Apps with Milvus: Tutorials & Notebooks](https://zilliz.com/learn/milvus-notebooks)

- [Top Performing AI Models for Your GenAI Apps | Zilliz](https://zilliz.com/ai-models)

- [What is RAG?](https://zilliz.com/learn/Retrieval-Augmented-Generation)

- [Generative AI Resource Hub | Zilliz](https://zilliz.com/learn/generative-ai)

How to Contribute to Milvus: A Quick Start for Developers




[Semantic search](https://zilliz.com/glossary/semantic-search) allows machines to understand language and yield better search results, which is essential in AI and data analytics. Once the language is represented as [embeddings](https://zilliz.com/learn/everything-you-should-know-about-vector-embeddings), the search can be performed using exact or approximate methods. Approximate Nearest Neighbor ([ANN](https://zilliz.com/glossary/anns)) search is a method used to quickly find points in a dataset that are closest to a given query point, unlike [exact nearest neighbor search](https://zilliz.com/blog/k-nearest-neighbor-algorithm-for-machine-learning), which can be computationally expensive for high-dimensional data. ANN allows faster retrieval by providing results that are approximately close to the nearest neighbors. 

One of the algorithms for Approximate Nearest Neighbor (ANN) search is [HNSW](https://zilliz.com/learn/hierarchical-navigable-small-worlds-HNSW) (Hierarchical Navigable Small Worlds), implemented under [HNSWlib](https://zilliz.com/learn/learn-hnswlib-graph-based-library-for-fast-anns), which will be the focus of today's discussion. In this blog, we will:

- Understand the HNSW algorithm.

- Explore HNSWlib and its key features.

- Set up HNSWlib, covering index building and search implementation.

- Compare it with Milvus.


## Understanding HNSW

**Hierarchical Navigable Small Worlds (**[**HNSW**](https://zilliz.com/learn/hierarchical-navigable-small-worlds-HNSW)**)** is a graph-based data structure that allows efficient similarity searches, particularly in high-dimensional spaces, by building a multi-layered graph of "small world" networks. Introduced in [2016](https://arxiv.org/abs/1603.09320), HNSW addresses the scalability issues associated with traditional search methods like brute-force and tree-based searches. It is ideal for applications involving large datasets, such as recommendation systems, image recognition, and [retrieval-augmented generation (RAG)](https://zilliz.com/vector-database-use-cases/llm-retrieval-augmented-generation).


### Why HNSW Matters

HNSW significantly enhances the performance of nearest-neighbor search in high-dimensional spaces. Combining the hierarchical structure with small-world navigability avoids the computational inefficiency of older methods, enabling it to perform well even with massive, complex datasets. To understand this better, let’s look at how it works now.


### How HNSW Works

1. **Hierarchical Layers:** HNSW organizes data into a hierarchy of layers, where each layer contains nodes connected by edges. The top layers are sparser, allowing for broad "skips" across the graph, much like zooming out on a map to see only major highways between cities. Lower layers increase in density, providing finer detail and more connections between closer neighbors.

2. **Navigable Small Worlds Concept:** Each layer in HNSW builds on the concept of a "small world" network, where nodes (data points) are only a few "hops" away from each other. The search algorithm begins at the highest, sparsest layer and works downward, moving to progressively denser layers to refine the search. This approach is like moving from a global view down to neighborhood-level details, gradually narrowing the search area.

![](https://assets.zilliz.com/Fig_1_An_Example_of_a_Navigable_Small_World_Graph_afa737ee9f.png)

[Fig 1](https://daniel-at-world.blogspot.com/2019/04/navigable-small-world-graphs-for.html): An Example of a Navigable Small World Graph

3. **Skip List-Like Structure:** The hierarchical aspect of HNSW resembles a skip list, a probabilistic data structure where higher layers have fewer nodes, allowing for faster initial searches. 

![](https://assets.zilliz.com/Fig_2_An_Example_of_Skip_List_Structure_f41b07234d.png)

[Fig 2](https://www.cs.cmu.edu/~ckingsf/bioinfo-lectures/skiplists.pdf): An Example of Skip List Structure

To search for 96 in the given skip list, we begin at the top level on the far left at the header node. Moving to the right, we encounter 31, less than 96, so we continue to the next node. Now, we need to move down a level where we see 31 again; since it’s still less than 96, we descend another level. Finding 31 once more, we then move right and reach 96, our target value. Thus, we locate 96 without needing to descend to the lowest levels of the skip list.

4. **Search Efficiency:** The HNSW algorithm starts from an entry node at the highest layer, progressing to closer neighbors with each step. It descends through the layers, using each one for coarse-to-fine-grained exploration, until it reaches the lowest layer where the most similar nodes are likely found. This layered navigation reduces the number of nodes and edges that need to be explored, making the search fast and accurate.

5. **Insertion and Maintenance**: When adding a new node, the algorithm determines its entry layer based on probability and connects it to nearby nodes using a neighbor selection heuristic. The heuristic aims to optimize connectivity, creating links that improve navigability while balancing graph density. This approach keeps the structure robust and adaptable to new data points.

While we have a foundational understanding of the HNSW algorithm, implementing it from scratch can be overwhelming. Fortunately, the community has developed libraries like [HNSWlib](https://github.com/nmslib/hnswlib) to simplify usage, making it accessible without scratching your head. So, let’s take a closer look at HNSWlib.


## Overview of HNSWlib

HNSWlib, a popular library implementing HNSW, is highly efficient and scalable, performing well even with millions of points. It achieves sublinear time complexity by allowing quick jumps between graph layers and optimizing the search for dense, high-dimensional data. Here are the key features of HNSWlib include:

- **Graph-Based Structure:** A multi-layered graph represents data points, allowing fast, nearest-neighbor searches.

- **High-Dimensional Efficiency:** Optimized for high-dimensional data, providing quick and accurate approximate searches.

- **Sublinear Search Time:** Achieves sublinear complexity by skipping layers, improving speed significantly.

- **Dynamic Updates:** Supports real-time insertion and deletion of nodes without requiring a complete graph rebuild.

- **Memory Efficiency:** Efficient memory usage, suitable for large datasets.

- **Scalability:** Scales well to millions of data points, making it ideal for medium-scale applications like recommendation systems.

**Note:** HNSWlib is excellent for creating simple prototypes for vector search applications. However, due to scalability limitations, there may be better choices such as [purpose-built vector databases](https://zilliz.com/blog/what-is-a-real-vector-database) for more complex scenarios involving hundreds of millions or even billions of data points. Let’s see that in action.


## Getting Started with HNSWlib: A Step-by-Step Guide

This section will demonstrate using HNSWlib as a [vector search library](https://zilliz.com/learn/comparing-vector-database-vector-search-library-and-vector-search-plugin) by creating an HNSW index, inserting data, and performing searches. Let’s start with installation:


### Setup and Imports

To get started with HNSWlib in Python, first install it using pip:

```
pip install hnswlib
```

Then, import the necessary libraries:

```
import hnswlib 
import numpy as np
```

### Preparing Data

In this example, we’ll use `NumPy`to generate a random dataset with 10,000 elements, each with a dimension size 256.

```
dim = 256  # Dimensionality of your vectors
num_elements = 10000  # Number of elements to insert
```

Let’s create the data:

```
data = np.random.rand(num_elements, dim).astype(np.float32)  # Example data
```

Now our data is ready, let’s build an index.


### Building an Index

In building an index, we need to define the dimensionality of the vectors and the space type. Let’s create an index:

```
p = hnswlib.Index(space='l2', dim=dim)
```
- `space='l2'`: This parameter defines the distance metric used for similarity. Setting it to `'l2'` means using the Euclidean distance (L2 norm). If you instead set it to `'ip'`, it would use the inner product, which is helpful for tasks like cosine similarity.

* `dim=dim`: This parameter specifies the dimensionality of the data points you’ll be working with. It must match the dimension of the data you plan to add to the index.

Here’s how to initialize an index:

```
p.init_index(max_elements=num_elements, ef_construction=200, M=16)
```

- `max_elements=num_elements`: This sets the maximum number of elements that can be added to the index. `Num_elements` is the maximum capacity, so we set this to 10,000 as we are working with 10,000 data points.

* `ef_construction=200`: This parameter controls the accuracy vs. construction speed trade-off during index creation. A higher value improves recall (accuracy) but increases memory usage and build time. Common values range from 100 to 200.

- `M=16`: This parameter determines the number of bi-directional links created for each data point, influencing accuracy and search speed. Typical values are between 12 and 48; 16 is often a good balance for moderate accuracy and speed.

```
p.set_ef(50)  # This parameter controls the speed/accuracy trade-off
```

- `ef`: The `ef` parameter, short for “exploration factor,” determines how many neighbors are examined during a search. A higher `ef` value results in more neighbors being explored, which generally increases the accuracy (recall) of the search but also makes it slower. Conversely, a lower `ef` value can search faster but might reduce accuracy.

In this case, Setting `ef` to 50 means the search algorithm will evaluate up to 50 neighbors when finding the most similar data points.

Note: `ef_construction` sets neighbor search effort during index creation, enhancing accuracy but slowing construction. `ef` controls search effort during querying, balancing speed and recall dynamically for each query.


### Performing Searches

To perform a nearest neighbor search using HNSWlib, we first create a random query vector. In this example, the vector's dimensionality matches the indexed data.

```
query_vector = np.random.rand(dim).astype(np.float32)  # Example query

labels, distances = p.knn_query(query_vector, k=5)  # k is the number of nearest neighbors
```

- `query_vector`: This line generates a random vector with the same dimensionality as the indexed data, ensuring compatibility for the nearest neighbor search.
- `knn_query`: The method searches for the `k` nearest neighbors of the `query_vector` within the index `p`. It returns two arrays: `labels`, which contain the indices of the nearest neighbors, and `distances`, which indicate the distances from the query vector to each of these neighbors. Here, `k=5` specifies that we want to find the five closest neighbors.

Here are the results after printing the labels and distances:

```
print("Nearest neighbors' labels:", labels)
print("Distances:", distances)
```

```
> Nearest neighbors' labels: [[4498 1751 5647 4483 2471]]
> Distances: [[33.718    35.484592 35.627766 35.828312 35.91495 ]]
```

Here we have it, a simple guide to get your wheels started with HNSWlib. 

As mentioned, HNSWlib is a great vector search engine for prototyping or experimenting with medium-sized datasets. If you have higher scalability requirements or need other enterprise-level features, you may need to choose a purpose-built vector database like the open-source [Milvus](https://zilliz.com/what-is-milvus) or its fully managed service on [Zilliz Cloud](https://zilliz.com/cloud). So, in the following section, we will compare HNSWlib with Milvus. 


## HNSWlib vs. Purpose-Built Vector Databases Like Milvus

A [vector database](https://zilliz.com/learn/what-is-vector-database) stores data as mathematical representations, enabling [machine learning models](https://zilliz.com/ai-models) to power search, recommendations, and text generation by identifying data through [similarity metrics](https://zilliz.com/blog/similarity-metrics-for-vector-search) for contextual understanding. 

Vector indices libraries like HNSWlib improve v[ector search](https://zilliz.com/learn/vector-similarity-search) and retrieval but lack the management features of a full database. On the other hand, vector databases, like [Milvus](https://milvus.io/), are designed to handle vector embeddings at scale, providing advantages in data management, indexing, and querying capabilities that standalone libraries typically lack. Here are some other benefits of using Milvus:

- **High-Speed Vector Similarity Search**: Milvus provides millisecond-level search performance across billion-scale vector datasets, ideal for applications like image retrieval, recommendation systems, natural language processing ([NLP](https://zilliz.com/learn/A-Beginner-Guide-to-Natural-Language-Processing)), and retrieval augmented generation ([RAG](https://zilliz.com/learn/Retrieval-Augmented-Generation)). 

- **Scalability and High Availability:** Built to handle massive data volumes, Milvus scales horizontally and includes replication and failover mechanisms for reliability.

- **Distributed Architecture:** Milvus uses a distributed, scalable architecture that separates storage and computing across multiple nodes for flexibility and robustness.

- [**Hybrid search**](https://zilliz.com/blog/a-review-of-hybrid-search-in-milvus)**:** Milvus supports multimodal search, hybrid [sparse and dense search](https://zilliz.com/learn/sparse-and-dense-embeddings), and hybrid dense and [full-text search](https://thenewstack.io/elasticsearch-was-great-but-vector-databases-are-the-future/), offering versatile and flexible search functionality.

- **Flexible Data Support**: Milvus supports various data types—vectors, scalars, and structured data—allowing seamless management and analysis within a single system.

- [**Active Community**](https://discord.com/invite/8uyFbECzPX) **and Support**: A thriving community provides regular updates, tutorials, and support, ensuring Milvus remains aligned with user needs and advances in the field. 

- [AI integration](https://milvus.io/docs/integrations_overview.md): Milvus has integrated with various popular AI frameworks and technologies, making it easier for developers to build applications with their familiar tech stacks. 

Milvus also provides a fully managed service on [Ziliz Cloud](https://zilliz.com/cloud), which is hassle-free and 10x faster than Milvus. 


### Comparison: Milvus vs. HNSWlib

| **Feature** |                                      **Milvus**                                     |                   **HNSWlib**                   |
| :---------: | :---------------------------------------------------------------------------------: | :---------------------------------------------: |
| Scalability |                        Handles billions of vectors with ease                        |    Fit for smaller datasets due to RAM usage    |
|  Ideal for  |            Prototyping, experimenting, and enterprise-level applications            | Focuses on prototypes and lightweight ANN tasks |
|   Indexing  | Supports 10+ indexing algorithms, including HNSW, DiskANN, Quantization, and Binary |           Uses a graph-based HNSW only          |
| Integration |                        Offers APIs and cloud-native services                        |   Serves as a lightweight, standalone library   |
| Performance |                    Optimizes for large data, distributed queries                    |    Offers high speed but limited scalability    |

Overall, Milvus is generally preferable for large-scale, production-grade applications with complex indexing needs, while HNSWlib is ideal for prototyping and more straightforward use cases.


## Conclusion

Semantic search can be resource-intensive, so internal data structuring, like that performed by HNSW, is essential for faster data retrieval. Libraries like HNSWlib care about the implementation, so the developers have the recipes ready to prototype vector capabilities. With just a few lines of code, we can build up our own index and perform searches.

HNSWlib is a great way to start. However, if you want to build complex and production-ready AI applications, purpose-built vector databases are the best option. For example, [Milvus](https://milvus.io/) is an open-source vector database with many enterprise-ready features such as high-speed vector search, scalability, availability, and flexibility in terms of data types and programming language.  


## Further Reading

- [What is Faiss (Facebook AI Similarity Search)? ](https://zilliz.com/learn/faiss)

- [What is HNSWlib? A Graph-based Library for Fast ANN Search ](https://zilliz.com/learn/learn-hnswlib-graph-based-library-for-fast-anns)

- [What is ScaNN (Scalable Nearest Neighbors)? ](https://zilliz.com/learn/what-is-scann-scalable-nearest-neighbors-google)

- [VectorDBBench: An Open-Source VectorDB Benchmark Tool](https://zilliz.com/vector-database-benchmark-tool?database=ZillizCloud%2CMilvus%2CElasticCloud%2CPgVector%2CPinecone%2CQdrantCloud%2CWeaviateCloud&dataset=medium&filter=none%2Clow%2Chigh&tab=1)

- [Generative AI Resource Hub | Zilliz](https://zilliz.com/learn/generative-ai)

- [What are Vector Databases and How Do They Work? ](https://zilliz.com/learn/what-is-vector-database)

- [What is RAG? ](https://zilliz.com/learn/Retrieval-Augmented-Generation)

- [Top Performing AI Models for Your GenAI Apps | Zilliz](https://zilliz.com/ai-models)



Getting Started with HNSWlib


Get expert help with your Milvus [vector database](https://zilliz.com/learn/what-is-vector-database) projects. Book a 20-minute one-on-one session to get insights, guidance, and answers to your questions. Whether you’re just getting started or need help with a specific issue, we’re here to help you make the most out of Milvus.

[![](https://assets.zilliz.com/milvus_office_hours_09848e076b.png)](https://meetings.hubspot.com/chloe-williams1/milvus-office-hour)


## Why Book a Session?

Here are a few great reasons to book time during our office hours:

- **Optimizing Milvus Performance:** Get tips on tuning configurations to boost database performance.

- **Data Schema Design Advice:** Learn best practices for data schema and index types to best fit your use case.

- **Scaling Your Milvus Deployment:** Get advice on managing and scaling Milvus for larger data volumes.

- **Troubleshooting Specific Issues:** Stuck on an error or unusual behavior? We can help troubleshoot and resolve it.

- **Integration Assistance:** Need guidance on using Milvus with other integrated frameworks like LangChain, LlamaIndex, Spark, or your data processing pipelines?

- **Learning about New Features:** Stay updated with the latest Milvus capabilities and how they could benefit your project.

Don’t miss this opportunity to level up your Milvus implementation with expert guidance.

If you’re unable to join us live, don’t worry—there are still plenty of ways to get support! The Milvus community is active and ready to help on both [Discord](https://discord.com/invite/8uyFbECzPX) and[ GitHub Discussions](https://github.com/search?q=milvus&type=discussions). Whether you prefer live chat or asynchronous discussions, these platforms are a great place to ask questions, share knowledge, and connect with other Milvus users and contributors.



Join Milvus Office Hours to Get Support from Vector DB Experts!



## What are Matryoshka Embeddings? 

When building efficient [vector search](https://zilliz.com/learn/vector-similarity-search) systems, one key challenge is managing storage costs while maintaining acceptable latency and recall. Modern [embedding models](https://zilliz.com/blog/choosing-the-right-embedding-model-for-your-data) output vectors with hundreds or thousands of dimensions, creating significant storage and computational overhead for the raw vector and index.

Traditionally, the storage requirements are reduced by applying a quantization or dimensionality reduction method just before building the index. For instance, we can save storage by lowering the precision using [Product Quantization](https://zilliz.com/learn/scalar-quantization-and-product-quantization) (PQ) or the number of dimensions using Principal Component Analysis (PCA). These methods analyze the entire vector set to find a more compact one that maintains the semantic relationships between vectors.

While effective, these standard approaches reduce precision or dimensionality only once and at a single scale. But what if we could maintain multiple layers of detail simultaneously, like a pyramid of increasingly precise representations?

Enter [**Matryoshka embeddings**](https://arxiv.org/abs/2205.13147). Named after Russian nesting dolls (see illustration), these clever constructs embed multiple scales of representation within a single vector. Unlike traditional post-processing methods, Matryoshka embeddings learn this multi-scale structure during the initial training process. The result is remarkable: **not only does the full embedding capture input semantics, but each nested subset prefix (first half, first quarter, etc.) provides a coherent, if less detailed, representation.**


![Figure: Visualization of Matryoshka embeddings with multiple layers of detail](https://assets.zilliz.com/Visualization_of_Matryoshka_embeddings_with_multiple_layers_of_detail_274f2c7aba.png)

_Figure: Visualization of Matryoshka embeddings with multiple layers of detail_

This approach contrasts sharply with conventional [embeddings](https://zilliz.com/glossary/vector-embeddings), where using arbitrary subsets of the vector dimensions typically destroys semantic meaning. With Matryoshka embeddings, you can choose the granularity that best balances your specific task's precision and computational cost. 

Need a quick approximate search? Use the smallest "doll." Need maximum accuracy? Use the full embedding. This flexibility makes them particularly valuable for systems adapting to different performance requirements or resource constraints.


## Inference

A valuable application of Matryoshka embeddings is accelerating similarity searches without sacrificing recall. By leveraging smaller subsets of query and database embeddings—such as the first 1/32 of their dimensions—we can build an index over this reduced space that still preserves much of the similarity information. Initial results from this smaller embedding space can be used directly. Still, there’s also a technique to boost recall and account for any minor reduction from the dimensional cutback, making this approach both efficient and effective for similarity search tasks.

![Figure: How the funnel search works with Matryoshka embeddings](https://assets.zilliz.com/How_the_funnel_search_works_with_Matryoshka_embeddings_8fa05a2fe7.png)

_Figure: How the funnel search works with Matryoshka embeddings_ 

To efficiently speed up similarity search while maintaining accuracy, we can use a "funnel search" approach. First, we perform an initial similarity search using only the first 1/32 of the embedding dimensions, generating a broad pool of candidate items. We then rerank these candidates based on their similarity to the query using the first 1/16 of the dimensions, pruning a portion of the list. This process continues iteratively, reranking and pruning with increasingly larger subsets of the embedding dimensions—1/8, 1/4, and so on. Importantly, we only perform one initial similarity search in this lower-dimensional space, and a single pass of the embedding model computes the query embedding. This funneling process narrows down candidates at each step and is faster and more efficient than directly searching in the full-dimensional space. Drawing many matches from the 1/32-dimensional space and refining them through funnel search can significantly accelerate similarity search while preserving strong recall.


## Training

Let’s go into a few of the technical details. The method is very simple to apply. Consider the context of fine-tuning a [BERT model](https://zilliz.com/learn/what-is-bert) for sentence embedding. To convert a BERT model, which has been pre-trained on the masked-token loss, into a sentence embedding model, we form the sentence embedding as the average of the final layer, that is, the average of the per-token contextualized embeddings. 

One choice of training objective is the [Cosine Sentence (CoSENT) loss](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#cosentloss), $L(u, v; s)$. It inputs a pair of sentence embeddings, $u,v$, and their desired similarity score, $s$ (see the link above for the formula). Now, to learn Matryoshka embeddings, we make a small modification to the training objective:

$L_M(u, v) = w_0L(u_{1:d}, v_{1:d}) + w_1L(u_{1:d/2}, v_{1:d/2}) + w_2L(u_{1:d/4}, v_{1:d/4}) + \cdots$

where the sum is continued by calculating the loss on half of the input to the previous term until an information bottleneck is reached. The authors suggest setting

$w_0=w_1=\cdots=1$.

_Simply put, the Matryoshka loss is a weighted sum of the original loss over recursive subsets of the input._

One key takeaway from the equation above is that the Matryoshka loss achieves efficient learning of representations at multiple scales by sharing weights across the embedding models (the same model is used to encode, for example, $u_{1:d}$ and $u_{1:d/2}$) and sharing dimensions across scales ($u_{1:d/2}$ is a subset of $u$).


## Matryoshka Embeddings and Milvus

Milvus seamlessly supports any Matryoshka embedding model that can be loaded via standard libraries such as [pymilvus.model](https://milvus.io/docs/embeddings.md), [sentence-transformers](https://milvus.io/docs/integrate_with_sentencetransformers.md), or other similar tools. From the system’s perspective, there’s no functional difference between a regular embedding model and one specifically trained to generate Matryoshka embeddings. 

Popular Matryoshka embedding models include: 

- OpenAI's [`text-embedding-3-large`](https://zilliz.com/ai-models/text-embedding-3-large)

- Nomic’s [`nomic-embed-text-v1`](https://huggingface.co/nomic-ai/nomic-embed-text-v1)

- Alibaba’s [`gte-multilingual-base`](https://huggingface.co/Alibaba-NLP/gte-multilingual-base)

For a complete guide on using Matryoshka embeddings with Milvus, see the notebook _[Funnel Search with Matryoshka Embeddings](https://github.com/milvus-io/bootcamp/blob/master/bootcamp/tutorials/quickstart/funnel_search_with_matryoshka.ipynb)_.


## Summary

Matryoshka embedding lets developers create shortened embeddings without sacrificing semantic integrity, making them ideal for more efficient search and storage. While you can modify an existing model, pre-trained options, such as those from [OpenAI](https://zilliz.com/ai-models) and [Hugging Face](https://zilliz.com/ai-models), are also available. 

However, a current limitation is the scarcity of open-source Matryoshka embeddings, with few available on the Hugging Face hub. Additionally, these models are often not explicitly labeled as “Matryoshka,” making them harder to locate. Hopefully, with growing interest, broader availability and clearer labeling may soon follow. 

Ready to streamline your search capabilities? Get started with Milvus + Matryoshka embeddings today!


## Resources

- Notebook: [Funnel Search with Matryoshka Embeddings](https://github.com/milvus-io/bootcamp/blob/master/bootcamp/tutorials/quickstart/funnel_search_with_matryoshka.ipynb) 

- Paper: [Matryoshka Representation Learning](https://arxiv.org/abs/2205.13147)

- Paper: [mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrieval](https://arxiv.org/pdf/2407.19669)

- [Introducing PyMilvus Integration with Embedding Models ](https://milvus.io/blog/introducing-pymilvus-integrations-with-embedding-models.md)

- [Exploring BGE-M3: The Future of Information Retrieval with Milvus ](https://zilliz.com/learn/Exploring-BGE-M3-the-future-of-information-retrieval-with-milvus)

- [Nomic Embed: Training a Reproducible Long Context Text Embedder](https://static.nomic.ai/reports/2024_Nomic_Embed_Text_Technical_Report.pdf)

- [Training Matryoshka Embeddings with Sentence Transformers Library](https://sbert.net/examples/training/matryoshka/README.html)

- [Milvus Bootcamp](https://milvus.io/bootcamp)



What are Matryoshka Embeddings?

Matryoshka Embeddings: Detail at Multiple Scales

How to Use the Milvus Backup Tool: A Step-by-Step Guide



[**Milvus**](https://zilliz.com/what-is-milvus) is an open-source [vector database](https://zilliz.com/learn/what-is-vector-database) designed to store, index, and search massive amounts of [unstructured data](https://zilliz.com/learn/introduction-to-unstructured-data) through vector representations, making it perfect for AI-driven applications, such as similarity search, [semantic search](https://zilliz.com/glossary/semantic-search), retrieval augmented generation ([RAG](https://zilliz.com/learn/Retrieval-Augmented-Generation)), recommendation engines, and other machine learning tasks. 

But what makes Milvus even more powerful is its seamless integration with Kubernetes. If you're a Kubernetes aficionado, you know the platform is perfect for orchestrating scalable, distributed systems. Milvus takes full advantage of Kubernetes' capabilities, allowing you to easily deploy, scale, and manage distributed Milvus clusters. This guide will provide a clear, step-by-step walkthrough for setting up Milvus on Kubernetes using the Milvus Operator. 


## Prerequisites

Before we begin, ensure you have the following prerequisites in place:

- A Kubernetes cluster up and running. If you're testing locally, `minikube` is a great choice.

- `kubectl` installed and configured to interact with your Kubernetes cluster.

- Familiarity with basic Kubernetes concepts like pods, services, and deployments.


## Step 1: Installing Minikube (For Local Testing)

If you need to set up a local Kubernetes environment, `minikube` is the tool for you. Official installation instructions are on the [minikube getting started page](https://minikube.sigs.k8s.io/docs/start/).


### 1. Install Minikube

Visit the[ minikube releases page](https://github.com/kubernetes/minikube/releases) and download the appropriate version for your operating system. For macOS/Linux, you can use the following command:

```
$ curl -Lo minikube https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64
$ sudo install minikube-linux-amd64 /usr/local/bin/minikube && rm minikube-linux-amd64
```


### 2. Start Minikube

```
$ minikube start
```


### 3. Interact with Cluster

Now, you can interact with your clusters with the kubectl inside minikube. If you haven’t installed kubectl, minikube will download the appropriate version by default.

```
$ minikube kubectl cluster-info
```

Alternatively, you can create a symbolic link to minikube’s binary named `kubectl` for easier usage.

```
$ sudo ln -s $(which minikube) /usr/local/bin/kubectl
$ kubectl cluster-info
```


## Step 2: Configuring the StorageClass

In Kubernetes, a **StorageClass** defines the types of storage available for your workloads, providing flexibility in managing different storage configurations. Before proceeding, you must ensure a default StorageClass is available in your cluster. Here’s how to check and configure one if necessary.


### 1. Check Installed StorageClasses

To see the available StorageClasses in your Kubernetes cluster, run the following command:

```
$ kubectl get sc
```

This will display the list of storage classes installed in your cluster. If a default StorageClass is already configured, it will be marked with `(default)`.

![](https://assets.zilliz.com/storage_classes_installed_in_your_cluster_21d36d6ac8.png)


### 2. Configure a Default StorageClass (if necessary)

If no default StorageClass is set, you can create one by defining it in a YAML file. Use the following example to create a default StorageClass:

```
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
 name: default-storageclass
provisioner: k8s.io/minikube-hostpath
```

This YAML configuration defines a `StorageClass` called `default-storageclass` that uses the `minikube-hostpath` provisioner, commonly used in local development environments.


### 3. Apply the StorageClass

Once the `default-storageclass.yaml` file is created, apply it to your cluster using the following command:

```
$ kubectl apply -f default-storageclass.yaml
```

This will set up the default StorageClass for your cluster, ensuring that your storage needs are properly managed in the future.


## Step 3: Installing Milvus Using the Milvus Operator

The Milvus Operator simplifies deploying Milvus on Kubernetes, managing the deployment, scaling, and updates. Before installing the Milvus Operator, you'll need to install the **cert-manager**, which provides certificates for the webhook server used by the Milvus Operator.


### 1. Install cert-manager

Milvus Operator requires a [cert-manager](https://cert-manager.io/docs/installation/supported-releases/) to manage certificates for secure communication. Make sure you install **cert-manager version 1.1.3** or later. To install it, run the following command:

```
$ kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.5.3/cert-manager.yaml
```

After the installation, verify that the cert-manager pods are running by executing:

```
$ kubectl get pods -n cert-manager
```


![](https://assets.zilliz.com/verify_that_the_cert_manager_pods_are_running_bb44c2b6d5.png)


### 2. Install the Milvus Operator

Once the cert-manager is up and running, you can install the Milvus Operator. Run the following command to deploy it using `kubectl`:

```
$ kubectl apply -f https://raw.githubusercontent.com/zilliztech/milvus-operator/main/deploy/manifests/deployment.yaml
```

You can check if the Milvus Operator pod is running using the following command: 

```
$ kubectl get pods -n milvus-operator
```


![](https://assets.zilliz.com/check_if_the_Milvus_Operator_pod_is_running_6e7ac41ebf.png)

### 3. Deploy Milvus Cluster

Once the Milvus Operator pod is running, you can deploy a Milvus cluster with the operator. The following command deploys a Milvus cluster with its components and dependencies in separate pods using default configurations:

```
$ kubectl apply -f https://raw.githubusercontent.com/zilliztech/milvus-operator/main/config/samples/milvus_cluster_default.yaml
```



![](https://assets.zilliz.com/Deploy_Milvus_Cluster_8b5d5343af.png)

To customize the Milvus settings, you will need to replace the YAML file with your own configuration YAML file. In addition to manually editing or creating the file, you can use the Milvus Sizing Tool to adjust the configurations and then download the corresponding YAML file.

Alternatively, you can use the [**Milvus Sizing Tool**](https://milvus.io/tools/sizing) for a more streamlined approach. This tool allows you to adjust various settings, such as resource allocation and storage options, and then download the corresponding YAML file with your desired configurations. This ensures that your Milvus deployment is optimized for your specific use case.

![](https://assets.zilliz.com/Figure_Milvus_sizing_tool_024693df9d.png)

Figure: Milvus sizing tool 

It may take some time to finish the deployment. You can check the status of your Milvus cluster via the command:

```
$ kubectl get milvus my-release
```


![](https://assets.zilliz.com/check_the_status_of_your_Milvus_cluster_bcbd85fd70.png)

Once your Milvus cluster is ready, all pods in the Milvus cluster should be running or completed:

```
$ kubectl get pods
```


## Step 4: Accessing Your Milvus Cluster

Once your Milvus cluster is deployed, you need to access it by forwarding a local port to the Milvus service port. Follow these steps to retrieve the service port and set up port forwarding.


#### **1. Get the Service Port**

First, identify the service port by using the following command. Replace `<YOUR_MILVUS_PROXY_POD>` with the name of your Milvus proxy pod, which typically starts with `my-release-milvus-proxy-`:

```
$ kubectl get pod <YOUR_MILVUS_PROXY_POD> --template ='{{(index (index .spec.containers 0).ports 0).containerPort}}{{"\n"}}'
```

This command will return the port number that your Milvus service is using.


#### **2. Forward the Port**

To access your Milvus cluster locally, forward a local port to the service port using the following command. Replace `<YOUR_LOCAL_PORT>` with the local port you want to use and `<YOUR_SERVICE_PORT>` with the service port retrieved in the previous step:

```
$ kubectl port-forward --address 0.0.0.0 service/my-release-milvus <YOUR_LOCAL_PORT>:<YOUR_SERVICE_PORT>
```

This command allows port-forwarding to listen on all IP addresses of the host machine. If you only need the service to listen on `localhost`, you can omit the `--address 0.0.0.0` option.

Once the port-forwarding is set up, you can access your Milvus cluster via the specified local port for further operations or integrations.


## Step 5: Connecting to Milvus Using Python SDK

With your Milvus cluster up and running, you can now interact with it using any Milvus SDK. In this example, we'll use [PyMilvus](https://zilliz.com/blog/what-is-pymilvus), Milvus’s **Python SDK,**  to connect to the cluster and perform basic operations.


### 1. Install PyMilvus

To interact with Milvus via Python, you need to install the `pymilvus` package:

```
$ pip install pymilvus
```


### 2. Connect to Milvus

The following is a sample Python script that connects to your Milvus cluster and demonstrates how to perform basic operations such as creating a collection.

```
from pymilvus import MilvusClient
```

```
# Connect to the Milvus server
client = MilvusClient(uri="http://localhost:<YOUR_LOCAL_PORT>")
```

```
# Create a collection
collection_name = "example_collection"
if client.has_collection(collection_name):
   client.drop_collection(collection_name)
client.create_collection(
   collection_name=collection_name,
   dimension=768,  # The vectors we will use in this demo has 768 dimensions
)
```


#### Explanation:

- Connect to Milvus: The script connects to the Milvus server running on `localhost` using the local port you set up in Step 4.

- Create a Collection: It checks if a collection named `example_collection` already exists, drops it if so, and then creates a new collection with vectors of 768 dimensions.

This script establishes a connection to the Milvus cluster and creates a collection, serving as a starting point for more complex operations like inserting vectors and performing similarity searches.


## Conclusion

Deploying Milvus in a distributed setup on Kubernetes unlocks powerful capabilities for managing large-scale vector data, enabling seamless scalability and high-performance AI-driven applications. Following this guide, you've learned how to set up Milvus using the Milvus Operator, making the process streamlined and efficient.

As you continue to explore Milvus, consider scaling your cluster to meet growing demands or deploying it on cloud platforms such as Amazon EKS, Google Cloud, or Microsoft Azure. For enhanced management and monitoring, tools like [**Milvus Backup**](https://milvus.io/docs/milvus_backup_overview.md), [**Birdwatcher**](https://milvus.io/docs/birdwatcher_overview.md), and [**Attu**](https://github.com/zilliztech/attu) offer valuable support for maintaining the health and performance of your deployments.

You're now ready to harness the full potential of Milvus on Kubernetes—happy deploying! 🚀


## Further Resources

- [Milvus Documentation](https://milvus.io/docs/overview.md)

- [Milvus Lite vs. Standalone vs. Distributed: Which Mode is Right for You? ](https://zilliz.com/blog/choose-the-right-milvus-deployment-mode-ai-applications)

- [Supercharging Vector Search: Milvus on GPUs with NVIDIA RAPIDS cuVS](https://zilliz.com/blog/milvus-on-gpu-with-nvidia-rapids-cuvs)

- [What is RAG? ](https://zilliz.com/learn/Retrieval-Augmented-Generation)

- [Generative AI Resource Hub | Zilliz](https://zilliz.com/learn/generative-ai)

- [Top Performing AI Models for Your GenAI Apps | Zilliz](https://zilliz.com/ai-models)


Deploying Milvus on Kubernetes: A Step-by-Step Guide for Kubernetes Users

How to Deploy the Open-Source Milvus Vector Database on Amazon EKS


[Milvus](https://milvus.io/intro) is an open-source vector database designed specifically for AI applications. Whether you're working on machine learning, deep learning, or any other AI-related project, Milvus offers a robust and efficient way to handle large-scale vector data.

Now, with the [model module integration](https://milvus.io/docs/embeddings.md) in PyMilvus, the Python SDK for Milvus, it's even easier to add Embedding and Reranking models. This integration simplifies transforming your data into searchable vectors or reranking results for more accurate outcomes, such as in [Retrieval Augmented Generation (RAG)](https://zilliz.com/learn/Retrieval-Augmented-Generation).

In this blog, we will review dense embedding models, sparse embedding models, and re-rankers and demonstrate how to use them in practice using [Milvus Lite](https://milvus.io/blog/introducing-milvus-lite.md), a lightweight version of Milvus that can run locally in your Python applications. 


## Dense vs Sparse Embeddings 

Before we walk you through how to use our integrations, let’s look at two main categories of vector embeddings. 

[Vector Embeddings](https://zilliz.com/glossary/vector-embeddings) generally fall into two main categories: [**Dense Embeddings** and **Sparse Embeddings**](https://zilliz.com/learn/sparse-and-dense-embeddings).

- Dense Embeddings are high-dimensional vectors in which most or all elements are non-zero, making them ideal for encoding text semantics or fuzzy meaning.

- Sparse Embeddings are high-dimensional vectors with many zero elements, better suited for encoding exact or adjacent concepts.

Milvus supports both types of embeddings and offers hybrid search. [Hybrid Search](https://zilliz.com/blog/hybrid-search-with-milvus) allows you to conduct searches across various vector fields within the same collection. These vectors can represent different facets of data, use diverse embedding models, or employ distinct data processing methods, combining the results using re-rankers.

## How to Use Our Embedding and Reranking Integrations

In the following sections, we’ll demonstrate three practical examples of using our integrations to generate embeddings and conduct vector searches. 

### Example 1: Use the Default Embedding Function to Generate Dense Vectors 

You must install the `pymilvus` client with the `model` package to use embedding and reranking functions with Milvus. 

```
pip install "pymilvus[model]"
```

This step will install [Milvus Lite](https://milvus.io/docs/quickstart.md), allowing you to run Milvus locally within your Python application. It also includes the model subpackage, which includes all utilities for Embedding and reranking.

The model subpackage supports various embedding models, including those from OpenAI, [Sentence Transformers](https://zilliz.com/learn/Sentence-Transformers-for-Long-Form-Text), [BGE-M3](https://zilliz.com/learn/bge-m3-and-splade-two-machine-learning-models-for-generating-sparse-embeddings), BM25, [SPLADE](https://zilliz.com/learn/bge-m3-and-splade-two-machine-learning-models-for-generating-sparse-embeddings), and Jina AI pre-trained models.

This example uses the `DefaultEmbeddingFunction`, based on the `all-MiniLM-L6-v2` Sentence Transformer model for simplicity. The model is about 70MB and will be downloaded during the first use:

```
from pymilvus import model

# This will download "all-MiniLM-L6-v2", a lightweight model.
ef = model.DefaultEmbeddingFunction()

# Data from which embeddings are to be generated
docs = [
   "Artificial intelligence was founded as an academic discipline in 1956.",
   "Alan Turing was the first person to conduct substantial research in AI.",
   "Born in Maida Vale, London, Turing was raised in southern England.",
]

embeddings = ef.encode_documents(docs)

print("Embeddings:", embeddings)
# Print dimension and shape of embeddings
print("Dim:", ef.dim, embeddings[0].shape)
```

The expected output should be something like the following: 

```
Embeddings: [array([-3.09392996e-02, -1.80662833e-02,  1.34775648e-02,  2.77156215e-02,
      -4.86349640e-03, -3.12581174e-02, -3.55921760e-02,  5.76934684e-03,
       2.80773244e-03,  1.35783911e-01,  3.59678417e-02,  6.17732145e-02,
...
      -4.61330153e-02, -4.85207550e-02,  3.13997865e-02,  7.82178566e-02,
      -4.75336798e-02,  5.21207601e-02,  9.04406682e-02, -5.36676683e-02],
     dtype=float32)]
Dim: 384 (384,)
```


### Example 2: Generate Sparse Vectors Using The BM25 Model

BM25 is a well-known method that uses word occurrence frequencies to determine the relevance between queries and documents. In this example, we’ll show how to use `BM25EmbeddingFunction` to generate sparse embeddings for queries and documents.

In BM25, it's important to calculate the statistics in your documents to obtain the IDF (Inverse Document Frequency), which can represent the patterns in your documents. The IDF measures how much information a word provides, whether it's common or rare across all documents.

```
from pymilvus.model.sparse import BM25EmbeddingFunction

# 1. Prepare a small corpus to search
docs = [
   "Artificial intelligence was founded as an academic discipline in 1956.",
   "Alan Turing was the first person to conduct substantial research in AI.",
   "Born in Maida Vale, London, Turing was raised in southern England.",
]
query = "Where was Turing born?"
bm25_ef = BM25EmbeddingFunction()

# 2. Fit the corpus to get BM25 model parameters on your documents.
bm25_ef.fit(docs)

# 3. Store the fitted parameters to expedite future processing.
bm25_ef.save("bm25_params.json")

# 4. Load the saved params
new_bm25_ef = BM25EmbeddingFunction()
new_bm25_ef.load("bm25_params.json")

docs_embeddings = new_bm25_ef.encode_documents(docs)
query_embeddings = new_bm25_ef.encode_queries([query])
print("Dim:", new_bm25_ef.dim, list(docs_embeddings)[0].shape)
```

### Example 3: Using a ReRanker 

A search system aims to find the most relevant results quickly and efficiently. Traditionally, methods like BM25 or TF-IDF have been used to rank search results based on keyword matching. Recent methods, such as embedding-based cosine similarity, are straightforward but can sometimes miss the subtleties of language and, most importantly, the interaction between documents and a query's intent.

This is where using a [re-ranker](https://zilliz.com/learn/optimize-rag-with-rerankers-the-role-and-tradeoffs) helps. A re-ranker is an advanced AI model that takes the initial set of results from a search—often provided by an embeddings/token-based search—and re-evaluates them to ensure they align more closely with the user's intent. It looks beyond the surface-level matching of terms to consider the deeper interaction between the search query and the content of the documents.

For this example, we’ll use the [Jina AI Reranker](https://milvus.io/docs/integrate_with_jina.md).



```
from pymilvus.model.reranker import JinaRerankFunction

jina_api_key = "<YOUR_JINA_API_KEY>"

rf = JinaRerankFunction("jina-reranker-v1-base-en", jina_api_key)

query = "What event in 1956 marked the official birth of artificial intelligence as a discipline?"

documents = [
   "In 1950, Alan Turing published his seminal paper, 'Computing Machinery and Intelligence,' proposing the Turing Test as a criterion of intelligence, a foundational concept in the philosophy and development of artificial intelligence.",
   "The Dartmouth Conference in 1956 is considered the birthplace of artificial intelligence as a field; here, John McCarthy and others coined the term 'artificial intelligence' and laid out its basic goals.",
   "In 1951, British mathematician and computer scientist Alan Turing also developed the first program designed to play chess, demonstrating an early example of AI in game strategy.",
   "The invention of the Logic Theorist by Allen Newell, Herbert A. Simon, and Cliff Shaw in 1955 marked the creation of the first true AI program, which was capable of solving logic problems, akin to proving mathematical theorems."
]

results = rf(query, documents)

for result in results:
   print(f"Index: {result.index}")
   print(f"Score: {result.score:.6f}")
   print(f"Text: {result.text}\n")
```

The expected output is similar to the following:

```
Index: 1
Score: 0.937096
Text: The Dartmouth Conference in 1956 is considered the birthplace of artificial intelligence as a field; here, John McCarthy and others coined the term 'artificial intelligence' and laid out its basic goals.

Index: 3
Score: 0.354210
Text: The invention of the Logic Theorist by Allen Newell, Herbert A. Simon, and Cliff Shaw in 1955 marked the creation of the first true AI program, which was capable of solving logic problems, akin to proving mathematical theorems.

Index: 0
Score: 0.349866
Text: In 1950, Alan Turing published his seminal paper, 'Computing Machinery and Intelligence,' proposing the Turing Test as a criterion of intelligence, a foundational concept in the philosophy and development of artificial intelligence.

Index: 2
Score: 0.272896
Text: In 1951, British mathematician and computer scientist Alan Turing also developed the first program designed to play chess, demonstrating an early example of AI in game strategy.
```

## Star Us On GitHub and Join Our Discord!

If you liked this blog post, consider starring Milvus on [GitHub](https://github.com/milvus-io/milvus), and feel free to join our [Discord](https://discord.gg/FG6hMJStWu)! 💙


Introducing PyMilvus Integration with Embedding Models


![](https://assets.zilliz.com/2_72e444c8dc.JPG)

We are excited to introduce [Milvus Lite](https://milvus.io/docs/milvus_lite.md), a lightweight vector database that runs locally within your Python application. Based on the popular open-source [Milvus](https://milvus.io/intro) vector database, Milvus Lite reuses the core components for vector indexing and query parsing while removing elements designed for high scalability in distributed systems. This design makes a compact and efficient solution ideal for environments with limited computing resources, such as laptops, Jupyter Notebooks, and mobile or edge devices.

Milvus Lite integrates with various AI development stacks like LangChain and LlamaIndex, enabling its use as a vector store in Retrieval Augmented Generation (RAG) pipelines without the need for server setup. Simply run `pip install pymilvus` (version 2.4.3 or above) to incorporate it into your AI application as a Python library.

Milvus Lite shares the Milvus API, ensuring that your client-side code works for both small-scale local deployments and Milvus servers deployed on Docker or Kubernetes with billions of vectors.

<iframe style="border-radius:12px" src="https://open.spotify.com/embed/episode/5bMcZgPgPVxSuoi1M2vn1p?utm_source=generator" width="100%" height="152" frameBorder="0" allowfullscreen="" allow="autoplay; clipboard-write; encrypted-media; fullscreen; picture-in-picture" loading="lazy"></iframe>

## Why We Built Milvus Lite

Many AI applications require vector similarity search for unstructured data, including text, images, voices, and videos, for applications such as chatbots and shopping assistants. Vector databases are crafted for storing and searching vector embeddings and are a crucial part of the AI development stack, particularly for generative AI use cases like [Retrieval Augmented Generation (RAG)](https://zilliz.com/learn/Retrieval-Augmented-Generation).

Despite the availability of numerous vector search solutions, an easy-to-start option that also works for large-scale production deployments was missing. As the creators of Milvus, we designed Milvus Lite to help AI developers build applications faster while ensuring a consistent experience across various deployment options, including Milvus on Kubernetes, Docker, and managed cloud services.

Milvus Lite is a crucial addition to our suite of offerings within the Milvus ecosystem. It provides developers with a versatile tool that supports every stage of their development journey. From prototyping to production environments and from edge computing to large-scale deployments, Milvus is now the only vector database that covers use cases of any size and all stages of development.

## How Milvus Lite Works 

Milvus Lite supports all the basic operations available in Milvus, such as creating collections and inserting, searching, and deleting vectors. It will soon support advanced features like hybrid search. Milvus Lite loads data into memory for efficient searches and persists it as an SQLite file.

Milvus Lite is included in the [Python SDK of Milvus](https://github.com/milvus-io/pymilvus) and can be deployed with a simple `pip install pymilvus`. The following code snippet demonstrates how to set up a vector database with Milvus Lite by specifying a local file name and then creating a new collection. For those familiar with the Milvus API, the only difference is that the `uri` refers to a local file name instead of a network endpoint, e.g., `"milvus_demo.db"` instead of `"http://localhost:19530"` for a Milvus server. Everything else remains the same. Milvus Lite also supports storing raw text and other labels as metadata, using a dynamic or explicitly defined schema, as shown below.

```
from pymilvus import MilvusClient

client = MilvusClient("milvus_demo.db")
# This collection can take input with mandatory fields named "id", "vector" and
# any other fields as "dynamic schema". You can also define the schema explicitly.
client.create_collection(
    collection_name="demo_collection",
    dimension=384  # Dimension for vectors.
)
```
For scalability, an AI application developed with Milvus Lite can easily transition to using Milvus deployed on Docker or Kubernetes by simply specifying the `uri` with the server endpoint.


## Integration with AI Development Stack

In addition to introducing Milvus Lite to make vector search easy to start with, Milvus also integrates with many frameworks and providers of the AI development stack, including [LangChain](https://python.langchain.com/v0.2/docs/integrations/vectorstores/milvus/), [LlamaIndex](https://docs.llamaindex.ai/en/stable/examples/vector_stores/MilvusIndexDemo/), [Haystack](https://haystack.deepset.ai/integrations/milvus-document-store), [Voyage AI](https://blog.voyageai.com/2024/05/30/semantic-search-with-milvus-lite-and-voyage-ai/), [Ragas](https://milvus.io/docs/integrate_with_ragas.md), [Jina AI](https://jina.ai/news/implementing-a-chat-history-rag-with-jina-ai-and-milvus-lite/), [DSPy](https://dspy-docs.vercel.app/docs/deep-dive/retrieval_models_clients/MilvusRM), [BentoML](https://www.bentoml.com/blog/building-a-rag-app-with-bentocloud-and-milvus-lite), [WhyHow](https://chiajy.medium.com/70873c7576f1), [Relari AI](https://blog.relari.ai/case-study-using-synthetic-data-to-benchmark-rag-systems-be324904ace1), [Airbyte](https://docs.airbyte.com/integrations/destinations/milvus), [HuggingFace](https://milvus.io/docs/integrate_with_hugging-face.md) and [MemGPT](https://memgpt.readme.io/docs/storage#milvus). Thanks to their extensive tooling and services, these integrations simplify the development of AI applications with vector search capability.

And this is just the beginning—many more exciting integrations are coming soon! Stay tuned! 


## More Resources and Examples

Explore [Milvus quickstart documentation](https://milvus.io/docs/quickstart.md) for detailed guides and code examples on using Milvus Lite to build AI applications like Retrieval-Augmented Generation ([RAG](https://github.com/milvus-io/bootcamp/blob/master/bootcamp/tutorials/quickstart/build_RAG_with_milvus.ipynb)) and [image search](https://github.com/milvus-io/bootcamp/blob/master/bootcamp/tutorials/quickstart/image_search_with_milvus.ipynb).

Milvus Lite is an open-source project, and we welcome your contributions. Check out our [Contributing Guide](https://github.com/milvus-io/milvus-lite/blob/main/CONTRIBUTING.md) to get started. You can also report bugs or request features by filing an issue on the [Milvus Lite GitHub](https://github.com/milvus-io/milvus-lite) repository.


Introducing Milvus Lite: Start Building a GenAI Application in Seconds


*This post was originally published on [Intel's Medium Channel](https://medium.com/intel-tech/optimize-vector-databases-enhance-rag-driven-generative-ai-90c10416cb9c) and is reposted here with permission.*


<br>

Two methods to optimize your vector database when using RAG


![](https://miro.medium.com/v2/resize:fit:1400/1*FRWBVwOHPYFDIVTp_ylZNQ.jpeg)


Photo by [Ilya Pavlov](https://unsplash.com/@ilyapavlov?utm_content=creditCopyText\&utm_medium=referral\&utm_source=unsplash) on [Unsplash](https://unsplash.com/photos/monitor-showing-java-programming-OqtafYT5kTw?utm_content=creditCopyText\&utm_medium=referral\&utm_source=unsplash)



By Cathy Zhang and Dr. Malini Bhandaru
Contributors: Lin Yang and Changyan Liu


Generative AI (GenAI) models, which are seeing exponential adoption in our daily lives, are being improved by [retrieval-augmented generation (RAG)](https://www.techtarget.com/searchenterpriseai/definition/retrieval-augmented-generation), a technique used to enhance response accuracy and reliability by fetching facts from external sources. RAG helps a regular [large language model (LLM)](https://www.techtarget.com/whatis/definition/large-language-model-LLM) understand context and reduce [hallucinations](https://en.wikipedia.org/wiki/Hallucination_$artificial_intelligence$) by leveraging a giant database of unstructured data stored as vectors — a mathematical presentation that helps capture context and relationships between data.


RAG helps to retrieve more contextual information and thus generate better responses, but the vector databases they rely on are getting ever larger to provide rich content to draw upon. Just as trillion-parameter LLMs are on the horizon, vector databases of billions of vectors are not far behind. As optimization engineers, we were curious to see if we could make vector databases more performant, load data faster, and create indices faster to ensure retrieval speed even as new data is added. Doing so would not only result in reduced user wait time, but also make RAG-based AI solutions a little more sustainable.


In this article, you’ll learn more about vector databases and their benchmarking frameworks, datasets to tackle different aspects, and the tools used for performance analysis — everything you need to start optimizing vector databases. We will also share our optimization achievements on two popular vector database solutions to inspire you on your optimization journey of performance and sustainability impact.




## Understanding Vector Databases


Unlike traditional relational or non-relational databases where data is stored in a structured manner, a vector database contains a mathematical representation of individual data items, called a vector, constructed using an embedding or transformation function. The vector commonly represents features or semantic meanings and can be short or long. Vector databases do vector retrieval by similarity search using a distance metric (where closer means the results are more similar) such as [Euclidean, dot product, or cosine similarity](https://www.pinecone.io/learn/vector-similarity/).


To accelerate the retrieval process, the vector data is organized using an indexing mechanism. Examples of these organization methods include flat structures, [inverted file (IVF),](https://arxiv.org/abs/2002.09094) [Hierarchical Navigable Small Worlds (HNSW),](https://arxiv.org/abs/1603.09320) and [locality-sensitive hashing (LSH)](https://en.wikipedia.org/wiki/Locality-sensitive_hashing), among others. Each of these methods contributes to the efficiency and effectiveness of retrieving similar vectors when needed.


Let’s examine how you would use a vector database in a GenAI system. Figure 1 illustrates both the loading of data into a vector database and using it in the context of a GenAI application. When you input your prompt, it undergoes a transformation process identical to the one used to generate vectors in the database. This transformed vector prompt is then used to retrieve similar vectors from the vector database. These retrieved items essentially serve as conversational memory, furnishing contextual history for prompts, akin to how LLMs operate. This feature proves particularly advantageous in natural language processing, computer vision, recommendation systems, and other domains requiring semantic comprehension and data matching. Your initial prompt is subsequently “merged” with the retrieved elements, supplying context, and assisting the LLM in formulating responses based on the provided context rather than solely relying on its original training data.


![](https://miro.medium.com/v2/resize:fit:1400/1*zQj_YJdWc2xKB6Vv89lzDQ.jpeg)


Figure 1. A RAG application architecture.


Vectors are stored and indexed for speedy retrieval. Vector databases come in two main flavors, traditional databases that have been extended to store vectors, and purpose-built vector databases. Some examples of traditional databases that provide vector support are [Redis](https://redis.io/), [pgvector](https://github.com/pgvector/pgvector), [Elasticsearch](https://www.elastic.co/elasticsearch), and [OpenSearch](https://opensearch.org/). Examples of purpose-built vector databases include proprietary solutions [Zilliz](https://zilliz.com/) and [Pinecone](https://www.pinecone.io/), and open source projects [Milvus](https://milvus.io/), [Weaviate](https://weaviate.io/), [Qdrant](https://qdrant.tech/), [Faiss](https://github.com/facebookresearch/faiss), and [Chroma](https://www.trychroma.com/). You can learn more about vector databases on GitHub via [LangChain ](https://github.com/langchain-ai/langchain/tree/master/libs/langchain/langchain/vectorstores)and [OpenAI Cookbook](https://github.com/openai/openai-cookbook/tree/main/examples/vector_databases).


We’ll take a closer look at one from each category, Milvus and Redis.




## Improving Performance


Before diving into the optimizations, let’s review how vector databases are evaluated, some evaluation frameworks, and available performance analysis tools.




### Performance Metrics


Let’s look at key metrics that can help you measure vector database performance.


- **Load latency** measures the time required to load data into the vector database’s memory and build an index. An index is a data structure used to efficiently organize and retrieve vector data based on its similarity or distance. Types of [in-memory indices](https://milvus.io/docs/index.md#In-memory-Index) include [flat index](https://thedataquarry.com/posts/vector-db-3/#flat-indexes), [IVF\_FLAT](https://supabase.com/docs/guides/ai/vector-indexes/ivf-indexes), [IVF\_PQ, HNSW](https://towardsdatascience.com/ivfpq-hnsw-for-billion-scale-similarity-search-89ff2f89d90e), [scalable nearest neighbors (ScaNN),](https://github.com/google-research/google-research/tree/master/scann)and [DiskANN](https://milvus.io/docs/disk_index.md).
- **Recall** is the proportion of true matches, or relevant items, found in the [Top K](https://redis.io/docs/data-types/probabilistic/top-k/) results retrieved by the search algorithm. Higher recall values indicate better retrieval of relevant items.
- **Queries per second (QPS)** is the rate at which the vector database can process incoming queries. Higher QPS values imply better query processing capability and system throughput.




### Benchmarking Frameworks


![](https://miro.medium.com/v2/resize:fit:920/1*mssEjZAuXg6nf-pad67rHA.jpeg)


Figure 2. The vector database benchmarking framework.


Benchmarking a vector database requires a vector database server and clients. In our performance tests, we used two popular open source tools.


- [**VectorDBBench**](https://github.com/zilliztech/VectorDBBench/tree/main)**:** Developed and open sourced by Zilliz, VectorDBBench helps test different vector databases with different index types and provides a convenient web interface.
- [**vector-db-benchmark**](https://github.com/qdrant/vector-db-benchmark/tree/master)**:** Developed and open sourced by Qdrant, vector-db-benchmark helps test several typical vector databases for the [HNSW](https://www.datastax.com/guides/hierarchical-navigable-small-worlds) index type. It runs tests through the command line and provides a [Docker Compose](https://docs.docker.com/compose/) __file to simplify starting server components.


![](https://miro.medium.com/v2/resize:fit:1400/1*NpHHEFV0TxRMse83hK6H1A.jpeg)


Figure 3. An example vector-db-benchmark command used to run the benchmark test.


But the benchmark framework is only part of the equation. We need data that exercises different aspects of the vector database solution itself, such as its ability to handle large volumes of data, various vector sizes, and speed of retrieval.With that, let’s look at some available public datasets.




### Open Datasets to Exercise Vector Databases


Large datasets are good candidates to test load latency and resource allocation. Some datasets have high dimensional data and are good for testing speed of computing similarity.


Datasets range from a dimension of 25 to a dimension of 2048. The [LAION](https://laion.ai/) dataset, an open image collection, has been used for training very large visual and language deep-neural models like stable diffusion generative models. OpenAI’s dataset of 5M vectors, each with a dimension of 1536, was created by VectorDBBench by running OpenAI on [raw data](https://huggingface.co/datasets/allenai/c4). Given each vector element is of type FLOAT, to save the vectors alone, approximately 29 GB (5M \* 1536 \* 4) of memory is needed, plus a similar amount extra to hold indices and other metadata for a total of 58 GB of memory for testing. When using the vector-db-benchmark tool, ensure adequate disk storage to save results.


To test for load latency, we needed a large collection of vectors, which [deep-image-96-angular](https://docs.hippo.transwarp.io/docs/performance-dataset) offers. To test performance of index generation and similarity computation, high dimensional vectors provide more stress. To this end we chose the 500K dataset of 1536 dimension vectors.




### Performance Tools


We’ve covered ways to stress the system to identify metrics of interest, but let’s examine what’s happening at a lower level: How busy is the computing unit, memory consumption, waits on locks, and more? These provide clues to databasebehavior, particularly useful in identifying problem areas.


The Linux [top](https://www.redhat.com/sysadmin/interpret-top-output) utility provides system-performance information. However, the [perf](https://perf.wiki.kernel.org/index.php/Main_Page) tool in Linux provides a deeper set of insights. To learn more, we also recommend reading [Linux perf examples](https://www.brendangregg.com/perf.html) and the [Intel top-down microarchitecture analysis method](https://www.intel.com/content/www/us/en/docs/vtune-profiler/cookbook/2023-0/top-down-microarchitecture-analysis-method.html). Yet another tool is the [Intel® vTune™ Profiler](https://www.intel.com/content/www/us/en/developer/tools/oneapi/vtune-profiler.html), which is useful when optimizing not just application but also system performance and configuration for a variety of workloads spanning HPC, cloud, IoT, media, storage, and more.




## Milvus Vector Database Optimizations


Let’s walk through some examples of how we attempted to improve the performance of the Milvus vector database.




### Reducing Memory Movement Overhead in Datanode Buffer Write


Milvus’s write path proxies write data into a log broker via _MsgStream_. The data nodes then consume the data, converting and storing it into segments. Segments will merge the newly inserted data. The merge logic allocates a new buffer to hold/move both the old data and the new data to be inserted and then returns the new buffer as old data for the next data merge. This results in the old data getting successively larger, which in turn makes data movement slower. Perf profiles showed a high overhead for this logic.


![](https://miro.medium.com/v2/resize:fit:1400/1*Az4dMVBcGmdeyKNrwpR19g.jpeg)


Figure 4. Merging and moving data in the vector database generates a high-performance overhead.


We changed the _merge buffer_ logic to directly append the new data to be inserted into the old data, avoiding allocating a new buffer and moving the large old data. Perf profiles confirm that there is no overhead to this logic. The microcode metrics _metric\_CPU operating frequency_ and _metric\_CPU utilization_ indicate an improvement that is consistent with the system not having to wait for the long memory movement anymore. Load latency improved by more than 60 percent. The improvement is captured on [GitHub](https://github.com/milvus-io/milvus/pull/26839).


![](https://miro.medium.com/v2/resize:fit:1400/1*MmaUtBTdqmMvC5MlQ8V0wQ.jpeg)


Figure 5. With less copying we see a performance improvement of more than 50 percent in load latency.




### Inverted Index Building with Reduced Memory Allocation Overhead


The Milvus search engine, [Knowhere](https://milvus.io/docs/knowhere.md), employs the [Elkan k-means algorithm](https://www.vlfeat.org/api/kmeans-fundamentals.html#kmeans-elkan) to train cluster data for creating [inverted file (IVF) indices](https://milvus.io/docs/v1.1.1/index.md). Each round of data training defines an iteration count. The larger the count, the better the training results. However, it also implies that the Elkan algorithm will be called more frequently.


The Elkan algorithm handles memory allocation and deallocation each time it’s executed. Specifically, it allocates memory to store half the size of symmetric matrix data, excluding the diagonal elements. In Knowhere, the symmetric matrix dimension used by the Elkan algorithm is set to 1024, resulting in a memory size of approximately 2 MB. This means for each training round Elkan repeatedly allocates and deallocates 2 MB memory.


Perf profiling data indicated frequent large memory allocation activity. In fact, it triggered [virtual memory area (VMA)](https://www.oreilly.com/library/view/linux-device-drivers/9781785280009/4759692f-43fb-4066-86b2-76a90f0707a2.xhtml)allocation, physical page allocation, page map setup, and updating of memory cgroup statistics in the kernel. This pattern of large memory allocation/deallocation activity can, in some situations, also aggravate memory fragmentation. This is a significant tax.


The _IndexFlatElkan_ structure is specifically designed and constructed to support the Elkan algorithm. Each data training process will have an _IndexFlatElkan_ instance initialized. To mitigate the performance impact resulting from frequent memory allocation and deallocation in the Elkan algorithm, we refactored the code logic, moving the memory management outside of the Elkan algorithm function up into the construction process of _IndexFlatElkan_. This enables memory allocation to occur only once during the initialization phase while serving all subsequent Elkan algorithm function calls from the current data training process and helps to improve load latency by around 3 percent. Find the [Knowhere patch here](https://github.com/zilliztech/knowhere/pull/280).




## Redis Vector Search Acceleration through Software Prefetch


Redis, a popular traditional in-memory key-value data store, recently began supporting vector search. To go beyond a typical key-value store, it offers extensibility modules; the [RediSearch](https://github.com/RediSearch/RediSearch) module facilitates the storage and search of vectors directly within Redis.


For vector similarity search, Redis supports two algorithms, namely brute force and HNSW. The HNSW algorithm is specifically crafted for efficiently locating approximate nearest neighbors in high-dimensional spaces. It uses a priority queue named _candidate\_set_ to manage all vector candidates for distance computing.


Each vector candidate encompasses substantial metadata in addition to the vector data. As a result, when loading a candidate from memory it can cause data cache misses, which incur processing delays. Our optimization introduces software prefetching to proactively load the next candidate while processing the current one. This enhancement has resulted in a 2 to 3 percent throughput improvement for vector similarity searches in a single instance Redis setup. The patch is in the process of being upstreamed.




## GCC Default Behavior Change to Prevent Mixed Assembly Code Penalties


To drive maximum performance, frequently used sections of code are often handwritten in assembly. However, when different segments of code are written either by different people or at different points in time, the instructions used may come from incompatible assembly instruction sets such as [Intel® Advanced Vector Extensions 512 (Intel® AVX-512)](https://www.intel.com/content/www/us/en/architecture-and-technology/avx-512-overview.html) and [Streaming SIMD Extensions (SSE)](https://en.wikipedia.org/wiki/Streaming_SIMD_Extensions). If not compiled appropriately, the mixed code results in a performance penalty. [Learn more about mixing Intel AVX and SSE instructions here](https://www.intel.com/content/dam/develop/external/us/en/documents/11mc12-avoiding-2bavx-sse-2btransition-2bpenalties-2brh-2bfinal-809104.pdf).


You can easily determine if you’re using mixed-mode assembly code and have not compiled the code with _VZEROUPPER_, incurring the performance penalty. It can be observed through a perf command like _sudo perf stat -e ‘assists.sse\_avx\_mix/event/event=0xc1,umask=0x10/’ \<workload>_. If your OS doesn’t have support for the event, use _cpu/event=0xc1,umask=0x10,name=assists\_sse\_avx\_mix/_.


The Clang compiler by default inserts _VZEROUPPER_, avoiding any mixed mode penalty. But the GCC compiler only inserted _VZEROUPPER_ when the -O2 or -O3 compiler flags were specified. We contacted the GCC team and explained the issue and they now, by default, correctly handle mixed mode assembly code.



## Start Optimizing Your Vector Databases


Vector databases are playing an integral role in GenAI, and they are growing ever larger to generate higher-quality responses. With respect to optimization, AI applications are no different from other software applications in that they reveal their secrets when one employs standard performance analysis tools along with benchmark frameworks and stress input.

Using these tools, we uncovered performance traps pertaining to unnecessary memory allocation, failing to prefetch instructions, and using incorrect compiler options. Based on our findings, we upstreamed enhancements to Milvus, Knowhere, Redis, and the GCC compiler to help make AI a little more performant and sustainable. Vector databases are an important class of applications worthy of your optimization efforts. We hope this article helps you get started.


Optimize Vector Databases, Enhance RAG-Driven Generative AI



## Introduction

Milvus is a distributed vector database that aims to store, index and manage massive embedding vectors. Its ability to efficiently index and search through trillions of vectors makes Milvus a go-to choice for AI and machine learning workloads.

Kubernetes (K8s), on the other hand, excels in managing and scaling containerized applications. It provides features like auto-scaling, self-healing, and load balancing, which are crucial for maintaining high availability and performance in production environments.


## Why Use Them Together? 

K8s can automatically scale the Milvus clusters based on the workload. As your data grows or the number of queries increases, K8s can spin up more Milvus instances to handle the load, ensuring your applications remain responsive.

One of the standout features of K8s is its horizontal scaling, which makes expanding your Milvus cluster a breeze. As your dataset grows, K8s effortlessly accommodates this growth, making it a straightforward and efficient solution.

In addition, the ability to handle queries also scales horizontally with K8s. As the query load increases, K8s can deploy more Milvus instances to handle the increased similarity search queries, ensuring low latency responses even under heavy loads.


## Prerequisites & Setting Up K8s

### Prerequisites

- **Docker** - Ensure Docker is installed on your system.

- **Kubernetes** - Have a Kubernetes cluster ready. You can use `minikube` for local development or a cloud provider's Kubernetes service for production environments.

- **Helm** - Install Helm, a package manager for Kubernetes, to help you manage Kubernetes applications, you can check our documentation to see how to do that <https://milvus.io/docs/install_cluster-helm.md>

- **Kubectl** - Install `kubectl`, a command-line tool for interacting with Kubernetes clusters, to deploy applications, inspect and manage cluster resources, and view logs.

### Setting Up K8s

After installing everything needed to run a K8s cluster, and if you used `minikube`, start your cluster with: 

```
minikube start
```

Check the status of your K8s cluster with:

```
kubectl cluster-info
```

### Deploying Milvus on K8s

For this deployment, we're opting for Milvus in cluster-mode to leverage its full distributed capabilities. We'll be using Helm, to streamline the installation process.

**1. Helm Installation Command** 

```
helm install my-milvus milvus/milvus --set pulsar.enabled=false --set kafka.enabled=true
```

This command installs Milvus on your K8s cluster with Kafka enabled and Pulsar disabled. Kafka serves as the messaging system within Milvus, handling data streaming between different components. Disabling Pulsar and enabling Kafka tailors the deployment to our specific messaging preferences and requirements.

**2. Port Forwarding**

To access Milvus from your local machine, create a port forward: `kubectl port-forward svc/my-milvus 27017:19530`.

This command maps port `19530` from the Milvus service `svc/my-milvus` to the same port on your local machine, allowing you to connect to Milvus using local tools. If you leave the local port unspecified (as in `:19530`), K8s will allocate an available port, making it dynamic. Ensure you note the allocated local port if you choose this method.

**3. Verifying the Deployment:**

```
kubectl get pods 

NAME                                    READY   STATUS    RESTARTS   AGE
my-milvus-datacoord-595b996bd4-zprpd    1/1     Running   0          85m
my-milvus-datanode-d9d555785-47nkt      1/1     Running   0          85m
my-milvus-etcd-0                        1/1     Running   0          84m
my-milvus-etcd-1                        1/1     Running   0          85m
my-milvus-etcd-2                        1/1     Running   0          85m
my-milvus-indexcoord-65bc68968c-6jg6q   1/1     Running   0          85m
my-milvus-indexnode-54586f55d-z9vx4     1/1     Running   0          85m
my-milvus-kafka-0                       1/1     Running   0          85m
my-milvus-kafka-1                       1/1     Running   0          85m
my-milvus-kafka-2                       1/1     Running   0          85m
my-milvus-minio-0                       1/1     Running   0          96m
my-milvus-minio-1                       1/1     Running   0          96m
my-milvus-minio-2                       1/1     Running   0          96m
my-milvus-minio-3                       1/1     Running   0          96m
my-milvus-proxy-76bb7d497f-sqwvd        1/1     Running   0          85m
my-milvus-querycoord-6f4c7b7598-b6twj   1/1     Running   0          85m
my-milvus-querynode-677bdf485b-ktc6m    1/1     Running   0          85m
my-milvus-rootcoord-7498fddfd8-v5zw8    1/1     Running   0          85m
my-milvus-zookeeper-0                   1/1     Running   0          85m
my-milvus-zookeeper-1                   1/1     Running   0          85m
my-milvus-zookeeper-2                   1/1     Running   0          85m
```


You should see a list of pods similar to the output above, all in the Running state. This indicates that your Milvus cluster is operational. Specifically, look for the 1/1 under the `READY` column, which signifies that each pod is fully ready and running. If any pods are not in the Running state, you may need to investigate further to ensure a successful deployment.

With your Milvus cluster deployed and all components confirmed running, you're now ready to proceed to data ingestion and indexing. This will involve connecting to your Milvus instance, creating collections, and inserting vectors for search and retrieval.


## Data Ingestion and Indexing

To start ingesting and indexing data in our Milvus cluster, we'll use the pymilvus SDK. There are two installation options:

- Basic SDK: `pip install pymilvus`

- For rich text embeddings and advanced models: `pip install pymilvus[model]`

Time to insert data in our cluster, we’ll be using `pymilvus`, you can either install the SDK only with `pip install pymilvus` or if you want to extract rich text embeddings, you can also use `PyMilvus Models` by installing `pip install pymilvus[model]`. 

### Connecting and Creating a Collection:

First, connect to your Milvus instance using the port you forwarded earlier. Ensure the URI matches the local port assigned by K8s:

```
from pymilvus import MilvusClient

client = MilvusClient(
        uri="http://127.0.0.1:52070",
    )

client.create_collection(collection_name="quick_setup", dimension=5)
```

The `dimension=5` parameter defines the vector size for this collection, essential for the vector search capabilities.


### Insert Data

Here's how to insert an initial set of data, where each vector represents an item, and the color field adds a descriptive attribute:

```
data=[
    {"id": 0, "vector": [0.3580376395471989, -0.6023495712049978, 0.18414012509913835, -0.26286205330961354, 0.9029438446296592], "color": "pink_8682"},
    {"id": 1, "vector": [0.19886812562848388, 0.06023560599112088, 0.6976963061752597, 0.2614474506242501, 0.838729485096104], "color": "red_7025"},
    {"id": 2, "vector": [0.43742130801983836, -0.5597502546264526, 0.6457887650909682, 0.7894058910881185, 0.20785793220625592], "color": "orange_6781"},
    {"id": 3, "vector": [0.3172005263489739, 0.9719044792798428, -0.36981146090600725, -0.4860894583077995, 0.95791889146345], "color": "pink_9298"},
    {"id": 4, "vector": [0.4452349528804562, -0.8757026943054742, 0.8220779437047674, 0.46406290649483184, 0.30337481143159106], "color": "red_4794"},
    {"id": 5, "vector": [0.985825131989184, -0.8144651566660419, 0.6299267002202009, 0.1206906911183383, -0.1446277761879955], "color": "yellow_4222"},
    {"id": 6, "vector": [0.8371977790571115, -0.015764369584852833, -0.31062937026679327, -0.562666951622192, -0.8984947637863987], "color": "red_9392"},
    {"id": 7, "vector": [-0.33445148015177995, -0.2567135004164067, 0.8987539745369246, 0.9402995886420709, 0.5378064918413052], "color": "grey_8510"},
    {"id": 8, "vector": [0.39524717779832685, 0.4000257286739164, -0.5890507376891594, -0.8650502298996872, -0.6140360785406336], "color": "white_9381"},
    {"id": 9, "vector": [0.5718280481994695, 0.24070317428066512, -0.3737913482606834, -0.06726932177492717, -0.6980531615588608], "color": "purple_4976"}
]

res = client.insert(
    collection_name="quick_setup",
    data=data
)

print(res)
```


The provided code assumes that you have created a collection in the Quick Setup manner. As shown in the above code,

The data to insert is organized into a list of dictionaries, where each dictionary represents a data record, termed as an entity.

Each dictionary contains a non-schema-defined field named color.

Each dictionary contains the keys corresponding to both pre-defined and dynamic fields.


### Insert Even More Data

```
colors = ["green", "blue", "yellow", "red", "black", "white", "purple", "pink", "orange", "brown", "grey"]
data = [ {
    "id": i, 
    "vector": [ random.uniform(-1, 1) for _ in range(5) ], 
    "color": f"{random.choice(colors)}_{str(random.randint(1000, 9999))}" 
} for i in range(1000) ]

res = client.insert(
    collection_name="quick_setup",
    data=data[10:]
)

print(res)
```


## Similarity Search

After populating the collection, you can perform a similarity search to find vectors close to a query vector. The value of the query_vectors variable is a list containing a sub-list of floats. The sub-list represents a vector embedding of 5 dimensions.

```
query_vectors = [
    [0.041732933, 0.013779674, -0.027564144, -0.013061441, 0.009748648]
]

res = client.search(
    collection_name="quick_setup",     # target collection
    data=query_vectors,                # query vectors
    limit=3,                           # number of returned entities
)

print(res)
```


This query searches for the top 3 vectors most similar to our query vector, demonstrating Milvus's powerful search capabilities.


## Uninstall Milvus from K8s

Once you are done with this tutorial, feel free to uninstall Milvus from your K8s cluster with:`helm uninstall my-milvus`.

This command will remove all Milvus components deployed in the `my-milvus` release, freeing up cluster resources.

## Conclusion

- Deploying Milvus on a Kubernetes cluster showcases the scalability and flexibility of vector databases in handling AI and machine learning workloads. Through this tutorial, you've learned the basics of setting up Milvus with Helm, creating a collection, and performing data ingestion and similarity searches.

- Installing Milvus on a Kubernetes cluster with Helm should be straightforward. To go deeper into scaling Milvus clusters for larger datasets or more intensive workloads, our documentation offers detailed guidance <https://milvus.io/docs/scaleout.md>

Feel free to check out the code on [Github](https://github.com/stephen37/K8s-tutorial-milvus), check out [Milvus](https://github.com/milvus-io/milvus), experiment with different configurations and use cases, and share your experiences with the community by joining our [Discord](https://discord.gg/FG6hMJStWu).


Getting started with Milvus cluster and K8s


We are happy to announce the launch of Milvus 2.4, a major advancement in enhancing search capabilities for large-scale datasets. This latest release adds new features, such as support for the GPU-based CAGRA index, beta support for [sparse embeddings](https://zilliz.com/learn/sparse-and-dense-embeddings), group search, and various other improvements in search capabilities. These developments reinforce our commitment to the community by offering developers like you a powerful and efficient tool for handling and querying vector data. Let's jump into the key benefits of Milvus 2.4 together.


## Enabled Multi-vector Search for Simplified Multimodal Searches

Milvus 2.4 provides multivector search capability, allowing simultaneous search and reranking of different vector types within the same Milvus system. This feature streamlines multimodal searches, significantly enhancing recall rates and enabling developers to effortlessly manage intricate AI applications with varied data types. Additionally, this functionality simplifies the integration and fine-tuning of custom reranking models, aiding in the creation of advanced search functions like precise [recommender systems](https://zilliz.com/vector-database-use-cases/recommender-system) that utilize insights from multidimensional data.

![How the Milti-Vector Search Feature Works](https://assets.zilliz.com/How_the_multi_vector_search_feature_works_6c85961349.png)

Multivector support in Milvus has two components: 

1. The ability to store/query multiple vectors for a single entity within a collection, which is a more natural way to organize data 

2. The ability to build/optimize a reranking algorithm by leveraging the prebuilt reranking algorithms in Milvus

Besides being a highly [requested feature](https://github.com/milvus-io/milvus/issues/25639), we built this capability because the industry is moving towards multimodal models with the release of GPT-4 and Claude 3. Reranking is a commonly used technique to further improve query performance in search. We aimed to make it easy for developers to build and optimize their rerankers within the Milvus ecosystem.


## Grouping Search Support for Enhanced Compute Efficiency

Grouping Search is another often [requested feature](https://github.com/milvus-io/milvus/issues/25343) we added to Milvus 2.4. It integrates a group-by operation designed for fields of types BOOL, INT, or VARCHAR, filling a crucial efficiency gap in executing large-scale grouping queries. 

Traditionally, developers relied on extensive Top-K searches followed by manual post-processing to distill group-specific results, a compute-intensive and code-heavy method. Grouping Search refines this process by efficiently linking query outcomes to aggregate group identifiers like document or video names, streamlining the handling of segmented entities within larger datasets.

Milvus distinguishes its Grouping Search with an iterator-based implementation, offering a marked improvement in computational efficiency over similar technologies. This choice ensures superior performance scalability, particularly in production environments where compute resource optimization is paramount. By reducing data traversal and computation overhead, Milvus supports more efficient query processing, significantly reducing response times and operational costs compared to other vector databases. 

Grouping Search bolsters Milvus's capability to manage high-volume, complex queries and aligns with high-performance computing practices for robust data management solutions.

## Beta Support for Sparse Vector Embeddings

[Sparse embeddings](https://zilliz.com/learn/sparse-and-dense-embeddings) represent a paradigm shift from traditional dense vector approaches, catering to the nuances of semantic similarity rather than mere keyword frequency. This distinction allows for a more nuanced search capability, aligning closely with the semantic content of the query and the documents. Sparse vector models, particularly useful in information retrieval and natural language processing, offer powerful out-of-domain search capabilities and interpretability compared to their dense counterparts.

In Milvus 2.4, we have expanded the Hybrid Search to include sparse embeddings generated by advanced neural models like SPLADEv2 or statistical models such as BM25. In Milvus, sparse vectors are treated on par with dense vectors, enabling the creation of collections with sparse vector fields, data insertion, index building, and performing similarity searches. Notably, sparse embeddings in Milvus support the [Inner Product](https://zilliz.com/blog/similarity-metrics-for-vector-search#Inner-Product) (IP) distance metric, which is advantageous given their high-dimensional nature, making other metrics less effective. This functionality also supports data types with a dimension as an unsigned 32-bit integer and a 32-bit float for the value, thus facilitating a broad spectrum of applications, from nuanced text searches to elaborate [information retrieval](https://zilliz.com/learn/information-retrieval-metrics) systems.

With this new feature, Milvus allows for hybrid search methodologies that meld keyword and embedding-based techniques, offering a seamless transition for users moving from keyword-centric search frameworks seeking a comprehensive, low-maintenance solution.

We are labeling this feature as “Beta” to continue our performance testing of the feature and gather feedback from the community. The general availability (GA) of sparse vector support is anticipated with the release of Milvus 3.0.


## CAGRA Index Support for Advanced GPU-Accelerated Graph Indexing

Developed by NVIDIA, [CAGRA](https://arxiv.org/abs/2308.15136) (Cuda Anns GRAph-based) is a GPU-based graph indexing technology that significantly surpasses traditional CPU-based methods like the HNSW index in efficiency and performance, especially in high-throughput environments.

With the introduction of the CAGRA Index, Milvus 2.4 provides enhanced GPU-accelerated graph indexing capability. This enhancement is ideal for building similarity search applications requiring minimal latency. Additionally, Milvus 2.4 integrates a brute-force search with the CAGRA index to achieve maximum recall rates in applications. For detailed insights, explore the [introduction blog on CAGRA](https://zilliz.com/blog/Milvus-introduces-GPU-index-CAGRA).

![Milvus Raft CAGRA vs. Milvus HNSW](https://assets.zilliz.com/Milvus_raft_cagra_vs_milvus_hnsw_ffe0415ff5.png)

## Additional Enhancements and Features

Milvus 2.4 also includes other key enhancements, such as Regular Expression support for enhanced substring matching in [metadata filtering](https://zilliz.com/blog/metadata-filtering-with-zilliz-cloud-pipelines), a new scalar inverted index for efficient scalar data type filtering, and a Change Data Capture tool for monitoring and replicating changes in Milvus collections. These updates collectively enhance Milvus's performance and versatility, making it a comprehensive solution for complex data operations.

For more details, see [Milvus 2.4 documentation](https://milvus.io/docs/release_notes.md). 


## Stay Connected!

Excited to learn more about Milvus 2.4? [Join our upcoming webinar](https://zilliz.com/event/unlocking-advanced-search-capabilities-milvus) with James Luan, Zilliz’s VP of Engineering, for an in-depth discussion on the capabilities of this latest release. If you have questions or feedback, join our [Discord channel](https://discord.com/invite/8uyFbECzPX) to engage with our engineers and community members. Don’t forget to follow us on [Twitter](https://twitter.com/milvusio) or [LinkedIn](https://www.linkedin.com/company/the-milvus-project) for the latest news and updates about Milvus.


Milvus 2.4 Supports Multi-vector Search, Sparse Vector, CAGRA, and More!

Unveiling Milvus 2.4: Multi-vector Search, Sparse Vector, CAGRA Index, and More!


[Milvus](https://zilliz.com/what-is-milvus), the highly scalable vector database renowned for its ability to handle vast datasets, takes a significant step forward by introducing Parquet file support in [version 2.3.4](https://zilliz.com/blog/what-is-new-in-milvus-2-3-4). By embracing Apache Parquet, users can streamline their data import processes and enjoy substantial savings in storage and computation costs.

In our latest post, we explore Parquet's advantages and the benefits it brings to Milvus users. We discuss the motivation behind integrating this feature and provide a step-by-step guide on seamlessly importing Parquet files into Milvus, unlocking new possibilities for efficient data management and analysis.

## What Is Apache Parquet? 

[Apache Parquet](https://parquet.apache.org/) is a popular open-source column-oriented data file format designed to enhance the efficiency of storing and processing large-scale datasets. In contrast to traditional row-oriented data formats like CSV or JSON, Parquet stores data by column, offering more efficient data compression and encoding schemes. This approach translates to improved performance, reduced storage requirements, and enhanced processing power, making it ideal for handling complex data in bulk. 

## How Milvus Users Benefit from the Support for Parquet File Imports

Milvus extends support for Parquet file imports, providing users with optimized experiences and various advantages, including lowered storage and computation expenses, streamlined data management, and a simplified importing process.

### Optimized Storage Efficiency and Streamlined Data Management 

Parquet provides flexible compression options and efficient encoding schemes catering to different data types, ensuring optimal storage efficiency. This flexibility is particularly valuable in cloud environments where every ounce of storage savings directly correlates to tangible cost reductions. With this new feature in Milvus, users can effortlessly consolidate all their diverse data into a single file, streamlining data management and enhancing the overall user experience. This feature is particularly beneficial for users working with variable-length Array data types, who can now enjoy a simplified data import process. 

### Improved Query Performance 

Parquet's columnar storage design and advanced compression methods significantly enhance query performance. When conducting queries, users can focus solely on the pertinent data without scanning through the irrelevant data. This selective column reading minimizes CPU usage, resulting in faster query times. 

### Broad Language Compatibility

Parquet is available in multiple languages such as Java, C++, and Python and is compatible with a large number of data processing tools. With the support of Parquet files, Milvus users using different SDKs can seamlessly generate Parquet files for parsing within the database. 

## How to Import Parquet Files into Milvus 

If your data is already in Parquet file format, importing is easy. Upload the Parquet file to an object storage system such as MinIO, and you're ready to import. 

The code snippet below is an example of importing Parquet files into Milvus.


```
remote_files = []
try:
    print("Prepare upload files")
    minio_client = Minio(endpoint=MINIO_ADDRESS, access_key=MINIO_ACCESS_KEY, secret_key=MINIO_SECRET_KEY,
                         secure=False)
    found = minio_client.bucket_exists(bucket_name)
    if not found:
        minio_client.make_bucket(bucket_name)
        print("MinIO bucket '{}' doesn't exist".format(bucket_name))
        return False, []

    # set your remote data path
    remote_data_path = "milvus_bulkinsert"

    def upload_file(f: str):
        file_name = os.path.basename(f)
        minio_file_path = os.path.join(remote_data_path, "parquet", file_name)
        minio_client.fput_object(bucket_name, minio_file_path, f)
        print("Upload file '{}' to '{}'".format(f, minio_file_path))
        remote_files.append(minio_file_path)

    upload_file(data_file)

except S3Error as e:
    print("Failed to connect MinIO server {}, error: {}".format(MINIO_ADDRESS, e))
    return False, []

print("Successfully upload files: {}".format(remote_files))
return True, remote_files
```

If your data is not Parquet files or has dynamic fields, you can leverage BulkWriter, our data format conversion tool, to help you generate Parquet files. BulkWriter has now embraced Parquet as its default output data format, ensuring a more intuitive experience for developers. 

The code snippet below is an example of using BulkWriter to generate Parquet files. 

```
import numpy as np
import json

from pymilvus import (
    RemoteBulkWriter,
    BulkFileType,
)

remote_writer = RemoteBulkWriter(
        schema=your_collection_schema,
        remote_path="your_remote_data_path",
        connect_param=RemoteBulkWriter.ConnectParam(
            endpoint=YOUR_MINIO_ADDRESS,
            access_key=YOUR_MINIO_ACCESS_KEY,
            secret_key=YOUR_MINIO_SECRET_KEY,
            bucket_name="a-bucket",
        ),
        file_type=BulkFileType.PARQUET,
)

# append your data
batch_count = 10000
for i in range(batch_count):
    row = {
        "id": i,
        "bool": True if i % 5 == 0 else False,
        "int8": i % 128,
        "int16": i % 1000,
        "int32": i % 100000,
        "int64": i,
        "float": i / 3,
        "double": i / 7,
        "varchar": f"varchar_{i}",
        "json": {"dummy": i, "ok": f"name_{i}"},
        "vector": gen_binary_vector() if bin_vec else gen_float_vector(),
        f"dynamic_{i}": i,
    }
    remote_writer.append_row(row)

# append rows by numpy type
for i in range(batch_count):
    remote_writer.append_row({
        "id": np.int64(i + batch_count),
        "bool": True if i % 3 == 0 else False,
        "int8": np.int8(i % 128),
        "int16": np.int16(i % 1000),
        "int32": np.int32(i % 100000),
        "int64": np.int64(i),
        "float": np.float32(i / 3),
        "double": np.float64(i / 7),
        "varchar": f"varchar_{i}",
        "json": json.dumps({"dummy": i, "ok": f"name_{i}"}),
        "vector": gen_binary_vector() if bin_vec else gen_float_vector(),
        f"dynamic_{i}": i,
    })

print(f"{remote_writer.total_row_count} rows appends")
print(f"{remote_writer.buffer_row_count} rows in buffer not flushed")
print("Generate data files...")
remote_writer.commit()
print(f"Data files have been uploaded: {remote_writer.batch_files}")
remote_files = remote_writer.batch_files
```

Then, you can start to import your Parquet files into Milvus. 

```
remote_files = [remote_file_path]
task_id = utility.do_bulk_insert(collection_name=collection_name,
                                 files=remote_files)

task_ids = [task_id]         
states = wait_tasks_to_state(task_ids, BulkInsertState.ImportCompleted)
complete_count = 0
for state in states:
    if state.state == BulkInsertState.ImportCompleted:
        complete_count = complete_count + 1
 ```

Now, your data is seamlessly integrated into Milvus.

## What's Next?

As Milvus continues to support ever-growing data volumes, the challenge arises in managing sizable imports, particularly when Parquet files surpass 10GB. To tackle this challenge, we plan to segregate the import data into scalar and vector columns, creating two Parquet files per import to alleviate the I/O pressure. For datasets exceeding several hundred gigabytes, we recommend importing the data multiple times. 



Milvus Supports Imports of Apache Parquet Files

Milvus Supports Imports of Apache Parquet Files for Enhanced Data Processing Efficiency


# Introduction to the Milvus versions

Selecting the appropriate Milvus version is foremost to the success of any project leveraging vector search technology. With different Milvus versions tailored to varying requirements, understanding the importance of selecting the correct version is crucial for achieving the desired results.

The right Milvus version can help a developer to learn and prototype quickly or help optimize resource utilization, streamline development efforts, and ensure compatibility with existing infrastructure and tools. Ultimately, it is about maintaining developer productivity and improving efficiency, reliability, and user satisfaction.


## Available Milvus versions 

Three versions of Milvus are available for developers, and all are open source. The three versions are Milvus Lite, Milvus Standalone, and Milvus Cluster, which differ in features and how users plan to use Milvus in the short and long term. So, let's explore these individually.


## Milvus Lite

As the name suggests, Milvus Lite is a lightweight version that integrates seamlessly with Google Colab and Jupyter Notebook. It is packaged as a single binary with no additional dependencies, making it easy to install and run on your machine or embed in Python applications. Additionally, Milvus Lite includes a CLI-based Milvus standalone server, providing flexibility for running it directly on your machine. Whether you embed it within your Python code or utilize it as a standalone server is entirely up to your preference and specific application requirements.


### Features and Capabilities

Milvus Lite includes all core Milvus vector search features. 

- **Search Capabilities**: Supports top-k, range, and hybrid searches, including metadata filtering, to cater to diverse search requirements.

- **Index Types and Similarity Metrics**: Offers support for 11 index types and five similarity metrics, providing flexibility and customization options for your specific use case.

- **Data Processing**: Enables batch (Apache Parquet, Arrays, JSON) and stream processing, with seamless integration through connectors for Airbyte, Apache Kafka, and Apache Spark.

- **CRUD Operations**: Offers full CRUD support (create, read, update/upsert, delete), empowering users with comprehensive data management capabilities.


### Applications and limitations

Milvus Lite is ideal for rapid prototyping and local development, offering support for quick setup and experimentation with small-scale datasets on your machine. However, its limitations become apparent when transitioning to production environments with larger datasets and more demanding infrastructure requirements. As such, while Milvus Lite is an excellent tool for initial exploration and testing, it may not be suitable for deploying applications in high-volume or production-ready settings.


### Available Resources

- [Documentation](https://milvus.io/docs/milvus_lite.md)

- [Github Repository](https://github.com/milvus-io/milvus-lite/)

- [Google Colab Example](https://github.com/milvus-io/milvus-lite/tree/main/examples)

- [Getting Started Video](https://www.youtube.com/watch?v=IgJdrGiB5ZY)


## Milvus Standalone

Milvus offers two operational modes: Standalone and Cluster. Both modes are identical in core vector database features and differ in data size support and scalability requirements. This distinction allows you to select the mode that best aligns with your dataset size, traffic volume, and other infrastructure requirements for production. 

Milvus Standalone is a mode of operation for the Milvus vector database system where it operates independently as a single instance without any clustering or distributed setup. Milvus runs on a single server or machine in this mode, providing functionalities such as indexing and searching for vectors. It is suitable for situations where the data and traffic volume scale is relatively small and does not require the distributed capabilities provided by a clustered setup.


### Features and Capabilities

- **High Performance**: Conduct vector searches on massive datasets (billions or more) with exceptional speed and efficiency.

- **Search Capabilities**: Supports top-k, range, and hybrid searches, including metadata filtering, to cater to diverse search requirements.

- **Index Types and Similarity Metrics**: Offers support for 11 index types and 5 similarity metrics, providing flexibility and customization options for your specific use case.

- **Data Processing**: Enables both batch (Apache Parquet, Arrays, Json) and stream processing, with seamless integration through connectors for Airbyte, Apache Kafka, and Apache Spark.

- **Scalability**: Achieve dynamic scalability with component-level scaling, allowing for seamless scaling up and down based on demand. Milvus can autoscale at a component level, optimizing resource allocation for enhanced efficiency.

- **Multi-Tenancy**: Supports multi-tenancy with the capability to manage up to 10,000 collections/partitions in a cluster, providing efficient resource utilization and isolation for different users or applications.

- **CRUD Operations**: Offers full CRUD support (create, read, update/upsert, delete), empowering users with comprehensive data management capabilities.


### Essential components:

- Milvus: The core functional component.

- etcd: The metadata engine responsible for accessing and storing metadata from Milvus' internal components, including proxies, index nodes, and more.

- MinIO: The storage engine responsible for data persistence within Milvus.

![](https://assets.zilliz.com/Screenshot_2024_02_19_at_4_16_41_PM_5e635586a7.png)

Figure 1: Milvus Standalone Architecture


### Available Resources

- Documentation

  - [Environment Checklist for Milvus with Docker Compose](https://milvus.io/docs/prerequisite-docker.md)

  - [Install Milvus Standalone with Docker](https://milvus.io/docs/install_standalone-docker.md)

- [Github Repository](https://github.com/milvus-io/milvus)


## Milvus Cluster

Milvus Cluster is a mode of operation for the Milvus vector database system where it operates and is distributed across multiple nodes or servers. In this mode, Milvus instances are clustered together to form a unified system that can handle larger volumes of data and higher traffic loads compared to a standalone setup. Milvus Cluster offers scalability, fault tolerance, and load balancing features, making it suitable for scenarios that need to handle big data and serve many concurrent queries efficiently.


### Features and Capabilities

- Inherits all features available in Milvus Standalone, including high-performance vector search, support for multiple index types and similarity metrics, and seamless integration with batch and stream processing frameworks.

- Offers unparalleled availability, performance, and cost optimization by leveraging distributed computing and load balancing across multiple nodes.

- Enables deploying and scaling secure, enterprise-grade workloads with lower total costs by efficiently utilizing resources across the cluster and optimizing resource allocation based on workload demands.


### Essential components:

Milvus Cluster includes eight microservice components and three third-party dependencies. All microservices can be deployed on Kubernetes independently from each other.


#### Microservice components

- Root coord

- Proxy

- Query coord

- Query node

- Index coord

- Index node

- Data coord

- Data node


#### Third-party dependencies

- etcd: Stores metadata for various components in the cluster.

- MinIO: Responsible for data persistence of large files in the cluster, such as index and binary log files.

- Pulsar: Manages logs of recent mutation operations, outputs streaming log, and provides log publish-subscribe services.

![](https://assets.zilliz.com/Screenshot_2024_02_19_at_4_18_01_PM_88971280ed.png)

Figure 2: Milvus Cluster Architecture


#### Available Resources

- [Documentation](https://milvus.io/docs/install_cluster-milvusoperator.md) | How to get started

  - [Install Milvus Cluster with Milvus Operator](https://milvus.io/docs/install_cluster-milvusoperator.md)

  - [Install Milvus Cluster with Helm](https://milvus.io/docs/install_cluster-helm.md)

  - [How to scale a Milvus Cluster](https://milvus.io/docs/scaleout.md)

- [Github Repository](https://github.com/milvus-io/milvus)


## Making the Decision on which Milvus version to use

When deciding which version of Milvus to use for your project, you must consider factors such as your dataset size, traffic volume, scalability requirements, and production environment constraints. Milvus Lite is perfect for prototyping on your laptop. Milvus Standalone offers high performance and flexibility for conducting vector searches on your datasets, making it suitable for smaller-scale deployments, CI/CD, and offline deployments when you have no Kubernetes support.. And finally, Milvus Cluster provides unparalleled availability, scalability, and cost optimization for enterprise-grade workloads, making it the preferred choice for large-scale, highly available production environments.

There is another version that is a hassle-free version, and that is a managed version of Milvus called [Zilliz Cloud](https://cloud.zilliz.com/signup). 

Ultimately, the Milvus version will depend on your specific use case, infrastructure requirements, and long-term goals. By carefully evaluating these factors and understanding the features and capabilities of each version, you can make an informed decision that aligns with your project's needs and objectives. Whether you choose Milvus Standalone or Milvus Cluster, you can leverage the power of vector databases to enhance the performance and efficiency of your AI applications.


What Milvus version to start with


As we conclude 2023, let's review the Milvus community's remarkable journey: boasting [25,000 GitHub Stars](https://github.com/milvus-io/milvus), the launch of [Milvus 2.3.0](https://milvus.io/blog/unveiling-milvus-2-3-milestone-release-offering-support-for-gpu-arm64-cdc-and-other-features.md), and exceeding 10 million [Docker image](https://hub.docker.com/r/milvusdb/milvus) downloads. This post explores the heart of the community by analyzing chat histories and revealing the top 10 keywords in discussions.


<iframe class="video-player" src="https://www.youtube.com/embed/o5uMdNLioQ0?list=PLPg7_faNDlT5Fb8WN8r1PzzQTNzdechnS" title="Mastering Milvus: Turbocharge Your Vector Database with Optimization Secrets!" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>

## #1 Version — The rise of AIGC drives rapid Milvus iteration

Surprisingly, "Version" emerged as the most discussed keyword in 2023. This revelation is rooted in the year's AI wave, with vector databases as a critical infrastructure to tackle challenges in AIGC applications' hallucination issues.

The enthusiasm around vector databases drives Milvus into a stage of swift iteration. The community witnessed the release of Twenty versions in 2023 alone, accommodating the demands of AIGC developers flooding the community with inquiries about choosing the optimal version of Milvus for various applications. For users navigating these updates, we recommend embracing the latest version for enhanced features and performance. 

If you are interested in Milvus's release planning, refer to the [Milvus Roadmap](https://wiki.lfaidata.foundation/display/MIL/Milvus+Long+Term+Roadmap+and+Time+schedule) page on the official website. 


## #2 Search — beyond Vector Search

"Search" takes second place, reflecting its fundamental role in database operations. Milvus supports various search capabilities, from Top-K ANN search to scalar filtered search and range search. The imminent release of Milvus 3.0 (Beta) promises keyword search (sparse embeddings), which many RAG app developers eagerly await.

Community discussions about searching focus on performance, capabilities, and principles. Users often ask questions about attribute filtering, setting index threshold values, and addressing latency concerns. Resources like [query and search documentation](https://milvus.io/docs/v2.0.x/search.md), [Milvus Enhancement Proposals (MEPs)](https://wiki.lfaidata.foundation/pages/viewpage.action?pageId=43287103), and Discord discussions have become the go-to references for unraveling the intricacies of searching within Milvus.


## #3 Memory — trade-offs between performance and accuracy for minimized memory overhead

“Memory” also took center stage in community discussions over the past year. As a distinctive data type, vectors inherently have high dimensions. Storing vectors in memory is a common practice for optimal performance, but the escalating data volume limits available memory. Milvus optimizes memory usage by adopting techniques like [MMap](https://zilliz.com/blog/milvus-introduced-mmap-for-redefined-data-management-increased-storage-capability) and DiskANN.

However, achieving low memory usage, excellent performance, and high accuracy simultaneously in a database system remains complex, necessitating trade-offs between performance and accuracy to minimize memory overhead.

In the case of Artificial Intelligence Generated Content (AIGC), developers usually prioritize fast responses and result accuracy over stringent performance requirements. Milvus’s addition of MMap and DiskANN minimizes memory usage while maximizing data processing and result accuracy, striking a balance that aligns with the practical needs of AIGC applications. 


## #4 Insert — smooth sailing through data insertion

Efficient data insertion is a crucial concern for developers, sparking frequent discussions on optimizing insertion speed within the Milvus community. Milvus excels in the efficient insertion of streaming data and the building of indexes, thanks to its adept separation of streaming and batch data. This capability sets it apart as a highly performant solution compared to other vector database providers, such as Pinecone.

Here are some valuable insights and recommendations about data insertions: 

- **Batch Insertion:** Opt for batch over single-row insertion for enhanced efficiency. Notably, insertion from files surpasses batch insertion in speed. When handling large datasets exceeding ten million records, consider using the `bulk_insert` interface for a streamlined and accelerated import process.

- **Strategic `flush()` Usage:** Rather than invoking the `flush()` interface after each batch, make a single call after completing all data insertion. Excessive use of the `flush()` interface between batches can lead to the generation of fragmented segment files, placing a considerable compaction burden on the system.

- **Primary Key Deduplication:** Milvus does not perform primary key deduplication when using the `insert` interface for data insertion. If you need to deduplicate primary keys, we recommend you deploy the `upsert` interface. However, the insertion performance of `upsert `is lower than that of `insert`, owing to an additional internal query operation. 


## #5 Configuration — decoding the parameter maze

Milvus is a distributed vector database that integrates many third-party components like object storage, message queues, and Etcd. Users grappled with adjusting parameters and understanding their impact on Milvus's performance, making "Configuration" a frequently discussed topic.

Among all the questions about configurations, "which parameters to adjust" is arguably the most challenging aspect, as the parameters vary in different situations. For instance, optimizing search performance parameters differs from optimizing insertion performance parameters and relies heavily on practical experience.

Once users identify "which parameters to adjust," the subsequent questions of "how to adjust" become more manageable. For specific procedures, refer to our documentation [Configure Milvus](https://milvus.io/docs/configure-helm.md). The great news is that Milvus has supported dynamic parameter adjustments since version 2.3.0, eliminating the need for restarts for changes to take effect. For specific procedures, refer to [Configure Milvus on the Fly](https://milvus.io/docs/dynamic_config.md). 


## #6 Logs — navigating the troubleshooting compass

"Logs" serve as the troubleshooter's compass. Users sought guidance in the community on exporting Milvus logs, adjusting log levels, and integrating with systems like Grafana’s Loki. Here are some suggestions about Milvus logs. 

- **How to view and export Milvus logs:** You can easily export Milvus logs with the one-click script [export-milvus-log.sh](https://github.com/milvus-io/milvus/tree/master/deployments/export-log) which is available on the GitHub repository. 

- **Log level:** Milvus has multiple log levels to accommodate diverse use cases. The info level is enough for most cases, and the debug level is for debugging. An excess of Milvus logs may signal misconfigured log levels.

- **We recommend integrating Milvus logs with a log collection system** like Loki for streamlined log retrieval in future troubleshooting.


## #7 Cluster — scaling for production environments

Given Milvus's identity as a distributed vector database, the term "cluster" is a frequent topic of discussion in the community. Conversations revolve around scaling data in a cluster, data migration, and data backup and synchronization. 

In production environments, robust scalability and high availability are standard requirements for distributed database systems. Milvus's storage-computation separation architecture allows seamless data scalability by expanding resources for computation and storage nodes, accommodating limitless data scales. Milvus also provides high availability with a multi-replica architecture and robust backup and syncing capabilities.  For more information, refer to [Coordinator HA](https://milvus.io/docs/coordinator_ha.md#Coordinator-HA).


## #8 Documentation — the gateway to understanding Milvus

"Documentation" is another frequently raised keyword in community discussions, often tied to questions about whether there is any documentation page for a specific feature and where to find it. 

Serving as the gateway to understanding Milvus, around 80% of community inquiries find answers in the [official documentation](https://milvus.io/docs). We recommend you read our documentation before using Milvus or encountering any problems. In addition, you can explore code examples in various SDK repositories for insights into using Milvus. 


## #9 Deployment — simplifying the Milvus journey

Simple deployment remains the Milvus team’s ongoing goal. To fulfill this commitment, we introduced [Milvus Lite](https://milvus.io/docs/milvus_lite.md#Get-Started-with-Milvus-Lite), a lightweight alternative to Milvus that is fully functional but has no K8s or Docker dependencies. 

We further streamlined deployment by introducing the lighter [NATS](https://zilliz.com/blog/optimizing-data-communication-milvus-embraces-nats-messaging) messaging solution and consolidating node components. Responding to user feedback, we're gearing up to release a standalone version without dependencies, with ongoing efforts to enhance features and simplify deployment operations. The rapid iteration of Milvus showcases the community's continuous commitment to the continued refinement of the deployment process.


## #10 Deletion — unraveling the impact

The prevalent discussions on "deletion" revolve around unchanged data counts after deletion, the continued retrievability of deleted data, and the failure of disk space recovery after deletion.

Milvus 2.3 introduces the `count(*)` expression to address delayed entity count updates. The persistence of deleted data in queries is probably due to the inappropriate use of [data consistency models](https://zilliz.com/blog/understand-consistency-models-for-vector-databases). Disk space recovery failure concerns prompt insights into redesigning Milvus's garbage collection mechanism, which sets a waiting period before the complete deletion of data. This approach allows a time window for potential recovery.


## Conclusion

The top 10 keywords offer a glimpse into the vibrant discussions within the Milvus community. As Milvus continues to evolve, the community remains an invaluable resource for developers seeking solutions, sharing experiences, and contributing to advancing vector databases in the era of AI. 

Join this exciting journey by joining our [Discord channel](https://discord.com/invite/8uyFbECzPX) in 2024. There, you can engage with our brilliant engineers and connect with like-minded Milvus enthusiasts. Also, attend the [Milvus Community Lunch and Learn](https://discord.com/invite/RjNbk8RR4f) every Tuesday from 12:00 to 12:30 PM PST. Share your thoughts, questions, and feedback, as every contribution adds to the collaborative spirit propelling Milvus forward. Your active participation is not just welcomed; it's appreciated. Let's innovate together!


Top 10 Keywords Dominating the Milvus Community in 2023

Unveiling the Top 10 Keywords Dominating the Milvus Community in 2023


![](https://assets.zilliz.com/What_is_new_in_Milvus_2_3_4_1847b0fa8a.png)


We are excited to unveil the latest release of Milvus 2.3.4. This update introduces a suite of features and enhancements meticulously crafted to optimize performance, boost efficiency, and deliver a seamless user experience. In this blog post, we'll delve into the highlights of Milvus 2.3.4. 

## Access logs for improved monitoring

Milvus now supports access logs, offering invaluable insights into interactions with external interfaces. These logs record method names, user requests, response times, error codes, and other interaction information, empowering developers and system administrators to conduct performance analysis, security auditing, and efficient troubleshooting. 

**_Note:_** _Currently, access logs only support gRPC interactions. However, our commitment to improvement continues, and future versions will extend this capability to include RESTful request logs._ 

For more detailed information, refer to [Configure Access Logs](https://milvus.io/docs/configure_access_logs.md).

## Parquet file imports for enhanced data processing efficiency

Milvus 2.3.4 now supports importing Parquet files, a widely embraced columnar storage format designed to enhance the efficiency of storing and processing large-scale datasets. This addition gives users increased flexibility and efficiency in their data processing endeavors. By eliminating the need for laborious data format conversions, users managing substantial datasets in the Parquet format will experience a streamlined data import process, significantly reducing the time from initial data preparation to subsequent vector retrieval.

Furthermore, our data format conversion tool, BulkWriter, has now embraced Parquet as its default output data format, ensuring a more intuitive experience for developers. 

## Binlog index on growing segments for faster searches

Milvus now leverages a binlog index on growing segments, resulting in up to tenfold faster searches in growing segments. This enhancement significantly boosts search efficiency and supports advanced indices like IVF or Fast Scan, improving the overall user experience.

## Support for up to 10,000 collections/partitions

Like tables and partitions in relational databases, collections and partitions are the core units for storing and managing vector data in Milvus. Responding to users' evolving needs for nuanced data organization, Milvus 2.3.4 now supports up to 10,000 collections/partitions in a cluster, a significant jump from the previous limit of 4,096. This enhancement benefits diverse use cases, such as knowledge base management and multi-tenant environments. The expanded support for collections/partitions stems from refinements to the time tick mechanism, goroutine management, and memory usage.

 **_Note:_** _The recommended limit for the number of collections/partitions is 10,000, as exceeding this limit may impact failure recovery and resource usage._

## Other enhancements 

In addition to the features above, Milvus 2.3.4 includes various improvements and bug fixes. These include reduced memory usage during data retrieval and variable-length data handling, refined error messaging, accelerated loading speed, and improved query shard balance. These collective enhancements contribute to a smoother and more efficient overall user experience.

For a comprehensive overview of all the changes introduced in Milvus 2.3.4, refer to our [Release Notes](https://milvus.io/docs/release_notes.md#v234).

## Stay connected!

If you have questions or feedback about Milvus, join our [Discord channel](https://discord.com/invite/8uyFbECzPX) to engage with our engineers and the community directly or join our [Milvus Community Lunch and Learn](https://discord.com/invite/RjNbk8RR4f) Every Tuesday from 12-12:30 PM PST. You’re also welcome to follow us on [Twitter](https://twitter.com/milvusio) or [LinkedIn](https://www.linkedin.com/company/the-milvus-project) for the latest news and updates about Milvus.


Milvus 2.3.4: Faster Searches, Expanded Data Support, Improved Monitoring, and More


![This image is generated by AI. ](https://assets.zilliz.com/image_7_1c3b05e71c.jpg)
# Milvus in 2023: An Unprecedented Vector Database Amidst Tech Buzz


*This post was written by James Luan with the help of ChatGPT. James primarily wrote prompts and reviewed and polished the AI-generated content.*

## 2023: the year of AI 

2023 marks a pivotal turning point in artificial intelligence (AI). [Large Language Models (LLMs)](https://zilliz.com/glossary/large-language-models-$llms$) have taken center stage, garnering widespread recognition for their exceptional natural language processing capabilities. This surge in popularity has substantially expanded the possibilities of machine learning applications, enabling developers to construct more intelligent and interactive applications.


Amidst this revolution, [vector databases](https://zilliz.com/learn/what-is-vector-database) have emerged as a crucial component, acting as the long-term memory for LLMs. The rise of [Retrieval-Augmented Generation (RAG)](https://zilliz.com/use-cases/llm-retrieval-augmented-generation) models, intelligent agents, and multimodal retrieval apps has demonstrated the vast potential of vector databases in enhancing multimodal data retrieval efficiency, reducing hallucinations in LLMs, and supplementing domain knowledge.


The LLM evolution has also catalyzed significant advancements in embedding technologies. According to the [Massive Text Embedding Benchmark (MTEB) Leaderboard](https://huggingface.co/spaces/mteb/leaderboard) on HuggingFace, leading embedding models such as UAE, VoyageAI, CohereV3, and Bge were all released in 2023. These advancements have bolstered the vector retrieval effectiveness of various vector search technologies like Milvus, providing more precise and efficient data processing capabilities for AI applications.


However, with the growing popularity of vector databases, debates arose about the necessity of specialized solutions. Tens of startups have entered the vector database arena. Many traditional relational and NoSQL databases have started treating vectors as a significant data type, and many claim to be capable of substituting specialized vector databases in every situation. 


As we enter 2024, it's an sensible moment to reflect on the entire vector database industry, with a special focus on Milvus—a standout product in this landscape. 




## Milvus in 2023: numbers don't lie


First launched in 2019, [Milvus](https://zilliz.com/what-is-milvus) has pioneered the concept of vector databases and consistently maintained a reputation for high reliability, scalability, search quality, and performance. In 2023, Milvus achieved impressive results and underwent significant shifts, primarily driven by the rapid advancement of LLMs and the boom of AIGC applications. Here are some key figures that best represent Milvus's progress in 2023. 




### ZERO downtime during rolling upgrades


For those new to vector databases, their primary focus centers on functionality rather than operational maintenance. Many application developers also pay less attention to stability in their vector databases than transactional databases since their applications are often in the early stages of exploration. However, stability becomes indispensable if you aim to deploy your AIGC application in a production environment and achieve the best user experience. 


Milvus distinguishes itself by prioritizing not just functionality but also operational stability. We added rolling upgrades to Milvus starting from version 2.2.3. After continuous refinement, this feature can ensure zero downtime during upgrades without interrupting business processes. 




### 3x performance improvement in production environments


Boosting vector search performance needs to be a primary goal for vector databases. Many vector search solutions chose to base their solution on adapting the [HNSW](https://zilliz.com/learn/hierarchical-navigable-small-worlds-HNSW) algorithm to get to market quickly; unfortunately, this leads to them facing significant challenges in real-world production environments, especially with highly filtered searches (over 90%) and frequent data deletions. Milvus considers performance from the get-go and excels in optimizing performance during any phase of development, especially in production environments, achieving a threefold improvement in search performance, especially in filtered search and streaming insert/search situations.


To further assist the vector database community, we introduced [VectorDBBench](https://github.com/zilliztech/VectorDBBench), an open-source benchmarking tool, last year. This tool is vital for early evaluations of vector databases across different conditions. Unlike traditional evaluation methods, VectorDBBench assesses databases using real-world data, including super large datasets or those closely resembling data from actual embedding models, providing users with more insightful information for informed decision-making.




### 5% recall improvement on the Beir dataset


While [dense embeddings](https://zilliz.com/learn/sparse-and-dense-embeddings) have proven effective in vector search, they must catch up when searching for names, objects, abbreviations, and short query contexts. In response to their limitations, Milvus has introduced a hybrid query approach that integrates dense embeddings with [sparse embeddings](https://zilliz.com/learn/sparse-and-dense-embeddings) to enhance the quality of search results. The synergy of this hybrid solution with a reranking model has resulted in a substantial 5% improvement in the recall rate on the Beir dataset, as validated by our tests.


Going beyond improvements in search quality, Milvus has also unveiled a graph-based retrieval solution tailored for sparse embeddings, surpassing the performance of conventional search algorithms like WAND.


At the 2023 NeurIPS BigANN competition, Zihao Wang, a talented engineer at Zilliz, presented [Pyanns](https://big-ann-benchmarks.com/neurips23.html#winners), a search algorithm that demonstrated significant superiority over other entries in the sparse embedding search track. This breakthrough solution is a precursor to our sparse embedding search algorithms for production environments.




### 10x memory saving on large datasets


[Retrieval Augmented Generation](https://zilliz.com/use-cases/llm-retrieval-augmented-generation) (RAG) was the most popular use case for vector databases in 2023. However, the increase in vector data volumes with RAG applications presents a storage challenge for these applications. This challenge is especially true when the volume of transformed vectors exceeds that of the original document chunks, potentially escalating memory usage costs. For example, after dividing documents into chunks, the size of a 1536-dimensional float32 vector (roughly 3kb) transformed from a 500-token chunk (about 1kb) is greater than the 500-token chunk. 


Milvus is the first open-source vector database to support disk-based indexing, bringing about a remarkable 5x memory saving. By the close of 2023, we introduced [Milvus 2.3.4](https://milvus.io/docs/release_notes.md#v234), enabling the capability to load scalar and vector data/indexes onto the disk using memory-mapped files ([MMap](https://zilliz.com/blog/milvus-introduced-mmap-for-redefined-data-management-increased-storage-capability)). This advancement offers more than a 10x reduction in memory usage compared to traditional in-memory indexing. 




### 20 Milvus releases


In 2023, Milvus underwent a transformative journey marked by significant milestones. Over the year, we launched 20 releases, a testament to the dedication of over 300 community developers and the realization of our commitment to a user-driven approach in development.


To illustrate, Milvus 2.2.9 introduced [dynamic schema](https://zilliz.com/blog/what-is-dynamic-schema), marking a crucial shift from prioritizing performance to enhancing usability. Building on this, [Milvus 2.3](https://milvus.io/blog/unveiling-milvus-2-3-milestone-release-offering-support-for-gpu-arm64-cdc-and-other-features.md) introduced critical features such as Upsert, [Range Search](https://zilliz.com/blog/unlock-advanced-recommendation-engines-with-milvus-new-range-search), Cosine metrics, and more, all driven by our user community's specific needs and feedback. This iterative development process underscores our commitment to continually aligning Milvus with the evolving requirements of our users.




### 1,000,000 tenants in a Single Custer 


Implementing multi-tenancy is crucial for developing RAG systems, AI agents, and other LLM applications, meeting the heightened user demands for data isolation. For B2C businesses, tenant numbers can skyrocket into the millions, making physical isolation of user data impractical (as an example, it's unlikely that anyone would create millions of tables in a relational database). Milvus introduced the Partition Key feature, allowing for efficient, logical isolation and data filtering based on partition keys, which is handy at a large scale.


Conversely, B2B enterprises, accustomed to dealing with tens of thousands of tenants, benefit from a more nuanced strategy involving physical resource isolation. The latest Milvus 2.3.4 brings enhanced memory management, coroutine handling, and CPU optimization, making creating tens of thousands of tables within a single cluster easier. This enhancement also accommodates the needs of B2B businesses with enhanced efficiency and control.




### 10,000,000 Docker image pulls


As 2023 drew to a close, Milvus reached an impressive milestone with [10 million Docker pull](https://hub.docker.com/r/milvusdb/milvus) downloads. This accomplishment signals the increasing fascination of the developer community with Milvus and emphasizes its rising significance within the vector database domain. 


As the world's first cloud-native vector database, Milvus boasts seamless integration with Kubernetes and the broader container ecosystem. Gazing into the future, one can't help but ponder the next focal point in the ever-evolving vector database landscape. Could it be the rise of Serverless services?




### 10 billion entities in a single collection


While scalability might not currently steal the spotlight in the AI phenomenon, it certainly plays a pivotal role, far from being a mere sideshow. Milvus vector database can seamlessly scale out to accommodate billions of vector data without breaking a sweat. Take a look at one of our LLM customers, for example. Milvus effortlessly helped this customer store, process, and retrieve an astounding 10 billion data points. But how do you balance the cost and performance when dealing with such a massive volume of data? Rest assured, Mivus has various capabilities to help you address that challenge and elevate your experience. 




## Beyond the numbers: the new insights into vector databases 


Beyond the numerical milestones, 2023 has enriched us with valuable insights. We've delved into the intricacies of the vector database landscape, moving beyond mere statistics to grasp the subtle nuances and evolving dynamics of vector search technology.




### LLM apps are still in the early stages. 


Reflecting on the early days of the mobile internet boom, many developers created simple apps like flashlights or weather forecasts, which eventually were integrated into smartphone operating systems. Last year, most AI Native applications, like AutoGPT, which rapidly hit 100,000 stars on GitHub, didn't deliver practical value but only represented meaningful experiments. For vector database applications, the current use cases may just be the first wave of AI Native transformations, and I eagerly anticipate more killer use cases to emerge.




### Vector databases go toward diversification. 


Similar to the evolution of databases into categories like OLTP, OLAP, and NoSQL, vector databases show a clear trend toward diversification. Departing from the conventional focus on online services, offline analysis has gained significant traction. Another notable instance of this shift is the introduction of [GPTCache](https://zilliz.com/blog/building-llm-apps-100x-faster-responses-drastic-cost-reduction-using-gptcache), an open-sourced semantic cache released in 2023. It enhances the efficiency and speed of GPT-based applications by storing and retrieving responses generated by language models.


We are hopeful and excited to witness even more diversified applications and system designs in vector databases in the coming year.




### Vector operations are becoming more complicated. 


While supporting [Approximate Nearest Neighbor (ANN)](https://zilliz.com/glossary/anns) search is a defining feature of vector databases, it doesn't stand alone. The common belief that merely keeping Nearest Neighbour Search is sufficient to classify a database as a vector or AI native database oversimplifies the intricacies of vector operations. Beyond the basic capabilities of hybrid scalar filtering and vector search, databases tailored for AI native applications should support more sophisticated semantic capabilities like NN Filtering, KNN Join, and cluster querying. 




### Elastic scalability is essential for AI native applications. 


The exponential growth of AI applications, exemplified by ChatGPT amassing over 100 million monthly active users in two months, surpasses any prior business trajectory. Swiftly scaling from 1 million to 1 billion data points becomes paramount once businesses hit their stride in growth. AI application developers benefit from the pay-as-you-go service model set by LLM providers, leading to substantial reductions in operational costs. Similarly, storing data that aligns with this pricing model proves advantageous for developers, allowing them to channel more attention toward core business. 


Unlike Language Models (LLMs) and various other technological systems, vector databases operate in a stateful manner, demanding persistent data storage for their functionality. Consequently, when selecting vector databases, it is crucial to prioritize elasticity and scalability. This prioritization ensures alignment with the dynamic demands of evolving AI applications, highlighting the need for seamless adaptability to changing workloads.




### Leveraging machine learning in vector databases can yield extraordinary results. 


In 2023, our substantial investment in the AI4DB (AI for Database) projects yielded remarkable success. As part of our endeavors, we introduced two pivotal capabilities to [Zilliz Cloud](https://zilliz.com/cloud), the fully managed Milvus solution: 1) AutoIndex, an auto-parameter-tuning index rooted in machine learning, and 2) a data partitioning strategy based on data clustering. Both innovations played a crucial role in significantly enhancing the search performance of Zilliz Cloud.




### Open source vs. closed source


Closed-source LLMs like OpenAI's GPT series and Claude currently take the lead, placing the open-source community disadvantaged due to the absence of comparable computational and data resources.


However, within vector databases, open source will eventually become the favored choice for users. Opting for open source introduces many advantages, including more diverse use cases, expedited iteration, and cultivating a more robust ecosystem. Furthermore, database systems are so intricate that they cannot afford the opacity often associated with LLMs. Users must thoroughly understand the database before choosing the most reasonable approach for its utilization. Moreover, the transparency ingrained in open source empowers users to possess the liberty and the control to customize the database according to their needs.




## Epilogue - And a new beginning!


As 2023 swiftly passes amidst transformative changes, the story of vector databases is just beginning. Our journey with the Milvus vector database is about something other than getting lost in the hype of AIGC. Instead, we focus on meticulously developing our product, identifying and nurturing application use cases that align with our strengths, and unwaveringly serving our users. Our commitment to open source aims to bridge the gap between us and our users, allowing them to sense our dedication and craftsmanship, even from a distance.


2023 also saw many AI startups being founded and getting their first funding rounds. It is exciting to see the innovation from these developers, and it reminds me of why I got into VectorDB development in the first place. 2024 will be a year for all these innovative applications to gain real traction, attracting not just funding but real paying customers. Customer revenue will bring different requirements for these developers, as building a fully scalable solution with little to no downtime is paramount. 


Let's make extraordinary things happen in 2024!


Milvus in 2023: An Unprecedented Vector Database Amidst Tech Buzz

Efficient Vector Similarity Search in Recommender Workflows Using Milvus with NVIDIA Merlin


![](https://assets.zilliz.com/How_to_Get_the_Right_Vector_Embedding_d9ebcacbbb.png)


*This article was originally published in [The New Stack](https://thenewstack.io/how-to-get-the-right-vector-embeddings/) and is reposted here with permission.*

**A comprehensive introduction to vector embeddings and how to generate them with popular open source models.**

![Image by Денис Марчук from Pixabay](https://assets.zilliz.com/how_to_get_right_vector_embeddings_e0838623b7.png)


Vector embeddings are critical when working with  [semantic similarity](https://zilliz.com/blog/vector-similarity-search). However, a vector is simply a series of numbers; a vector embedding is a series of numbers representing input data. Using vector embeddings, we can structure  [unstructured data](https://zilliz.com/blog/introduction-to-unstructured-data)  or work with any type of data by converting it into a series of numbers. This approach allows us to perform mathematical operations on the input data, rather than relying on qualitative comparisons.

Vector embeddings are influential for many tasks, particularly for  [semantic search](https://zilliz.com/glossary/semantic-search). However, it is crucial to obtain the appropriate vector embeddings before using them. For instance, if you use an image model to vectorize text, or vice versa, you will probably get poor results.

In this post, we will learn what vector embeddings mean, how to generate the right vector embeddings for your applications using different models and how to make the best use of vector embeddings with vector databases like  [Milvus](https://milvus.io/)  and  [Zilliz Cloud](https://zilliz.com/).

## How are vector embeddings created?

![](https://assets.zilliz.com/how_vector_embeddings_are_created_03f9b60c68.png)


Now that we understand the importance of vector embeddings, let’s learn how they work. A vector embedding is the internal representation of input data in a deep learning model, also known as embedding models or a deep neural network. So, how do we extract this information?

We obtain vectors by removing the last layer and taking the output from the second-to-last layer. The last layer of a neural network usually outputs the model’s prediction, so we take the output of the second-to-last layer. The vector embedding is the data fed to a neural network’s predictive layer.

The dimensionality of a vector embedding is equivalent to the size of the second-to-last layer in the model and, thus, interchangeable with the vector’s size or length. Common vector dimensionalities include 384 (generated by Sentence Transformers Mini-LM), 768 (by Sentence Transformers MPNet), 1,536 (by OpenAI) and 2,048 (by ResNet-50).


## What does a vector embedding mean?

Someone once asked me about the meaning of each dimension in a vector embedding. The short answer is nothing. A single dimension in a vector embedding does not mean anything, as it is too abstract to determine its meaning. However, when we take all dimensions together, they provide the semantic meaning of the input data.

The dimensions of the vector are high-level, abstract representations of different attributes. The represented attributes depend on the training data and the model itself. Text and image models generate different embeddings because they’re trained for fundamentally different data types. Even different text models generate different embeddings. Sometimes they differ in size; other times, they differ in the attributes they represent. For instance, a model trained on legal data will learn different things than one trained on health-care data. I explored this topic in my post  [comparing vector embeddings](https://zilliz.com/blog/comparing-different-vector-embeddings).

## Generate the right vector embeddings

How do you obtain the proper vector embeddings? It all starts with identifying the type of data you wish to embed. This section covers embedding five different types of data: images, text, audio, videos and multimodal data. All models we introduce here are open source and come from Hugging Face or PyTorch.

### Image embeddings

Image recognition took off in 2012 after AlexNet hit the scene. Since then, the field of computer vision has witnessed numerous advancements. The latest notable image recognition model is ResNet-50, a 50-layer deep residual network based on the former ResNet-34 architecture.

Residual neural networks (ResNet) solve the vanishing gradient problem in deep convolutional neural networks using shortcut connections. These connections allow the output from earlier layers to go to later layers directly without passing through all the intermediate layers, thus avoiding the vanishing gradient problem. This design makes ResNet less complex than VGGNet (Visual Geometry Group), a previously top-performing convolutional neural network.

I recommend two ResNet-50 implementations as examples:  [ResNet 50 on Hugging Face](https://huggingface.co/microsoft/resnet-50)  and  [ResNet 50 on PyTorch Hub](https://pytorch.org/vision/main/models/generated/torchvision.models.resnet50.html). While the networks are the same, the process of obtaining embeddings differs.

The code sample below demonstrates how to use PyTorch to obtain vector embeddings. First, we load the model from PyTorch Hub. Next, we remove the last layer and call  `.eval()`  to instruct the model to behave like it’s running for inference. Then, the  `embed`  function generates the vector embedding.


```
# Load the embedding model with the last layer removed
model = torch.hub.load('pytorch/vision:v0.10.0', 'resnet50', pretrained=True) model = torch.nn.Sequential(*(list(model.children())[:-1]))
model.eval()


def embed(data):
with torch.no_grad():
output = model(torch.stack(data[0])).squeeze()
return output
```


HuggingFace uses a slightly different setup. The code below demonstrates how to obtain a vector embedding from Hugging Face. First, we need a feature extractor and model from the  `transformers`  library. We will use the feature extractor to get inputs for the model and use the model to obtain outputs and extract the last hidden state.


```
# Load model directly
from transformers import AutoFeatureExtractor, AutoModelForImageClassification


extractor = AutoFeatureExtractor.from_pretrained("microsoft/resnet-50")
model = AutoModelForImageClassification.from_pretrained("microsoft/resnet-50")


from PIL import Image


image = Image.open("<image path>")
# image = Resize(size=(256, 256))(image)


inputs = extractor(images=image, return_tensors="pt")
# print(inputs)


outputs = model(**inputs)
vector_embeddings = outputs[1][-1].squeeze()
```


### Text embeddings

Engineers and researchers have been experimenting with natural language and AI since the invention of AI. Some of the earliest experiments include:

-   ELIZA, the first AI therapist chatbot.
-   John Searle’s Chinese Room, a thought experiment that examines whether the ability to translate between Chinese and English requires an understanding of the language.
-   Rule-based translations between English and Russian.

AI’s operation on natural language has evolved significantly from its rule-based embeddings. Starting with primary neural networks, we added recurrence relations through RNNs to keep track of steps in time. From there, we used transformers to solve the sequence transduction problem.

Transformers consist of an encoder, which encodes an input into a matrix representing the state, an attention matrix and a decoder. The decoder decodes the state and attention matrix to predict the correct next token to finish the output sequence. GPT-3, the most popular language model to date, comprises strict decoders. They encode the input and predict the right next token(s).

Here are two models from the  `sentence-transformers`  library by Hugging Face that you can use in addition to OpenAI’s embeddings:

-   [MiniLM-L6-v2](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2): a 384-dimensional model
-   [MPNet-Base-V2](https://huggingface.co/sentence-transformers/all-mpnet-base-v2): a 768-dimensional model

You can access embeddings from both models in the same way.


```
from sentence_transformers import SentenceTransformer


model = SentenceTransformer("<model-name>")
vector_embeddings = model.encode(“<input>”)
```


### Multimodal embeddings

Multimodal models are less well-developed than image or text models. They often relate images to text.

The most useful open source example is [CLIP VIT](https://huggingface.co/openai/clip-vit-large-patch14), an image-to-text model. You can access CLIP VIT’s embeddings in the same way as you would an image model, as shown in the code below.


```
# Load model directly
from transformers import AutoProcessor, AutoModelForZeroShotImageClassification


processor = AutoProcessor.from_pretrained("openai/clip-vit-large-patch14")
model = AutoModelForZeroShotImageClassification.from_pretrained("openai/clip-vit-large-patch14")
from PIL import Image


image = Image.open("<image path>")
# image = Resize(size=(256, 256))(image)


inputs = extractor(images=image, return_tensors="pt")
# print(inputs)


outputs = model(**inputs)
vector_embeddings = outputs[1][-1].squeeze()
```


### Audio embeddings

AI for audio has received less attention than AI for text or images. The most common use case for audio is speech-to-text for industries such as call centers, medical technology and accessibility. One popular open source model for speech-to-text is  [Whisper from OpenAI](https://huggingface.co/openai/whisper-large-v2). The code below shows how to obtain vector embeddings from the speech-to-text model.  


```
import torch
from transformers import AutoFeatureExtractor, WhisperModel
from datasets import load_dataset


model = WhisperModel.from_pretrained("openai/whisper-base")
feature_extractor = AutoFeatureExtractor.from_pretrained("openai/whisper-base")
ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
inputs = feature_extractor(ds[0]["audio"]["array"], return_tensors="pt")
input_features = inputs.input_features
decoder_input_ids = torch.tensor([[1, 1]]) * model.config.decoder_start_token_id
vector_embedding = model(input_features, decoder_input_ids=decoder_input_ids).last_hidden_state
```


### Video embeddings

Video embeddings are more complex than audio or image embeddings. A multimodal approach is necessary when working with videos, as they include synchronized audio and images. One popular video model is the  [multimodal perceiver](https://huggingface.co/deepmind/multimodal-perceiver)  from DeepMind. This  [notebook tutorial](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/Perceiver/Perceiver_for_Multimodal_Autoencoding.ipynb)  shows how to use the model to classify a video.

To get the embeddings of the input, use  `outputs[1][-1].squeeze()`  from the code shown in the notebook instead of deleting the outputs. I highlight this code snippet in the  `autoencode`  function.


```
def autoencode_video(images, audio):
     # only create entire video once as inputs
     inputs = {'image': torch.from_numpy(np.moveaxis(images, -1, 2)).float().to(device),
               'audio': torch.from_numpy(audio).to(device),
               'label': torch.zeros((images.shape[0], 700)).to(device)}
     nchunks = 128
     reconstruction = {}
     for chunk_idx in tqdm(range(nchunks)):
          image_chunk_size = np.prod(images.shape[1:-1]) // nchunks
          audio_chunk_size = audio.shape[1] // SAMPLES_PER_PATCH // nchunks
          subsampling = {
               'image': torch.arange(
                    image_chunk_size * chunk_idx, image_chunk_size * (chunk_idx + 1)),
               'audio': torch.arange(
                    audio_chunk_size * chunk_idx, audio_chunk_size * (chunk_idx + 1)),
               'label': None,
          }
     # forward pass
          with torch.no_grad():
               outputs = model(inputs=inputs, subsampled_output_points=subsampling)


          output = {k:v.cpu() for k,v in outputs.logits.items()}
          reconstruction['label'] = output['label']
          if 'image' not in reconstruction:
               reconstruction['image'] = output['image']
               reconstruction['audio'] = output['audio']
          else:
               reconstruction['image'] = torch.cat(
                    [reconstruction['image'], output['image']], dim=1)
               reconstruction['audio'] = torch.cat(
                    [reconstruction['audio'], output['audio']], dim=1)
          vector_embeddings = outputs[1][-1].squeeze()
# finally, reshape image and audio modalities back to original shape
     reconstruction['image'] = torch.reshape(reconstruction['image'], images.shape)
     reconstruction['audio'] = torch.reshape(reconstruction['audio'], audio.shape)
     return reconstruction


     return None
```


## Storing, indexing, and searching vector embeddings with vector databases

Now that we understand what vector embeddings are and how to generate them using various powerful embedding models, the next question is how to store and take advantage of them. Vector databases are the answer.

Vector databases like  [Milvus](https://zilliz.com/what-is-milvus)  and  [Zilliz Cloud](https://zilliz.com/cloud)  are purposely built for storing, indexing, and searching across massive datasets of unstructured data through vector embeddings. They are also one of the most critical infrastructures for various AI stacks.

Vector databases usually use the  [Approximate Nearest Neighbor (ANN)](https://zilliz.com/glossary/anns)  algorithm to calculate the spatial distance between the query vector and vectors stored in the database. The closer the two vectors are located, the more relevant they are. Then the algorithm finds the top k nearest neighbors and delivers them to the user.

Vector databases are popular in use cases such as  [LLM retrieval augmented generation](https://zilliz.com/use-cases/llm-retrieval-augmented-generation)  (RAG), question and answer systems, recommender systems, semantic searches, and image, video and audio similarity searches.

To learn more about vector embeddings, unstructured data, and vector databases, consider starting with the  [Vector Database 101](https://zilliz.com/blog?tag=39&page=1)  series.

## Summary

Vectors are a powerful tool for working with unstructured data. Using vectors, we can mathematically compare different pieces of unstructured data based on semantic similarity. Choosing the right vector-embedding model is critical for building a vector search engine for any application.

In this post, we learned that vector embeddings are the internal representation of input data in a neural network. As a result, they depend highly on the network architecture and the data used to train the model. Different data types (such as images, text, and audio) require specific models. Fortunately, many pretrained open source models are available for use. In this post, we covered models for the five most common types of data: images, text, multimodal, audio, and video. In addition, if you want to make the best use of vector embeddings, vector databases are the most popular tool.

How to Get the Right Vector Embeddings


![](https://assets.zilliz.com/How_to_Migrate_Your_Data_to_Milvus_with_Ease_485dcb8b22.png)


[Milvus](https://milvus.io/) is a robust open-source vector database for [similarity search](https://zilliz.com/learn/vector-similarity-search) that can store, process, and retrieve billions and even trillions of vector data with minimal latency. It is also highly scalable, reliable, cloud-native, and feature-rich. [The newest release of Milvus](https://milvus.io/blog/unveiling-milvus-2-3-milestone-release-offering-support-for-gpu-arm64-cdc-and-other-features.md) introduces even more exciting features and improvements, including [GPU support](https://zilliz.com/blog/getting-started-with-gpu-powered-milvus-unlocking-10x-higher-performance) for over 10x faster performance and MMap for greater storage capacity on a single machine.

As of September 2023, Milvus has earned almost 23,000 stars on GitHub and has tens of thousands of users from diverse industries with varying needs. It is becoming even more popular as Generative AI technology like [ChatGPT](https://zilliz.com/learn/ChatGPT-Vector-Database-Prompt-as-code) becomes more prevalent. It is an essential component of various AI stacks, especially the [retrieval augmented generation](https://zilliz.com/use-cases/llm-retrieval-augmented-generation) framework, which addresses the hallucination problem of large language models.

To meet the growing demand from new users who want to migrate to Milvus and existing users who wish to upgrade to the latest Milvus versions, we developed  [Milvus Migration](https://github.com/zilliztech/milvus-migration). In this blog, we'll explore the features of Milvus Migration and guide you through quickly transitioning your data to Milvus from Milvus 1.x, [FAISS](https://zilliz.com/blog/set-up-with-facebook-ai-similarity-search-faiss), and [Elasticsearch 7.0](https://zilliz.com/comparison/elastic-vs-milvus) and beyond.

## Milvus Migration, a powerful data migration tool

[Milvus Migration](https://github.com/zilliztech/milvus-migration) is a data migration tool written in Go. It enables users to move their data seamlessly from older versions of Milvus (1.x), FAISS, and Elasticsearch 7.0 and beyond to Milvus 2.x versions.

The diagram below demonstrates how we built Milvus Migration and how it works.

![](https://assets.zilliz.com/milvus_migration_architecture_144e22f499.png)


### How Milvus Migration migrates data

#### From Milvus 1.x and FAISS to Milvus 2.x

The data migration from Milvus 1.x and FAISS involves parsing the content of the original data files, transforming them into the data storage format of Milvus 2.x, and writing the data using Milvus SDK's `bulkInsert`. This entire process is stream-based, theoretically limited only by disk space, and stores data files on your local disk, S3, OSS, GCP, or Minio.

#### From Elasticsearch to Milvus 2.x

In the Elasticsearch data migration, data retrieval is different. Data is not obtained from files but sequentially fetched using Elasticsearch's scroll API. The data is then parsed and transformed into Milvus 2.x storage format, followed by writing it using `bulkInsert`. Besides migrating `dense_vector` type vectors stored in Elasticsearch, Milvus Migration also supports migrating other field types, including long, integer, short, boolean, keyword, text, and double.

### Milvus Migration feature set

Milvus Migration simplifies the migration process through its robust feature set:

- **Supported Data Sources:**

	- Milvus 1.x to Milvus 2.x

	- Elasticsearch 7.0 and beyond to Milvus 2.x

	- FAISS to Milvus 2.x

  

- **Multiple Interaction Modes:**

	- Command-line interface (CLI) using the Cobra framework

	- Restful API with a built-in Swagger UI

	- Integration as a Go module in other tools

  

- **Versatile File Format Support:**

	- Local files

	- Amazon S3

	- Object Storage Service (OSS)

	- Google Cloud Platform (GCP)

  

- **Flexible Elasticsearch Integration:**

	- Migration of `dense_vector` type vectors from Elasticsearch

	- Support for migrating other field types such as long, integer, short, boolean, keyword, text, and double
    

### Interface definitions

Milvus Migration provides the following key interfaces:

-   `/start`: Initiates a migration job (equivalent to a combination of dump and load, currently only supports ES migration).
    
-   `/dump`: Initiates a dump job (writes source data into the target storage medium).
    
-   `/load`: Initiates a load job (writes data from the target storage medium into Milvus 2.x).
    
-   `/get_job`: Allows users to view job execution results. (For more details, refer to [the project's server.go](https://github.com/zilliztech/milvus-migration/blob/main/server/server.go))
    

  

Next, let's use some example data to explore how to use Milvus Migration in this section. You can find these examples [here](https://github.com/zilliztech/milvus-migration#migration-examples-migrationyaml-details) on GitHub.

## Migration from Elasticsearch to Milvus 2.x

1.  Prepare Elasticsearch Data
    

To [migrate Elasticsearch](https://zilliz.com/blog/elasticsearch-cloud-vs-zilliz) data, you should already set up your own Elasticsearch server. You should store vector data in the `dense_vector` field and index them with other fields. The index mappings are as shown below.

![](https://assets.zilliz.com/migrate_elasticsearch_data_milvus_index_mappings_59370f9596.png)



2.  Compile and Build
    

First, download the Milvus Migration’s [source code from GitHub](https://github.com/zilliztech/milvus-migration). Then, run the following commands to compile it.


```
go get
go build
```

This step will generate an executable file named `milvus-migration`.

  

3.  Configure `migration.yaml`
    

Before starting the migration, you must prepare a configuration file named `migration.yaml` that includes information about the data source, target, and other relevant settings. Here's an example configuration:



```
# Configuration for Elasticsearch to Milvus 2.x migration


dumper:
  worker:
    workMode: Elasticsearch
    reader:
      bufferSize: 2500
meta:
  mode: config
  index: test_index
  fields:
    - name: id
      pk: true
      type: long
    - name: other_field
      maxLen: 60
      type: keyword
    - name: data
      type: dense_vector
      dims: 512
  milvus:
      collection: "rename_index_test"
      closeDynamicField: false
      consistencyLevel: Eventually
      shardNum: 1


source:
  es:
    urls:
      - http://localhost:9200
    username: xxx
    password: xxx


target:
  mode: remote
  remote:
    outputDir: outputPath/migration/test1
    cloud: aws
    region: us-west-2
    bucket: xxx
    useIAM: true
    checkBucket: false
  milvus2x:
    endpoint: {yourMilvusAddress}:{port}
    username: ******
    password: ******
```
  

For a more detailed explanation of the configuration file, refer to [this page](https://github.com/zilliztech/milvus-migration/blob/main/README_ES.md#elasticsearch-to-milvus-2x-migrationyaml-example) on GitHub.

  

4.  Execute the migration job
    

Now that you have configured your `migration.yaml` file, you can start the migration task by running the following command:


```
./milvus-migration start --config=/{YourConfigFilePath}/migration.yaml
```

  

Observe the log output. When you see logs similar to the following, it means the migration was successful.

  
```
[task/load_base_task.go:94] ["[LoadTasker] Dec Task Processing-------------->"] [Count=0] [fileName=testfiles/output/zwh/migration/test_mul_field4/data_1_1.json] [taskId=442665677354739304][task/load_base_task.go:76] ["[LoadTasker] Progress Task --------------->"] [fileName=testfiles/output/zwh/migration/test_mul_field4/data_1_1.json] [taskId=442665677354739304][dbclient/cus_field_milvus2x.go:86] ["[Milvus2x] begin to ShowCollectionRows"][loader/cus_milvus2x_loader.go:66] ["[Loader] Static: "] [collection=test_mul_field4_rename1] [beforeCount=50000] [afterCount=100000] [increase=50000][loader/cus_milvus2x_loader.go:66] ["[Loader] Static Total"] ["Total Collections"=1] [beforeTotalCount=50000] [afterTotalCount=100000] [totalIncrease=50000][migration/es_starter.go:25] ["[Starter] migration ES to Milvus finish!!!"] [Cost=80.009174459][starter/starter.go:106] ["[Starter] Migration Success!"] [Cost=80.00928425][cleaner/remote_cleaner.go:27] ["[Remote Cleaner] Begin to clean files"] [bucket=a-bucket] [rootPath=testfiles/output/zwh/migration][cmd/start.go:32] ["[Cleaner] clean file success!"]
```


  

In addition to the command-line approach, Milvus Migration also supports migration using Restful API.

To use the Restful API, start the API server using the following command:


```
./milvus-migration server run -p 8080
```

Once the service runs, you can initiate the migration by calling the API.


```
curl -XPOST http://localhost:8080/api/v1/start
```

  

When the migration is complete, you can use [Attu](https://zilliz.com/attu), an all-in-one vector database administration tool, to view the total number of successful rows migrated and perform other collection-related operations.


![The Attu interface](https://assets.zilliz.com/attu_interface_vector_database_admin_4893a31f6d.png)



## Migration from Milvus 1.x to Milvus 2.x

1.  Prepare Milvus 1.x Data
    

To help you quickly experience the migration process, we’ve put 10,000 Milvus 1.x [test data](https://github.com/zilliztech/milvus-migration/blob/main/README_1X.md) records in the source code of Milvus Migration. However, in real cases, you must export your own `meta.json` file from your Milvus 1.x instance before starting the migration process.

-   You can export the data with the following command.
    

```
./milvus-migration export -m "user:password@tcp(adderss)/milvus?charset=utf8mb4&parseTime=True&loc=Local" -o outputDir
```

  

Make sure to:

-   Replace the placeholders with your actual MySQL credentials.
    
-   Stop the Milvus 1.x server or halt data writes before performing this export.
    
-   Copy the Milvus `tables` folder and the `meta.json` file to the same directory.
    

**Note:** If you use Milvus 2.x on [Zilliz Cloud](https://zilliz.com/cloud) (the fully managed service of Milvus), you can start the migration using Cloud Console.

2.  Compile and Build
    

First, download the Milvus Migration’s [source code from GitHub](https://github.com/zilliztech/milvus-migration). Then, run the following commands to compile it.


```
go get
go build
```


This step will generate an executable file named `milvus-migration`.

3.  Configure `migration.yaml`
    

Prepare a `migration.yaml` configuration file, specifying details about the source, target, and other relevant settings. Here's an example configuration:


```
# Configuration for Milvus 1.x to Milvus 2.x migration


dumper:
  worker:
    limit: 2
    workMode: milvus1x
    reader:
      bufferSize: 1024
    writer:
      bufferSize: 1024
loader:
  worker:
    limit: 16
meta:
  mode: local
  localFile: /outputDir/test/meta.json


source:
  mode: local
  local:
    tablesDir: /db/tables/


target:
  mode: remote
  remote:
    outputDir: "migration/test/xx"
    ak: xxxx
    sk: xxxx
    cloud: aws
    endpoint: 0.0.0.0:9000
    region: ap-southeast-1
    bucket: a-bucket
    useIAM: false
    useSSL: false
    checkBucket: true
  milvus2x:
    endpoint: localhost:19530
    username: xxxxx
    password: xxxxx
```


  

For a more detailed explanation of the configuration file, refer to [this page](https://github.com/zilliztech/milvus-migration/blob/main/README_1X.md) on GitHub.

  

4.  Execute Migration Job
    

You must execute the `dump` and `load` commands separately to finish the migration. These commands convert the data and import it into Milvus 2.x.

**Note:** We’ll simplify this step and enable users to finish migration using just one command shortly. Stay tuned.

  
**Dump Command:**

```
./milvus-migration dump --config=/{YourConfigFilePath}/migration.yaml
```
  

**Load Command:**


```
./milvus-migration load --config=/{YourConfigFilePath}/migration.yaml
```

  

After the migration, the generated collection in Milvus 2.x will contain two fields: `id` and `data`. You can view more details using [Attu](https://zilliz.com/attu), an all-in-one vector database administration tool.

## Migration from FAISS to Milvus 2.x

1.  Prepare FAISS Data
    

To migrate Elasticsearch data, you should have your own FAISS data ready. To help you quickly experience the migration process, we’ve put some [FAISS test data](https://github.com/zilliztech/milvus-migration/blob/main/README_FAISS.md) in the source code of Milvus Migration.

2.  Compile and Build
    

First, download the Milvus Migration’s [source code from GitHub](https://github.com/zilliztech/milvus-migration). Then, run the following commands to compile it.


```
go get
go build
```

This step will generate an executable file named `milvus-migration`.

  

3.  Configure `migration.yaml`
    

Prepare a `migration.yaml` configuration file for FAISS migration, specifying details about the source, target, and other relevant settings. Here's an example configuration:

```
# Configuration for FAISS to Milvus 2.x migration


dumper:
  worker:
    limit: 2
    workMode: FAISS
    reader:
      bufferSize: 1024
    writer:
      bufferSize: 1024
loader:
  worker:
    limit: 2
source:
  mode: local
  local:
    FAISSFile: ./testfiles/FAISS/FAISS_ivf_flat.index


target:
  create:
    collection:
      name: test1w
      shardsNums: 2
      dim: 256
      metricType: L2
  mode: remote
  remote:
    outputDir: testfiles/output/
    cloud: aws
    endpoint: 0.0.0.0:9000
    region: ap-southeast-1
    bucket: a-bucket
    ak: minioadmin
    sk: minioadmin
    useIAM: false
    useSSL: false
    checkBucket: true
  milvus2x:
    endpoint: localhost:19530
    username: xxxxx
    password: xxxxx
```
  

For a more detailed explanation of the configuration file, refer to [this page](https://github.com/zilliztech/milvus-migration/blob/main/README_FAISS.md) on GitHub.

  

4.  Execute Migration Job
    

Like Milvus 1.x to Milvus 2.x migration, FAISS migration requires executing both the `dump` and `load` commands. These commands convert the data and import it into Milvus 2.x.

**Note:** We’ll simplify this step and enable users to finish migration using just one command shortly. Stay tuned.

**Dump Command:**
```
./milvus-migration dump --config=/{YourConfigFilePath}/migration.yaml
```

**Load Command:**

```
./milvus-migration load --config=/{YourConfigFilePath}/migration.yaml
```



  

You can view more details using [Attu](https://zilliz.com/attu), an all-in-one vector database administration tool.

## Stay tuned for future migration plans

In the future, we’ll support migration from more data sources and add more migration features, including:

-   Support migration from Redis to Milvus.
    
-   Support migration from MongoDB to Milvus.
    
-   Support resumable migration.
    
-   Simplify migration commands by merging the dump and load processes into one.
    
-   Support migration from other mainstream data sources to Milvus.
    

## Conclusion

Milvus 2.3, the latest release of Milvus, brings exciting new features and performance improvements that cater to the growing needs of data management. Migrating your data to Milvus 2.x can unlock these benefits, and the Milvus Migration project makes the migration process streamlined and easy. Give it a try, and you won't be disappointed.

***Note:** The information in this blog is based on the state of the Milvus and [Milvus Migration](https://github.com/zilliztech/milvus-migration) projects as of September 2023. Check the official [Milvus documentation](https://milvus.io/docs) for the most up-to-date information and instructions.*

How to Migrate Your Data to Milvus Seamlessly: A Comprehensive Guide


![](https://assets.zilliz.com/Exploring_NATS_878f48c848.png)

In the intricate tapestry of data processing, seamless communication is the thread that binds operations together. [Milvus](https://zilliz.com/what-is-milvus), the trailblazing [open-source vector database](https://zilliz.com/cloud), has embarked on a transformative journey with its latest feature: NATS messaging integration. In this comprehensive blog post, we'll unravel the intricacies of this integration, exploring its core features, setup process, migration benefits, and how it stacks up against its predecessor, RocksMQ.

## Understanding the role of message queues in Milvus

In Milvus’ cloud-native architecture, the message queue, or Log Broker, holds pivotal importance. It’s the backbone ensuring persistent data streams, synchronization, event notifications, and data integrity during system recoveries. Traditionally, RocksMQ was the most straightforward choice in Milvus Standalone mode, especially when compared with Pulsar and Kafka, but its limitations became evident with extensive data and complex scenarios.

Milvus 2.3 introduces NATS, a single-node MQ implementation, redefining how to manage data streams. Unlike its predecessors, NATS liberates Milvus users from performance constraints, delivering a seamless experience in handling substantial data volumes.

## What is NATS?

NATS is a distributed system connectivity technology implemented in Go. It supports various communication modes like Request-Reply and Publish-Subscribe across systems, provides data persistence through JetStream, and offers distributed capabilities through built-in RAFT. You can refer to the [NATS official website](https://nats.io/) for a more detailed understanding of NATS.

In Milvus 2.3 Standalone mode, NATS, JetStream, and PubSub provide Milvus with robust MQ capabilities.

## Enabling NATS

Milvus 2.3 offers a new control option, `mq.type`, which allows users to specify the type of MQ they want to use. To enable NATS, set `mq.type=natsmq`. If you see logs similar to the ones below after you initiate Milvus instances, you have successfully enabled NATS as the message queue.

  
```
[INFO] [dependency/factory.go:83] ["try to init mq"] [standalone=true] [mqType=natsmq]
```
  

## Configuring NATS for Milvus

NATS customization options include specifying the listening port, JetStream storage directory, maximum payload size, and initialization timeout. Fine-tuning these settings ensures optimal performance and reliability.


```
natsmq:
server: # server side configuration for natsmq.
port: 4222 # 4222 by default, Port for nats server listening.
storeDir: /var/lib/milvus/nats # /var/lib/milvus/nats by default, directory to use for JetStream storage of nats.
maxFileStore: 17179869184 # (B) 16GB by default, Maximum size of the 'file' storage.
maxPayload: 8388608 # (B) 8MB by default, Maximum number of bytes in a message payload.
maxPending: 67108864 # (B) 64MB by default, Maximum number of bytes buffered for a connection Applies to client connections.
initializeTimeout: 4000 # (ms) 4s by default, waiting for initialization of natsmq finished.
monitor:
trace: false # false by default, If true enable protocol trace log messages.
debug: false # false by default, If true enable debug log messages.
logTime: true # true by default, If set to false, log without timestamps.
logFile: /tmp/milvus/logs/nats.log # /tmp/milvus/logs/nats.log by default, Log file path relative to .. of milvus binary if use relative path.
logSizeLimit: 536870912 # (B) 512MB by default, Size in bytes after the log file rolls over to a new one.
retention:
maxAge: 4320 # (min) 3 days by default, Maximum age of any message in the P-channel.
maxBytes: # (B) None by default, How many bytes the single P-channel may contain. Removing oldest messages if the P-channel exceeds this size.
maxMsgs: # None by default, How many message the single P-channel may contain. Removing oldest messages if the P-channel exceeds this limit.
```
  

**Note:**

-   You must specify `server.port` for NATS server listening. If there is a port conflict, Milvus cannot start. Set `server.port=-1` to randomly select a port.
    
-   `storeDir` specifies the directory for JetStream storage. We recommend storing the directory in a high-performant solid-state drive (SSD) for better read/write throughput of Milvus.
    
-   `maxFileStore` sets the upper limit of JetStream storage size. Exceeding this limit will prevent further data writing.
    
-   `maxPayload` limits individual message size. You should keep it above 5MB to avoid any write rejections.
    
-   `initializeTimeout`controls NATS server startup timeout.
    
-   `monitor` configures NATS’ independent logs.
    
-   `retention` controls the retention mechanism of NATS messages.
    

For more information, refer to [NATS official documentation](https://docs.nats.io/running-a-nats-service/configuration).

## Migrating from RocksMQ to NATS

Migrating from RocksMQ to NATS is a seamless process involving steps like stopping write operations, flushing data, modifying configurations, and verifying the migration through Milvus logs.

1.  Before initiating the migration, stop all write operations in Milvus.
    
2.  Execute the `FlushALL` operation in Milvus and wait for its completion. This step ensures that all pending data is flushed and the system is ready for shutdown.
    
3.  Modify the Milvus configuration file by setting `mq.type=natsmq` and adjusting relevant options under the `natsmq` section.
    
4.  Start the Milvus 2.3.
    
5.  Back up and clean the original data stored in the `rocksmq.path` directory. (Optional)
    

## NATS vs. RocksMQ: A Performance Showdown

### Pub/Sub Performance Testing

-   **Testing Platform:** M1 Pro Chip / Memory: 16GB
    
-   **Testing Scenario:** Subscribing and publishing random data packets to a topic repeatedly until the last published result is received.
    
-   **Results:**
    
    -   For smaller data packets (< 64kb), RocksMQ outperforms NATS regarding memory, CPU, and response speed.
    
    -   For larger data packets (> 64kb), NATS outshines RocksMQ, offering much faster response times.
    

  

| Test Type           | MQ      | op count | cost per op      | Memory cost | CPU Total Time | Storage cost |
| ------------------- | ------- | -------- | ---------------- | ----------- | -------------- | ------------ |
| 5MB\*100 Pub/Sub    | NATS    | 50       | 1.650328186 s/op | 4.29 GB     | 85.58          | 25G          |
| 5MB\*100 Pub/Sub    | RocksMQ | 50       | 2.475595131 s/op | 1.18 GB     | 81.42          | 19G          |
| 1MB\*500 Pub/Sub    | NATS    | 50       | 2.248722593 s/op | 2.60 GB     | 96.50          | 25G          |
| 1MB\*500 Pub/Sub    | RocksMQ | 50       | 2.554614279 s/op | 614.9 MB    | 80.19          | 19G          |
| 64KB\*10000 Pub/Sub | NATS    | 50       | 2.133345262 s/op | 3.29 GB     | 97.59          | 31G          |
| 64KB\*10000 Pub/Sub | RocksMQ | 50       | 3.253778195 s/op | 331.2 MB    | 134.6          | 24G          |
| 1KB\*50000 Pub/Sub  | NATS    | 50       | 2.629391004 s/op | 635.1 MB    | 179.67         | 2.6G         |
| 1KB\*50000 Pub/Sub  | RocksMQ | 50       | 0.897638581 s/op | 232.3 MB    | 60.42          | 521M         |

  

Table 1: Pub/Sub performance testing results

### Milvus Integration Testing

**Data size:** 100M

**Result:** In extensive testing with a 100 million vectors dataset, NATS showcased lower vector search and query latency.

| Metrics                                 | RocksMQ (ms) | NATS (ms) |
| --------------------------------------- | ------------ | --------- |
| Average vector search latency           | 23.55        | 20.17     |
| Vector search requests per second (RPS) | 2.95         | 3.07      |
| Average query latency                   | 7.2          | 6.74      |
| Query requests per second (RPS)         | 1.47         | 1.54      |

Table 2: Milvus integration testing results with 100m dataset

  

**Dataset: <100M**

**Result:** For datasets smaller than 100M, NATS and RocksMQ show similar performance.

## Conclusion: Empowering Milvus with NATS messaging

The integration of NATS within Milvus marks a significant stride in data processing. Whether delving into real-time analytics, machine learning applications, or any data-intensive venture, NATS empowers your projects with efficiency, reliability, and speed. As the data landscape evolves, having a robust messaging system like NATS within Milvus ensures seamless, reliable, and high-performing data communication.

Optimizing Data Communication: Milvus Embraces NATS Messaging

![](https://assets.zilliz.com/What_s_New_in_Milvus_2_3_2_and_2_3_3_d3d0db03c3.png)

In the ever-evolving landscape of vector search technologies, Milvus remains at the forefront, pushing boundaries and setting new standards. Today, we are thrilled to announce the release of Milvus 2.3.2 and 2.3.3! These updates bring many exciting features, optimizations, and improvements, enhancing system performance, flexibility, and overall user experience.

## Support for Array data types - making search results more accurate and relevant

Adding the Array data type support is a pivotal enhancement for Milvus, particularly in query filtering scenarios like intersection and union. This addition ensures that search results are not only more accurate but also more relevant. In practical terms, for instance, within the e-commerce sector, product tags stored as string Arrays allow consumers to perform advanced searches, filtering out irrelevant results.

Dive into our comprehensive [documentation](https://milvus.io/docs/array_data_type.md) for an in-depth guide on leveraging Array types in Milvus.

## Support for complex delete expressions - improving your data management

In previous versions, Milvus supported primary key deletion expressions, providing a stable and streamlined architecture. With Milvus 2.3.2 or 2.3.3, users can employ complex delete expressions, facilitating sophisticated data management tasks such as rolling cleanup of old data or GDPR compliance-driven data deletion based on user IDs.

Note: Ensure you’ve loaded collections before utilizing complex expressions. Additionally, it's important to note that the deletion process does not guarantee atomicity.

## TiKV integration - scalable metadata storage with stability

Previously relying on Etcd for metadata storage, Milvus faced limited capacity and scalability challenges in metadata storage. To address these problems, Milvus added TiKV, an open-source key-value store, as one more option for metadata storage. TiKV offers enhanced scalability, stability, and efficiency, making it an ideal solution for Milvus's evolving requirements. Starting from Milvus 2.3.2, users can seamlessly transition to TiKV for their metadata storage by modifying the configuration.

## Support for FP16 vector type - embracing machine learning efficiency

Milvus 2.3.2 and later versions now support the FP16 vector type at the interface level. FP16, or 16-bit floating point, is a data format widely used in deep learning and machine learning, providing efficient representation and calculation of numerical values. While full support for FP16 is underway, various indexes in the indexing layer require converting FP16 to FP32 during construction.

We will fully support FP16, BF16, and int8 data types in later versions of Milvus. Stay tuned.

## Significant improvement in the rolling upgrade experience - seamless transition for users

Rolling upgrade is a critical feature for distributed systems, enabling system upgrades without disrupting business services or experiencing downtimes. In the latest Milvus releases, we’ve strengthened Milvus’s rolling upgrade feature, ensuring a more streamlined and efficient transition for users’ upgrading  from version 2.2.15 to 2.3.3 and all later versions. The community has also invested in extensive testing and optimizations, reducing query impact during the upgrade to less than 5 minutes, providing users with a hassle-free experience.

## Performance optimization

In addition to introducing new features, we’ve significantly optimized the performance of Milvus in the latest two releases.

-   Minimized data copy operations for optimized data loading
    
-   Simplified large-capacity inserts using batch varchar reading
    
-   Removed unnecessary offset checks during data padding to improve recall phase performance.
    
-   Addressed high CPU consumption issues in scenarios with substantial data insertions
    

These optimizations collectively contribute to a faster and more efficient Milvus experience. Check out our monitoring dashboard for a quick glance at how Milvus improved its performance.

  

## Incompatible changes

-   Permanently deleted TimeTravel-related code.
    
-   Deprecated support for MySQL as the metadata store.
    

Refer to the [Milvus release notes](https://milvus.io/docs/release_notes.md) for more detailed information about all the new features and enhancements.

## Conclusion

With the latest Milvus 2.3.2 and 2.3.3 releases, we're committed to providing a robust, feature-rich, high-performance database solution. Explore these new features, take advantage of the optimizations, and join us on this exciting journey as we evolve Milvus to meet the demands of modern data management. Download the latest version now and experience the future of data storage with Milvus!

## Let’s keep in touch!

If you have questions or feedback about Milvus, join our [Discord channel](https://discord.com/invite/8uyFbECzPX) to engage with our engineers and the community directly or join our [Milvus Community Lunch and Learn](https://discord.com/invite/RjNbk8RR4f) Every Tuesday from 12-12:30 PM PST. You’re also welcome to follow us on [Twitter](https://twitter.com/milvusio) or [LinkedIn](https://www.linkedin.com/company/the-milvus-project) for the latest news and updates about Milvus.

Revealing Milvus 2.3.2 & 2.3.3: Support for Array Data Types, Complex Delete, TiKV Integration, and More


![](https://assets.zilliz.com/Exploring_M_Map_5086d652bd.png)

[Milvus](https://zilliz.com/what-is-milvus) is the fastest solution in open-source [vector databases](https://zilliz.com/blog/what-is-a-real-vector-database), catering to users with intensive performance requirements. However, the diversity of user needs mirrors the data they work with. Some prioritize budget-friendly solutions and expansive storage over sheer speed. Understanding this spectrum of demands, Milvus introduces the MMap feature, redefining how we handle large data volumes while promising cost efficiency without sacrificing functionality.

## What is MMap?

MMap, short for memory-mapped files, bridges the gap between files and memory within operating systems. This technology allows Milvus to map large files directly into the system's memory space, transforming files into contiguous memory blocks. This integration eliminates the need for explicit read or write operations, fundamentally changing how Milvus manages data. It ensures seamless access and efficient storage for large files or situations where users need to access files randomly.

## Who benefits from MMap?

Vector databases demand substantial memory capacity due to the storage requirements of vector data. With the MMap feature, processing more data within limited memory becomes a reality. However, this increased capability comes at a performance cost. The system intelligently manages memory, evicting some data based on load and usage. This eviction allows Milvus to process more data within the same memory capacity.

During our tests, we observed that with ample memory, all data resides in memory after a warm-up period, preserving system performance. However, as data volume grows, performance gradually decreases. **Therefore, we recommend the MMap feature for users less sensitive to performance fluctuations.**

## Enabling MMap in Milvus: a simple configuration

Enabling MMap in Milvus is remarkably straightforward. All you need to do is modify the `milvus.yaml` file: add the `mmapDirPath` item under the `queryNode` configuration and set a valid path as its value.

![](https://assets.zilliz.com/enabling_mmap_a2df88276b.png)

## Striking the balance: performance, storage, and system limits

Data access patterns significantly impact performance. Milvus's MMap feature optimizes data access based on locality. MMap enables Milvus to write scalar data directly to the disk for sequentially accessed data segments. Variable-length data such as strings undergoes flattening and is indexed using an offsets array in memory. This approach ensures data access locality and eliminates the overhead of storing each variable-length data separately. Optimizations for vector indexes are meticulous. MMap is selectively employed for vector data while retaining adjacency lists in memory, conserving significant memory without compromising performance.

In addition, MMap maximizes data processing by minimizing memory usage. Unlike previous Milvus versions where QueryNode copied entire datasets, MMap adopts a streamlined, copy-free streaming process during development. This optimization drastically reduces memory overhead.

**Our internal testing results show that Milvus can efficiently handle double the data volume when enabling MMap.**

## The road ahead: continuous innovation and user-centric enhancements

While the MMap feature is in its beta phase, Milvus's team is committed to continuous improvement. Future updates will refine the system's memory usage, enabling Milvus to support even more extensive data volumes on a single node. Users can anticipate more granular control over the MMap feature, enabling dynamic changes to collections and advanced field loading modes. These enhancements provide unprecedented flexibility, allowing users to tailor their data processing strategies to specific requirements.

## Conclusion: redefining data processing excellence with Milvus MMap

Milvus 2.3's MMap feature marks a significant leap in data processing technology. By striking a delicate balance between performance, cost, and system limits, Milvus empowers users to handle vast amounts of data efficiently and cost-effectively. As Milvus continues to evolve, it remains at the forefront of innovative solutions, redefining the boundaries of what's achievable in data management.

Stay tuned for more groundbreaking developments as Milvus continues its journey toward unparalleled data processing excellence.


Milvus Introduced MMap for Redefined Data Management and Increased Storage Capability



![](https://assets.zilliz.com/Vector_Databases_vs_Vector_Search_Plugins_vs_Vector_Search_Libraries_74def521ad.png)

Hey there - welcome back to Vector Database 101!

The surge in [ChatGPT](https://zilliz.com/learn/ChatGPT-Vector-Database-Prompt-as-code) and other large language models (LLMs) has driven the growth of vector search technologies, featuring specialized vector databases like [Milvus](https://zilliz.com/what-is-milvus) and [Zilliz Cloud](https://zilliz.com/cloud) alongside libraries such as [FAISS](https://zilliz.com/blog/set-up-with-facebook-ai-similarity-search-faiss) and integrated vector search plugins within conventional databases.


In our [previous series post](https://zilliz.com/learn/what-is-vector-database), we delved into the fundamentals of vector databases. In this post, we’ll continue to explore the intricate realm of vector search, comparing vector databases, vector search plugins, and vector search libraries.


## What is vector search?


[Vector search](https://zilliz.com/learn/vector-similarity-search), also known as vector similarity search, is a technique for retrieving the top-k results that are most similar or semantically related to a given query vector among an extensive collection of dense vector data. Before conducting similarity searches, we leverage neural networks to transform [unstructured data](https://zilliz.com/learn/introduction-to-unstructured-data), such as text, images, videos, and audio, into high-dimensional numerical vectors called embedding vectors. After generating embedding vectors, vector search engines compare the spatial distance between the input query vector and the vectors in the vector stores. The closer they are in space, the more similar they are.


Multiple vector search technologies are available in the market, including machine learning libraries like Python's NumPy, vector search libraries like FAISS, vector search plugins built on traditional databases, and specialized vector databases like Milvus and Zilliz Cloud.




## Vector databases vs. vector search libraries


[Specialized vector databases](https://zilliz.com/blog/what-is-a-real-vector-database) are not the only stack for similarity searches. Before the advent of vector databases, many vector searching libraries, such as FAISS, ScaNN, and HNSW, were used for vector retrieval.


Vector search libraries can help you quickly build a high-performance prototype vector search system. Taking FAISS as an example, it is open-source and developed by Meta for efficient similarity search and dense vector clustering. FAISS can handle vector collections of any size, even those that cannot be fully loaded into memory. Additionally, FAISS offers tools for evaluation and parameter tuning. Even though written in C++, FAISS provides a Python/NumPy interface.


However, vector search libraries are merely lightweight ANN libraries rather than managed solutions, and they have limited functionality. If your dataset is small and limited, these libraries can be sufficient for unstructured data processing, even for systems running in production. However, as dataset sizes increase and more users are onboarded, the scale problem becomes increasingly difficult to solve. Moreover, they don’t allow any modifications to their index data and cannot be queried during data import.


By contrast, vector databases are a more optimal solution for unstructured data storage and retrieval. They can store and query millions or even billions of vectors while providing real-time responses simultaneously; they’re highly scalable to meet users’ growing business needs.


In addition, vector databases like Milvus have much more user-friendly features for structured/semi-structured data:  cloud-nativity, multi-tenancy, scalability, etc. These features will become clear as we dive deeper into this tutorial.


They also operate in a totally different layer of abstraction from vector search libraries - vector databases are full-fledged services, while ANN libraries are meant to be integrated into the application that you’re developing. In this sense, ANN libraries are one of the many components that vector databases are built on top of, similar to how Elasticsearch is built on top of Apache Lucene.


To give an example of why this abstraction is so important, let’s look at inserting a new unstructured data element into a vector database. This is super easy in Milvus:


```
from pymilvus import Collectioncollection = Collection('book')mr = collection.insert(data)
```


It’s really as easy as that - 3 lines of code. With a library such as FAISS or ScaNN, there is, unfortunately, no easy way of doing this without manually re-creating the entire index at certain checkpoints. Even if you could, vector search libraries still lack scalability and multi-tenancy, two of the most important vector database features.


## Vector databases vs. vector search plugins for traditional databases


Great, now that we’ve established the difference between vector search libraries and vector databases, let’s take a look at how vector databases differ from **vector search plugins**.


An increasing number of traditional relational databases, and search systems such as Clickhouse and [Elasticsearch](https://zilliz.com/blog/elasticsearch-cloud-vs-zilliz) are including built-in vector search plugins. Elasticsearch 8.0, for example, includes vector insertion and ANN search functionality that can be called via restful API endpoints. The problem with vector search plugins should be clear as night and day - **these solutions do not take a full-stack approach to embedding management and vector search**. Instead, these plugins are meant to be enhancements on top of existing architectures, thereby making them limited and unoptimized. Developing an unstructured data application atop a traditional database would be like trying to fit lithium batteries and electric motors inside the frame of a gas-powered car - not a great idea!


To illustrate why this is, let’s go back to the list of features that a vector database should implement (from the first section). Vector search plugins are missing two of these features - tunability and user-friendly APIs/SDKs. I’ll continue to use Elasticsearch’s ANN engine as an example; other vector search plugins operate very similarly so I won’t go too much further into detail. Elasticsearch supports vector storage via the `dense_vector` data field type and allows for querying via the `knnsearch endpoint`:


```json
PUT index
{
"mappings": {
  "properties": {
    "image-vector": {
      "type": "dense_vector",
      "dims": 128,
      "index": true,
      "similarity": "l2_norm"
    }
  }
}
}


PUT index/_doc
{
"image-vector": [0.12, 1.34, ...]
}
```


```json
GET index/_knn_search
{
"knn": {
  "field": "image-vector",
  "query_vector": [-0.5, 9.4, ...],
  "k": 10,
  "num_candidates": 100
}
}
```


Elasticsearch's ANN plugin supports only one indexing algorithm: Hierarchical Navigable Small Worlds, also known as HNSW (I like to think that the creator was ahead of Marvel when it came to popularizing the multiverse). On top of that, only L2/Euclidean distance is supported as a distance metric. This is an okay start, but let's compare it to Milvus, a full-fledged vector database. Using `pymilvus`:


```python
>>> field1 = FieldSchema(name='id', dtype=DataType.INT64, description='int64', is_primary=True)
>>> field2 = FieldSchema(name='embedding', dtype=DataType.FLOAT_VECTOR, description='embedding', dim=128, is_primary=False)
>>> schema = CollectionSchema(fields=[field1, field2], description='hello world collection')
>>> collection = Collection(name='my_collection', data=None, schema=schema)
>>> index_params = {
       'index_type': 'IVF_FLAT',
       'params': {'nlist': 1024},
       "metric_type": 'L2'}
>>> collection.create_index('embedding', index_params)
```


```python
>>> search_param = {
       'data': vector,
       'anns_field': 'embedding',
       'param': {'metric_type': 'L2', 'params': {'nprobe': 16}},
       'limit': 10,
       'expr': 'id_field > 0'
   }
>>> results = collection.search(**search_param)
```


While both [Elasticsearch and Milvus](https://zilliz.com/comparison/elastic-vs-milvus) have methods for creating indexes, inserting embedding vectors, and performing nearest neighbor search, it’s clear from these examples that Milvus has a more intuitive vector search API (better user-facing API) and broader vector index + distance metric support (better tunability). Milvus also plans to support more vector indices and allow for querying via SQL-like statements in the future, further improving both tunability and usability.


We just blew through quite a bit of content. This section was admittedly fairly long, so for those of you who skimmed it, here’s a quick tl;dr: Milvus is better than vector search plugins because Milvus was built from the ground-up as a vector database, allowing for a richer set of features and an architecture more suited towards unstructured data.


## How to choose from different vector search technologies?


Not all vector databases are created equal; each possesses unique traits that cater to specific applications. Vector search libraries and plugins are user-friendly and ideal for handling small-scale production environments with millions of vectors. If your data size is small and you just require basic vector search functionality, these technologies are sufficient for your business.


However, a specialized vector database should be your top choice for data-intensive businesses dealing with hundreds of millions of vectors and demanding real-time responses. Milvus, for instance, effortlessly manages billions of vectors, offering lightning-fast query speeds and rich functionality. Moreover, fully managed solutions like Zilliz prove even more advantageous, liberating you from operational challenges and enabling an exclusive focus on your core business activities.


## Take another look at the Vector Database 101 courses


1. [Introduction to Unstructured Data](https://zilliz.com/blog/introduction-to-unstructured-data)
2. [What is a Vector Database?](https://zilliz.com/learn/what-is-vector-database)
3. [Comparing Vector Databases, Vector Search Libraries, and Vector Search Plugins](https://zilliz.com/learn/comparing-vector-database-vector-search-library-and-vector-search-plugin)
4. [Introduction to Milvus](https://zilliz.com/blog/introduction-to-milvus-vector-database)
5. [Milvus Quickstart](https://zilliz.com/blog/milvus-vector-database-quickstart)
6. [Introduction to Vector Similarity Search](https://zilliz.com/blog/vector-similarity-search)
7. [Vector Index Basics and the Inverted File Index](https://zilliz.com/blog/vector-index)
8. [Scalar Quantization and Product Quantization](https://zilliz.com/blog/scalar-quantization-and-product-quantization)
9. [Hierarchical Navigable Small Worlds (HNSW)](https://zilliz.com/blog/hierarchical-navigable-small-worlds-HNSW)
10. [Approximate Nearest Neighbors Oh Yeah (ANNOY)](https://zilliz.com/learn/approximate-nearest-neighbor-oh-yeah-ANNOY)
11. [Choosing the Right Vector Index for Your Project](https://zilliz.com/learn/choosing-right-vector-index-for-your-project)
12. [DiskANN and the Vamana Algorithm](https://zilliz.com/learn/DiskANN-and-the-Vamana-Algorithm)


Comparing Vector Databases, Vector Search Libraries, and Vector Search Plugins




![](https://assets.zilliz.com/Ingesting_Chaos_20231017_110103_54fe2009cb.png)


Data is being generated faster than ever before in every form imaginable. This data is the gasoline that will power a new wave of artificial intelligence applications, but these productivity enhancement engines need help ingesting this fuel. The wide range of scenarios and edge cases surrounding unstructured data makes it challenging to use in production AI systems.

For starters, there are a vast number of data sources. These export data in various file formats, each with its eccentricities. For example, how you process a PDF varies greatly depending on where it comes from. Ingesting a PDF for a securities litigation case will likely focus on textual data. In contrast, a system design specification for a rocket engineer will be full of diagrams that require visual processing. The lack of a defined schema in unstructured data further adds complexity. Even when the challenge of processing the data is overcome, the issue of ingesting it at scale remains. Files can vary significantly in size, which changes how they are processed. You can quickly process a 1MB upload on an API over HTTP, but reading in dozens of GBs from a single file requires streaming and a dedicated worker. 

Overcoming these traditional data engineering challenges is table stakes for connecting raw data to [LLMs](https://zilliz.com/glossary/large-language-models-(llms)) via [vector databases](https://zilliz.com/learn/what-is-vector-database) like [Milvus](https://github.com/milvus-io/milvus). However, emerging use cases such as performing semantic similarity searches with the help of a vector database require new processing steps like chunking the source data, orchestrating metadata for hybrid searches, picking the suitable vector embedding model, and tuning the search parameters to determine what data to feed to the LLM. These workflows are so new that there are no established best practices for developers to follow. Instead, they must experiment to find the correct configuration and use case for their data. To speed up this process, using a vector embedding pipeline to handle the data ingestion into the vector database is invaluable. 

A vector embedding pipeline like [VectorFlow](https://github.com/dgarnitz/vectorflow) will connect your raw data to your vector database, including chunking, metadata orchestration, embedding, and upload. VectorFlow enables engineering teams to focus on the core application logic, experimenting with the different retrieval parameters generated from the embedding model, the chunking strategy, the metadata fields, and aspects of the search to see what performs best. 

In our work helping engineering teams move their [retrieval augmented generation (RAG)](https://zilliz.com/use-cases/llm-retrieval-augmented-generation) systems from prototype to production, we have observed the following approach to be successful in testing the different parameters of a RAG search pipeline:

1. Use a small set of the data you are familiar with for speed of iteration, like a few PDFs where they have relevant chunks for the search queries.
2. Make a standard set of questions and answers about that subset of the data. For example, after reading the PDFs, write a list of questions and have your team agree on the answers.
3. Create an automated evaluation system that scores how the retrieval does on each question. One way to do this is to take the answer from the RAG system and run it back through the LLM with a prompt that asks if this RAG result answers the question given the correct answer. This should be a “yes” or “no” answer. For example, if you have 25 questions on your documents, and the system gets 20 correct, you can use this to benchmark against other approaches.  
4. Ensure you use a different LLM for the evaluation than you used to encode the vector embeddings stored in the database. The evaluation LLM is typically a decoder-type of a model like GPT-4. One thing to remember is the cost of these evaluations when run repeatedly. Open-source models like Llama2 70B or the Deci AI LLM 6B, which can run on a single, smaller GPU, have roughly the same performance at a fraction of the cost.
5. Run each test multiple times and average the score to smooth out the stochasticity of the LLM.

Holding every option constant except one, you can quickly determine which parameters work best for your use case. A vector embedding pipeline like VectorFlow makes this especially easy on the ingestion side, where you can quickly try out different chunking strategies, chunk lengths, chunk overlaps, and open-source embedding models to see what leads to the best results. This is especially useful when your dataset has various file types and data sources that require custom logic. 

Once the team knows what works for its use case, the vector embedding pipeline enables them to quickly move to production without having to redesign the system to consider things like reliability and monitoring. With technologies like VectorFlow and [Milvus](https://zilliz.com/what-is-milvus), which are open-source and platform-agnostic, the team can efficiently test across different environments while complying with privacy and security requirements. 





Ingesting Chaos: The MLOps Behind Handling Unstructured Data Reliably at Scale for RAG 



![](https://assets.zilliz.com/Why_you_need_a_real_vector_database2_20230830_075505_1_4b32582c87.png)

*This article was originally published on [AIAI](https://www.aiacceleratorinstitute.com/why-and-when-do-you-need-a-purpose-built-vector-database/) and is reposted here with permission.*


The increasing popularity of [ChatGPT](https://zilliz.com/learn/ChatGPT-Vector-Database-Prompt-as-code) and other large language models (LLMs) has fueled the rise of vector search technologies, including purpose-built vector databases such as [Milvus](https://milvus.io/docs/overview.md) and [Zilliz Cloud](https://zilliz.com/cloud), vector search libraries such as [FAISS](https://zilliz.com/blog/set-up-with-facebook-ai-similarity-search-faiss), and vector search plugins integrated with traditional databases. However, choosing the best solution for your needs can be challenging. Like choosing between a high-end restaurant and a fast-food chain, selecting the right vector search technology depends on your needs and expectations.

In this post, I will provide an overview of vector search and its functioning, compare different vector search technologies, and explain why opting for a purpose-built vector database is crucial.

## What is vector search, and how does it work?

[Vector search](https://zilliz.com/blog/vector-similarity-search), also known as vector similarity search, is a technique for retrieving the top-k results that are most similar or semantically related to a given query vector among an extensive collection of dense vector data.

Before conducting similarity searches, we leverage neural networks to transform [unstructured data](https://zilliz.com/blog/introduction-to-unstructured-data), such as text, images, videos, and audio, into high-dimensional numerical vectors called embedding vectors. For example, we can use the pre-trained ResNet-50 convolutional neural network to transform a bird image into a collection of embeddings with 2,048 dimensions. Here, we list the first three and last three vector elements: `[0.1392, 0.3572, 0.1988, ..., 0.2888, 0.6611, 0.2909]`.

![A bird image by Patrice Bouchard](https://assets.zilliz.com/bird_image_4a1be18f99.png)

After generating embedding vectors, vector search engines compare the spatial distance between the input query vector and the vectors in the vector stores. The closer they are in space, the more similar they are. 

![Embedding arithmetic](https://assets.zilliz.com/Frame_3732_20230510_073643_25f985523e.png)

## Popular vector search technologies 

Multiple vector search technologies are available in the market, including machine learning libraries like Python's NumPy, vector search libraries like FAISS, vector search plugins built on traditional databases, and specialized vector databases like Milvus and Zilliz Cloud. 

### Machine learning libraries 

Using machine learning libraries is the easiest way to implement vector searches. For instance, we can use Python's NumPy to implement a nearest neighbor algorithm in less than 20 lines of code.

```
import numpy as np

# Function to calculate euclidean distance
def euclidean_distance(a, b):
return np.linalg.norm(a - b)

# Function to perform knn
def knn(data, target, k):
# Calculate distances between target and all points in the data
distances = [euclidean_distance(d, target) for d in data]
# Combine distances with data indices
distances = np.array(list(zip(distances, np.arange(len(data)))))

# Sort by distance
sorted_distances = distances[distances[:, 0].argsort()]

# Get the top k closest indices
closest_k_indices = sorted_distances[:k, 1].astype(int)

# Return the top k closest vectors
return data[closest_k_indices]
```

We can generate 100 two-dimensional vectors and find the nearest neighbor to the vector [0.5, 0.5].

```
# Define some 2D vectors
data = np.random.rand(100, 2)

# Define a target vector
target = np.array([0.5, 0.5])

# Define k
k = 3

# Perform knn
closest_vectors = knn(data, target, k)

# Print the result
print("The closest vectors are:")
print(closest_vectors)
```

Machine learning libraries, such as Python's NumPy, offer great flexibility at a low cost. However, they do have some limitations. For instance, they can only handle a small amount of data and do not ensure data persistence.

I only recommend using NumPy or other machine learning libraries for vector search when:

- You need quick prototyping.
- You don't care about data persistence.
- Your data size is under one million, and you do not require scalar filtering.
- You do not need high performance.

### Vector search libraries

Vector search libraries can help you quickly build a high-performance prototype vector search system. FAISS is a typical example. It is open-source and developed by Meta for efficient similarity search and dense vector clustering. FAISS can handle vector collections of any size, even those that cannot be fully loaded into memory. Additionally, FAISS offers tools for evaluation and parameter tuning. Even though written in C++, FAISS provides a Python/NumPy interface.

Below is the code for an example vector search based on FAISS:

```
import numpy as np
import faiss

# Generate some example data
dimension = 64 # dimension of the vector space
database_size = 10000 # size of the database
query_size = 100 # number of queries to perform
np.random.seed(123) # make the random numbers predictable

# Generating vectors to index in the database (db_vectors)
db_vectors = np.random.random((database_size, dimension)).astype('float32')

# Generating vectors for query (query_vectors)
query_vectors = np.random.random((query_size, dimension)).astype('float32')

# Building the index
index = faiss.IndexFlatL2(dimension) # using the L2 distance metric
print(index.is_trained) # should return True

# Adding vectors to the index
index.add(db_vectors)
print(index.ntotal) # should return database_size (10000)

# Perform a search
k = 4 # we want to see 4 nearest neighbors
distances, indices = index.search(query_vectors, k)

# Print the results
print("Indices of nearest neighbors: \n", indices)
print("\nL2 distances to the nearest neighbors: \n", distances)
```

Vector search libraries such as FAISS are easy to use and fast enough to handle small-scale production environments with millions of vectors. You can enhance their query performance by utilizing quantization and GPUs and reducing data dimensions.

However, these libraries have some limitations when used in production. For example, FAISS does not support real-time data addition and deletion, remote calls, multiple languages, scalar filtering, scalability, or disaster recovery.

### Different types of vector databases 

Vector databases have emerged to address the limitations of the libraries above, providing a more comprehensive and practical solution for production applications.

Four types of vector databases are available on the battlefield:

- Existing relational or columnar databases that incorporate a vector search plugin. PG Vector is an example. 
- Traditional inverted index search engines with support for dense vector indexing. [ElasticSearch](https://zilliz.com/comparison/elastic-vs-milvus) is an example.
- Lightweight vector databases built on vector search libraries. Chroma is an example.
- **Purpose-built vector databases**. This type of database is specifically designed and optimized for vector searching from the bottom up. Purpose-built vector databases typically offer more advanced features, including distributed computing, disaster recovery, and data persistence. [Milvus](https://zilliz.com/what-is-milvus) is a primary example.

Not all vector databases are created equal. Each stack has unique advantages and limitations, making them more or less suitable for different applications. 

I prefer specialized vector databases over other solutions because they are the most efficient and convenient option, offering numerous unique benefits. In the following sections, I will use Milvus as an example to explain the reasons for my preference. 

## Key benefits of purpose-built vector databases

[Milvus](https://milvus.io/) is an open-source, distributed, purpose-built vector database that can store, index, manage, and retrieve billions of embedding vectors. It is also one of the most popular vector databases for [LLM retrieval augmented generation](https://zilliz.com/use-cases/llm-retrieval-augmented-generation). As an exemplary instance of purpose-built vector databases, Milvus shares many unique advantages with its counterparts.

### Data Persistence and Cost-Effective Storage

While preventing data loss is the minimum requirement for a database, many single-machine and lightweight vector databases do not prioritize data reliability. By contrast, purpose-built distributed vector databases like [Milvus](https://zilliz.com/what-is-milvus) prioritize system resilience, scalability, and data persistence by separating storage and computation. 

Moreover, most vector databases that utilize approximate nearest neighbor (ANN) indexes need a lot of memory to perform vector searching, as they load ANN indexes purely into memory. However, Milvus supports disk indexes, making storage over ten times more cost-effective than in-memory indexes.

### Optimal Query Performance 

A specialized vector database provides optimal query performance compared to other vector search options. For example, Milvus is ten times faster at handling queries than vector search plugins. Milvus uses the [ANN algorithm](https://zilliz.com/glossary/anns) instead of the KNN brutal search algorithm for faster vector searching. Additionally, it shards its indexes, reducing the time it takes to construct an index as the data volume increases. This approach enables Milvus to easily handle billions of vectors with real-time data additions and deletions. In contrast, other vector search add-ons are only suitable for scenarios with fewer than tens of millions of data and infrequent additions and deletions.

Milvus also supports GPU acceleration. Internal testing shows that GPU-accelerated vector indexing can achieve 10,000+ QPS when searching tens of millions of data, which is at least ten times faster than traditional CPU indexing for single-machine query performance.

### System Reliability 

Many applications use vector databases for online queries that require low query latency and high throughput. These applications demand single-machine failover at the minute level, and some even require cross-region disaster recovery for critical scenarios. Traditional replication strategies based on Raft/Paxos suffer from serious resource waste and need help to pre-shard the data, leading to poor reliability. In contrast, Milvus has a distributed architecture that leverages K8s message queues for high availability, reducing recovery time and saving resources. 

### Operability and Observability

To better serve enterprise users, vector databases must offer a range of enterprise-level features for better operability and observability. Milvus supports multiple deployment methods, including K8s Operator and Helm chart, docker-compose, and pip install, making it accessible to users with different needs. Milvus also provides a monitoring and alarm system based on Grafana, Prometheus, and Loki, improving its observability. With a distributed cloud-native architecture, Milvus is the industry's first vector database to support multi-tenant isolation, RBAC, quota limiting, and rolling upgrades. All of these approaches make managing and monitoring Milvus much simpler.

## Getting started with Milvus in 3 simple steps within 10 minutes 

Building a vector database is a complex task, but using one is as simple as using Numpy and FAISS. Even students unfamiliar with AI can implement vector search based on Milvus in just ten minutes. To experience highly scalable and high-performance vector search services, follow these three steps:

- Deploy Milvus on your server with the help of the [Milvus deployment document](https://milvus.io/docs/install_standalone-docker.md).
- Implement vector search with just 50 lines of code by referring to the [Hello Milvus document](https://milvus.io/docs/example_code.md).
- Explore the [example documents of Towhee](https://github.com/towhee-io/examples/) to gain insight into popular [use cases of vector databases](https://zilliz.com/use-cases).


Why and When Do You Need a Purpose-Built Vector Database?


![](https://assets.zilliz.com/Milvus_2_3_Milvus_io_2e3b0eb55c.jpeg)

Exciting news! After eight months of concerted effort, we're thrilled to announce the release of Milvus 2.3, a milestone version that brings numerous highly anticipated features, including support for GPU, Arm64, upsert, change data capture, ScaNN index, and MMap technology. Milvus 2.3 also introduces improved query performance, more robust load balancing and scheduling, and better observability and operability. 

Join me to look at these new features and enhancements and learn how you can benefit from this release. 

## Support for GPU index that leads to 3-10 times faster in QPS 

GPU index is a highly anticipated feature in the Milvus community. Thanks to a great collaboration with the Nvidia engineers, Milvus 2.3 has supported GPU indexing with the robust RAFT algorithm added to Knowhere, the Milvus index engine. With GPU support, Milvus 2.3 is more than three times faster in QPS than older versions using the CPU HNSW index and almost ten times faster for specific datasets that require heavy computation. 

## Arm64 support to accommodate growing user demand

Arm CPUs are becoming increasingly popular among cloud providers and developers. To meet this growing demand, Milvus now provides Docker images for the ARM64 architecture. With this new CPU support, MacOS users can build their applications with Milvus more seamlessly. 

## Upsert support for better user experience 

Milvus 2.3 introduces a notable enhancement by supporting the upsert operation. This new functionality allows users to update or insert data seamlessly and empowers them to perform both operations in a single request through the Upsert interface. This feature streamlines data management and brings efficiency to the table.

**Note**:
- The upsert feature does not apply to auto-increment IDs.
- Upsert is implemented as a combination of `delete` and `insert`, which may result in some performance loss. We recommend using `insert` if you use Milvus in write-heavy scenarios.

## Range search for more accurate results 
Milvus 2.3 allows users to specify the distance between the input vector and the vectors stored in Milvus during a query. Milvus then returns all matching results within the set range. Below is an example of specifying the search distance using the range search feature.

```
// add radius and range_filter to params in search_params
search_params = {"params": {"nprobe": 10, "radius": 10, "range_filter" : 20}, "metric_type": "L2"}
res = collection.search(
vectors, "float_vector", search_params, topK,
"int64 > 100", output_fields=["int64", "float"]
)
```

In this example, the user requires Milvus to return vectors within a distance of 10 to 20 units from the input vector.

**Note**: Different distance metrics vary in how they calculate distances, resulting in distinct value ranges and sorting strategies. Therefore, it is essential to understand their characteristics before using the range search feature.

## ScaNN index for faster query speed 

Milvus 2.3 now supports the ScaNN index, an open-source [approximate nearest neighbor (ANN)](https://zilliz.com/glossary/anns) index developed by Google. ScaNN index has demonstrated superior performance in various benchmarks, outperforming HNSW by around 20% and being approximately seven times faster than IVFFlat. With the support for the ScaNN index, Milvus achieves much faster query speed compared to older versions.

## Growing index for stable and better query performance 

Milvus includes two categories of data: indexed data and streaming data. Milvus can use indexes to search indexed data quickly but can only brutely search streaming data row by row, which can impact performance. Milvus 2.3 introduces the Growing Index, which automatically creates real-time indexes for streaming data to improve query performance.

## Iterator for data retrieval in batches 

In Milvus 2.3, Pymilvus has introduced an iterator interface that allows users to retrieve more than 16,384 entities in a search or range search. This feature is handy when users need to export tens of thousands or even more vectors in batches.

## Support for MMap for increased capacity

MMap is a UNIX system call used to map files and other objects into memory. Milvus 2.3 supports MMap, which enables users to load data onto local disks and map it to memory, thereby increasing single-machine capacity.

Our testing results indicate that using MMap technology, Milvus can double its data capacity while limiting performance degradation to within 20%. This approach significantly reduces overall costs, making it particularly beneficial for users on a tight budget who do not mind compromising performance.

## CDC support for higher system availability 

Change Data Capture (CDC) is a commonly used feature in database systems that captures and replicates data changes to a designated destination. With the CDC feature, Milvus 2.3 enables users to synchronize data across data centers, back up incremental data, and seamlessly migrate data, making the system more available. 

In addition to the features above, Milvus 2.3 introduces a count interface to accurately calculate the number of rows of data stored in a collection in real-time, supports the Cosine metric to measure vector distance, and more operations on JSON arrays. For more features and detailed information, refer to [Milvus 2.3 release notes](https://milvus.io/docs/release_notes.md). 

## Enhancements and bug fixes 

In addition to new features, Milvus 2.3 includes many improvements and bug fixes for older versions.

### Improved performance for data filtering

Milvus performs scalar filtering before vector searching in hybrid scalar and vector data queries to achieve more accurate results. However, the indexing performance may decline if the user has filtered out too much data after scalar filtering. In Milvus 2.3, we optimized the filtering strategy of HNSW to address this issue, resulting in improved query performance.

### Increased multi-core CPU usage 

Approximate nearest search (ANN) is a computationally intensive task that requires massive CPU resources. In previous releases, Milvus could only utilize around 70% of the available multi-core CPU resources. However, with the latest release, Milvus has overcome this limitation and can fully utilize all available multi-core CPU resources, resulting in improved query performance and reduced resource waste.

### Refactored QueryNode

QueryNode is a crucial component in Milvus that is responsible for vector searching. However, in older versions, QueryNode had complex states, duplicate message queues, an unorganized code structure, and non-intuitive error messages.

In Milvus 2.3, we've upgraded QueryNode by introducing a stateless code structure and removing the message queue for deleting data. These updates result in less resource waste and faster and more stable vector searching.

### Enhanced message queues based on NATS

We built Milvus on a log-based architecture, and in previous versions, we used Pulsar and Kafka as the core log brokers. However, this combination faced three key challenges:

- It was unstable in multi-topic situations.
- It consumed resources when idle and struggled to deduplicate messages.
- Pulsar and Kafka are closely tied to the Java ecosystem, so their community rarely maintains and updates their Go SDKs.

To address these problems, we have combined NATS and Bookeeper as our new log broker for Milvus, which fits users' needs better.

### Optimized load balancer

Milvus 2.3 has adopted a more flexible load-balancing algorithm based on the system's real loads. This optimized algorithm lets users quickly detect node failures and unbalanced loads and adjust schedulings accordingly. According to our testing results, Milvus 2.3 can detect faults, unbalanced load, abnormal node status, and other events within seconds and make adjustments promptly.

For more information about Milvus 2.3, refer to [Milvus 2.3 release notes](https://milvus.io/docs/release_notes.md). 

## Tool upgrades

We have also upgraded Birdwatcher and Attu, two valuable tools for operating and maintaining Milvus, along with Milvus 2.3.

### Birdwatcher update 

We've upgraded [Birdwatcher](https://github.com/milvus-io/birdwatcher), the debug tool of Milvus, introducing numerous features and enhancements, including: 
- RESTful API for seamless integration with other diagnostic systems.
- PProf command support to facilitate integration with the Go pprof tool.
- Storage usage analysis capabilities.
- Efficient log analysis functionality.
- Support for viewing and modifying configurations in etcd.

### Attu update 

We’ve launched a brand-new interface for [Attu](https://zilliz.com/attu), an all-in-one vector database administration tool. The new interface has a more straightforward design and is easier to understand.

![](https://assets.zilliz.com/Attu_s_new_interface_e24dd0d670.png)

For more details, refer to [Milvus 2.3 release notes](https://milvus.io/docs/release_notes.md). 

## Let’s keep in touch!

If you have questions or feedback about Milvus, please don't hesitate to contact us through [Twitter](https://twitter.com/milvusio) or [LinkedIn](https://www.linkedin.com/company/the-milvus-project). You're also welcome to join our [Slack channel](https://milvus.io/slack/) to chat with our engineers and the community directly or check out our [Tuesday office hours](https://us02web.zoom.us/meeting/register/tZ0pcO6vrzsuEtVAuGTpNdb6lGnsPBzGfQ1T#/registration)!


Smarter Retrieval for RAG: Late Chunking with Jina Embeddings v2 and Milvus