The vector database for RAG: retrieval that holds up in production

Milvus is an open-source vector database for RAG: keyword, grep-style and semantic search, tenant filters, in-database reranking and disk indexes in one Apache-2.0 package, from a laptop to billions of chunks.

Build RAG with the Milvus Quickstart Prefer managed? Try Zilliz Cloud free

40K+ GitHub stars, Apache-2.0 open source
BM25 full-text search built in
Meituan: 10B+ vectors, RAG to agent memory
Read AI: sub 20-50 ms retrieval

Milvus covers the retrieval half of a RAG pipeline.

Milvus and Zilliz Cloud adopters

Read.ai
OpenEvidence
Filevine
Notta.ai
Airtable
Salesforce
NVIDIA
IBM

Do I need a vector database for RAG? What it has to do well

Keyword plus meaning

Built-in BM25 full-text search fuses with dense vectors in one hybrid search, so one vector store for RAG retrieves exact IDs and paraphrased questions alike.

Metadata filtering, multi-tenant RAG

Filter expressions (==, in, ranges and JSON paths) run with vector search in one request. Include the correct tenant filter on every request when sharing a collection through a partition key.

Answers stay current

Upsert replaces chunks by primary key, deletes remove them, and per-request consistency levels decide how fresh each read must be. No second index to keep in sync.

Large knowledge bases, smaller bill

Disk-based indexes, mmap and tiered storage keep most of a large knowledge base on SSD or object storage instead of RAM.

10B+vectors on self-run open-source Milvus at Meituan, across 100+ use casesView source

Sub 20-50 msretrieval latency at Read AI for millions of monthly usersView source

Hybrid search, RAG reranking and disk indexes

Hybrid search for RAG, plus grep

RAG questions mix meaning with exact strings: error codes, SKUs, clause numbers. Hybrid search for RAG runs dense and BM25 retrieval in one hybrid_search request, and pattern filters narrow both routes.

Fuse dense and BM25 results with RRF or weighted rankers
Regex filters (=~, !~, RE2) on VARCHAR, JSON strings and arrays; LIKE accelerated by NGRAM indexes
TEXT_MATCH, PHRASE_MATCH and TEXT_MATCH_FUZZY for term, phrase and typo-tolerant matching

RAG reranking pipelines with Function Chain

Since Milvus 3.0, a Function Chain runs an ordered rerank pipeline inside the search request, per segment, per query node and after the global merge, with no client-side requery round trip.

Blend vector score with freshness (decay) and popularity (weighted combine)
Call a model reranker such as Cohere, Voyage AI, vLLM or TEI
Sort and trim candidates; works in search and hybrid search

Learning to rerank

Train an XGBoost ranking model offline on relevance or click labels, register it with Milvus, and let a Function Chain score recalled chunks.

Features come from scalar fields: recency, source authority, usage counts
Native XGBoost scoring runs on query nodes, next to the data
Rule-based options too: boost ranker and decay rankers

Disk indexes to cut cost

Not every chunk needs to live in memory: HNSW for the hot set, disk and quantized indexes for the rest. Pick the footprint for your latency budget.

DiskANN keeps the graph and raw vectors on SSD; AISAQ moves nearly everything to SSD
mmap and tiered storage load data from disk or object storage on demand
Quantized indexes: IVF_RABITQ (up to 1:32 compression), IVF_SQ8, IVF_PQ

How to choose a vector database for RAG: a five-point checklist

How to choose a vector database for RAG: what to check, why it matters, and the Milvus answer (milvus.io/docs)
What to check	Why it matters for RAG	Milvus
Hybrid search (keyword + vector)	Error codes, SKUs and clause numbers miss with embeddings alone	BM25 + dense hybrid_search; regex, LIKE, TEXT_MATCH filters
Metadata filtering and multi-tenancy	Each user must see only their documents, at query time	Filter expressions inside ANN; partition key for millions of tenants
Reranking	Top-k order decides what the LLM reads	RRF / weighted rankers, model rankers, Function Chain, XGBoost scoring
Freshness	Stale chunks produce confident wrong answers	Upsert, delete, consistency levels
Scale, cost, managed vs self-hosted	Pilots start small; production reaches billions of chunks	Milvus Lite → Standalone → Distributed, or Zilliz Cloud; DiskANN, mmap, tiered storage

RAG vector database example: tenant-scoped retrieval

Store chunks with a tenant field, then retrieve the top matches for one customer and pass them to your LLM.

python

# pip install -U "pymilvus[milvus-lite]"
from pymilvus import MilvusClient

client = MilvusClient("rag_demo.db")  # same code against http://localhost:19530 or Zilliz Cloud
client.create_collection(collection_name="chunks", dimension=768)

# embed() is your embedding model (768 dimensions here)
client.insert(collection_name="chunks", data=[
    {"id": 1, "vector": embed("Rotate API keys under Settings > Security."), "tenant": "acme",
     "text": "Rotate API keys under Settings > Security."},
])

hits = client.search(
    collection_name="chunks",
    data=[embed("How do I rotate API keys?")],
    filter='tenant == "acme"',          # retrieve only this customer's documents
    limit=5,
    output_fields=["text"],
)
context = "\n\n".join(h["entity"]["text"] for h in hits[0])  # put into your LLM prompt

embed() is your embedding model. For BM25 + dense hybrid retrieval and regex filters, see the hybrid search and pattern matching guides.

Explore the concepts

Frequently asked questions

Which vector database is best for RAG?

One that covers retrieval end to end: keyword plus semantic search, tenant filters, reranking, storage that fits the corpus. Milvus ships all four, open source, one API from Lite to Distributed.

Do I need a dedicated vector database for RAG, or is pgvector enough?

pgvector fits when the corpus is small and already lives in Postgres. A dedicated vector database for RAG pays off when you need hybrid search, filtered search at scale, tenant isolation and disk indexes; Milvus Lite keeps the start just as light.

Do I need Elasticsearch for keyword matching in RAG?

Not for retrieval. Milvus runs BM25 natively, adds regex, LIKE and TEXT_MATCH filters, and hybrid_search fuses keyword and dense results in one request.

Does Milvus handle chunking and embeddings?

Chunking stays in your loader (LangChain, LlamaIndex, Dify). Embeddings can move into the database: Embedding Functions (Milvus 2.6+) call OpenAI, Cohere, Voyage AI, Gemini and others on insert and at query time, so you store text and search with text.

Can I rerank RAG results with my own model, and how do I measure retrieval quality?

Yes. In Milvus 3.0+, a Function Chain can score candidates with your XGBoost model or call a hosted reranker inside the search request. Measure recall@k and NDCG on labeled queries before and after.

Can I run a vector database for RAG locally with LangChain or LlamaIndex?

Yes. pip install Milvus Lite for local RAG, then use the official LangChain, LlamaIndex, Haystack or Dify integrations; the docs also include a Graph RAG tutorial for multi-hop questions. The same code later targets a cluster or Zilliz Cloud.

Put a production RAG retrieval layer behind your LLM

Start with Milvus Lite in a notebook, plug it into LangChain or LlamaIndex, then add hybrid search, metadata filtering, rerank chains and disk indexes as your corpus grows.

Build RAG with the Milvus Quickstart Prefer managed? Try Zilliz Cloud free

Sources and technical documentation (45)