Milvus architecture: a distributed vector database built on storage-compute separation
Milvus architecture is built on storage-compute separation: stateless proxies and workers over object storage that holds every segment and index. That is what makes it a distributed vector database, one that spreads vectors, writes, and search across many nodes so each can grow on its own. Add the write and search paths, quantized and disk-based indexes, and it scales to tens of billions of vectors.
- Apache-2.0, 40K+ GitHub stars
- Scales to tens of billions of vectors
- Object storage is the source of truth
- IVF_RABITQ: 1-to-32 compression
- Partition key: millions of tenants
- Salesforce
- Walmart
- eBay
- NVIDIA
- Roblox
- Filevine
- Shell
What does the Milvus architecture look like as a distributed vector database?
The Milvus cluster architecture splits into four layers that scale and recover independently. Stream processing (new writes) runs on streaming nodes; batch processing (sealed data) runs on query and data nodes, so write processing and sealed-data search can scale independently; growing-data search still runs on streaming nodes.
| Component | Layer | Role | Scale it when |
|---|---|---|---|
| Proxy (stateless) | Access | Validates requests, routes by shard, merges results | Connections or request rate grow |
| Coordinator | Control | Topology, DDL, timestamps, load balancing, task scheduling; one active instance | Not scaled out; standby for HA |
| Streaming node | Worker | Owns each shard's WAL, orders writes, searches growing data | Write throughput grows |
| Query node | Worker | Loads sealed segments and indexes, runs search | Data loaded or QPS grows |
| Data node | Worker | Compaction and index building | Index or compaction backlog grows |
| Object, WAL, meta storage | Storage | Segments and indexes; write-ahead log; schemas and checkpoints | Data volume grows |
How does storage-compute separation work in a vector database?
In Milvus, object storage is the source of truth. Every sealed segment, index file, and log snapshot lands in S3, MinIO, Azure Blob, or Google Cloud Storage. Worker nodes are stateless executors: a new query node pulls its segments from object storage, so scaling out never copies data between peers.
The write-ahead log follows the same design. Since Milvus 2.6, Woodpecker writes the WAL directly to object storage with no local disks, replacing Kafka or Pulsar; Milvus 3.0 can also run Woodpecker as its own service.
- Scale reads, writes, and indexing independently by adding only the node type you lack.
- Lose a node without losing data; the coordinator reassigns its segments or shards.
- Keep durable storage on cheap object storage while compute stays elastic.
- Isolate workloads with resource groups instead of separate clusters.
How does data flow through the Milvus architecture from insert to search?
Writes pass through the WAL into growing segments. Visibility to a particular query depends on its consistency level. Sealed segments are persisted in object storage, indexed by data nodes and loaded by query nodes; background compaction merges data and applies deletes.
- Search path: the proxy sends the request to each shard's streaming node, which searches its growing segment and fans sealed-segment work out to query nodes.
- Each node returns a local top-k; results merge per shard, then across shards at the proxy.
- Consistency is chosen per request: Strong, Bounded (default), Session or Eventually.
| Stage | What happens | Component |
|---|---|---|
| Route | Proxy validates the request, hashes the primary key to a shard (vchannel) | Proxy |
| Log | Timestamp assigned, row appended to the WAL, write acknowledged | Streaming node → WAL |
| Growing segment | Resident in memory; visibility follows the selected consistency level | Streaming node |
| Seal and flush | Segment becomes immutable and lands in object storage | Streaming node → object storage |
| Index and compact | Index built; small segments merged; deletes applied | Data node |
| Load | Sealed segment and index loaded; matching growing data released | Query node |
How does a scalable vector database grow reads, writes, and indexing?
Writes pass through proxies, streaming nodes and the WAL. Growing and sealed data serve searches under the requested consistency level. Sealed data is persisted in object storage, indexed on data nodes and loaded by query nodes, with separate scaling levers.
| Bottleneck | Lever |
|---|---|
| Insert throughput | More shards (up to 16 per collection) and streaming nodes |
| Index build lags ingest | More data nodes |
| Data exceeds query-node memory | Quantized or disk indexes, mmap, tiered storage, more query nodes |
| Search QPS | In-memory replicas on separate query nodes |
| Noisy neighbors | Resource groups that pin query nodes to collections |
| Many tenants | Partition key: one collection, millions of tenants |
Which vector quantization indexes cut memory for billion-scale vector search?
Raw float32 vectors cost dimensions × 4 bytes each, and HNSW memory usage adds graph links on top. At billions of vectors, memory is the bill, so Milvus offers vector quantization at several compression levels, most with an optional refine step that reranks candidates at higher precision to recover recall.
| Index | Vector memory footprint | When to use |
|---|---|---|
| HNSW (FP32) | Full raw vectors plus graph links | Data fits in RAM; lowest latency, highest recall |
| HNSW_SQ (SQ8, SQ6, SQ4U, FP16, BF16) | 8, 6, 4, or 16 bits per dimension; SQ8 cuts 75% vs FP32 | Keep HNSW speed with smaller memory |
| HNSW_PQ / HNSW_PRQ | PQ compresses 4–32x; PRQ adds residual quantization for accuracy | Graph search under a tight memory budget |
| IVF_SQ8 / IVF_PQ | 1 byte per dimension / PQ codes | Large batches, high throughput, cheaper builds |
| IVF_RABITQ | 1 bit per dimension: 1-to-32 compression; refine with SQ6, SQ8, FP16, BF16, or FP32 | Maximum compression; replaces IVF_FLAT or IVF_SQ8 when memory is short |
| GPU_CAGRA / GPU_IVF_PQ | Data loaded into GPU memory | High-throughput, high-recall search on GPUs |
RaBitQ (Gao & Long, SIGMOD 2024) gives a theoretical error bound for 1-bit codes; it runs faster on CPUs with AVX-512 VPOPCNTDQ.
DiskANN vs HNSW, mmap, and tiered storage: what handles data larger than memory?
When even compressed vectors outgrow RAM, move the index to local NVMe or leave cold data in object storage. The Milvus docs suggest DiskANN once only about a quarter of raw data fits in memory.
| Option | What stays in RAM | Where the rest lives | When to use |
|---|---|---|---|
| DiskANN | PQ codes, 1/8 of raw vector size by default | Vamana graph and raw vectors on NVMe SSD | Billion-scale search with stable latency |
| AISAQ (Milvus 2.6.4+) | Near-zero DRAM; optional PQ cache | All index structures on SSD | Ultra-large data on standard servers; RAG and offline search |
| mmap | Hot pages only | Fields and indexes memory-mapped from local disk | Data slightly above RAM; quantized IVF + mmap |
| Tiered storage (Milvus 2.6.4+) | Metadata plus cached hot chunks | Object storage, fetched on demand and evicted when cold | Collections beyond a node's memory or disk; warm up critical fields |
How far does Milvus scale?
Milvus standalone vs distributed is mostly a question of scale: Standalone packs the same components into one process, Distributed spreads them across nodes. Entities per collection are unlimited either way; capacity grows with segments, shards, and nodes. These defaults come from the Milvus docs and config.
| Item | Default / documented scale |
|---|---|
| Vectors | Tens of billions |
| Collections per cluster | 65,536 (maxCollectionNum, configurable) |
| Databases per cluster | 64 |
| Partitions per collection | 1,024 |
| Tenants with partition key | Millions, in one collection |
| Shards per collection | 16 |
| Vector dimensions | 32,768 |
Customer stories
Meituan
Grew self-run Milvus to 10B+ vectors and 100+ use cases in about a year, using DiskANN to keep a 2B-vector collection mostly on disk.
Read the storyKling AI
Scales storage and compute separately, using SQ8 and DiskANN to cut memory, with 10B+ vectors and 10,000+ QPS peaks in one cluster.
Read the storyExplore the concepts
- How does the Milvus architecture separate ingestion, storage, and search?
- What determines vector-database scalability?
- How do sharding and replication differ in a vector database?
- What is IVF_RABITQ, and how should you evaluate it?
- When should you use DISKANN in Milvus?
- How does mmap differ from a disk-oriented ANN index?
- What is vector quantization, and what does it trade off?
- How do you turn a Milvus workload into a capacity and cost estimate?
Frequently asked questions
What is a distributed vector database?
A vector database that shards data and spreads ingestion, indexing, and search across many nodes, so each capacity grows by adding machines. Milvus Distributed is an Apache-2.0 example built on shared object storage.
What does storage-compute separation mean for a vector database?
Durable data lives in object storage, and compute nodes are stateless caches and executors. You add query, streaming, or data nodes without moving data between them, and a failed node is replaced, not repaired.
How does data flow through the Milvus architecture from insert to search?
Insert: proxy → streaming node → WAL, then a growing segment that is searchable at once. Once sealed, the segment goes to object storage, a data node indexes it, and query nodes load it. Search: proxy → streaming nodes and query nodes → merged top-k.
How do distributed vector databases handle sharding and replication?
In Milvus, shards split a collection's write stream by primary key (up to 16 per collection), each owned by a streaming node, and proxies route by shard. Replicas are in-memory copies of loaded segments on separate query nodes that raise search QPS; data itself is replicated once, durably, in object storage.
How much memory does billion-scale vector search need?
Raw float32 vectors need dimensions × 4 bytes each before index overhead. Quantization (SQ8, PQ, RaBitQ), DiskANN, and tiered storage shrink what must sit in RAM; check sizes with milvus.io/tools/sizing.
DiskANN vs HNSW: when should I use which?
Use HNSW when raw vectors fit in memory and latency matters most. Use DiskANN when data is far larger than RAM: the graph and raw vectors sit on NVMe SSD and only PQ codes stay in memory.
What consistency levels does Milvus support?
Four, set per search or query: Strong reads the latest data; Bounded (the default) tolerates a short staleness window; Session guarantees read-your-own-writes within one client; Eventually trades freshness for the fastest reads.
Prefer not to operate the cluster yourself?
Zilliz Cloud runs managed Milvus with the same MilvusClient API; node counts, upgrades, and storage tiering are handled for you. Filevine manages over 30 billion vectors there.
Sources and technical documentation (31)
- https://milvus.io/
- https://milvus.io/docs/architecture_overview.md
- https://milvus.io/docs/woodpecker_architecture.md
- https://milvus.io/docs/release_notes.md
- https://milvus.io/docs/scaleout.md
- https://milvus.io/docs/resource_group.md
- https://milvus.io/docs/replica.md
- https://milvus.io/docs/multi_tenancy.md
- https://milvus.io/docs/index-explained.md
- https://milvus.io/docs/hnsw-sq.md
- https://milvus.io/docs/hnsw-pq.md
- https://milvus.io/docs/hnsw-prq.md
- https://milvus.io/docs/ivf-rabitq.md
- https://milvus.io/docs/diskann.md
- https://milvus.io/docs/aisaq.md
- https://milvus.io/docs/mmap.md
- https://milvus.io/docs/tiered-storage-overview.md
- https://milvus.io/docs/gpu-cagra.md
- https://milvus.io/docs/gpu-index-overview.md
- https://milvus.io/docs/gcs.md
- https://milvus.io/docs/limitations.md
- https://github.com/milvus-io/milvus/blob/v3.0.3/configs/milvus.yaml
- https://github.com/gaoj0017/RaBitQ
- https://zilliz.com/customers/meituan
- https://zilliz.com/customers/kling-ai
- https://zilliz.com/customers/filevine
- https://github.com/milvus-io/milvus
- https://github.com/milvus-io/milvus-docs/tree/v3.0.x
- https://milvus.io/docs/consistency.md
- https://milvus.io/docs/glossary.md
- https://milvus.io/docs/install-overview.md