How do vector databases like Milvus or Weaviate handle storage of vectors and indexes under the hood (e.g., do they use memory-mapped files, proprietary storage engines, etc.)?

Vector databases like Milvus and Weaviate handle vector storage and indexing through a combination of custom storage engines, memory-mapped files, and distributed systems. The specifics vary between platforms, but both prioritize efficient data access and scalability. Here’s a breakdown of their approaches:

Storage Mechanisms Milvus uses a distributed architecture that separates storage and compute. It relies on object storage (e.g., Amazon S3, MinIO) for persisting raw vector data and metadata, while intermediate data like indexes and query results are managed using memory-mapped files. This design allows Milvus to scale horizontally by offloading large datasets to cost-effective cloud storage while keeping frequently accessed data in memory for low-latency queries. For example, when a query is executed, Milvus loads relevant index segments into memory via memory-mapped files, avoiding full-file reads. Weaviate, on the other hand, employs a custom storage engine based on an LSM (Log-Structured Merge) tree, which batches writes into immutable files and merges them in the background. This approach optimizes write-heavy workloads while enabling efficient compaction. Weaviate also supports pluggable backends like disk-based storage or S3 for snapshots, balancing performance and durability.

Indexing Strategies Both databases use approximate nearest neighbor (ANN) indexes to accelerate similarity searches. Milvus supports multiple index types (e.g., IVF, HNSW, DiskANN) and decouples index creation from storage. Indexes are built separately and loaded into memory during queries, often using memory-mapped files to reduce overhead. For instance, HNSW graphs in Milvus are stored in memory for fast traversal but can be persisted to disk when idle. Weaviate defaults to HNSW for in-memory indexing but also supports disk-based ANN alternatives like Spotify’s Annoy. Its LSM-based storage allows incremental index updates, where new vectors are added to a write-ahead log and merged into the main index during compaction. This avoids rebuilding the entire index for small updates, improving write efficiency.

Underlying Infrastructure Milvus leverages distributed systems for scalability. It uses etcd for metadata management, object storage for bulk data, and message brokers like Kafka/Pulsar for streaming updates. This modular design lets it handle large datasets across clusters while maintaining low-latency queries through memory-mapped index access. Weaviate opts for a simpler, single-node architecture with optional replication. Its LSM engine handles vector storage and indexing in a unified layer, reducing complexity for smaller deployments. Both systems use memory-mapped files to bridge disk and memory, enabling fast access without loading entire datasets into RAM. For example, Milvus might map HNSW graph segments directly from disk, while Weaviate’s LSM trees use memory mapping to accelerate compaction and querying. These choices reflect trade-offs between scalability (Milvus) and simplicity (Weaviate) while ensuring efficient vector operations.