Vector.dev is an observability data pipeline tool that can send data to AWS S3, but it’s important to understand that Vector.dev’s AWS S3 sink is designed for standard S3 buckets and file-based data, not for AWS S3 Vector’s specialized vector storage format. The Vector.dev AWS S3 sink streams observability events like logs, metrics, and traces to S3 as files using various encodings (JSON, Avro, etc.) and compression formats, which is fundamentally different from S3 Vector’s requirement for structured vector embeddings and specialized APIs.
To use Vector.dev in an S3 Vector workflow, you would need to implement a multi-stage pipeline where Vector.dev handles data collection and preprocessing, while a separate component manages vector generation and S3 Vector ingestion. For example, Vector.dev could collect and transform raw text data, stream it to a standard S3 bucket or processing queue, then trigger a downstream process that generates embeddings using services like Amazon Bedrock and stores the resulting vectors in S3 Vector using the PutVectors API. This approach leverages Vector.dev’s strengths in data pipeline management while properly interfacing with S3 Vector’s vector-specific requirements.
A practical implementation might involve Vector.dev collecting application logs or documents, transforming and enriching the data, then sending it to Amazon SQS or Kinesis for further processing. A Lambda function or container-based application could then consume these messages, generate vector embeddings using embedding models, and store the results in S3 Vector with appropriate metadata. While this adds complexity compared to direct file storage, it allows you to leverage Vector.dev’s robust data pipeline capabilities for data collection and preprocessing while properly utilizing S3 Vector’s semantic search capabilities. Alternatively, you could extend Vector.dev with custom transforms or sinks specifically designed for S3 Vector integration, though this would require custom development and wouldn’t be available as a standard Vector.dev component.
Will Amazon S3 vectors kill vector databases or save them?
S3 vectors looks great particularly in terms of price and integration into the AWS ecosystem. So naturally, there are a lot of hot takes. I’ve seen folks on social media and in engineering circles say this could be the end of purpose-built vector databases—Milvus, Pinecone, Qdrant, and others included. Bold claim, right?
As a group of people who’s spent way too many late nights thinking about vector search, we have to admit that: S3 Vectors does bring something interesting to the table, especially around cost and integration within the AWS ecosystem. But instead of “killing” vector databases, I see it fitting into the ecosystem as a complementary piece. In fact, its real future probably lies in working with professional vector databases, not replacing them.
Check out James’ post to learn why we think that—looking at it from three angles: the tech itself, what it can and can’t do, and what it means for the market. We’ll also share S3 vectors’ strenghs and weakness and in what situations you should choose an alternative such as Milvus and Zilliz Cloud.
Will Amazon S3 Vectors Kill Vector Databases—or Save Them?
Or if you’d like to compare Amazon S3 vectors with other specialized vector databases, visit our comparison page for more details: Vector Database Comparison