How does a distributed log differ from a message queue?

A distributed log and a message queue are both systems for handling data streams, but they serve different purposes and have distinct design characteristics. A distributed log, such as Apache Kafka or Amazon Kinesis, is designed to persist and replicate an ordered sequence of records across multiple nodes. It emphasizes durability, strict ordering, and the ability for multiple consumers to read data at their own pace. In contrast, a message queue, like RabbitMQ or Amazon SQS, focuses on transient communication between producers and consumers, ensuring messages are processed once (or a limited number of times) and then removed. The key difference lies in their primary use cases: distributed logs excel at long-term data retention and replayability, while message queues prioritize ephemeral task delivery and decoupling services.

One major distinction is how they handle message retention and consumption. In a distributed log, messages are stored for extended periods (days, weeks, or indefinitely) and remain accessible even after being read. Consumers track their progress via offsets, allowing them to re-read messages or process historical data. For example, a fraud detection system might use a distributed log to replay transaction events for analysis. Message queues, however, typically delete messages once they’re acknowledged by a consumer. This suits scenarios like task processing, where each message represents a job (e.g., sending an email) that shouldn’t be handled multiple times. Some queues support retention policies, but this isn’t their primary focus.

Another difference is their approach to ordering and scalability. Distributed logs maintain a strict, global order of messages within a partition or topic, which is critical for use cases like event sourcing or maintaining transaction consistency. Message queues often prioritize scalability over strict ordering, distributing messages across multiple consumers in parallel. For instance, a queue might process user uploads in any order to maximize throughput, while a distributed log might enforce order for financial transactions. Additionally, distributed logs are often partitioned to scale horizontally, whereas queues rely on competing consumers or sharding. Developers choose between them based on whether they need durable, ordered data streams (distributed logs) or transient, at-least-once delivery (queues).