What is micro-batching in data streaming?

Micro-batching in data streaming is a processing technique where data is grouped into small batches and processed at regular intervals, balancing the trade-offs between real-time streaming and traditional batch processing. Instead of handling each record individually as it arrives (pure streaming) or waiting to process large datasets all at once (batch), micro-batching divides the stream into tiny batches, often measured in seconds. This approach reduces overhead by amortizing the cost of processing across multiple records while maintaining near-real-time latency. For example, a system might collect data for 1–5 seconds, process the batch, then repeat, ensuring data is handled quickly without overwhelming resources.

A common example of micro-batching is Apache Spark Streaming. Spark processes data in fixed-time intervals (e.g., 2-second batches), allowing it to reuse batch processing logic while achieving low-latency results. This is useful for scenarios like aggregating metrics (e.g., counting user clicks per minute) or transforming data before storage. Micro-batching also simplifies fault tolerance: if a batch fails, the system can reprocess just that batch instead of the entire stream. Tools like Apache Flink also use micro-batching under the hood for windowed operations, such as calculating moving averages, where grouping data into small windows aligns naturally with batch boundaries.

The main trade-off with micro-batching is latency versus throughput. Smaller batches reduce latency but increase overhead due to frequent batch commits, while larger batches improve throughput at the cost of slower responses. For instance, a fraud detection system might use 1-second batches to balance timely alerts with efficient resource use. Developers should consider their latency requirements: pure streaming (e.g., Apache Kafka Streams) is better for sub-second needs, while micro-batching suits applications where a slight delay (e.g., 5–10 seconds) is acceptable for simpler scaling and error handling. It’s a practical middle ground for use cases like ETL pipelines or dashboard updates that don’t require instant results.