Data Streaming vs Batch Processing at Scale

The default answer to "how do we move data between systems?" used to be a nightly batch job. ETL pipelines would run at 2 AM, shuffle gigabytes of records between databases, and by morning the warehouse was updated. This model worked fine — until businesses needed to act on data in seconds, not hours.

Batch Processing: Where It Still Wins

Batch is not dead. For workloads where data recency doesn't affect business decisions — end-of-month financial reports, weekly ML model retraining, bulk data migrations — batch remains the right tool. It's simpler to reason about, easier to debug, and far cheaper to operate. A scheduled Airflow DAG reading from PostgreSQL and writing to BigQuery costs a fraction of a Kafka cluster running 24/7.

The trap is treating batch as the default for everything, including use cases where staleness is actually harmful. A fraud detection system running on yesterday's transaction patterns is a liability. A personalisation engine surfacing last week's browsing history is irrelevant. These workloads need streaming.

When Streaming Becomes Necessary

• Sub-minute latency requirements: Fraud detection, live inventory, real-time pricing.
• Event-driven architecture: Microservices that must react to state changes in other services immediately.
• Continuous ML inference: Models that score every transaction or click as it happens.
• Audit trails: Immutable, ordered event logs that must be replayed on demand.

Apache Kafka: The Standard and Its Trade-offs

Kafka has become the de facto backbone for streaming architectures. Its durable, partitioned log model means producers and consumers are fully decoupled, and any consumer can replay the entire event history from offset zero. This makes it extraordinarily flexible — but Kafka is not trivial to operate. Partition rebalancing, consumer lag monitoring, schema evolution with Avro or Protobuf, and exactly-once delivery semantics all require dedicated expertise.

The Hybrid Architecture Pattern

Most production systems at scale run both paradigms simultaneously. Kafka handles the real-time event stream. A separate process — often Apache Flink or Spark Structured Streaming — aggregates those events into micro-batches (every 5–30 seconds) for analytical queries. The data warehouse receives hourly or daily compacted snapshots for historical reporting. Understanding which layer each query belongs to is the core skill of modern data engineering.