Data & Architecture
From Data-at-Rest to Data-in-Motion
The traditional approach to data storage is based on the concept of ‘data-at-rest’. Applications store the current state of the world in a relational database and query it when necessary. However, in the era of IoT, real-time financial transactions, and personalized e-commerce, the current *status* of data is often less important than the *flow* and the change (the events) that led to that status. This insight has led to the massive rise of Event-Driven Architecture (EDA), with Apache Kafka as the undisputed beating heart of modern enterprise systems.
In an Event-Driven architecture, microservices do not communicate directly with each other (leading to tight couplings and failure under overload), but instead publish events to a central stream. This enables applications to respond to data in motion in real-time, independently, and scalably.
What Makes Apache Kafka Unique?
Apache Kafka, originally developed at LinkedIn, is fundamentally different from traditional message queues such as RabbitMQ or ActiveMQ. Whereas a message queue deletes a message as soon as it has been processed by a consumer, Kafka functions as a distributed ‘append-only’ log. Events (such as ‘Order #123 placed’) are stored chronologically and immutably on disk in so-called ‘topics’.
This design offers unprecedented benefits. Because messages are retained for a configured retention period (days, weeks, or even indefinitely), multiple, independent applications (‘consumers’) can read the same data stream at their own pace. If a new machine learning application is deployed today, it can play back the entire history of the Kafka stream from the beginning (‘offset 0’) to train its model. This principle, known as Event Sourcing, guarantees that not a single piece of business logic is lost.
Scalability and Partitions
The enormous scalability of Kafka — it can process millions of messages per second — is achieved through ‘Partitioning’. A topic is divided into multiple partitions, which are distributed across multiple servers (brokers) in the cluster. Kafka guarantees that the order of events is preserved within a specific partition by using a ‘Partition Key’ (for example, a Customer ID). This means that all orders, payments, and returns from customer X are always processed in chronological order by the same consumer, while the data of thousands of other customers is streamed in parallel over the network.
Kafka Streams and Real-time Data Processing
Storing events is only the first step. The true power emerges with ‘Stream Processing’ via frameworks such as Kafka Streams or Apache Flink. This allows developers to write continuous queries that directly intervene incoming data streams, without the data having to be written to a database first.
Consider a fraud detection system at a bank: a Kafka Streams application can aggregate a stream of credit card transactions in real time (e.g., “count the number of transactions on this card in the last 5 minutes”). As soon as the threshold is exceeded, the stream immediately generates a new ‘PossibleFraud’ event, whereupon another system immediately blocks the account. All of this happens within milliseconds. For more context regarding the architectural shift towards microservices and distributed systems, you can see the background articles on AG Connect consult.
Domain-Driven Design (DDD): The Bridge Between Complex Business Logic and Code
