Event-Driven Architecture with Apache Kafka: Principles and Practical Patterns

We have all built systems where one service calls another, which calls a third, which calls a database, and the whole chain collapses because a minor payment gateway went down for three seconds. Synchronous HTTP calls look clean on a whiteboard, but they fail in production. The moment you introduce network boundaries, temporal coupling becomes your worst enemy. If Service A must talk to Service B in real-time to complete a transaction, Service A is only as reliable as Service B.

Event-driven architecture changes this dynamic. Instead of asking services to perform actions, we publish facts. Something happened, and anyone who cares can listen. Apache Kafka has become the default engine for this shift. It is not just a message queue; it is a distributed commit log designed to handle massive throughput while keeping services decoupled.

The Core Shift: Command vs. Event

To build these systems without running into constant consistency issues, it helps to document your design choices in architecture decision records. First, you have to understand the difference between a command and an event.

A command is an instruction. "CreateInvoice" or "SendEmail" are commands. They have intent, they expect a response, and they usually target a specific recipient. If the recipient is down or slow, the command fails or hangs. This creates a tight coupling between the sender and the receiver.

An event is a statement of fact. "InvoiceCreated" or "EmailSent" are events. They represent things that have already happened in the past. You cannot change the past, and the publisher does not care who reacts to it. The publisher simply writes this fact to Kafka and moves on. If the email service is down for maintenance, the event sits in Kafka, waiting for the service to wake up and process it. This decoupling of time is the real strength of event-driven systems.

Kafka's Design Principles

At its core, Kafka is simple. It is an append-only log structured on disk. When a producer sends a message, Kafka appends it to the end of the log. There are no complex routing tables, no transient message states, and no automatic deletions upon delivery.

This simplicity is why Kafka scales so well. Traditional message brokers track which consumer has read which message by updating database indexes or state tables. Kafka does not do this. Consumers track their own position using an offset: a simple integer pointing to their place in the log. Reading a message is just a sequential disk read, which modern operating systems optimize heavily using page caches.

But a single log cannot scale infinitely. Kafka solves this by splitting topics into partitions. Partitions are the unit of scalability. Each partition is a physical log file distributed across different brokers in the cluster. If you want to scale your writes, you increase partitions.

However, partitions introduce a major constraint: order is only guaranteed within a single partition. If you publish message A and message B to different partitions, Kafka makes no promises about which one a consumer will see first. If order matters, such as processing deposits before withdrawals for a specific bank account, you must route those messages to the same partition using a partition key. Usually, this key is something like an account_id or user_id.

Pattern 1: The Outbox Pattern

One of the hardest problems in distributed systems is the dual-write problem. Imagine a user signs up. You need to save the user to your PostgreSQL database and publish a UserRegistered event to Kafka.

If you save to the database first and the network drops before you publish to Kafka, your system is inconsistent. If you publish to Kafka first and the database transaction fails, you have announced a user that does not exist.

The transactional outbox pattern solves this. Instead of writing to two different systems, you write to two tables in the same database within a single transaction. You save the user to the users table, and write the event payload to an outbox table. Since both writes happen in the same Postgres transaction, they either both succeed or both fail.

┌──────────────────────────────────────────────┐
│             Database Transaction             │
│                                              │
│  ┌──────────────┐      ┌────────────────┐    │
│  │ users Table  │      │  outbox Table  │    │
│  │              │      │                │    │
│  │ Insert User  │      │  Insert Event  │    │
│  └──────────────┘      └────────────────┘    │
└──────────────────────────────────────────────┘
                       │
                       ▼ (Transaction Committed)
┌──────────────────────────────────────────────┐
│          Message Relay / CDC Engine          │
│  (Reads outbox table, publishes to Kafka)    │
└──────────────────────────────────────────────┘
                       │
                       ▼
               ┌──────────────┐
               │ Kafka Topic  │
               └──────────────┘

A separate process, often a Debezium connector using Change Data Capture (CDC) or a simple polling worker, reads the outbox table, publishes the events to Kafka, and marks them as sent. This guarantees at least-once delivery without relying on slow distributed transactions.

Pattern 2: Event Sourcing and CQRS

Most traditional systems store the current state. When a user updates their address, you overwrite the old address in the database. The history is gone unless you have audit logs.

Event sourcing flips this. Instead of storing the current state, you store the sequence of events that led to that state. To find a user's current address, you read all their address change events from the beginning of time and replay them.

Kafka is an excellent fit for event sourcing because of its durability. You can configure topics with infinite retention, turning Kafka into your primary database of historical facts.

But replaying thousands of events every time you need to check a user's balance is too slow. That is where Command Query Responsibility Segregation (CQRS) comes in. You separate the write model (the event store in Kafka) from the read model. A projection worker listens to the Kafka event log, processes the events, and updates a fast read-optimized database like Redis or Elasticsearch. Your API queries the read database, while your business logic writes to the event log.

Pattern 3: Saga Pattern (Choreography)

Distributed transactions across microservices are a recipe for failure. Two-phase commit (2PC) protocols lock resources across services, destroying performance and availability. The Saga pattern offers an alternative by breaking a global transaction into a series of local transactions.

In a choreographed saga, services communicate via Kafka without a central coordinator. Let us trace an e-commerce order:

1. The Order Service creates an order in a PENDING state and publishes an OrderCreated event.
2. The Payment Service listens to OrderCreated, charges the customer, and publishes PaymentAuthorized.
3. The Inventory Service listens to PaymentAuthorized, reserves the items, and publishes InventoryReserved.
4. The Order Service listens to InventoryReserved and updates the order status to CONFIRMED.

If the Payment Service fails to charge the card, it publishes PaymentFailed. The Order Service listens to this and marks the order as CANCELLED. If inventory was already reserved, the Inventory Service would listen to PaymentFailed and release the items. These are compensating transactions. They do not roll back state in the database sense; they apply a new, corrective state to keep the system balanced.

Handling Poison Pills and Retries

Eventually, a consumer will encounter a message it cannot process. Maybe the payload is corrupted, or a required field is missing. This is a poison pill. If your consumer crashes and restarts, it will try to read the same message again, getting stuck in an infinite loop.

You need a resilient retry strategy. Do not block the main processing loop. If a message fails, catch the exception and send the message to a retry topic. A separate consumer can process the retry topic with a delay.

If the message fails multiple times in the retry topic, move it to a Dead Letter Queue (DLQ) topic. The DLQ is a holding area. You can set up alerts on the DLQ, write tools to inspect the bad messages, fix the underlying bug, and republish them to the main topic once the system is ready.

               ┌──────────────┐
               │  Main Kafka  │
               │    Topic     │
               └──────────────┘
                       │
                       ▼
               ┌──────────────┐
               │Main Consumer │
               └──────────────┘
                       │
           (Processing Fails / Error)
                       ▼
               ┌──────────────┐
               │ Retry Kafka  │
               │    Topic     │
               └──────────────┘
                       │
                       ▼
               ┌──────────────┐
               │Retry Consumer│
               └──────────────┘
                       │
              (Repeated Failures)
                       ▼
               ┌──────────────┐
               │ Dead Letter  │
               │ Queue (DLQ)  │
               └──────────────┘

Schema Evolution

In a shared event-driven system, schemas are your API contracts. If the Order Service changes the format of the OrderCreated event, every downstream service might break.

You cannot just send raw JSON without guardrails. You need a schema registry. Confluent's Schema Registry is the standard tool here. It stores schemas (typically Avro or Protobuf) and enforces compatibility rules.

When a producer attempts to publish an event, it checks the schema against the registry. If the schema breaks backward compatibility, such as by removing a required field, the registry rejects it. Downstream consumers can safely assume that the data they read matches a structure they understand.

The Partition Key Trap

Choosing the right partition key is critical. If you choose a key with low cardinality, like country_code, you will end up with hot partitions. A few partitions will process millions of messages while others sit idle. This limits your ability to scale.

Conversely, if you do not use a key, Kafka uses round-robin routing. This distributes the load perfectly but destroys message ordering. You must balance the need for ordering with the need for even distribution. A good key has high cardinality and groups related events together, such as tenant_id combined with entity_id.

Operational Reality

Kafka is powerful, but it comes with a steep operational tax. Running a self-managed Kafka cluster requires managing Zookeeper (or KRaft metadata logs), tuning JVM garbage collection, monitoring disk I/O, and handling broker failures.

Before adopting Kafka, ask yourself if you need its scale. For simple message passing, lighter alternatives like RabbitMQ, AWS SQS, or Redis Pub/Sub and Streams might be enough. But if you need an immutable, replayable log of facts to serve as the nervous system of your microservices, Kafka remains the standard. At the end of the day, developers choose tools they trust to keep their production systems running. Plan your schemas, protect your database transactions with the outbox pattern, and build robust error handling from day one.