Event-Driven Architectures with Apache Kafka

Introduction

Event-driven architecture has become a go-to pattern for systems that need to stay responsive as they grow. Instead of services calling each other directly, they publish and react to events — which makes them easier to scale independently and harder to knock over when one part has a bad day. Kafka sits at the center of this for a lot of teams: it handles high throughput, survives failures, and doesn't force producers and consumers to know about each other. This post walks through how that actually works, using a notification system as the example.

Kafka: The Heartbeat of Event-Driven Architecture

Event-driven architecture is all about components reacting to events — significant changes in state that need to be communicated across different parts of a system. Kafka, with its high-throughput, fault-tolerant, and scalable nature, excels as the messaging backbone that efficiently distributes these events.

The Kafka Mechanism

In an event-driven architecture, Kafka serves as the central hub for events. Producers publish events to Kafka topics, and consumers subscribe to these topics to process the events. This decoupled mechanism allows for high flexibility and scalability, as services can independently produce, consume, and react to events in real-time.

Generic Example: Notification System

Consider a notification system where various services need to send notifications based on different triggers, such as a new user registration, a purchase completion, or a system alert.

Kafka in Action

1. Event Production: Each service sends events to a Kafka topic whenever a trigger condition is met. For instance, the user service publishes a "New User Registered" event to the user_events topic.

2. Event Consumption: A notification service consumes events from these topics. Based on the event type and content, it processes and sends out appropriate notifications via email, SMS, or in-app messages.

3. Scalability and Flexibility: As the system grows, more services can be added that produce or consume events. Kafka's scalability ensures that the system can handle increasing volumes of events without a hitch.

Implementing Kafka with Kotlin

To bring this example to life, let's sketch out simple Kotlin code snippets for a Kafka producer and consumer within this notification system.

Kotlin Producer for User Registration Events

import org.apache.kafka.clients.producer.KafkaProducer
import org.apache.kafka.clients.producer.ProducerRecord
import org.apache.kafka.common.serialization.StringSerializer
import java.util.*

fun main() {
    val props = Properties().apply {
        put("bootstrap.servers", "localhost:9092")
        put("key.serializer", StringSerializer::class.java.name)
        put("value.serializer", StringSerializer::class.java.name)
    }

    KafkaProducer<String, String>(props).use { producer ->
        val topic = "user_events"
        val event = "New User Registered: UserID"

        producer.send(ProducerRecord(topic, "UserID", event))
    }
}

Kotlin Consumer for Processing Notifications

import org.apache.kafka.clients.consumer.KafkaConsumer
import org.apache.kafka.common.serialization.StringDeserializer
import java.time.Duration
import java.util.*

fun main() {
    val props = Properties().apply {
        put("bootstrap.servers", "localhost:9092")
        put("group.id", "notification-service")
        put("key.deserializer", StringDeserializer::class.java.name)
        put("value.deserializer", StringDeserializer::class.java.name)
    }

    KafkaConsumer<String, String>(props).use { consumer ->
        consumer.subscribe(listOf("user_events"))

        while (true) {
            val records = consumer.poll(Duration.ofMillis(100))
            for (record in records) {
                val event = record.value()
                // Process and send notification based on the event
            }
        }
    }
}

Conclusion

Kafka earns its place in event-driven systems because it solves a real coordination problem: how do you get many services talking to each other without them becoming tightly coupled? The producer/consumer model, combined with durable topics, means you can add consumers without touching the producer and replay events when something downstream breaks. For a notification system like this one, that translates directly to reliability and flexibility as the number of event types grows.