Messaging Communication Among Services With Kafka

Knowing about Publish and Subscribe

Communication methods among services can be achieved directly and indirectly.

• Direct communication commonly uses RPC (Remote Procedure Call), with RESTful API as the most popular choice today. The caller knows exactly who it is talking to and waits for a response.
• Indirect communication uses a messaging or publish/subscribe model. This method needs an application called a "Message Broker" to act as a mediator — the publisher sends data into the broker, and subscribers consume it on their own schedule.

The key advantage of Pub/Sub: the publisher doesn't need to know anything about who consumes its data or how they use it. It just needs to confirm the data landed in the broker. That's its entire job.

With direct communication, every time a new service needs access to data produced by another service, the producer has to be updated — who to notify, in what format, at what endpoint. This coupling compounds with every new service added to the system.

With a message broker, a new consumer simply subscribes to the relevant topic. The producer is untouched. The data flows as it always did.

That said, Pub/Sub is not without trade-offs:

• Delay. Because communication is indirect, there is latency between when a message is published and when it is processed. If a subscriber is slow or the broker is under load, that delay is unpredictable.
• No built-in status feedback. By default, the publisher has no way of knowing whether a subscriber successfully processed a message. Subscribers need to handle their own retry logic and failure cases.

Why Kafka

Kafka is one of the most widely adopted message brokers available. It is open source under the Apache 2.0 licence, which means you can use it for personal or commercial purposes at no cost. A few reasons it stands out:

• Scalable. Kafka is built to handle very high data volumes. As load grows, you scale horizontally by adding partitions and brokers.
• High throughput. Kafka processes messages quickly and consistently, outperforming many similar systems in benchmark comparisons.
• Persistence. Kafka stores messages on disk, not just in memory. If a subscriber fails mid-processing, the message is still there and can be reprocessed.
• Rich ecosystem. Kafka integrates with Java, Python, Go, Node.js, C++, and others. It also connects well with data infrastructure like Elasticsearch and Apache Hadoop.

Core Concepts

Before you can use Kafka effectively, you need a clear mental model of its building blocks.

Topics

A topic is a named channel for a category of messages. Producers publish to a topic; consumers subscribe to it. Think of it as a logically named stream — order-placed, payment-processed, user-registered.

Partitions

Each topic is split into one or more partitions. Partitions are the unit of parallelism in Kafka. Messages within a partition are ordered and assigned a sequential offset. Across partitions, order is not guaranteed — so if ordering matters, publish related messages to the same partition using a consistent key.

Brokers

A broker is a Kafka server. A Kafka cluster is made up of multiple brokers. Each broker holds a subset of partitions and handles reads and writes for them. If one broker goes down, replicas on other brokers keep the data available.

Producers

A producer publishes messages to a topic. It can choose which partition to send a message to — either explicitly, or implicitly via a key (same key always routes to the same partition), or via round-robin if no key is set.

Consumers and Consumer Groups

A consumer reads messages from a topic partition and tracks its position using an offset. Consumers are organized into consumer groups. Within a group, each partition is assigned to exactly one consumer — this is how Kafka achieves parallel processing without double-delivery. Different groups each receive all messages independently, making it easy to have multiple services consuming the same topic for different purposes.

Message Flow in Practice

A concrete example: an e-commerce system where placing an order should trigger an inventory update, a notification email, and an analytics event.

With direct communication, the order service would need to call three services synchronously. If any of them is slow or down, the order request slows down or fails.

With Kafka:

1. The order service publishes one message to the order-placed topic.
2. The inventory service, notification service, and analytics service each subscribe to that topic in their own consumer groups.
3. Each processes the message independently, at its own pace, without the order service knowing or caring.

The order service is now fully decoupled. Adding a new downstream service requires zero changes to the producer.

A Minimal Go Example

Using the segmentio/kafka-go library:

// Producer
writer := kafka.NewWriter(kafka.WriterConfig{
    Brokers: []string{"localhost:9092"},
    Topic:   "order-placed",
})
writer.WriteMessages(ctx, kafka.Message{
    Key:   []byte(orderID),
    Value: []byte(orderJSON),
})

// Consumer
reader := kafka.NewReader(kafka.ReaderConfig{
    Brokers: []string{"localhost:9092"},
    Topic:   "order-placed",
    GroupID: "inventory-service",
})
for {
    msg, _ := reader.ReadMessage(ctx)
    processOrder(msg.Value)
}

The consumer group ID is what ties a consumer to its offset history. Restart the process with the same group ID and it picks up exactly where it left off.

When Kafka Is Not the Answer

Kafka adds operational complexity. For small systems with a handful of services and modest traffic, the overhead of running and managing a Kafka cluster outweighs the benefits. A simpler broker like RabbitMQ, or even direct HTTP calls, is often the right call at that scale.

Kafka earns its place when you need high throughput, durable message storage, or true fan-out to many independent consumers. If none of those requirements apply yet, keep it simple and reach for Kafka when you actually need it.

Summary

Kafka solves a real problem: as systems grow, direct service-to-service calls create a web of dependencies that is hard to change and easy to break. The publish/subscribe model cleanly separates producers from consumers, and Kafka's design — partitioned logs, consumer groups, on-disk persistence — makes that model reliable and scalable.

Understanding topics, partitions, brokers, and consumer groups is enough to get started. The rest is operational experience — tuning replication, managing offsets, sizing partitions for your throughput — which comes with time.