Apache Kafka - Introduction
Apache Kafka is a distributed event streaming platform designed for high-throughput, fault-tolerant, real-time data pipelines. Originally built at LinkedIn, it is now used by thousands of companies to decouple producers and consumers, buffer high-volume event streams, and enable event-driven architectures.
Contents
What is Kafka?
Kafka acts as a distributed commit log. Events (messages) are appended to topics and retained for a configurable period: consumers can replay from any offset. This differs from traditional queues where messages are deleted after consumption.
- Publish-Subscribe: many producers publish events; many consumer groups read independently.
- Durable: messages are written to disk and replicated across brokers.
- Scalable: topics are split into partitions distributed across a cluster.
- High-throughput: sequential disk writes + batching achieve millions of events/second.
Core Components
| Component | Role |
|---|---|
| Broker | A Kafka server that stores and serves messages. A cluster has multiple brokers for fault tolerance. |
| Topic | A named stream of messages, such as orderspayments |
| Partition | One of the ordered logs a topic is split into. Partitions are what let Kafka work in parallel: different partitions can live on different brokers, and different consumers in a group can read them at the same time. |
| Producer | Application that publishes messages to a topic. |
| Consumer | Application that reads messages from a topic. |
| Consumer Group | A group of consumers that together consume a topic: each partition is assigned to exactly one consumer in the group. |
| Offset | A unique sequential ID for each message within a partition. Consumers track their position via offsets. |
| KRaft controllers | Manage the cluster's metadata: which brokers are alive, which broker leads each partition, and topic settings. Older versions used a separate ZooKeeper cluster for this; Kafka 4.0 removed ZooKeeper, so every new cluster uses KRaft. |
Topics and Partitions
A topic is a logical channel. Each topic is split into partitions, and each partition is an append-only log stored on disk by one broker (as a series of segment files). Partitions enable parallelism: multiple consumers in a group can read different partitions simultaneously.
- Messages within a partition are strictly ordered.
- Across partitions, there is no global ordering guarantee.
- A replication factor of N means N copies exist: one leader + (N-1) followers. If the leader fails, a follower takes over.
- A message with a key goes to the partition chosen by a hash of the key, so all messages with the same key stay in order in one partition. Messages without a key are spread across partitions in batches.
The number of partitions is the upper limit on how many consumers in one group can work in parallel, so choose it from the throughput you need. You can add partitions later, but not remove them, and adding them changes which partition each key maps to, which breaks per-key ordering for keys that move. Each partition also costs the brokers memory and files, so do not create far more than you need.
Producers and Consumers
Producers write messages to topics. Key settings:
acks01allalldelivery.timeout.msbatch.sizelinger.ms
Consumers read messages from topics. Key concepts:
- Consumers belong to a consumer group. Each partition is assigned to exactly one consumer per group.
- Adding consumers to a group increases parallelism (up to the number of partitions).
- Consumers commit offsets to track their position. On restart, they resume from the committed offset.
- Commit strategies: auto-commit (simple; after a crash, some messages are processed again) or manual commit (you decide when a message counts as done).
Use Cases
- Real-time analytics: stream user clicks, transactions, sensor data to analytics pipelines.
- Microservices communication: decouple services via events instead of synchronous REST calls.
- Log aggregation: centralise application and system logs from many services.
- Event sourcing: store every state change as an immutable event log.
- Data integration (ETL): move data between databases, data warehouses, and search indexes.
- Stream processing: Kafka Streams or Apache Flink for real-time transformations.
Kafka vs Traditional Message Queues
| Feature | Kafka | Traditional Queue (RabbitMQ, SQS) |
|---|---|---|
| Message retention | Configurable (hours/days/forever) | Deleted after consumption |
| Replay | Yes: reset the offset to read again | Not in classic queues (RabbitMQ Streams add it) |
| Consumers | Many independent groups, each reading every message | Competing consumers: each message goes to one of them |
| Ordering | Per partition | Per queue, while there is a single consumer |
| Throughput | Very high; scales by adding partitions and brokers | Usually lower per queue |
| Best for | Event streaming, analytics, log pipelines | Task queues, job dispatching |