Kafka - Topics and Partitions

A topic is a named, append-only log. Producers write to it; consumers read from it. Topics are identified by name and can have multiple partitions.

• Topic names are case-sensitive and conventionally use lowercase and hyphens (e.g. user-events).
• Messages are not deleted on consumption: they are retained for a configurable period.
• Multiple consumer groups can read the same topic independently.

A partition is an ordered, immutable sequence of records, each identified by a monotonically increasing offset. Each partition has one leader broker that receives its writes, and copies on other brokers (see Replication, below).

• Partitions are how Kafka spreads work: within one consumer group, each partition is read by one consumer, so the partition count is the most consumers that can work at the same time.
• Within a partition, ordering is guaranteed. Across partitions, it is not.
• Messages with the same key always land in the same partition (hash-based routing).
• The partition count can be increased but never decreased, so plan it before the topic is in use.

Each partition has one leader and zero or more followers (determined by the replication factor).

• All writes go to the leader. Reads go to the leader too by default; since Kafka 2.4, consumers can be set up to read from a follower in their own data centre.
• Followers copy new data from the leader continuously.
• In-Sync Replicas (ISR): the set of replicas that are caught up to the leader. When acks=all, all ISR brokers must acknowledge.
• If the leader fails, one of the ISR followers is elected the new leader automatically.

The table assumes the common production setting min.insync.replicas=2. With acks=all, a write succeeds only if at least that many replicas have it, so a confirmed message always exists on two brokers. If fewer replicas are in sync, the broker rejects writes instead of risking data loss. With the default of 1, acks=all can succeed with a single copy.

Kafka retains messages for a configurable period regardless of whether they were consumed. Two retention modes:

• Time-based (retention.ms): delete messages older than the configured time (default: 7 days).
• Size-based (retention.bytes): delete the oldest data when a partition grows beyond the limit. It applies to each partition separately and is off by default.

Kafka deletes whole segment files, not single messages, so a message can stay somewhat longer than the retention time until its segment is closed and old enough.

# Create topic with 1-day retention
kafka-topics.sh --create \
  --topic user-events \
  --partitions 6 \
  --replication-factor 3 \
  --config retention.ms=86400000 \
  --bootstrap-server localhost:9092

Log compaction is an alternative to time-based deletion. Instead of deleting old data by time, Kafka keeps only the latest record for each key. Older records with the same key are removed. This is ideal for maintaining a changelog or snapshot of the latest state per entity.

# Enable log compaction on a topic
kafka-topics.sh --create \
  --topic user-profiles \
  --partitions 3 \
  --replication-factor 3 \
  --config cleanup.policy=compact \
  --bootstrap-server localhost:9092

Topics can also be managed from Java with Admin. Each method returns futures; .get() waits for the result:

import java.util.List;
import java.util.Map;
import java.util.Properties;
import java.util.Set;

import org.apache.kafka.clients.admin.Admin;
import org.apache.kafka.clients.admin.AdminClientConfig;
import org.apache.kafka.clients.admin.NewTopic;
import org.apache.kafka.clients.admin.TopicDescription;

public class TopicAdmin {

    public static void main(String[] args) throws Exception {
        Properties props = new Properties();
        props.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");

        try (Admin admin = Admin.create(props)) {
            // Create a topic: 6 partitions, 3 replicas, 7-day retention
            NewTopic orders = new NewTopic("orders", 6, (short) 3)
                    .configs(Map.of("retention.ms", "604800000"));
            admin.createTopics(List.of(orders)).all().get();

            // List topics
            Set<String> topics = admin.listTopics().names().get();
            System.out.println("Topics: " + topics);

            // Describe a topic: partitions, leaders, replicas
            Map<String, TopicDescription> descriptions =
                    admin.describeTopics(List.of("orders")).allTopicNames().get();
            descriptions.forEach((name, description) -> System.out.println(name + ": " + description));

            // Delete a topic
            admin.deleteTopics(List.of("old-topic")).all().get();
        }
    }
}