Kafka Performance Tuning & Monitoring

Kafka is built for high throughput, but reaching peak performance requires tuning producers, consumers, and brokers for your specific workload. Key levers include batching and compression on the producer side, fetch tuning on the consumer side, and partition strategy across the cluster. Monitoring consumer lag, broker I/O utilization, and under-replicated partitions is essential for maintaining a healthy production cluster.

Share

Contents

Producer Throughput Tuning

The Kafka producer batches records into requests before sending them to brokers. Tuning batch size, linger time, buffer memory, and compression directly controls throughput and latency trade-offs.

batch.size

Controls the maximum number of bytes per partition batch. The producer accumulates records into a batch until it reaches this size or linger.ms expires. Increasing batch size improves throughput by reducing the number of network round-trips, but adds memory pressure.

Properties
# Default is 16384 (16 KB). For high-throughput workloads, increase to 64 KB or 128 KB.
batch.size=131072
linger.ms

Adds an artificial delay before sending a batch, giving more records a chance to arrive and fill the batch. A value of 0 (the default) means the producer sends immediately. Setting it to 5-20 ms can significantly improve throughput with minimal latency impact.

Properties
# Wait up to 10 ms for more records to fill the batch
linger.ms=10
buffer.memory

The total memory available for buffering unsent records across all partitions. If the buffer fills up, send() blocks for up to max.block.ms before throwing an exception.

Properties
# Default is 33554432 (32 MB). Increase for bursty workloads.
buffer.memory=67108864
Putting it together
Java
Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG,      "broker1:9092,broker2:9092");
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG,   StringSerializer.class);
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class);

// Throughput tuning
props.put(ProducerConfig.BATCH_SIZE_CONFIG,             131072);   // 128 KB batches
props.put(ProducerConfig.LINGER_MS_CONFIG,              10);       // wait up to 10 ms
props.put(ProducerConfig.BUFFER_MEMORY_CONFIG,          67108864); // 64 MB buffer
props.put(ProducerConfig.COMPRESSION_TYPE_CONFIG,       "lz4");    // fast compression

KafkaProducer<String, String> producer = new KafkaProducer<>(props);
Increasing batch.size and linger.ms trades latency for throughput. For latency-sensitive applications (sub-10 ms requirements), keep linger.ms=0 and rely on natural batching from high-volume send loops.

Consumer Throughput Tuning

Consumer throughput depends on how much data each fetch request retrieves and how many records are processed per poll cycle. Tuning fetch parameters minimizes idle time between network requests.

fetch.min.bytes

The minimum amount of data the broker should return for a fetch request. If fewer bytes are available, the broker waits until this threshold is met or fetch.max.wait.ms expires. Increasing this value reduces the number of fetch requests for low-volume topics.

Properties
# Default is 1 byte. Increase for better batching on low-throughput topics.
fetch.min.bytes=65536
fetch.max.wait.ms

The maximum time the broker waits for fetch.min.bytes to accumulate before responding. Works together with fetch.min.bytes to control fetch frequency.

Properties
# Default is 500 ms. Increase for better batching, decrease for lower latency.
fetch.max.wait.ms=500
max.poll.records

Limits the number of records returned in a single poll() call. If your processing logic is slow, reducing this value prevents rebalances caused by exceeding max.poll.interval.ms.

Properties
# Default is 500. Increase for fast processing pipelines.
max.poll.records=1000
max.partition.fetch.bytes

The maximum amount of data per partition the broker returns per fetch. If a single partition has large messages, increase this to avoid truncation. The total fetch size is bounded by fetch.max.bytes.

Properties
# Default is 1048576 (1 MB). Increase for topics with large message values.
max.partition.fetch.bytes=2097152
If your processing takes longer than max.poll.interval.ms (default 5 minutes), the consumer is removed from the group and a rebalance is triggered. Either reduce max.poll.records or increase max.poll.interval.ms to match your processing time.
Java
Properties props = new Properties();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG,        "broker1:9092");
props.put(ConsumerConfig.GROUP_ID_CONFIG,                 "perf-consumer-group");
props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG,   StringDeserializer.class);
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);

// Throughput tuning
props.put(ConsumerConfig.FETCH_MIN_BYTES_CONFIG,          65536);   // 64 KB minimum fetch
props.put(ConsumerConfig.FETCH_MAX_WAIT_MS_CONFIG,        500);     // wait up to 500 ms
props.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG,         1000);    // 1000 records per poll
props.put(ConsumerConfig.MAX_PARTITION_FETCH_BYTES_CONFIG, 2097152); // 2 MB per partition

KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);

Compression

Compression reduces the size of record batches on the wire and on disk. The producer compresses each batch; the broker stores the compressed batch as-is (unless re-compression is needed due to a format mismatch); the consumer decompresses transparently.

Compression codecs compared

Kafka supports four compression types, each with different trade-offs between CPU cost and compression ratio:

Java
/*
  Codec       Ratio      CPU Cost     Best For
  ──────────────────────────────────────────────────
  gzip        Highest    Highest      Archival, storage-constrained
  snappy      Low        Lowest       Real-time, CPU-constrained
  lz4         Medium     Low          General purpose, high throughput
  zstd        High       Medium       Best ratio-to-speed balance
  ──────────────────────────────────────────────────

  Recommendation:
  - Start with lz4 for most workloads
  - Use zstd when storage or bandwidth is the bottleneck
  - Use snappy when CPU is the bottleneck
  - Use gzip only for cold-storage / archival topics
*/
Enabling producer-side compression
Properties
# Producer config — apply lz4 compression to all batches
compression.type=lz4
Broker-side compression configuration

By default, the broker keeps the producer's compression format (compression.type=producer). You can override this at the topic level, but be aware that re-compression adds CPU overhead on the broker.

Shell
# Set topic-level compression to zstd
bin/kafka-configs.sh --bootstrap-server localhost:9092 \
  --entity-type topics --entity-name orders \
  --alter --add-config compression.type=zstd
If the broker's compression.type differs from the producer's, the broker must decompress and re-compress every batch. This can significantly increase broker CPU usage. Keep producer and broker compression types aligned whenever possible.

Broker Configuration

Broker-level tuning focuses on I/O threads, network threads, log segment management, and operating system settings that affect disk and memory performance.

Thread pools
Properties
# Number of threads handling network requests (reading from / writing to sockets)
# Default: 3. Increase on high-connection-count brokers.
num.network.threads=8

# Number of threads performing disk I/O (log appends, index updates)
# Default: 8. Increase if disk throughput is the bottleneck.
num.io.threads=16
Log segment settings
Properties
# Maximum size of a single log segment file before rolling to a new one
# Default: 1073741824 (1 GB). Smaller segments enable faster cleanup but increase file count.
log.segment.bytes=1073741824

# How long to retain log segments (in hours)
log.retention.hours=168

# Alternatively, retain by total size per partition
log.retention.bytes=10737418240
OS-level tuning (Linux)

Kafka relies heavily on the OS page cache for read performance. The following kernel parameters should be tuned on production brokers:

Shell
# Increase the number of open file descriptors (Kafka opens many segment files)
ulimit -n 100000

# Reduce swappiness — keep Kafka's JVM heap in RAM
sudo sysctl -w vm.swappiness=1

# Increase socket buffer sizes for high-throughput networking
sudo sysctl -w net.core.rmem_max=16777216
sudo sysctl -w net.core.wmem_max=16777216
sudo sysctl -w net.ipv4.tcp_rmem="4096 87380 16777216"
sudo sysctl -w net.ipv4.tcp_wmem="4096 65536 16777216"

# Set dirty page ratio — flush to disk less frequently for better throughput
sudo sysctl -w vm.dirty_ratio=80
sudo sysctl -w vm.dirty_background_ratio=5
Kafka reads from the page cache for most consumer fetches. Ensure your brokers have enough free RAM (beyond JVM heap) for the page cache to hold the most recent log segments. A common rule of thumb is to leave at least 50% of physical memory for the page cache.

Partitioning for Parallelism

Partitions are the unit of parallelism in Kafka. More partitions allow more consumers to read in parallel, but over-partitioning has costs. Choosing the right partition count depends on your target throughput and consumer processing capacity.

Choosing partition count
Java
/*
  Rule of thumb for partition count:

  partitions = max(T/Pp, T/Pc)

  Where:
    T  = target throughput (MB/s)
    Pp = throughput per producer partition (measure with perf test)
    Pc = throughput per consumer partition (measure with your processing logic)

  Example:
    Target: 100 MB/s
    Producer can push 10 MB/s per partition  → 10 partitions
    Consumer can process 5 MB/s per partition → 20 partitions
    Answer: 20 partitions (consumer is the bottleneck)
*/
Over-partitioning pitfalls

While more partitions increase parallelism, excessive partitions cause problems:

Java
/*
  Risks of too many partitions:
  ─────────────────────────────────────────────────────
  1. More open file handles — each partition has segment files,
     index files, and time-index files on every replica.

  2. Longer leader election time — on broker failure, each
     partition needs a new leader elected sequentially.

  3. Higher end-to-end latency — the producer waits for the
     slowest partition in a batch before completing.

  4. More memory on the producer — each partition gets its
     own accumulator buffer (batch.size bytes).

  5. Rebalance storms — consumer group rebalances take longer
     with more partitions to reassign.
  ─────────────────────────────────────────────────────
  Start with fewer partitions and scale up. You can always
  increase the partition count, but you can never decrease it.
*/
Key-based partitioning

When a message key is provided, Kafka's default partitioner hashes the key to determine the partition. This guarantees ordering per key but can cause hot partitions if keys are skewed.

Java
// Records with the same key always go to the same partition
producer.send(new ProducerRecord<>("orders", customerId, orderJson));

// If key distribution is skewed, consider a custom partitioner:
public class RoundRobinByRegionPartitioner implements Partitioner {
    @Override
    public int partition(String topic, Object key, byte[] keyBytes,
                         Object value, byte[] valueBytes, Cluster cluster) {
        List<PartitionInfo> partitions = cluster.partitionsForTopic(topic);
        // Custom logic to spread load across partitions
        String region = extractRegion((String) key);
        return Math.abs(region.hashCode()) % partitions.size();
    }
}
Changing the partition count for a topic with keyed messages breaks key-to-partition mapping. Existing keys will hash to different partitions, disrupting ordering guarantees. Plan your partition count carefully before producing keyed data.

Monitoring Key Metrics

Kafka exposes hundreds of JMX metrics. Focus on the critical ones that signal trouble before it affects your applications.

Essential broker metrics
Java
/*
  Metric                                  What it tells you
  ──────────────────────────────────────────────────────────────────
  UnderReplicatedPartitions               Partitions where follower replicas
  (kafka.server:type=                     are behind the leader. Non-zero
   ReplicaManager,name=                   means data is at risk.
   UnderReplicatedPartitions)

  RequestHandlerAvgIdlePercent            How busy the broker's I/O threads
  (kafka.server:type=                     are. Below 0.3 (30% idle) means
   KafkaRequestHandlerPool,name=          the broker is overloaded.
   RequestHandlerAvgIdlePercent)

  BytesInPerSec / BytesOutPerSec          Network throughput per broker.
  (kafka.server:type=                     Use to detect imbalanced load
   BrokerTopicMetrics,name=               across the cluster.
   BytesInPerSec)

  ActiveControllerCount                   Should be exactly 1 across the
  (kafka.controller:type=                 entire cluster. Zero means no
   KafkaController,name=                  controller — a critical problem.
   ActiveControllerCount)

  IsrShrinkRate / IsrExpandRate           Rate of ISR changes. Frequent
  (kafka.server:type=                     shrinks indicate slow followers
   ReplicaManager)                        or network issues.
  ──────────────────────────────────────────────────────────────────
*/
Querying JMX metrics
Shell
# Enable JMX on a Kafka broker (set before starting the broker)
export JMX_PORT=9999
export KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote \
  -Dcom.sun.management.jmxremote.authenticate=false \
  -Dcom.sun.management.jmxremote.ssl=false"

# Use jmxterm or JConsole to query metrics
java -jar jmxterm.jar -l localhost:9999

# Inside jmxterm:
# bean kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions
# get Value
Set up alerting on UnderReplicatedPartitions > 0 and RequestHandlerAvgIdlePercent < 0.3. These two metrics catch most broker-level issues before they cascade into consumer-visible problems.

Consumer Lag Monitoring

Consumer lag is the difference between the latest offset in a partition and the consumer group's committed offset. It measures how far behind a consumer is from real-time data. Growing lag indicates that consumers cannot keep up with producers.

Using kafka-consumer-groups.sh
Shell
# Describe a consumer group — shows lag per partition
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
  --describe --group my-consumer-group

# Output columns:
# TOPIC  PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG  CONSUMER-ID  HOST  CLIENT-ID

# Check all consumer groups at once
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
  --describe --all-groups
Lag as a health signal
Java
/*
  Interpreting consumer lag:
  ─────────────────────────────────────────────────────
  Lag = 0          Consumer is caught up (real-time)
  Lag steady       Consumer keeps up but with a fixed delay
  Lag growing      Consumer is falling behind — action needed
  Lag decreasing   Consumer is catching up (e.g. after a restart)
  ─────────────────────────────────────────────────────

  Common causes of growing lag:
  - Slow processing logic in the consumer
  - Too few consumers relative to partitions
  - GC pauses in the consumer JVM
  - Network issues between consumer and broker
  - Skewed partition assignment (one partition much larger)
*/
Burrow for automated lag monitoring

Burrow (by LinkedIn) evaluates lag as a sliding window instead of a point-in-time number. It classifies consumer status as OK, WARNING, or ERROR based on lag trends rather than absolute values.

Shell
# Burrow API — check consumer group status
curl -s http://localhost:8000/v3/kafka/local/consumer/my-consumer-group/status | jq .

# Response includes:
# "status": "OK" | "WARNING" | "ERR"
# "maxlag": { "partition": 3, "offset_lag": 15000 }
# "totallag": 42000
Alerting thresholds
Java
/*
  Recommended alerting thresholds (adjust per workload):
  ─────────────────────────────────────────────────────
  WARNING:  lag > 10,000 offsets for > 5 minutes
  CRITICAL: lag > 100,000 offsets for > 5 minutes
            OR lag is monotonically increasing for > 15 minutes

  For time-sensitive workloads, convert lag to time:
    lag_seconds = lag_offsets / (messages_per_second)
  Alert when lag_seconds > your SLA (e.g. 60 seconds)
  ─────────────────────────────────────────────────────
*/
A consumer group with zero lag but zero throughput is not healthy: it means no data is flowing. Monitor both lag and consumption rate to get the full picture.

Benchmarking with kafka-producer-perf-test

Kafka ships with built-in performance testing tools that let you establish throughput baselines without writing custom code. Use these tools to measure the impact of configuration changes.

Producer performance test
Shell
# Send 1 million records of 1 KB each to the 'perf-test' topic
bin/kafka-producer-perf-test.sh \
  --topic perf-test \
  --num-records 1000000 \
  --record-size 1024 \
  --throughput -1 \
  --producer-props \
    bootstrap.servers=localhost:9092 \
    batch.size=131072 \
    linger.ms=10 \
    compression.type=lz4

# Output:
# 1000000 records sent, 95238.1 records/sec (93.01 MB/sec),
# 12.3 ms avg latency, 245.0 ms max latency,
# 8 ms 50th, 28 ms 95th, 45 ms 99th, 62 ms 99.9th.
Consumer performance test
Shell
# Consume 1 million messages from 'perf-test' topic
bin/kafka-consumer-perf-test.sh \
  --bootstrap-server localhost:9092 \
  --topic perf-test \
  --messages 1000000 \
  --threads 1

# Output:
# start.time, end.time, data.consumed.in.MB, MB.sec,
# data.consumed.in.nMsg, nMsg.sec
Benchmarking methodology
Java
/*
  Steps to benchmark effectively:
  ─────────────────────────────────────────────────────
  1. Create a dedicated topic with your target partition count
     bin/kafka-topics.sh --create --topic perf-test \
       --partitions 12 --replication-factor 3

  2. Establish a BASELINE with default configs

  3. Change ONE parameter at a time and re-run

  4. Record: throughput (MB/s), avg latency, p99 latency

  5. Compare against baseline to measure improvement

  Example test matrix:
  ─────────────────────────────────────────────────────
  Test   batch.size   linger.ms   compression   Result
  ─────────────────────────────────────────────────────
  Base   16384        0           none          50 MB/s
  A      131072       0           none          72 MB/s
  B      131072       10          none          85 MB/s
  C      131072       10          lz4           93 MB/s
  D      131072       10          zstd          88 MB/s
  ─────────────────────────────────────────────────────
*/
Always run benchmarks on infrastructure that matches production. Network bandwidth, disk type (SSD vs HDD), and the number of replicas significantly affect results. A benchmark on a single local broker does not predict production performance.

Related articles

All Kafka articles →
Kafka Producer & Consumer ConfigurationEssential Kafka producer and consumer configuration: acks, retries, batch.size, linger.ms, compression, and tuning for throughput vs reliability.Kafka Monitoring: Metrics & Consumer LagKafka Monitoring: track broker health, consumer lag, producer metrics, and JMX-based observability.Kafka Consumer Lag Monitoring: Prometheus, Grafana & AlertingKafka consumer lag monitoring: expose lag metrics with JMX Exporter and Kafka Exporter, build Grafana dashboards, and set Prometheus alerting rules.Kafka - ProducersKafka Producer: API, acks, retries, idempotent producer configuration with Java code examples.