Kafka Producer & Consumer Configuration

Getting the right Kafka configuration is critical for balancing throughput, latency, and reliability. This article covers the most impactful producer and consumer settings and explains the trade-offs so you can tune for your specific use case.

Kafka

Key Producer Configs
Key Consumer Configs
Tuning for Throughput vs Latency
Tuning for Reliability

Key Producer Configs

Config	Default	Description
`acks`	`1`	`0`=no ack, `1`=leader only, `all`=all ISR replicas must acknowledge
`retries`	`2147483647`	Number of retries on transient failures
`retry.backoff.ms`	`100`	Backoff time between retries
`delivery.timeout.ms`	`120000`	Max total time for a send call including retries (2 min)
`batch.size`	`16384`	Max bytes to batch per partition before sending (16 KB)
`linger.ms`	`0`	Wait up to this many ms for additional records to fill the batch
`buffer.memory`	`33554432`	Total memory for the send buffer (32 MB)
`max.block.ms`	`60000`	How long `send()` blocks when buffer is full
`compression.type`	`none`	`none`, `gzip`, `snappy`, `lz4`, `zstd`
`max.in.flight.requests.per.connection`	`5`	Unacknowledged requests per broker. Set to 1 for strict ordering without idempotence.
`enable.idempotence`	`true` (Kafka 3+)	Ensures exactly-once delivery per producer session

Java

Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());

// Reliability
props.put(ProducerConfig.ACKS_CONFIG, "all");
props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, true);
props.put(ProducerConfig.RETRIES_CONFIG, Integer.MAX_VALUE);
props.put(ProducerConfig.MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION, 5); // safe with idempotence

// Throughput
props.put(ProducerConfig.BATCH_SIZE_CONFIG, 65536);       // 64 KB batch
props.put(ProducerConfig.LINGER_MS_CONFIG, 20);           // wait 20 ms to fill batch
props.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, "lz4"); // fast compression
props.put(ProducerConfig.BUFFER_MEMORY_CONFIG, 67108864); // 64 MB send buffer

Key Consumer Configs

Config	Default	Description
`fetch.min.bytes`	`1`	Minimum bytes the broker should return per fetch request. Increase to reduce CPU at cost of latency.
`fetch.max.wait.ms`	`500`	Max time broker waits before returning a response if `fetch.min.bytes` is not met
`max.poll.records`	`500`	Max records returned per `poll()` call
`max.poll.interval.ms`	`300000`	Max time between polls before consumer is evicted from group (5 min)
`session.timeout.ms`	`45000`	Heartbeat timeout: consumer is evicted after this many ms without a heartbeat
`heartbeat.interval.ms`	`3000`	Frequency of heartbeats. Should be < `session.timeout.ms` / 3.
`enable.auto.commit`	`true`	Auto-commit offsets in background
`auto.commit.interval.ms`	`5000`	Auto-commit frequency
`auto.offset.reset`	`latest`	Where to start when no committed offset: `earliest` or `latest`
`isolation.level`	`read_uncommitted`	Set to `read_committed` to skip transactional messages that were aborted

Tuning for Throughput vs Latency

These two goals pull in opposite directions: optimise for one tends to hurt the other.

High Throughput (batch more, compress, buffer):

Java

// Producer — maximise batching
props.put(ProducerConfig.LINGER_MS_CONFIG, 50);            // wait longer to fill batches
props.put(ProducerConfig.BATCH_SIZE_CONFIG, 131072);       // 128 KB batch
props.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, "zstd"); // best compression ratio
props.put(ProducerConfig.BUFFER_MEMORY_CONFIG, 134217728); // 128 MB buffer

// Consumer — fetch more at once
props.put(ConsumerConfig.FETCH_MIN_BYTES_CONFIG, 65536);   // 64 KB min fetch
props.put(ConsumerConfig.FETCH_MAX_WAIT_MS_CONFIG, 500);
props.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG, 2000);

Low Latency (send immediately, fetch immediately):

Java

// Producer — send as soon as possible
props.put(ProducerConfig.LINGER_MS_CONFIG, 0);         // no waiting
props.put(ProducerConfig.BATCH_SIZE_CONFIG, 1);        // effectively no batching
props.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, "none");

// Consumer — return data immediately
props.put(ConsumerConfig.FETCH_MIN_BYTES_CONFIG, 1);   // return as soon as anything is available
props.put(ConsumerConfig.FETCH_MAX_WAIT_MS_CONFIG, 0);
props.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG, 1);  // process one at a time

For most use cases, a linger.ms of 5–20 ms combined with batch.size of 64–128 KB and lz4 compression gives a good throughput-latency balance.

Tuning for Reliability

For reliable, loss-free messaging:

Java

// Producer — strongest durability guarantees
props.put(ProducerConfig.ACKS_CONFIG, "all");                // all in-sync replicas must ack
props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, true);   // no duplicates on retry
props.put(ProducerConfig.RETRIES_CONFIG, Integer.MAX_VALUE); // retry indefinitely
props.put(ProducerConfig.DELIVERY_TIMEOUT_MS_CONFIG, 300000); // 5 min total timeout
props.put(ProducerConfig.MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION, 5); // safe with idempotence

// Consumer — manual commit for at-least-once
props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, false);
props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
props.put(ConsumerConfig.ISOLATION_LEVEL_CONFIG, "read_committed"); // skip aborted txn messages

At-least-once: disable auto-commit, commit after processing. Messages may be reprocessed on failure: make your consumers idempotent.
Exactly-once: use idempotent producer + transactions + isolation.level=read_committed consumer. See the Kafka Transactions article for details.

Broker-side settings that affect producer durability:

Broker Config	Recommended	Effect
`min.insync.replicas`	2	Minimum replicas that must acknowledge a write. Producer with `acks=all` will block if fewer replicas are in sync.
`unclean.leader.election.enable`	false	Prevents a lagging replica from becoming leader (avoids data loss)
`replication.factor`	3	Number of replicas per partition

Contents

Key Producer Configs

Key Consumer Configs

Tuning for Throughput vs Latency

Tuning for Reliability

Related articles