Kafka Message Serialization

Kafka transmits messages as raw bytes. Serializers convert Java objects to bytes before sending; deserializers convert bytes back to objects on the consumer side. Choosing the right serialization format affects schema evolution, compatibility, and performance.

Kafka

String Serializer (default)
JSON Serialization with Jackson
Custom Serializer / Deserializer
Avro Serialization & Schema Registry
Format Comparison

String Serializer (default)

The simplest approach: suitable for plain text, CSV, or JSON strings. Both key and value are serialized as UTF-8 strings.

Java

// Producer
Properties producerProps = new Properties();
producerProps.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
producerProps.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
producerProps.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());

KafkaProducer<String, String> producer = new KafkaProducer<>(producerProps);
producer.send(new ProducerRecord<>("orders", "order-123", "{'item':'book','qty':2}"));

// Consumer
Properties consumerProps = new Properties();
consumerProps.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
consumerProps.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());

JSON Serialization with Jackson

For structured data, serialize POJOs to JSON using a custom serializer backed by Jackson's ObjectMapper.

Java

// Shared POJO
public class Order {
    public String orderId;
    public String item;
    public int quantity;
    // constructors, getters...
}

// Custom JSON Serializer
public class JsonSerializer<T> implements Serializer<T> {
    private final ObjectMapper mapper = new ObjectMapper();

    @Override
    public byte[] serialize(String topic, T data) {
        try {
            return mapper.writeValueAsBytes(data);
        } catch (JsonProcessingException e) {
            throw new SerializationException("Error serializing to JSON", e);
        }
    }
}

// Custom JSON Deserializer
public class JsonDeserializer<T> implements Deserializer<T> {
    private final ObjectMapper mapper = new ObjectMapper();
    private Class<T> targetType;

    public JsonDeserializer(Class<T> targetType) {
        this.targetType = targetType;
    }

    @Override
    public T deserialize(String topic, byte[] data) {
        try {
            return mapper.readValue(data, targetType);
        } catch (IOException e) {
            throw new SerializationException("Error deserializing JSON", e);
        }
    }
}

// Producer usage
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, JsonSerializer.class.getName());
KafkaProducer<String, Order> producer = new KafkaProducer<>(props);
producer.send(new ProducerRecord<>("orders", "order-1", new Order("order-1", "book", 2)));

// Consumer usage
KafkaConsumer<String, Order> consumer = new KafkaConsumer<>(props,
    new StringDeserializer(), new JsonDeserializer<>(Order.class));

Spring Kafka provides JsonSerializer and JsonDeserializer out of the box via spring-kafka: no need to write your own.

Custom Serializer / Deserializer

Implement Serializer<T> and Deserializer<T> for full control: useful for binary formats or when integrating with existing wire formats.

Java

public class OrderSerializer implements Serializer<Order> {
    @Override
    public byte[] serialize(String topic, Order order) {
        if (order == null) return null;
        // Simple binary layout: orderId length (4 bytes) + orderId + quantity (4 bytes)
        byte[] orderIdBytes = order.orderId.getBytes(StandardCharsets.UTF_8);
        ByteBuffer buffer = ByteBuffer.allocate(4 + orderIdBytes.length + 4);
        buffer.putInt(orderIdBytes.length);
        buffer.put(orderIdBytes);
        buffer.putInt(order.quantity);
        return buffer.array();
    }
}

public class OrderDeserializer implements Deserializer<Order> {
    @Override
    public Order deserialize(String topic, byte[] data) {
        if (data == null) return null;
        ByteBuffer buffer = ByteBuffer.wrap(data);
        int idLen = buffer.getInt();
        byte[] idBytes = new byte[idLen];
        buffer.get(idBytes);
        String orderId = new String(idBytes, StandardCharsets.UTF_8);
        int quantity = buffer.getInt();
        return new Order(orderId, quantity);
    }
}

Avro Serialization & Schema Registry

Apache Avro is a compact binary format with schema support. Combined with Confluent Schema Registry, schemas are stored centrally and versioned: producers and consumers reference schema IDs instead of embedding the full schema in every message.

XML

<!-- pom.xml dependencies -->
<dependency>
    <groupId>io.confluent</groupId>
    <artifactId>kafka-avro-serializer</artifactId>
    <version>7.5.0</version>
</dependency>

Java

// Producer with Avro
Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, KafkaAvroSerializer.class.getName());
props.put("schema.registry.url", "http://localhost:8081");

// Avro GenericRecord (schema-driven)
Schema schema = new Schema.Parser().parse(new File("order.avsc"));
GenericRecord record = new GenericData.Record(schema);
record.put("orderId", "order-123");
record.put("quantity", 2);

KafkaProducer<String, GenericRecord> producer = new KafkaProducer<>(props);
producer.send(new ProducerRecord<>("orders", "order-123", record));

// Consumer with Avro
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, KafkaAvroDeserializer.class.getName());
props.put("specific.avro.reader", true); // use generated Avro classes

Schema Registry must be running and reachable by both producers and consumers. If the registry is unavailable, serialization will fail.

Format Comparison

Format	Size	Human readable	Schema evolution	Best for
String / JSON	Large	✅ Yes	Manual	Simple messages, debugging, early development
Avro	Small	❌ Binary	✅ Schema Registry	High-volume production pipelines
Protobuf	Very small	❌ Binary	✅ Registry	Cross-language, performance-critical
Custom binary	Smallest	❌ Binary	Manual	Full control, proprietary formats

Contents

String Serializer (default)

JSON Serialization with Jackson

Custom Serializer / Deserializer

Avro Serialization & Schema Registry

Format Comparison

Related articles