Idempotency Patterns: Keys, Deduplication & Safe Retries

In distributed systems, retries are inevitable: network timeouts, partial failures, and Kafka at-least-once delivery all mean your service may receive the same request more than once. An idempotent operation produces the same result no matter how many times it is executed. Building idempotency in from the start prevents duplicate orders, double charges, and corrupted state.

Share

Contents

REST API: Idempotency Key Header

Clients generate a unique Idempotency-Key (UUID) for each logical operation and include it in the request header. If the server has already processed a request with that key, it returns the stored response instead of processing it again.

Java
// POST /api/payments  with header  Idempotency-Key: 550e8400-e29b-41d4-a716-446655440000
@RestController
@RequestMapping("/api/payments")
public class PaymentController {

    private final PaymentService paymentService;
    private final IdempotencyStore idempotencyStore;

    @PostMapping
    public ResponseEntity<PaymentResponse> createPayment(
            @RequestHeader("Idempotency-Key") String idempotencyKey,
            @RequestBody PaymentRequest req) {

        // 1. Check if already processed
        return idempotencyStore.get(idempotencyKey, PaymentResponse.class)
            .map(cached -> ResponseEntity.ok(cached))                     // replay cached response
            .orElseGet(() -> {
                // 2. Process for the first time
                PaymentResponse response = paymentService.charge(req);

                // 3. Store result with TTL
                idempotencyStore.put(idempotencyKey, response, Duration.ofHours(24));

                return ResponseEntity.status(HttpStatus.CREATED).body(response);
            });
    }
}

Storing & Checking Idempotency Keys

Use Redis for fast, TTL-aware key storage, or a database table for durability.

Java
import com.fasterxml.jackson.databind.ObjectMapper;
import org.springframework.data.redis.core.StringRedisTemplate;
import org.springframework.stereotype.Component;

import java.time.Duration;
import java.util.Optional;

@Component
public class RedisIdempotencyStore {

    private final StringRedisTemplate redis;
    private final ObjectMapper mapper;
    private static final String PREFIX = "idem:";

    public <T> Optional<T> get(String key, Class<T> type) {
        String json = redis.opsForValue().get(PREFIX + key);
        if (json == null) return Optional.empty();
        try {
            return Optional.of(mapper.readValue(json, type));
        } catch (Exception e) { return Optional.empty(); }
    }

    public void put(String key, Object value, Duration ttl) {
        try {
            redis.opsForValue().set(PREFIX + key, mapper.writeValueAsString(value), ttl);
        } catch (Exception e) { throw new RuntimeException("Failed to store idempotency key", e); }
    }
}
SQL
-- Database alternative — durable but slower
CREATE TABLE idempotency_keys (
  key         VARCHAR(100) PRIMARY KEY,
  response    JSONB        NOT NULL,
  created_at  TIMESTAMPTZ  NOT NULL DEFAULT NOW(),
  expires_at  TIMESTAMPTZ  NOT NULL
);
CREATE INDEX idx_idem_expires ON idempotency_keys(expires_at);

Database-Level Idempotency: Unique Constraints

The simplest form of idempotency: let the database enforce uniqueness. A duplicate insert throws a DataIntegrityViolationException which you catch and convert to the stored result.

Java
@Service
public class OrderService {

    private final OrderRepository orderRepo;

    @Transactional
    public Order placeOrder(PlaceOrderRequest req) {
        // If the same (customerId, externalOrderId) already exists, return it
        return orderRepo.findByExternalOrderId(req.externalOrderId())
            .orElseGet(() -> {
                try {
                    Order order = new Order(req.customerId(), req.externalOrderId(),
                                           req.amount(), OrderStatus.PENDING);
                    return orderRepo.save(order);
                } catch (DataIntegrityViolationException e) {
                    // Race condition — another thread just inserted; fetch and return it
                    return orderRepo.findByExternalOrderId(req.externalOrderId())
                        .orElseThrow();
                }
            });
    }
}
SQL
ALTER TABLE orders ADD CONSTRAINT uq_orders_external_id
  UNIQUE (customer_id, external_order_id);

Kafka Consumer Deduplication

Kafka's at-least-once delivery means your consumer may receive duplicate messages. Track processed event IDs to skip duplicates.

Java
@Component
public class PaymentEventConsumer {

    private final ProcessedEventRepository processedRepo;
    private final PaymentService paymentService;

    @KafkaListener(topics = "payment.requested", groupId = "payment-service")
    @Transactional
    public void consume(PaymentRequestedEvent event,
                        @Header(KafkaHeaders.OFFSET) long offset,
                        @Header(KafkaHeaders.RECEIVED_PARTITION) int partition) {

        String eventId = event.eventId();  // unique ID in the event payload

        // Skip if already processed
        if (processedRepo.existsById(eventId)) {
            log.debug("Skipping duplicate event id={}", eventId);
            return;
        }

        // Process
        paymentService.charge(event.orderId(), event.amount());

        // Record as processed — same transaction as the business operation
        processedRepo.save(new ProcessedEvent(eventId, Instant.now()));
    }
}
SQL
CREATE TABLE processed_events (
  event_id   VARCHAR(100) PRIMARY KEY,
  processed_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
-- Clean up old entries periodically
DELETE FROM processed_events WHERE processed_at < NOW() - INTERVAL '7 days';
Save the processed event record in the same database transaction as the business operation. If the business operation rolls back, so does the deduplication record, preventing a scenario where the event is marked "processed" but the work wasn't actually done.

Natural Idempotency Keys

Many operations have natural keys that make them idempotent without extra infrastructure:

OperationNatural Idempotency Key
Create order(customerId, cartId) or externalOrderId
Send email(userId, emailTemplateId, date)
Reserve inventory(orderId, productId)
Kafka eventeventId field in the payload (UUID generated by producer)
Webhook deliveryWebhook provider's delivery ID in the header

Spring HandlerInterceptor for Idempotency

Extract idempotency key checking into a reusable interceptor so controllers stay clean.

Java
import jakarta.servlet.http.*;
import org.springframework.web.servlet.HandlerInterceptor;

@Component
public class IdempotencyInterceptor implements HandlerInterceptor {

    private final RedisIdempotencyStore store;

    @Override
    public boolean preHandle(HttpServletRequest request,
                             HttpServletResponse response,
                             Object handler) throws Exception {
        if (!request.getMethod().equals("POST")) return true;

        String key = request.getHeader("Idempotency-Key");
        if (key == null || key.isBlank()) return true;  // optional — skip if not provided

        store.get(key, String.class).ifPresent(cached -> {
            try {
                response.setStatus(200);
                response.setContentType("application/json");
                response.getWriter().write(cached);
            } catch (Exception e) { throw new RuntimeException(e); }
        });

        // Store key in request attribute for the controller to save the response
        request.setAttribute("idempotencyKey", key);
        return true;
    }
}