Graceful shutdown and zero-downtime deploys for Spring Boot on Kubernetes

Every rolling update stops old pods, and each stop is a chance to drop requests, interrupt jobs or replay Kafka records. Two things happen at once when a pod is deleted: Kubernetes stops routing to it, and the kubelet tells the application to stop. This page follows both paths from the pod's side, then shows each part of the shutdown in a running Spring Boot 4.1 application, most of it under a real SIGTERM.

Share

Contents

What happens when a pod is deleted

A pod can be deleted for several reasons: a rolling update replaces it, a scale-down removes it, or a node is drained for maintenance. In these cases the API server does not stop anything itself. It marks the pod as terminating, records a deadline based on terminationGracePeriodSeconds (30 seconds by default), and leaves the rest to two groups of components that watch the pod. Eviction for node pressure is different: the kubelet does it and ignores terminationGracePeriodSeconds, and a hard eviction threshold allows no grace period at all.

  • The node path: the kubelet is the agent on each node that runs pods' containers. It runs each container's preStop hook, if there is one, then asks the container runtime to send SIGTERM to the container's main process, the one started by ENTRYPOINT or CMD (process 1 inside the container). Only process 1 gets the signal, so it must forward the signal to any child processes. If the container is still running when the grace period ends, the kubelet sends SIGKILL. SIGTERM asks the process to stop, and the JVM runs its shutdown code; SIGKILL ends the process at once, and no Java code runs.
  • The network path: EndpointSlices are the API objects that list the pod addresses behind a Service. Their controller marks the pod's endpoint as terminating, with ready: false. kube-proxy, which turns Services into forwarding rules on every node, rewrites those rules. Ingress controllers and cloud load balancers update their lists of backends. Each of these watches the API server and reacts at its own pace.

The Kubernetes documentation states that, "at the same time as the kubelet is starting graceful shutdown of the Pod", the control plane evaluates whether to remove the pod from EndpointSlices. There is no ordering between the two paths: they race. If the application stops listening as soon as it receives SIGTERM, a node or load balancer can still route new connections to it for a second or two, and those clients get a connection error.

Zero-downtime shutdown therefore needs settings in the pod spec, the image and the application:

The pieces of a clean shutdown and where each is set
PieceSet inWhat it does
preStop sleepPod specDelays SIGTERM so the network path can finish first
terminationGracePeriodSecondsPod specTotal time, including preStop, before SIGKILL
Exec-form entrypointDockerfileMakes the JVM process 1, so the signal reaches it
server.shutdown=gracefulSpring Boot (default since 3.4)Stops accepting connections, lets in-flight requests finish
spring.lifecycle.timeout-per-shutdown-phaseSpring Boot (default 30s)The longest Spring waits for each shutdown phase
Executor and scheduler await settingsSpring BootWhether queued @Async and @Scheduled work finishes
Kafka listener container stopSpring for Apache KafkaCommits offsets and leaves the consumer group

The exec form and process 1 are covered in Docker for Java: with a shell-form ENTRYPOINT, the shell is process 1 and usually does not pass SIGTERM on; dash, the /bin/sh of Debian and Ubuntu images, does not. The JVM then never hears it, and the pod is killed with SIGKILL at the end of the grace period. This page assumes the entrypoint and the probes are correct and follows what happens after the signal arrives.

The programs below run Spring Boot 4.1.2 on Java 21. Three of them start the application in a child JVM and send it a real SIGTERM or SIGKILL, as the kubelet would. The other two call context.close() directly, which is what the SIGTERM shutdown hook does. The Kubernetes manifest was validated but not applied to a cluster, so no kubectl output is shown.

preStop and the grace period

The usual fix for the race is to wait a few seconds before the application hears about the shutdown. A preStop hook that sleeps keeps the container serving while the endpoint removal reaches every node and load balancer. The kubelet sends SIGTERM only after the hook returns:

YAML
apiVersion: apps/v1
kind: Deployment
metadata:
  name: orders
spec:
  replicas: 3
  selector:
    matchLabels:
      app: orders
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  template:
    metadata:
      labels:
        app: orders
    spec:
      terminationGracePeriodSeconds: 45  # preStop 10 + phase 20 + executor await 10 + margin 5
      containers:
        - name: app
          image: registry.example.com/orders:1.4.0
          ports:
            - containerPort: 8080
          env:
            - name: SPRING_LIFECYCLE_TIMEOUT_PER_SHUTDOWN_PHASE
              value: "20s"
            - name: SPRING_TASK_EXECUTION_SHUTDOWN_AWAIT_TERMINATION
              value: "true"
            - name: SPRING_TASK_EXECUTION_SHUTDOWN_AWAIT_TERMINATION_PERIOD
              value: "10s"
          readinessProbe:
            httpGet:
              path: /actuator/health/readiness
              port: 8080
            periodSeconds: 5
          lifecycle:
            preStop:
              sleep:
                seconds: 10

This manifest passes kubernetes-validate --strict for Kubernetes 1.33 and 1.34 (not applied to a cluster). The env entries are Spring Boot properties written as environment variables; the checklist shows them as application.yaml. Three details decide whether the manifest works:

  • The sleep action: preStop.sleep runs inside the kubelet, so it works in distroless images that have no shell or sleep binary. It became beta, on by default, in Kubernetes 1.30 and stable in 1.34. On older clusters, use exec with ["sleep", "10"], which needs a sleep binary in the image.
  • The grace period includes the hook: the documentation says the countdown "begins before the PreStop hook is executed". With a 10-second sleep and the default grace period of 30 seconds, Spring has only 20 seconds left. If the hook is still running when the period ends, the kubelet sends SIGTERM anyway and allows a one-time 2 more seconds before SIGKILL.
  • The sleep length: it has to cover the slowest component on the network path. A sleep of 5 to 10 seconds usually covers kube-proxy alone; an external cloud load balancer that checks its targets every 10 seconds needs longer. There is no signal that tells the pod the removal has finished, so the value is an estimate.

The readiness probe does not close the race. A terminating pod's endpoint is marked not ready because the pod is terminating, not because a probe failed, so the probe adds nothing on the network path. It is still needed so that a new pod gets traffic only once it is ready. Spring Boot does switch its readiness state to REFUSING_TRAFFIC on shutdown, as the next program shows, but only after SIGTERM, too late to help.

With maxUnavailable: 0 and maxSurge: 1, a rollout starts a new pod and waits for it to become ready before it deletes an old one, so capacity never drops while old pods drain.

Three setups need more than a sleep:

  • Load balancers that target pod IPs directly, such as the AWS Load Balancer Controller in IP target mode or GKE container-native load balancing. They deregister a pod through a cloud API, which can lag behind kube-proxy, so the preStop sleep must cover that lag. Pod readiness gates make a rollout wait until new pods are healthy in the load balancer: GKE adds them on its own, the AWS controller only in namespaces labelled elbv2.k8s.aws/pod-readiness-gate-inject: enabled. AWS also keeps an old target draining for its deregistration delay, 300 seconds by default; lower it with the target group attribute deregistration_delay.timeout_seconds rather than raising the grace period to match.
  • Long-lived connections (HTTP/2, gRPC, WebSocket, server-sent events). Endpoint removal steers only new connections, so the server must close existing ones during the drain (HTTP/2 does this with a GOAWAY frame) and clients must reconnect.
  • Service meshes: a proxy sidecar that stops at the same time as the application breaks the application's outbound calls during the drain. A native sidecar, stable since Kubernetes 1.33 and injected by default since Istio 1.27, is stopped only after the application container has exited.

SIGTERM in Spring Boot: draining an in-flight request

When the JVM receives SIGTERM, it runs its shutdown hooks. Spring Boot registers one that closes the application context, and closing the context performs the graceful shutdown. To show each step, this program starts a Spring Boot application with a /slow endpoint that takes 3 seconds, sends SIGTERM while a request is in progress, and then tries two more requests: one on a new connection, and one on a keep-alive connection a second client opened earlier.

Java
package demo.shutdown.drain;

import jakarta.annotation.PreDestroy;
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.net.Socket;
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.nio.charset.StandardCharsets;
import java.util.Map;
import java.util.concurrent.BlockingQueue;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.LinkedBlockingQueue;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.boot.availability.AvailabilityChangeEvent;
import org.springframework.boot.web.server.context.WebServerInitializedEvent;
import org.springframework.context.event.ContextClosedEvent;
import org.springframework.context.event.EventListener;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RestController;

@SpringBootApplication
@RestController
public class SlowRequestDrain {

    // ---------- the application: runs in a child JVM ----------

    @GetMapping("/slow")
    String slow() throws InterruptedException {
        System.out.println("handler: /slow started");
        Thread.sleep(3000);
        System.out.println("handler: /slow finished");
        return "slow result";
    }

    @GetMapping("/fast")
    String fast() {
        return "fast result";
    }

    @EventListener
    void started(WebServerInitializedEvent e) {
        System.out.println("PORT=" + e.getWebServer().getPort());
    }

    @EventListener
    void availability(AvailabilityChangeEvent<?> e) {
        System.out.println("event: " + e.getState().getClass().getSimpleName()
                + " = " + e.getState());
    }

    @EventListener
    void closed(ContextClosedEvent e) {
        System.out.println("event: ContextClosedEvent");
    }

    @PreDestroy
    void destroy() {
        System.out.println("bean: @PreDestroy");
    }

    static void runApp() {
        SpringApplication app = new SpringApplication(SlowRequestDrain.class);
        app.setDefaultProperties(Map.of(
                "server.port", "0",
                "spring.main.banner-mode", "off",
                "logging.level.root", "WARN",
                "logging.level.org.springframework.boot.tomcat.GracefulShutdown", "INFO",
                "logging.pattern.console", "log: %logger{0}: %msg%n"));
        app.run();
    }

    // ---------- the test driver: plays the kubelet and a client ----------

    static final BlockingQueue<String> LINES = new LinkedBlockingQueue<>();

    static String printUntil(String marker) throws InterruptedException {
        while (true) {
            String line = LINES.take();
            if (line.equals("EOF")) {
                return line;
            }
            if (!line.startsWith("PORT=")) {
                System.out.println("  [app] " + line);
            }
            if (line.contains(marker)) {
                return line;
            }
        }
    }

    // one HTTP/1.1 request on an open connection; returns the status line and Connection header
    static String fastOn(Socket socket) throws IOException {
        socket.getOutputStream().write("GET /fast HTTP/1.1\r\nHost: localhost\r\n\r\n"
                .getBytes(StandardCharsets.US_ASCII));
        BufferedReader in = new BufferedReader(new InputStreamReader(socket.getInputStream()));
        String status = in.readLine();
        if (status == null) {
            return "connection closed by the server, no response";
        }
        String line;
        String connection = "";
        int length = 0;
        while (!(line = in.readLine()).isEmpty()) {
            if (line.startsWith("Content-Length:")) {
                length = Integer.parseInt(line.substring(15).trim());
            } else if (line.startsWith("Connection:")) {
                connection = ", " + line;
            }
        }
        in.skip(length);
        return status.trim() + connection;
    }

    public static void main(String[] args) throws Exception {
        if (args.length > 0) {
            runApp();
            return;
        }
        String java = ProcessHandle.current().info().command().orElseThrow();
        Process app = new ProcessBuilder(java, "-cp", System.getProperty("java.class.path"),
                SlowRequestDrain.class.getName(), "app").redirectErrorStream(true).start();
        Thread.ofPlatform().daemon().start(() -> {
            try (BufferedReader r = new BufferedReader(
                    new InputStreamReader(app.getInputStream()))) {
                r.lines().forEach(LINES::add);
            } catch (Exception ignored) {
                // the process is gone
            }
            LINES.add("EOF");
        });
        String port = printUntil("PORT=").substring(5);
        HttpClient client = HttpClient.newHttpClient();
        String base = "http://localhost:" + port;
        Socket keepAlive = new Socket("localhost", Integer.parseInt(port));
        System.out.println("client 2: GET /fast -> " + fastOn(keepAlive)
                + " (keeps its connection open)");

        System.out.println("client: GET /slow");
        CompletableFuture<HttpResponse<String>> slow = client.sendAsync(
                HttpRequest.newBuilder(URI.create(base + "/slow")).build(),
                HttpResponse.BodyHandlers.ofString());
        printUntil("/slow started");

        long t0 = System.nanoTime();
        System.out.println("kubelet: SIGTERM");
        app.toHandle().destroy(); // sends SIGTERM on Linux and macOS
        printUntil("Commencing graceful shutdown");

        System.out.println("client: GET /fast on a new connection");
        try {
            HttpResponse<String> fast = HttpClient.newHttpClient().send(
                    HttpRequest.newBuilder(URI.create(base + "/fast")).build(),
                    HttpResponse.BodyHandlers.ofString());
            System.out.println("client: /fast -> " + fast.statusCode());
        } catch (Exception e) {
            System.out.println("client: /fast -> " + e.getClass().getSimpleName()
                    + (e.getMessage() == null ? "" : ": " + e.getMessage()));
        }

        System.out.println("client 2: GET /fast on its open connection -> "
                + fastOn(keepAlive));

        printUntil("Graceful shutdown complete");
        HttpResponse<String> r = slow.join();
        System.out.println("client: /slow -> " + r.statusCode() + " " + r.body());
        printUntil("EOF");
        int code = app.waitFor();
        long ms = (System.nanoTime() - t0) / 1_000_000;
        System.out.printf("process exited with code %d, %.0f s after SIGTERM%n",
                code, ms / 1000.0);
    }
}
Output
client 2: GET /fast -> HTTP/1.1 200 (keeps its connection open)
client: GET /slow
  [app] event: LivenessState = CORRECT
  [app] event: ReadinessState = ACCEPTING_TRAFFIC
  [app] handler: /slow started
kubelet: SIGTERM
  [app] event: ReadinessState = REFUSING_TRAFFIC
  [app] event: ContextClosedEvent
  [app] log: GracefulShutdown: Commencing graceful shutdown. Waiting for active requests to complete
client: GET /fast on a new connection
client: /fast -> ConnectException
client 2: GET /fast on its open connection -> connection closed by the server, no response
  [app] handler: /slow finished
  [app] log: GracefulShutdown: Graceful shutdown complete
client: /slow -> 200 slow result
  [app] bean: @PreDestroy
process exited with code 143, 3 s after SIGTERM
  • The LivenessState and ReadinessState lines come from startup. The driver prints the application's output only while it waits for the next line it expects, so they appear after GET /slow.
  • After SIGTERM, the readiness state changed to REFUSING_TRAFFIC, the ContextClosedEvent was published, and Tomcat logged Commencing graceful shutdown, at which point it stopped accepting connections.
  • The request on a new connection failed with ConnectException (connection refused) in this run. It races Tomcat closing its socket, so results vary: in 20 runs, 16 gave ConnectException and 4 gave an IOException, "header parser received no bytes", where the connection was accepted and then closed without a reply. A request can also get in before the socket closes and be answered with 200, though none of the 20 runs did. The failures are what a client gets when the network path is slower than the node path and no preStop sleep covers the gap.
  • The next request on the idle keep-alive connection got no response: the server closed the connection instead. Clients and proxies that reuse connections need to retry a request that fails this way; many do so automatically for idempotent requests such as GET.
  • The in-flight /slow request ran to the end and returned 200. Only after it finished did Tomcat log Graceful shutdown complete.
  • @PreDestroy ran after the web server had drained, then the JVM exited with code 143. That is 128 plus 15, the signal number of SIGTERM. It shows how the JVM stopped, not that every request finished: for that, the log must say Graceful shutdown complete, not Graceful shutdown aborted (see the last program).

Graceful shutdown applies to Tomcat, Jetty and Reactor Netty, the three servers Spring Boot 4 supports. server.shutdown=graceful has been the default since Spring Boot 3.4; server.shutdown=immediate turns it off. An IDE's stop button may not send SIGTERM, so test with kill -TERM <pid>.

Shutdown phases: the order in which beans stop

Closing the context takes several steps. Spring publishes the ContextClosedEvent, then stops every SmartLifecycle bean: a bean with its own start and stop, such as the web server, the @Async executor or a Kafka listener container. Each has a phase, an int, and Spring stops the highest phase first. It then destroys the beans, which runs @PreDestroy and DisposableBean.destroy(). Beans in the same phase stop together, and Spring waits for each phase up to spring.lifecycle.timeout-per-shutdown-phase. This program adds two lifecycle beans to a small web application, lists every SmartLifecycle bean, and closes the context with a 1-second phase timeout:

Java
package demo.shutdown.phases;

import jakarta.annotation.PreDestroy;
import java.util.Comparator;
import java.util.Map;
import org.springframework.beans.factory.DisposableBean;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.context.ConfigurableApplicationContext;
import org.springframework.context.SmartLifecycle;
import org.springframework.context.annotation.Bean;
import org.springframework.context.event.ContextClosedEvent;
import org.springframework.context.event.EventListener;
import org.springframework.stereotype.Component;

@SpringBootApplication
public class ShutdownPhases {

    // a component that is started and stopped with the context, in a given phase
    static class Worker implements SmartLifecycle {
        private final String name;
        private final int phase;
        private final long stopMillis;
        private volatile boolean running;

        Worker(String name, int phase, long stopMillis) {
            this.name = name;
            this.phase = phase;
            this.stopMillis = stopMillis;
        }

        @Override
        public void start() {
            running = true;
        }

        @Override
        public void stop() {
            running = false;
        }

        // the asynchronous variant: Spring waits until the callback runs or the phase times out
        @Override
        public void stop(Runnable callback) {
            System.out.println("stop " + name + " (phase " + phase + ")");
            Thread.ofPlatform().start(() -> {
                try {
                    Thread.sleep(stopMillis);
                } catch (InterruptedException e) {
                    Thread.currentThread().interrupt();
                }
                running = false;
                System.out.println("  " + name + " stopped after " + stopMillis + " ms");
                callback.run();
            });
        }

        @Override
        public boolean isRunning() {
            return running;
        }

        @Override
        public int getPhase() {
            return phase;
        }
    }

    @Bean
    Worker messageConsumer() {
        return new Worker("messageConsumer", SmartLifecycle.DEFAULT_PHASE, 200);
    }

    @Bean
    Worker bufferFlusher() {
        return new Worker("bufferFlusher", 1000, 2500); // slower than the phase timeout
    }

    @Component
    static class Cache implements DisposableBean {
        @PreDestroy
        void preDestroy() {
            System.out.println("Cache @PreDestroy");
        }

        @Override
        public void destroy() {
            System.out.println("Cache DisposableBean.destroy()");
        }
    }

    @EventListener
    void closed(ContextClosedEvent e) {
        System.out.println("ContextClosedEvent");
    }

    public static void main(String[] args) {
        SpringApplication app = new SpringApplication(ShutdownPhases.class);
        app.setDefaultProperties(Map.of(
                "server.port", "0",
                "spring.lifecycle.timeout-per-shutdown-phase", "1s",
                "spring.main.banner-mode", "off",
                "logging.level.root", "WARN",
                "logging.level.org.springframework.boot.tomcat.GracefulShutdown", "INFO",
                "logging.level.org.springframework.context.support.DefaultLifecycleProcessor",
                "INFO",
                "logging.pattern.console", "log: %logger{0}: %msg%n"));
        ConfigurableApplicationContext context = app.run();

        System.out.println("Lifecycle beans, in the order they stop:");
        context.getBeansOfType(SmartLifecycle.class).entrySet().stream()
                .sorted(Comparator.comparingInt(
                        (Map.Entry<String, SmartLifecycle> e) -> e.getValue().getPhase())
                        .reversed())
                .forEach(e -> System.out.printf("  %-11d %s%n",
                        e.getValue().getPhase(), e.getKey()));

        System.out.println("close() (what the SIGTERM shutdown hook calls):");
        context.close();
        System.out.println("closed");
    }
}
Output
Lifecycle beans, in the order they stop:
  2147483647  messageConsumer
  2147483547  org.springframework.kafka.config.internalKafkaListenerEndpointRegistry
  2147482623  webServerGracefulShutdown
  2147481599  webServerStartStop
  1073741823  applicationTaskExecutor
  1000        bufferFlusher
  -2147483647 springBootLoggingLifecycle
  -2147483648 kafkaProducerFactory
close() (what the SIGTERM shutdown hook calls):
ContextClosedEvent
stop messageConsumer (phase 2147483647)
  messageConsumer stopped after 200 ms
log: GracefulShutdown: Commencing graceful shutdown. Waiting for active requests to complete
log: GracefulShutdown: Graceful shutdown complete
stop bufferFlusher (phase 1000)
log: DefaultLifecycleProcessor: Shutdown phase 1000 ends with 1 bean still running after timeout of 1000ms: [bufferFlusher]
Cache @PreDestroy
Cache DisposableBean.destroy()
closed
  bufferFlusher stopped after 2500 ms
  • The list is the stop order. messageConsumer uses the default phase, Integer.MAX_VALUE (2147483647), so it stops first, before the web server drains. The Kafka starter is on this program's class path too, and its listener registry, at Integer.MAX_VALUE - 100, also stops ahead of the web server. The Kafka producer factory has the lowest possible phase, so it stops last and producers stay usable while requests drain.
  • webServerGracefulShutdown is the drain from the previous section, at Integer.MAX_VALUE - 1024. webServerStartStop, at Integer.MAX_VALUE - 2048, then stops the server itself. A bean left at the default phase stops before the drain, as messageConsumer did; a bean that in-flight requests still use needs a phase below webServerGracefulShutdown.
  • applicationTaskExecutor, the executor behind @Async, is at Integer.MAX_VALUE / 2, the default phase for executors and schedulers since Spring Framework 6.2. It stops after the web server, so a request that starts an @Async task during the drain still finds the executor running.
  • bufferFlusher, standing in for a bean that flushes buffered work, needed 2.5 seconds to stop, but its phase timed out after 1 second. Spring logged that the phase ended with a bean still running and moved on: Cache's @PreDestroy and destroy() ran, and close() returned while the flusher was still working. Its last line appears after closed only because main called close() and the JVM waited for the flusher's thread. After a real SIGTERM, the JVM exits when the shutdown hook returns, cutting the flush off.

The timeout is per phase, not for the whole shutdown. Three slow phases with a 20-second timeout can take a minute in total. When you size terminationGracePeriodSeconds, count every phase that can be slow, not just the web server's drain.

@Async tasks and @Scheduled jobs

The web server's drain does not wait for work on Spring's executors. That depends on two Spring Boot properties, set separately for the task executor and the task scheduler. This program starts a @Scheduled job and queues three @Async exports on a one-thread pool, so one export runs and two wait. It closes the context while the job and the first export are running, once with the defaults and once with await-termination turned on:

Java
package demo.shutdown.executors;

import java.util.HashMap;
import java.util.Map;
import java.util.concurrent.CountDownLatch;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.context.ConfigurableApplicationContext;
import org.springframework.scheduling.annotation.Async;
import org.springframework.scheduling.annotation.EnableAsync;
import org.springframework.scheduling.annotation.EnableScheduling;
import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.stereotype.Component;

@SpringBootApplication
@EnableAsync
@EnableScheduling
public class ExecutorShutdown {

    static volatile CountDownLatch jobStarted;

    static void work(String what, long millis) {
        System.out.println("  " + what + " started");
        try {
            Thread.sleep(millis);
            System.out.println("  " + what + " finished");
        } catch (InterruptedException e) {
            System.out.println("  " + what + " interrupted");
        }
    }

    @Component
    static class Exporter {
        @Async
        public void export(int id) {
            work("@Async export " + id, 1500);
        }
    }

    @Component
    static class Cleanup {
        @Scheduled(initialDelay = 0, fixedDelay = 60_000)
        void run() {
            jobStarted.countDown();
            work("@Scheduled cleanup", 2000);
        }
    }

    static void scenario(String title, boolean await) throws InterruptedException {
        System.out.println("== " + title);
        jobStarted = new CountDownLatch(1);
        SpringApplication app = new SpringApplication(ExecutorShutdown.class);
        Map<String, Object> props = new HashMap<>(Map.of(
                "spring.main.web-application-type", "none",
                "spring.main.banner-mode", "off",
                "logging.level.root", "ERROR",
                "spring.task.execution.pool.core-size", "1", // one thread: others queue
                "spring.task.execution.shutdown.await-termination", String.valueOf(await),
                "spring.task.scheduling.shutdown.await-termination", String.valueOf(await)));
        if (await) {
            props.put("spring.task.execution.shutdown.await-termination-period", "10s");
            props.put("spring.task.scheduling.shutdown.await-termination-period", "10s");
        }
        app.setDefaultProperties(props);
        ConfigurableApplicationContext context = app.run();
        Exporter exporter = context.getBean(Exporter.class);
        for (int id = 1; id <= 3; id++) {
            exporter.export(id);
        }
        jobStarted.await();
        Thread.sleep(300); // both tasks are now running
        System.out.println("  -- closing the context");
        long t0 = System.nanoTime();
        context.close();
        System.out.printf("  -- close() returned after %.1f s%n",
                (System.nanoTime() - t0) / 1e9);
        try {
            exporter.export(4);
        } catch (RuntimeException e) {
            System.out.println("  export 4 after close: " + e.getClass().getSimpleName());
        }
    }

    public static void main(String[] args) throws InterruptedException {
        scenario("Defaults: await-termination=false", false);
        Thread.sleep(2500); // let anything still running print
        scenario("await-termination=true, period 10s", true);
    }
}
Output
== Defaults: await-termination=false
  @Async export 1 started
  @Scheduled cleanup started
  -- closing the context
  @Async export 1 finished
  @Async export 2 started
  @Scheduled cleanup finished
  @Async export 2 interrupted
  -- close() returned after 1.7 s
  export 4 after close: TaskRejectedException
== await-termination=true, period 10s
  @Scheduled cleanup started
  @Async export 1 started
  -- closing the context
  @Async export 1 finished
  @Async export 2 started
  @Scheduled cleanup finished
  @Async export 2 finished
  @Async export 3 started
  @Async export 3 finished
  -- close() returned after 4.2 s
  export 4 after close: TaskRejectedException
  • With the defaults, the executor and the scheduler are both lifecycle beans, and each one's stop waits, within the phase timeout, for its running tasks: export 1 and the cleanup job both finished. Meanwhile export 2 started from the queue. When the context then destroyed the executor, the executor's shutdownNow() interrupted export 2 and dropped export 3 from the queue.
  • With await-termination=true, the executor ran everything it had, including the queued exports 2 and 3, and close() took 4.2 seconds to return instead of 1.7. The period, 10 seconds here, caps the wait. After it, Spring logs a warning and carries on closing without interrupting anything; a task still running is cut off when the JVM exits.
  • In both runs, the executor kept accepting tasks during the close and rejected them only once destroyed: export 4, submitted after close(), got TaskRejectedException.
Spring Boot executor and scheduler shutdown properties
PropertyDefaultEffect
spring.task.execution.shutdown.await-terminationfalseLet queued @Async tasks finish instead of interrupting and discarding them
spring.task.execution.shutdown.await-termination-periodnot setThe longest the executor blocks the shutdown for its tasks; without it, the shutdown does not wait at all
spring.task.scheduling.shutdown.await-terminationfalseThe same for the @Scheduled task scheduler
spring.task.scheduling.shutdown.await-termination-periodnot setThe longest the scheduler blocks the shutdown

An interrupted task stops only if it reacts to interruption. Thread.sleep and blocking I/O do; a CPU loop that never checks Thread.interrupted() does not, and runs on until the JVM exits, which ends it wherever it is. For work that must not be lost, such as sending an email after an order, an executor queue is the wrong place: it lives in memory and disappears with the pod. Write the work to an outbox table in the same database transaction as the order, and have a separate process deliver it from there, for example to a message broker.

Kafka consumers: commit and leave the group

A Kafka consumer has two duties when it stops. It should commit the offsets of the records it has processed, so the next consumer does not repeat them, and tell the group coordinator it is leaving, so its partitions are reassigned at once. A killed consumer does neither. This program runs an embedded Kafka broker, writes 8 records to each of two topics, one per run, and consumes them with a Spring Boot application in a child JVM. The consumer, pod A, is stopped after it processes its second record, with SIGTERM in the first run and SIGKILL in the second. Pod B then starts in the same consumer group:

Java
package demo.shutdown.kafka;

import ch.qos.logback.classic.Level;
import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.io.OutputStream;
import java.io.PrintStream;
import java.util.Collection;
import java.util.Map;
import java.util.concurrent.BlockingQueue;
import java.util.concurrent.LinkedBlockingQueue;
import org.apache.kafka.clients.consumer.Consumer;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.common.TopicPartition;
import org.apache.kafka.common.serialization.StringSerializer;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.context.annotation.Bean;
import org.springframework.context.event.EventListener;
import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.kafka.event.ConsumerStartingEvent;
import org.springframework.kafka.event.ConsumerStoppedEvent;
import org.springframework.kafka.listener.ConsumerAwareRebalanceListener;
import org.springframework.kafka.test.EmbeddedKafkaBroker;
import org.springframework.kafka.test.EmbeddedKafkaKraftBroker;

@SpringBootApplication
public class KafkaConsumerShutdown {

    // ---------- the consumer application: runs in a child JVM ----------

    static volatile long consumerStart;

    @EventListener
    void starting(ConsumerStartingEvent e) {
        consumerStart = System.nanoTime();
    }

    @KafkaListener(id = "billing", topics = "${app.topic}", groupId = "billing")
    void on(ConsumerRecord<String, String> r) throws InterruptedException {
        Thread.sleep(400); // the work for one record
        System.out.println("processed " + r.value() + " (offset " + r.offset() + ")");
    }

    @Bean
    ConsumerAwareRebalanceListener rebalanceListener() {
        return new ConsumerAwareRebalanceListener() {
            @Override
            public void onPartitionsAssigned(Consumer<?, ?> consumer,
                    Collection<TopicPartition> partitions) {
                if (!partitions.isEmpty()) {
                    System.out.printf("partition assigned %.0f s after the consumer started%n",
                            (System.nanoTime() - consumerStart) / 1e9);
                }
            }

            @Override
            public void onPartitionsRevokedBeforeCommit(Consumer<?, ?> consumer,
                    Collection<TopicPartition> partitions) {
                System.out.println("partition revoked, offsets about to be committed");
            }
        };
    }

    @EventListener
    void stopped(ConsumerStoppedEvent e) {
        System.out.println("consumer closed: " + e.getReason());
    }

    static void runApp(String bootstrap, String topic) {
        SpringApplication app = new SpringApplication(KafkaConsumerShutdown.class);
        app.setDefaultProperties(Map.of(
                "spring.kafka.bootstrap-servers", bootstrap,
                "app.topic", topic,
                "spring.kafka.consumer.auto-offset-reset", "earliest",
                "spring.kafka.consumer.max-poll-records", "4",
                "spring.kafka.consumer.properties.session.timeout.ms", "10000",
                "spring.main.web-application-type", "none",
                "spring.main.banner-mode", "off",
                "logging.level.root", "ERROR"));
        app.run();
    }

    // ---------- the test driver: a broker, a producer, and the kubelet ----------

    // a child JVM, with its output collected line by line
    record Pod(String name, Process process, BlockingQueue<String> lines) {

        static Pod start(String name, String bootstrap, String topic) throws Exception {
            String java = ProcessHandle.current().info().command().orElseThrow();
            Process p = new ProcessBuilder(java, "-cp", System.getProperty("java.class.path"),
                    KafkaConsumerShutdown.class.getName(), bootstrap, topic)
                    .redirectErrorStream(true).start();
            BlockingQueue<String> lines = new LinkedBlockingQueue<>();
            Thread.ofPlatform().daemon().start(() -> {
                try (BufferedReader r = new BufferedReader(
                        new InputStreamReader(p.getInputStream()))) {
                    r.lines().forEach(lines::add);
                } catch (Exception ignored) {
                    // the process is gone
                }
                lines.add("EOF");
            });
            return new Pod(name, p, lines);
        }

        // print the pod's output until a line contains the marker, or the pod ends
        void printUntil(String marker) throws InterruptedException {
            String line;
            while (!(line = lines.take()).equals("EOF")) {
                System.out.println("  [" + name + "] " + line);
                if (line.contains(marker)) {
                    return;
                }
            }
        }
    }

    static void scenario(String title, String bootstrap, String topic, boolean kill)
            throws Exception {
        System.out.println("== " + title);
        Pod a = Pod.start("pod A", bootstrap, topic);
        a.printUntil("order 2 (");
        if (kill) {
            System.out.println("kubelet: SIGKILL pod A");
            a.process().toHandle().destroyForcibly();
        } else {
            System.out.println("kubelet: SIGTERM pod A");
            a.process().toHandle().destroy();
        }
        a.printUntil("EOF");
        System.out.println("pod A exit code " + a.process().waitFor());

        Pod b = Pod.start("pod B", bootstrap, topic);
        b.printUntil("order 8 (");
        b.process().toHandle().destroy();
        b.process().waitFor();
    }

    public static void main(String[] args) throws Exception {
        if (args.length > 0) {
            runApp(args[0], args[1]);
            return;
        }
        EmbeddedKafkaBroker broker = startBroker();
        String bootstrap = broker.getBrokersAsString();
        try (var producer = new KafkaProducer<String, String>(Map.of(
                "bootstrap.servers", bootstrap), new StringSerializer(), new StringSerializer())) {
            for (String topic : new String[] {"orders-a", "orders-b"}) {
                for (int i = 1; i <= 8; i++) {
                    producer.send(new ProducerRecord<>(topic, "order " + i)).get();
                }
            }
        }
        scenario("SIGTERM: graceful stop", bootstrap, "orders-a", false);
        scenario("SIGKILL: no shutdown at all", bootstrap, "orders-b", true);
        broker.destroy();
    }

    static EmbeddedKafkaBroker startBroker() {
        // the broker logs at INFO, and prints its storage setup, so both are silenced.
        // Logback's Logger is named in full: org.slf4j.Logger, imported, has no setLevel
        ((ch.qos.logback.classic.Logger) LoggerFactory.getLogger(Logger.ROOT_LOGGER_NAME))
                .setLevel(Level.ERROR);
        PrintStream stdout = System.out;
        System.setOut(new PrintStream(OutputStream.nullOutputStream()));
        try {
            EmbeddedKafkaBroker broker = new EmbeddedKafkaKraftBroker(1, 1,
                    "orders-a", "orders-b").brokerProperties(Map.of(
                            "transaction.state.log.replication.factor", "1",
                            "transaction.state.log.min.isr", "1",
                            // no 3 s wait before the first assignment of an empty group
                            "group.initial.rebalance.delay.ms", "0"));
            broker.afterPropertiesSet();
            return broker;
        } finally {
            System.setOut(stdout);
        }
    }
}
Output
== SIGTERM: graceful stop
  [pod A] partition assigned 1 s after the consumer started
  [pod A] processed order 1 (offset 0)
  [pod A] processed order 2 (offset 1)
kubelet: SIGTERM pod A
  [pod A] processed order 3 (offset 2)
  [pod A] processed order 4 (offset 3)
  [pod A] partition revoked, offsets about to be committed
  [pod A] consumer closed: NORMAL
pod A exit code 143
  [pod B] partition assigned 0 s after the consumer started
  [pod B] processed order 5 (offset 4)
  [pod B] processed order 6 (offset 5)
  [pod B] processed order 7 (offset 6)
  [pod B] processed order 8 (offset 7)
== SIGKILL: no shutdown at all
  [pod A] partition assigned 0 s after the consumer started
  [pod A] processed order 1 (offset 0)
  [pod A] processed order 2 (offset 1)
kubelet: SIGKILL pod A
pod A exit code 137
  [pod B] partition assigned 7 s after the consumer started
  [pod B] processed order 1 (offset 0)
  [pod B] processed order 2 (offset 1)
  [pod B] processed order 3 (offset 2)
  [pod B] processed order 4 (offset 3)
  [pod B] processed order 5 (offset 4)
  [pod B] processed order 6 (offset 5)
  [pod B] processed order 7 (offset 6)
  [pod B] processed order 8 (offset 7)
  • After SIGTERM, pod A still processed orders 3 and 4. With max.poll.records set to 4, they were part of the batch it had already fetched, and by default a listener container stops only after finishing the records of the current poll.
  • Pod A then gave up its partition, committed the offsets of all four records, and closed the consumer, which sent the coordinator a request to leave the group. Pod B was given the partition almost as soon as its consumer started, and began at order 5. Nothing was processed twice. This assumes dynamic membership, the default: a consumer with a group.instance.id (static membership) stays in the group on close, and its partitions wait until its session times out.
  • After SIGKILL, pod A committed nothing. Pod B started from offset 0 and processed orders 1 and 2 a second time.
  • Pod B also waited 7 seconds for its partition in this run. The coordinator still counted the killed consumer as a member until its session timed out, 10 seconds after its last heartbeat in this program; the rest of that time passed while pod B's JVM started. With the Kafka default session.timeout.ms of 45 seconds, each killed consumer pauses its partitions for up to that long.

Spring for Apache Kafka commits and leaves the group on a normal context close, so a graceful SIGTERM is enough. Two settings decide how long the consumer takes to stop:

  • spring.kafka.listener.immediate-stop: false by default, so the container finishes every record of the current poll. With the default max.poll.records of 500 and slow processing, that can outlast the grace period. Set it to true to stop after the current record; the offsets of the records already processed are still committed.
  • The phase timeout: on a context close, the listener registry stops its containers asynchronously, and Spring waits for them up to spring.lifecycle.timeout-per-shutdown-phase. When it runs out, the shutdown moves on, but the consumer thread is not interrupted: the record in progress runs until the JVM exits or is killed, and its offset is not committed. ContainerProperties.setShutdownTimeout does not apply here; it bounds only a direct call to a container's stop().

At-least-once delivery means a consumer must cope with duplicates anyway, because a pod can always be killed. A clean shutdown makes them rare rather than routine. The @KafkaListener page explains acknowledgment modes and committed offsets in more depth.

When the grace period runs out

Two timers can end a shutdown early. Spring's phase timeout gives up on a phase and carries on closing; Kubernetes' grace period ends the process with SIGKILL. This program runs twice, each time sending SIGTERM during a 5-second request. The first run has a 2-second Spring phase timeout and a 30-second grace period. The second has a 30-second phase timeout and a 2-second grace period, after which the driver sends SIGKILL, as the kubelet would. A raw-socket client shows exactly what came back:

Java
package demo.shutdown.timeout;

import jakarta.annotation.PreDestroy;
import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.io.IOException;
import java.net.Socket;
import java.nio.charset.StandardCharsets;
import java.util.Map;
import java.util.concurrent.BlockingQueue;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.LinkedBlockingQueue;
import java.util.concurrent.TimeUnit;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.boot.web.server.context.WebServerInitializedEvent;
import org.springframework.context.event.EventListener;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RestController;

@SpringBootApplication
@RestController
public class GracePeriodRunsOut {

    // ---------- the application: runs in a child JVM ----------

    @GetMapping("/report")
    String report() throws InterruptedException {
        System.out.println("handler: /report started, needs 5 s");
        try {
            Thread.sleep(5000);
        } catch (InterruptedException e) {
            System.out.println("handler: interrupted");
            throw e;
        }
        System.out.println("handler: /report finished");
        return "report";
    }

    @EventListener
    void started(WebServerInitializedEvent e) {
        System.out.println("PORT=" + e.getWebServer().getPort());
    }

    @PreDestroy
    void destroy() {
        System.out.println("bean: @PreDestroy");
    }

    static void runApp(String timeoutPerPhase) {
        SpringApplication app = new SpringApplication(GracePeriodRunsOut.class);
        app.setDefaultProperties(Map.of(
                "server.port", "0",
                "spring.lifecycle.timeout-per-shutdown-phase", timeoutPerPhase,
                "spring.main.banner-mode", "off",
                "logging.level.root", "WARN",
                "logging.level.org.springframework.boot.tomcat.GracefulShutdown", "INFO",
                "logging.level.org.springframework.context.support", "WARN",
                "logging.pattern.console", "log: %logger{0}: %msg%n"));
        app.run();
    }

    // ---------- the test driver: plays the kubelet and a client ----------

    static String printUntil(BlockingQueue<String> lines, String marker)
            throws InterruptedException {
        while (true) {
            String line = lines.take();
            if (line.equals("EOF")) {
                return line;
            }
            if (!line.startsWith("PORT=")) {
                System.out.println("  [app] " + line);
            }
            if (line.contains(marker)) {
                return line;
            }
        }
    }

    static void scenario(String title, String timeoutPerPhase, int graceSeconds)
            throws Exception {
        System.out.println("== " + title);
        String java = ProcessHandle.current().info().command().orElseThrow();
        Process app = new ProcessBuilder(java, "-cp", System.getProperty("java.class.path"),
                GracePeriodRunsOut.class.getName(), timeoutPerPhase)
                .redirectErrorStream(true).start();
        BlockingQueue<String> lines = new LinkedBlockingQueue<>();
        Thread.ofPlatform().daemon().start(() -> {
            try (BufferedReader r = new BufferedReader(
                    new InputStreamReader(app.getInputStream()))) {
                r.lines().forEach(lines::add);
            } catch (Exception ignored) {
                // the process is gone
            }
            lines.add("EOF");
        });
        String port = printUntil(lines, "PORT=").substring(5);
        CompletableFuture<String> call = CompletableFuture.supplyAsync(() -> get(port));
        printUntil(lines, "/report started");

        System.out.println("kubelet: SIGTERM, grace period " + graceSeconds + " s");
        app.toHandle().destroy();
        printUntil(lines, "Commencing graceful shutdown");
        if (!app.waitFor(graceSeconds, TimeUnit.SECONDS)) {
            System.out.println("kubelet: grace period over, SIGKILL");
            app.toHandle().destroyForcibly();
        }
        printUntil(lines, "EOF");
        System.out.println("process exit code " + app.waitFor());
        System.out.println("client: " + call.join());
    }

    // a raw HTTP request, so that the client shows exactly what came back
    static String get(String port) {
        try (Socket socket = new Socket("localhost", Integer.parseInt(port))) {
            socket.getOutputStream().write(("GET /report HTTP/1.1\r\nHost: localhost\r\n"
                    + "Connection: close\r\n\r\n").getBytes(StandardCharsets.US_ASCII));
            String response = new String(socket.getInputStream().readAllBytes(),
                    StandardCharsets.US_ASCII);
            return response.isEmpty() ? "connection closed, no response"
                    : response.lines().findFirst().orElseThrow();
        } catch (IOException e) {
            return e.getClass().getSimpleName() + ": " + e.getMessage();
        }
    }

    public static void main(String[] args) throws Exception {
        if (args.length > 0) {
            runApp(args[0]);
            return;
        }
        scenario("Spring's shutdown phase times out first", "2s", 30);
        scenario("The kubelet's grace period runs out first", "30s", 2);
    }
}
Output
== Spring's shutdown phase times out first
  [app] handler: /report started, needs 5 s
kubelet: SIGTERM, grace period 30 s
  [app] log: GracefulShutdown: Commencing graceful shutdown. Waiting for active requests to complete
  [app] handler: interrupted
  [app] log: GracefulShutdown: Graceful shutdown aborted with one or more requests still active
  [app] bean: @PreDestroy
process exit code 143
client: connection closed, no response
== The kubelet's grace period runs out first
  [app] handler: /report started, needs 5 s
kubelet: SIGTERM, grace period 2 s
  [app] log: GracefulShutdown: Commencing graceful shutdown. Waiting for active requests to complete
kubelet: grace period over, SIGKILL
process exit code 137
client: connection closed, no response
  • In the first run, the drain timed out after 2 seconds. Spring moved on to stop Tomcat, which interrupted the request thread, and Tomcat logged Graceful shutdown aborted with one or more requests still active; the client's connection was closed with no response. The rest of the shutdown still ran: @PreDestroy printed and the exit code was 143.
  • In the second run, the process was killed during the drain. No shutdown callback ran after that point, not even @PreDestroy, and the exit code was 137, which is 128 plus 9, the signal number of SIGKILL. The client again got a closed connection.

From the client's side the two failures look the same. For the application, the first is much better: pools close, buffered logs and metrics flush, and @PreDestroy runs, so Spring's timeout should end before the grace period. An exit code of 137 in a container's last state, with a reason other than OOMKilled, usually means the grace period was too short or the signal never reached the JVM. That state is kept only when a container restarts in place, for example after a failed liveness probe; a pod deleted by a rollout is gone once it stops, so check its log instead.

Checklist

Size the grace period as a sum. With the values from the manifest above:

Text
terminationGracePeriodSeconds  >=  preStop sleep
                                   + timeout-per-shutdown-phase for each phase that can be slow
                                   + executor await-termination-period, if set
                                   + a few seconds of margin
45                             >=  10 + 20 + 10 + 5

This assumes one slow phase, the web server's drain. Kafka listeners add a second slow phase with the same 20-second cap, which gives 65 seconds. With the defaults, a 30-second phase timeout and a 30-second grace period, a drain that uses its full timeout is cut off by SIGKILL, because the grace period started earlier and also covers any preStop sleep. Lower the phase timeout or raise the grace period. The Spring side of the manifest, as application.yaml:

YAML
spring:
  lifecycle:
    timeout-per-shutdown-phase: 20s
  task:
    execution:
      shutdown:
        await-termination: true
        await-termination-period: 10s
# server.shutdown is graceful by default
Settings for zero-downtime shutdown of a Spring Boot pod
ConcernSettingStarting point
Signal reaches the JVMExec-form ENTRYPOINT, Java as process 1See Docker for Java
Endpoint removal racelifecycle.preStop.sleep.seconds5 to 10 seconds; longer behind external load balancers
In-flight HTTP requestsserver.shutdown=graceful (default)Keep the default
Time for each shutdown phasespring.lifecycle.timeout-per-shutdown-phaseA little above your slowest normal request, such as 20s
@Async and @Scheduled workspring.task.*.shutdown.await-termination and spring.task.*.shutdown.await-termination-periodtrue with a period of 10 to 20 seconds, if the work matters
Kafka consumersGraceful close (default); spring.kafka.listener.immediate-stopimmediate-stop=true when batches are slow to process; count the listener phase in the budget
Hard deadlineterminationGracePeriodSecondsThe sum above; the default of 30 is often too short with a preStop sleep
Node drainsPodDisruptionBudgetmaxUnavailable: 1; limits voluntary evictions such as drains, not rolling updates
Capacity during rolloutsmaxUnavailable: 0, maxSurge: 1New pod ready before an old one is deleted
VerificationExit code and log143: stopped on SIGTERM, but check the log for Graceful shutdown aborted; 137: SIGKILL

Related pages: Kubernetes probes for Spring Boot for readiness and liveness, deploying Spring Boot on Kubernetes for Deployments and rolling updates, Spring @Async for executor configuration, and Spring Boot Actuator for the health endpoints behind the probes.