Graceful shutdown and zero-downtime deploys for Spring Boot on Kubernetes
Every rolling update stops old pods, and each stop is a chance to drop requests, interrupt jobs or replay Kafka records. Two things happen at once when a pod is deleted: Kubernetes stops routing to it, and the kubelet tells the application to stop. This page follows both paths from the pod's side, then shows each part of the shutdown in a running Spring Boot 4.1 application, most of it under a real SIGTERM.
Contents
What happens when a pod is deleted
A pod can be deleted for several reasons: a rolling update replaces it, a scale-down removes it, or a node is drained for maintenance. In these cases the API server does not stop anything itself. It marks the pod as terminating, records a deadline based on terminationGracePeriodSecondsterminationGracePeriodSeconds
- The node path: the kubelet is the agent on each node that runs pods' containers. It runs each container's
hook, if there is one, then asks the container runtime to sendpreStop to the container's main process, the one started bySIGTERM orENTRYPOINT (process 1 inside the container). Only process 1 gets the signal, so it must forward the signal to any child processes. If the container is still running when the grace period ends, the kubelet sendsCMD .SIGKILL asks the process to stop, and the JVM runs its shutdown code;SIGTERM ends the process at once, and no Java code runs.SIGKILL - The network path: EndpointSlices are the API objects that list the pod addresses behind a Service. Their controller marks the pod's endpoint as terminating, with
. kube-proxy, which turns Services into forwarding rules on every node, rewrites those rules. Ingress controllers and cloud load balancers update their lists of backends. Each of these watches the API server and reacts at its own pace.ready: false
The Kubernetes documentation states that, "at the same time as the kubelet is starting graceful shutdown of the Pod", the control plane evaluates whether to remove the pod from EndpointSlices. There is no ordering between the two paths: they race. If the application stops listening as soon as it receives SIGTERM
Zero-downtime shutdown therefore needs settings in the pod spec, the image and the application:
| Piece | Set in | What it does |
|---|---|---|
preStop | Pod spec | Delays SIGTERM |
terminationGracePeriodSeconds | Pod spec | Total time, including preStopSIGKILL |
| Exec-form entrypoint | Dockerfile | Makes the JVM process 1, so the signal reaches it |
server.shutdown=graceful | Spring Boot (default since 3.4) | Stops accepting connections, lets in-flight requests finish |
spring.lifecycle.timeout-per-shutdown-phase | Spring Boot (default 30s) | The longest Spring waits for each shutdown phase |
| Executor and scheduler await settings | Spring Boot | Whether queued @Async@Scheduled |
| Kafka listener container stop | Spring for Apache Kafka | Commits offsets and leaves the consumer group |
The exec form and process 1 are covered in ENTRYPOINTSIGTERM/bin/shSIGKILL
The programs below run Spring Boot 4.1.2 on Java 21. Three of them start the application in a child JVM and send it a real SIGTERMSIGKILLcontext.close()SIGTERMkubectl
preStop and the grace period
The usual fix for the race is to wait a few seconds before the application hears about the shutdown. A preStopSIGTERM
apiVersion: apps/v1
kind: Deployment
metadata:
name: orders
spec:
replicas: 3
selector:
matchLabels:
app: orders
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
maxUnavailable: 0
template:
metadata:
labels:
app: orders
spec:
terminationGracePeriodSeconds: 45 # preStop 10 + phase 20 + executor await 10 + margin 5
containers:
- name: app
image: registry.example.com/orders:1.4.0
ports:
- containerPort: 8080
env:
- name: SPRING_LIFECYCLE_TIMEOUT_PER_SHUTDOWN_PHASE
value: "20s"
- name: SPRING_TASK_EXECUTION_SHUTDOWN_AWAIT_TERMINATION
value: "true"
- name: SPRING_TASK_EXECUTION_SHUTDOWN_AWAIT_TERMINATION_PERIOD
value: "10s"
readinessProbe:
httpGet:
path: /actuator/health/readiness
port: 8080
periodSeconds: 5
lifecycle:
preStop:
sleep:
seconds: 10
This manifest passes kubernetes-validate --strictenvapplication.yaml
- The sleep action:
runs inside the kubelet, so it works in distroless images that have no shell orpreStop.sleep binary. It became beta, on by default, in Kubernetes 1.30 and stable in 1.34. On older clusters, usesleep withexec , which needs a["sleep", "10"] binary in the image.sleep - The grace period includes the hook: the documentation says the countdown "begins before the PreStop hook is executed". With a 10-second sleep and the default grace period of 30 seconds, Spring has only 20 seconds left. If the hook is still running when the period ends, the kubelet sends
anyway and allows a one-time 2 more seconds beforeSIGTERM .SIGKILL - The sleep length: it has to cover the slowest component on the network path. A sleep of 5 to 10 seconds usually covers kube-proxy alone; an external cloud load balancer that checks its targets every 10 seconds needs longer. There is no signal that tells the pod the removal has finished, so the value is an estimate.
The readiness probe does not close the race. A terminating pod's endpoint is marked not ready because the pod is terminating, not because a probe failed, so the probe adds nothing on the network path. It is still needed so that a new pod gets traffic only once it is ready. Spring Boot does switch its readiness state to REFUSING_TRAFFICSIGTERM
With maxUnavailable: 0maxSurge: 1
Three setups need more than a sleep:
- Load balancers that target pod IPs directly, such as the AWS Load Balancer Controller in IP target mode or GKE container-native load balancing. They deregister a pod through a cloud API, which can lag behind kube-proxy, so the
sleep must cover that lag. Pod readiness gates make a rollout wait until new pods are healthy in the load balancer: GKE adds them on its own, the AWS controller only in namespaces labelledpreStop . AWS also keeps an old target draining for its deregistration delay, 300 seconds by default; lower it with the target group attributeelbv2.k8s.aws/pod-readiness-gate-inject: enabled rather than raising the grace period to match.deregistration_delay.timeout_seconds - Long-lived connections (HTTP/2, gRPC, WebSocket, server-sent events). Endpoint removal steers only new connections, so the server must close existing ones during the drain (HTTP/2 does this with a
frame) and clients must reconnect.GOAWAY - Service meshes: a proxy sidecar that stops at the same time as the application breaks the application's outbound calls during the drain. A native sidecar, stable since Kubernetes 1.33 and injected by default since Istio 1.27, is stopped only after the application container has exited.
SIGTERM in Spring Boot: draining an in-flight request
When the JVM receives SIGTERM/slowSIGTERM
package demo.shutdown.drain;
import jakarta.annotation.PreDestroy;
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.net.Socket;
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.nio.charset.StandardCharsets;
import java.util.Map;
import java.util.concurrent.BlockingQueue;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.LinkedBlockingQueue;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.boot.availability.AvailabilityChangeEvent;
import org.springframework.boot.web.server.context.WebServerInitializedEvent;
import org.springframework.context.event.ContextClosedEvent;
import org.springframework.context.event.EventListener;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RestController;
@SpringBootApplication
@RestController
public class SlowRequestDrain {
// ---------- the application: runs in a child JVM ----------
@GetMapping("/slow")
String slow() throws InterruptedException {
System.out.println("handler: /slow started");
Thread.sleep(3000);
System.out.println("handler: /slow finished");
return "slow result";
}
@GetMapping("/fast")
String fast() {
return "fast result";
}
@EventListener
void started(WebServerInitializedEvent e) {
System.out.println("PORT=" + e.getWebServer().getPort());
}
@EventListener
void availability(AvailabilityChangeEvent<?> e) {
System.out.println("event: " + e.getState().getClass().getSimpleName()
+ " = " + e.getState());
}
@EventListener
void closed(ContextClosedEvent e) {
System.out.println("event: ContextClosedEvent");
}
@PreDestroy
void destroy() {
System.out.println("bean: @PreDestroy");
}
static void runApp() {
SpringApplication app = new SpringApplication(SlowRequestDrain.class);
app.setDefaultProperties(Map.of(
"server.port", "0",
"spring.main.banner-mode", "off",
"logging.level.root", "WARN",
"logging.level.org.springframework.boot.tomcat.GracefulShutdown", "INFO",
"logging.pattern.console", "log: %logger{0}: %msg%n"));
app.run();
}
// ---------- the test driver: plays the kubelet and a client ----------
static final BlockingQueue<String> LINES = new LinkedBlockingQueue<>();
static String printUntil(String marker) throws InterruptedException {
while (true) {
String line = LINES.take();
if (line.equals("EOF")) {
return line;
}
if (!line.startsWith("PORT=")) {
System.out.println(" [app] " + line);
}
if (line.contains(marker)) {
return line;
}
}
}
// one HTTP/1.1 request on an open connection; returns the status line and Connection header
static String fastOn(Socket socket) throws IOException {
socket.getOutputStream().write("GET /fast HTTP/1.1\r\nHost: localhost\r\n\r\n"
.getBytes(StandardCharsets.US_ASCII));
BufferedReader in = new BufferedReader(new InputStreamReader(socket.getInputStream()));
String status = in.readLine();
if (status == null) {
return "connection closed by the server, no response";
}
String line;
String connection = "";
int length = 0;
while (!(line = in.readLine()).isEmpty()) {
if (line.startsWith("Content-Length:")) {
length = Integer.parseInt(line.substring(15).trim());
} else if (line.startsWith("Connection:")) {
connection = ", " + line;
}
}
in.skip(length);
return status.trim() + connection;
}
public static void main(String[] args) throws Exception {
if (args.length > 0) {
runApp();
return;
}
String java = ProcessHandle.current().info().command().orElseThrow();
Process app = new ProcessBuilder(java, "-cp", System.getProperty("java.class.path"),
SlowRequestDrain.class.getName(), "app").redirectErrorStream(true).start();
Thread.ofPlatform().daemon().start(() -> {
try (BufferedReader r = new BufferedReader(
new InputStreamReader(app.getInputStream()))) {
r.lines().forEach(LINES::add);
} catch (Exception ignored) {
// the process is gone
}
LINES.add("EOF");
});
String port = printUntil("PORT=").substring(5);
HttpClient client = HttpClient.newHttpClient();
String base = "http://localhost:" + port;
Socket keepAlive = new Socket("localhost", Integer.parseInt(port));
System.out.println("client 2: GET /fast -> " + fastOn(keepAlive)
+ " (keeps its connection open)");
System.out.println("client: GET /slow");
CompletableFuture<HttpResponse<String>> slow = client.sendAsync(
HttpRequest.newBuilder(URI.create(base + "/slow")).build(),
HttpResponse.BodyHandlers.ofString());
printUntil("/slow started");
long t0 = System.nanoTime();
System.out.println("kubelet: SIGTERM");
app.toHandle().destroy(); // sends SIGTERM on Linux and macOS
printUntil("Commencing graceful shutdown");
System.out.println("client: GET /fast on a new connection");
try {
HttpResponse<String> fast = HttpClient.newHttpClient().send(
HttpRequest.newBuilder(URI.create(base + "/fast")).build(),
HttpResponse.BodyHandlers.ofString());
System.out.println("client: /fast -> " + fast.statusCode());
} catch (Exception e) {
System.out.println("client: /fast -> " + e.getClass().getSimpleName()
+ (e.getMessage() == null ? "" : ": " + e.getMessage()));
}
System.out.println("client 2: GET /fast on its open connection -> "
+ fastOn(keepAlive));
printUntil("Graceful shutdown complete");
HttpResponse<String> r = slow.join();
System.out.println("client: /slow -> " + r.statusCode() + " " + r.body());
printUntil("EOF");
int code = app.waitFor();
long ms = (System.nanoTime() - t0) / 1_000_000;
System.out.printf("process exited with code %d, %.0f s after SIGTERM%n",
code, ms / 1000.0);
}
}
client 2: GET /fast -> HTTP/1.1 200 (keeps its connection open)
client: GET /slow
[app] event: LivenessState = CORRECT
[app] event: ReadinessState = ACCEPTING_TRAFFIC
[app] handler: /slow started
kubelet: SIGTERM
[app] event: ReadinessState = REFUSING_TRAFFIC
[app] event: ContextClosedEvent
[app] log: GracefulShutdown: Commencing graceful shutdown. Waiting for active requests to complete
client: GET /fast on a new connection
client: /fast -> ConnectException
client 2: GET /fast on its open connection -> connection closed by the server, no response
[app] handler: /slow finished
[app] log: GracefulShutdown: Graceful shutdown complete
client: /slow -> 200 slow result
[app] bean: @PreDestroy
process exited with code 143, 3 s after SIGTERM
- The
andLivenessState lines come from startup. The driver prints the application's output only while it waits for the next line it expects, so they appear afterReadinessState .GET /slow - After
, the readiness state changed toSIGTERM , theREFUSING_TRAFFIC was published, and Tomcat loggedContextClosedEvent , at which point it stopped accepting connections.Commencing graceful shutdown - The request on a new connection failed with
(connection refused) in this run. It races Tomcat closing its socket, so results vary: in 20 runs, 16 gaveConnectException and 4 gave anConnectException , "header parser received no bytes", where the connection was accepted and then closed without a reply. A request can also get in before the socket closes and be answered with 200, though none of the 20 runs did. The failures are what a client gets when the network path is slower than the node path and noIOException sleep covers the gap.preStop - The next request on the idle keep-alive connection got no response: the server closed the connection instead. Clients and proxies that reuse connections need to retry a request that fails this way; many do so automatically for idempotent requests such as
.GET - The in-flight
request ran to the end and returned/slow . Only after it finished did Tomcat log200 .Graceful shutdown complete ran after the web server had drained, then the JVM exited with code 143. That is 128 plus 15, the signal number of@PreDestroy . It shows how the JVM stopped, not that every request finished: for that, the log must saySIGTERM , notGraceful shutdown complete (see the last program).Graceful shutdown aborted
Graceful shutdown applies to Tomcat, Jetty and Reactor Netty, the three servers Spring Boot 4 supports. server.shutdown=gracefulserver.shutdown=immediateSIGTERMkill -TERM <pid>
Shutdown phases: the order in which beans stop
Closing the context takes several steps. Spring publishes the ContextClosedEventSmartLifecycle@Asyncint@PreDestroyDisposableBean.destroy()spring.lifecycle.timeout-per-shutdown-phaseSmartLifecycle
package demo.shutdown.phases;
import jakarta.annotation.PreDestroy;
import java.util.Comparator;
import java.util.Map;
import org.springframework.beans.factory.DisposableBean;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.context.ConfigurableApplicationContext;
import org.springframework.context.SmartLifecycle;
import org.springframework.context.annotation.Bean;
import org.springframework.context.event.ContextClosedEvent;
import org.springframework.context.event.EventListener;
import org.springframework.stereotype.Component;
@SpringBootApplication
public class ShutdownPhases {
// a component that is started and stopped with the context, in a given phase
static class Worker implements SmartLifecycle {
private final String name;
private final int phase;
private final long stopMillis;
private volatile boolean running;
Worker(String name, int phase, long stopMillis) {
this.name = name;
this.phase = phase;
this.stopMillis = stopMillis;
}
@Override
public void start() {
running = true;
}
@Override
public void stop() {
running = false;
}
// the asynchronous variant: Spring waits until the callback runs or the phase times out
@Override
public void stop(Runnable callback) {
System.out.println("stop " + name + " (phase " + phase + ")");
Thread.ofPlatform().start(() -> {
try {
Thread.sleep(stopMillis);
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
}
running = false;
System.out.println(" " + name + " stopped after " + stopMillis + " ms");
callback.run();
});
}
@Override
public boolean isRunning() {
return running;
}
@Override
public int getPhase() {
return phase;
}
}
@Bean
Worker messageConsumer() {
return new Worker("messageConsumer", SmartLifecycle.DEFAULT_PHASE, 200);
}
@Bean
Worker bufferFlusher() {
return new Worker("bufferFlusher", 1000, 2500); // slower than the phase timeout
}
@Component
static class Cache implements DisposableBean {
@PreDestroy
void preDestroy() {
System.out.println("Cache @PreDestroy");
}
@Override
public void destroy() {
System.out.println("Cache DisposableBean.destroy()");
}
}
@EventListener
void closed(ContextClosedEvent e) {
System.out.println("ContextClosedEvent");
}
public static void main(String[] args) {
SpringApplication app = new SpringApplication(ShutdownPhases.class);
app.setDefaultProperties(Map.of(
"server.port", "0",
"spring.lifecycle.timeout-per-shutdown-phase", "1s",
"spring.main.banner-mode", "off",
"logging.level.root", "WARN",
"logging.level.org.springframework.boot.tomcat.GracefulShutdown", "INFO",
"logging.level.org.springframework.context.support.DefaultLifecycleProcessor",
"INFO",
"logging.pattern.console", "log: %logger{0}: %msg%n"));
ConfigurableApplicationContext context = app.run();
System.out.println("Lifecycle beans, in the order they stop:");
context.getBeansOfType(SmartLifecycle.class).entrySet().stream()
.sorted(Comparator.comparingInt(
(Map.Entry<String, SmartLifecycle> e) -> e.getValue().getPhase())
.reversed())
.forEach(e -> System.out.printf(" %-11d %s%n",
e.getValue().getPhase(), e.getKey()));
System.out.println("close() (what the SIGTERM shutdown hook calls):");
context.close();
System.out.println("closed");
}
}
Lifecycle beans, in the order they stop:
2147483647 messageConsumer
2147483547 org.springframework.kafka.config.internalKafkaListenerEndpointRegistry
2147482623 webServerGracefulShutdown
2147481599 webServerStartStop
1073741823 applicationTaskExecutor
1000 bufferFlusher
-2147483647 springBootLoggingLifecycle
-2147483648 kafkaProducerFactory
close() (what the SIGTERM shutdown hook calls):
ContextClosedEvent
stop messageConsumer (phase 2147483647)
messageConsumer stopped after 200 ms
log: GracefulShutdown: Commencing graceful shutdown. Waiting for active requests to complete
log: GracefulShutdown: Graceful shutdown complete
stop bufferFlusher (phase 1000)
log: DefaultLifecycleProcessor: Shutdown phase 1000 ends with 1 bean still running after timeout of 1000ms: [bufferFlusher]
Cache @PreDestroy
Cache DisposableBean.destroy()
closed
bufferFlusher stopped after 2500 ms
- The list is the stop order.
uses the default phase,messageConsumer (2147483647), so it stops first, before the web server drains. The Kafka starter is on this program's class path too, and its listener registry, atInteger.MAX_VALUE , also stops ahead of the web server. The Kafka producer factory has the lowest possible phase, so it stops last and producers stay usable while requests drain.Integer.MAX_VALUE - 100 is the drain from the previous section, atwebServerGracefulShutdown .Integer.MAX_VALUE - 1024 , atwebServerStartStop , then stops the server itself. A bean left at the default phase stops before the drain, asInteger.MAX_VALUE - 2048 did; a bean that in-flight requests still use needs a phase belowmessageConsumer .webServerGracefulShutdown , the executor behindapplicationTaskExecutor , is at@Async , the default phase for executors and schedulers since Spring Framework 6.2. It stops after the web server, so a request that starts anInteger.MAX_VALUE / 2 task during the drain still finds the executor running.@Async , standing in for a bean that flushes buffered work, needed 2.5 seconds to stop, but its phase timed out after 1 second. Spring logged that the phase ended with a bean still running and moved on:bufferFlusher 'sCache and@PreDestroy ran, anddestroy() returned while the flusher was still working. Its last line appears afterclose() only becauseclosed calledmain and the JVM waited for the flusher's thread. After a realclose() , the JVM exits when the shutdown hook returns, cutting the flush off.SIGTERM
The timeout is per phase, not for the whole shutdown. Three slow phases with a 20-second timeout can take a minute in total. When you size terminationGracePeriodSeconds
@Async tasks and @Scheduled jobs
The web server's drain does not wait for work on Spring's executors. That depends on two Spring Boot properties, set separately for the task executor and the task scheduler. This program starts a @Scheduled@Asyncawait-termination
package demo.shutdown.executors;
import java.util.HashMap;
import java.util.Map;
import java.util.concurrent.CountDownLatch;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.context.ConfigurableApplicationContext;
import org.springframework.scheduling.annotation.Async;
import org.springframework.scheduling.annotation.EnableAsync;
import org.springframework.scheduling.annotation.EnableScheduling;
import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.stereotype.Component;
@SpringBootApplication
@EnableAsync
@EnableScheduling
public class ExecutorShutdown {
static volatile CountDownLatch jobStarted;
static void work(String what, long millis) {
System.out.println(" " + what + " started");
try {
Thread.sleep(millis);
System.out.println(" " + what + " finished");
} catch (InterruptedException e) {
System.out.println(" " + what + " interrupted");
}
}
@Component
static class Exporter {
@Async
public void export(int id) {
work("@Async export " + id, 1500);
}
}
@Component
static class Cleanup {
@Scheduled(initialDelay = 0, fixedDelay = 60_000)
void run() {
jobStarted.countDown();
work("@Scheduled cleanup", 2000);
}
}
static void scenario(String title, boolean await) throws InterruptedException {
System.out.println("== " + title);
jobStarted = new CountDownLatch(1);
SpringApplication app = new SpringApplication(ExecutorShutdown.class);
Map<String, Object> props = new HashMap<>(Map.of(
"spring.main.web-application-type", "none",
"spring.main.banner-mode", "off",
"logging.level.root", "ERROR",
"spring.task.execution.pool.core-size", "1", // one thread: others queue
"spring.task.execution.shutdown.await-termination", String.valueOf(await),
"spring.task.scheduling.shutdown.await-termination", String.valueOf(await)));
if (await) {
props.put("spring.task.execution.shutdown.await-termination-period", "10s");
props.put("spring.task.scheduling.shutdown.await-termination-period", "10s");
}
app.setDefaultProperties(props);
ConfigurableApplicationContext context = app.run();
Exporter exporter = context.getBean(Exporter.class);
for (int id = 1; id <= 3; id++) {
exporter.export(id);
}
jobStarted.await();
Thread.sleep(300); // both tasks are now running
System.out.println(" -- closing the context");
long t0 = System.nanoTime();
context.close();
System.out.printf(" -- close() returned after %.1f s%n",
(System.nanoTime() - t0) / 1e9);
try {
exporter.export(4);
} catch (RuntimeException e) {
System.out.println(" export 4 after close: " + e.getClass().getSimpleName());
}
}
public static void main(String[] args) throws InterruptedException {
scenario("Defaults: await-termination=false", false);
Thread.sleep(2500); // let anything still running print
scenario("await-termination=true, period 10s", true);
}
}
== Defaults: await-termination=false
@Async export 1 started
@Scheduled cleanup started
-- closing the context
@Async export 1 finished
@Async export 2 started
@Scheduled cleanup finished
@Async export 2 interrupted
-- close() returned after 1.7 s
export 4 after close: TaskRejectedException
== await-termination=true, period 10s
@Scheduled cleanup started
@Async export 1 started
-- closing the context
@Async export 1 finished
@Async export 2 started
@Scheduled cleanup finished
@Async export 2 finished
@Async export 3 started
@Async export 3 finished
-- close() returned after 4.2 s
export 4 after close: TaskRejectedException
- With the defaults, the executor and the scheduler are both lifecycle beans, and each one's stop waits, within the phase timeout, for its running tasks: export 1 and the cleanup job both finished. Meanwhile export 2 started from the queue. When the context then destroyed the executor, the executor's
interrupted export 2 and dropped export 3 from the queue.shutdownNow() - With
, the executor ran everything it had, including the queued exports 2 and 3, andawait-termination=true took 4.2 seconds to return instead of 1.7. The period, 10 seconds here, caps the wait. After it, Spring logs a warning and carries on closing without interrupting anything; a task still running is cut off when the JVM exits.close() - In both runs, the executor kept accepting tasks during the close and rejected them only once destroyed: export 4, submitted after
, gotclose() .TaskRejectedException
| Property | Default | Effect |
|---|---|---|
spring.task.execution.shutdown.await-termination | false | Let queued @Async |
spring.task.execution.shutdown.await-termination-period | not set | The longest the executor blocks the shutdown for its tasks; without it, the shutdown does not wait at all |
spring.task.scheduling.shutdown.await-termination | false | The same for the @Scheduled |
spring.task.scheduling.shutdown.await-termination-period | not set | The longest the scheduler blocks the shutdown |
An interrupted task stops only if it reacts to interruption. Thread.sleepThread.interrupted()
Kafka consumers: commit and leave the group
A Kafka consumer has two duties when it stops. It should commit the offsets of the records it has processed, so the next consumer does not repeat them, and tell the group coordinator it is leaving, so its partitions are reassigned at once. A killed consumer does neither. This program runs an embedded Kafka broker, writes 8 records to each of two topics, one per run, and consumes them with a Spring Boot application in a child JVM. The consumer, pod A, is stopped after it processes its second record, with SIGTERMSIGKILL
package demo.shutdown.kafka;
import ch.qos.logback.classic.Level;
import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.io.OutputStream;
import java.io.PrintStream;
import java.util.Collection;
import java.util.Map;
import java.util.concurrent.BlockingQueue;
import java.util.concurrent.LinkedBlockingQueue;
import org.apache.kafka.clients.consumer.Consumer;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.common.TopicPartition;
import org.apache.kafka.common.serialization.StringSerializer;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.context.annotation.Bean;
import org.springframework.context.event.EventListener;
import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.kafka.event.ConsumerStartingEvent;
import org.springframework.kafka.event.ConsumerStoppedEvent;
import org.springframework.kafka.listener.ConsumerAwareRebalanceListener;
import org.springframework.kafka.test.EmbeddedKafkaBroker;
import org.springframework.kafka.test.EmbeddedKafkaKraftBroker;
@SpringBootApplication
public class KafkaConsumerShutdown {
// ---------- the consumer application: runs in a child JVM ----------
static volatile long consumerStart;
@EventListener
void starting(ConsumerStartingEvent e) {
consumerStart = System.nanoTime();
}
@KafkaListener(id = "billing", topics = "${app.topic}", groupId = "billing")
void on(ConsumerRecord<String, String> r) throws InterruptedException {
Thread.sleep(400); // the work for one record
System.out.println("processed " + r.value() + " (offset " + r.offset() + ")");
}
@Bean
ConsumerAwareRebalanceListener rebalanceListener() {
return new ConsumerAwareRebalanceListener() {
@Override
public void onPartitionsAssigned(Consumer<?, ?> consumer,
Collection<TopicPartition> partitions) {
if (!partitions.isEmpty()) {
System.out.printf("partition assigned %.0f s after the consumer started%n",
(System.nanoTime() - consumerStart) / 1e9);
}
}
@Override
public void onPartitionsRevokedBeforeCommit(Consumer<?, ?> consumer,
Collection<TopicPartition> partitions) {
System.out.println("partition revoked, offsets about to be committed");
}
};
}
@EventListener
void stopped(ConsumerStoppedEvent e) {
System.out.println("consumer closed: " + e.getReason());
}
static void runApp(String bootstrap, String topic) {
SpringApplication app = new SpringApplication(KafkaConsumerShutdown.class);
app.setDefaultProperties(Map.of(
"spring.kafka.bootstrap-servers", bootstrap,
"app.topic", topic,
"spring.kafka.consumer.auto-offset-reset", "earliest",
"spring.kafka.consumer.max-poll-records", "4",
"spring.kafka.consumer.properties.session.timeout.ms", "10000",
"spring.main.web-application-type", "none",
"spring.main.banner-mode", "off",
"logging.level.root", "ERROR"));
app.run();
}
// ---------- the test driver: a broker, a producer, and the kubelet ----------
// a child JVM, with its output collected line by line
record Pod(String name, Process process, BlockingQueue<String> lines) {
static Pod start(String name, String bootstrap, String topic) throws Exception {
String java = ProcessHandle.current().info().command().orElseThrow();
Process p = new ProcessBuilder(java, "-cp", System.getProperty("java.class.path"),
KafkaConsumerShutdown.class.getName(), bootstrap, topic)
.redirectErrorStream(true).start();
BlockingQueue<String> lines = new LinkedBlockingQueue<>();
Thread.ofPlatform().daemon().start(() -> {
try (BufferedReader r = new BufferedReader(
new InputStreamReader(p.getInputStream()))) {
r.lines().forEach(lines::add);
} catch (Exception ignored) {
// the process is gone
}
lines.add("EOF");
});
return new Pod(name, p, lines);
}
// print the pod's output until a line contains the marker, or the pod ends
void printUntil(String marker) throws InterruptedException {
String line;
while (!(line = lines.take()).equals("EOF")) {
System.out.println(" [" + name + "] " + line);
if (line.contains(marker)) {
return;
}
}
}
}
static void scenario(String title, String bootstrap, String topic, boolean kill)
throws Exception {
System.out.println("== " + title);
Pod a = Pod.start("pod A", bootstrap, topic);
a.printUntil("order 2 (");
if (kill) {
System.out.println("kubelet: SIGKILL pod A");
a.process().toHandle().destroyForcibly();
} else {
System.out.println("kubelet: SIGTERM pod A");
a.process().toHandle().destroy();
}
a.printUntil("EOF");
System.out.println("pod A exit code " + a.process().waitFor());
Pod b = Pod.start("pod B", bootstrap, topic);
b.printUntil("order 8 (");
b.process().toHandle().destroy();
b.process().waitFor();
}
public static void main(String[] args) throws Exception {
if (args.length > 0) {
runApp(args[0], args[1]);
return;
}
EmbeddedKafkaBroker broker = startBroker();
String bootstrap = broker.getBrokersAsString();
try (var producer = new KafkaProducer<String, String>(Map.of(
"bootstrap.servers", bootstrap), new StringSerializer(), new StringSerializer())) {
for (String topic : new String[] {"orders-a", "orders-b"}) {
for (int i = 1; i <= 8; i++) {
producer.send(new ProducerRecord<>(topic, "order " + i)).get();
}
}
}
scenario("SIGTERM: graceful stop", bootstrap, "orders-a", false);
scenario("SIGKILL: no shutdown at all", bootstrap, "orders-b", true);
broker.destroy();
}
static EmbeddedKafkaBroker startBroker() {
// the broker logs at INFO, and prints its storage setup, so both are silenced.
// Logback's Logger is named in full: org.slf4j.Logger, imported, has no setLevel
((ch.qos.logback.classic.Logger) LoggerFactory.getLogger(Logger.ROOT_LOGGER_NAME))
.setLevel(Level.ERROR);
PrintStream stdout = System.out;
System.setOut(new PrintStream(OutputStream.nullOutputStream()));
try {
EmbeddedKafkaBroker broker = new EmbeddedKafkaKraftBroker(1, 1,
"orders-a", "orders-b").brokerProperties(Map.of(
"transaction.state.log.replication.factor", "1",
"transaction.state.log.min.isr", "1",
// no 3 s wait before the first assignment of an empty group
"group.initial.rebalance.delay.ms", "0"));
broker.afterPropertiesSet();
return broker;
} finally {
System.setOut(stdout);
}
}
}
== SIGTERM: graceful stop
[pod A] partition assigned 1 s after the consumer started
[pod A] processed order 1 (offset 0)
[pod A] processed order 2 (offset 1)
kubelet: SIGTERM pod A
[pod A] processed order 3 (offset 2)
[pod A] processed order 4 (offset 3)
[pod A] partition revoked, offsets about to be committed
[pod A] consumer closed: NORMAL
pod A exit code 143
[pod B] partition assigned 0 s after the consumer started
[pod B] processed order 5 (offset 4)
[pod B] processed order 6 (offset 5)
[pod B] processed order 7 (offset 6)
[pod B] processed order 8 (offset 7)
== SIGKILL: no shutdown at all
[pod A] partition assigned 0 s after the consumer started
[pod A] processed order 1 (offset 0)
[pod A] processed order 2 (offset 1)
kubelet: SIGKILL pod A
pod A exit code 137
[pod B] partition assigned 7 s after the consumer started
[pod B] processed order 1 (offset 0)
[pod B] processed order 2 (offset 1)
[pod B] processed order 3 (offset 2)
[pod B] processed order 4 (offset 3)
[pod B] processed order 5 (offset 4)
[pod B] processed order 6 (offset 5)
[pod B] processed order 7 (offset 6)
[pod B] processed order 8 (offset 7)
- After
, pod A still processed orders 3 and 4. WithSIGTERM set to 4, they were part of the batch it had already fetched, and by default a listener container stops only after finishing the records of the current poll.max.poll.records - Pod A then gave up its partition, committed the offsets of all four records, and closed the consumer, which sent the coordinator a request to leave the group. Pod B was given the partition almost as soon as its consumer started, and began at order 5. Nothing was processed twice. This assumes dynamic membership, the default: a consumer with a
(static membership) stays in the group on close, and its partitions wait until its session times out.group.instance.id - After
, pod A committed nothing. Pod B started from offset 0 and processed orders 1 and 2 a second time.SIGKILL - Pod B also waited 7 seconds for its partition in this run. The coordinator still counted the killed consumer as a member until its session timed out, 10 seconds after its last heartbeat in this program; the rest of that time passed while pod B's JVM started. With the Kafka default
of 45 seconds, each killed consumer pauses its partitions for up to that long.session.timeout.ms
Spring for Apache Kafka commits and leaves the group on a normal context close, so a graceful SIGTERM
:spring.kafka.listener.immediate-stop by default, so the container finishes every record of the current poll. With the defaultfalse of 500 and slow processing, that can outlast the grace period. Set it tomax.poll.records to stop after the current record; the offsets of the records already processed are still committed.true- The phase timeout: on a context close, the listener registry stops its containers asynchronously, and Spring waits for them up to
. When it runs out, the shutdown moves on, but the consumer thread is not interrupted: the record in progress runs until the JVM exits or is killed, and its offset is not committed.spring.lifecycle.timeout-per-shutdown-phase does not apply here; it bounds only a direct call to a container'sContainerProperties.setShutdownTimeout .stop()
At-least-once delivery means a consumer must cope with duplicates anyway, because a pod can always be killed. A clean shutdown makes them rare rather than routine. The
When the grace period runs out
Two timers can end a shutdown early. Spring's phase timeout gives up on a phase and carries on closing; Kubernetes' grace period ends the process with SIGKILLSIGTERMSIGKILL
package demo.shutdown.timeout;
import jakarta.annotation.PreDestroy;
import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.io.IOException;
import java.net.Socket;
import java.nio.charset.StandardCharsets;
import java.util.Map;
import java.util.concurrent.BlockingQueue;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.LinkedBlockingQueue;
import java.util.concurrent.TimeUnit;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.boot.web.server.context.WebServerInitializedEvent;
import org.springframework.context.event.EventListener;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RestController;
@SpringBootApplication
@RestController
public class GracePeriodRunsOut {
// ---------- the application: runs in a child JVM ----------
@GetMapping("/report")
String report() throws InterruptedException {
System.out.println("handler: /report started, needs 5 s");
try {
Thread.sleep(5000);
} catch (InterruptedException e) {
System.out.println("handler: interrupted");
throw e;
}
System.out.println("handler: /report finished");
return "report";
}
@EventListener
void started(WebServerInitializedEvent e) {
System.out.println("PORT=" + e.getWebServer().getPort());
}
@PreDestroy
void destroy() {
System.out.println("bean: @PreDestroy");
}
static void runApp(String timeoutPerPhase) {
SpringApplication app = new SpringApplication(GracePeriodRunsOut.class);
app.setDefaultProperties(Map.of(
"server.port", "0",
"spring.lifecycle.timeout-per-shutdown-phase", timeoutPerPhase,
"spring.main.banner-mode", "off",
"logging.level.root", "WARN",
"logging.level.org.springframework.boot.tomcat.GracefulShutdown", "INFO",
"logging.level.org.springframework.context.support", "WARN",
"logging.pattern.console", "log: %logger{0}: %msg%n"));
app.run();
}
// ---------- the test driver: plays the kubelet and a client ----------
static String printUntil(BlockingQueue<String> lines, String marker)
throws InterruptedException {
while (true) {
String line = lines.take();
if (line.equals("EOF")) {
return line;
}
if (!line.startsWith("PORT=")) {
System.out.println(" [app] " + line);
}
if (line.contains(marker)) {
return line;
}
}
}
static void scenario(String title, String timeoutPerPhase, int graceSeconds)
throws Exception {
System.out.println("== " + title);
String java = ProcessHandle.current().info().command().orElseThrow();
Process app = new ProcessBuilder(java, "-cp", System.getProperty("java.class.path"),
GracePeriodRunsOut.class.getName(), timeoutPerPhase)
.redirectErrorStream(true).start();
BlockingQueue<String> lines = new LinkedBlockingQueue<>();
Thread.ofPlatform().daemon().start(() -> {
try (BufferedReader r = new BufferedReader(
new InputStreamReader(app.getInputStream()))) {
r.lines().forEach(lines::add);
} catch (Exception ignored) {
// the process is gone
}
lines.add("EOF");
});
String port = printUntil(lines, "PORT=").substring(5);
CompletableFuture<String> call = CompletableFuture.supplyAsync(() -> get(port));
printUntil(lines, "/report started");
System.out.println("kubelet: SIGTERM, grace period " + graceSeconds + " s");
app.toHandle().destroy();
printUntil(lines, "Commencing graceful shutdown");
if (!app.waitFor(graceSeconds, TimeUnit.SECONDS)) {
System.out.println("kubelet: grace period over, SIGKILL");
app.toHandle().destroyForcibly();
}
printUntil(lines, "EOF");
System.out.println("process exit code " + app.waitFor());
System.out.println("client: " + call.join());
}
// a raw HTTP request, so that the client shows exactly what came back
static String get(String port) {
try (Socket socket = new Socket("localhost", Integer.parseInt(port))) {
socket.getOutputStream().write(("GET /report HTTP/1.1\r\nHost: localhost\r\n"
+ "Connection: close\r\n\r\n").getBytes(StandardCharsets.US_ASCII));
String response = new String(socket.getInputStream().readAllBytes(),
StandardCharsets.US_ASCII);
return response.isEmpty() ? "connection closed, no response"
: response.lines().findFirst().orElseThrow();
} catch (IOException e) {
return e.getClass().getSimpleName() + ": " + e.getMessage();
}
}
public static void main(String[] args) throws Exception {
if (args.length > 0) {
runApp(args[0]);
return;
}
scenario("Spring's shutdown phase times out first", "2s", 30);
scenario("The kubelet's grace period runs out first", "30s", 2);
}
}
== Spring's shutdown phase times out first
[app] handler: /report started, needs 5 s
kubelet: SIGTERM, grace period 30 s
[app] log: GracefulShutdown: Commencing graceful shutdown. Waiting for active requests to complete
[app] handler: interrupted
[app] log: GracefulShutdown: Graceful shutdown aborted with one or more requests still active
[app] bean: @PreDestroy
process exit code 143
client: connection closed, no response
== The kubelet's grace period runs out first
[app] handler: /report started, needs 5 s
kubelet: SIGTERM, grace period 2 s
[app] log: GracefulShutdown: Commencing graceful shutdown. Waiting for active requests to complete
kubelet: grace period over, SIGKILL
process exit code 137
client: connection closed, no response
- In the first run, the drain timed out after 2 seconds. Spring moved on to stop Tomcat, which interrupted the request thread, and Tomcat logged
; the client's connection was closed with no response. The rest of the shutdown still ran:Graceful shutdown aborted with one or more requests still active printed and the exit code was 143.@PreDestroy - In the second run, the process was killed during the drain. No shutdown callback ran after that point, not even
, and the exit code was 137, which is 128 plus 9, the signal number of@PreDestroy . The client again got a closed connection.SIGKILL
From the client's side the two failures look the same. For the application, the first is much better: pools close, buffered logs and metrics flush, and @PreDestroyOOMKilled
Checklist
Size the grace period as a sum. With the values from the manifest above:
terminationGracePeriodSeconds >= preStop sleep
+ timeout-per-shutdown-phase for each phase that can be slow
+ executor await-termination-period, if set
+ a few seconds of margin
45 >= 10 + 20 + 10 + 5
This assumes one slow phase, the web server's drain. Kafka listeners add a second slow phase with the same 20-second cap, which gives 65 seconds. With the defaults, a 30-second phase timeout and a 30-second grace period, a drain that uses its full timeout is cut off by SIGKILLpreStopapplication.yaml
spring:
lifecycle:
timeout-per-shutdown-phase: 20s
task:
execution:
shutdown:
await-termination: true
await-termination-period: 10s
# server.shutdown is graceful by default
| Concern | Setting | Starting point |
|---|---|---|
| Signal reaches the JVM | Exec-form ENTRYPOINT | See |
| Endpoint removal race | lifecycle.preStop.sleep.seconds | 5 to 10 seconds; longer behind external load balancers |
| In-flight HTTP requests | server.shutdown=graceful | Keep the default |
| Time for each shutdown phase | spring.lifecycle.timeout-per-shutdown-phase | A little above your slowest normal request, such as 20s |
@Async@Scheduled | spring.task.*.shutdown.await-terminationspring.task.*.shutdown.await-termination-period | true |
| Kafka consumers | Graceful close (default); spring.kafka.listener.immediate-stop | immediate-stop=true |
| Hard deadline | terminationGracePeriodSeconds | The sum above; the default of 30 is often too short with a preStop |
| Node drains | PodDisruptionBudget | maxUnavailable: 1 |
| Capacity during rollouts | maxUnavailable: 0maxSurge: 1 | New pod ready before an old one is deleted |
| Verification | Exit code and log | 143: stopped on SIGTERMGraceful shutdown abortedSIGKILL |
Related pages: