Autoscaling Spring Boot on Kubernetes

The HorizontalPodAutoscaler adds and removes pods by comparing a metric with a target, and most examples use CPU. A JVM makes that metric unreliable: it burns CPU while starting and while the JIT compiler warms up, exactly when the autoscaler is deciding. This page measures that effect on a real Spring Boot service, runs the HPA's own formula against it, and shows the settings and the metrics that scale a Java service steadily.

Share

Contents

How the HorizontalPodAutoscaler decides

A HorizontalPodAutoscaler (HPA) is a control loop in the kube-controller-manager. Every 15 seconds by default, it reads metrics for a Deployment's pods, compares them with a target and writes a new replica count into the Deployment. The basic manifest and the Metrics Server it needs are covered in Deploying Spring Boot on Kubernetes; this page is about what happens when the pods are JVMs.

The core is one formula from the Kubernetes documentation:

desiredReplicas = ceil(currentReplicas × currentMetricValue / desiredMetricValue)

Two pods each using 140% of their CPU request, against a 70% target, give ceil(2 × 140 / 70) = 4. When the ratio is between 0.9 and 1.1 (the default 10% tolerance), nothing changes. The behavior section then smooths the result: a stabilization window acts on the lowest recommendation of a recent period when scaling up (default 0 seconds) and on the highest when scaling down (default 5 minutes), and rate limits cap the pods added or removed per period, within minReplicas and maxReplicas.

The table lists the five metric types; three APIs serve them:

HPA metric types in autoscaling/v2
TypeExampleServed byBest for
ResourcePod CPU as a percentage of its requestmetrics.k8s.io, from Metrics Server (samples every 15 s)CPU-bound services with a steady cost per request
ContainerResource (stable since 1.30)CPU of one container, ignoring sidecarsmetrics.k8s.io, from Metrics ServerPods with a service-mesh proxy or log shipper beside the application
PodsRequests per second per podcustom.metrics.k8s.io, from an adapter such as Prometheus AdapterWeb services, scaled on average load per pod
ObjectRequests per second on an Ingresscustom.metrics.k8s.ioLoad measured at one shared object, such as an Ingress
ExternalKafka consumer lag, queue depthexternal.metrics.k8s.io, from an adapter or KEDAWorkers that drain a queue or topic

The Kubernetes behaviour here was checked against the 1.33 and 1.34 documentation and source; the programs ran on Java 21 with Spring Boot 4.1. No cluster was available: the manifests were validated against the 1.33 and 1.34 schemas (and KEDA 2.21) but not applied, and a program implementing the HPA algorithm stands in for kubectl.

Why CPU misleads for a JVM

CPU utilization works well as a scaling signal when each request costs about the same CPU at all times. A JVM, and the way Kubernetes measures its CPU, break that assumption:

  • Startup: Spring Boot scans the classpath, loads thousands of classes and creates beans before it serves anything. A new pod uses nearly all the CPU it can get while this happens.
  • JIT warm-up: the JVM interprets bytecode, then compiles hot methods with the fast C1 and later the optimizing C2 compiler. Compilation takes CPU and interpreted code is slower, so a fresh pod spends several times more CPU per request than a warm one.
  • Garbage collection: GC threads use CPU that tracks allocation, not request count. A pod near its maximum heap collects more often, so its CPU rises without any rise in requests.
  • The request is the denominator: utilization is usage divided by the CPU request. With a 250m request (a quarter of a core), a JVM reports 350% or more while it starts, because startup keeps about a full core busy.
  • Limits and throttling: a CPU limit is a CPU-time quota per 100 ms period. A JVM with many GC and compiler threads can spend it early in the period and then wait, which shows up as latency, not CPU.

To see the first two effects, this program starts a small Spring Boot service and drives it from the same JVM at a fixed 300 requests per second for one minute; each request builds and serializes a 40-line quote. It measures process CPU in 5-second windows, in millicores, minus the load generator's threads (JIT and GC work caused by the client still counts, so the figure is somewhat high):

Java
package demo.warmup;

import java.lang.management.CompilationMXBean;
import java.lang.management.GarbageCollectorMXBean;
import java.lang.management.ManagementFactory;
import java.lang.management.ThreadInfo;
import java.lang.management.ThreadMXBean;
import java.math.BigDecimal;
import java.math.RoundingMode;
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicInteger;

import com.sun.management.OperatingSystemMXBean;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.boot.web.server.context.WebServerApplicationContext;
import org.springframework.context.ConfigurableApplicationContext;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;

@SpringBootApplication
@RestController
public class WarmupCpu {

    record Line(String sku, int qty, BigDecimal price, BigDecimal total) { }
    record Quote(String customer, List<Line> lines, BigDecimal net, BigDecimal tax) { }

    // A small but real piece of work per request: build 40 lines, price them, serialize to JSON
    @GetMapping("/quote")
    Quote quote(@RequestParam("customer") String customer) {
        List<Line> lines = new ArrayList<>();
        BigDecimal net = BigDecimal.ZERO;
        for (int i = 0; i < 40; i++) {
            BigDecimal price = BigDecimal.valueOf(1999 + i * 37L, 2);
            BigDecimal total = price.multiply(BigDecimal.valueOf(i % 5 + 1));
            lines.add(new Line("SKU-" + i, i % 5 + 1, price, total));
            net = net.add(total);
        }
        BigDecimal tax = net.multiply(new BigDecimal("0.2")).setScale(2, RoundingMode.HALF_UP);
        return new Quote(customer, lines, net, tax);
    }

    static final OperatingSystemMXBean OS =
            (OperatingSystemMXBean) ManagementFactory.getOperatingSystemMXBean();
    static final ThreadMXBean THREADS = ManagementFactory.getThreadMXBean();
    static final CompilationMXBean JIT = ManagementFactory.getCompilationMXBean();

    // CPU used by the load generator's own threads, so it can be taken out of the process total
    static long clientCpuNanos() {
        long sum = 0;
        for (ThreadInfo t : THREADS.getThreadInfo(THREADS.getAllThreadIds())) {
            if (t != null && (t.getThreadName().startsWith("load-")
                    || t.getThreadName().startsWith("HttpClient"))) {
                sum += Math.max(0, THREADS.getThreadCpuTime(t.getThreadId()));
            }
        }
        return sum;
    }

    static long gcMillis() {
        long sum = 0;
        for (GarbageCollectorMXBean gc : ManagementFactory.getGarbageCollectorMXBeans()) {
            sum += gc.getCollectionTime();
        }
        return sum;
    }

    public static void main(String[] args) throws Exception {
        ConfigurableApplicationContext ctx = SpringApplication.run(WarmupCpu.class,
                "--server.port=0", "--spring.main.banner-mode=off", "--logging.level.root=WARN");
        long startupCpuMs = OS.getProcessCpuTime() / 1_000_000;
        long startupWallMs = ManagementFactory.getRuntimeMXBean().getUptime();
        System.out.printf("CPUs available to the JVM: %d%n", Runtime.getRuntime()
                .availableProcessors());
        System.out.printf("Startup: %.1f CPU-seconds in %.1f s = %.2f cores on average%n",
                startupCpuMs / 1000.0, startupWallMs / 1000.0,
                (double) startupCpuMs / startupWallMs);

        int port = ((WebServerApplicationContext) ctx).getWebServer().getPort();
        URI uri = URI.create("http://localhost:" + port + "/quote?customer=acme");
        AtomicInteger n = new AtomicInteger();
        ExecutorService pool = Executors.newFixedThreadPool(8,
                r -> new Thread(r, "load-" + n.incrementAndGet()));
        HttpClient client = HttpClient.newBuilder().executor(pool).build();
        HttpRequest request = HttpRequest.newBuilder(uri).build();

        // Open-loop load: the same request rate every second, like steady production traffic
        int perSecond = 300;
        int windows = 12;
        int windowSeconds = 5;
        ScheduledExecutorService ticker = Executors.newSingleThreadScheduledExecutor();
        AtomicInteger ok = new AtomicInteger();
        ticker.scheduleAtFixedRate(() -> client
                .sendAsync(request, HttpResponse.BodyHandlers.ofString())
                .thenAccept(r -> {
                    if (r.statusCode() == 200) {
                        ok.incrementAndGet();
                    }
                }), 0, 1_000_000 / perSecond, TimeUnit.MICROSECONDS);

        System.out.printf("Load: %d requests per second for %d s%n",
                perSecond, windows * windowSeconds);
        System.out.println("seconds  service CPU (millicores)  JIT ms  GC ms  requests");
        List<Long> perWindow = new ArrayList<>();
        long cpu0 = OS.getProcessCpuTime() - clientCpuNanos();
        long jit0 = JIT.getTotalCompilationTime();
        long gc0 = gcMillis();
        int ok0 = 0;
        for (int w = 0; w < windows; w++) {
            Thread.sleep(windowSeconds * 1000L);
            long cpu1 = OS.getProcessCpuTime() - clientCpuNanos();
            long jit1 = JIT.getTotalCompilationTime();
            long gc1 = gcMillis();
            int ok1 = ok.get();
            // CPU-milliseconds per second of wall time = millicores, as Kubernetes counts CPU
            long milli = Math.round((cpu1 - cpu0) / 1e6 / windowSeconds / 10) * 10;
            perWindow.add(milli);
            System.out.printf("%3d-%-3d  %25d  %6d  %5d  %8d%n", w * windowSeconds,
                    (w + 1) * windowSeconds, milli, jit1 - jit0, gc1 - gc0, ok1 - ok0);
            cpu0 = cpu1; jit0 = jit1; gc0 = gc1; ok0 = ok1;
        }
        ticker.shutdownNow();
        pool.shutdown();
        pool.awaitTermination(5, TimeUnit.SECONDS);

        long first = perWindow.get(0);
        double steady = perWindow.subList(windows - 3, windows).stream()
                .mapToLong(Long::longValue).average().orElse(1);
        System.out.printf("First 5 s / last 15 s: %.1fx%n", first / steady);
        double[] q = new double[3];
        for (int i = 0; i < 3; i++) {
            q[i] = perWindow.subList(i * 3, i * 3 + 3).stream()
                    .mapToLong(Long::longValue).average().orElse(0) / steady;
        }
        System.out.printf("First three 15 s windows / last 15 s: %.1fx %.1fx %.1fx%n",
                q[0], q[1], q[2]);
        System.out.printf("Startup average / steady load: %.1fx%n",
                (double) startupCpuMs / startupWallMs * 1000 / steady);
        ctx.close();
    }
}
Output
CPUs available to the JVM: 2
Startup: 5.3 CPU-seconds in 2.9 s = 1.86 cores on average
Load: 300 requests per second for 60 s
seconds  service CPU (millicores)  JIT ms  GC ms  requests
  0-5                         1240    5717     83      1502
  5-10                         740    2885     56      1501
 10-15                         560    2177     39      1502
 15-20                         500    1740     28      1502
 20-25                         400    1209     28      1501
 25-30                         430    1349     23      1500
 30-35                         260     443     25      1501
 35-40                         210     197     28      1501
 40-45                         190     163     11      1501
 45-50                         230     381     24      1502
 50-55                         350    1068     30      1500
 55-60                         360     913     24      1502
First 5 s / last 15 s: 4.0x
First three 15 s windows / last 15 s: 2.7x 1.4x 0.7x
Startup average / steady load: 5.9x

Figures from this shared 2-core machine vary between runs, but the shape stayed the same:

  • Startup used 0.9 to 1.9 cores on average, depending on what else the machine was doing: 180% to 380% utilization on a 500m request, and 5.9 times a warm pod's CPU in this run (Startup average line).
  • At a fixed 300 requests per second, the first 5 seconds cost 3.5 to 6 times the CPU of the last 15 seconds across runs (4.0x here). The same work on the same pod looks like a heavily loaded service, then like an idle one.
  • JIT ms is compile time added across compiler threads, so it can exceed the 5,000 ms window. It falls in step with the CPU, which shows where the extra CPU went.
  • CPU kept falling, with some noise, for most of the minute: C2 compiles a method only after thousands of invocations.

A CPU limit makes startup take longer. This program starts the same kind of service three times pinned with taskset to CPU number 0 (CPUs 0) and three times to CPUs 0 and 1. The JVM then sizes itself as under a CPU limit of 1, though a real limit throttles in bursts instead of running one thread at a time:

Java
package demo.startup;

import java.lang.management.GarbageCollectorMXBean;
import java.lang.management.ManagementFactory;
import java.util.ArrayList;
import java.util.List;

import com.sun.management.OperatingSystemMXBean;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.context.ConfigurableApplicationContext;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RestController;

@SpringBootApplication
@RestController
public class StartupCpus {

    @GetMapping("/hello")
    String hello() {
        return "hello";
    }

    public static void main(String[] args) throws Exception {
        if (args.length > 0) {
            child();
            return;
        }
        String java = ProcessHandle.current().info().command().orElseThrow();
        String cp = System.getProperty("java.class.path");
        for (String cpus : List.of("0", "0,1")) {
            long best = Long.MAX_VALUE;
            String detail = "";
            for (int run = 0; run < 3; run++) {
                // taskset pins the JVM to the given CPUs, the way a CPU limit caps a container
                List<String> cmd = new ArrayList<>(List.of("taskset", "-c", cpus, java,
                        "-cp", cp, StartupCpus.class.getName(), "child"));
                Process p = new ProcessBuilder(cmd).redirectErrorStream(true).start();
                String out = new String(p.getInputStream().readAllBytes()).trim();
                p.waitFor();
                String[] f = out.split(" ");
                long ready = Long.parseLong(f[0]);
                if (ready < best) {
                    best = ready;
                    detail = String.format("%.1f CPU-seconds, GC %s", Long.parseLong(f[1])
                            / 1000.0, f[2]);
                }
            }
            System.out.printf("CPUs %-4s  ready after %.1f s (best of 3), %s%n",
                    cpus, best / 1000.0, detail);
        }
    }

    // Starts the application, prints time to ready, CPU used and the collector, then exits
    static void child() {
        ConfigurableApplicationContext ctx = SpringApplication.run(StartupCpus.class,
                "--server.port=0", "--spring.main.banner-mode=off", "--logging.level.root=WARN");
        long ready = ManagementFactory.getRuntimeMXBean().getUptime();
        OperatingSystemMXBean os =
                (OperatingSystemMXBean) ManagementFactory.getOperatingSystemMXBean();
        long cpu = os.getProcessCpuTime() / 1_000_000;
        List<GarbageCollectorMXBean> gcs = ManagementFactory.getGarbageCollectorMXBeans();
        String gc = gcs.get(0).getName().startsWith("G1") ? "G1" : "Serial";
        System.out.println(ready + " " + cpu + " " + gc);
        ctx.close();
    }
}
Output
CPUs 0     ready after 5.3 s (best of 3), 5.3 CPU-seconds, GC Serial
CPUs 0,1   ready after 3.1 s (best of 3), 5.6 CPU-seconds, GC G1

Both starts needed about the same CPU-seconds, so with half the CPUs the wall time was 1.7 times as long in this run, and with one CPU the JVM chose the Serial collector instead of G1 (see Docker for Java). A tight CPU limit makes the startup spike longer rather than lower, and the startup probe must allow for it.

What the HPA does with starting pods

The Kubernetes documentation names Java warm-up as the example of this problem. When the HPA scales on CPU, it sets some pods aside before averaging:

  • Starting pods that are not ready: a pod whose readiness probe has not yet passed is first left out of the average. If the ready pods alone call for a scale-up, the HPA recomputes with each unready pod counted at 0% of its request, which makes the scale-up smaller rather than larger.
  • The initialization period: for 5 minutes after a pod starts (--horizontal-pod-autoscaler-cpu-initialization-period), a CPU sample counts only if the pod is ready and the whole sample window was collected after it became ready.
  • The readiness delay: after the initialization period, an unready pod is still set aside if its last readiness change came within 30 seconds of its start (--horizontal-pod-autoscaler-initial-readiness-delay), which the controller takes to mean it has never been ready.

Both durations are cluster-wide kube-controller-manager flags that managed clusters usually do not let you change. These rules protect against CPU used before the pod is ready. They do nothing about the JIT warm-up measured above, which happens after the readiness probe passes.

The simulator below (its source follows the results) implements the Kubernetes 1.33 controller's calculation: GetResourceReplicas with the initialization-period rule and the 10% tolerance, then the stabilization windows and the default scale-up limit (double or add 4 pods per 15 seconds). Each pod has a 1-CPU request, a warm pod uses 1 millicore per request per second, and a starting pod burns a full core. After readiness, a pod uses 3.2, 1.6 and 1.1 times the warm CPU in its first three 15-second windows, the ratios of one WarmupCpu run (the run above, with a noisier last 15 seconds, printed 2.7, 1.4 and 0.7). Traffic jumps from 1,000 to 2,400 requests per second, which needs 2,400 / 700 = 3.4, so 4 pods at a 70% target:

Output
== Ready after 30 s, JIT warm-up happens while serving traffic
  time   rps  pods ready  CPU seen  counted  recommended  new count
  1:00  1000     2     2       50%      50%            2          2
  1:15  2400     2     2      120%     120%            4          4
  1:30  2400     4     2      110%     120%            4          4
  1:45  2400     4     2      110%     120%            4          4
  2:00  2400     4     4      126%     126%            8          8
  2:15  2400     8     4       89%      78%            8          8
  2:30  2400     8     4       82%      63%            8          8
  2:45  2400     8     8       63%      63%            8          8
  3:00  2400     8     8       39%      39%            5          8
  3:15  2400     8     8       32%      32%            4          8
  3:30  2400     8     8       30%      30%            4          8
  3:45  2400     8     8       30%      30%            4          8
  7:45  2400     8     8       30%      30%            4          5
  8:00  2400     5     5       48%      48%            4          4
pods started: 6, peak replicas: 8, replicas at the end: 4

== Same, with scaleUp.stabilizationWindowSeconds: 60
  time   rps  pods ready  CPU seen  counted  recommended  new count
  1:00  1000     2     2       50%      50%            2          2
  1:15  2400     2     2      120%     120%            4          2
  1:30  2400     2     2      120%     120%            4          2
  1:45  2400     2     2      120%     120%            4          2
  2:00  2400     2     2      120%     120%            4          4
  2:15  2400     4     2      110%     120%            4          4
  2:30  2400     4     2      110%     120%            4          4
  2:45  2400     4     4      126%     126%            8          4
  3:00  2400     4     4       78%      78%            5          4
  3:15  2400     4     4       63%      63%            4          4
  3:30  2400     4     4       60%      60%            4          4
  3:45  2400     4     4       60%      60%            4          4
pods started: 2, peak replicas: 4, replicas at the end: 4

== Application warms itself up before reporting ready (ready after 75 s)
  time   rps  pods ready  CPU seen  counted  recommended  new count
  1:00  1000     2     2       50%      50%            2          2
  1:15  2400     2     2      120%     120%            4          4
  1:30  2400     4     2      110%     120%            4          4
  1:45  2400     4     2      110%     120%            4          4
  2:00  2400     4     2      110%     120%            4          4
  2:15  2400     4     2      110%     120%            4          4
  2:30  2400     4     2      110%     120%            4          4
  2:45  2400     4     4       60%      60%            4          4
  3:00  2400     4     4       60%      60%            4          4
  3:15  2400     4     4       60%      60%            4          4
  3:30  2400     4     4       60%      60%            4          4
  3:45  2400     4     4       60%      60%            4          4
pods started: 2, peak replicas: 4, replicas at the end: 4

== Swinging traffic, scaleDown.stabilizationWindowSeconds: 0
pods started: 10, peak replicas: 4, replicas at the end: 2
replicas at each minute: 4 4 2 2 4 4 2 2 4 4 2 2 4 4 2 2 4 4 2 2

== Swinging traffic, scaleDown.stabilizationWindowSeconds: 300 (default)
pods started: 2, peak replicas: 4, replicas at the end: 4
replicas at each minute: 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4

Each row is one HPA evaluation; after 3:45, only rows that change the count are printed. The rps column is incoming requests per second, pods the replica count, ready the pods that pass their readiness probe and get traffic. CPU seen averages utilization over all pods, counted over the pods the HPA kept. The recommended column is the formula's raw result, new count the count after the stabilization windows and the scale-up limit.

  • In the first scenario the HPA correctly asks for 4 pods at 1:15. While the two new pods start, it sets them aside: at 1:30 CPU seen is 110%, but the recommendation stays at 4.
  • At 2:00 the new pods have been ready for one full window and are counted, while their JIT compilers are still busy. Counted CPU reads 126%, and ceil(4 × 126 / 70) = 8, twice what the traffic needs. The scale-down window keeps the highest recommendation of the last 5 minutes, so the 8 pods run for five and a half minutes, more than four of them at 30%, until the count falls to 5 at 7:45 and to 4 at 8:00.
  • A 60-second scale-up stabilization window acts on the lowest recommendation of the last minute. At 2:45 the formula asks for 8 again, but the window still holds the recommendations of 4 made while the new pods were unready, so the count stays at 4; a window shorter than the warm-up would miss this. The price: the genuine scale-up waits 45 seconds longer.
  • In the third scenario the application warms itself up before it reports ready, and the simulation treats it as fully warm from then on. The HPA never counts a cold pod, and the count goes straight to 4. A real warm-up removes most of the cold cost but not all, so a scale-up window is still worth keeping.

The simulation is deliberately simple: pods start when created, traffic spreads evenly, and CPU is a fixed cost per request. Image pulls and scheduling make the spike land later, not smaller, and a rolling update brings in cold pods too, so a CPU-based HPA can scale up during every rollout. The source:

Java
import java.util.ArrayList;
import java.util.List;
import java.util.function.IntUnaryOperator;

public class HpaSimulator {

    static final int STEP = 15;              // HPA sync period and metrics window, seconds
    static final double REQUEST_M = 1000;    // CPU request per pod, millicores
    static final double TARGET = 0.70;       // averageUtilization: 70
    static final double TOLERANCE = 0.10;    // default cluster-wide tolerance
    static final int CPU_INIT_PERIOD = 300;  // cpu-initialization-period flag, 5 minutes
    static final int MIN = 2;
    static final int MAX = 20;
    // CPU per request after the pod becomes ready, relative to a warm JVM, per 15 s window.
    // From a WarmupCpu run, 15 s averages against the last 15 s: 3.2x, 1.6x, 1.1x, then warm.
    static final double[] COLD = {3.2, 1.6, 1.1};

    record Pod(int start, int readyAt, boolean warmBeforeReady) { }
    record Rec(int time, int replicas) { }

    static double factor(Pod p, int t) {
        if (p.warmBeforeReady()) {
            return 1.0;
        }
        int w = (t - p.readyAt()) / STEP;
        return w < COLD.length ? COLD[w] : 1.0;
    }

    /** Runs one scenario and prints what the HPA saw and decided at each evaluation. */
    static void run(String title, IntUnaryOperator rps, int seconds, int readySeconds,
                    boolean warmBeforeReady, int upWindow, int downWindow, boolean verbose) {
        System.out.println("== " + title);
        List<Pod> pods = new ArrayList<>();
        pods.add(new Pod(-600, -570, warmBeforeReady));
        pods.add(new Pod(-600, -570, warmBeforeReady));
        List<Rec> history = new ArrayList<>();
        int started = 0;
        int peak = pods.size();
        StringBuilder timeline = new StringBuilder();
        if (verbose) {
            System.out.println("  time   rps  pods ready  CPU seen  counted"
                    + "  recommended  new count");
        }
        for (int t = 0; t < seconds; t += STEP) {
            final int now = t;
            List<Pod> ready = pods.stream().filter(p -> p.readyAt() <= now).toList();
            double perPod = (double) rps.applyAsInt(t) / ready.size();
            // CPU each pod used during [t, t + STEP): a pod that is still starting burns
            // a full core, which is 100% of its 1000m request
            double sumAll = 0;
            double sumCounted = 0;
            int counted = 0;
            boolean anyUnready = false;
            for (Pod p : pods) {
                double cpu = p.readyAt() <= t ? perPod * factor(p, t) : 1000;
                sumAll += cpu;
                int sampleEnd = t + STEP;
                // groupPods(): inside the initialization period a sample counts only if the
                // pod is ready and the whole window was collected after it became ready
                boolean unready = sampleEnd - p.start() < CPU_INIT_PERIOD
                        ? p.readyAt() > sampleEnd || sampleEnd < p.readyAt() + STEP
                        : p.readyAt() > sampleEnd;
                if (unready) {
                    anyUnready = true;
                } else {
                    sumCounted += cpu;
                    counted++;
                }
            }
            int current = pods.size();
            double ratio = sumCounted / (counted * REQUEST_M) / TARGET;
            int desired;
            if (!(anyUnready && ratio > 1) && Math.abs(ratio - 1) <= TOLERANCE) {
                desired = current;
            } else if (!(anyUnready && ratio > 1)) {
                desired = (int) Math.ceil(ratio * counted);
            } else {
                // scale-up with unready pods: assume they use 0% and recompute
                double newRatio = sumCounted / (current * REQUEST_M) / TARGET;
                desired = newRatio <= 1 + TOLERANCE ? current : (int) Math.ceil(newRatio * current);
            }
            history.add(new Rec(t + STEP, desired));
            // stabilization: lowest recommendation in the up window, highest in the down window
            int up = desired;
            int down = desired;
            for (Rec r : history) {
                if (r.time() > t + STEP - upWindow) {
                    up = Math.min(up, r.replicas());
                }
                if (r.time() > t + STEP - downWindow) {
                    down = Math.max(down, r.replicas());
                }
            }
            int next = Math.min(Math.max(current, up), down);
            next = Math.min(next, Math.max(current + 4, current * 2)); // default scale-up policy
            next = Math.max(MIN, Math.min(MAX, next)); // then minReplicas and maxReplicas
            // from 1:00 print every evaluation up to 3:45, then only those that change the count
            if (verbose && ((t + STEP >= 60 && t + STEP <= 225) || next != current)) {
                System.out.printf("%6s %5d  %4d %5d  %7.0f%%  %6.0f%%  %11d  %9d%n",
                        clock(t + STEP),
                        rps.applyAsInt(t), current, ready.size(),
                        sumAll / (current * REQUEST_M) * 100,
                        counted == 0 ? 0 : sumCounted / (counted * REQUEST_M) * 100, desired, next);
            }
            while (pods.size() < next) {
                pods.add(new Pod(t + STEP, t + STEP + readySeconds, warmBeforeReady));
                started++;
            }
            while (pods.size() > next) {
                pods.removeLast();
            }
            peak = Math.max(peak, pods.size());
            if ((t + STEP) % 60 == 0) {
                timeline.append(pods.size()).append(' ');
            }
        }
        System.out.printf("pods started: %d, peak replicas: %d, replicas at the end: %d%n",
                started, peak, pods.size());
        if (!verbose) {
            System.out.println("replicas at each minute: " + timeline.toString().trim());
        }
        System.out.println();
    }

    static String clock(int s) {
        return String.format("%d:%02d", s / 60, s % 60);
    }

    public static void main(String[] args) {
        // Traffic jumps from 1,000 to 2,400 requests/s at 1:00. A warm pod uses 1 millicore
        // per request/s, so 2,400 requests/s need 2,400 / 700 = 3.4, that is 4 pods at 70%.
        IntUnaryOperator step = t -> t < 60 ? 1000 : 2400;
        run("Ready after 30 s, JIT warm-up happens while serving traffic",
                step, 600, 30, false, 0, 300, true);
        run("Same, with scaleUp.stabilizationWindowSeconds: 60",
                step, 600, 30, false, 60, 300, true);
        run("Application warms itself up before reporting ready (ready after 75 s)",
                step, 600, 75, true, 0, 300, true);

        // Traffic that swings between 1,200 and 2,800 requests/s every 2 minutes, for 20 min
        IntUnaryOperator wave = t -> (t / 120) % 2 == 0 ? 2800 : 1200;
        run("Swinging traffic, scaleDown.stabilizationWindowSeconds: 0",
                wave, 1200, 30, true, 0, 0, false);
        run("Swinging traffic, scaleDown.stabilizationWindowSeconds: 300 (default)",
                wave, 1200, 30, true, 0, 300, false);
    }
}

Warming up before the pod reports ready

After the context has started, Spring Boot sets liveness to CORRECT, runs every ApplicationRunner and CommandLineRunner, and only then sets readiness to ACCEPTING_TRAFFIC. The web server is already listening, so a runner that calls the application's own endpoints warms the JIT while the readiness probe keeps the pod out of the Service:

Java
package demo.readywarm;

import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.util.ArrayList;
import java.util.List;

import org.springframework.boot.ApplicationArguments;
import org.springframework.boot.ApplicationRunner;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.boot.web.server.context.WebServerApplicationContext;
import org.springframework.context.ConfigurableApplicationContext;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;

@SpringBootApplication
@RestController
public class WarmupBeforeReady implements ApplicationRunner {

    record Item(String sku, int qty, long cents) { }

    private final WebServerApplicationContext web;

    WarmupBeforeReady(WebServerApplicationContext web) {
        this.web = web;
    }

    @GetMapping("/quote")
    List<Item> quote(@RequestParam("customer") String customer) {
        List<Item> items = new ArrayList<>();
        for (int i = 0; i < 40; i++) {
            items.add(new Item(customer + "-" + i, i % 5 + 1, 1999L + i * 37L));
        }
        return items;
    }

    // Runners run after the web server has started but before Spring Boot publishes
    // ReadinessState.ACCEPTING_TRAFFIC, so the readiness probe fails while this runs
    @Override
    public void run(ApplicationArguments args) throws Exception {
        String base = "http://localhost:" + web.getWebServer().getPort();
        HttpClient client = HttpClient.newHttpClient();
        System.out.println("During warm-up, liveness:  " + get(client, base
                + "/actuator/health/liveness"));
        System.out.println("During warm-up, readiness: " + get(client, base
                + "/actuator/health/readiness"));
        HttpRequest quote = HttpRequest.newBuilder(URI.create(base + "/quote?customer=w"))
                .build();
        List<Long> batches = new ArrayList<>();
        for (int batch = 0; batch < 6; batch++) {
            long start = System.nanoTime();
            for (int i = 0; i < 500; i++) {
                client.send(quote, HttpResponse.BodyHandlers.discarding());
            }
            batches.add(System.nanoTime() - start);
        }
        long total = batches.stream().mapToLong(Long::longValue).sum();
        System.out.printf("Warm-up: 6 batches of 500 requests in %.1f s, last batch %.1fx "
                + "faster than the first%n", total / 1e9, (double) batches.get(0) / batches.get(5));
    }

    static String get(HttpClient client, String url) throws Exception {
        HttpResponse<String> r = client.send(HttpRequest.newBuilder(URI.create(url)).build(),
                HttpResponse.BodyHandlers.ofString());
        return r.statusCode() + " " + r.body();
    }

    public static void main(String[] args) throws Exception {
        ConfigurableApplicationContext ctx = SpringApplication.run(WarmupBeforeReady.class,
                "--server.port=0", "--spring.main.banner-mode=off", "--logging.level.root=WARN");
        String base = "http://localhost:"
                + ((WebServerApplicationContext) ctx).getWebServer().getPort();
        System.out.println("After startup, readiness: " + get(HttpClient.newHttpClient(),
                base + "/actuator/health/readiness"));
        ctx.close();
    }
}
Output
During warm-up, liveness:  200 {"status":"UP"}
During warm-up, readiness: 503 {"status":"OUT_OF_SERVICE"}
Warm-up: 6 batches of 500 requests in 4.6 s, last batch 3.1x faster than the first
After startup, readiness: 200 {"status":"UP"}
  • During the runner, liveness answers UP (the CORRECT state), so Kubernetes does not restart the pod; readiness answers 503 OUT_OF_SERVICE, so the pod gets no traffic and the HPA sets its CPU aside.
  • The last batch of 500 requests ran 3.1 times faster than the first in this run, and between 2.5 and 3.4 times across runs. After run returns, readiness turns UP and the pod joins the Service warm.

In Spring Boot 4 the liveness and readiness groups are on by default (management.endpoint.health.probes.enabled defaults to true), inside a cluster or not. Warm up only with requests that have no side effects, and keep the warm-up short (4.6 seconds here): it adds directly to the time a scale-up takes.

The Deployment needs probes that allow for the extra time, and a CPU request that matches the warm load:

YAML
# deployment.yaml: settings that matter for autoscaling
apiVersion: apps/v1
kind: Deployment
metadata:
  name: payment-service
  namespace: payments
spec:
  # no replicas field: the HPA owns the count
  selector:
    matchLabels:
      app: payment-service
  template:
    metadata:
      labels:
        app: payment-service
    spec:
      terminationGracePeriodSeconds: 45
      containers:
      - name: app
        image: registry.example.com/payment-service:4f2a91c
        ports:
        - containerPort: 8080
        resources:
          requests:
            cpu: "1"            # the HPA's 100%: size it for warm, steady load
            memory: 2Gi
          limits:
            memory: 2Gi         # no CPU limit, so startup is not throttled
        env:
        - name: JAVA_TOOL_OPTIONS
          value: "-XX:ActiveProcessorCount=2 -XX:+UseG1GC -XX:MaxRAMPercentage=75"
        startupProbe:           # allows 60 s for startup before liveness takes over
          httpGet:
            path: /actuator/health/liveness
            port: 8080
          periodSeconds: 5
          failureThreshold: 12
        readinessProbe:         # fails until the warm-up runner has finished
          httpGet:
            path: /actuator/health/readiness
            port: 8080
          periodSeconds: 5
        livenessProbe:
          httpGet:
            path: /actuator/health/liveness
            port: 8080
          periodSeconds: 10
        lifecycle:
          preStop:
            sleep:
              seconds: 10       # keep serving while endpoints are removed
  • No replicas field: with an HPA in charge, every kubectl apply of a manifest that sets replicas resets the count. On an existing Deployment, deleting the field and running a client-side kubectl apply drops the count to 1 for a moment; remove it first with kubectl apply edit-last-applied, or use server-side apply.
  • CPU request: this is what 100% utilization means to the HPA. Set it to what a warm pod uses at the load you want each pod to carry, not to the startup peak.
  • No CPU limit: the pod may borrow idle CPU while it starts and compiles. The JVM then sees every CPU on the node, so -XX:ActiveProcessorCount=2 caps the GC, JIT and common-pool threads it sizes from the CPU count (2 because the pod may exceed its request). Check that no namespace LimitRange adds a default CPU limit.
  • Memory and collector: the JVM also picks Serial when it sees less than 1,792 MB, so a 1Gi limit gives Serial even with two processors; -XX:+UseG1GC makes the choice explicit. The 2Gi limit leaves a quarter outside the 75% heap for metaspace, thread stacks, code cache and direct buffers.
  • Probes: the startup probe allows 60 seconds; readiness waits for the warm-up runner. Kubernetes probes for Spring Boot covers the timing.
  • preStop and grace period: see Scale-down below.

Tuning a CPU-based HPA for a JVM

If CPU is the signal you have, these settings address the overshoot shown in the simulation:

YAML
# hpa-cpu.yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: payment-service
  namespace: payments
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: payment-service
  minReplicas: 2
  maxReplicas: 20
  metrics:
  - type: ContainerResource
    containerResource:
      name: cpu
      container: app              # the JVM only, not sidecars
      target:
        type: Utilization
        averageUtilization: 70
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 60   # act on the lowest recommendation of the last minute
      policies:
      - type: Percent
        value: 100                     # at most double per minute; replaces the default add-4
        periodSeconds: 60
    scaleDown:
      stabilizationWindowSeconds: 300  # act on the highest recommendation of the last 5 min
      policies:
      - type: Pods
        value: 2                       # remove at most 2 pods per minute
        periodSeconds: 60
  • Scale-up stabilization window: the default is 0, so a single high reading scales up at once. Make the window a little longer than your service's warm-up after readiness: 60 seconds fit the 45 seconds simulated above, but a service with many endpoints can take minutes. Measure yours by starting one pod under steady load and watching its CPU until it stops falling. A longer window reacts later to a real surge.
  • Scale-up policy: the default allows doubling, or adding 4 pods, every 15 seconds. Listing policies replaces both defaults, so this one adds only 2 pods per minute at 2 replicas; add a Pods policy if small Deployments must grow faster (the policy that allows the most change wins). A slower rate gives new pods time to warm up.
  • Target utilization: 60% to 75% leaves headroom while new pods start and warm up; the longer the startup, the lower the target.
  • ContainerResource: with a sidecar in the pod, Resource divides the total usage of all containers by their total requests. Naming the application container keeps a busy proxy from hiding an idle JVM, or the reverse.

Do not scale a JVM on memory. The heap grows to what the collector finds convenient and is given back slowly, if at all, so after a burst the HPA may not scale down for a long time.

Scaling on request rate, backlog or latency

A metric that counts the work itself does not overshoot during warm-up: a cold pod and a warm pod receive the same requests per second, though the cold one uses several times the CPU. Spring Boot Actuator records such metrics through Micrometer. This program serves two endpoints, registers a gauge for a work backlog, sends 25 GET and 7 POST requests, and prints the matching lines from /actuator/prometheus:

Java
package demo.hpametrics;

import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.util.concurrent.BlockingQueue;
import java.util.concurrent.LinkedBlockingQueue;

import io.micrometer.core.instrument.Gauge;
import io.micrometer.core.instrument.MeterRegistry;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.boot.web.server.context.WebServerApplicationContext;
import org.springframework.context.ConfigurableApplicationContext;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RestController;

@SpringBootApplication
@RestController
public class HpaMetrics {

    // Work accepted but not yet processed: the backlog a worker deployment would scale on
    private final BlockingQueue<String> backlog = new LinkedBlockingQueue<>();

    HpaMetrics(MeterRegistry registry) {
        Gauge.builder("orders.backlog", backlog, BlockingQueue::size)
                .description("Orders waiting to be processed")
                .register(registry);
    }

    @PostMapping("/orders")
    String accept() {
        backlog.add("order");
        return "accepted";
    }

    @GetMapping("/orders/{id}")
    String find(@PathVariable("id") String id) {
        return "order " + id;
    }

    public static void main(String[] args) throws Exception {
        ConfigurableApplicationContext ctx = SpringApplication.run(HpaMetrics.class,
                "--server.port=0", "--spring.main.banner-mode=off", "--logging.level.root=WARN",
                "--management.endpoints.web.exposure.include=prometheus",
                "--management.metrics.distribution.slo.http.server.requests=250ms,1s",
                "--management.metrics.tags.application=orders");
        int port = ((WebServerApplicationContext) ctx).getWebServer().getPort();
        String base = "http://localhost:" + port;
        HttpClient client = HttpClient.newHttpClient();
        // one scrape first, so the first measured request is not the one that starts MVC
        send(client, HttpRequest.newBuilder(URI.create(base + "/actuator/prometheus")).build());

        for (int i = 0; i < 25; i++) {
            send(client, HttpRequest.newBuilder(URI.create(base + "/orders/" + i)).build());
        }
        for (int i = 0; i < 7; i++) {
            send(client, HttpRequest.newBuilder(URI.create(base + "/orders"))
                    .POST(HttpRequest.BodyPublishers.noBody()).build());
        }
        String scrape = client.send(
                HttpRequest.newBuilder(URI.create(base + "/actuator/prometheus")).build(),
                HttpResponse.BodyHandlers.ofString()).body();

        System.out.println("Sent 25 GET /orders/{id} and 7 POST /orders. Scrape, filtered:");
        for (String line : scrape.split("\n")) {
            boolean wanted = line.startsWith("orders_backlog")
                    || line.startsWith("# TYPE orders_backlog")
                    || line.startsWith("# TYPE http_server_requests_seconds ")
                    || line.startsWith("http_server_requests_active_seconds_gcount{")
                    || (line.startsWith("http_server_requests_seconds_")
                        && line.contains("uri=\"/orders"));
            // the _sum and _max series hold measured times, which differ on every run
            if (wanted && !line.contains("_sum{") && !line.contains("_max{")) {
                System.out.println(line);
            }
        }
        ctx.close();
    }

    static void send(HttpClient client, HttpRequest request) throws Exception {
        client.send(request, HttpResponse.BodyHandlers.discarding());
    }
}
Output
Sent 25 GET /orders/{id} and 7 POST /orders. Scrape, filtered:
http_server_requests_active_seconds_gcount{application="orders",exception="none",method="GET",outcome="SUCCESS",status="200",uri="UNKNOWN"} 1
http_server_requests_active_seconds_gcount{application="orders",exception="none",method="POST",outcome="SUCCESS",status="200",uri="UNKNOWN"} 0
# TYPE http_server_requests_seconds histogram
http_server_requests_seconds_bucket{application="orders",error="none",exception="none",method="GET",outcome="SUCCESS",status="200",uri="/orders/{id}",le="0.25"} 25
http_server_requests_seconds_bucket{application="orders",error="none",exception="none",method="GET",outcome="SUCCESS",status="200",uri="/orders/{id}",le="1.0"} 25
http_server_requests_seconds_bucket{application="orders",error="none",exception="none",method="GET",outcome="SUCCESS",status="200",uri="/orders/{id}",le="+Inf"} 25
http_server_requests_seconds_count{application="orders",error="none",exception="none",method="GET",outcome="SUCCESS",status="200",uri="/orders/{id}"} 25
http_server_requests_seconds_bucket{application="orders",error="none",exception="none",method="POST",outcome="SUCCESS",status="200",uri="/orders",le="0.25"} 7
http_server_requests_seconds_bucket{application="orders",error="none",exception="none",method="POST",outcome="SUCCESS",status="200",uri="/orders",le="1.0"} 7
http_server_requests_seconds_bucket{application="orders",error="none",exception="none",method="POST",outcome="SUCCESS",status="200",uri="/orders",le="+Inf"} 7
http_server_requests_seconds_count{application="orders",error="none",exception="none",method="POST",outcome="SUCCESS",status="200",uri="/orders"} 7
# TYPE orders_backlog gauge
orders_backlog{application="orders"} 7.0
  • http_server_requests_seconds_count is a counter per endpoint, method and status: 25 for GET /orders/{id} and 7 for POST /orders. Prometheus's rate() turns it into requests per second, usually the best scaling metric for a web service. The uri label is the route template, so the 25 ids share one series.
  • The le="0.25" and le="1.0" buckets come from the slo property: they count requests that took at most 250 ms and at most 1 s, enough to compute the share of slow requests.
  • http_server_requests_active_seconds_gcount is the number of requests in flight, from a long task timer (hence _gcount). Its uri is UNKNOWN because it is tagged before Spring has matched a route; the one active GET is the scrape itself. It suits slow, blocking endpoints, where CPU stays low while threads wait.
  • orders_backlog is the custom gauge: 7 orders accepted and not yet processed. Workers scale on this kind of number. In production, keep the backlog in a broker or database: a gauge published by the workers disappears when they scale to zero, and nothing scales them back up.
  • The application="orders" tag comes from management.metrics.tags.application and keeps services apart in a shared Prometheus.

The program needs spring-boot-starter-actuator and micrometer-registry-prometheus; Spring Boot Actuator covers the endpoints.

Latency makes a poor scaling target: it rises for reasons more pods do not fix, such as a slow database, and more pods add connections to the bottleneck. Scale on request rate and alert on latency. A slow dependency raises in-flight requests too, so scale on them only where the slowness is the pod's own work.

The HPA cannot query Prometheus itself. It reads the custom.metrics.k8s.io API, which Prometheus Adapter implements by running PromQL queries. A rule names the series, maps its labels to Kubernetes objects and gives the query that turns it into a value. This rule, in the format of the adapter's walkthrough (with the Helm chart, under rules.custom), publishes requests per second per pod. It assumes Prometheus adds namespace and pod labels when it scrapes, as the usual Kubernetes scrape configurations do. Not run here:

YAML
# prometheus-adapter rules
rules:
- seriesQuery: 'http_server_requests_seconds_count{namespace!="",pod!=""}'
  resources:
    overrides:
      namespace: {resource: "namespace"}
      pod: {resource: "pod"}
  name:
    matches: "^http_server_requests_seconds_count$"
    as: "http_server_requests_per_second"
  metricsQuery: >-
    sum(rate(<<.Series>>{<<.LabelMatchers>>,uri!~"/actuator.*"}[2m]))
    by (<<.GroupBy>>)

The adapter fills in the series name, the namespace and pod selectors, and the pod label. The uri!~"/actuator.*" matcher leaves out probe and scrape requests, which arrive at the same rate whatever the load. The HPA then targets an average value per pod:

YAML
# hpa-requests.yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: payment-service
  namespace: payments
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: payment-service
  minReplicas: 2
  maxReplicas: 20
  metrics:
  - type: Pods
    pods:
      metric:
        name: http_server_requests_per_second
      target:
        type: AverageValue
        averageValue: "400"     # 70-80% of what one warm pod handles at acceptable latency
  - type: ContainerResource     # safety net; the higher recommendation wins
    containerResource:
      name: cpu
      container: app
      target:
        type: Utilization
        averageUtilization: 150 # well above warm use at 400 requests/s
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300

With several metrics, the HPA computes a replica count for each and uses the largest. The CPU target must sit clearly above what a warm pod uses at 400 requests per second, or CPU decides every scale-up, cold pods included. With the request sized for that load, warm use is near 100%, hence 150 (allowed without a CPU limit). The safety net then acts only if requests become much more expensive, for example after a slower release. Find the per-pod rate by load-testing one warm pod until latency or errors exceed what you accept, and take 70% to 80% of that rate.

KEDA: queue lag and Prometheus queries without an adapter

KEDA is a CNCF project that serves external.metrics.k8s.io itself. You describe scaling in a ScaledObject, a custom resource, and KEDA creates the HPA from it. Each trigger uses a scaler that reads one source: a Prometheus query, Kafka consumer lag, RabbitMQ queue length and many more. KEDA also scales to and from zero, which the plain HPA needs an alpha feature gate for in Kubernetes 1.33 and 1.34. The examples follow the KEDA 2.21 documentation and were validated, not run:

YAML
# scaledobject-requests.yaml: the same request-rate scaling, with KEDA
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: payment-service
  namespace: payments
spec:
  scaleTargetRef:
    name: payment-service
  minReplicaCount: 2
  maxReplicaCount: 20
  advanced:
    horizontalPodAutoscalerConfig:
      behavior:                 # passed through to the HPA KEDA creates
        scaleDown:
          stabilizationWindowSeconds: 300
  triggers:
  - type: prometheus
    metadata:
      serverAddress: http://prometheus.monitoring.svc:9090
      query: >-
        sum(rate(http_server_requests_seconds_count{namespace="payments",
        application="payment-service",uri!~"/actuator.*"}[2m]))
      threshold: "400"          # requests/s per pod

Use this ScaledObject in place of hpa-requests.yaml: KEDA creates its own HPA, and when two HPAs select the same pods, the controller marks both with AmbiguousSelector and neither scales. The query's application label must match management.metrics.tags.application. KEDA passes the query's total to the HPA as an AverageValue target by default, so the HPA divides it by the pod count and compares the result with 400.

For a Kafka consumer, such as one built with @KafkaListener, the backlog is the consumer group's lag:

YAML
# scaledobject-kafka.yaml: scale consumers on lag
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: order-consumer
  namespace: payments
spec:
  scaleTargetRef:
    name: order-consumer
  pollingInterval: 30
  minReplicaCount: 1
  maxReplicaCount: 12           # no more than the topic's partitions
  advanced:
    horizontalPodAutoscalerConfig:
      behavior:
        scaleUp:
          stabilizationWindowSeconds: 60   # new consumers start slowly
  triggers:
  - type: kafka
    metadata:
      bootstrapServers: kafka.kafka.svc:9092
      consumerGroup: order-consumer
      topic: orders
      lagThreshold: "500"       # target lag per replica
  • Partitions cap the useful replicas: each partition is read by only one consumer in a group, so extra pods sit idle. KEDA's Kafka scaler stops at the partition count unless allowIdleConsumers is set.
  • Every scale event is a rebalance: the group reassigns partitions and consumption pauses meanwhile. A scale-down stabilization window keeps a fluctuating lag from causing a rebalance every minute.
  • New consumers start slowly: a new consumer pod processes slowly for its first seconds, so lag may keep rising briefly after a scale-up; the scale-up window in the example covers that.
  • KEDA's cooldownPeriod (300 seconds by default) only governs the last step to zero replicas, with minReplicaCount: 0. Between 1 and N replicas, the HPA's own behavior settings apply.
  • Scale from zero is a cold start: the first message waits for the next poll (pollingInterval, 30 seconds by default), scheduling, JVM startup and warm-up. An activation threshold (activationLagThreshold for Kafka) sets how much lag wakes the workload.

Scale-down: stabilization and graceful shutdown

The last two scenarios of the simulation show why the default 5-minute scale-down window matters, with traffic swinging between 1,200 and 2,800 requests per second every two minutes:

  • With scaleDown.stabilizationWindowSeconds: 0, the count followed every swing, 4, 2, 4, 2, and started 10 pods in 20 minutes, each paying for a JVM startup and warm-up. These simulated pods warm up before they are ready; without that, each would also report high CPU while warming, which can trigger the next scale-up.
  • With the default 300 seconds, the HPA kept the highest recommendation of the last 5 minutes and stayed at 4 pods throughout, starting only the 2 pods the first peak needed.

Churn costs Java services more. Among otherwise equal pods, the ReplicaSet controller removes the ones that became ready most recently first (on a logarithmic time scale), so pods that have just finished their warm-up usually go first. A removed pod may also be in the middle of requests or messages. Each deleted pod goes through the same sequence as in a rolling update:

  1. The pod is removed from the Service's endpoints. Ingress controllers and other nodes learn this a little later, so some requests still arrive.
  2. At the same time, the preStop hook runs; the 10-second sleep above keeps the pod serving meanwhile. The sleep action is stable from Kubernetes 1.34, so the image needs no shell.
  3. The kubelet sends SIGTERM. Spring Boot's graceful shutdown (server.shutdown=graceful, the default) stops accepting new requests and waits up to spring.lifecycle.timeout-per-shutdown-phase (30 seconds by default) for the ones in progress.
  4. If the pod is still running when terminationGracePeriodSeconds runs out, the kubelet sends SIGKILL.

The grace period must cover the preStop sleep plus the shutdown: 10 + 30 fits the 45 seconds above for a web service. The timeout applies per lifecycle phase, so a service that also stops listener containers needs more. The page Graceful shutdown on Kubernetes runs this sequence against a real JVM.

Common pitfalls

  • A maximum the cluster cannot place: if the nodes are full, new pods stay Pending. The HPA counts them as replicas, the running pods keep the whole load, and nothing improves until the cluster autoscaler adds nodes, minutes before the JVM even starts. Plan maxReplicas with node capacity.
  • HPA and VPA on the same metric: a Vertical Pod Autoscaler that changes the CPU request moves the HPA's 100% mark. Use different signals, or run VPA with updateMode: "Off", which only recommends. An in-place resize (beta in Kubernetes 1.33 and 1.34) does not change a running JVM's maximum heap or the thread counts it chose at startup.
  • Metrics with no data: when Prometheus or the adapter is down, the HPA cannot compute that metric. The CPU safety net in hpa-requests.yaml can still scale up, but the HPA skips any scale-down while a metric is missing. KEDA's fallback instead sets a fixed replica count after a number of failed reads.
  • Slow starts: every minute of startup and warm-up is a minute of extra pods. Spring Boot's CDS training run and the JDK's AOT cache (JEP 483) shorten startup; AOT method profiles (JEP 515) and CRaC checkpoints also shorten the JIT warm-up the HPA sees. Neither JEP is in Java 21.

Which signal to use

Scaling signals for Spring Boot services
WorkloadSignalHowWatch out for
CPU-bound web service, no PrometheusCPU utilizationContainerResource with a scale-up window longer than the warm-upJIT warm-up after readiness; size the request for warm load
Web service with PrometheusRequests per second per podPrometheus Adapter and a Pods metric, or a KEDA prometheus triggerExclude /actuator traffic; keep CPU as a safety net
Slow or blocking endpointsIn-flight requests per podThe _gcount gauge through the adapter or KEDA, summed without rate()A slow dependency raises it too
Kafka consumersConsumer group lagKEDA kafka triggerPartition count caps replicas; rebalance on every change
Queue or batch workersBacklog sizeThe broker's own metric through KEDA, or a gauge published outside the workersScale to zero needs a metric that exists while no worker runs, and an acceptable cold start
Memory-heavy serviceRequests or CPU, not memorySize the heap separatelyThe JVM returns heap slowly, so memory stays high after load drops

Whichever signal you choose, warm new pods before they are ready, keep a scale-down window of several minutes, and let a removed pod finish its work; with CPU or queue lag, add a scale-up window that covers the warm-up. See also Deploying Spring Boot on Kubernetes, Kubernetes probes for Spring Boot and Helm charts for Spring Boot.