Database migrations for Spring Boot on Kubernetes

Spring Boot can apply database migrations when the application starts: Flyway or Liquibase checks which scripts the database has already run and applies only the new ones, each versioned script once. For a single instance, that is all it takes. On Kubernetes, three things change: several replicas start at once and find the same new script; old pods keep serving while new ones arrive; and a rollback brings old code back to a new schema. This page compares where to run migrations (at startup, in an init container, in a Job or in a Helm hook) and runs three situations against a shared H2 database server: concurrent migrations and their locks, a column rename that breaks running pods, and the expand/contract sequence that makes the same change safely.

Share

Contents

Why migrations are different on Kubernetes

Flyway and Liquibase keep a history table in the database, compare it with the scripts shipped in the application, and run the missing ones. A Kubernetes Deployment changes three things:

  • Several replicas start at once: the first pods of a new release find the same pending scripts, and each script must run exactly once.
  • Old and new versions run side by side: a rolling update replaces pods a few at a time, so for minutes, old pods serve traffic against the schema the new code needs.
  • Rollbacks bring old code back: kubectl rollout undo or helm rollback restores the previous image, but neither undoes the migrations. The old code then runs against the new schema until the next release.

The tools solve the first problem with a database lock, which has traps of its own. The other two need migrations that the previous release can live with. First, decide where the migration runs:

Where a Spring Boot service can run its migrations
WhereHowAdvantagesWatch out for
Application startupSpring Boot runs Flyway or Liquibase when spring-boot-starter-flyway or spring-boot-starter-liquibase is on the class path (Flyway also needs its database module, such as flyway-database-postgresql)Nothing to set up; schema and code always ship togetherEvery pod competes for the lock; a long migration delays readiness and needs a startup probe; the app's database user needs rights to change the schema (DDL)
Init containerThe same image in a migrate-only mode, before the application containerProbes do not run until it finishes; separate logs and exit codeRuns again in every new pod (scale-up, replacement), though not when the application container restarts; a failure shows as Init:Error or Init:CrashLoopBackOff
Kubernetes JobA Job applied by the pipeline before the DeploymentOne run per release, normally; separate credentials and timeout; a failed Job stops the pipelineThe pipeline must run the Job before it updates the Deployment; a Job's pod template is immutable, so each release needs a new Job name or must delete the old Job first
Helm hookThe Job above, annotated helm.sh/hook: pre-install,pre-upgradeHelm runs it and waits; a failed hook stops the upgradeHooks are created before the chart's own Secrets and ConfigMaps, so on a first install a hook Job that reads the chart's Secret cannot start, and the install fails when Helm's --timeout runs out; create that Secret outside the chart

This page uses Spring Boot 4.1.2 with Flyway 12.4 and Liquibase 5.0, on Java 21. To make several pods share one database here, the programs start an H2 database server and run each pod as a separate JVM. The Kubernetes manifests were validated against the Kubernetes 1.33 and 1.34 schemas but not applied to a cluster (not run). Database, Helm, Argo CD, Flux and service-mesh details come from their documentation.

Migrating at startup: replicas and the Flyway lock

When every pod migrates at startup, Flyway makes the new pods take turns with a database lock. On H2 it locks the history table's rows with SELECT ... FOR UPDATE. On PostgreSQL it takes an advisory lock: a lock identified by a number, which the database holds for a session or a transaction without locking any table. On MySQL and SQL Server it takes a named lock. This program starts three pods of one release half a second apart against an empty database. V2 is a Java migration that takes 3 seconds, standing in for a slow data change such as a backfill (an UPDATE that fills a new column in existing rows), so the other pods arrive while it runs:

SQL
-- db/shop/V1__create_customer.sql
CREATE TABLE customer (
    id   BIGINT PRIMARY KEY,
    name VARCHAR(100) NOT NULL
);

-- db/shop/V3__add_email.sql
ALTER TABLE customer ADD COLUMN email VARCHAR(200);
Java
package demo.migrations.concurrent;

import java.io.IOException;
import java.net.ServerSocket;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import org.flywaydb.core.api.migration.BaseJavaMigration;
import org.flywaydb.core.api.migration.Context;
import org.flywaydb.core.api.output.MigrateResult;
import org.h2.tools.Server;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.boot.flyway.autoconfigure.FlywayConfigurationCustomizer;
import org.springframework.boot.flyway.autoconfigure.FlywayMigrationStrategy;
import org.springframework.context.annotation.Bean;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.jdbc.datasource.DriverManagerDataSource;

@SpringBootApplication
public class ConcurrentReplicas {

    // ---------- one pod: a Spring Boot application that runs Flyway at startup ----------

    // a migration that takes a while, like a backfill on a large table
    static class V2__slow_backfill extends BaseJavaMigration {
        @Override
        public void migrate(Context context) throws Exception {
            Thread.sleep(3000);
        }
    }

    @Bean
    FlywayConfigurationCustomizer javaMigrations() {
        return configuration -> configuration.javaMigrations(new V2__slow_backfill());
    }

    // the default strategy just calls migrate(); this one also reports what it did
    @Bean
    FlywayMigrationStrategy reportingStrategy() {
        return flyway -> {
            long t0 = System.nanoTime();
            MigrateResult result = flyway.migrate();
            List<String> versions = result.migrations.stream().map(m -> "V" + m.version).toList();
            System.out.printf("result applied %s, migrate() took %.0f s%n",
                    versions.isEmpty() ? "nothing" : String.join(", ", versions),
                    (System.nanoTime() - t0) / 1e9);
        };
    }

    static void runPod(String url) {
        SpringApplication app = new SpringApplication(ConcurrentReplicas.class);
        app.setDefaultProperties(Map.of(
                "spring.main.web-application-type", "none",
                "spring.main.banner-mode", "off",
                "logging.level.root", "OFF",
                "spring.liquibase.enabled", "false", // Liquibase is on this class path too
                "spring.flyway.locations", "classpath:db/shop",
                "spring.datasource.url", url));
        app.run().close();
    }

    // ---------- the driver: a shared database server and pods in separate JVMs ----------

    static Process startPod(String url) throws IOException {
        return new ProcessBuilder(ProcessHandle.current().info().command().orElseThrow(),
                "-cp", System.getProperty("java.class.path"),
                ConcurrentReplicas.class.getName(), url)
                .redirectErrorStream(true).start();
    }

    static String result(Process pod) throws IOException {
        return new String(pod.getInputStream().readAllBytes()).lines()
                .filter(l -> l.startsWith("result ")).map(l -> l.substring(7))
                .findFirst().orElse("no result");
    }

    static void threePods(String url) throws Exception {
        List<Process> pods = new ArrayList<>();
        for (int i = 0; i < 3; i++) {
            pods.add(startPod(url));
            Thread.sleep(500);
        }
        for (int i = 0; i < 3; i++) {
            System.out.println("  pod-" + (i + 1) + ": " + result(pods.get(i)));
        }
        System.out.println("  flyway_schema_history:");
        new JdbcTemplate(new DriverManagerDataSource(url)).queryForList(
                "SELECT \"installed_rank\", \"version\", \"description\", \"success\" "
                        + "FROM \"flyway_schema_history\" ORDER BY 1")
                .forEach(row -> System.out.println("    " + row.values()));
    }

    public static void main(String[] args) throws Exception {
        if (args.length > 0) {
            runPod(args[0]);
            return;
        }
        int port;
        try (ServerSocket free = new ServerSocket(0)) {
            port = free.getLocalPort();
        }
        Server server = Server.createTcpServer("-tcpPort", String.valueOf(port),
                "-ifNotExists").start();
        String url = "jdbc:h2:tcp://localhost:" + port + "/mem:";
        // wait up to 30 s for a row lock (H2's default is 2 s)
        threePods(url + "shop;DB_CLOSE_DELAY=-1;LOCK_TIMEOUT=30000");
        server.stop();
    }
}
Output
pod-1: applied V1, V2, migrate() took 4 s
  pod-2: applied V3, migrate() took 3 s
  pod-3: applied nothing, migrate() took 2 s
  flyway_schema_history:
    [-1, null, << Flyway Schema History table created >>, true]
    [1, 1, create customer, true]
    [2, 2, slow backfill, true]
    [3, 3, add email, true]
  • The history table has each version exactly once, after the marker row Flyway writes when it creates the table. The lock did its job.
  • Flyway takes the lock and re-reads the history for each migration in turn, so when pod 1 finishes one script, a waiting pod can win the lock and apply the next. Here pod 1 applied V1 and V2; a waiting pod applied V3. The split changes from run to run; the database ended up the same every time.
  • With spring.flyway.group=true, one pod applies all pending migrations in one transaction. On PostgreSQL they succeed or roll back together; on MySQL, every DDL statement commits on its own, so a failure leaves the earlier ones applied.
  • Pods 2 and 3 spent seconds in migrate(), waiting for the lock. Every new replica stays not ready for as long as the migrations take, so a 10-minute data migration needs a startup probe that allows for it (see Kubernetes probes).

On PostgreSQL, Flyway 12.4 does not block on its advisory lock: it tries to take it once a second and fails after spring.flyway.lock-retry-count retries (default 50) with "Number of retries exceeded while attempting to acquire PostgreSQL advisory lock". With a migration longer than about 50 seconds, the other new pods fail and crash-loop. Raise the count (or set it to -1 to wait without limit) and give the startup probe the same room, or move long migrations to a Job. On MySQL, Flyway waits for its lock without a limit, and on SQL Server for up to an hour.

Liquibase: the lock table and the stuck lock

Liquibase keeps a one-row table, DATABASECHANGELOGLOCK. A pod that wants to migrate sets LOCKED to true and writes its host name to LOCKEDBY, runs its changesets (Liquibase's unit of change, like one Flyway script), then sets LOCKED back to false. Because the lock is an ordinary row and not a database session lock, it survives the pod that took it. This program runs three pods of a release with a 3-second change, then simulates a pod killed while it held the lock, and starts more pods before and after the lock is cleared. release2.yaml adds the slow change to release 1's table, and release3.yaml adds a column:

YAML
# db/liquibase/release2.yaml
databaseChangeLog:
  - include: {file: db/liquibase/1-customer.yaml}
  - include: {file: db/liquibase/2-backfill.yaml}
YAML
# db/liquibase/2-backfill.yaml
databaseChangeLog:
  - changeSet:
      id: 2-slow-backfill
      author: shop
      changes:
        # stands in for a long data change: a Java class that takes 3 seconds
        - customChange:
            class: demo.migrations.liquibase.LiquibaseLocks$SlowBackfill
Java
package demo.migrations.liquibase;

import java.io.IOException;
import java.net.ServerSocket;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import liquibase.change.custom.CustomTaskChange;
import liquibase.database.Database;
import liquibase.exception.ValidationErrors;
import liquibase.resource.ResourceAccessor;
import org.h2.tools.Server;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.jdbc.datasource.DriverManagerDataSource;

@SpringBootApplication
public class LiquibaseLocks {

    // ---------- one pod: a Spring Boot application that runs Liquibase at startup ----------

    static volatile boolean ranBackfill;

    // the change in 2-backfill.yaml: stands in for a slow data change
    public static class SlowBackfill implements CustomTaskChange {
        @Override
        public void execute(Database database) {
            try {
                Thread.sleep(3000);
            } catch (InterruptedException e) {
                Thread.currentThread().interrupt();
            }
            ranBackfill = true;
        }

        @Override
        public String getConfirmationMessage() {
            return "backfill done";
        }

        @Override
        public void setUp() {
        }

        @Override
        public void setFileOpener(ResourceAccessor resourceAccessor) {
        }

        @Override
        public ValidationErrors validate(Database database) {
            return new ValidationErrors();
        }
    }

    static void runPod(String url, String changelog) {
        SpringApplication app = new SpringApplication(LiquibaseLocks.class);
        app.setDefaultProperties(Map.of(
                "spring.main.web-application-type", "none",
                "spring.main.banner-mode", "off",
                "logging.level.root", "OFF",
                "spring.flyway.enabled", "false", // Flyway is on this class path too
                "spring.liquibase.change-log", "classpath:db/liquibase/" + changelog,
                "spring.datasource.url", url));
        long t0 = System.nanoTime();
        try {
            app.run().close();
            System.out.printf("result started after %.0f s%s%n", (System.nanoTime() - t0) / 1e9,
                    ranBackfill ? ", ran the 3-second change" : "");
        } catch (RuntimeException e) {
            System.out.println("result failed: " + rootMessage(e));
        }
    }

    static String rootMessage(Throwable e) {
        while (e.getCause() != null) {
            e = e.getCause();
        }
        String message = e.getMessage();
        int cut = message.indexOf(" since "); // drop " since <time>", which differs on every run
        return e.getClass().getSimpleName() + ": "
                + (cut > 0 ? message.substring(0, cut) : message);
    }

    // ---------- the driver: a shared database server and pods in separate JVMs ----------

    static Process startPod(String url, String changelog, String... jvmArgs) throws IOException {
        List<String> command = new ArrayList<>();
        command.add(ProcessHandle.current().info().command().orElseThrow());
        command.addAll(List.of(jvmArgs));
        command.addAll(List.of("-cp", System.getProperty("java.class.path"),
                LiquibaseLocks.class.getName(), url, changelog));
        return new ProcessBuilder(command).redirectErrorStream(true).start();
    }

    static String result(Process pod) throws IOException {
        return new String(pod.getInputStream().readAllBytes()).lines()
                .filter(l -> l.startsWith("result ")).map(l -> l.substring(7))
                .findFirst().orElse("no result");
    }

    // three pods of one Deployment, starting half a second apart
    static void threePods(String url, String changelog) throws Exception {
        List<Process> pods = new ArrayList<>();
        for (int i = 0; i < 3; i++) {
            pods.add(startPod(url, changelog));
            Thread.sleep(500);
        }
        for (int i = 0; i < 3; i++) {
            System.out.println("  pod-" + (i + 1) + ": " + result(pods.get(i)));
        }
    }

    static void printLock(JdbcTemplate jdbc) {
        jdbc.queryForList("SELECT ID, LOCKED, LOCKEDBY FROM DATABASECHANGELOGLOCK")
                .forEach(row -> System.out.println("  DATABASECHANGELOGLOCK " + row.values()));
    }

    public static void main(String[] args) throws Exception {
        if (args.length > 0) {
            runPod(args[0], args[1]);
            return;
        }
        int port;
        try (ServerSocket free = new ServerSocket(0)) {
            port = free.getLocalPort();
        }
        Server server = Server.createTcpServer("-tcpPort", String.valueOf(port),
                "-ifNotExists").start();
        String base = "jdbc:h2:tcp://localhost:" + port + "/mem:";

        System.out.println("== second release with a 3-second change, three pods");
        String shop = base + "shop;DB_CLOSE_DELAY=-1";
        System.out.println("  (release 1 alone: " + result(startPod(shop, "release1.yaml")) + ")");
        threePods(shop, "release2.yaml");

        System.out.println("== a pod was killed while it held the lock");
        JdbcTemplate jdbc = new JdbcTemplate(new DriverManagerDataSource(shop));
        jdbc.update("UPDATE DATABASECHANGELOGLOCK SET LOCKED = TRUE, "
                + "LOCKGRANTED = CURRENT_TIMESTAMP, LOCKEDBY = 'orders-7c9d4-k2x8p (10.1.4.23)'");
        printLock(jdbc);
        System.out.println("  release 2 again: " + result(startPod(shop, "release2.yaml",
                "-Dliquibase.changelogLockWaitTimeInMinutes=0"))); // give up at once
        System.out.println("  release 3: " + result(startPod(shop, "release3.yaml",
                "-Dliquibase.changelogLockWaitTimeInMinutes=0")));

        System.out.println("== after releasing the lock by hand");
        jdbc.update("UPDATE DATABASECHANGELOGLOCK SET LOCKED = FALSE, LOCKGRANTED = NULL, "
                + "LOCKEDBY = NULL");
        System.out.println("  release 3: " + result(startPod(shop, "release3.yaml")));
        server.stop();
    }
}
Output
== second release with a 3-second change, three pods
  (release 1 alone: started after 3 s)
  pod-1: started after 10 s, ran the 3-second change
  pod-2: started after 17 s
  pod-3: started after 16 s
== a pod was killed while it held the lock
  DATABASECHANGELOGLOCK [1, true, orders-7c9d4-k2x8p (10.1.4.23)]
  release 2 again: started after 2 s
  release 3: failed: LockException: Could not acquire change log lock.  Currently locked by orders-7c9d4-k2x8p (10.1.4.23)
== after releasing the lock by hand
  release 3: started after 2 s
  • Each time runs inside the pod's JVM, from SpringApplication.run until the application had started. Pod 1 ran the change and took longer than release 1 alone plus the change, because three JVMs were starting at once on this machine's 2 cores.
  • Pods 2 and 3 found the lock taken and waited. Liquibase checks the lock every 10 seconds by default (liquibase.changelogLockPollRate), so each started at its next check after pod 1 had finished. Each waiting pod gives up after 5 minutes (liquibase.changelogLockWaitTimeInMinutes).
  • The simulated kill left LOCKED set, with the dead pod's name in LOCKEDBY. Nothing clears it automatically: Liquibase cannot tell a dead holder from a slow one.
  • Release 2 started anyway. All its changesets had already run, and Liquibase checks for pending changesets before it asks for the lock, so a pod with nothing to apply does not notice the stuck lock.
  • Release 3 had a new changeset, so it needed the lock and failed with LockException. With the default 5-minute wait, every pod of the release would hang for 5 minutes, fail, and restart, until someone cleared the lock.
  • After the row was reset, release 3 started normally.

To clear the lock, run the Liquibase CLI's liquibase release-locks, or the program's second UPDATE. Run it only after checking that no pod is still migrating, because releasing a lock that is really held lets two pods run changesets at once. The community extension liquibase-sessionlock replaces the row with a session lock on PostgreSQL, MySQL, MariaDB and Oracle (not run here). Flyway's locks already belong to the database session or transaction, so the database releases them when the connection dies. Its default PostgreSQL lock is transaction-level and works through PgBouncer in transaction mode; a session-level lock (spring.flyway.postgresql.transactional-lock=false, or liquibase-sessionlock) does not, so point such migrations at the database itself.

Migrating in a separate step: a Job or an init container

Running the migration as its own step gives it its own logs and exit code; a Job also gets its own timeout and credentials and ends the competition between pods. The application image can both migrate and serve if it has a mode switch, and the switch belongs in main. A FlywayMigrationStrategy alone does not end the application: a web image started with APP_MODE=migrate would migrate, start its web server and never exit, so the Job or init container would never finish.

In this program, migrate mode builds the application with WebApplicationType.NONE and passes the exit code from SpringApplication.exit to System.exit. Anything that picks up work, such as Kafka listeners and schedulers, must stay off in that mode; the @ConditionalOnProperty bean here shows how. In serve mode, Flyway only validates, so a pod that starts against an old schema fails at once. The driver runs each pod as a separate JVM with APP_MODE set, as Kubernetes would; Spring Boot's relaxed binding maps it to app.mode, which defaults to serve:

Java
package demo.migrations.job;

import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.net.ServerSocket;
import java.util.Map;
import org.flywaydb.core.Flyway;
import org.h2.tools.Server;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.WebApplicationType;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
import org.springframework.boot.context.event.ApplicationReadyEvent;
import org.springframework.boot.flyway.autoconfigure.FlywayMigrationStrategy;
import org.springframework.context.ApplicationListener;
import org.springframework.context.annotation.Bean;

@SpringBootApplication
public class MigrationJob {

    // ---------- the application image: one image, two modes ----------

    // the Job migrates; the application pods only check that the schema matches the code
    @Bean
    FlywayMigrationStrategy strategy(@Value("${app.mode:serve}") String mode) {
        return switch (mode) {
            case "migrate" -> flyway -> System.out.println("applied "
                    + flyway.migrate().migrationsExecuted + " migration(s)");
            case "serve" -> Flyway::validate;
            default -> throw new IllegalArgumentException(mode);
        };
    }

    // stands in for everything that picks up work: Kafka listeners, schedulers, queue consumers
    @Bean
    @ConditionalOnProperty(name = "app.mode", havingValue = "serve", matchIfMissing = true)
    ApplicationListener<ApplicationReadyEvent> startTakingWork() {
        return event -> System.out.println("web server started, accepting traffic");
    }

    static void runApplication(String[] args) {
        SpringApplication app = new SpringApplication(MigrationJob.class);
        app.setDefaultProperties(Map.of( // in a real image: application.properties
                "server.port", "0",
                "spring.main.banner-mode", "off",
                "logging.level.root", "OFF",
                "spring.liquibase.enabled", "false", // Liquibase is on this class path too
                "spring.flyway.locations", "classpath:db/job"));
        if ("migrate".equals(System.getenv("APP_MODE"))) {
            // no web server: migrate, close the context, and exit with its exit code
            app.setWebApplicationType(WebApplicationType.NONE);
            System.exit(SpringApplication.exit(app.run(args)));
        }
        app.run(args);
    }

    public static void main(String[] args) throws Exception {
        if (System.getenv("APP_MODE") != null) {
            runApplication(args);
        } else {
            simulateRelease();
        }
    }

    // ---------- the driver: a shared database server and pods in separate JVMs ----------

    static Process startPod(String mode, String url) throws IOException {
        ProcessBuilder pod = new ProcessBuilder(
                ProcessHandle.current().info().command().orElseThrow(),
                "-cp", System.getProperty("java.class.path"), MigrationJob.class.getName());
        pod.environment().put("APP_MODE", mode); // what the Job or Deployment sets
        pod.environment().put("SPRING_DATASOURCE_URL", url);
        return pod.redirectErrorStream(true).start();
    }

    // prints the pod's own lines and the cause of a failed start (not the stack trace),
    // and stops a serving pod once it takes traffic
    static void watch(Process pod) throws Exception {
        BufferedReader out = new BufferedReader(new InputStreamReader(pod.getInputStream()));
        boolean inCause = false;
        for (String line; (line = out.readLine()) != null; ) {
            if (line.startsWith("Caused by: ")) { // "Caused by: org.flywaydb...Exception: text"
                inCause = true;
                int colon = line.indexOf(':', "Caused by: ".length());
                System.out.println("  " + line.substring(line.lastIndexOf('.', colon) + 1));
            } else if (line.startsWith("applied ") || line.startsWith("web server")
                    || inCause && line.startsWith("Detected ")) {
                System.out.println("  " + line);
            }
            if (line.startsWith("web server")) {
                pod.destroy(); // the demo ends here; on Kubernetes the pod keeps serving
                System.out.println("  (still running: stopped by the demo)");
                pod.waitFor();
                return;
            }
        }
        System.out.println("  exit code " + pod.waitFor());
    }

    static void simulateRelease() throws Exception {
        int port;
        try (ServerSocket free = new ServerSocket(0)) {
            port = free.getLocalPort();
        }
        Server server = Server.createTcpServer("-tcpPort", String.valueOf(port),
                "-ifNotExists").start();
        String url = "jdbc:h2:tcp://localhost:" + port + "/mem:job;DB_CLOSE_DELAY=-1";

        System.out.println("== application pod (APP_MODE=serve) before the Job has run");
        watch(startPod("serve", url));
        System.out.println("== migration Job (APP_MODE=migrate), same image");
        watch(startPod("migrate", url));
        System.out.println("== application pod (APP_MODE=serve) after the Job");
        watch(startPod("serve", url));
        server.stop();
    }
}
Output
== application pod (APP_MODE=serve) before the Job has run
  FlywayValidateException: Validate failed: Migrations have failed validation
  Detected resolved migration not applied to database: 1.
  Detected resolved migration not applied to database: 2.
  exit code 1
== migration Job (APP_MODE=migrate), same image
  applied 2 migration(s)
  exit code 0
== application pod (APP_MODE=serve) after the Job
  web server started, accepting traffic
  (still running: stopped by the demo)
  • Before the Job ran, the application pod refused to start. Flyway.validate() reported both scripts as not applied, and the JVM exited with code 1. On Kubernetes, it would restart until the Job had run, while the previous version kept serving.
  • In migrate mode, the same image applied two scripts, started no web server, and exited with code 0. Exit code 0 tells Kubernetes that the Job or init container succeeded.
  • After the Job, a new application pod started its web server and took traffic. Validation does not block a rollback: an older release finds applied migrations it does not know, and Flyway ignores those by default.

A Job that runs once per release (not run here, validated only):

YAML
apiVersion: batch/v1
kind: Job
metadata:
  name: orders-migrate-1-5-0     # a new name per release: a Job's pod template is immutable
spec:
  backoffLimit: 2                 # up to 2 retries after a failed pod
  activeDeadlineSeconds: 900      # fail the release rather than wait forever
  ttlSecondsAfterFinished: 86400  # delete the finished Job, and its pod's logs, after a day
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: migrate
          image: registry.example.com/orders:1.5.0
          env:
            - name: APP_MODE
              value: migrate
            - name: SPRING_DATASOURCE_URL
              value: jdbc:postgresql://postgres:5432/orders
            - name: SPRING_DATASOURCE_USERNAME
              valueFrom:
                secretKeyRef:
                  name: orders-db
                  key: username
            - name: SPRING_DATASOURCE_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: orders-db
                  key: password

The pipeline applies the Job, waits with kubectl wait --for=condition=complete --timeout=15m job/orders-migrate-1-5-0, and only then updates the Deployment. Without --timeout, kubectl wait gives up after 30 seconds, and a failed Job shows up only as that timeout unless the pipeline also waits for condition=failed. As a Helm hook, the Job also falls under helm upgrade --timeout (5 minutes by default), so pass a longer one: if Helm gives up, the next upgrade deletes the Job, killing a migration that may still be running, because the default helm.sh/hook-delete-policy is before-hook-creation.

A Job can still run a migration twice or stop it midway:

  • Two runs: the Job documentation warns that a Job may occasionally start its pod twice, and backoffLimit retries failures; the migration lock keeps those runs apart.
  • Killed midway: when activeDeadlineSeconds fires or a node is drained, the migration stops halfway, so set the deadline well above the slowest migration. A pod lost to a drain also counts against backoffLimit, unless a podFailurePolicy rule with action: Ignore matches the DisruptionTarget condition (stable since Kubernetes 1.31).
  • Cleaning up: PostgreSQL rolls back the open transaction, so a rerun of a transactional script starts clean. But the killed backend notices the lost connection only when its current statement ends (sooner with client_connection_check_interval, PostgreSQL 14 and later) and holds Flyway's lock until then, and a script outside a transaction, such as CREATE INDEX CONCURRENTLY, can leave partial work. MySQL commits each DDL statement as it runs, so the rerun fails until someone removes the half-applied changes by hand and, if Flyway recorded a failure, runs flyway repair.
  • Service mesh: an injected Istio or Linkerd sidecar keeps running after the migration exits, so the Job never completes. Use the mesh's native-sidecar mode (Kubernetes 1.29 and later) or turn off injection for the Job.

An init container runs the same migrate container before the application container in every pod (not run here, validated only):

YAML
# part of the Deployment's pod template
    spec:
      initContainers:
        - name: migrate
          image: registry.example.com/orders:1.5.0
          env:
            - name: APP_MODE
              value: migrate
            - name: SPRING_DATASOURCE_URL
              value: jdbc:postgresql://postgres:5432/orders
            # plus the migration credentials, as in the Job
      containers:
        - name: app
          image: registry.example.com/orders:1.5.0
          env:
            - name: APP_MODE
              value: serve
            - name: SPRING_DATASOURCE_URL
              value: jdbc:postgresql://postgres:5432/orders
            # plus the application's own credentials

The init container needs no pipeline step, but it runs in every new pod, so the replicas compete for the lock as before. On PostgreSQL, raise spring.flyway.lock-retry-count for the init container too: otherwise the waiting pods fail after about 50 seconds and go into Init:CrashLoopBackOff, whose back-off (up to 5 minutes) delays the rollout after the migration has finished. The Deployment's progressDeadlineSeconds (600 by default) still applies: a rollout that makes no progress for longer is reported as failed. Either way, the migration finishes while the old version still serves all traffic. That is the real problem.

The rolling-update trap: old pods on the new schema

Every option above has a window in which the old code runs against the new schema. With a Job or hook, the window is the whole rollout: the schema changes first, then pods are replaced a few at a time. With migration at startup, it opens when the first new pod migrates. A column rename is the classic change that breaks in that window, so this program renames name to full_name in one step and runs the data access code of both versions:

SQL
-- db/rename/V2__rename_name_to_full_name.sql
ALTER TABLE customer RENAME COLUMN name TO full_name;
Java
package demo.migrations.rename;

import ch.qos.logback.classic.Level;
import java.util.List;
import org.flywaydb.core.Flyway;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.dao.DataAccessException;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.jdbc.datasource.DriverManagerDataSource;

public class BreakingRename {

    static final String URL = "jdbc:h2:mem:rename;DB_CLOSE_DELAY=-1";
    static final JdbcTemplate JDBC = new JdbcTemplate(new DriverManagerDataSource(URL));

    // the data access code of each application version
    interface CustomerCode {
        void save(long id, String name);

        List<String> names();
    }

    static class Version1 implements CustomerCode {
        public void save(long id, String name) {
            JDBC.update("INSERT INTO customer (id, name) VALUES (?, ?)", id, name);
        }

        public List<String> names() {
            return JDBC.queryForList("SELECT name FROM customer ORDER BY id", String.class);
        }
    }

    static class Version2 implements CustomerCode {
        public void save(long id, String name) {
            JDBC.update("INSERT INTO customer (id, full_name) VALUES (?, ?)", id, name);
        }

        public List<String> names() {
            return JDBC.queryForList("SELECT full_name FROM customer ORDER BY id", String.class);
        }
    }

    static void migrate(String target) {
        int applied = Flyway.configure().dataSource(URL, "", "")
                .locations("classpath:db/rename").target(target).load()
                .migrate().migrationsExecuted;
        System.out.println("Flyway: applied " + applied + ", schema now at V" + target);
    }

    static void use(String pod, CustomerCode code, long id, String name) {
        try {
            code.save(id, name);
            System.out.println("  " + pod + ": saved " + name + ", reads " + code.names());
        } catch (DataAccessException e) {
            String m = e.getMostSpecificCause().getMessage();
            System.out.println("  " + pod + ": FAILED " + m.substring(0, m.indexOf(';')));
        }
    }

    public static void main(String[] args) {
        // without Spring Boot, Logback logs everything at DEBUG; keep only errors
        // Logback's Logger is named in full: org.slf4j.Logger has no setLevel
        ((ch.qos.logback.classic.Logger) LoggerFactory.getLogger(Logger.ROOT_LOGGER_NAME))
                .setLevel(Level.ERROR);
        System.out.println("== release 1 is running");
        migrate("1");
        use("v1 pod", new Version1(), 1, "Ada");

        System.out.println("== rolling update to release 2: the first new pod migrates");
        migrate("2");
        use("v2 pod", new Version2(), 2, "Grace");
        use("v1 pod", new Version1(), 3, "Linus"); // old pods still take traffic

        System.out.println("== release 2 fails, so the Deployment is rolled back to release 1");
        use("v1 pod", new Version1(), 4, "Barbara");
    }
}
Output
== release 1 is running
Flyway: applied 1, schema now at V1
  v1 pod: saved Ada, reads [Ada]
== rolling update to release 2: the first new pod migrates
Flyway: applied 1, schema now at V2
  v2 pod: saved Grace, reads [Ada, Grace]
  v1 pod: FAILED Column "NAME" not found
== release 2 fails, so the Deployment is rolled back to release 1
  v1 pod: FAILED Column "NAME" not found
  • The release-2 code worked at once and read Ada, saved by release 1, under the new column name.
  • The release-1 pods, still serving part of the traffic, failed every query that mentions name, so until the rollout finishes, every request that reaches an old pod fails.
  • The rollback made it worse: Flyway does not undo migrations, so the restored release-1 code failed on every request, not just those on old pods.

The Recreate strategy, which stops all old pods before starting new ones, removes the mixed-version window at the cost of downtime on every release, and still leaves rollbacks broken. The general fix is to make every migration compatible with the previous release's code.

Expand and contract

The expand/contract pattern, also called parallel change, splits a breaking change into steps that each keep the previous release working. The same rename takes three steps, one per release: expand (release 2) adds full_name and writes both columns; switch (release 3) moves all code to full_name and makes that column mandatory; contract (release 4) drops name.

The rename, release by release
ReleaseSchema after its migrationCode writesCode readsSafe rollback to
1name NOT NULLnamenameNot applicable
2 (expand)name NOT NULL; full_name nullable, copied from nameBothCOALESCE(full_name, name)1
3 (switch)full_name NOT NULL, name nullablefull_namefull_name2 (not 1)
4 (contract)full_name onlyfull_namefull_name3
SQL
-- db/expand/V2__expand_add_full_name.sql
-- release 2: add the new column next to the old one; nothing is removed
ALTER TABLE customer ADD COLUMN full_name VARCHAR(100);
UPDATE customer SET full_name = name;

-- db/expand/V3__backfill_and_relax_name.sql
-- release 3: release 2 is fully rolled out, so every running pod writes both columns;
-- copy every row that release-1 pods inserted or updated during that rollout
UPDATE customer SET full_name = name WHERE full_name IS DISTINCT FROM name;
ALTER TABLE customer ALTER COLUMN full_name SET NOT NULL;
ALTER TABLE customer ALTER COLUMN name DROP NOT NULL;

-- db/expand/V4__contract_drop_name.sql
-- release 4: no running code reads or writes name any more
ALTER TABLE customer DROP COLUMN name;

The scripts use PostgreSQL and H2 syntax. On MySQL, V3 compares with WHERE NOT (full_name <=> name) and changes the constraints with ALTER TABLE customer MODIFY full_name VARCHAR(100) NOT NULL, MODIFY name VARCHAR(100) NULL, repeating each column's full definition. In the program, a release-1 pod also updates a row during the release 2 rollout:

Java
package demo.migrations.expand;

import ch.qos.logback.classic.Level;
import java.util.List;
import org.flywaydb.core.Flyway;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.dao.DataAccessException;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.jdbc.datasource.DriverManagerDataSource;

public class ExpandContract {

    static final String URL = "jdbc:h2:mem:expand;DB_CLOSE_DELAY=-1";
    static final JdbcTemplate JDBC = new JdbcTemplate(new DriverManagerDataSource(URL));

    // the data access code of each application version
    interface CustomerCode {
        void save(long id, String name);

        void rename(long id, String name);

        List<String> names();
    }

    // release 1: knows only name
    static class Version1 implements CustomerCode {
        public void save(long id, String name) {
            JDBC.update("INSERT INTO customer (id, name) VALUES (?, ?)", id, name);
        }

        public void rename(long id, String name) {
            JDBC.update("UPDATE customer SET name = ? WHERE id = ?", name, id);
        }

        public List<String> names() {
            return JDBC.queryForList("SELECT name FROM customer ORDER BY id", String.class);
        }
    }

    // release 2: writes both columns, reads the new one with a fallback to the old one
    static class Version2 implements CustomerCode {
        public void save(long id, String name) {
            JDBC.update("INSERT INTO customer (id, name, full_name) VALUES (?, ?, ?)",
                    id, name, name);
        }

        public void rename(long id, String name) {
            JDBC.update("UPDATE customer SET name = ?, full_name = ? WHERE id = ?", name, name, id);
        }

        public List<String> names() {
            return JDBC.queryForList(
                    "SELECT COALESCE(full_name, name) FROM customer ORDER BY id", String.class);
        }
    }

    // release 3 and later: knows only full_name
    static class Version3 implements CustomerCode {
        public void save(long id, String name) {
            JDBC.update("INSERT INTO customer (id, full_name) VALUES (?, ?)", id, name);
        }

        public void rename(long id, String name) {
            JDBC.update("UPDATE customer SET full_name = ? WHERE id = ?", name, id);
        }

        public List<String> names() {
            return JDBC.queryForList("SELECT full_name FROM customer ORDER BY id", String.class);
        }
    }

    static long nextId = 1;

    static void migrate(String target) {
        Flyway.configure().dataSource(URL, "", "").locations("classpath:db/expand")
                .target(target).load().migrate();
        System.out.println("  Flyway: schema now at V" + target);
    }

    static void use(String pod, CustomerCode code, String name) {
        try {
            code.save(nextId++, name);
            System.out.println("  " + pod + ": saved " + name + ", reads " + code.names());
        } catch (DataAccessException e) {
            String m = e.getMostSpecificCause().getMessage();
            System.out.println("  " + pod + ": FAILED " + m.substring(0, m.indexOf(';')));
        }
    }

    static void printStaleCount() {
        System.out.println("  rows where full_name differs from name: " + JDBC.queryForObject(
                "SELECT COUNT(*) FROM customer WHERE full_name IS DISTINCT FROM name",
                Integer.class));
    }

    public static void main(String[] args) {
        // Logback's Logger is named in full: org.slf4j.Logger has no setLevel
        ((ch.qos.logback.classic.Logger) LoggerFactory.getLogger(Logger.ROOT_LOGGER_NAME))
                .setLevel(Level.ERROR);
        System.out.println("== release 1");
        migrate("1");
        use("v1 pod", new Version1(), "Ada");

        System.out.println("== release 2 (expand), old and new pods side by side");
        migrate("2");
        use("v1 pod", new Version1(), "Grace");
        new Version1().rename(1, "Ada Lovelace"); // release-1 code updates name only
        System.out.println("  v1 pod: renamed 1 to Ada Lovelace, reads " + new Version1().names());
        use("v2 pod", new Version2(), "Linus");
        printStaleCount();

        System.out.println("== release 3 (switch to the new column), old and new pods");
        migrate("3");
        use("v2 pod", new Version2(), "Barbara");
        use("v3 pod", new Version3(), "Edsger");
        use("v1 pod", new Version1(), "Alan"); // a rollback two releases back

        System.out.println("== release 4 (contract), only v3 code is running");
        migrate("4");
        use("v3 pod", new Version3(), "Margaret");
        use("v2 pod", new Version2(), "Ken"); // a rollback two releases back
    }
}
Output
== release 1
  Flyway: schema now at V1
  v1 pod: saved Ada, reads [Ada]
== release 2 (expand), old and new pods side by side
  Flyway: schema now at V2
  v1 pod: saved Grace, reads [Ada, Grace]
  v1 pod: renamed 1 to Ada Lovelace, reads [Ada Lovelace, Grace]
  v2 pod: saved Linus, reads [Ada, Grace, Linus]
  rows where full_name differs from name: 2
== release 3 (switch to the new column), old and new pods
  Flyway: schema now at V3
  v2 pod: saved Barbara, reads [Ada Lovelace, Grace, Linus, Barbara]
  v3 pod: saved Edsger, reads [Ada Lovelace, Grace, Linus, Barbara, Edsger]
  v1 pod: FAILED NULL not allowed for column "FULL_NAME"
== release 4 (contract), only v3 code is running
  Flyway: schema now at V4
  v3 pod: saved Margaret, reads [Ada Lovelace, Grace, Linus, Barbara, Edsger, Margaret]
  v2 pod: FAILED Column "NAME" not found
  • Release 2: V2 added full_name, copied from name. The release-1 pod saved Grace into name only, and the release-2 code showed her through COALESCE. It also renamed Ada in name only, so release 2 kept showing her old name: until V3 runs, release 2 shows old values for rows that release 1 updates. The double write, which name's NOT NULL also demands, keeps a rollback to release 1 safe: every row still has a name.
  • Release 3: V3 copied name into full_name wherever the two differ, which fixed Grace and Ada Lovelace, then made full_name mandatory and name optional. No release-3 code has written yet when V3 runs, so where the columns differ, name is newer. A V3 that copied only WHERE full_name IS NULL kept "Ada" in a run of the same program, and V4 then dropped the only copy of "Ada Lovelace". Release-2 pods worked unchanged, so a rollback to release 2 is safe; release-1 code failed, because its inserts omit full_name, now NOT NULL.
  • Release 4: V4 dropped name. Release-3 code never uses it; release-2 code failed. Each step keeps only the previous release working, so run the contract step only when nobody will roll back to a release that uses name.

Each release is still an ordinary rolling update with one migration step, and the two versions that can run together always agree on the schema. The price is three releases instead of one, and code that writes two columns for a while.

These scripts are fine for small tables. On a large PostgreSQL table, V2 holds an ACCESS EXCLUSIVE lock, which blocks even reads, for the whole UPDATE, and SET NOT NULL in V3 scans the whole table under that lock, because Flyway runs each script in one transaction. For a large table, put the DDL in its own script and backfill in batches outside a transaction or from a Kubernetes Job. Replace SET NOT NULL with three scripts, so that no lock lasts through a scan: add CHECK (full_name IS NOT NULL) NOT VALID; run VALIDATE CONSTRAINT, which does not block reads or writes; then run SET NOT NULL. PostgreSQL 12 and later prove it from the constraint without a scan; then drop the CHECK.

Which changes are safe in one step

The test for every migration: can the previous release, unchanged, work against the schema this script leaves?

Schema changes and the previous release
ChangePrevious release keeps working?Safe way
Add a tableYesOne step
Add a nullable column, or one with a defaultYes, if old code names its columns in INSERT and does not use SELECT * into fixed positionsOne step; with a default, instant on PostgreSQL 11 and later for a non-volatile default, such as a constant, and on MySQL 8.0.12 and later (last column only before 8.0.29); otherwise the table is rebuilt
Add an indexYes, but on PostgreSQL a plain index build blocks writes until it finishesCREATE INDEX CONCURRENTLY on PostgreSQL. It cannot run in a transaction, so put it alone in its script and set spring.flyway.postgresql.transactional-lock=false (Liquibase: runInTransaction: false). A failed build leaves an INVALID index to drop before retrying; do not add IF NOT EXISTS, which keeps the invalid one
Add a NOT NULL column without a defaultNo: old code's inserts failAdd it nullable, backfill, then add the constraint in a later release
Rename a column or tableNoExpand/contract, as above
Change a column's typeWidening, yes; narrowing or a new meaning, noWidening a VARCHAR changes only metadata (on MySQL, while the length prefix stays 1 or 2 bytes). Others, such as INT to BIGINT, rewrite the table: new column, double write, backfill, switch, drop
Drop a column or tableNo, if the previous release uses itStop using it in one release (for a column, that release's migration also makes it nullable or gives it a default); drop it in a later release
Add a constraint, such as a foreign key or CHECKOnly if old code never writes violating rowsFirst stop the code from writing violating rows; on PostgreSQL, add the constraint NOT VALID, then VALIDATE CONSTRAINT in a separate script

Even an instant change needs a lock, and waiting for that lock can itself cause the outage. On PostgreSQL, most ALTER TABLE forms wait for an ACCESS EXCLUSIVE lock behind any long-running query or open transaction that has used the table, and every new query on the table queues behind the waiting ALTER. Start the script with SET LOCAL lock_timeout = '5s', which ends with the script's transaction, and let the pod or Job retry when it expires. A plain SET would stay on the connection, and at startup Flyway borrows it from the application's pool unless spring.flyway.url or user is set. MySQL does the same with metadata locks: a pending ALTER blocks later statements on the table, even with ALGORITHM=INPLACE and LOCK=NONE, and lock_wait_timeout defaults to a year. For large MySQL tables, gh-ost and pt-online-schema-change copy the table in the background instead.

The Hibernate setting spring.jpa.hibernate.ddl-auto must be validate or none in production when a migration tool owns the schema; with update, every pod would change the schema on its own, with no lock and no history. A JPA entity names every mapped column in its INSERT and every SELECT, so the old entity survives an expand step only if the new columns are nullable or have defaults, and a column can be dropped only after a release whose entity no longer maps it. To get only a better Java name, rename the field and keep @Column(name = "name"); no migration is needed.

Which to use

Choosing where and how to migrate
SituationRecommendation
Small service, fast migrations, few replicasMigrate at startup; size the startup probe (and, on PostgreSQL, spring.flyway.lock-retry-count) for the slowest migration
Long data migrations, or many replicasA Job per release, or a Helm pre-upgrade hook with a longer --timeout; application pods validate only
App database user should not change the schemaA Job or hook with its own credentials. On PostgreSQL, with the Job as migrator and the pods as app, ALTER DEFAULT PRIVILEGES FOR ROLE migrator GRANT SELECT, INSERT, UPDATE, DELETE ON TABLES TO app and ... GRANT USAGE, SELECT ON SEQUENCES TO app cover later migrations; for existing ones, use GRANT ... ON ALL TABLES (and SEQUENCES) IN SCHEMA
No pipeline step for migrations, and migrations longer than the startup probe allowsInit container in migrate mode, with spring.flyway.lock-retry-count raised on PostgreSQL; keep the migration within progressDeadlineSeconds
Deploying with Argo CD or FluxA Job annotated argocd.argoproj.io/hook: PreSync, with the default BeforeHookCreation delete policy. Argo CD maps Helm's pre-install and pre-upgrade hooks to PreSync and runs them on every sync. Flux's helm-controller runs Helm hooks as Helm does
Several services share one databaseA schema per service (spring.flyway.default-schema); or a history table per service (spring.flyway.table) with spring.flyway.baseline-on-migrate=true and spring.flyway.baseline-version=0; without them, Flyway refuses the non-empty schema or skips V1
Liquibase on any of the aboveThe liquibase-sessionlock extension, or a runbook for release-locks; keep each changeset short
Rename, type change, drop, new NOT NULLExpand/contract across releases
Rollbacks must keep workingMake each migration compatible with the previous release. A Kubernetes rollback never runs undo scripts (U scripts in Flyway's paid editions, Liquibase rollback)

Related pages: the Helm hooks section for running a migration Job from a chart, Kubernetes probes for startup probes, graceful shutdown for how pods are stopped during a rollout, and deploying Spring Boot on Kubernetes for rolling update settings.