Database migrations for Spring Boot on Kubernetes
Spring Boot can apply database migrations when the application starts: Flyway or Liquibase checks which scripts the database has already run and applies only the new ones, each versioned script once. For a single instance, that is all it takes. On Kubernetes, three things change: several replicas start at once and find the same new script; old pods keep serving while new ones arrive; and a rollback brings old code back to a new schema. This page compares where to run migrations (at startup, in an init container, in a Job or in a Helm hook) and runs three situations against a shared H2 database server: concurrent migrations and their locks, a column rename that breaks running pods, and the expand/contract sequence that makes the same change safely.
Contents
Why migrations are different on Kubernetes Migrating at startup: replicas and the Flyway lock Liquibase: the lock table and the stuck lock Migrating in a separate step: a Job or an init container The rolling-update trap: old pods on the new schema Expand and contract Which changes are safe in one step Which to use
Why migrations are different on Kubernetes
- Several replicas start at once: the first pods of a new release find the same pending scripts, and each script must run exactly once.
- Old and new versions run side by side: a rolling update replaces pods a few at a time, so for minutes, old pods serve traffic against the schema the new code needs.
- Rollbacks bring old code back:
orkubectl rollout undo restores the previous image, but neither undoes the migrations. The old code then runs against the new schema until the next release.helm rollback
The tools solve the first problem with a database lock, which has traps of its own. The other two need migrations that the previous release can live with. First, decide where the migration runs:
| Where | How | Advantages | Watch out for |
|---|---|---|---|
| Application startup | Spring Boot runs Flyway or Liquibase when spring-boot-starter-flywayspring-boot-starter-liquibaseflyway-database-postgresql | Nothing to set up; schema and code always ship together | Every pod competes for the lock; a long migration delays readiness and needs a startup probe; the app's database user needs rights to change the schema (DDL) |
| Init container | The same image in a migrate-only mode, before the application container | Probes do not run until it finishes; separate logs and exit code | Runs again in every new pod (scale-up, replacement), though not when the application container restarts; a failure shows as Init:ErrorInit:CrashLoopBackOff |
| Kubernetes Job | A Job applied by the pipeline before the Deployment | One run per release, normally; separate credentials and timeout; a failed Job stops the pipeline | The pipeline must run the Job before it updates the Deployment; a Job's pod template is immutable, so each release needs a new Job name or must delete the old Job first |
| Helm hook | The Job above, annotated helm.sh/hook: pre-install,pre-upgrade | Helm runs it and waits; a failed hook stops the upgrade | Hooks are created before the chart's own Secrets and ConfigMaps, so on a first install a hook Job that reads the chart's Secret cannot start, and the install fails when Helm's --timeout |
This page uses Spring Boot 4.1.2 with Flyway 12.4 and Liquibase 5.0, on Java 21. To make several pods share one database here, the programs start an H2 database server and run each pod as a separate JVM. The Kubernetes manifests were validated against the Kubernetes 1.33 and 1.34 schemas but not applied to a cluster (not run). Database, Helm, Argo CD, Flux and service-mesh details come from their documentation.
Migrating at startup: replicas and the Flyway lock
When every pod migrates at startup, Flyway makes the new pods take turns with a database lock. On H2 it locks the history table's rows with SELECT ... FOR UPDATEUPDATE
-- db/shop/V1__create_customer.sql
CREATE TABLE customer (
id BIGINT PRIMARY KEY,
name VARCHAR(100) NOT NULL
);
-- db/shop/V3__add_email.sql
ALTER TABLE customer ADD COLUMN email VARCHAR(200);
package demo.migrations.concurrent;
import java.io.IOException;
import java.net.ServerSocket;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import org.flywaydb.core.api.migration.BaseJavaMigration;
import org.flywaydb.core.api.migration.Context;
import org.flywaydb.core.api.output.MigrateResult;
import org.h2.tools.Server;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.boot.flyway.autoconfigure.FlywayConfigurationCustomizer;
import org.springframework.boot.flyway.autoconfigure.FlywayMigrationStrategy;
import org.springframework.context.annotation.Bean;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.jdbc.datasource.DriverManagerDataSource;
@SpringBootApplication
public class ConcurrentReplicas {
// ---------- one pod: a Spring Boot application that runs Flyway at startup ----------
// a migration that takes a while, like a backfill on a large table
static class V2__slow_backfill extends BaseJavaMigration {
@Override
public void migrate(Context context) throws Exception {
Thread.sleep(3000);
}
}
@Bean
FlywayConfigurationCustomizer javaMigrations() {
return configuration -> configuration.javaMigrations(new V2__slow_backfill());
}
// the default strategy just calls migrate(); this one also reports what it did
@Bean
FlywayMigrationStrategy reportingStrategy() {
return flyway -> {
long t0 = System.nanoTime();
MigrateResult result = flyway.migrate();
List<String> versions = result.migrations.stream().map(m -> "V" + m.version).toList();
System.out.printf("result applied %s, migrate() took %.0f s%n",
versions.isEmpty() ? "nothing" : String.join(", ", versions),
(System.nanoTime() - t0) / 1e9);
};
}
static void runPod(String url) {
SpringApplication app = new SpringApplication(ConcurrentReplicas.class);
app.setDefaultProperties(Map.of(
"spring.main.web-application-type", "none",
"spring.main.banner-mode", "off",
"logging.level.root", "OFF",
"spring.liquibase.enabled", "false", // Liquibase is on this class path too
"spring.flyway.locations", "classpath:db/shop",
"spring.datasource.url", url));
app.run().close();
}
// ---------- the driver: a shared database server and pods in separate JVMs ----------
static Process startPod(String url) throws IOException {
return new ProcessBuilder(ProcessHandle.current().info().command().orElseThrow(),
"-cp", System.getProperty("java.class.path"),
ConcurrentReplicas.class.getName(), url)
.redirectErrorStream(true).start();
}
static String result(Process pod) throws IOException {
return new String(pod.getInputStream().readAllBytes()).lines()
.filter(l -> l.startsWith("result ")).map(l -> l.substring(7))
.findFirst().orElse("no result");
}
static void threePods(String url) throws Exception {
List<Process> pods = new ArrayList<>();
for (int i = 0; i < 3; i++) {
pods.add(startPod(url));
Thread.sleep(500);
}
for (int i = 0; i < 3; i++) {
System.out.println(" pod-" + (i + 1) + ": " + result(pods.get(i)));
}
System.out.println(" flyway_schema_history:");
new JdbcTemplate(new DriverManagerDataSource(url)).queryForList(
"SELECT \"installed_rank\", \"version\", \"description\", \"success\" "
+ "FROM \"flyway_schema_history\" ORDER BY 1")
.forEach(row -> System.out.println(" " + row.values()));
}
public static void main(String[] args) throws Exception {
if (args.length > 0) {
runPod(args[0]);
return;
}
int port;
try (ServerSocket free = new ServerSocket(0)) {
port = free.getLocalPort();
}
Server server = Server.createTcpServer("-tcpPort", String.valueOf(port),
"-ifNotExists").start();
String url = "jdbc:h2:tcp://localhost:" + port + "/mem:";
// wait up to 30 s for a row lock (H2's default is 2 s)
threePods(url + "shop;DB_CLOSE_DELAY=-1;LOCK_TIMEOUT=30000");
server.stop();
}
}
pod-1: applied V1, V2, migrate() took 4 s
pod-2: applied V3, migrate() took 3 s
pod-3: applied nothing, migrate() took 2 s
flyway_schema_history:
[-1, null, << Flyway Schema History table created >>, true]
[1, 1, create customer, true]
[2, 2, slow backfill, true]
[3, 3, add email, true]
- The history table has each version exactly once, after the marker row Flyway writes when it creates the table. The lock did its job.
- Flyway takes the lock and re-reads the history for each migration in turn, so when pod 1 finishes one script, a waiting pod can win the lock and apply the next. Here pod 1 applied V1 and V2; a waiting pod applied V3. The split changes from run to run; the database ended up the same every time.
- With
, one pod applies all pending migrations in one transaction. On PostgreSQL they succeed or roll back together; on MySQL, every DDL statement commits on its own, so a failure leaves the earlier ones applied.spring.flyway.group=true - Pods 2 and 3 spent seconds in
, waiting for the lock. Every new replica stays not ready for as long as the migrations take, so a 10-minute data migration needs a startup probe that allows for it (seemigrate()Kubernetes probes ).
On PostgreSQL, Flyway 12.4 does not block on its advisory lock: it tries to take it once a second and fails after spring.flyway.lock-retry-count-1
Liquibase: the lock table and the stuck lock
Liquibase keeps a one-row table, DATABASECHANGELOGLOCKLOCKEDLOCKEDBYLOCKEDrelease2.yamlrelease3.yaml
# db/liquibase/release2.yaml
databaseChangeLog:
- include: {file: db/liquibase/1-customer.yaml}
- include: {file: db/liquibase/2-backfill.yaml}
# db/liquibase/2-backfill.yaml
databaseChangeLog:
- changeSet:
id: 2-slow-backfill
author: shop
changes:
# stands in for a long data change: a Java class that takes 3 seconds
- customChange:
class: demo.migrations.liquibase.LiquibaseLocks$SlowBackfill
package demo.migrations.liquibase;
import java.io.IOException;
import java.net.ServerSocket;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import liquibase.change.custom.CustomTaskChange;
import liquibase.database.Database;
import liquibase.exception.ValidationErrors;
import liquibase.resource.ResourceAccessor;
import org.h2.tools.Server;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.jdbc.datasource.DriverManagerDataSource;
@SpringBootApplication
public class LiquibaseLocks {
// ---------- one pod: a Spring Boot application that runs Liquibase at startup ----------
static volatile boolean ranBackfill;
// the change in 2-backfill.yaml: stands in for a slow data change
public static class SlowBackfill implements CustomTaskChange {
@Override
public void execute(Database database) {
try {
Thread.sleep(3000);
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
}
ranBackfill = true;
}
@Override
public String getConfirmationMessage() {
return "backfill done";
}
@Override
public void setUp() {
}
@Override
public void setFileOpener(ResourceAccessor resourceAccessor) {
}
@Override
public ValidationErrors validate(Database database) {
return new ValidationErrors();
}
}
static void runPod(String url, String changelog) {
SpringApplication app = new SpringApplication(LiquibaseLocks.class);
app.setDefaultProperties(Map.of(
"spring.main.web-application-type", "none",
"spring.main.banner-mode", "off",
"logging.level.root", "OFF",
"spring.flyway.enabled", "false", // Flyway is on this class path too
"spring.liquibase.change-log", "classpath:db/liquibase/" + changelog,
"spring.datasource.url", url));
long t0 = System.nanoTime();
try {
app.run().close();
System.out.printf("result started after %.0f s%s%n", (System.nanoTime() - t0) / 1e9,
ranBackfill ? ", ran the 3-second change" : "");
} catch (RuntimeException e) {
System.out.println("result failed: " + rootMessage(e));
}
}
static String rootMessage(Throwable e) {
while (e.getCause() != null) {
e = e.getCause();
}
String message = e.getMessage();
int cut = message.indexOf(" since "); // drop " since <time>", which differs on every run
return e.getClass().getSimpleName() + ": "
+ (cut > 0 ? message.substring(0, cut) : message);
}
// ---------- the driver: a shared database server and pods in separate JVMs ----------
static Process startPod(String url, String changelog, String... jvmArgs) throws IOException {
List<String> command = new ArrayList<>();
command.add(ProcessHandle.current().info().command().orElseThrow());
command.addAll(List.of(jvmArgs));
command.addAll(List.of("-cp", System.getProperty("java.class.path"),
LiquibaseLocks.class.getName(), url, changelog));
return new ProcessBuilder(command).redirectErrorStream(true).start();
}
static String result(Process pod) throws IOException {
return new String(pod.getInputStream().readAllBytes()).lines()
.filter(l -> l.startsWith("result ")).map(l -> l.substring(7))
.findFirst().orElse("no result");
}
// three pods of one Deployment, starting half a second apart
static void threePods(String url, String changelog) throws Exception {
List<Process> pods = new ArrayList<>();
for (int i = 0; i < 3; i++) {
pods.add(startPod(url, changelog));
Thread.sleep(500);
}
for (int i = 0; i < 3; i++) {
System.out.println(" pod-" + (i + 1) + ": " + result(pods.get(i)));
}
}
static void printLock(JdbcTemplate jdbc) {
jdbc.queryForList("SELECT ID, LOCKED, LOCKEDBY FROM DATABASECHANGELOGLOCK")
.forEach(row -> System.out.println(" DATABASECHANGELOGLOCK " + row.values()));
}
public static void main(String[] args) throws Exception {
if (args.length > 0) {
runPod(args[0], args[1]);
return;
}
int port;
try (ServerSocket free = new ServerSocket(0)) {
port = free.getLocalPort();
}
Server server = Server.createTcpServer("-tcpPort", String.valueOf(port),
"-ifNotExists").start();
String base = "jdbc:h2:tcp://localhost:" + port + "/mem:";
System.out.println("== second release with a 3-second change, three pods");
String shop = base + "shop;DB_CLOSE_DELAY=-1";
System.out.println(" (release 1 alone: " + result(startPod(shop, "release1.yaml")) + ")");
threePods(shop, "release2.yaml");
System.out.println("== a pod was killed while it held the lock");
JdbcTemplate jdbc = new JdbcTemplate(new DriverManagerDataSource(shop));
jdbc.update("UPDATE DATABASECHANGELOGLOCK SET LOCKED = TRUE, "
+ "LOCKGRANTED = CURRENT_TIMESTAMP, LOCKEDBY = 'orders-7c9d4-k2x8p (10.1.4.23)'");
printLock(jdbc);
System.out.println(" release 2 again: " + result(startPod(shop, "release2.yaml",
"-Dliquibase.changelogLockWaitTimeInMinutes=0"))); // give up at once
System.out.println(" release 3: " + result(startPod(shop, "release3.yaml",
"-Dliquibase.changelogLockWaitTimeInMinutes=0")));
System.out.println("== after releasing the lock by hand");
jdbc.update("UPDATE DATABASECHANGELOGLOCK SET LOCKED = FALSE, LOCKGRANTED = NULL, "
+ "LOCKEDBY = NULL");
System.out.println(" release 3: " + result(startPod(shop, "release3.yaml")));
server.stop();
}
}
== second release with a 3-second change, three pods
(release 1 alone: started after 3 s)
pod-1: started after 10 s, ran the 3-second change
pod-2: started after 17 s
pod-3: started after 16 s
== a pod was killed while it held the lock
DATABASECHANGELOGLOCK [1, true, orders-7c9d4-k2x8p (10.1.4.23)]
release 2 again: started after 2 s
release 3: failed: LockException: Could not acquire change log lock. Currently locked by orders-7c9d4-k2x8p (10.1.4.23)
== after releasing the lock by hand
release 3: started after 2 s
- Each time runs inside the pod's JVM, from
until the application had started. Pod 1 ran the change and took longer than release 1 alone plus the change, because three JVMs were starting at once on this machine's 2 cores.SpringApplication.run - Pods 2 and 3 found the lock taken and waited. Liquibase checks the lock every 10 seconds by default (
), so each started at its next check after pod 1 had finished. Each waiting pod gives up after 5 minutes (liquibase.changelogLockPollRate ).liquibase.changelogLockWaitTimeInMinutes - The simulated kill left
set, with the dead pod's name inLOCKED . Nothing clears it automatically: Liquibase cannot tell a dead holder from a slow one.LOCKEDBY - Release 2 started anyway. All its changesets had already run, and Liquibase checks for pending changesets before it asks for the lock, so a pod with nothing to apply does not notice the stuck lock.
- Release 3 had a new changeset, so it needed the lock and failed with
. With the default 5-minute wait, every pod of the release would hang for 5 minutes, fail, and restart, until someone cleared the lock.LockException - After the row was reset, release 3 started normally.
To clear the lock, run the Liquibase CLI's liquibase release-locksUPDATEliquibase-sessionlockspring.flyway.postgresql.transactional-lock=falseliquibase-sessionlock
Migrating in a separate step: a Job or an init container
Running the migration as its own step gives it its own logs and exit code; a Job also gets its own timeout and credentials and ends the competition between pods. The application image can both migrate and serve if it has a mode switch, and the switch belongs in mainFlywayMigrationStrategyAPP_MODE=migrate
In this program, migrateWebApplicationType.NONESpringApplication.exitSystem.exit@ConditionalOnPropertyserveAPP_MODEapp.modeserve
package demo.migrations.job;
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.net.ServerSocket;
import java.util.Map;
import org.flywaydb.core.Flyway;
import org.h2.tools.Server;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.WebApplicationType;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
import org.springframework.boot.context.event.ApplicationReadyEvent;
import org.springframework.boot.flyway.autoconfigure.FlywayMigrationStrategy;
import org.springframework.context.ApplicationListener;
import org.springframework.context.annotation.Bean;
@SpringBootApplication
public class MigrationJob {
// ---------- the application image: one image, two modes ----------
// the Job migrates; the application pods only check that the schema matches the code
@Bean
FlywayMigrationStrategy strategy(@Value("${app.mode:serve}") String mode) {
return switch (mode) {
case "migrate" -> flyway -> System.out.println("applied "
+ flyway.migrate().migrationsExecuted + " migration(s)");
case "serve" -> Flyway::validate;
default -> throw new IllegalArgumentException(mode);
};
}
// stands in for everything that picks up work: Kafka listeners, schedulers, queue consumers
@Bean
@ConditionalOnProperty(name = "app.mode", havingValue = "serve", matchIfMissing = true)
ApplicationListener<ApplicationReadyEvent> startTakingWork() {
return event -> System.out.println("web server started, accepting traffic");
}
static void runApplication(String[] args) {
SpringApplication app = new SpringApplication(MigrationJob.class);
app.setDefaultProperties(Map.of( // in a real image: application.properties
"server.port", "0",
"spring.main.banner-mode", "off",
"logging.level.root", "OFF",
"spring.liquibase.enabled", "false", // Liquibase is on this class path too
"spring.flyway.locations", "classpath:db/job"));
if ("migrate".equals(System.getenv("APP_MODE"))) {
// no web server: migrate, close the context, and exit with its exit code
app.setWebApplicationType(WebApplicationType.NONE);
System.exit(SpringApplication.exit(app.run(args)));
}
app.run(args);
}
public static void main(String[] args) throws Exception {
if (System.getenv("APP_MODE") != null) {
runApplication(args);
} else {
simulateRelease();
}
}
// ---------- the driver: a shared database server and pods in separate JVMs ----------
static Process startPod(String mode, String url) throws IOException {
ProcessBuilder pod = new ProcessBuilder(
ProcessHandle.current().info().command().orElseThrow(),
"-cp", System.getProperty("java.class.path"), MigrationJob.class.getName());
pod.environment().put("APP_MODE", mode); // what the Job or Deployment sets
pod.environment().put("SPRING_DATASOURCE_URL", url);
return pod.redirectErrorStream(true).start();
}
// prints the pod's own lines and the cause of a failed start (not the stack trace),
// and stops a serving pod once it takes traffic
static void watch(Process pod) throws Exception {
BufferedReader out = new BufferedReader(new InputStreamReader(pod.getInputStream()));
boolean inCause = false;
for (String line; (line = out.readLine()) != null; ) {
if (line.startsWith("Caused by: ")) { // "Caused by: org.flywaydb...Exception: text"
inCause = true;
int colon = line.indexOf(':', "Caused by: ".length());
System.out.println(" " + line.substring(line.lastIndexOf('.', colon) + 1));
} else if (line.startsWith("applied ") || line.startsWith("web server")
|| inCause && line.startsWith("Detected ")) {
System.out.println(" " + line);
}
if (line.startsWith("web server")) {
pod.destroy(); // the demo ends here; on Kubernetes the pod keeps serving
System.out.println(" (still running: stopped by the demo)");
pod.waitFor();
return;
}
}
System.out.println(" exit code " + pod.waitFor());
}
static void simulateRelease() throws Exception {
int port;
try (ServerSocket free = new ServerSocket(0)) {
port = free.getLocalPort();
}
Server server = Server.createTcpServer("-tcpPort", String.valueOf(port),
"-ifNotExists").start();
String url = "jdbc:h2:tcp://localhost:" + port + "/mem:job;DB_CLOSE_DELAY=-1";
System.out.println("== application pod (APP_MODE=serve) before the Job has run");
watch(startPod("serve", url));
System.out.println("== migration Job (APP_MODE=migrate), same image");
watch(startPod("migrate", url));
System.out.println("== application pod (APP_MODE=serve) after the Job");
watch(startPod("serve", url));
server.stop();
}
}
== application pod (APP_MODE=serve) before the Job has run
FlywayValidateException: Validate failed: Migrations have failed validation
Detected resolved migration not applied to database: 1.
Detected resolved migration not applied to database: 2.
exit code 1
== migration Job (APP_MODE=migrate), same image
applied 2 migration(s)
exit code 0
== application pod (APP_MODE=serve) after the Job
web server started, accepting traffic
(still running: stopped by the demo)
- Before the Job ran, the application pod refused to start.
reported both scripts as not applied, and the JVM exited with code 1. On Kubernetes, it would restart until the Job had run, while the previous version kept serving.Flyway.validate() - In
mode, the same image applied two scripts, started no web server, and exited with code 0. Exit code 0 tells Kubernetes that the Job or init container succeeded.migrate - After the Job, a new application pod started its web server and took traffic. Validation does not block a rollback: an older release finds applied migrations it does not know, and Flyway ignores those by default.
A Job that runs once per release (not run here, validated only):
apiVersion: batch/v1
kind: Job
metadata:
name: orders-migrate-1-5-0 # a new name per release: a Job's pod template is immutable
spec:
backoffLimit: 2 # up to 2 retries after a failed pod
activeDeadlineSeconds: 900 # fail the release rather than wait forever
ttlSecondsAfterFinished: 86400 # delete the finished Job, and its pod's logs, after a day
template:
spec:
restartPolicy: Never
containers:
- name: migrate
image: registry.example.com/orders:1.5.0
env:
- name: APP_MODE
value: migrate
- name: SPRING_DATASOURCE_URL
value: jdbc:postgresql://postgres:5432/orders
- name: SPRING_DATASOURCE_USERNAME
valueFrom:
secretKeyRef:
name: orders-db
key: username
- name: SPRING_DATASOURCE_PASSWORD
valueFrom:
secretKeyRef:
name: orders-db
key: password
The pipeline applies the Job, waits with kubectl wait --for=condition=complete --timeout=15m job/orders-migrate-1-5-0--timeoutkubectl waitcondition=failedhelm upgrade --timeouthelm.sh/hook-delete-policybefore-hook-creation
A Job can still run a migration twice or stop it midway:
- Two runs: the Job documentation warns that a Job may occasionally start its pod twice, and
retries failures; the migration lock keeps those runs apart.backoffLimit - Killed midway: when
fires or a node is drained, the migration stops halfway, so set the deadline well above the slowest migration. A pod lost to a drain also counts againstactiveDeadlineSeconds , unless abackoffLimit rule withpodFailurePolicy matches theaction: Ignore condition (stable since Kubernetes 1.31).DisruptionTarget - Cleaning up: PostgreSQL rolls back the open transaction, so a rerun of a transactional script starts clean. But the killed backend notices the lost connection only when its current statement ends (sooner with
, PostgreSQL 14 and later) and holds Flyway's lock until then, and a script outside a transaction, such asclient_connection_check_interval , can leave partial work. MySQL commits each DDL statement as it runs, so the rerun fails until someone removes the half-applied changes by hand and, if Flyway recorded a failure, runsCREATE INDEX CONCURRENTLY .flyway repair - Service mesh: an injected Istio or Linkerd sidecar keeps running after the migration exits, so the Job never completes. Use the mesh's native-sidecar mode (Kubernetes 1.29 and later) or turn off injection for the Job.
An init container runs the same migrate container before the application container in every pod (not run here, validated only):
# part of the Deployment's pod template
spec:
initContainers:
- name: migrate
image: registry.example.com/orders:1.5.0
env:
- name: APP_MODE
value: migrate
- name: SPRING_DATASOURCE_URL
value: jdbc:postgresql://postgres:5432/orders
# plus the migration credentials, as in the Job
containers:
- name: app
image: registry.example.com/orders:1.5.0
env:
- name: APP_MODE
value: serve
- name: SPRING_DATASOURCE_URL
value: jdbc:postgresql://postgres:5432/orders
# plus the application's own credentials
The init container needs no pipeline step, but it runs in every new pod, so the replicas compete for the lock as before. On PostgreSQL, raise spring.flyway.lock-retry-countInit:CrashLoopBackOffprogressDeadlineSeconds
The rolling-update trap: old pods on the new schema
Every option above has a window in which the old code runs against the new schema. With a Job or hook, the window is the whole rollout: the schema changes first, then pods are replaced a few at a time. With migration at startup, it opens when the first new pod migrates. A column rename is the classic change that breaks in that window, so this program renames namefull_name
-- db/rename/V2__rename_name_to_full_name.sql
ALTER TABLE customer RENAME COLUMN name TO full_name;
package demo.migrations.rename;
import ch.qos.logback.classic.Level;
import java.util.List;
import org.flywaydb.core.Flyway;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.dao.DataAccessException;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.jdbc.datasource.DriverManagerDataSource;
public class BreakingRename {
static final String URL = "jdbc:h2:mem:rename;DB_CLOSE_DELAY=-1";
static final JdbcTemplate JDBC = new JdbcTemplate(new DriverManagerDataSource(URL));
// the data access code of each application version
interface CustomerCode {
void save(long id, String name);
List<String> names();
}
static class Version1 implements CustomerCode {
public void save(long id, String name) {
JDBC.update("INSERT INTO customer (id, name) VALUES (?, ?)", id, name);
}
public List<String> names() {
return JDBC.queryForList("SELECT name FROM customer ORDER BY id", String.class);
}
}
static class Version2 implements CustomerCode {
public void save(long id, String name) {
JDBC.update("INSERT INTO customer (id, full_name) VALUES (?, ?)", id, name);
}
public List<String> names() {
return JDBC.queryForList("SELECT full_name FROM customer ORDER BY id", String.class);
}
}
static void migrate(String target) {
int applied = Flyway.configure().dataSource(URL, "", "")
.locations("classpath:db/rename").target(target).load()
.migrate().migrationsExecuted;
System.out.println("Flyway: applied " + applied + ", schema now at V" + target);
}
static void use(String pod, CustomerCode code, long id, String name) {
try {
code.save(id, name);
System.out.println(" " + pod + ": saved " + name + ", reads " + code.names());
} catch (DataAccessException e) {
String m = e.getMostSpecificCause().getMessage();
System.out.println(" " + pod + ": FAILED " + m.substring(0, m.indexOf(';')));
}
}
public static void main(String[] args) {
// without Spring Boot, Logback logs everything at DEBUG; keep only errors
// Logback's Logger is named in full: org.slf4j.Logger has no setLevel
((ch.qos.logback.classic.Logger) LoggerFactory.getLogger(Logger.ROOT_LOGGER_NAME))
.setLevel(Level.ERROR);
System.out.println("== release 1 is running");
migrate("1");
use("v1 pod", new Version1(), 1, "Ada");
System.out.println("== rolling update to release 2: the first new pod migrates");
migrate("2");
use("v2 pod", new Version2(), 2, "Grace");
use("v1 pod", new Version1(), 3, "Linus"); // old pods still take traffic
System.out.println("== release 2 fails, so the Deployment is rolled back to release 1");
use("v1 pod", new Version1(), 4, "Barbara");
}
}
== release 1 is running
Flyway: applied 1, schema now at V1
v1 pod: saved Ada, reads [Ada]
== rolling update to release 2: the first new pod migrates
Flyway: applied 1, schema now at V2
v2 pod: saved Grace, reads [Ada, Grace]
v1 pod: FAILED Column "NAME" not found
== release 2 fails, so the Deployment is rolled back to release 1
v1 pod: FAILED Column "NAME" not found
- The release-2 code worked at once and read Ada, saved by release 1, under the new column name.
- The release-1 pods, still serving part of the traffic, failed every query that mentions
, so until the rollout finishes, every request that reaches an old pod fails.name - The rollback made it worse: Flyway does not undo migrations, so the restored release-1 code failed on every request, not just those on old pods.
The Recreate
Expand and contract
The expand/contract pattern, also called parallel change, splits a breaking change into steps that each keep the previous release working. The same rename takes three steps, one per release: expand (release 2) adds full_namefull_namename
| Release | Schema after its migration | Code writes | Code reads | Safe rollback to |
|---|---|---|---|---|
| 1 | name | name | name | Not applicable |
| 2 (expand) | namefull_namename | Both | COALESCE(full_name, name) | 1 |
| 3 (switch) | full_namename | full_name | full_name | 2 (not 1) |
| 4 (contract) | full_name | full_name | full_name | 3 |
-- db/expand/V2__expand_add_full_name.sql
-- release 2: add the new column next to the old one; nothing is removed
ALTER TABLE customer ADD COLUMN full_name VARCHAR(100);
UPDATE customer SET full_name = name;
-- db/expand/V3__backfill_and_relax_name.sql
-- release 3: release 2 is fully rolled out, so every running pod writes both columns;
-- copy every row that release-1 pods inserted or updated during that rollout
UPDATE customer SET full_name = name WHERE full_name IS DISTINCT FROM name;
ALTER TABLE customer ALTER COLUMN full_name SET NOT NULL;
ALTER TABLE customer ALTER COLUMN name DROP NOT NULL;
-- db/expand/V4__contract_drop_name.sql
-- release 4: no running code reads or writes name any more
ALTER TABLE customer DROP COLUMN name;
The scripts use PostgreSQL and H2 syntax. On MySQL, V3 compares with WHERE NOT (full_name <=> name)ALTER TABLE customer MODIFY full_name VARCHAR(100) NOT NULL, MODIFY name VARCHAR(100) NULL
package demo.migrations.expand;
import ch.qos.logback.classic.Level;
import java.util.List;
import org.flywaydb.core.Flyway;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.dao.DataAccessException;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.jdbc.datasource.DriverManagerDataSource;
public class ExpandContract {
static final String URL = "jdbc:h2:mem:expand;DB_CLOSE_DELAY=-1";
static final JdbcTemplate JDBC = new JdbcTemplate(new DriverManagerDataSource(URL));
// the data access code of each application version
interface CustomerCode {
void save(long id, String name);
void rename(long id, String name);
List<String> names();
}
// release 1: knows only name
static class Version1 implements CustomerCode {
public void save(long id, String name) {
JDBC.update("INSERT INTO customer (id, name) VALUES (?, ?)", id, name);
}
public void rename(long id, String name) {
JDBC.update("UPDATE customer SET name = ? WHERE id = ?", name, id);
}
public List<String> names() {
return JDBC.queryForList("SELECT name FROM customer ORDER BY id", String.class);
}
}
// release 2: writes both columns, reads the new one with a fallback to the old one
static class Version2 implements CustomerCode {
public void save(long id, String name) {
JDBC.update("INSERT INTO customer (id, name, full_name) VALUES (?, ?, ?)",
id, name, name);
}
public void rename(long id, String name) {
JDBC.update("UPDATE customer SET name = ?, full_name = ? WHERE id = ?", name, name, id);
}
public List<String> names() {
return JDBC.queryForList(
"SELECT COALESCE(full_name, name) FROM customer ORDER BY id", String.class);
}
}
// release 3 and later: knows only full_name
static class Version3 implements CustomerCode {
public void save(long id, String name) {
JDBC.update("INSERT INTO customer (id, full_name) VALUES (?, ?)", id, name);
}
public void rename(long id, String name) {
JDBC.update("UPDATE customer SET full_name = ? WHERE id = ?", name, id);
}
public List<String> names() {
return JDBC.queryForList("SELECT full_name FROM customer ORDER BY id", String.class);
}
}
static long nextId = 1;
static void migrate(String target) {
Flyway.configure().dataSource(URL, "", "").locations("classpath:db/expand")
.target(target).load().migrate();
System.out.println(" Flyway: schema now at V" + target);
}
static void use(String pod, CustomerCode code, String name) {
try {
code.save(nextId++, name);
System.out.println(" " + pod + ": saved " + name + ", reads " + code.names());
} catch (DataAccessException e) {
String m = e.getMostSpecificCause().getMessage();
System.out.println(" " + pod + ": FAILED " + m.substring(0, m.indexOf(';')));
}
}
static void printStaleCount() {
System.out.println(" rows where full_name differs from name: " + JDBC.queryForObject(
"SELECT COUNT(*) FROM customer WHERE full_name IS DISTINCT FROM name",
Integer.class));
}
public static void main(String[] args) {
// Logback's Logger is named in full: org.slf4j.Logger has no setLevel
((ch.qos.logback.classic.Logger) LoggerFactory.getLogger(Logger.ROOT_LOGGER_NAME))
.setLevel(Level.ERROR);
System.out.println("== release 1");
migrate("1");
use("v1 pod", new Version1(), "Ada");
System.out.println("== release 2 (expand), old and new pods side by side");
migrate("2");
use("v1 pod", new Version1(), "Grace");
new Version1().rename(1, "Ada Lovelace"); // release-1 code updates name only
System.out.println(" v1 pod: renamed 1 to Ada Lovelace, reads " + new Version1().names());
use("v2 pod", new Version2(), "Linus");
printStaleCount();
System.out.println("== release 3 (switch to the new column), old and new pods");
migrate("3");
use("v2 pod", new Version2(), "Barbara");
use("v3 pod", new Version3(), "Edsger");
use("v1 pod", new Version1(), "Alan"); // a rollback two releases back
System.out.println("== release 4 (contract), only v3 code is running");
migrate("4");
use("v3 pod", new Version3(), "Margaret");
use("v2 pod", new Version2(), "Ken"); // a rollback two releases back
}
}
== release 1
Flyway: schema now at V1
v1 pod: saved Ada, reads [Ada]
== release 2 (expand), old and new pods side by side
Flyway: schema now at V2
v1 pod: saved Grace, reads [Ada, Grace]
v1 pod: renamed 1 to Ada Lovelace, reads [Ada Lovelace, Grace]
v2 pod: saved Linus, reads [Ada, Grace, Linus]
rows where full_name differs from name: 2
== release 3 (switch to the new column), old and new pods
Flyway: schema now at V3
v2 pod: saved Barbara, reads [Ada Lovelace, Grace, Linus, Barbara]
v3 pod: saved Edsger, reads [Ada Lovelace, Grace, Linus, Barbara, Edsger]
v1 pod: FAILED NULL not allowed for column "FULL_NAME"
== release 4 (contract), only v3 code is running
Flyway: schema now at V4
v3 pod: saved Margaret, reads [Ada Lovelace, Grace, Linus, Barbara, Edsger, Margaret]
v2 pod: FAILED Column "NAME" not found
- Release 2: V2 added
, copied fromfull_name . The release-1 pod saved Grace intoname only, and the release-2 code showed her throughname . It also renamed Ada inCOALESCE only, so release 2 kept showing her old name: until V3 runs, release 2 shows old values for rows that release 1 updates. The double write, whichname 's NOT NULL also demands, keeps a rollback to release 1 safe: every row still has aname .name - Release 3: V3 copied
intoname wherever the two differ, which fixed Grace and Ada Lovelace, then madefull_name mandatory andfull_name optional. No release-3 code has written yet when V3 runs, so where the columns differ,name is newer. A V3 that copied onlyname kept "Ada" in a run of the same program, and V4 then dropped the only copy of "Ada Lovelace". Release-2 pods worked unchanged, so a rollback to release 2 is safe; release-1 code failed, because its inserts omitWHERE full_name IS NULL , now NOT NULL.full_name - Release 4: V4 dropped
. Release-3 code never uses it; release-2 code failed. Each step keeps only the previous release working, so run the contract step only when nobody will roll back to a release that usesname .name
Each release is still an ordinary rolling update with one migration step, and the two versions that can run together always agree on the schema. The price is three releases instead of one, and code that writes two columns for a while.
These scripts are fine for small tables. On a large PostgreSQL table, V2 holds an ACCESS EXCLUSIVEUPDATESET NOT NULLSET NOT NULLCHECK (full_name IS NOT NULL) NOT VALIDVALIDATE CONSTRAINTSET NOT NULLCHECK
Which changes are safe in one step
The test for every migration: can the previous release, unchanged, work against the schema this script leaves?
| Change | Previous release keeps working? | Safe way |
|---|---|---|
| Add a table | Yes | One step |
| Add a nullable column, or one with a default | Yes, if old code names its columns in INSERTSELECT * | One step; with a default, instant on PostgreSQL 11 and later for a non-volatile default, such as a constant, and on MySQL 8.0.12 and later (last column only before 8.0.29); otherwise the table is rebuilt |
| Add an index | Yes, but on PostgreSQL a plain index build blocks writes until it finishes | CREATE INDEX CONCURRENTLYspring.flyway.postgresql.transactional-lock=falserunInTransaction: falseINVALIDIF NOT EXISTS |
Add a NOT NULL | No: old code's inserts fail | Add it nullable, backfill, then add the constraint in a later release |
| Rename a column or table | No | Expand/contract, as above |
| Change a column's type | Widening, yes; narrowing or a new meaning, no | Widening a VARCHARINTBIGINT |
| Drop a column or table | No, if the previous release uses it | Stop using it in one release (for a column, that release's migration also makes it nullable or gives it a default); drop it in a later release |
Add a constraint, such as a foreign key or CHECK | Only if old code never writes violating rows | First stop the code from writing violating rows; on PostgreSQL, add the constraint NOT VALIDVALIDATE CONSTRAINT |
Even an instant change needs a lock, and waiting for that lock can itself cause the outage. On PostgreSQL, most ALTER TABLEACCESS EXCLUSIVEALTERSET LOCAL lock_timeout = '5s'SETspring.flyway.urluserALTERALGORITHM=INPLACELOCK=NONElock_wait_timeout
The Hibernate setting spring.jpa.hibernate.ddl-autovalidatenoneupdateINSERTSELECT@Column(name = "name")
Which to use
| Situation | Recommendation |
|---|---|
| Small service, fast migrations, few replicas | Migrate at startup; size the startup probe (and, on PostgreSQL, spring.flyway.lock-retry-count |
| Long data migrations, or many replicas | A Job per release, or a Helm pre-upgrade--timeout |
| App database user should not change the schema | A Job or hook with its own credentials. On PostgreSQL, with the Job as migratorappALTER DEFAULT PRIVILEGES FOR ROLE migrator GRANT SELECT, INSERT, UPDATE, DELETE ON TABLES TO app... GRANT USAGE, SELECT ON SEQUENCES TO appGRANT ... ON ALL TABLESSEQUENCESIN SCHEMA |
| No pipeline step for migrations, and migrations longer than the startup probe allows | Init container in migrate mode, with spring.flyway.lock-retry-countprogressDeadlineSeconds |
| Deploying with Argo CD or Flux | A Job annotated argocd.argoproj.io/hook: PreSyncBeforeHookCreationpre-installpre-upgradePreSync |
| Several services share one database | A schema per service (spring.flyway.default-schemaspring.flyway.tablespring.flyway.baseline-on-migrate=truespring.flyway.baseline-version=0 |
| Liquibase on any of the above | The liquibase-sessionlockrelease-locks |
Rename, type change, drop, new NOT NULL | Expand/contract across releases |
| Rollbacks must keep working | Make each migration compatible with the previous release. A Kubernetes rollback never runs undo scripts (Urollback |
Related pages: the