Liquibase Tutorial
Part 21 of 24
Abhay 8 min read

Kubernetes Deployments: Jobs, Init Containers, and Helm Hooks

Spring Boot’s Liquibase auto-run works fine for a single instance. In Kubernetes, where multiple pods start simultaneously, auto-run at application startup creates a race: every pod acquires DATABASECHANGELOGLOCK, one holds it, the rest wait, and if Kubernetes kills a pod mid-migration (because it failed its readiness probe while waiting on the lock), the lock remains set and blocks every subsequent pod.

The solution is to run migrations exactly once, before application pods start, using either an init container or a Helm pre-upgrade Job.

The Problem with Auto-Run in Kubernetes

When a Kubernetes Deployment scales from 0 to 3 replicas, all three pods start simultaneously. Each Spring Boot pod tries to run Liquibase:

Pod A: Acquires DATABASECHANGELOGLOCK → running migrations...
Pod B: Waiting for lock (LOCKED=1)...
Pod C: Waiting for lock (LOCKED=1)...

If Pod A takes more than the readiness probe timeout to complete migrations:

Kubernetes: Pod A failed readiness probe → killing Pod A
Pod A killed mid-migration → LOCKED=1 left in DATABASECHANGELOGLOCK
Pod B: Still waiting for lock...
Pod C: Still waiting for lock...

Now all pods are stuck waiting for a lock held by a dead process. The deployment hangs until someone manually runs liquibase releaseLocks.

The fix: Run Liquibase once, as a separate process, before application pods start. When application pods start, spring.liquibase.enabled=false — they connect to an already-migrated database.

A Kubernetes Job runs migration to completion before Helm proceeds with the deployment. The helm.sh/hook: pre-install,pre-upgrade annotation tells Helm to create and complete this Job before updating the application Deployment.

Helm chart structure

charts/ecommerce/
├── Chart.yaml
├── values.yaml
├── templates/
│   ├── deployment.yaml
│   ├── service.yaml
│   ├── configmap-changelog.yaml    ← Liquibase changelogs as ConfigMap
│   └── job-migration.yaml          ← Migration Job with Helm hook

templates/configmap-changelog.yaml

For small changelog sets, store them in a ConfigMap (mounted into the migration Job):

apiVersion: v1
kind: ConfigMap
metadata:
  name: {{ include "ecommerce.fullname" . }}-changelog
  labels:
    {{- include "ecommerce.labels" . | nindent 4 }}
data:
  db.changelog-master.yaml: |
    databaseChangeLog:
      - includeAll:
          path: /liquibase/changelog/migrations/
          relativeToChangelogFile: false

For larger changelog sets, package the changelogs into the application Docker image and reference them from there (shown in the Job spec below).

templates/job-migration.yaml

apiVersion: batch/v1
kind: Job
metadata:
  name: {{ include "ecommerce.fullname" . }}-db-migration-{{ .Release.Revision }}
  labels:
    {{- include "ecommerce.labels" . | nindent 4 }}
  annotations:
    # Run before install and upgrade; delete Job after completion
    "helm.sh/hook": pre-install,pre-upgrade
    "helm.sh/hook-weight": "-5"
    "helm.sh/hook-delete-policy": hook-succeeded
spec:
  backoffLimit: 3          # Retry up to 3 times on failure
  activeDeadlineSeconds: 600   # Fail the hook after 10 minutes
  template:
    metadata:
      labels:
        app: {{ include "ecommerce.fullname" . }}-migration
    spec:
      restartPolicy: OnFailure
      containers:
        - name: liquibase
          image: liquibase/liquibase:4.31.1
          args:
            - "--changelog-file=/liquibase/changelog/db.changelog-master.yaml"
            - "--url=$(DB_URL)"
            - "--username=$(DB_MIGRATION_USER)"
            - "--password=$(DB_MIGRATION_PASSWORD)"
            - "--contexts=$(LIQUIBASE_CONTEXTS)"
            - "update"
          env:
            - name: DB_URL
              valueFrom:
                secretKeyRef:
                  name: {{ include "ecommerce.fullname" . }}-db-secret
                  key: migration-url
            - name: DB_MIGRATION_USER
              valueFrom:
                secretKeyRef:
                  name: {{ include "ecommerce.fullname" . }}-db-secret
                  key: migration-username
            - name: DB_MIGRATION_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: {{ include "ecommerce.fullname" . }}-db-secret
                  key: migration-password
            - name: LIQUIBASE_CONTEXTS
              value: {{ .Values.liquibase.contexts | default "default" | quote }}
          volumeMounts:
            - name: changelog
              mountPath: /liquibase/changelog
          resources:
            requests:
              cpu: 100m
              memory: 256Mi
            limits:
              cpu: 500m
              memory: 512Mi
      volumes:
        - name: changelog
          # Mount changelogs from the application image via an init container
          # OR use a ConfigMap for small changelog sets
          emptyDir: {}
      initContainers:
        - name: copy-changelog
          image: {{ .Values.image.repository }}:{{ .Values.image.tag }}
          command: ["/bin/sh", "-c", "cp -r /app/resources/db/changelog/* /changelog/"]
          volumeMounts:
            - name: changelog
              mountPath: /changelog

values.yaml

image:
  repository: your-registry/ecommerce-api
  tag: latest

liquibase:
  contexts: default

db:
  secretName: ecommerce-db-secret

Secret (applied separately, not in Helm chart):

kubectl create secret generic ecommerce-db-secret \
  --from-literal=migration-url='jdbc:mysql://mysql-host:3306/ecommerce?useSSL=false&allowPublicKeyRetrieval=true&serverTimezone=UTC' \
  --from-literal=migration-username='lb_migration_user' \
  --from-literal=migration-password='supersecret'

Deployment with Liquibase disabled:

# templates/deployment.yaml
env:
  - name: SPRING_LIQUIBASE_ENABLED
    value: "false"   # Migration runs in the Job; app pods skip it

Approach 2: Init Container

An init container runs inside each pod before the main container starts. Migrations are guaranteed to complete before the application container starts for that specific pod.

# templates/deployment.yaml (relevant section)
spec:
  initContainers:
    - name: liquibase-migration
      image: liquibase/liquibase:4.31.1
      args:
        - "--changelog-file=/changelog/db.changelog-master.yaml"
        - "--url=$(DB_URL)"
        - "--username=$(DB_MIGRATION_USER)"
        - "--password=$(DB_MIGRATION_PASSWORD)"
        - "update"
      env:
        - name: DB_URL
          valueFrom:
            secretKeyRef:
              name: ecommerce-db-secret
              key: migration-url
        - name: DB_MIGRATION_USER
          valueFrom:
            secretKeyRef:
              name: ecommerce-db-secret
              key: migration-username
        - name: DB_MIGRATION_PASSWORD
          valueFrom:
            secretKeyRef:
              name: ecommerce-db-secret
              key: migration-password
      volumeMounts:
        - name: changelog
          mountPath: /changelog
  containers:
    - name: ecommerce-api
      image: your-registry/ecommerce-api:latest
      env:
        - name: SPRING_LIQUIBASE_ENABLED
          value: "false"

Init Container vs Helm Job: Choosing the Right Approach

ConsiderationInit ContainerHelm Job
Runs on every pod startYes — every new/restarted podNo — once per Helm release
Multiple pods starting simultaneouslyEach runs update (idempotent, lock-protected)Only one Job runs
Migration failsPod stays in Init:Error, app pod doesn’t startHelm deploy fails, existing pods unchanged
Lock stuck from previous pod killNext pod waits for lock timeoutJob retry handles it
Rollback possible via HelmNo — migrations already ranNo — migrations already ran

Use Helm Job (pre-upgrade hook) when:

  • You want migrations to run exactly once per deployment, not per pod
  • Your deployment scales to many replicas (N init containers all acquiring the lock is wasteful)
  • You want to decouple migration completion from pod health

Use init container when:

  • You don’t use Helm
  • Your deployment is single-replica
  • You want the guarantee that each pod individually verified migrations before starting

Handling DATABASECHANGELOGLOCK in Kubernetes

The biggest operational issue with Liquibase in Kubernetes is a stuck lock from a killed pod. Options for handling it:

Option 1: Configure lock wait time

# In application.yml or passed to liquibase CLI
spring:
  liquibase:
    parameters:
      liquibase.lock-wait-time-in-minutes: 2    # Default is 5 minutes

Or via Liquibase CLI:

liquibase --changelog-lock-wait-time-in-minutes=2 update

After 2 minutes of waiting for the lock, Liquibase fails the Job. Helm marks the hook as failed. The deployment does not proceed. Kubernetes restarts the Job (up to backoffLimit times).

Option 2: Release locks before migration (risky if another process is running)

In the Helm Job, add a releaseLocks step before update:

args:
  - /bin/sh
  - -c
  - |
    liquibase --url="${DB_URL}" --username="${DB_MIGRATION_USER}" --password="${DB_MIGRATION_PASSWORD}" \
      --changelog-file=/changelog/db.changelog-master.yaml \
      releaseLocks
    liquibase --url="${DB_URL}" --username="${DB_MIGRATION_USER}" --password="${DB_MIGRATION_PASSWORD}" \
      --changelog-file=/changelog/db.changelog-master.yaml \
      update

Only use this if you are certain no other Liquibase process is actively holding the lock — which is guaranteed with a single Helm Job.

Option 3: Kubernetes liveness probe timeout

Size your Job’s activeDeadlineSeconds to be longer than any expected migration. If migration takes more than activeDeadlineSeconds, Kubernetes kills the Job and Helm fails the hook.

Disabling Spring Boot Liquibase Auto-Run

When migrations run in a separate Job or init container, the application must not also run Liquibase at startup:

# values.yaml
env:
  - name: SPRING_LIQUIBASE_ENABLED
    value: "false"

Or in application-kubernetes.yml:

spring:
  liquibase:
    enabled: false

Activate via profile:

# In the Deployment
env:
  - name: SPRING_PROFILES_ACTIVE
    value: kubernetes

Docker Image for Liquibase CLI

Use the official Liquibase Docker image in Jobs and init containers:

liquibase/liquibase:4.31.1

If you need the MySQL JDBC driver bundled (the official image includes common drivers), verify it:

docker run --rm liquibase/liquibase:4.31.1 \
  --url=jdbc:mysql://host:3306/db \
  --username=user --password=pass \
  validate

If the driver is missing, build a custom image:

FROM liquibase/liquibase:4.31.1
COPY mysql-connector-j-*.jar /liquibase/lib/

Complete Helm Deployment Flow

# 1. Build and push application image (includes changelogs)
docker build -t your-registry/ecommerce-api:v1.2.0 .
docker push your-registry/ecommerce-api:v1.2.0

# 2. Update values.yaml image tag
# image.tag: v1.2.0

# 3. Helm upgrade (triggers pre-upgrade Job automatically)
helm upgrade --install ecommerce ./charts/ecommerce \
  --namespace ecommerce \
  --set image.tag=v1.2.0 \
  --set liquibase.contexts=prod,default \
  --wait \             # Wait for all resources including hooks to complete
  --timeout 10m

# What happens:
# Step 1: Helm runs the pre-upgrade Job (migration)
# Step 2: Job runs: validate → update
# Step 3: Job completes successfully
# Step 4: Helm updates the Deployment (with spring.liquibase.enabled=false)
# Step 5: New pods start (migration already done)

If the migration Job fails, helm upgrade exits with an error. The existing Deployment is unchanged — the old pods continue serving traffic. No partial deployment.

Common Mistakes

Leaving spring.liquibase.enabled=true alongside the init container: Both the init container and each Spring Boot pod try to run migrations. The init container holds the lock while running; the application container waits for it to release — which defeats the purpose of the init container. Always disable Spring Boot auto-run when using external migration.

Not setting backoffLimit on the migration Job: A migration that fails once (e.g., transient network issue connecting to MySQL) would mark the entire Helm hook as failed with no retry. Set backoffLimit: 3 to allow retries.

Using the same database user for migration and runtime: The migration Job needs DDL privileges (CREATE, ALTER, DROP). The application pods need only DML (SELECT, INSERT, UPDATE, DELETE). Use separate secrets for DB_MIGRATION_USER and DB_RUNTIME_USER.

Best Practices

  • Helm pre-upgrade Job over init containers for multi-replica deployments — runs once, not per pod
  • --wait in helm upgrade — ensures the migration Job completes before Helm returns success
  • activeDeadlineSeconds on the Job — prevents a stuck migration from blocking the deployment indefinitely
  • Separate migration user from runtime user — scope credentials to the minimum required for each role
  • SPRING_LIQUIBASE_ENABLED=false in application pods — migrations run in the Job; the app must not re-run them
  • Package changelogs in the application image — ensures the migration Job and application pods always use the same changelog version

What You’ve Learned

  • Spring Boot auto-run in Kubernetes causes lock contention when multiple pods start simultaneously
  • Helm pre-upgrade Job runs migrations once before the Deployment is updated — the cleanest Kubernetes pattern
  • Init containers run migrations per pod — simpler but wasteful for multi-replica deployments
  • SPRING_LIQUIBASE_ENABLED=false disables Spring auto-run when an external migration process is used
  • backoffLimit and activeDeadlineSeconds on the Job control retry and timeout behaviour
  • The migration user needs DDL privileges; the runtime application user needs only DML

Next: Article 22 — Adopting Liquibase on an Existing Production Database — the full step-by-step process for onboarding a database that has been running in production without version control.