Kubernetes Upgrade Runbook: Plan, Test, and Safely Upgrade Production Clusters

Overview

Begin with the exact target version, the current cluster version, and the infrastructure model you operate. Managed Kubernetes services, self-managed clusters, and distributions can use different upgrade commands, support windows, node-image processes, and rollback options. Treat the provider or distribution documentation as the authority for the final procedure.

The safest production Kubernetes upgrade is normally a staged change:

1. Inventory the cluster and define the target version.
2. Review version-skew rules and deprecated or removed APIs.
3. Test applications and add-ons against the target version.
4. Validate backups and document recovery steps.
5. Upgrade a non-production environment, then a representative production cluster.
6. Upgrade nodes through replacement or a provider-supported rolling process.
7. Verify control-plane health, workloads, networking, storage, security, and observability.

Plan rollback before starting, but do not assume that downgrading a cluster is a supported or safe recovery method. Database migrations, persisted resources, API conversion, and node-image changes can make a downgrade impractical. A tested restore, redeployment, or failover procedure is often more reliable than an attempted version reversal.

Checklist by scenario

1. Before any upgrade

• Record the baseline: capture Kubernetes versions, node pools, operating system and image versions, critical add-ons, admission webhooks, storage drivers, ingress components, autoscalers, policy engines, and observability agents.
• Confirm the target: write down the current version, target version, supported upgrade path, maintenance window, owners, communication plan, and success criteria.
• Check version skew: review the Kubernetes version-skew policy and your provider's rules for control-plane components, kubelets, kubectl, controllers, and clients. Do not infer compatibility from a previous upgrade.
• Scan for deprecated APIs: inspect manifests, Helm charts, operators, Git repositories, and live objects for APIs removed or changed in the target version. Verify both the API version in source files and the version actually stored in the cluster.
• Validate backups: confirm that cluster-state backups complete successfully, persistent-volume data is protected, and the team has performed a restore test or has a documented recovery path. A backup that has never been restored is an assumption, not evidence.
• Freeze unrelated changes: limit application releases, infrastructure modifications, and policy changes during the upgrade window so that failures remain diagnosable.

2. When preparing a staging cluster

• Use representative workloads, node sizes, storage classes, network policies, ingress routes, secrets management, and admission policies.
• Install the same major add-ons used in production, including the cloud controller, CSI drivers, DNS components, metrics and tracing agents, ingress controllers, policy tools, and GitOps controllers.
• Deploy workloads from the same release artifacts that production will use. A staging cluster running different chart versions does not provide a meaningful upgrade rehearsal.
• Exercise pod scheduling, rolling updates, disruption budgets, autoscaling, service discovery, ingress, persistent-volume attach and mount operations, job execution, and node drains.
• Record warnings, admission failures, pending pods, restart increases, latency changes, and reconciliation errors before and after the upgrade.

3. During the control-plane upgrade

• Confirm that monitoring, incident ownership, and provider activity logs are visible before making the change.
• Follow the provider or distribution's documented sequence rather than combining independent control-plane changes into one untested operation.
• Watch API-server availability, authentication, authorization, admission latency, controller errors, scheduler activity, and cluster events.
• Test read and write operations for representative resources. A responsive command-line client alone does not prove that admission, controllers, and custom resources are healthy.
• Pause if the documented health criteria fail. Escalate using the prepared recovery plan instead of repeatedly retrying an unknown state.

4. During node-pool replacement

• Confirm capacity for surge nodes, including CPU, memory, IP addresses, quotas, storage attachments, and disruption budgets.
• Review taints, labels, affinity rules, topology constraints, and workload tolerations before introducing the new pool.
• Drain nodes using the supported process and observe evictions. Pay particular attention to single-replica workloads, strict disruption budgets, local storage, and pods that cannot be rescheduled.
• Check that the new node image contains compatible kernel, runtime, networking, storage, and security-agent components.
• Keep the old pool available until workloads have been verified and the recovery decision is clear. Remove it only after the defined observation period.

5. For a GitOps-managed cluster

• Review the desired-state repository for deprecated manifests, chart dependencies, CRDs, and operator versions.
• Prevent the reconciler from masking upgrade symptoms. Know whether it will immediately recreate a resource, revert a manual change, or apply an incompatible manifest.
• Check application health and synchronization after the upgrade. The Argo CD application health checklist can help structure this verification.
• Use a staged pull request or environment promotion process so that the cluster version change and application changes remain traceable.

Version-compatibility reference

Use this table as a review prompt, not as a substitute for release-specific documentation. Compatibility depends on the Kubernetes version, provider, distribution, component release, and configuration.

What to double-check

Deprecated APIs and custom resources

Do not search only the main application repository. Deprecated objects may live in Helm dependencies, operators, admission configurations, monitoring rules, generated manifests, or abandoned namespaces. Confirm that CRDs are upgraded in the order required by their operator and that stored versions can be served or converted by the target release.

Admission webhooks

A webhook that cannot reach its service, uses an invalid certificate, or rejects an unfamiliar API request can block deployments and routine cluster operations. Test both expected and failure paths. Review namespace selectors, object selectors, timeout settings, and failure behavior before the maintenance window.

Workload disruption

Pod disruption budgets improve safety only when they match the actual redundancy of the workload. Check replicas, topology spread, readiness probes, termination behavior, graceful shutdown, and capacity headroom. A workload with one replica and a strict disruption budget may prevent draining; a workload with weak readiness checks may accept traffic before it is ready.

Observability and recovery

Define what “healthy” means before the upgrade: error rate, latency, saturation, restart rate, queue depth, reconciliation status, and business-level checks. Keep an incident runbook available; the SRE runbook template provides a useful structure for ownership, signals, commands, and escalation.

Common mistakes

• Upgrading directly in production: rehearse the exact topology and add-ons in a representative environment first.
• Checking only the control plane: node images, storage, ingress, policy, observability, and workload behavior are part of the upgrade.
• Trusting a successful command: a completed provider operation does not prove that applications can serve traffic or persist data.
• Ignoring capacity: rolling node replacement needs room for rescheduled pods and may consume cloud quotas or IP space.
• Assuming rollback is one command: define restore, failover, redeployment, and communication actions before the change.
• Mixing maintenance with feature work: separate version upgrades from application migrations, policy rewrites, and unrelated infrastructure changes whenever possible.

When to revisit

Review this runbook before every Kubernetes minor-version upgrade, node-image refresh, managed-cluster maintenance event, platform migration, and quarterly disaster-recovery exercise. Recheck it whenever Kubernetes removes or changes an API, your cloud provider changes upgrade procedures, or an add-on publishes a compatibility requirement.

At the end of each upgrade, record the actual duration, alerts, blocked evictions, failed tests, manual workarounds, and follow-up actions. Update the checklist while the details are fresh. Before the next window, confirm the target version and all compatibility links again, run the preflight checks, announce the maintenance plan, and assign one person to make the go or no-go decision.

For related preparation, compare the cluster's deployment practices with these Kubernetes deployment best practices, and review supply-chain controls in the Cosign image-signing workflow. A repeatable upgrade is the result of tested procedures, current compatibility records, observable workloads, and a recovery plan—not confidence in a single command.