A Kubernetes cluster upgrade is a controlled change to the control plane, node pools, APIs, admission policies, and supporting tools. This runbook gives platform teams a reusable Kubernetes upgrade checklist: inventory dependencies, test workloads, upgrade in a safe sequence, verify behavior, and choose a recovery path when validation fails.
Overview
Upgrade planning should begin with the target version and the exact procedure supported by your Kubernetes distribution or managed-cloud provider. Do not treat a version change as only a node-image refresh. The change can affect API availability, scheduling, networking, storage, admission controllers, observability agents, autoscalers, Helm charts, and CI/CD automation.
Use this compatibility model as a planning reference rather than a substitute for provider documentation:
| Area | What to confirm | Evidence to retain |
|---|---|---|
| Control plane | Supported source and target versions, upgrade order, maintenance behavior, and rollback limitations | Approved change record and provider procedure |
| Node pools | Supported node image, runtime, operating system, architecture, and replacement strategy | Node-pool plan and capacity calculation |
| APIs | Removed or deprecated APIs used by manifests, charts, controllers, and custom resources | Scan output and migrated manifests |
| Extensions | Compatibility of ingress, CNI, CSI, DNS, metrics, policy, admission, and autoscaling components | Version matrix and test results |
| Workloads | Readiness, disruption budgets, probes, storage behavior, and application-level smoke tests | Staging and production verification notes |
Define success before the maintenance window. A useful success statement might include: all nodes report the intended version, critical workloads are available, error rates remain within the service objective, scheduled jobs run, ingress and service discovery work, and no unresolved warning appears in cluster events. For background on connecting upgrade checks to reliability targets, see the SLO and error budget workflow.
Checklist by scenario
Before any upgrade
- Record the current Kubernetes version, provider or distribution, regions, clusters, node pools, zones, and maintenance constraints.
- Export a dependency inventory covering CNI, CSI drivers, ingress controllers, DNS, metrics and logging agents, policy engines, certificate automation, autoscalers, service meshes, operators, and admission webhooks.
- List every production namespace, owner, critical workload, persistent volume, scheduled job, and external dependency.
- Confirm that backups are recent and restorable. Test the recovery procedure for cluster state and application data; a backup that has never been restored is only an assumption.
- Check capacity for replacement nodes. Include pod density, daemonsets, topology constraints, disruption budgets, quotas, and temporary headroom.
- Freeze unrelated infrastructure and application changes for the maintenance window, or document how concurrent changes will be distinguished from upgrade effects.
- Review the target release notes and provider-specific upgrade guide for API removals, sequencing rules, and known compatibility requirements.
When deprecated APIs or manifests are involved
- Search source repositories, Helm values, GitOps configuration, admission policies, and generated manifests for deprecated API groups and versions.
- Inspect live objects as well as source files. An old object can remain in the cluster even after its source manifest has changed.
- Render charts for the target version and validate them with the appropriate schema or server-side dry-run process.
- Upgrade operators and custom resources according to their project documentation, paying particular attention to conversion webhooks.
- Test create, update, and delete operations for resources managed by automation. Read-only validation does not prove that reconciliation still works.
In staging or a representative test cluster
- Restore representative application data where practical, then deploy the same charts, policies, agents, and admission configuration used in production.
- Exercise normal traffic, rolling deployments, pod eviction, node drain, autoscaling, persistent storage attachment, ingress routing, and scheduled workloads.
- Run application smoke tests and confirm telemetry. The OpenTelemetry setup guide can help verify that traces, metrics, and logs remain useful during the change.
- Test the exact automation that will execute the production change. A manually successful test does not validate a Terraform, Helm, or GitOps workflow.
- Capture baseline measurements for availability, latency, saturation, restart counts, pending pods, and failed jobs.
During the production change
- Announce the start, owners, stop conditions, communication channel, and maintenance boundaries.
- Confirm backups, monitoring, access credentials, and emergency contacts immediately before changing the cluster.
- Upgrade the control plane using the supported provider or distribution sequence. Do not improvise a downgrade plan while the upgrade is running.
- Upgrade node pools in a controlled order. Use surge or replacement capacity where supported, drain conservatively, and watch disruption budgets and pending pods.
- Upgrade or validate cluster extensions at the point required by their compatibility guidance. Verify webhook reachability and certificate validity.
- Pause after each major stage. Record health signals before continuing to the next node pool or workload group.
After the upgrade
Run the following checks with appropriate permissions and adapt resource names to your environment:
kubectl version
kubectl get nodes -o wide
kubectl get pods -A
kubectl get events -A --sort-by=.lastTimestamp
kubectl get --raw='/readyz?verbose'
kubectl get --raw='/livez?verbose'
Also verify ingress and service-to-service traffic, DNS resolution, persistent volume attachment and mounting, secrets and certificates, deployment rollouts, autoscaling signals, backup jobs, scheduled jobs, and alert delivery. Compare results with the baseline rather than relying only on a green node status. Review the Kubernetes deployment readiness checklist for workload-level checks.
What to double-check
Rollback is not always a version downgrade
Many Kubernetes upgrade paths do not provide a simple, supported control-plane downgrade. Define recovery as a decision tree before the change:
- Application failure, cluster healthy: stop the rollout, revert the application or configuration through the normal deployment system, and preserve evidence.
- Node-pool issue: cordon or remove affected nodes only if safe, restore the previous node image or pool according to provider guidance, and confirm capacity first.
- Extension incompatibility: restore a compatible controller, webhook, driver, or chart version, then re-run focused tests.
- Control-plane or data risk: stop making changes, engage the provider or incident commander, and use the tested backup or cluster-rebuild procedure. Do not delete resources in an attempt to make the control plane appear healthy.
Record the exact point at which the recovery decision changes. Include time limits, owners, and objective thresholds such as sustained API errors, failed readiness checks, unavailable replicas, or storage failures.
Admission, security, and supply-chain dependencies
Admission webhooks can block deployments and even routine cluster operations when they are unreachable or use incompatible APIs. Test failure behavior, timeouts, namespace selectors, certificates, and service endpoints. Revalidate image policy, signature checks, SBOM or provenance enforcement, and CI credentials after the upgrade. The software supply-chain security guide provides a useful companion checklist.
Automation and ownership
Make the inventory actionable by assigning an owner to each extension and critical workload. A service catalog can make ownership and lifecycle metadata easier to find; the Backstage software catalog guide covers that operating model. Store the upgrade plan, test evidence, commands, dashboards, and recovery notes with the infrastructure code or runbook.
Common mistakes
- Checking only node versions: a healthy node does not prove that webhooks, storage, ingress, policies, or applications work.
- Skipping live-object inspection: stale resources and manually created objects can escape repository searches.
- Ignoring disruption budgets: strict budgets can make drains stall; permissive budgets can remove too much capacity at once.
- Upgrading every dependency simultaneously: separate changes where possible so failures remain diagnosable.
- Trusting a backup without a restore test: recovery time and data completeness must be demonstrated, not assumed.
- Using a generic rollback statement: write the provider-specific recovery procedure before the maintenance window.
- Failing to remove temporary exceptions: restore normal alerting, admission policy, maintenance settings, and change controls after validation.
When to revisit
Revisit this Kubernetes upgrade runbook before each planned version change, node-image refresh, provider maintenance cycle, or seasonal planning cycle. Update it whenever the provider changes upgrade sequencing, the target release changes API behavior, an extension changes its compatibility range, or the platform adopts a new CNI, CSI driver, admission controller, autoscaler, GitOps workflow, or backup system.
After every upgrade, turn observations into durable improvements. Add missing commands, dashboards, owners, capacity assumptions, and stop conditions. Review failed validation runs and incidents separately from successful changes; both reveal where the runbook is incomplete. A practical final action is to schedule the next review, link the current provider documentation and release notes, and open one small tracked task for each unresolved dependency. This keeps the checklist useful instead of allowing it to become a historical record that no longer matches the cluster.