Kubernetes Deployment Best Practices: A Production Readiness Checklist

Overview

A Kubernetes Deployment can be syntactically valid and still be unready for production. A workload may start successfully but fail under load, remain unavailable during a rollout, expose an unintended network path, or become difficult to diagnose when something changes. Production readiness is therefore less about one manifest field and more about whether the application, cluster, delivery process, and operating team have clear behavior under normal and abnormal conditions.

This checklist is designed for application teams, platform engineers, and administrators reviewing a new service or a significant change to an existing one. Adapt the thresholds to your workload rather than copying them blindly. A stateless HTTP service, a batch worker, and a stateful system need different readiness criteria.

Before using the checklist, document the deployment's owner, dependencies, expected traffic pattern, availability objective, data-handling requirements, and rollback owner. That context makes the technical review useful instead of turning it into a collection of disconnected YAML checks.

For background on selecting a delivery method, see Kubernetes deployment strategies explained. If you are deciding whether Kubernetes is appropriate for the workload at all, compare the operational trade-offs in Docker Compose versus Kubernetes.

Checklist by scenario

1. Every production deployment

• Workload ownership: Record the owning team, escalation path, repository, deployment pipeline, and service documentation.
• Immutable image: Build from a reviewed source revision and reference an image digest where your delivery process supports it. Avoid relying on a mutable tag whose contents can change without a manifest change.
• Version compatibility: Confirm that the application, configuration, database schema, message formats, and dependent services can coexist during a rolling update.
• Resource requests: Set CPU and memory requests based on measured usage or an explicit initial estimate. Requests influence scheduling and should represent what the container needs to operate reliably.
• Resource limits: Set limits deliberately, especially for memory. Understand how the runtime behaves when it approaches a limit and whether a limit could cause throttling or termination during legitimate bursts.
• Health checks: Configure a startup probe for slow initialization, a readiness probe for traffic eligibility, and a liveness probe only when the process can be safely restarted after the selected failure condition.
• Graceful termination: Handle termination signals, stop accepting new work, finish or safely abandon in-flight work, and set a termination grace period appropriate to the application.
• Configuration: Keep environment-specific settings outside the image. Separate ordinary configuration from sensitive values and verify that required keys exist before rollout.
• Security context: Run as a non-root user when possible, use a read-only root filesystem when compatible, drop unnecessary Linux capabilities, and avoid privileged access unless the workload has a documented requirement.
• Observability: Confirm that logs, application metrics, Kubernetes events, and traces where applicable can be correlated to the workload version and request or job context.

2. Services exposed to users or other applications

• Service selection: Verify that the Service selector matches only the intended Pods and that the target port is correct.
• Readiness behavior: Make readiness represent the ability to serve the required request, not merely whether the process is alive. Check behavior during dependency failure and shutdown.
• Ingress or gateway rules: Review hostnames, paths, TLS configuration, timeouts, body-size limits, and authentication boundaries.
• Network policy: Define which namespaces, workloads, and external destinations may connect. Test both an allowed path and a denied path.
• Capacity signals: Identify the metric that should drive scaling and confirm that it reflects user demand, queue pressure, or another meaningful saturation signal.
• Disruption tolerance: Use a PodDisruptionBudget only when it reflects a real availability requirement and does not prevent necessary maintenance. Pair it with enough replicas and suitable topology placement.

3. Deployments that need controlled change

• Rollout strategy: Select rolling, blue-green, canary, or another progressive approach based on blast radius, compatibility, and verification needs. The strategy should be explicit in the delivery design, not an accidental default.
• Availability controls: Review maxUnavailable and maxSurge against replica count, scheduling capacity, and application startup time.
• Automated verification: Gate promotion on health, error rate, latency, business checks, or queue behavior that can reveal a bad release.
• Pause and resume: Ensure operators know how to pause a rollout, inspect it, and continue it safely.
• Rollback: Confirm that the previous image and configuration remain available and that rollback is compatible with database and message-schema changes.

For teams evaluating progressive delivery controllers, the comparison of Argo Rollouts and Flagger provides a useful starting point. The tool matters less than having measurable promotion and rollback criteria.

4. Workloads with autoscaling or variable demand

• Horizontal scaling: Confirm that the selected metric, target value, minimum, and maximum replica counts match the workload's behavior.
• Startup cost: Account for image pull time, initialization, cache warming, and dependency connection setup before assuming a new Pod can serve traffic immediately.
• Cluster capacity: Check that the cluster can schedule the maximum expected replicas and that node pools, quotas, and disruption rules do not block scale-out.
• Vertical changes: Treat changes to requests and limits as capacity changes. Review their effect on scheduling, cost, eviction risk, and autoscaling signals.
• Load test: Exercise scale-up, sustained load, recovery after a dependency interruption, and scale-down behavior in an environment that resembles production.

What to double-check

Many production incidents occur at the boundaries between Kubernetes objects rather than inside a single manifest. Review these relationships explicitly:

• Probe and application semantics: A liveness probe that checks a database or remote API can restart healthy application processes during a dependency outage. Keep liveness narrowly focused on whether the process needs replacement; use readiness to remove a Pod from service.
• Requests and observed usage: Compare declared requests with actual CPU and memory behavior over representative periods. Undersized requests can cause contention, while oversized requests can leave capacity unusable and increase avoidable spend. The Kubernetes cost optimization checklist covers this review in more detail.
• Rollout and capacity: A surge setting may temporarily require additional nodes. Confirm that the scheduler and cluster autoscaler, if used, can provide that capacity without violating quotas or placement rules.
• Configuration and secrets: Check precedence, defaults, rotation behavior, and failure handling. A secret reference that exists in one namespace or environment may not exist in another.
• Identity and permissions: Inspect the ServiceAccount, Role or ClusterRole, and bindings. Remove permissions that the application no longer needs and verify that cloud-provider identity mappings are scoped to the workload.
• Supply chain: Scan images and dependencies in the CI/CD process, preserve provenance appropriate to your environment, and define how vulnerable components are handled. See the software supply chain security checklist for a broader review.
• Operational visibility: Make sure dashboards and alerts distinguish a failed rollout, an unscheduled Pod, a readiness failure, an OOM termination, and an application-level error. These conditions require different responses.

Before approval, run the exact deployment through a non-production environment using the same manifests, policy controls, and delivery steps where practical. Record the commands or dashboard views an operator needs to determine whether the release is healthy.

Common mistakes

• Using only a liveness probe: A process can be alive but unable to serve requests. Without readiness, traffic may reach an unhealthy instance.
• Copying resource values: Values from another service rarely describe the same runtime, traffic, or memory profile. Start with evidence and revisit the estimate.
• Making every probe identical: Startup, readiness, and liveness answer different questions. Reusing one endpoint and timing configuration can create false failures.
• Assuming a rollout equals a rollback plan: Kubernetes can restore an earlier ReplicaSet, but application data changes may not be reversible. Test the complete recovery path.
• Ignoring termination: Short grace periods, missing signal handling, or incorrect connection draining can cause dropped requests and duplicated jobs.
• Scaling on an easy but weak metric: CPU may not represent demand for an I/O-bound service, queue worker, or rate-limited API. Select a signal tied to the actual bottleneck.
• Relying on manual edits: Changes made directly in a cluster can drift from source control and make later diagnosis difficult. Use a controlled CI/CD or GitOps workflow and document exceptions.
• Skipping failure tests: A deployment is not ready because the happy path works. Test unavailable dependencies, failed readiness, insufficient capacity, interrupted nodes, and a release that must be stopped.

You can turn this article into a downloadable team checklist by copying the following compact review into a Markdown file:

- [ ] Owner, dependencies, compatibility, and rollback owner documented
- [ ] Image source and version are reviewed and reproducible
- [ ] CPU and memory requests and limits are justified
- [ ] Startup, readiness, and liveness behavior are tested
- [ ] Graceful shutdown and termination timing are verified
- [ ] Configuration, secrets, identity, and permissions are reviewed
- [ ] Service, ingress, TLS, and network policy paths are tested
- [ ] Replicas, disruption rules, placement, and capacity are appropriate
- [ ] Rollout gates, pause, promotion, and rollback are documented
- [ ] Autoscaling and load behavior are tested where applicable
- [ ] Logs, metrics, alerts, and runbook steps are available
- [ ] Security and supply-chain checks pass
- [ ] The change is applied through the approved delivery workflow

When to revisit

Keep this checklist with the deployment workflow rather than treating it as a one-time launch document. Revisit it before a major traffic or seasonal planning cycle, after changing the container base image, runtime, Kubernetes version, ingress controller, service mesh, autoscaling method, policy engine, or delivery tooling. It should also be reviewed after a significant incident, an unexpected scaling event, a rollback, or a change to data schemas and external dependencies.

Set an owner for the checklist and record the last review date in the repository or platform documentation. During each review, compare resource and scaling assumptions with observed behavior, confirm that probes still match application startup and shutdown, and execute a rollback or recovery exercise at a frequency appropriate to the service's risk.

The practical next step is to select one production Deployment, run the compact checklist with its owner and on-call representative, and turn every unchecked item into a clearly assigned task. Once the review is complete, store the evidence alongside the manifest and pipeline definition so the next change starts with a current baseline.