Argo CD Application Health Checklist: Diagnose Sync Drift and Safe Rollouts

Overview

Argo CD reports two related but different conditions: sync status and health status. An application can be synced to the manifests in Git while still being unhealthy because a pod is crashing, a service has no endpoints, or a custom resource has not reached its expected state. Conversely, an application can be healthy but out of sync because someone changed a live resource, a generated value differs, or the desired revision has not been applied.

Start every investigation by recording the application name, destination cluster and namespace, desired Git revision, live revision, first observed failure, and the most recent deployment change. This creates a short operational timeline and prevents several people from making unrelated changes while troubleshooting.

Useful first checks include:

• Review the application summary, sync status, health status, conditions, and operation history in the Argo CD interface or CLI.
• Inspect the application diff before syncing. Identify whether the difference is expected, generated, or caused by an out-of-band edit.
• Check Kubernetes events, controller logs, pod status, readiness probes, and admission or policy errors.
• Confirm that the destination cluster, namespace, project, source path, revision, and values are the ones intended.
• Compare the failure with the latest Git change and with any dependency that should have been created first.

For broader operational context, pair this process with a documented SRE runbook template and the production checks in Kubernetes deployment best practices.

Checklist by scenario

When the application is out of sync

1. Read the diff before taking action. Determine which fields differ and whether the live value is managed by Kubernetes, a controller, a mutating webhook, or an operator.
2. Check for an intentional Git change. Confirm the commit, branch or tag, rendered values, and environment-specific overlays.
3. Look for unmanaged edits. A manual scale change, emergency annotation, or direct patch may explain the drift. Decide whether to preserve it, encode it in Git, or remove it.
4. Check ignore rules carefully. Ignoring a noisy field can reduce false drift, but broad ignore rules can hide meaningful changes. Limit them to known controller-managed fields.
5. Sync only after the desired state is understood. Use a preview or dry-run where available, then monitor the operation and resulting resource health.

Do not treat every out-of-sync result as an error. GitOps depends on the difference being visible. The important question is whether the difference is authorized and reproducible.

When the application is unhealthy

1. Identify the resource that determines the degraded state. Start with the resource tree rather than the application label alone.
2. Inspect workload evidence. Review pod conditions, container termination reasons, logs, image pull events, resource scheduling, and readiness or startup probes.
3. Check dependencies. A deployment may be healthy at the workload level while its service, ingress, secret, config map, database migration, or external dependency is not ready.
4. Verify namespace and permissions. Failed creation, denied updates, and missing service-account permissions can look like a stalled rollout.
5. Confirm custom-resource health logic. If a custom resource remains progressing indefinitely, its status fields or Argo CD health customization may not reflect the controller's actual lifecycle.

For custom resources, health evaluation should be based on stable status signals rather than timestamps or implementation details. A simplified Lua customization may look like this:

hs = {}
if obj.status ~= nil and obj.status.ready == true then
  hs.status = "Healthy"
  hs.message = "Resource reports ready"
elseif obj.status ~= nil and obj.status.phase == "Failed" then
  hs.status = "Degraded"
  hs.message = "Resource reports failure"
else
  hs.status = "Progressing"
  hs.message = "Waiting for the controller"
end
return hs

The exact fields must match the controller's documented status contract. Test the customization against successful, progressing, and failed objects before using it in production.

When a sync is stuck or fails

• Read the operation message and Kubernetes events for the first failing resource, not only the final application summary.
• Check whether a hook, job, or health gate is waiting indefinitely. Review hook logs, completion status, and cleanup behavior.
• Look for immutable-field errors, invalid manifests, missing CRDs, admission policy failures, quota limits, and insufficient permissions.
• Confirm that resources are being applied in a safe order. Use sync-wave annotations when a dependency must exist before another resource is created.
• Cancel or terminate an operation only after capturing its error and checking whether partial changes were applied.

When a rollout needs controlled ordering

Sync waves allow related resources to be grouped into an intentional sequence. A typical pattern is to apply foundational resources first, then configuration, workloads, and traffic-related objects:

metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "-1"

Use waves for real dependencies, such as a CRD before its custom resource or a migration job before an application that requires the schema. Avoid turning every object into its own wave; excessive ordering increases waiting time and makes failures harder to interpret. Also verify that hooks, health checks, and controller reconciliation time are compatible with the chosen sequence.

When a rollback is necessary

1. Decide whether the failure is caused by the Git revision, the cluster, or an external dependency.
2. Preserve evidence: application history, diff, events, logs, and the failed commit.
3. Prefer reverting the problematic change in Git so the desired state remains auditable and reproducible.
4. If an urgent operational rollback is supported by your process, select a known-good application revision and verify the resulting diff before applying it.
5. Monitor health after the rollback and create a follow-up change that addresses the original failure rather than leaving the environment at an undocumented state.

Rollback is not a substitute for understanding persistent drift. If a controller or manual process will recreate the same change, correct the source of that change before declaring recovery.

What to double-check before forcing a sync

A forced sync can overwrite live fields or bypass assumptions that made the application safe to operate. Before using force or replace-style behavior, confirm all of the following:

• The diff is understood and the target revision is correct.
• No emergency manual change is protecting availability or data integrity.
• Stateful workloads, persistent volumes, secrets, and database migrations have an explicit recovery plan.
• Health checks will detect a failed rollout instead of reporting success as soon as an object is accepted by the API server.
• Sync windows, project policies, admission controls, and change-approval requirements permit the action.
• There is an operator watching the rollout, with a clear stop or rollback condition.

Consider using sync options only at the narrowest scope needed. A resource-specific exception is easier to review than a broad application-level override. Record why the option was used and whether it should be removed afterward.

Common mistakes

• Confusing synced with available: API acceptance does not prove that traffic can reach a ready workload.
• Forcing a sync without reading the diff: This can replace an intentional live change or conceal the actual source of drift.
• Fixing the cluster but not Git: The next reconciliation may restore the failing configuration.
• Using overly broad health rules: A customization that marks every object healthy can hide real controller failures.
• Adding too many sync waves: Complex ordering can create deadlocks, long waits, and unclear ownership.
• Ignoring hooks and jobs: A completed migration, failed cleanup hook, or never-ending job can block the application even when pods look normal.
• Changing several variables at once: Keep the investigation reversible by changing one suspected cause, observing the result, and recording the outcome.

If your team is building a broader GitOps operating model, compare the workflow assumptions in GitOps vs. CI/CD and document deployment patterns alongside your release strategy.

When to revisit

Review this checklist whenever an application is onboarded, a deployment workflow changes, or an Argo CD or Kubernetes upgrade may affect resource health, sync options, API behavior, or supported integrations. Revisit it before seasonal planning cycles and before periods when fewer operators will be available to respond.

As part of each review, validate one representative application from every important workload pattern: a stateless service, a stateful component, a custom-resource-based service, and an application with hooks or ordered dependencies. Confirm that:

• the resource tree identifies the real failure clearly;
• health checks distinguish progressing, healthy, and failed states;
• diffs exclude only fields that are intentionally controller-managed;
• sync waves reflect actual dependencies;
• rollback instructions point to a known-good revision; and
• the runbook records who may force, terminate, or roll back an operation.

Keep the checklist with the application or platform documentation, and update examples when workflows or tools change. A short, tested procedure is more useful during an incident than a large collection of undocumented sync options.