SRE Runbook Template: Build Reliable Procedures for Incidents and Operations

Overview

An SRE runbook is an operational guide for responding to a known condition. It is not a replacement for engineering judgment or an incident commander. Its purpose is to help the responder establish context quickly, avoid unsafe assumptions, and take the least risky effective action.

A useful incident response runbook should be written for a specific service or failure mode. “Check the logs and restart the application” is too vague to guide a production decision. A stronger procedure identifies the alert, expected symptoms, relevant dashboards, safe commands, decision points, escalation path, and recovery criteria.

For cloud-native systems, runbooks often need to cover Kubernetes workloads, managed databases, queues, ingress, deployment automation, and observability tooling. Keep commands version-aware and explain what each command is intended to reveal. Read-only diagnostics should be clearly separated from commands that change production state.

Copyable SRE runbook structure

1. Service: Name, environment, critical dependencies, and business function.
2. Owner: Primary team, on-call rotation, technical lead, and escalation contact.
3. Severity: Conditions that distinguish an urgent customer-impacting incident from a lower-risk warning.
4. Alerts: Alert names, links, thresholds, expected firing behavior, and known noisy conditions.
5. Symptoms: What users, support teams, dashboards, and logs are likely to show.
6. Diagnostics: A short, ordered set of read-only checks with expected results.
7. Mitigation: Reversible actions, prerequisites, risks, and stop conditions.
8. Escalation: When to involve another team, a vendor, or an incident commander.
9. Recovery: Signals that confirm the service is healthy and the incident is contained.
10. Follow-up: Evidence to preserve, ticket ownership, and review requirements.

Checklist by scenario

Before an incident

• Confirm that the service name, repository, deployment location, and production environment are unambiguous.
• Record the team responsible for the service and the approved escalation path.
• Link to the service dashboard, alert definitions, logs, traces, deployment history, and dependency documentation.
• List normal operating signals such as request rate, error rate, latency, saturation, queue depth, and replica availability.
• Document safe access requirements without placing credentials, tokens, or sensitive data in the runbook.
• Mark every command as read-only or state-changing. Include the namespace, context, region, or environment where it must run.
• Define success and failure criteria for each mitigation step.
• Test the procedure in a non-production environment when practical, and identify steps that require an authorized approver.

When an alert fires

1. Acknowledge and classify: Record the alert time, affected service, suspected severity, and current customer impact.
2. Check for correlation: Compare the alert with recent deployments, infrastructure changes, dependency incidents, traffic changes, and other alerts.
3. Establish scope: Determine whether the issue affects one pod, one zone, one tenant, one endpoint, or the entire service.
4. Preserve evidence: Capture relevant dashboard views, alert details, deployment identifiers, logs, and trace examples before making disruptive changes.
5. Run diagnostics in order: Start with low-risk checks that narrow the problem without altering production state.
6. Choose a mitigation: Select the smallest reversible action that addresses the likely failure and has a defined rollback.
7. Communicate: State what is known, what is being investigated, customer impact, and the next update point.

For a Kubernetes workload failure

Begin with the workload status, recent events, rollout state, and pod-level symptoms. Check whether the issue is caused by scheduling, failed readiness or liveness probes, image retrieval, configuration, resource pressure, or an application error. Avoid treating a restart as a diagnosis. If a rollback is considered, verify the previous revision, database compatibility, configuration changes, and the rollback command before executing it.

Useful runbook entries can include commands such as kubectl get deployment, kubectl describe pod, and kubectl rollout history, but they should be accompanied by the correct context and namespace. The exact command set should match the cluster access model and approved operating practices.

For elevated latency or error rates

Compare latency and errors by endpoint, status code, region, version, and dependency where those dimensions are available. Use traces to identify whether time is spent in the application, database, queue, network, or an external service. Check whether a recent release or configuration change aligns with the start of the event. If traffic is being shifted or a deployment is paused, document the decision threshold and the signal that will permit a return to normal operation.

For capacity, queue, or resource pressure

Check whether demand increased, capacity decreased, or resource requests and limits are misaligned with actual behavior. Inspect queue age rather than only queue length when processing time varies. For Kubernetes, review pending pods, node pressure, autoscaler activity, throttling, and eviction signals. A mitigation such as scaling may reduce symptoms while increasing cost or exposing a downstream bottleneck, so include those trade-offs in the runbook.

What to double-check

• Ownership: The listed team still owns the service, and the on-call route reaches a monitored channel.
• Links: Dashboards, log queries, trace views, repositories, and deployment records still resolve and show the intended environment.
• Commands: Examples use the current tooling, correct namespaces and regions, and safe authentication practices.
• Permissions: Responders have the access needed for diagnostics, while destructive actions remain restricted or require approval.
• Dependencies: Database, cache, queue, identity, network, and third-party dependencies have current diagnostic guidance.
• Rollback: The documented version is available, compatible with data and configuration changes, and reversible if the rollback fails.
• Communication: Customer-facing language is factual, avoids speculation, and identifies the next update owner.
• Recovery: The runbook explains how to confirm improvement over time rather than relying on a single green check.

Observability should support the procedure rather than merely decorate it. If a step asks a responder to inspect traces, the relevant instrumentation and collector path must be understood. If the service depends on metrics retention, make sure the required time range is available. A practical OpenTelemetry setup can make cross-service diagnosis easier, but the runbook should still identify the exact service, operation, and signal to inspect.

Common mistakes

• Writing for an ideal system: Document the failure modes responders actually encounter, including partial outages and degraded dependencies.
• Mixing diagnosis and mitigation: Separate observation from action so a responder does not make a change before understanding scope.
• Using vague instructions: Replace “check Kubernetes” with a named resource, command, signal, and interpretation.
• Leaving out stop conditions: Every risky action should state when to stop, roll back, or escalate.
• Assuming alerts are sufficient: An alert may identify a symptom without identifying the cause. Include supporting logs, traces, and dependency checks.
• Embedding secrets: Use approved secret-management and access processes; never copy credentials into operational documentation.
• Ignoring communication: Technical recovery without a clear incident record can leave support teams and customers without reliable information.
• Never rehearsing the runbook: An untested procedure often fails at the exact step where time matters most.

When to revisit

Review a runbook after every incident that exposes a missing, misleading, or slow step. Also revisit it before seasonal planning cycles, major traffic changes, platform migrations, and planned resilience exercises. Tool changes are another trigger: update the procedure when the Kubernetes version, deployment workflow, observability stack, access model, alerting rules, or cloud architecture changes.

Set a lightweight ownership model. The service team owns technical accuracy, the on-call group validates usability, and incident leadership confirms that escalation and communication guidance remain clear. A review does not need to rewrite the entire document. Ask responders to verify the links, run a safe diagnostic path, confirm the rollback target, and remove steps that no longer reflect production.

To put this template into practice, choose one high-impact alert and write the smallest complete procedure for it. Add the owner, symptoms, read-only checks, one reversible mitigation, escalation criteria, and recovery signals. Test it during a scheduled review, capture the gaps, and update the runbook while the details are still easy to verify. Repeat for the next important failure mode rather than trying to document every possible incident at once.