This OpenTelemetry setup guide shows how to build a maintainable cloud-native observability pipeline: instrument services, deploy an OpenTelemetry Collector, route traces, metrics, and logs to suitable backends, and track the signals that reveal whether the implementation is helping engineers troubleshoot production systems. It also provides a recurring review process so configuration, sampling, ownership, and telemetry costs do not quietly drift.
Overview
OpenTelemetry is a set of APIs, SDKs, instrumentation libraries, and collection components for generating and transporting telemetry. Its core signals are traces, metrics, and logs. The project is designed to separate application instrumentation from the systems that receive and process telemetry, which makes it useful across changing cloud-native environments.
A practical implementation usually has four layers:
- Instrumentation: Application code, libraries, agents, or sidecars create telemetry.
- Transport: Telemetry is sent using an OpenTelemetry protocol or another supported protocol.
- Collection and processing: An OpenTelemetry Collector receives, batches, filters, enriches, samples, and forwards data.
- Backends and workflows: Traces, metrics, and logs are stored and queried in systems used by developers and SRE teams.
Start with one service and one troubleshooting workflow rather than instrumenting every workload at once. For example, select an HTTP API with a known dependency chain, make request traces searchable by service and environment, and verify that an engineer can follow a slow request into its database or downstream calls. This creates a concrete acceptance test for the rest of the rollout.
The Collector is not mandatory in every small deployment, but it becomes valuable when teams need a consistent place to apply batching, retries, redaction, routing, and sampling. It also reduces the number of application configuration changes required when telemetry destinations change.
What to track
Instrumentation coverage
Maintain an inventory of services and record whether each one produces traces, metrics, and logs. Coverage should be measured against useful service boundaries, not simply the number of installed libraries. Track the language and runtime, instrumentation method, service name, environment, deployment version, and owner.
Review resource attributes carefully. Consistent values for fields such as service.name, service.version, deployment.environment, and cloud or Kubernetes identity make telemetry easier to group. Avoid putting request IDs, user IDs, URLs with unbounded paths, or other high-cardinality values into metric labels. These may be useful in traces or logs, but they can make metrics expensive or difficult to query.
Trace quality
Track whether important requests have complete, connected traces. Useful checks include:
- Root spans identify the correct service and operation.
- Parent-child relationships connect inbound requests to outbound calls.
- Database, queue, and external service spans contain enough context to explain latency.
- Errors include a status, exception details where appropriate, and a link to the relevant deployment.
- Trace context is preserved across asynchronous boundaries and service-to-service calls.
Do not judge tracing only by volume. A large number of spans is not useful if services cannot be correlated or if sensitive values are collected accidentally. Define a small set of representative journeys and check them after deployments.
Collector health
For each Collector instance or gateway, monitor received items, exported items, rejected items, export failures, queue depth, retry activity, and process resource use. A Collector that accepts telemetry but cannot export it is a reliability problem, so dashboards should distinguish received data from successfully delivered data.
Keep the configuration in version control. A basic pipeline might look like this:
receivers:
otlp:
protocols:
grpc:
http:
processors:
memory_limiter:
check_interval: 1s
limit_mib: 512
batch:
timeout: 5s
send_batch_size: 512
exporters:
otlphttp/observability:
endpoint: https://observability.example.invalid
service:
pipelines:
traces:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [otlphttp/observability]This is a structural example, not a drop-in production configuration. Exporter names, authentication, TLS settings, limits, and backend endpoints must match the Collector distribution and destination you operate. Validate configuration in CI and test failure behavior before using it for critical workloads.
Metrics, logs, and cost controls
Track whether metric names and dimensions remain stable, whether logs can be correlated with trace IDs, and whether the selected retention period matches investigative needs. Monitor telemetry volume by service, signal, environment, and destination. A sudden increase can indicate a useful new workload, an instrumentation change, a retry loop, or an accidental high-cardinality attribute.
Sampling should be treated as an explicit policy. Head sampling makes an early decision, while tail sampling can use information gathered during a trace, such as errors or latency, before deciding what to retain. Whichever approach you use, document which traffic is sampled, which traffic is retained preferentially, and how operators can investigate incidents when a trace is absent.
Cadence and checkpoints
Use different review intervals for implementation health and design decisions. A short operational check catches failures quickly; a monthly or quarterly review catches drift.
Per deployment
- Confirm the service identity and version changed as expected.
- Run one representative request and inspect its trace across dependencies.
- Check that error status, latency, and important attributes are present.
- Verify that no credentials, tokens, personal data, or unnecessary payloads are emitted.
Weekly or during on-call review
- Inspect Collector export failures, queue pressure, and rejected telemetry.
- Compare telemetry volume with deployment and traffic changes.
- Test links between logs, metrics, and traces for a recently investigated issue.
- Record missing instrumentation or broken service ownership metadata.
Monthly or quarterly
- Review sampling rules against current incident and debugging requirements.
- Remove unused dashboards, processors, exporters, and instrumentation packages.
- Recheck retention, access controls, redaction rules, and destination contracts.
- Compare the cost and operational value of each signal and environment.
- Test Collector upgrades in a non-production environment before broad rollout.
Keep a small change log for these reviews. Record what changed, why it changed, which services were affected, and how the result was verified. This turns an OpenTelemetry implementation into an operational system rather than a one-time configuration project.
How to interpret changes
A rise in trace volume is not automatically a problem. First compare it with request volume, deployment changes, sampling configuration, and the number of instrumented services. If telemetry grows faster than traffic, inspect span creation, retry behavior, duplicate exporters, and high-cardinality attributes.
A drop in telemetry can be more serious, but it needs context. Check application health, network connectivity, Collector queues, authentication, certificate changes, backend ingestion, and sampling decisions. If only one service is affected, compare its runtime configuration with a healthy peer. If several services fail at once, investigate the shared Collector or destination path first.
Use traces to explain latency, metrics to identify patterns and saturation, and logs to provide detailed event context. These signals should reinforce one another. For example, a latency metric can identify an affected operation, a trace can locate the slow dependency, and a correlated log can explain the application decision or error. If the signals cannot be joined, prioritize consistent service identity and trace-context propagation before adding more dashboards.
When reviewing sampling, preserve diagnostic value rather than maximizing retention. Error traces, slow traces, and traces associated with important workflows may deserve different treatment from routine successful requests. Document the policy so application teams understand why some traces are present and others are not.
When to revisit
Revisit this OpenTelemetry setup guide at least monthly for operational checks and quarterly for architecture review. Review it sooner when a new runtime or deployment platform is introduced, a backend changes, a service begins handling materially different traffic, or an incident exposes missing context.
Use the following rollout checklist:
- Select one service and define two or three troubleshooting journeys.
- Standardize service, environment, version, and ownership attributes.
- Enable instrumentation and verify trace propagation across one dependency chain.
- Deploy a Collector with bounded memory, batching, retries, and secure transport.
- Measure received versus exported telemetry and alert on sustained delivery failures.
- Set explicit redaction, retention, and sampling rules before expanding coverage.
- Roll out by service group, recording gaps and correcting the instrumentation pattern.
- Schedule the next monthly health review and quarterly design review.
As the surrounding platform evolves, connect this work to your broader cloud-native tooling practices. A deployment workflow should verify observability alongside application health; a Kubernetes operating model should assign ownership for Collector resources; and an incident runbook should state how to find related traces, metrics, and logs. For deployment context, see GitOps vs CI/CD and Kubernetes deployment strategies. For metrics capacity planning, the Prometheus retention and scaling guide provides related considerations.
The goal is not to collect everything forever. It is to maintain a dependable path from a user-visible symptom to the service, dependency, change, or resource condition that requires attention. Review the implementation on a schedule, adjust it when the system changes, and keep the configuration understandable enough for the next engineer to operate.
