This OpenTelemetry setup guide shows how to introduce tracing, metrics, and logs without creating an unmanageable stream of telemetry. You will learn how to choose signals, instrument services, configure the OpenTelemetry Collector, estimate data volume and operating cost, verify production behavior, and repeat the process safely as your system changes.
Overview
OpenTelemetry is most useful when it answers an operational question. For example: Which dependency is slowing checkout? Did the latest deployment increase error latency? Which Kubernetes workload is consuming more resources than expected? Instrumentation should support these questions rather than collect every possible event by default.
A practical architecture has four layers:
- Application instrumentation: An OpenTelemetry SDK or automatic instrumentation library creates telemetry inside the service.
- Resource metadata: Attributes identify the service, environment, version, namespace, and other ownership context.
- Collection: The OpenTelemetry Collector receives, processes, batches, samples, and exports telemetry.
- Backend analysis: A tracing, metrics, logging, or unified observability backend stores and queries the resulting data.
Start with one service and one operational workflow. A small pilot makes it easier to identify missing context, excessive cardinality, exporter problems, and unexpected volume before the same configuration reaches every workload.
For Kubernetes environments, keep application instrumentation separate from cluster collection where possible. A service can emit telemetry to a Collector through a stable endpoint, while the Collector deployment model can evolve independently. Review workload readiness and resource settings alongside your telemetry rollout using a Kubernetes deployment best practices checklist.
How to estimate
Telemetry cost is driven by volume, retention, processing, and the pricing model of the selected backend. Before choosing sampling or storage settings, estimate the signals your services will produce. The estimate will not be exact, but it provides a repeatable baseline for comparing configurations.
1. Count the producing units
Record the number of services, instances, pods, or nodes that will emit each signal. Do not assume that every signal has the same source. Application traces may be produced by each request, host metrics by each node, and container logs by each workload.
2. Estimate event rates
For each signal, estimate a representative rate:
- Traces: requests per second multiplied by the proportion of requests sampled.
- Spans: sampled traces multiplied by the average spans per trace.
- Metrics: number of instruments multiplied by collection frequency and emitting instances.
- Logs: log records per second multiplied by the average encoded record size.
A simple monthly volume model is:
monthly volume = events per second × average event size × active seconds per month
Use a stated operating window rather than presenting the result as a universal monthly fact. For example, you might model 30 days of continuous operation, then create a second estimate for a lower-traffic period. Include protocol encoding, attributes, indexes, and backend overhead if the vendor's billing model includes them. If those details are unknown, mark them as excluded instead of hiding them in a precise-looking number.
3. Apply sampling and filtering
Sampling should be deliberate. A head-sampling policy can reduce the number of traces sent from the application or Collector. Tail sampling can make decisions after a trace is observed, which can help preserve errors or slow requests, but it requires the relevant spans to reach the same sampling decision point. Use filtering to remove signals that are not useful, such as known noisy health checks, while retaining enough data to diagnose availability and dependency problems.
4. Compare scenarios
Create at least three scenarios:
- Baseline: current traffic, normal sampling, and expected retention.
- Growth: increased request rate, additional services, or more telemetry attributes.
- Incident: elevated error rate, increased traffic, or temporarily higher diagnostic sampling.
This approach is more useful than a single estimate because observability systems must remain useful during the incidents when teams need them most.
Inputs and assumptions
Keep the following inputs in a versioned worksheet, repository document, or platform catalog. Record the date, environment, and source of each estimate.
| Input | What to record | Why it matters |
|---|---|---|
| Service and instance count | Services, pods, hosts, and environments | Determines the number of producers and resource attributes. |
| Request rate | Normal, peak, and incident traffic | Drives trace and application metric volume. |
| Average spans per trace | Typical and high-complexity request paths | Shows how instrumentation depth affects volume. |
| Metric instruments | Instrument count, labels, and collection interval | High-cardinality attributes can multiply time series. |
| Log rate and size | Records per second and approximate payload size | Logs can dominate volume when verbose levels are enabled. |
| Sampling policy | Head or tail sampling rules and exceptions | Determines which traces are retained for analysis. |
| Retention | Hot, searchable, and archived periods | Connects ingest volume to storage requirements. |
| Collector resources | CPU, memory, replicas, queues, and limits | Prevents the collection tier from becoming a bottleneck. |
Use stable resource attributes such as service name, service version, deployment environment, cloud region, cluster, namespace, and workload name. Avoid putting request IDs, user IDs, full URLs, or unbounded values into metric labels. Those values may be useful in traces or logs, but they can create an excessive number of metric series and increase privacy and governance risk.
Choose exporters and protocol settings that your backend supports, then test them in a non-production environment. Collector configuration commonly includes receivers, processors, exporters, and pipelines. A minimal structure looks like this:
receivers:
otlp:
protocols:
grpc:
http:
processors:
memory_limiter:
batch:
exporters:
otlp:
endpoint: ${env:OTEL_EXPORTER_ENDPOINT}
service:
pipelines:
traces:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [otlp]Treat this as a configuration pattern, not a drop-in production file. Add authentication, TLS, queue behavior, retry policy, metrics and logs pipelines, and backend-specific exporter settings according to your environment. Keep secrets outside the configuration file and validate the configuration during CI or deployment.
For Kubernetes, decide whether the Collector runs as a deployment, daemonset, or both. A deployment is often suitable for receiving application telemetry through a shared endpoint. A daemonset can be useful when collection is tied to nodes or local workloads. The choice should follow traffic patterns, failure domains, and operational ownership rather than a default template.
Worked examples
Example 1: Estimating trace volume
Assume a service receives 40 requests per second, the average trace contains 12 spans, and the proposed sampling rate is 25%. For a continuously active 30-day planning window, the estimated sampled traces per second are:
40 × 0.25 = 10 sampled traces per second
The estimated span rate is:
10 × 12 = 120 spans per second
To convert this into storage volume, multiply 120 by the average encoded span size and the active seconds in your chosen planning window. Because span size varies with attributes and instrumentation, measure a representative batch from the Collector or backend rather than relying on a generic size assumption. Repeat the calculation for peak traffic and for a policy that retains all error traces.
Example 2: Estimating metric series growth
Suppose a service emits 20 metric instruments across 10 instances. A simple upper-bound estimate before label expansion is:
20 instruments × 10 instances = 200 instance-level series
If a metric also includes two bounded dimensions, such as environment and region, multiply only by the known number of values. Do not use this method for unbounded dimensions such as user ID or request path unless the possible values are controlled and explicitly counted. Compare the result with the backend's series and retention model before enabling the full set in production.
Example 3: Verifying a rollout
Instrument one service, deploy it to a test environment, and verify the complete path:
- Make a request that crosses at least one internal or external dependency.
- Confirm the trace contains the expected service and dependency spans.
- Check that resource attributes identify the environment and deployed version.
- Generate a controlled error and confirm it is visible in the selected signal and backend.
- Inspect Collector health, rejected data, retry counts, queue depth, and export errors.
- Compare observed event volume with the estimate and explain material differences.
When a deployment platform changes, repeat this verification. A GitOps workflow can help make Collector configuration reviewable; see the GitOps versus CI/CD decision guide for a broader workflow comparison.
When to recalculate
Recalculate telemetry volume and repeat production verification whenever an input changes. Important triggers include a new service, a material traffic increase, a change in sampling, new instrumentation libraries, additional metric labels, longer retention, a backend migration, or a Collector upgrade. Also revisit the estimate after an incident if emergency diagnostic settings increased volume or if missing telemetry delayed investigation.
Use this practical rollout checklist:
- Define the operational questions the telemetry must answer.
- Choose signals and semantic attributes before adding instrumentation broadly.
- Measure normal, peak, and incident event rates.
- Configure Collector batching, memory protection, retries, authentication, and transport security.
- Set sampling and filtering rules with explicit error and latency exceptions.
- Test exporter compatibility and backend queries in a non-production environment.
- Alert on Collector rejection, queue growth, export failure, and resource saturation.
- Verify traces, metrics, and logs from a known request after every relevant change.
- Record assumptions, observed volume, and configuration versions.
OpenTelemetry works best as a managed platform capability rather than a one-time instrumentation project. Give service teams a supported SDK pattern, a documented Collector endpoint, resource-attribute conventions, and a short verification procedure. Keep the volume worksheet with that documentation, and update it when traffic, pricing inputs, retention, or backend behavior changes. For incident operations, connect the resulting dashboards and traces to an actionable SRE runbook so telemetry leads to a decision, not just another screen to inspect.