Distributed Tracing

Follow the time across service boundaries

An illustrative login request takes five seconds, while individual service logs show only milliseconds of work. The missing time could be queueing, retries, connection setup, a blocked thread, or an uninstrumented dependency. Distributed tracing helps connect those observations into one request history. It reveals the instrumented path, not an automatically complete account of every operation.

Traces, spans, and context

A trace groups related work. A span represents an operation with start/end time, attributes, status, and links or a parent. Child spans can overlap, so adding every span duration is not the request's total latency. Inspect the critical path and unexplained gaps.

Context propagation carries identity between processes. A valid W3C traceparent header uses a 32-hex-character trace ID and a 16-hex-character parent ID:

traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01

Short names such as abc123 are useful diagram labels but are not valid trace IDs in that header. Use the SDK propagator to inject/extract context rather than concatenating headers manually. Preserve context through asynchronous tasks and message consumers as well as HTTP calls. Batches can need span links to multiple upstream messages.

The W3C Trace Context specification defines the wire format. A trace ID is correlation data, not proof of identity or permission.

Instrument without changing request behavior

OpenTelemetry's JavaScript instrumentation guide explains context and active spans. Initialize the SDK before loading automatically instrumented application libraries, use a compatible OTLP exporter, and configure a collector or backend endpoint for the installed versions.

The following describes the lifecycle without pretending to be a drop-in SDK setup:

extract incoming context
start server span in that context
  start a child operation as the active span
    execute the application operation
    record an error/status if it fails
  end the child in a finally block
  inject current context into permitted outgoing calls
end the server span in a finally block
export completed spans through a bounded batch queue

Use parameterized database queries. Do not add raw user-supplied SQL, credentials, session tokens, or full request bodies to spans. Attributes need a retention and access policy, and high-cardinality identifiers can increase cost. Instrumentation should not make an authorization failure return protected data or expose internal exception text to a caller.

Decide what to keep

A head sampler cannot reliably retain every failed or slow request by checking a status or duration that does not yet exist. Tail sampling needs memory limits, a decision timeout, and a policy for late spans. Keeping every trace can be reasonable at low volume; choose the rate from an observability budget and incident needs.

The OpenTelemetry sampling guide explains these distinctions. Metrics computed from sampled traces need care: a biased sample of slow requests is not a population-wide latency distribution.

A practical diagnostic exercise

Create a test route that performs one database call and two parallel downstream requests. Give each downstream call a controlled delay. Check that:

1. Every service's spans join the same trace and have the intended parent or links.
2. Parallel spans overlap instead of being added as sequential latency.
3. A failed dependency records an error and ends its span.
4. Disabling the collector does not indefinitely block application requests.
5. A queued job retains useful causal context after the original HTTP request ends.

Then add a retry. You should see each attempt and the delay between attempts. If the trace shows only the successful final call, it can hide the main cause of user-visible latency.

Tracing identifies a slow password check or query; it does not justify weakening password hashing or removing correctness checks. First investigate resource limits, queueing, and the applicable security requirements. Measure instrumentation overhead on your workload instead of assuming a universal millisecond cost.

Related concepts

• Observability stack — traces alongside logs and metrics.
• Microservices — service boundaries and failure modes.
• Backpressure — queueing and overload.