The Observability Stack

Monitoring vs Observability

• Monitoring: Tells you when something is wrong. ("CPU usage is 99%")
• Observability: Gives you evidence to investigate why it is wrong; telemetry does not establish causation by itself. ("Service A is retrying due to DB lock contention")

To achieve observability, we need three distinct types of data.

1. Logs (The "What")

Discrete events. "Something happened at time T".

Structure:

• Unstructured: 2023-10-01 Error: DB failed (Hard to query).
• Structured (JSON): {"retry": 3, "service": "payment", "error": "timeout"} (Easy to aggregate).

Centralized Logging Architecture

Don't SSH into servers to tail -f. Ship logs to a central backend.

1. Application: Writes to stdout/stderr.
2. Collector (Fluentd/Vector): Reads streams, parses JSON, enriches (adds Kubernetes pod name).
3. Storage (Elasticsearch/ClickHouse): Indexes fields.
4. UI (Kibana/Grafana): Search "error" AND service="payment".

2. Metrics (The "Health")

Aggregated numerical data. Cheap to store. Good for alerts.

Key Types:

• Counter: Increases during a process lifetime and can reset on restart (Total Requests, Errors). rate() gives requests/sec.
• Gauge: Goes up and down (Memory Usage, Queue Size).
• Histogram: Distribution of values (Request Latency: p50, p90, p99).

[!TIP] Cardinality Explosion: Avoid putting unique IDs (UserID, IP) in metric labels. It creates millions of time series and kills Prometheus.

Scraping vs Pushing

• Prometheus (Pull): Scrapes /metrics at a configured interval, for example every 15 seconds. Service doesn't need to know about monitoring server.
• StatsD (Push): App sends UDP packets to collector. Good for short-lived jobs (Lambdas).

3. Distributed Tracing (The "Where")

Follows a request across microservices.

Structure:

• Trace Context: A unique TraceID passed in HTTP Headers (W3C Trace Context).
• Span: A unit of work (DB Query, HTTP Call). Has SpanID, ParentID, Start/End time.

Visualization: Waterfalls showing gaps (latency) and errors.

Integration: Code Example (OpenTelemetry)

OpenTelemetry provides APIs, SDKs and collection protocols. This instrumentation fragment requires a configured SDK, exporters and logging correlation; by itself it does not send telemetry anywhere. OpenTelemetry Python instrumentation.

import logging
from opentelemetry import trace, metrics

logger = logging.getLogger(__name__)

# 1. Tracing Setup
tracer = trace.get_tracer(__name__)

# 2. Metrics Setup
meter = metrics.get_meter(__name__)
request_counter = meter.create_counter("requests_total")

def handle_request():
    # Start a span
    with tracer.start_as_current_span("checkout_flow") as span:
        span.set_attribute("checkout.channel", "web")
        
        # Increment metric
        request_counter.add(1)
        
        try:
            process_payment()
        except Exception as e:
            span.record_exception(e)
            span.set_status(trace.Status(trace.StatusCode.ERROR))
            # Correlation requires a configured logging integration
            logger.error("Payment failed", exc_info=e)

def process_payment():
    with tracer.start_as_current_span("db_update"):
        # Database logic...
        pass

The "Golden Signals" (SRE)

What should you alert on? (Google SRE Book)

1. Latency: Time to service a request.
2. Traffic: Demand on system (Req/sec).
3. Errors: Rate of failed requests (HTTP 500s).
4. Saturation: How "full" is the service? (CPU/Memory/Queue depth).

Interview Tips 💡

• "How do you debug a slow request in microservices?" — Distributed Tracing. Look for the "long bar" in the waterfall.
• "Push vs Pull Metrics?" — Prometheus (Pull) is standard for infrastructure. Push (Datadog) better for serverless.
• "Log Levels" — Explain DEBUG vs INFO vs WARN vs ERROR. Use targeted, time-bounded debug logging when needed and avoid secrets or personal data.
• "Sampling" — Full tracing can be practical at small scale; sampling can control cost and overhead as traffic grows. Tail-based sampling keeps only the interesting (slow/error) traces.

Worked investigation: fast CPU, slow checkout

Suppose checkout p99 rises while CPU stays at 30%. A CPU-only alert misses it. Compare checkout latency with dependency spans, connection-pool wait time and retry counts. If the database span is short but the gap before it is long, investigate queueing before tuning the query. Keep request IDs in logs or trace context rather than metric labels so each request does not create a new time series.

Related Concepts

• Distributed Tracing
• Prometheus Architecture
• ELK Stack