Monitoring vs Observability

• Monitoring: Tells you when something is wrong. ("CPU usage is 99%")
• Observability: Tells you why it is wrong. ("Service A is retrying due to DB lock contention")

To achieve observability, we need three distinct types of data.

1. Logs (The "What")

Discrete events. "Something happened at time T".

Structure:

• Unstructured: 2023-10-01 Error: DB failed (Hard to query).
• Structured (JSON): {"retry": 3, "service": "payment", "error": "timeout"} (Easy to aggregate).

Centralized Logging Architecture

Don't SSH into servers to tail -f. Ship logs to a central backend.

1. Application: Writes to stdout/stderr.
2. Collector (Fluentd/Vector): Reads streams, parses JSON, enriches (adds Kubernetes pod name).
3. Storage (Elasticsearch/ClickHouse): Indexes fields.
4. UI (Kibana/Grafana): Search "error" AND service="payment".

2. Metrics (The "Health")

Aggregated numerical data. Cheap to store. Good for alerts.

Key Types:

• Counter: Always goes up (Total Requests, Errors). rate() gives requests/sec.
• Gauge: Goes up and down (Memory Usage, Queue Size).
• Histogram: Distribution of values (Request Latency: p50, p90, p99).

[!TIP] Cardinality Explosion: Avoid putting unique IDs (UserID, IP) in metric labels. It creates millions of time series and kills Prometheus.

Scraping vs Pushing

• Prometheus (Pull): Scrapes /metrics endpoint every 15s. Service doesn't need to know about monitoring server.
• StatsD (Push): App sends UDP packets to collector. Good for short-lived jobs (Lambdas).

3. Distributed Tracing (The "Where")

Follows a request across microservices.

Structure:

• Trace Context: A unique TraceID passed in HTTP Headers (W3C Trace Context).
• Span: A unit of work (DB Query, HTTP Call). Has SpanID, ParentID, Start/End time.

Visualization: Waterfalls showing gaps (latency) and errors.

Integration: Code Example (OpenTelemetry)

OpenTelemetry (OTel) is the standard for generating all three.

from opentelemetry import trace, metrics

# 1. Tracing Setup
tracer = trace.get_tracer(__name__)

# 2. Metrics Setup
meter = metrics.get_meter(__name__)
request_counter = meter.create_counter("requests_total")

def handle_request():
    # Start a span
    with tracer.start_as_current_span("checkout_flow") as span:
        span.set_attribute("user.id", "123")
        
        # Increment metric
        request_counter.add(1)
        
        try:
            process_payment()
        except Exception as e:
            span.record_exception(e)
            span.set_status(trace.Status(trace.StatusCode.ERROR))
            # Log automatically correlated with TraceID
            logger.error("Payment failed", exc_info=e)

def process_payment():
    with tracer.start_as_current_span("db_update"):
        # Database logic...
        pass

The "Golden Signals" (SRE)

What should you alert on? (Google SRE Book)

1. Latency: Time to service a request.
2. Traffic: Demand on system (Req/sec).
3. Errors: Rate of failed requests (HTTP 500s).
4. Saturation: How "full" is the service? (CPU/Memory/Queue depth).

The Economics: Observability Bills Can Exceed Compute

A dirty secret of modern infrastructure: it is entirely possible to spend more observing a system than running it. Teams routinely discover that logging and APM invoices rival their entire compute budget. The costs concentrate in three places:

• Log volume: a chatty service emitting 10KB of DEBUG logs per request, at 1,000 req/s, produces ~860 GB/day — indexed, replicated, and retained for 30 days, that's tens of terabytes for one service. Structured logging discipline (one INFO event per request, DEBUG off in prod, sampling for high-volume paths) is a budget decision as much as a hygiene one.
• Metric cardinality: every unique label combination is a separate time series stored forever-ish. http_requests{path=...} where path includes raw URLs with IDs (/users/8231442/orders) silently creates millions of series. Normalize labels to route templates (/users/:id/orders).
• Trace volume: tracing every request at 50K req/s is mostly recording identical, boring successes. Sampling strategy — head-based (decide at request start, cheap) vs. tail-based (decide after completion, keeps all errors and slow requests, requires buffering) — is the main knob.

A practical retention tiering that most teams converge on: metrics for ~13 months (capacity trends), logs 7–30 days hot with cheap object-storage archive for compliance, traces 3–7 days (they're for active debugging, not history).

Correlation: Where the Three Pillars Become One Tool

Individually, each pillar answers a narrow question. The operational superpower is jumping between them without losing context:

1. An alert fires on the p99 latency metric for /checkout (metrics told you when).
2. You open the latency histogram, click an exemplar — a real trace attached to that histogram bucket — and land in the exact slow request's waterfall (traces tell you where: a 900ms UPDATE balance span).
3. The span links to the logs emitted during it, filtered by trace_id (logs tell you why: lock wait timeout, retried 3 times).

Making this work requires one discipline everywhere: propagate and log the trace context. Every log line carries trace_id; every metric that can carry exemplars does. OpenTelemetry's biggest contribution isn't any single signal — it's a shared context that stitches them together across every language and vendor.

Alerting: The Art of Not Crying Wolf

Observability data is only useful if the right human looks at the right time — and doesn't get paged for the wrong things. Hard-won principles:

• Alert on symptoms, not causes. Page on "checkout error rate > 1%" (users are hurting), not "CPU > 80%" (maybe fine, maybe not). Cause-based alerts become dashboards, not pages.
• Use burn-rate alerts for SLOs. If your SLO allows 0.1% errors per month, alert when you're burning that budget 14x faster than sustainable (page) or 2x (ticket). This catches both sharp outages and slow leaks with the same framework, without hair-trigger noise.
• Every page must be actionable. If the responder's action is "watched it, it self-resolved," delete or demote the alert. Alert fatigue is how real incidents get ignored — an on-call rotation receiving 50 pages a week is less safe than one receiving 5.
• Symptom durations matter: for: 5m clauses suppress transient blips, at the cost of 5 minutes of detection latency. Choose per-alert based on the blast radius of the underlying failure.

Interview Tips 💡

• "How do you debug a slow request in microservices?" — Distributed Tracing. Look for the "long bar" in the waterfall.
• "How do you keep observability costs sane?" — Route templates over raw paths in labels, sampled traces with tail-based retention of errors, log-level discipline, and tiered retention.
• "What do you page on?" — Symptom-based, SLO burn-rate alerts; everything else is a dashboard or ticket.
• "Push vs Pull Metrics?" — Prometheus (Pull) is standard for infrastructure. Push (Datadog) better for serverless.
• "Log Levels" — Explain DEBUG vs INFO vs WARN vs ERROR. Don't log DEBUG in prod (cost).
• "Sampling" — You can't trace 100% of requests (too expensive). Tail-based sampling keeps only the interesting (slow/error) traces.

Related Concepts

• Distributed Tracing
• Prometheus Architecture
• ELK Stack