Load Balancing

Why Load Balance?

A single server can handle only so many requests (e.g., 10k RPS). To handle millions, we need Horizontal Scaling — adding more servers to a fleet. A Load Balancer (LB) sits in front of these servers and distributes incoming traffic, ensuring no single server is overwhelmed while others sit idle.

Without a load balancer, if your one server goes down, your entire application is offline. With a fleet of servers behind a load balancer, routing can move to healthy servers, although in-flight requests may fail and need a safe retry.

Interactive: Algorithms

How does the LB decide which server gets the request?

1. Round Robin

• Logic: Send requests in rotation. Server 1 -> Server 2 -> Server 3 -> Server 1.
• Pros: Simple, with a local rotation counter; independent balancers need no shared global request counter. Fair distribution when servers are identical and requests are uniform.
• Cons: Doesn't account for server load. One request might take 1ms, another 10 seconds. A server handling expensive requests gets overwhelmed while others are idle.
• Variant — Weighted Round Robin: Each server has a weight. A server with weight 3 gets triple the traffic of a server with weight 1. Useful when servers have different hardware.

2. Least Connections

• Logic: Send to the server with the fewest active connections right now.
• Pros: Adapts to slow/fast requests. A server struggling with heavy queries naturally receives fewer new requests.
• Cons: Needs state — the LB must track active connections per server. Slightly more overhead than Round Robin.
• Variant — Weighted Least Connections: Combines connection count with server weights.

3. IP Hash (Sticky Sessions)

• Logic: hash(ClientIP) % NumServers. The mapping stays stable only while the key and relevant backend configuration stay stable.
• Pros: Offers session affinity under those assumptions. Useful when application state is stored in server memory (e.g., shopping carts, WebSocket connections).
• Cons: If a server dies, all its clients are disrupted. Uneven distribution if some IPs generate more traffic.

4. Least Response Time

• Logic: Send to the server with the lowest average response time AND fewest active connections.
• Pros: Optimizes for user-perceived latency.
• Cons: Requires continuous monitoring of response times. More complex to implement.

5. Random

• Logic: Pick a server at random.
• Pros: Extremely simple. Surprisingly effective with large server fleets.
• Cons: No guarantees about even distribution for small numbers of requests.

L4 vs L7 Load Balancing

Layer 4 (Transport Layer)

• Data: IP Address + Port (TCP/UDP).
• Behavior: "I see a packet for Port 80, I forward it to Server A." Routing is based mainly on transport metadata; implementation and listener configuration determine additional inspection.
• Speed: Extremely fast — operates at the kernel level using technologies like IPVS or eBPF.
• Encryption: Can pass TLS through; some L4 products also support TLS-terminating listeners.
• Examples: AWS NLB (Network Load Balancer), HAProxy in TCP mode, Linux IPVS.

Layer 7 (Application Layer)

• Data: HTTP Headers, URL path, Cookies, Request body.
• Behavior: "URL is /checkout → send to Payment Service. URL is /static/logo.png → send to CDN. Header has Content-Type: application/grpc → send to gRPC backend."
• Speed: Slower — must buffer enough packets to parse the HTTP request, decrypt SSL, inspect the payload.
• Smart Features: Content-based routing, A/B testing, authentication, Web Application Firewall (WAF), rate limiting, header injection.
• Examples: AWS ALB (Application Load Balancer), Nginx, Envoy, Traefik, Cloudflare.

Comparison

Health Checks

LBs must detect when a server is unhealthy and stop sending it traffic.

Active Health Checks

The LB proactively sends a request to each server every N seconds:

GET /health HTTP/1.1

Response 200 OK → Server is healthy
Response 503    → Server is unhealthy, remove from rotation
Timeout (>3s)   → Server is unhealthy

Healthy servers are checked every 10-30 seconds. Unhealthy servers are checked more frequently (every 2-5 seconds) to detect recovery quickly.

Passive Health Checks

The LB monitors actual traffic:

• If Server A returns 3 consecutive 5xx errors, it's temporarily removed.
• If Server A's response time exceeds 5 seconds for the past 10 requests, mark as degraded.

Use active and passive checks where supported, tuning their thresholds to the service rather than copying the example timings.

Code Example: Nginx L7 Load Balancer

http {
    upstream backend_servers {
        least_conn;  # Algorithm: Least Connections

        server 10.0.0.1:8080 weight=3;  # More powerful server
        server 10.0.0.2:8080 weight=1;
        server 10.0.0.3:8080 weight=1;
        server 10.0.0.4:8080 backup;    # Only used if others fail
    }

    server {
        listen 80;
        listen 443 ssl;

        ssl_certificate     /etc/ssl/cert.pem;
        ssl_certificate_key /etc/ssl/key.pem;

        location / {
            proxy_pass http://backend_servers;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        }

        # Static files served directly (no backend needed)
        location /static/ {
            root /var/www;
            expires 30d;
        }
    }
}

Global Load Balancing (DNS-Based)

For multi-region deployments, DNS-based load balancing routes users to the nearest data center:

1. User in Tokyo queries api.example.com.
2. DNS returns IP of the Tokyo data center.
3. User in London queries the same domain.
4. DNS returns IP of the London data center.

Geolocation and latency-based routing are different DNS policies. Resolver location, EDNS Client Subnet support, caches and provider measurements affect the selected answer; neither guarantees the fastest endpoint for every user.

Load Balancer High Availability

What if the load balancer itself fails? It becomes a Single Point of Failure (SPOF).

Solution: Deploy load balancers in Active-Passive or Active-Active pairs:

• Active-Passive: Two LB instances share a Virtual IP (VIP). If the active one fails, the passive takes over using VRRP (Virtual Router Redundancy Protocol).
• Active-Active: Both LBs handle traffic simultaneously. DNS returns both IPs. Failover also needs health-aware answers, cache expiry, client retry behavior and sufficient spare capacity.

Interview Tips 💡

1. Start with why: "A load balancer distributes traffic across servers for scalability and fault tolerance."
2. Choose the right algorithm: "Round Robin for stateless APIs, Least Connections for variable-latency backends, IP Hash for sticky sessions."
3. L4 vs L7: "Use L4 for raw performance (gaming, TCP), L7 when you need content-based routing (web apps, microservices)."
4. Don't forget health checks: "Active checks ping /health every 10 seconds. Passive checks monitor real traffic for errors."
5. Address the SPOF: "LBs must be deployed in HA pairs to avoid becoming a single point of failure."

Worked decision: three fast workers and one slow worker

Start with equal request counts and vary the processing time on one backend. Round robin still sends it a similar share, so its queue can grow. Least connections may help when each connection represents comparable work; HTTP/2 multiplexing and long-lived idle connections weaken that assumption. Measure active requests, queue time and p95 response latency before selecting an algorithm. Do not treat the animation’s equal three-second requests as a production benchmark.

The Nginx load balancing guide documents algorithm and health-check behavior for that implementation.

Related Concepts

• Consistent Hashing — Advanced balancing for stateful routing
• DNS Architecture — Global load balancing via DNS
• Proxies — Forward vs Reverse Proxy
• Horizontal Scaling — Why you need a load balancer
• Rate Limiting — Often implemented at the LB layer