Caching Overview

Start with the request path

A cache stores a reusable result closer to its reader. It can avoid a database query, computation or remote fetch. It also creates another copy that can become stale. The right question is which work can safely be skipped and how the application behaves when the cache misses.

Worked estimate: the cold-cache problem

Assume an illustrative service receives 1,000 requests per second, 90% are reads, and 80% of those reads hit a cache. All writes reach the origin. Normal origin load is:

Read misses = 1,000 × 0.90 × (1 − 0.80) = 180 requests/s
Writes      = 1,000 × 0.10              = 100 requests/s
Origin      = 180 + 100                = 280 requests/s
Cold cache  = 1,000 requests/s

An origin sized only for 280 requests/s can be overwhelmed when the cache restarts. The cold-cache load is about 3.57 times its normal load. This estimate assumes one origin operation per miss or write; joins, fan-out and retries can multiply the real work. Change the inputs in the capacity lab and compare both conditions.

Three responses are possible: provision enough origin headroom, admit only a bounded rate during warm-up, or serve eligible stale data while refreshing it. Each shifts a cost. Headroom costs capacity, admission creates rejected requests, and stale responses weaken freshness. The choice belongs in the product requirement.

Choose a cache boundary

Do not compare raw RAM access latency with an entire database transaction as if they measured the same operation. Measure the complete request path, including serialization and the network.

Design freshness explicitly

A time-to-live is an expiry rule, not a guarantee that data is correct until expiry. A value can become stale immediately after insertion. For a profile description, brief staleness may be acceptable; for an authorization decision, stale permission data can be dangerous. Keep authoritative checks on sensitive paths.

Use cache-aside when the application owns lookup and invalidation, and read caching strategies for race conditions. HTTP caches follow different rules: the HTTP caching standard defines directives such as private, no-store and no-cache.

What to test before depending on a cache

1. Empty it while maintaining the assumed traffic rate. Observe origin latency and rejected work.
2. Expire a popular key on many clients at once. Coalesce concurrent fetches and bound waiting time.
3. Update the database while a slow reader is filling the cache. Check that an old value cannot survive indefinitely.
4. Change authentication or tenant context. Confirm one user cannot receive another user’s cached response.
5. Fail the cache network path. Use deadlines and controlled fallback; unlimited fallback is another overload path.

Track hit rate by endpoint, payload size, miss latency, eviction rate and origin saturation. A high global hit rate can conceal a small, expensive class of misses.

Read eviction policies to plan memory limits and backpressure to bound overload.