What is Scaling?
Scaling is the ability of a system to handle increased load by adding resources. It's one of the most fundamental challenges in system design.
The question: Your app has 1,000 users today. How do you handle 1,000,000 users tomorrow?
The Growth Problem
Day 1: 1,000 users → Single server handles it fine Day 30: 10,000 users → Server at 80% CPU Day 90: 100,000 users → Server crashes Day 180: 1M users → ???
Two fundamental approaches: Scale Up or Scale Out
Vertical Scaling (Scale Up)
Add more resources to existing server: Bigger CPU, more RAM, faster disk.
Before: 4 CPU cores, 16GB RAM After: 32 CPU cores, 256GB RAM Same server, more powerful
Compare measured capacity
Choose instance sizes using current provider specifications and your own workload. CPU counts, RAM, and hourly price do not translate directly into an equal multiple of application throughput.
Pros & Cons
✅ Pros:
- Simple (no code changes)
- No distributed system complexity
- Consistent performance
- Easy to manage (one machine)
❌ Cons:
- Physical limits (can't buy infinite CPU)
- Single point of failure
- Downtime during upgrades
- Expensive at high end
- Diminishing returns (2x cost ≠ 2x performance)
When to Use
Good for: - Databases (PostgreSQL, MySQL) - Legacy monoliths - Early stage (< 100k users) - CPU-intensive tasks (video encoding) Bad for: - Web-scale applications - High availability requirements - Cost-sensitive projects
Horizontal Scaling (Scale Out)
Add more servers instead of making one server bigger.
Before: 1 server After: 100 servers Each server handles 1% of traffic
Architecture
Implementation
// Stateless application server (can scale horizontally)
const express = require('express');
const redis = require('redis');
const app = express();
// Initialize and connect the Redis client using your installed driver's API.
// The cache and db objects below represent application adapters.
app.get('/user/:id', async (req, res) => {
const userId = req.params.id;
// Check cache first
const cached = await cache.get(`user:${userId}`);
if (cached) {
return res.json(JSON.parse(cached));
}
// Fetch from database
const user = await db.users.findById(userId);
// Store in cache
await cache.set(`user:${userId}`, JSON.stringify(user), 'EX', 3600);
res.json(user);
});
app.listen(process.env.PORT || 3000);
Key: Servers are stateless - no session data stored locally.
Pros & Cons
✅ Pros:
- More aggregate capacity when work can be partitioned
- High availability (one server dies, others continue)
- Cost-effective (use commodity hardware)
- Gradual scaling (add servers as needed)
❌ Cons:
- Complexity (distributed systems)
- Stateless requirement (sessions in Redis/DB)
- Data consistency challenges
- Network overhead
When to Use
Good for: - Web applications - API servers - Microservices - High availability needs - Web-scale (> 1M users) Bad for: - Stateful applications - Single-threaded tasks - Tightly coupled systems
An illustrative scaling sequence
The following user counts are scenario labels, not thresholds. A thousand intensive users can require more capacity than a million occasional readers; advance only when measurements justify the extra moving parts.
Stage 1: Single Server (0-1,000 users)
┌─────────────────┐ │ Web + DB │ ← Everything on one box │ One Server │ └─────────────────┘
Complexity: Very Low
Stage 2: Separate Database (1k-10k users)
┌─────────────┐
│ Web Server │
└──────┬──────┘
│
┌──────▼──────┐
│ Database │
└─────────────┘
Why: Database and web server compete for resources
Stage 3: Load Balancer + Multiple Servers (10k-100k)
┌──────────────┐
Users ──▶ │Load Balancer │
└──────┬───────┘
│
┌────────────┼────────────┐
│ │ │
┌────▼───┐ ┌───▼────┐ ┌───▼────┐
│Server 1│ │Server 2│ │Server 3│
└────────┘ └────────┘ └────────┘
│ │ │
└────────────┼────────────┘
│
┌──────▼──────┐
│ Database │
└─────────────┘
Why: Single server can't handle traffic
Stage 4: Database Replication (100k-1M)
Web Servers
│
├──────────────────┐
│ (writes) │ (reads)
▼ ▼
┌─────────┐ ┌─────────┐
│ Master │─────▶│ Replica │
│ DB │ │ DB │
└─────────┘ └─────────┘
│
┌────▼────┐
│Replica 2│
└─────────┘
Why: Database becomes bottleneck
Pattern: Write to master, read from replicas
Stage 5: Caching Layer (1M-10M)
Web Servers
│
▼
┌─────────┐
│ Redis │ ← Cache hot data
│ Cache │
└─────────┘
│
▼
Databases
Why: 80% of requests hit same 20% of data
Impact: 10x reduction in database load
Stage 6: CDN + Asset Optimization (10M+)
Users │ ▼ ┌─────────┐ │ CDN │ ← Static assets (images, CSS, JS) └─────────┘ │ ▼ Load Balancer → Web Servers
Why: Reduce latency globally
Impact: 95% of bandwidth served from CDN
Stage 7: Database Sharding (10M-100M)
User ID % 4: 0 → Shard 0 1 → Shard 1 2 → Shard 2 3 → Shard 3
Why: Single database can't handle writes
Complexity: High
Database Scaling Strategies
1. Vertical Scaling (Database)
-- Upgrade database instance RDS db.t3.small → db.r5.24xlarge Limits: - Memory limits depend on the selected engine and instance class - CPU limits depend on the selected engine and instance class - Single point of failure
2. Read Replicas
Application
│
├─── Writes ──→ Master DB
│
└─── Reads ──→ Replica 1
Replica 2
Replica 3
Implementation:
const { Pool } = require('pg');
const masterPool = new Pool({ host: 'master.db.com' });
const replicaPool = new Pool({ host: 'replica.db.com' });
// Writes go to master
async function createUser(name, email) {
return masterPool.query(
'INSERT INTO users (name, email) VALUES ($1, $2)',
[name, email]
);
}
// Reads go to replica
async function getUser(id) {
return replicaPool.query(
'SELECT * FROM users WHERE id = $1',
[id]
);
}
Pros:
- Easy to implement
- Can increase read capacity, subject to lag and shared limits
- No application changes (mostly)
Cons:
- Replication lag (eventual consistency)
- Writes still bottlenecked
- Doesn't help with storage
3. Sharding (Horizontal Partitioning)
Split data across multiple databases by key.
// Hash-based sharding
function getUserShard(userId) {
const shardCount = 4;
return userId % shardCount;
// User 1 → Shard 1
// User 5 → Shard 1
// User 7 → Shard 3
}
async function getUser(userId) {
const shard = getUserShard(userId);
const db = databases[shard];
return db.query('SELECT * FROM users WHERE id = $1', [userId]);
}
Sharding strategies:
1. Hash-based: shard = user_id % num_shards 2. Range-based: users 1-1M → Shard 0 users 1M-2M → Shard 1 users 2M-3M → Shard 2 3. Geographic: US users → US Shard EU users → EU Shard Asia users → Asia Shard
Challenges:
-- Cross-shard queries are expensive -- User 1 (Shard 0) follows User 5 (Shard 1) SELECT posts.* FROM posts WHERE user_id IN ( SELECT followed_id FROM follows WHERE follower_id = 1 ) -- Requires querying multiple shards!
Pros:
- Scales writes and storage
- Each shard is independent
- Can use different hardware per shard
Cons:
- Complex application logic
- Difficult to rebalance
- Cross-shard queries expensive
- Cross-shard transactions require database support or application coordination
Caching Strategies
Cache Aside (Lazy Loading)
async function getUser(id) {
// 1. Try cache
const cached = await cache.get(`user:${id}`);
if (cached) return JSON.parse(cached);
// 2. Cache miss - fetch from DB
const user = await db.users.findById(id);
// 3. Store in cache
await cache.set(`user:${id}`, JSON.stringify(user), 'EX', 3600);
return user;
}
Write-Through Cache
async function updateUser(id, data) {
// 1. Update database
const user = await db.users.update(id, data);
// 2. Update cache immediately
await cache.set(`user:${id}`, JSON.stringify(user), 'EX', 3600);
return user;
}
Cache Eviction Policies
LRU (Least Recently Used): - Evict least recently accessed items - Redis offers approximate LRU; the default policy is noeviction LFU (Least Frequently Used): - Evict least frequently accessed items - Better for hot data TTL (Time To Live): - Expire after fixed time - Good for time-sensitive data
Work through a measurable example
Suppose an illustrative service receives 100 requests/s on average and peaks at five times that rate. If 90% are reads and a cache serves 80% of those reads without reaching the origin, peak origin load is:
500 × (10% writes + 90% reads × 20% misses) = 140 requests/s
A measured node limit of 300 requests/s with a 70% planning margin permits 210 requests/s per node. One node can carry the modeled peak; a second provides single-node failure headroom. With a cold cache, all 500 requests/s reach the origin and the same arithmetic needs three serving nodes plus one spare. These numbers are assumptions, not a benchmark or purchasing recommendation.
Try your own inputs in the capacity planning lab. Then test database limits, request cost, retry amplification, and redistribution after a failure. A user count alone cannot reveal any of those constraints.
Interview Tips 💡
When discussing scaling in system design interviews:
- Start simple: "Use the smallest design that meets measured load, recovery, and availability requirements."
- Identify bottleneck: "As traffic grows, database becomes bottleneck. Add read replicas..."
- Measure cache value: "The origin reduction depends on the fraction of cacheable reads and the observed hit rate."
- Estimate numbers: "One million users at ten requests/day is about 116 average requests/s. Measure peaks and per-node sustainable throughput before calculating node counts."
- Discuss trade-offs: "Shard when storage, throughput, or locality constraints justify it; user count is not a universal trigger."
- Validate failures: "Replay peak traffic with a node unavailable and a cold cache before accepting the capacity plan."
Related Concepts
- Load Balancing — Distribute traffic across servers
- Caching — Reduce database load
- Database Sharding — Horizontal database scaling
- CDN — Global content delivery
- Microservices — Service-based scaling
Source and next experiment
Google SRE: Handling Overload discusses why requests per second alone can hide resource constraints. Use the capacity lab to state assumptions explicitly, then measure tail latency and saturation while varying the most expensive request mix.
Keep reading
Vertical Scaling (Scaling Up)
A deep dive into upgrading server hardware to handle increased load, including its limits and best use cases.
API Gateway Pattern
The single entry point for microservices. Implementing rate limiting, authentication, and protocol translation/aggregation.
Caching Strategies
A breakdown of where to place your cache and how to keep it in sync with your database.
Event Sourcing & CQRS
Comprehensive guide to Event Sourcing and Command Query Responsibility Segregation (CQRS) patterns, covering immutable event logs, state reconstruction, read/write separation, and real-world implementations in banking, e-commerce, and audit systems.
Horizontal Scaling (Scaling Out)
The definitive guide to adding more servers to your infrastructure pool and testing the bottlenecks that limit growth.
Microservices Architecture
An architectural style that structures an application as a collection of loosely coupled, independently deployable services.