Message Queues

What Are Message Queues?

A Message Queue is a form of asynchronous communication between services. Instead of calling a service directly and waiting for a response, you put a message in a queue. The receiving service picks it up when ready.

Think of it like email vs. phone calls: email lets you communicate without both parties being available simultaneously.

Why Use Message Queues?

Core Concepts

Producer

The service that sends messages to the queue.

Consumer

The service that receives and processes messages from the queue.

Queue / Topic

The buffer that stores messages between producers and consumers.

Broker

The server that manages queues, routing, and delivery (e.g., RabbitMQ, Kafka).

Messaging Patterns

1. Point-to-Point (Queue)

Competing consumers normally give each delivery to one consumer. Failure and retry can deliver the same logical message again.

Producer → [Queue] → Consumer A
                     Consumer B (waiting)
                     
One consumer receives each delivery; duplicates can be retried

Use case: Task distribution (e.g., image processing jobs)

2. Publish-Subscribe (Topic)

One producer sends a message, and ALL subscribers receive a copy.

Producer → [Topic] → Subscriber A (gets copy)
                   → Subscriber B (gets copy)
                   → Subscriber C (gets copy)

Use case: Event broadcasting (e.g., order placed → notify inventory, billing, shipping)

3. Fan-Out

Combine pub-sub with queues. One message fans out to multiple queues.

Producer → [Exchange] → Queue A → Consumer A
                      → Queue B → Consumer B
                      → Queue C → Consumer C

Use case: Different services need the same event for different purposes

Delivery Guarantees

How do we ensure messages are delivered correctly?

Achieving Exactly-Once Semantics

True exactly-once is hard. Common approaches:

1. Idempotent consumers: Design processing so duplicates have no effect
2. Deduplication: Track processed message IDs
3. Transactional outbox: Atomically write business data and an outbox row in one database transaction; a relay later publishes the row, potentially more than once

# SQL-style pseudocode: perform this in the same transaction as the local effect.
# The UNIQUE constraint makes concurrent duplicate attempts conflict.
INSERT INTO processed_events (event_id) VALUES (:event_id)
ON CONFLICT (event_id) DO NOTHING
RETURNING event_id;
# Apply the local update only if a row was inserted, then commit and acknowledge.
# External payment calls require the provider’s idempotency key as well.

Message Ordering

FIFO (First In, First Out)

Messages processed in order. Important for:

• Financial transactions
• State machines
• Event sourcing

Partitioned Ordering

Order is guaranteed within a partition, but not across partitions.

Partition 0: [M1, M3, M5] → processed in order
Partition 1: [M2, M4, M6] → processed in order
But M1 might process after M2

Kafka uses this approach. Order key determines partition.

Dead Letter Queues (DLQ)

When a message can't be processed after multiple retries, it goes to a dead letter queue for investigation.

Main Queue → [Consumer fails 3 times] → Dead Letter Queue
                                         ↓
                                    Manual investigation

Popular Technologies

When to Choose What

• Need complex routing? → RabbitMQ
• High throughput + replay? → Kafka
• Serverless on AWS? → SQS/SNS
• Simple and fast? → Redis Streams

Real-World Examples

E-Commerce Order Processing

Order Service → [orders_topic → separate queue per service] 
                    ↓
                → Inventory Service (reserve stock)
                → Payment Service (charge card)
                → Email Service (send confirmation)
                → Analytics Service (track metrics)

Video Processing Pipeline

Upload Service → [video_queue] → Transcoding Workers (scale 1-100)
                                       ↓
                                 [thumbnail_queue] → Thumbnail Workers
                                       ↓
                                 [notification_queue] → Notify User

Social Media Timeline

In a hypothetical social-feed design:

1. Post goes to Kafka topic
2. Fanout service reads post
3. Writes to followers' timelines
4. Each follower’s timeline can be a materialized feed, rather than a message queue

Anti-Patterns to Avoid

❌ Synchronous over queue: Don't use queues when you need immediate response ❌ Unbounded queues: Set max size to prevent memory issues ❌ Ignoring errors: Always handle failed messages (DLQ) ❌ Tight coupling: Don't put business logic in message routing

Interview Tips 💡

When discussing message queues in system design:

1. Justify async: "We'd use a queue here because the downstream service is slow/unreliable..."
2. Choose technology: "For event streaming, we'd use Kafka; for task queues, SQS..."
3. Handle failures: "Messages would retry 3 times, then go to a dead letter queue..."
4. Discuss ordering: "Order matters for payments, so we'd partition by user_id..."
5. Consider idempotency: "Consumers would be idempotent to handle potential duplicates..."

Worked failure: a crash after charging a card

A consumer charges the payment provider, then crashes before acknowledging the message. The broker redelivers it. A SELECT followed by an INSERT is not sufficient: two workers can both pass the check. Use the same stable idempotency key for the provider call and record the result under a database uniqueness constraint. Acknowledge only after the outcome is safely recorded. Also test the reverse failure: the local commit succeeds but the acknowledgment is lost.

Delivery and acknowledgment boundaries are described in RabbitMQ consumer acknowledgments.

Related Concepts

• Kafka Architecture — Deep dive into Kafka's design
• Event Sourcing & CQRS — Event-driven architecture patterns
• Microservices — Message queues enable service decoupling
• Rate Limiting — Queues provide natural rate limiting
• Backpressure — Handling overwhelmed consumers