Apache Kafka Architecture

Why Kafka?

Traditional message queues (like RabbitMQ) push messages to consumers and delete them after acknowledgment. Kafka is fundamentally different: it is a Distributed Commit Log. Messages are persisted on disk for days or weeks. Consumers "pull" messages at their own pace, and multiple consumers can independently read the same data.

This design makes Kafka ideal for scenarios where multiple services need to process the same events, or where you need the ability to replay past events (e.g., rebuilding a derived view after a bug fix).

Interactive: The Commit Log

1. Core Concepts

A. Topic

A category of messages (e.g., user-clicks, payments, order-updates). Producers write to topics, and consumers read from topics.

• Topics are logical containers. Physically, they are split into Partitions — the actual unit of storage and parallelism.

B. Partition (The Unit of Scalability)

A Topic is divided into ordered, immutable sequences called partitions (P0, P1, P2).

• Ordering: Messages are ordered only within a single partition, not across partitions.
• Parallelism: In a conventional consumer group, three partitions allow at most three active consumers for that topic. Other groups can read independently.
• Storage: Each partition is an append-only log stored as segment files on disk.
• Sizing: Choose partitions using measured throughput, key distribution, recovery time and required consumer parallelism; there is no universal topic size.

C. Offset

Every message in a partition gets a sequential ID called an Offset (0, 1, 2, 3...).

• Kafka doesn't delete messages after consumption. Instead, each consumer tracks its own "offset pointer" — how far it has read.
• This enables Replayability: A consumer can rewind to an available offset; retention and compaction may have removed older records.
• Offsets are stored in a special internal topic called __consumer_offsets.

D. Broker

A single Kafka server is called a Broker. A Kafka Cluster consists of multiple brokers:

• Each broker stores one or more partitions.
• In KRaft, a controller quorum manages metadata; its active controller coordinates changes. Controller and broker roles may be separate.
• Size the broker count from measured storage, partition load, replication and recovery requirements; there is no universal production cluster size.

2. Producers & Keys

How does a producer know which partition to send to?

# Producer sends a message with a key
producer.send(
    topic="user-clicks",
    key=b"user:123",      # Partition key
    value=b'{"page": "/home", "time": "2026-01-15"}'
)

• Partition = hash(key) % num_partitions
• Why keys matter: A stable key normally maps to the same partition while the partition count and partitioner stay fixed. Changing either can remap it; producer retry settings and consumer processing also affect the ordering observed by an application.
• Null keys: If no key is provided, the producer uses a round-robin or sticky partitioner for even distribution.

3. Consumer Groups

This is the key to Kafka's scalability for consumption.

• Problem: We need to process 1 million messages per second.
• Solution: Create a Consumer Group with 10 consumer instances. Kafka automatically assigns partitions to consumers within the group.

How It Works

• Topic with 6 partitions + Consumer Group with 3 consumers:

  • Consumer 1 → P0, P1
  • Consumer 2 → P2, P3
  • Consumer 3 → P4, P5

• Rebalancing: If Consumer 2 dies, Kafka re-assigns P2 and P3 to the surviving consumers:

  • Consumer 1 → P0, P1, P2
  • Consumer 3 → P3, P4, P5

• Scaling rule: Adding consumers beyond the number of partitions is wasteful — the extra consumers sit idle.

Multiple Consumer Groups

Multiple consumer groups can read the same topic independently. Each group maintains its own offsets:

• Group A (Analytics): Counting clicks per page.
• Group B (Recommendations): Building user profiles.
• Group C (Alerting): Detecting anomalies.

All three groups read every message, but at their own pace.

4. Reliability (Replication & ISR)

Which failures should an acknowledged write survive?

Replication

• Replication Factor: Typically 3 (1 Leader, 2 Followers).
• The leader handles writes; reads normally use the leader, with supported replica-fetch configurations able to serve reads from followers.
• If the Leader broker fails, one of the Followers is promoted to Leader automatically.

In-Sync Replicas (ISR)

• ISR: The set of replicas considered in sync under the configured lag criteria; this does not mean their offsets are identical at every instant.
• The producer's acks setting controls durability:
  • acks=0: Don't wait for any acknowledgment (fastest, data loss possible).
  • acks=1: Wait for Leader to acknowledge (balanced).
  • acks=all: Wait for ALL replicas in the ISR to acknowledge (slowest, strongest guarantee).

# Producer configuration
acks=all
enable.idempotence=true

# Topic/broker configuration (not a producer property)
min.insync.replicas=2

5. ZooKeeper vs KRaft

The Old Way (Pre-2.8): ZooKeeper

• Kafka relied on Apache ZooKeeper for metadata management: tracking which brokers are alive, which broker is the Controller, and partition leadership.
• Problems: ZooKeeper was a separate system to deploy, monitor, and scale. It became a bottleneck as clusters grew beyond ~200,000 partitions.

The New Way: KRaft (Kafka Raft)

• Starting in Kafka 2.8 (GA in 3.3), Kafka manages its own metadata using an internal Raft consensus quorum.
• Benefits: Simpler operations (no ZooKeeper), faster Controller failover, different metadata-management and scaling characteristics; validate limits for the deployed version.
• Kafka 4.0 removed ZooKeeper mode. See the Kafka 4.0 upgrade guide.

6. Real-World Use Cases

1. Activity tracking: A hypothetical product can publish clicks and searches for independently operated analytics consumers.
2. Log Aggregation: Collect logs from thousands of microservices into a central system (ELK stack, Splunk) via Kafka.
3. Event Sourcing / CQRS: Store every state change as an immutable event. Replay events to rebuild system state.
4. Stream Processing: Process events in real-time with Kafka Streams or Apache Flink.
5. Change Data Capture (CDC): Capture row-level changes from databases (MySQL, PostgreSQL) and stream them to downstream systems via Debezium + Kafka.

Summary

Interview Tips 💡

1. Start with the problem: "Traditional queues delete messages after reading. Kafka retains them — enabling replay, multiple consumers, and event sourcing."
2. Ordering guarantee: "Kafka guarantees ordering within a partition, not across partitions. Choose partition keys carefully."
3. Scaling consumers: "The max parallelism is limited by the number of partitions. More consumers than partitions = idle consumers."
4. Durability vs. latency: "acks=all with topic min.insync.replicas=2 rejects writes when fewer than two replicas are in sync. It improves durability but does not guarantee survival of every correlated failure."
5. KRaft: "Modern Kafka (3.3+) no longer needs ZooKeeper. It uses an internal Raft quorum for metadata management."

Worked failure: three replicas, one survivor

Assume replication factor 3, producer acks=all, topic min.insync.replicas=2, and two replicas become unavailable. Rejecting new writes is the intended behavior: accepting with one copy would abandon the chosen durability requirement. Alert on under-replicated partitions and producer errors together. Recover replicas before relaxing the minimum; a successful retry after recovery still needs an idempotent downstream handler for external side effects.

Configuration scope and acknowledgment behavior are documented in Kafka topic settings.

Related Concepts

• Message Queues — Kafka vs. RabbitMQ vs. SQS
• Event Sourcing & CQRS — Pattern that leverages Kafka's log
• Distributed Transactions — Kafka as the event backbone
• Backpressure — Handling producer/consumer speed mismatches