Module 5: Streaming with Kafka · Lesson 5.2 · about 30 min
Kafka: the log at the centre
Topics, partitions, offsets, brokers, replication, producers and consumer groups, with a local Kafka you can start in one command and Python to talk to it.
Module 5 · about 30 minutes · read first: 5.1. You’ll be able to explain topics, partitions, offsets, brokers and replication, choose a message key, size a topic’s partitions, read consumer lag, and run Kafka on your laptop with a Python producer and consumer.
1. Kafka in one paragraph
Apache Kafka is a distributed, durable, append-only log. “Append-only” means new events are only ever added to the end; nothing is edited in place. It was built at LinkedIn around 2010 to move activity data between systems and became an Apache open-source project soon after. Today it sits in the middle of most event-driven platforms: tills, apps, ad servers and databases write to it; stream processors, warehouses, search indexes and machine learning systems read from it. Confluent, a company founded by Kafka’s creators, sells a managed version, and the big clouds offer their own (Amazon MSK, Google Cloud Managed Service for Apache Kafka, and Azure Event Hubs, which speaks the Kafka protocol).
Picture a set of conveyor belts with numbered trays. Each topic is a set of belts. Each belt is a partition. Every tray placed on a belt gets the next number, its offset, and stays on the belt for a set time, say seven days (retention), whether anyone has looked at it or not. Teams (consumer groups) stand along the belts, each with its own clipboard noting the last tray number it handled. Fraud, stock and marketing read the same trays at their own speed, and anyone can rewind to tray 1,000 and read again.
2. The parts
- Broker: one Kafka server. A cluster has several. Modern Kafka manages its own cluster metadata with KRaft; the old dependency on a separate ZooKeeper service was removed in Kafka 4.0.
- Topic: a named stream, such as
pos.sales.v1orads.impressions.v1. - Partition: a slice of a topic, spread across brokers. Each partition is its own ordered log. More partitions mean more work can happen in parallel.
- Offset: the position of a message inside one partition: 0, 1, 2, and so on.
- Message (record): a key, a value (bytes, often JSON, Avro or Protobuf), a timestamp and optional headers.
- Replication: each partition has a leader on one broker and followers with copies on others. A replication factor of 3 is the usual choice in production, so the data survives a broker failing.
- Retention: keep messages for a time (7 days), up to a size, or forever with log compaction, which keeps only the latest value per key (handy for “current price per product”).
3. Keys decide partitions, and partitions decide order
The producer turns the key into a number (a hash) and uses it to pick a partition. All events with key store-214 land in the same partition, in the order they were sent. So choose the key around what must stay in order: an order ID, a shopper ID, a store.
pos.sales.v1 has 3 partitions.A key that is far too popular, such as one giant advertiser, makes one hot partition that does most of the work while the others idle. It is the same skew problem you met in Making Spark fast and cheap.
- Topic
- A named stream, like a TV channel. Producers write to it, consumers subscribe.
- Partition
- One ordered slice of a topic. Order is promised only inside a partition.
- Offset
- A message's position number in its partition. A consumer's offset is its bookmark.
- Consumer group
- A team of consumers sharing one job. Each partition goes to exactly one member.
- Lag
- How far a group's bookmark is behind the newest message: latest offset − committed offset, added over partitions.
- Physical meaning
- When a grocer's website shows the right stock seconds after a till sale, a Kafka partition carried that store's sales in order, and a consumer group of a few small programs kept up with every store at once, each reading its own share of the belts.
4. Producers and consumers
- Producers send records, group them into batches for efficiency, and wait for an acknowledgement. The setting
acks=allwaits until every in-sync copy has the record: slower but safe, and the default in recent Kafka versions. - Consumers pull records and commit offsets to save their progress. If a consumer restarts, it carries on from the last committed offset.
- A consumer group shares a topic’s partitions among its members. To go faster, add consumers, up to the number of partitions. If one dies, its partitions are handed to the others.
Partitions therefore cap how many consumers can work in parallel. Choosing the count is a sizing sum:
The number every on-call engineer watches is consumer lag. If it keeps growing, the consumers are too slow; if it shrinks, they are catching up.
“Kafka is a queue, so a message disappears once it is read.” In a classic queue, yes. In Kafka, reading only moves your own bookmark. The message stays until retention removes it, so other groups can read it and you can replay it after fixing a bug.
5. Run it on your laptop
You need Docker. This starts a single-broker Kafka (good for learning, not for production) and creates a topic with 3 partitions:
docker run -d --name kafka -p 9092:9092 apache/kafka:latest
docker exec -it kafka /opt/kafka/bin/kafka-topics.sh --bootstrap-server localhost:9092 \
--create --topic pos.sales.v1 --partitions 3 --replication-factor 1
pip install confluent-kafka
A producer that sends 1,000 synthetic till sales, keyed by store:
# producer.py
import json, random, time
from confluent_kafka import Producer
p = Producer({"bootstrap.servers": "localhost:9092", "acks": "all"})
for i in range(1000):
store = f"store-{random.randint(1, 20)}"
evt = {"txn": i, "store": store, "sku": random.randint(1, 500),
"price": round(random.uniform(1, 40), 2), "ts": time.time()}
p.produce("pos.sales.v1", key=store, value=json.dumps(evt))
p.poll(0) # serve delivery reports
p.flush() # wait until every message is acknowledged
A consumer that keeps a running total per store and commits its bookmark after each message:
# consumer.py: run two copies in two terminals and watch them split the partitions
import json
from confluent_kafka import Consumer
c = Consumer({"bootstrap.servers": "localhost:9092", "group.id": "store-totals",
"auto.offset.reset": "earliest", "enable.auto.commit": False})
c.subscribe(["pos.sales.v1"])
totals = {}
while True:
m = c.poll(1.0)
if m is None or m.error():
continue
e = json.loads(m.value())
totals[e["store"]] = totals.get(e["store"], 0) + e["price"]
print(m.partition(), m.offset(), e["store"], round(totals[e["store"]], 2))
c.commit(message=m, asynchronous=False)
To see lag, run kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group store-totals inside the container: it lists each partition’s current offset, end offset and lag.
6. Kafka and Google Pub/Sub
| Kafka | Google Pub/Sub | |
|---|---|---|
| Model | Partitioned log; consumers track offsets | Topics and subscriptions; each message acknowledged |
| Order | Per partition | Optional, per ordering key |
| Replay | Move a group’s offset back, within retention | Seek to a time or snapshot, within retention |
| Running it | Clusters to size (unless managed) | Serverless: nothing to size |
| Ecosystem | Kafka Connect, Kafka Streams, Schema Registry, a huge community | Built into Google Cloud: Dataflow, BigQuery subscriptions |
Many companies on Google Cloud use both: Kafka as the company-wide event backbone, Pub/Sub for pipelines that live entirely inside Google Cloud.
7. Try it
- Run the producer and two consumers. Stop one consumer and watch the other take over its partitions within a few seconds.
- Start a consumer with a new
group.idand see it read everything from offset 0. - In two sentences, explain why
storeis a reasonable key here, and what would go wrong if you keyed bytxninstead.
What to check
- The surviving consumer starts printing partition numbers it did not print before.
- The new group’s first lines show offsets 0, 1, 2 on each partition.
- Keying by store keeps each store’s sales in order in one partition, so a running total per store is correct. Keying by
txnspreads one store’s sales over all partitions, so two consumers would each hold half a store’s total and nothing keeps that store’s events in order.
New words: broker, partition, offset, replication factor, retention, log compaction, key, consumer group, rebalance, consumer lag.
- Kafka is an append-only log: producers add to the end, consumers read with their own bookmark (offset), and reading never deletes.
- A topic is split into partitions; the key picks the partition, and order is kept only inside a partition.
- In a consumer group each partition has exactly one reader, so partitions cap parallelism: partitions ≥ throughput needed ÷ throughput per consumer.
- Replication (usually 3 copies) keeps data safe when a broker fails; lag tells you whether consumers are keeping up.
Check yourself
Sources
- Apache Kafka documentation: Design and Introduction https://kafka.apache.org/documentation/
- Kreps, J., Narkhede, N. and Rao, J. (2011), Kafka: a Distributed Messaging System for Log Processing, NetDB workshop
- Kleppmann, M. (2017), Designing Data-Intensive Applications, O'Reilly, Chapter 11 (Partitioned logs)
- Confluent documentation: confluent-kafka Python client
Your ticks stay in this browser. Save them as a file on the course home.
Updated 2026-11-22. All company data in the examples is synthetic. Tool versions and prices change: check the official docs before you rely on them.