The map
- Not started
- In progress
- Done
- Builds on
- 01The logAppend, offset, partition0/2
- 02Writing and readingProducers, groups, guarantees0/3
- 03Keeping the dataReplicas, retention, controllers0/3
- 04Building on itSchemas, tools, operations, choices0/4
- 05Next: system designSame ideas, at service scale
- Event streamingLibrary ↗Kafka is the reference implementation of event streaming. The library topic covers the pattern and its alternatives at a system design level.
- Competing consumersLibrary ↗A consumer group is competing consumers with one twist: the unit of work handed out is a partition, not a message, which caps parallelism at the partition count.
- Idempotency keysLibrary ↗Kafka delivers at least once by default, and exactly-once ends at Kafka's edge. Every consumer that changes something outside Kafka needs an idempotent handler.
- Transactional outboxLibrary ↗Writing to a database and producing to Kafka are two systems. The outbox commits the event with the data, and a relay or CDC connector publishes it.
- BackpressureLibrary ↗Kafka absorbs bursts because consumers pull at their own pace; consumer lag is the backpressure signal. The library topic covers what to do when lag keeps growing.
- Event sourcingLibrary ↗A compacted or long-retention topic can hold the events from which state is rebuilt. The library topic covers snapshots and replays.
- CQRSLibrary ↗Kafka is a common way to feed read models: commands write to one store, events flow through topics, and consumers build query-shaped views.
Written for Apache Kafka 3.x and 4.x; the links point to the 4.3 documentation (May 2026). The examples were run on Apache Kafka 4.1.0 from the apache/kafka Docker image, on a single-node cluster and on a three-node KRaft cluster with combined broker and controller roles. Exactly-once transactions, Kafka Streams and Connect were not run.
Recent changes worth knowing: Kafka 4.0 (March 2025) removed ZooKeeper entirely, made the new consumer rebalance protocol generally available and changed the producer's linger.ms default from 0 to 5. Kafka 4.1 (September 2025) turned on Eligible Leader Replicas for new clusters, and Kafka 4.2 (February 2026) made share groups ("queues for Kafka") production-ready. Schema Registry, mentioned in the schemas topic, is a Confluent component rather than part of Apache Kafka; other registries exist.