sdlc.dev

Kafka path · Building on it · Topic 11 of 12

Operational pitfalls: lag, partitions, rebalances

Not started

Kafka's guarantees hold as long as the cluster is healthy and consumers keep up. Most incidents are not data loss but a pipeline that silently falls behind, a cluster struggling under its own partition count, or a consumer group that spends more time reassigning partitions than processing them. Each has a recognisable signal.

Why it matters: Kafka rarely loses data when configured sensibly. What hurts in practice is consumers falling behind, clusters carrying far more partitions than they need, and groups that keep rebalancing instead of working.

  1. 1Watch lag, per partition

    Consumer lag is log end offset minus committed offset. Steady lag is a capacity problem; lag on one partition is a hot key or a stuck consumer.

  2. 2Partitions are not free

    Each one is files on disk, memory on brokers and clients, a leader to elect and a unit to rebalance. Size for the throughput and parallelism you need, plus headroom.

  3. 3Rebalance storms stop work

    Slow processing past max.poll.interval.ms, short session timeouts and rolling restarts can keep a group rebalancing. The new protocol, static membership and smaller batches help.

  4. 4Alert on replication health

    UnderReplicatedPartitions, UnderMinIsrPartitionCount and OfflinePartitionsCount should be zero; anything else means durability or availability is at risk.

1

How it works

The explanation, with examples you can run.

Consumer lag

bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group billing
GROUP    TOPIC   PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAGbilling  orders  0          3               4               1billing  orders  1          1               4               3billing  orders  2          0               0               0

Lag is the log end offset minus the committed offset, per partition. Read it as a trend: lag that rises during a burst and then drains is the log doing its job as a buffer; lag that climbs steadily means consumers are slower than producers, and they will eventually fall off the end of retention. Lag in time ("how old is the oldest unprocessed record") is easier to alert on than a record count, because a count means different things on different topics. Consumers publish records-lag-max; tools outside Kafka can also compute lag from committed offsets.

Fixes, by cause: add consumers if there are idle partitions to give them; add partitions if every partition already has an owner (keyed topics: plan this); speed up the handler (batching writes, fewer round trips); or deal with a hot key.

Too many partitions

Every partition replica is a directory of segment files and indexes, memory on the broker, entries in the metadata log, and work during leader elections and broker restarts. On the client side, producers keep a buffer per partition and consumers fetch per partition. KRaft raised how many partitions a cluster can manage, but the cost per partition does not vanish. Pick counts from the throughput you need per topic and the consumer parallelism you need, with headroom, rather than a large round number for every topic.

Rebalance storms

A rebalance pauses work on the partitions being moved. With the classic protocol and the old eager assignors, every rebalance stopped the whole group. Common triggers:

  • Slow batches. A member that does not call poll() within max.poll.interval.ms (5 minutes) is removed, its partitions move, it comes back and triggers another rebalance.
  • Missed heartbeats. Long GC pauses or overloaded hosts exceed the session timeout (45 s by default for classic consumers).
  • Rolling deploys. Each restart is a leave and a join.

Remedies: smaller max.poll.records or faster processing; static membership (group.instance.id), so a restart within the session timeout keeps its partitions; the cooperative sticky assignor on the classic protocol; or the new protocol (group.protocol=consumer, GA in 4.0), which moves partitions incrementally without stopping the group.

Broker health

kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions       ISR smaller than the replica listkafka.server:type=ReplicaManager,name=UnderMinIsrPartitionCount       acks=all writes are failingkafka.controller:type=KafkaController,name=OfflinePartitionsCount     partitions with no leaderkafka.controller:type=KafkaController,name=ActiveControllerCount      exactly one controller should report 1

Alongside these: disk usage per broker (a full disk takes a broker down), request latency percentiles, and leader balance. Kafka tries to move leadership back to each partition's preferred replica after a broker returns; kafka-reassign-partitions.sh moves replicas when you add brokers, because new brokers start empty.

2

Check yourself

3 questions. Get them all right to finish this topic.

  1. 1.Lag on a 12-partition topic grows on partition 7 only. What is the most likely cause?
  2. 2.Consumers take 8 minutes per poll batch on bad days and the group keeps rebalancing. Which change addresses the cause?
  3. 3.Why is "1,000 partitions per topic, to be safe" a poor default?

0 of 3 right so far. Get all three to finish this topic.

Go deeper