sdlc.dev

Kafka path · Building on it · Topic 12 of 12

Kafka or a queue (RabbitMQ, SQS)?

Not started

Kafka and message queues such as RabbitMQ and Amazon SQS all move messages from producers to consumers, but they are built on different ideas. A queue is a list of work: a message is delivered to one consumer, acknowledged, and deleted. Kafka is a log of facts: records are kept for a retention period, and every reader keeps its own position. Most choices between them follow from that difference.

Why it matters: Kafka and message queues overlap enough to be confused and differ enough to hurt when you pick the wrong one. The question is whether you are distributing work or publishing a history of facts.

  1. 1A queue hands out work

    Each message goes to one consumer, is acknowledged individually and is then deleted. Failed messages are redelivered or sent to a dead letter queue.

  2. 2A log publishes facts

    Records stay for the retention period; any number of groups read them independently, at their own pace, and can replay.

  3. 3Ordering and parallelism are tied together in Kafka

    Kafka orders per partition and gives each partition to one consumer per group. Queues spread messages freely and usually give up ordering to do it.

  4. 4Share groups blur the line

    Since Kafka 4.2, share groups let many consumers take records from the same partition with per-record acknowledgement and delivery counts.

1

How it works

The explanation, with examples you can run.

Side by side

Kafka (consumer groups)Work queue (RabbitMQ queues, SQS)
After processingRecord stays until retentionMessage is deleted on acknowledgement
Many independent readersEach group reads everythingFan out to one queue per subscriber (exchanges, SNS)
ReplayReset offsets within retentionNot possible once deleted
OrderingPer partition, by keyUsually none (SQS FIFO and single-consumer queues are exceptions)
ParallelismUp to the partition count per groupAdd consumers freely
Unit of acknowledgementAn offset per partitionOne message
A slow or poison messageHolds up its partitionOthers carry on; it is retried or dead-lettered
RoutingTopic and keyRich in RabbitMQ (exchanges, bindings); simple in SQS
RetentionConfigurable: hours, days or foreverUntil consumed; SQS also expires messages after 4 days by default, 14 at most

When Kafka fits

  • Events that several systems consume, now or later: order placed, payment captured, profile changed.
  • Replay and backfill: rebuilding a search index, recomputing analytics after a bug fix, onboarding a new service from history.
  • Streams processing and change data capture, where ordering per key matters.
  • High sustained throughput with consumers that read in batches.

When a queue fits

  • Task distribution: send this email, resize this image, call this webhook. Jobs vary in duration, ordering does not matter, and one stuck job should not delay the rest.
  • Per-message retries with delays and dead letter queues, without writing that machinery yourself.
  • Request/reply and routing patterns, where RabbitMQ's exchanges do the work.
  • Small volumes where running a Kafka cluster (or paying for a managed one) is not worth it; SQS is serverless and pay per request.

Share groups: queues inside Kafka

Kafka 4.2 made share groups (KIP-932, "queues for Kafka") production-ready. Consumers in a share group take records from the same partitions cooperatively instead of owning whole partitions. Each record is acquired with a time-limited lock (share.record.lock.duration.ms, 30 s by default), acknowledged individually, and redelivered if not, up to share.delivery.count.limit attempts (5 by default). That suits queue-shaped work on data that already lives in Kafka, without ordering guarantees. It is new, so check client support in your language before relying on it.

Both, often

Many systems use both: Kafka as the record of what happened, and queues (or share groups) for the tasks those events cause. In either case delivery is at least once, so the handlers are idempotent.

2

Check yourself

3 questions. Get them all right to finish this topic.

  1. 1.Three teams (billing, analytics, search) each need every order event, and analytics wants to reprocess last month after a bug fix. Which fits better?
  2. 2.Image-resize jobs take from 1 second to 5 minutes each, order does not matter, and one slow job should not hold up others. With classic consumer groups, what is the weakness of Kafka here?
  3. 3.In SQS, a consumer receives a message and does not delete it within the visibility timeout (30 seconds by default). What happens?

0 of 3 right so far. Get all three to finish this topic.

Go deeper