sdlc.dev

Redis path · Using it well · Topic 6 of 11

Pub/sub versus streams

Not started

Redis has two quite different ways to pass messages. Pub/sub broadcasts each message to whoever is subscribed right now and keeps nothing. Streams are a data type: an append-only log stored under a key, which consumers read at their own pace and acknowledge. Pick by asking whether a message may be lost when nobody is listening.

Why it matters: Pub/sub silently drops messages for anyone not listening; choosing it for work that must not be lost is a common and quiet bug.

  1. 1Pub/sub is at-most-once

    A published message goes to the subscribers connected at that moment and is then gone; nothing is stored.

  2. 2Streams are a stored log

    XADD appends entries with ids to a stream key; readers can read from any point, including history.

  3. 3Consumer groups share work

    XREADGROUP hands each entry to one consumer in the group and keeps it pending until XACK.

  4. 4Pending entries need care

    Entries a crashed consumer never acknowledged stay pending; XAUTOCLAIM lets another consumer take them over.

1

How it works

The explanation, with examples you can run.

Pub/sub

# Terminal 1> SUBSCRIBE news1) "subscribe"2) "news"3) (integer) 1 # Terminal 2> PUBLISH news hello(integer) 1          # number of subscribers that received it # Terminal 1 receives1) "message"2) "news"3) "hello"

PUBLISH returns how many subscribers got the message; with nobody subscribed it returns 0 and the message is gone. The Redis docs describe the delivery as at most once. A subscriber that disconnects, restarts, or falls too far behind (its output buffer overflows and Redis drops the connection) misses messages, with no way to replay them. Pub/sub suits signals where only the latest state matters: "config changed, reload", cache invalidation hints, live presence or typing indicators. Since Redis 7, sharded pub/sub (SSUBSCRIBE, SPUBLISH) keeps a channel's traffic on the shard that owns it in a cluster.

Streams

A stream is an append-only log. Each entry has an id (by default a millisecond timestamp and a sequence number) and a set of field-value pairs:

> XADD orders:events * type created order 1001"1760610000000-0"> XADD orders:events * type paid order 1001"1760610000001-0"

(The ids depend on the clock when you run it.) Entries stay until you trim them, for example with XADD ... MAXLEN ~ 100000 or XTRIM, so a consumer that was offline can catch up. XREAD reads from any id, and can block waiting for new entries.

Consumer groups

To spread work across several workers and know what has been done, create a group:

> XGROUP CREATE orders:events billing 0OK> XREADGROUP GROUP billing worker-1 COUNT 10 STREAMS orders:events >1) 1) "orders:events"   2) 1) 1) "1760610000000-0"         2) 1) "type"            2) "created"            ...      2) 1) "1760610000001-0"         ...> XPENDING orders:events billing1) (integer) 2...

> means "entries never delivered to this group". Each one goes to one consumer and is added to the group's pending entries list until the consumer runs XACK. If a worker crashes, its entries stay pending; another worker can take them over with XAUTOCLAIM (Redis 6.2 and later) once they have been idle long enough. That makes delivery at least once, so handlers must cope with seeing an entry twice. A second group, say analytics, reads the same entries independently.

Choosing

NeedUse
Broadcast a hint; losing one is finePub/sub
Every message processed, with retriesStream with a consumer group
A simple work queue, no historyA list with LMOVE/BLMOVE to a processing list
Long retention, replay from weeks ago, very high volumeA log built for it, such as Kafka

Streams live in memory like everything else in Redis, so trim them, and remember that their durability is Redis's: with asynchronous replication and periodic fsync, acknowledged entries can still be lost in a failover.

2

Check yourself

3 questions. Get them all right to finish this topic.

  1. 1.An order service publishes "order paid" on a pub/sub channel. The email worker is restarting for 10 seconds. What happens to messages published during that time?
  2. 2.Two workers in the same consumer group call XREADGROUP on a stream. How are new entries divided?
  3. 3.A worker read an entry with XREADGROUP and crashed before XACK. What happens to the entry?

0 of 3 right so far. Get all three to finish this topic.

Go deeper