sdlc.dev

AWS fundamentals path · Messaging · Topic 14 of 18

SQS: queues, visibility timeouts and dead letters

Not started

Amazon SQS is a managed message queue: producers send messages, consumers poll for them. The key idea is that **receiving a message does not remove it**. SQS hides it from other consumers for the **visibility timeout** (30 seconds by default, up to 12 hours), and the consumer must call DeleteMessage once it has finished. If the consumer crashes, or is simply too slow, the message becomes visible again and someone else gets it. That is what makes SQS reliable, and it is also where duplicates come from.

Why it matters: A queue between services absorbs spikes and outages. SQS's delivery rules (at least once, a visibility timeout, a receive count) decide whether that work is done once, twice or never.

  1. 1Receiving hides, deleting finishes

    A received message is hidden for the visibility timeout. Only DeleteMessage removes it; if the consumer does not delete it in time, it comes back.

  2. 2Standard is at least once

    Standard queues scale almost without limit but may deliver a message more than once and out of order. Handlers must be idempotent.

  3. 3FIFO orders per group

    FIFO queues preserve order within a message group and remove duplicates within a 5-minute window, at lower throughput.

  4. 4Dead-letter queues catch poison

    After maxReceiveCount receives without a delete, SQS moves the message to a dead-letter queue instead of retrying it forever.

1

How it works

The explanation, with examples you can run.

The life of a message

  1. SendMessage: stored redundantly (up to 1 MiB; larger payloads go to S3 with a pointer). Kept for 4 days by default, up to 14.
  2. ReceiveMessage: returned to one consumer with a fresh receipt handle, hidden for the visibility timeout, receive count incremented. Use long polling (WaitTimeSeconds up to 20) to avoid paying for empty responses.
  3. DeleteMessage with the latest receipt handle: gone. Otherwise it reappears when the timeout expires.
  4. With a redrive policy, once the receive count exceeds maxReceiveCount, SQS moves the message to a dead-letter queue (DLQ) instead.

Try it

A simulated clock with one message, a 30-second visibility timeout and maxReceiveCount 3. Consumer A is slower than the timeout, B and C crash.

⌘/Ctrl + Enter
OutputNot run yet

Press Run to see what your code prints, in order.

Runs in this browser, in an isolated frame with no access to this page. console.log, info, warn and error are shown; code runs as a classic script, so use an async function for await.

t= 0s  A receives order-42 (receive 1), hidden until t=30t=30s  B receives order-42 (receive 2), hidden until t=60t=40s  B crashes without deleting itt=45s  A finishes late; its receipt handle is stale, and another consumer already has the messaget=60s  C receives order-42 (receive 3), hidden until t=90t=65s  C crashes without deleting itt=90s  order-42 was received 3 times: moved to the DLQ

A did the work, but too late: the side effects happened, and the message was handed out again. Change visibilityTimeout to 60 and A deletes it cleanly.

Standard or FIFO

Standard queues support a nearly unlimited request rate, deliver at least once, and make a best effort at ordering. FIFO queues (names end in .fifo) deliver in order within each message group ID, and deduplicate messages with the same deduplication ID within 5 minutes. Without high-throughput mode they handle 300 API calls per second per action (3,000 messages with batches of 10); high-throughput mode raises that a lot, depending on the Region. A failing message blocks the rest of its group until it succeeds or reaches the DLQ.

Settings that prevent incidents

  • Visibility timeout above worst-case processing time; for a Lambda trigger, AWS recommends at least six times the function's timeout.
  • Always a DLQ, with an alarm on its depth (and on ApproximateAgeOfOldestMessage in the main queue).
  • Idempotent consumers, keyed on a business ID, because duplicates will happen on standard queues.
  • Batch sends, receives and deletes (up to 10 messages) to cut cost and API calls.
2

Check yourself

3 questions. Get them all right to finish this topic.

  1. 1.Processing usually takes 20 seconds but sometimes 50. The visibility timeout is the default 30 seconds. What goes wrong?
  2. 2.Why can a standard queue deliver the same message twice even when nothing timed out?
  3. 3.What does maxReceiveCount = 5 on a redrive policy do?

0 of 3 right so far. Get all three to finish this topic.

Go deeper