sdlc.dev

Kafka path · Writing and reading · Topic 3 of 12

Producers: acks, batching, idempotence

Not started

A producer turns send() calls into batches, sends each batch to the leader of its partition and waits for an acknowledgement. Three groups of settings matter: how many replicas must confirm a write (acks), how records are grouped (batching), and what happens when a send fails (retries and idempotence). The modern defaults are safe; the risk is usually code or old configuration that overrides them.

Why it matters: The producer settings decide whether an acknowledged record can be lost, whether retries create duplicates, and how much latency you trade for throughput. The defaults changed in 3.0 and again in 4.0.

  1. 1acks sets who must confirm

    acks=0 waits for nothing, acks=1 for the partition leader, acks=all for every in-sync replica. all has been the default since 3.0.

  2. 2Records travel in batches

    The producer groups records per partition up to batch.size (16 KB) or linger.ms (5 ms since 4.0, 0 before), and compresses whole batches if asked.

  3. 3Retries are on by default

    Failed sends are retried until delivery.timeout.ms (2 minutes) runs out, so a transient broker problem usually costs latency, not data.

  4. 4Idempotence removes retry duplicates

    With enable.idempotence (default true) the broker drops a batch it has already written, using a producer id and per-partition sequence numbers.

1

How it works

The explanation, with examples you can run.

acks

acksThe producer waits forAn acknowledged record can be lost if
0Nothing; offsets come back as -1Anything goes wrong
1The leader to write it to its logThe leader fails before a follower copies it
all (-1)Every in-sync replicaAll in-sync replicas fail together

acks=all has been the default since Kafka 3.0. It is only as strong as the in-sync replica set, which is why it is paired with min.insync.replicas (the replication topic).

Batching

The producer keeps a buffer per partition. A batch is sent when it reaches batch.size (16,384 bytes by default) or has waited linger.ms. Kafka 4.0 changed the linger.ms default from 0 to 5 ms because larger batches usually lower overall latency by reducing the work per record. Compression (compression.type: none by default; lz4 or zstd are common choices) is applied per batch, so better batching also compresses better.

Retries and idempotence

retries defaults to the maximum integer, bounded in time by delivery.timeout.ms (120,000 ms): the total time send() may take, including waiting in the buffer and every retry. When it expires, the send fails and your callback must decide what to do.

Retrying is where duplicates come from: the broker may have written the batch and only the response was lost. The idempotent producer (on by default since 3.0, and required for transactions) fixes this. The broker gives each producer an id, the producer numbers its batches per partition, and the broker refuses to append a sequence number it has already written. It also keeps batches in order with up to 5 requests in flight per connection, which is why idempotence requires max.in.flight.requests.per.connection of 5 or less.

A simplified model of the broker's check:

⌘/Ctrl + Enter
OutputNot run yet

Press Run to see what your code prints, in order.

Runs in this browser, in an isolated frame with no access to this page. console.log, info, warn and error are shown; code runs as a classic script, so use an async function for await.

The real broker tracks sequences per producer id and partition, for the last five batches, and the producer id changes when a non-transactional producer restarts.

A sensible starting configuration

acks=allenable.idempotence=truecompression.type=lz4linger.ms=5delivery.timeout.ms=120000

These are mostly the defaults made explicit, which protects you from a library or an old config file silently changing them. Note that a conflicting setting (for example acks=1) quietly disables idempotence unless you set enable.idempotence=true explicitly, in which case the producer refuses to start.

2

Check yourself

3 questions. Get them all right to finish this topic.

  1. 1.A producer with acks=1 gets an acknowledgement, then the leader's disk fails before followers copy the record. What happens?
  2. 2.The broker wrote a batch but the response timed out, so the producer retries. With idempotence enabled, what ends up in the partition?
  3. 3.Why did Kafka 4.0 change the default linger.ms from 0 to 5?

0 of 3 right so far. Get all three to finish this topic.

Go deeper