sdlc.dev

Redis path · Keeping it running · Topic 8 of 11

Persistence: RDB, AOF and what you can lose

Not started

Redis keeps the dataset in memory and replies to a write as soon as it has changed memory. Whether that write survives a crash depends on persistence, which comes in two forms: RDB snapshots of the whole dataset at intervals, and the AOF, an append-only file of every write. You can use either, both, or neither (for a pure cache, restarting empty may be fine). Each choice has a clear answer to "what do I lose if the process dies now?".

Why it matters: Redis acknowledges a write before it is on disk; how much you lose in a crash is a setting you should choose on purpose.

  1. 1RDB is a snapshot

    A forked child writes the whole dataset to a compact file every so often; a crash loses everything since the last snapshot.

  2. 2AOF is a log of writes

    Every write command is appended to a file; with appendfsync everysec, a crash loses about a second.

  3. 3Fork has a cost

    Snapshots and AOF rewrites fork the process; with a large dataset the fork can pause Redis, and copy-on-write needs spare memory.

  4. 4Persistence is not replication

    Disk files protect against a restart; they do not keep you running when the machine dies.

1

How it works

The explanation, with examples you can run.

RDB snapshots

Redis forks a child process that writes the dataset to a compact binary file (dump.rdb) while the parent keeps serving. A Redis 8 server with no configuration file reports these save points:

> CONFIG GET save1) "save"2) "3600 1 300 100 60 10000"

That means: snapshot after 3600 seconds if at least 1 key changed, after 300 seconds if 100 changed, after 60 seconds if 10,000 changed. Between snapshots nothing reaches the disk, so a crash can lose minutes of writes. In exchange, RDB files are small, fast to load at restart, and convenient for backups: copy the file somewhere else.

BGSAVE takes a snapshot on demand; SAVE does it on the main thread and blocks everyone, so avoid it.

AOF: the append-only file

With appendonly yes, every command that changes data is appended to a log, and replayed at startup. How often the log is forced to disk with fsync sets the loss window:

appendfsyncLoss on a crashCost
alwaysPractically noneAn fsync before replying; much slower
everysec (default)About one secondSmall
noWhatever the operating system has not flushed, often tens of secondsLeast

The log grows, so Redis rewrites it in the background into a minimal form. Since Redis 7, the AOF is multi-part: a base file (by default in RDB format, aof-use-rdb-preamble yes) plus incremental files, tracked by a manifest in the appenddirname directory.

Many deployments enable both: AOF for a short loss window, RDB for backups and fast restarts. When AOF is on, Redis uses it at startup, because it is the more complete of the two.

The fork and its memory

Both snapshots and AOF rewrites fork the process. The child shares memory with the parent through copy-on-write: pages are only copied when the parent changes them. With a write-heavy workload during a long save, much of the dataset may be copied, so the machine needs spare memory beyond the dataset itself, and Linux should be configured with vm.overcommit_memory = 1 as the Redis docs advise. Forking a process with a very large memory footprint also takes time, during which the main thread is paused; on some virtual machines this was notably slow.

What persistence does not do

Disk files protect against a restart. They do not keep the service available while a machine is replaced, and a disk that dies with the machine takes its files with it. For that you need replicas, and for backups you need copies elsewhere. And with any setting other than appendfsync always, an acknowledged write can still be lost, which matters for anything Redis is the only home for.

2

Check yourself

3 questions. Get them all right to finish this topic.

  1. 1.Redis runs with only the default RDB snapshots (save 3600 1 300 100 60 10000). The process is killed. What can be lost?
  2. 2.AOF is on with appendfsync everysec, the default. How much data can a server crash lose?
  3. 3.A Redis instance using 24 GB on a 32 GB machine starts a background save during heavy writes and is killed by the OOM killer. Why?

0 of 3 right so far. Get all three to finish this topic.

Go deeper