sdlc.dev

CI/CD and DevOps path · Measure and run · Topic 13 of 13

SLOs, error budgets and on-call

Not started

Fast delivery is only good if the service stays usable. Site reliability engineering gives you a way to put a number on "usable": a **service level objective (SLO)**. Whatever the SLO doesn't require is the **error budget**, room for failure that you can spend on releases, experiments and maintenance. When it runs out, reliability work comes first.

Why it matters: 100% reliability is impossible and chasing it stops all change. An SLO says how much unreliability users will tolerate, and the error budget turns that into a rule for when to ship and when to slow down.

  1. 1SLI, SLO, SLA

    An SLI is a measurement (the share of requests served successfully and fast). An SLO is a target for it over a window. An SLA is a contract with consequences, usually looser than the SLO.

  2. 2The budget is what's left

    A 99.9% SLO over 30 days allows 0.1% of requests, or about 43 minutes of full outage, to fail. That is the error budget.

  3. 3The budget sets the pace

    Budget left: ship. Budget burned: freeze risky changes and spend effort on reliability until it recovers.

  4. 4Alert on burn rate

    Page a human when the budget is being consumed fast enough to run out soon, not on every spike in a graph.

1

How it works

The explanation, with examples you can run.

SLI, SLO, SLA

  • An SLI (indicator) is a ratio of good events to all events, measured where users feel it: the share of HTTP requests that return non-5xx in under 300 ms, the share of jobs that finish within an hour.
  • An SLO (objective) is a target for an SLI over a window: 99.9% over a rolling 28 or 30 days.
  • An SLA (agreement) is a contract with refunds or penalties. Set it looser than the SLO, so you notice trouble before it costs money.

Pick SLOs from what users need, not from what the system happens to do today. Few users notice the difference between 99.99% and 99.999%, but the second costs far more.

The error budget

⌘/Ctrl + Enter
OutputNot run yet

Press Run to see what your code prints, in order.

Runs in this browser, in an isolated frame with no access to this page. console.log, info, warn and error are shown; code runs as a classic script, so use an async function for await.

A burn rate of 1 uses exactly the whole budget by the end of the window. The SRE Workbook suggests paging at fast burns (around 14.4× over an hour, which spends 2% of a 30-day budget in that hour) and filing a ticket for slow ones, using two windows so alerts both fire quickly and stop quickly.

An error budget policy

Agree in writing, before anything goes wrong, what happens as the budget runs down. For example: if the budget is exhausted, pause feature releases except security fixes, and put the next sprint into the top causes of failure; if a single incident burns more than a fifth of the budget, it gets a postmortem. The policy turns arguments about "are we being too risky?" into a shared rule.

On-call that people can sustain

  • Page on symptoms, like SLO burn, and route causes (disk filling, CPU) to tickets and dashboards.
  • Every page should need a human now. If the right response is "wait and see", it shouldn't page.
  • Runbooks for each alert: what it means, how to check, how to mitigate, and how to roll back the last deploy.
  • Blameless postmortems that end in actions on the system, not on a person.
  • Watch the load. Too many pages per shift is itself an incident to fix.
2

Check yourself

3 questions. Get them all right to finish this topic.

  1. 1.An SLO is 99.9% of requests successful over 30 days. The service handles 10 million requests in that window. How many failed requests does the error budget allow?
  2. 2.Halfway through the month, 90% of the error budget is gone. Under a typical error budget policy, what happens?
  3. 3.Why page on error budget burn rate rather than on CPU above 80%?

0 of 3 right so far. Get all three to finish this topic.

Go deeper