sdlc.dev

Backend engineer path · Running it in production · Topic 16 of 18

Testing in production, safely

Not started

Testing in production means checking new code against real traffic, while limiting how many users a problem can reach. Three tools do most of the work: feature flags turn a change on for a few users at a time, canary releases send a small share of traffic to the new version and compare it with the old, and synthetic checks run scripted journeys against the live system around the clock. None of them replaces the tests that run before a deploy.

Why it matters: Production has traffic, data and configuration that no test environment copies exactly. Some bugs only show up there, so limit how many users they can reach and find them before users report them.

  1. 1Not instead of, as well as

    Tests only cover scenarios someone thought of. Production checks catch the rest, but they don't replace the suite.

  2. 2Separate deploy from release

    Feature flags ship code turned off, then turn it on for staff, then 1%, then everyone, and can turn it off again in seconds.

  3. 3Canaries compare

    Send a small share of traffic to the new version, compare its errors and latency with the old one, and roll back automatically if it is worse.

  4. 4Synthetic checks

    A script runs a key journey against production every few minutes, as a dedicated test user, and alerts when it fails.

1

How it works

The explanation, with examples you can run.

Rolling out with a flag

A percentage rollout must give the same answer for the same user every time, or a user would flip between old and new on every page. Hashing the user id into a bucket does that:

⌘/Ctrl + Enter
OutputNot run yet

Press Run to see what your code prints, in order.

Runs in this browser, in an isolated frame with no access to this page. console.log, info, warn and error are shown; code runs as a classic script, so use an async function for await.

Including the flag name in the hash keeps the same users from being first in line for every experiment. The allow-list lets staff and test accounts see the feature before anyone else. Raise percent in steps while watching error rates. To roll back, set it to 0; there is no redeploy.

Testing flags before they reach production: run the suite with the configuration that will be live (new flag on) and with the fallback (new flag off). Then remove the flag once the rollout is done, because every flag left in the code doubles the paths someone has to reason about.

Canary releases

A canary deploys the new version next to the old one and routes a small slice of traffic to it: a random 1%, staff first, or a single region. Compare the canary with the baseline on the same signals you load test with (error rate, p95 and p99 latency, saturation) and on business signals such as checkouts per minute. Automate the decision: tools like Argo Rollouts and Flagger, or your cloud's deployment service, can step traffic up and roll back when a metric crosses a limit.

Synthetic checks

A synthetic check is a short script that runs a critical journey against production every few minutes, from outside your network, and pages someone when it fails. A Playwright test works well:

import { expect, test } from "@playwright/test" test("search to product page works in production", async ({ page }) => {  await page.goto("https://shop.example.com")  await page.getByRole("searchbox", { name: "Search" }).fill("mug")  await page.getByRole("searchbox", { name: "Search" }).press("Enter")  await page.getByRole("link", { name: /mug/i }).first().click()  await expect(page.getByRole("button", { name: "Add to cart" })).toBeEnabled()})

Doing it safely

  • Use dedicated test accounts that are marked as synthetic, so their traffic is left out of analytics, emails and billing.
  • Never move real money. Use the payment provider's test mode for those accounts, or stop the journey before payment.
  • Make checks read-only where you can. Where they write, clean up, and make it obvious which data is test data.
  • Keep a kill switch for every risky change, and practise using it.
2

Check yourself

3 questions. Get them all right to finish this topic.

  1. 1.A canary on 5% of traffic shows twice the error rate of the baseline. What should happen?
  2. 2.Which flag configurations does Pete Hodgson suggest testing, since testing every combination is impractical?
  3. 3.What does a synthetic check catch that unit and integration tests can't?

0 of 3 right so far. Get all three to finish this topic.

Go deeper