QA automation path · Trust at scale · Topic 9 of 12

Why tests are flaky, and how to fix them

Not started

A flaky test passes and fails on the same code. The cause is nearly always something the test doesn't control: timing, data left by another test, the order tests run in, a real network call, or the clock. Retrying the test only hides the problem. The fix is to find the thing that isn't under control and bring it under control.

Why it matters: A test that fails at random teaches people to ignore red builds. Once that habit exists, the next real failure gets ignored too.

  1. 1Flaky means non-deterministic

    Same code, same test, different result. Something the test depends on isn't under its control.

  2. 2Five usual causes

    Guessing at timing, state shared between tests, dependence on order, real networks and services, and the real clock.

  3. 3Make it fail on purpose

    Repeat the test many times, shuffle the order, run it in parallel. A flake you can reproduce is half fixed.

  4. 4Fix the cause, not the symptom

    Wait for a condition instead of sleeping, build fresh state, double the remote service, inject the clock.

1

How it works

The explanation, with examples you can run.

Shared state and order

Two tests that share a module-level cart. Each is correct on its own; whether they pass depends on which runs first.

⌘/Ctrl + Enter
OutputNot run yet

Press Run to see what your code prints, in order.

Runs in this browser, in an isolated frame with no access to this page. console.log, info, warn and error are shown; code runs as a classic script, so use an async function for await.

Under a test runner, the order changes when someone adds a file, when tests run in parallel, or when a runner shuffles. The fix is to create the cart inside each test (or in a beforeEach). The same goes for database rows, singletons, environment variables, and mocks that were never restored.

Guessing at time

A fixed sleep bets on how long something takes. Waiting for the condition doesn't need to guess:

⌘/Ctrl + Enter
OutputNot run yet

Press Run to see what your code prints, in order.

Runs in this browser, in an isolated frame with no access to this page. console.log, info, warn and error are shown; code runs as a classic script, so use an async function for await.

Playwright's web-first assertions (await expect(locator).toHaveText(...)) and expect.poll() do this for you. In unit tests, fake timers remove real waiting altogether.

The usual suspects

CauseSymptomFix
TimingFails on slow machines or under loadWait for conditions, never fixed sleeps
Shared statePasses alone, fails in the suiteFresh data per test, reset before not after
OrderFails when files are added or shuffledSame fix; run shuffled to prove it
Network and servicesFails when a sandbox API is slow or downDouble it, and keep it honest with contract tests
Clock and time zoneFails at midnight, month end, DSTInject the clock, pin the time zone
Resource leaksFails after N tests, "too many connections"Close what you open; small pools fail fast
Concurrency in the productFails one run in fiftyOften a real race condition: fix the code

Reproduce it first

Run the suspect many times and in different orders before changing anything: npx playwright test --repeat-each=20 path/to/spec.ts, Jest's --randomize, or Vitest's sequence.shuffle option. A failure you can reproduce in a minute can be fixed in an afternoon. Look hard at the last row of the table: sometimes the test is right and the product has the race.

2

Check yourself

3 questions. Get them all right to finish this topic.

  1. 1.A test passes when run on its own and fails when the whole suite runs. What is the most likely cause?
  2. 2.What should replace await sleep(2000) before an assertion?
  3. 3.A test of "orders placed today" fails only between 23:00 and 00:00 on CI. Why?

0 of 3 right so far. Get all three to finish this topic.