sdlc.dev

QA automation path · Trust at scale · Topic 10 of 12

Running tests in CI

Not started

In CI, every change runs the whole suite on a clean machine. To be useful it has to be fast enough that people wait for it, and trustworthy enough that a red build means a real problem. Parallel workers and shards take care of speed. Trust comes from how you handle retries and flaky tests.

Why it matters: A suite only protects you if it runs on every change, finishes while the author is still waiting, and is trusted when it goes red.

  1. 1Fastest checks first

    Types, lint and unit tests in minutes; integration and end-to-end after. Fail early on the cheap stuff.

  2. 2Parallel, then sharded

    Workers use the cores of one machine. Shards split the suite across machines, and the reports are merged at the end.

  3. 3Retries hide bugs

    A retry turns a red build green without fixing anything. Report every test that only passed on retry, and track how often it happens.

  4. 4Quarantine with a limit

    Move known-flaky tests out of the gate so the rest stays trusted, but cap how many and for how long.

1

How it works

The explanation, with examples you can run.

Order and speed

Run checks from cheapest to most expensive and stop at the first failure: type check and lint, then unit tests, then integration tests with containers, then end-to-end tests against a preview build. Cache dependencies, and with Playwright install only the browsers you use (npx playwright install --with-deps chromium).

Playwright runs test files in parallel workers on one machine (--workers, 50% of logical cores by default). For more speed, shard across machines. Each job runs one slice with --shard=1/4 and so on. Use the blob reporter, then merge the slices into one report:

# .github/workflows/e2e.yml (the test job)jobs:  e2e:    runs-on: ubuntu-latest    strategy:      fail-fast: false      matrix:        shard: [1/4, 2/4, 3/4, 4/4]    steps:      - uses: actions/checkout@v4      - uses: actions/setup-node@v4        with: { node-version: 22, cache: npm }      - run: npm ci      - run: npx playwright install --with-deps chromium      - run: npx playwright test --shard=${{ matrix.shard }} --reporter=blob      - uses: actions/upload-artifact@v4        if: ${{ !cancelled() }}        with:          name: blob-report-${{ strategy.job-index }}          path: blob-report# A later job downloads all blob-report-* artifacts into one folder and runs:#   npx playwright merge-reports --reporter html ./all-blob-reports

Set fullyParallel: true so the shards are balanced test by test rather than file by file.

What retries hide

Retries are tempting: set retries: 2 and the build goes green. But a retry can't tell a flaky test from a flaky product. Here a checkout fails one call in three, and every CI run still ends green:

⌘/Ctrl + Enter
OutputNot run yet

Press Run to see what your code prints, in order.

Runs in this browser, in an isolated frame with no access to this page. console.log, info, warn and error are shown; code runs as a classic script, so use an async function for await.

If you use retries, make them visible. Playwright already reports such tests as flaky. Keep the trace from the first retry (trace: "on-first-retry"), chart the flaky rate over time, and consider --fail-on-flaky-tests on the main branch. Never retry unit tests: they have no excuse to be non-deterministic.

Quarantine, with limits

When a flaky test can't be fixed today, move it out of the gating suite (a tag, a separate project, or test.fixme with a ticket) so everyone else can trust the build again. A quarantined test protects nothing, so give quarantine limits: an owner, a ticket, a maximum number of tests and a maximum time (Fowler suggests something like a week). When a limit is reached, fixing the quarantined tests comes before new work.

Make failures easy to act on

Upload reports, traces, screenshots and container logs as artifacts for every failed job. Show the failing test's name and first error in the pull request. Re-run only the failures with --last-failed while you debug, and on large suites use --only-changed for a quick first pass before the full run.

2

Check yourself

3 questions. Get them all right to finish this topic.

  1. 1.In Playwright, a test fails on the first try and passes on the retry. How is it reported?
  2. 2.Why can retries be harmful even when every flaky test really is just a test problem?
  3. 3.You shard a Playwright suite four ways without fullyParallel. How are tests divided?

0 of 3 right so far. Get all three to finish this topic.

Go deeper