sdlc.dev

QA automation path · Foundations · Topic 4 of 12

Coverage, its limits, and mutation testing

Not started

Coverage tells you which lines and branches ran during the tests. It is good at finding code that no test touches. It can't tell you whether the code that ran was actually checked. Mutation testing goes further: it plants small bugs in the code and counts how many of them your tests catch.

Why it matters: A coverage number is easy to raise without catching a single extra bug. Mutation testing measures what coverage can't: whether your tests notice when the code is wrong.

  1. 1Coverage finds gaps

    A line no test runs is definitely untested. That is the useful part of a coverage report.

  2. 2It can't judge quality

    A line that ran may have been checked by nothing. A test with no assertions still adds coverage.

  3. 3Mutants

    Mutation testing makes small changes to the code, such as >= to >, and reruns the tests. If every test still passes, the mutant survived, and you have found a gap.

  4. 4Use it where it matters

    Mutation runs are slow, so aim them at critical modules or at the files a change touches.

1

How it works

The explanation, with examples you can run.

100% coverage that misses a bug

Take isAdult(age), implemented as age >= 18. Two tests, ages 30 and 10, run every line and both outcomes, so coverage is 100%. Now run them against a few mutants, the same kind of changes a mutation tool makes:

⌘/Ctrl + Enter
OutputNot run yet

Press Run to see what your code prints, in order.

Runs in this browser, in an isolated frame with no access to this page. console.log, info, warn and error are shown; code runs as a classic script, so use an async function for await.

The weak suite lets age > 18 survive. It can't tell an 18-year-old from a 17-year-old, which is exactly the kind of off-by-one bug that reaches production. Two boundary tests kill it. The coverage number was 100% both times.

Reading coverage well

  • Prefer branch coverage to line coverage. A one-line a && b ? x : y counts as covered after one run, whichever path that run took.
  • Look at what is uncovered and ask whether it worries you: error handling, rarely used options, the retry path. That conversation is worth more than the percentage.
  • Expect thoughtful testing to land high (Fowler mentions the upper 80s or 90s), but be wary of a hard 100% gate. It rewards tests written for the number.
  • Ratchet rather than gate: fail the build if coverage on changed files drops, not if the whole repo misses a target.

Mutation testing in practice

For JavaScript and TypeScript, Stryker runs your existing test command once per mutant. It changes operators, flips conditions, empties blocks and replaces return values, then reports each mutant as killed (some test failed), survived (all passed), or not covered (no test reached it). The mutation score is the share of mutants your tests detect.

npm install --save-dev @stryker-mutator/corenpx stryker init   # asks about your test runner and writes a config filenpx stryker run    # HTML report with each surviving mutant in place

Because the suite runs once per mutant, a full run on a large codebase can take a long time. Scope it to the modules where a wrong answer is expensive (pricing, permissions, money) and run it nightly or on changed files. Some survivors are equivalent mutants: changes that behave the same as the original. Ignore those; the others usually point to a missing boundary test or a missing assertion.

2

Check yourself

3 questions. Get them all right to finish this topic.

  1. 1.A test calls parseInvoice(sample) and asserts nothing. What does the coverage report show?
  2. 2.Stryker reports that the mutant changing a > b to a >= b survived. What does that mean?
  3. 3.Management sets a hard 100% line-coverage gate. What usually happens?

0 of 3 right so far. Get all three to finish this topic.

Go deeper