sdlc.dev

Kubernetes path · Keeping it healthy · Topic 8 of 13

Probes: liveness, readiness and startup

Not started

The kubelet can check each container in three ways, and each check has a different consequence. A **liveness** probe that keeps failing gets the container killed and restarted. A **readiness** probe that fails takes the pod out of Service endpoints, so it gets no new traffic, without touching the container. A **startup** probe suspends the other two until the application has started. Getting these right is the difference between self-healing and self-harm.

Why it matters: Probes are how Kubernetes decides to restart a container or stop sending it traffic. Configured badly, they turn a slow database into a cluster-wide restart storm.

  1. 1Liveness restarts

    If the liveness probe fails failureThreshold times in a row, the kubelet kills and restarts the container.

  2. 2Readiness routes

    A failing readiness probe removes the pod from Service endpoints. The container keeps running, and returns to traffic when the probe passes.

  3. 3Startup waits

    A startup probe holds off liveness and readiness until it succeeds once, giving slow starters time without weakening liveness.

  4. 4Liveness must be local

    Never make liveness depend on a database or another service: when that dependency is slow, every replica is restarted at once.

1

How it works

The explanation, with examples you can run.

A sensible configuration

containers:  - name: app    image: ghcr.io/acme/web:1.4.2    ports:      - { name: http, containerPort: 3000 }    startupProbe:              # up to 30 x 5 s = 150 s to start; the others wait for it      httpGet: { path: /healthz, port: http }      periodSeconds: 5      failureThreshold: 30    readinessProbe:            # failing = taken out of the Service, not restarted      httpGet: { path: /ready, port: http }      periodSeconds: 5      failureThreshold: 2    livenessProbe:             # failing = container restarted; keep it cheap and local      httpGet: { path: /healthz, port: http }      periodSeconds: 10      timeoutSeconds: 2      failureThreshold: 3

Probes can be httpGet (any 2xx or 3xx status passes), tcpSocket, grpc or exec. The defaults are periodSeconds: 10, timeoutSeconds: 1, failureThreshold: 3, successThreshold: 1 and initialDelaySeconds: 0. Without a readiness probe, a container counts as ready as soon as it starts.

What each endpoint should check

  • /healthz (liveness): "is this process able to make progress?" Return 200 if the event loop or request threads respond. Do not check the database, the cache or other services. If the database is down, restarting your app will not bring it back, and if all replicas share the dependency, all of them restart together.
  • /ready (readiness): "should I get traffic right now?" False while warming caches, during shutdown, or when the pod is overloaded. Be careful here too with shared dependencies: if every pod reports not-ready because the database is slow, the Service has no endpoints and every request fails fast with nothing behind it. Often it is better to stay ready and return errors or degraded responses.
  • Startup: the same check as liveness, with a budget (failureThreshold x periodSeconds) that covers the slowest realistic start.

How bad probes cause outages

  • Liveness checks a dependency: a database blip becomes a fleet-wide restart, followed by a reconnection storm that keeps the database down.
  • Timeout too tight: under load, a probe that usually takes 50 ms takes 1.2 s, fails three times, and the busiest pods are restarted exactly when they are needed, pushing their load onto the others, which then fail too.
  • No startup probe for a slow starter: liveness kills the container before it finishes starting, forever (CrashLoopBackOff).
  • Expensive probes: a probe that runs a query every few seconds on every pod is load you created yourself.

Probes and shutdown

When a pod is deleted, it is removed from endpoints and its containers receive SIGTERM at about the same time, and proxies on other nodes may take a moment to notice. Keep serving while you drain: fail readiness, keep handling in-flight and late-arriving requests for a few seconds, then exit. A short preStop hook (for example the built-in sleep action) is a common way to delay SIGTERM until endpoints have updated.

2

Check yourself

3 questions. Get them all right to finish this topic.

  1. 1.Your liveness probe calls /health, which checks the database. The database is slow for two minutes. What happens?
  2. 2.A Java service needs up to 3 minutes to start. What is the right fix for liveness restarts during start-up?
  3. 3.What are the default probe timing values?

0 of 3 right so far. Get all three to finish this topic.

Go deeper