sdlc.dev

Kubernetes path · Keeping it healthy · Topic 9 of 13

Requests, limits, QoS, throttling and OOMKills

Not started

Each container can declare **requests**, what it needs reserved, and **limits**, the most it may use, for CPU and memory. Requests drive scheduling: a pod only lands on a node with enough unrequested capacity. Limits are enforced by the kernel through the same cgroup files you saw with docker run --memory --cpus. CPU and memory behave very differently at the limit, and that difference explains most resource incidents.

Why it matters: Requests decide where pods fit and limits decide how they fail. Most latency spikes and surprise restarts in Kubernetes trace back to these four numbers.

  1. 1Requests reserve

    The scheduler places pods by their requests, and under contention CPU is shared in proportion to them.

  2. 2Limits cap

    Over the CPU limit the container is throttled (paused until the next period); over the memory limit it is OOM-killed.

  3. 3QoS decides who goes first

    Guaranteed (requests equal limits), Burstable (some requests) and BestEffort (none). Under node pressure, BestEffort pods are evicted first.

  4. 4Resize without restarting

    In-place pod resize, stable since Kubernetes 1.35, changes a running container's CPU and memory without recreating the pod.

1

How it works

The explanation, with examples you can run.

Writing them down

containers:  - name: app    image: ghcr.io/acme/web:1.4.2    resources:      requests:        cpu: 250m          # what the scheduler reserves: a quarter of a core        memory: 256Mi      limits:        memory: 256Mi      # above this the container is OOM-killed        # no CPU limit: use spare CPU instead of being throttled

cpu: 250m means 250 millicores, a quarter of one core. Memory uses binary units (Mi, Gi).

CPU: shared by requests, throttled by limits

When the node is busy, CPU time is divided between containers in proportion to their requests. A CPU limit is a hard quota per scheduling period (by default 100 ms). With limits.cpu: 500m, the container gets 50 ms of CPU time per 100 ms, across all its threads. A request that needs 80 ms of CPU in a burst uses the 50 ms, is paused for the rest of the period, and finishes in the next one. Average utilisation can look low while p99 latency suffers. Multi-threaded runtimes (JVM, Go, Node worker pools) burn through a quota quickly.

Many teams therefore set CPU requests on everything but leave CPU limits off for latency-sensitive services, so they can use idle CPU, and rely on requests for fairness. If you do set CPU limits, watch the throttling metrics (container_cpu_cfs_throttled_periods_total).

Memory: no slowing down, only killing

Memory cannot be throttled. When a container goes over its memory limit, the kernel's OOM killer terminates a process in it, and the kubelet restarts the container:

kubectl describe pod web-6d4c...#   Last State:  Terminated#     Reason:    OOMKilled#     Exit Code: 137

Set memory requests equal to limits for predictable services; a memory request lower than the limit lets the pod overcommit the node, and if the node runs short, the kubelet evicts pods to protect itself. Make sure the runtime knows its limit: modern JVMs and Go (GOMEMLIMIT) read or accept it; Node's heap size may need --max-old-space-size.

QoS classes and eviction

Kubernetes derives a QoS class from these settings:

ClassRuleEvicted under node pressure
GuaranteedEvery container has CPU and memory requests equal to limitsLast
BurstableAt least one request or limit, not GuaranteedMiddle, roughly by how far over their requests they are
BestEffortNo requests or limits at allFirst

Production workloads should never be BestEffort: they are scheduled without any reservation and are the first to go.

Getting the numbers right, and changing them live

Start from measured usage (for example the p95 of memory and CPU over a week, plus headroom), not guesses; Vertical Pod Autoscaler in recommendation mode can suggest values. Namespace ResourceQuota and LimitRange objects enforce totals and defaults per team.

Changing resources used to mean replacing pods. In-place pod resize is stable since Kubernetes 1.35: you patch a running pod's resize subresource and the kubelet adjusts the cgroups, with a per-resource resizePolicy saying whether that resource can change without restarting the container. Controllers such as VPA can use it to right-size workloads with less disruption.

2

Check yourself

3 questions. Get them all right to finish this topic.

  1. 1.A container with limits.memory 256Mi allocates 300 MiB. What happens?
  2. 2.A service has limits.cpu: 500m and average CPU use of 30%, yet its p99 latency is bad. What is a likely cause?
  3. 3.Which pod gets the Guaranteed QoS class?

0 of 3 right so far. Get all three to finish this topic.

Go deeper