sdlc.dev

AWS fundamentals path · Running it · Topic 16 of 18

Observability: CloudWatch, X-Ray and OpenTelemetry

Not started

**CloudWatch** is where AWS services report on themselves. Every service publishes **metrics** (an ALB's request count and 5xx errors, an SQS queue's depth, a Lambda function's duration and throttles), most services can write **logs** to CloudWatch Logs, and **alarms** watch metrics and act when they cross thresholds. Tracing, following one request across services, comes from **X-Ray** or, increasingly, **OpenTelemetry** sending to CloudWatch. The data is mostly there by default; the judgement about what to watch is not.

Why it matters: Every AWS service emits metrics and can write logs, but nothing alerts you until you create alarms. Knowing what is there by default saves you from finding out from your users.

  1. 1Metrics come free, alarms do not exist yet

    AWS services publish metrics to CloudWatch automatically. You still have to decide which ones matter and alarm on them.

  2. 2Logs go to log groups

    Lambda, ECS, API Gateway and agents write to CloudWatch Logs. Retention defaults to never expire, so set it.

  3. 3Traces connect the hops

    Distributed tracing follows one request through services. AWS now recommends OpenTelemetry; the X-Ray SDKs entered maintenance mode in February 2026.

  4. 4Alarm on symptoms

    Alarm on what users feel (error rate, latency, queue age) rather than every CPU spike.

1

How it works

The explanation, with examples you can run.

Metrics

A metric is a time series in a namespace (AWS/SQS), identified by dimensions (QueueName=orders). AWS metrics are standard resolution (one-minute points or coarser); custom metrics can be high resolution, down to one second. Publish your own business metrics with PutMetricData, or more cheaply by writing structured log lines in the Embedded Metric Format, which CloudWatch turns into metrics. Watch dimension cardinality: each unique combination is a separately billed metric.

Logs

Logs land in log groups (one per function or service) and log streams. Logs Insights queries them with a pipe-based language; metric filters turn matching lines into metrics; subscription filters stream them to Lambda, Firehose or OpenSearch. Two habits pay off immediately: log JSON, so fields are queryable, and set retention on every group, because the default is to keep logs forever.

fields @timestamp, level, orderId, msg| filter level = "error"| stats count() by bin(5m)

Alarms

An alarm evaluates a metric (or a metric math expression, or an anomaly detection band) over periods: for example, "5xx rate above 1 % for 3 of 5 one-minute periods". It is in OK, ALARM or INSUFFICIENT_DATA, and on state change it can notify an SNS topic, scale an Auto Scaling group or run an action. Composite alarms combine others to cut noise. Good first alarms for a typical service:

  • ALB HTTPCode_Target_5XX_Count relative to requests, and TargetResponseTime p99.
  • UnHealthyHostCount above zero for several minutes.
  • SQS ApproximateAgeOfOldestMessage, and any messages in a DLQ.
  • Lambda Errors, Throttles, and duration near the timeout.
  • RDS FreeStorageSpace, DatabaseConnections and CPU credit balance on burstable instances.

Traces

A trace follows one request across the load balancer, services, queues and databases, so you can see which hop was slow. X-Ray stores and visualises traces; AWS announced that the X-Ray SDKs and daemon entered maintenance mode on 25 February 2026 and recommends instrumenting with OpenTelemetry (the AWS Distro for OpenTelemetry, or the CloudWatch agent as collector). CloudWatch Application Signals builds service maps and SLOs from that telemetry.

Also turn on CloudTrail in every account: it records API calls (who changed the security group, who deleted the bucket), which is an audit log rather than application telemetry, and invaluable after an incident.

2

Check yourself

3 questions. Get them all right to finish this topic.

  1. 1.Your Lambda function's logs from two years ago are still being stored and billed. Why?
  2. 2.Which alarm best catches a stuck SQS consumer?
  3. 3.An alarm on a metric that only exists when errors happen sits in INSUFFICIENT_DATA most of the time. What should you configure?

0 of 3 right so far. Get all three to finish this topic.

Go deeper