sdlc.dev

Linux path · The kernel boundary · Topic 7 of 12

System calls and strace

Not started

User programs on Linux run without direct access to hardware or to other processes' memory. Whenever they need something only the kernel can do, such as opening a file, reading from a socket, mapping memory or starting a process, they make a system call: a controlled jump into the kernel with a call number and a few arguments, returning a result or an error code. Every library and runtime, from the C library to the JVM, is built on these few hundred calls, and strace lets you watch them.

Why it matters: When a program hangs, can't find a file or is mysteriously slow, watching its system calls usually shows why in seconds, without reading its code.

  1. 1Programs can't touch hardware

    Opening files, sending packets, allocating memory and starting processes all go through system calls into the kernel.

  2. 2Libraries wrap them

    fopen, print and HTTP clients end up as openat, write, connect and friends. There are a few hundred system calls in total.

  3. 3Failures come back as errno

    A failed call returns -1 and an error code such as ENOENT, EACCES or EMFILE, which is often the real cause behind a vague error.

  4. 4strace shows them live

    It prints every call, its arguments and result. -f follows children, -e filters, -c counts, -T times each call.

1

How it works

The explanation, with examples you can run.

Watch cat read a file

echo hello > hi.txtstrace -e trace=openat,read,write,close cat hi.txt 2>&1 >/dev/null | tail -8
openat(AT_FDCWD, "hi.txt", O_RDONLY)    = 3read(3, "hello\n", 131072)              = 6write(1, "hello\n", 6)                  = 6read(3, "", 131072)                     = 0close(3)                                = 0close(1)                                = 0close(2)                                = 0+++ exited with 0 +++

(strace writes to stderr, so 2>&1 >/dev/null sends the trace through the pipe and drops cat's own output.) The earlier lines, cut off by tail, are the dynamic loader opening the C library. Then cat opens the file and gets descriptor 3, reads up to 128 KiB and gets 6 bytes, writes them to descriptor 1, reads again and gets 0 (end of file), and closes. That is the whole program, as the kernel sees it.

Find the file a program is looking for

strace -f -e trace=openat python3 -c 'open("/etc/app/config.yaml")' 2>&1 | grep config.yaml
openat(AT_FDCWD, "/etc/app/config.yaml", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)FileNotFoundError: [Errno 2] No such file or directory: '/etc/app/config.yaml'

A failed call returns -1 and sets errno; strace prints its name. The ones you meet most: ENOENT (no such file), EACCES (permission denied), EMFILE (too many open files), ECONNREFUSED (nothing listening), EAGAIN (would block, try later) and EINTR (interrupted by a signal). Real applications often swallow these into a vague message; the trace shows the actual call and path.

Count instead of print

strace -c -f ls /usr/bin > /dev/null 2> summary.txt; head -8 summary.txt
% time     seconds  usecs/call     calls    errors syscall------ ----------- ----------- --------- --------- ---------------- 33.69    0.000220         110         2           getdents64 15.47    0.000101           6        15           mmap  8.58    0.000056           7         8           mprotect  7.81    0.000051           8         6           openat  6.43    0.000042           5         8           close  6.13    0.000040           5         7           munmap

getdents64 reads directory entries; most of the rest is the loader mapping libraries. Other useful flags: -p <pid> attaches to a running process (a hung one is often sitting in futex, epoll_wait, read or connect), -T shows the time spent in each call, -tt adds timestamps, and -e trace=%network or %file selects groups of calls.

Cost and alternatives

strace uses ptrace, which stops the traced process at every system call so the tracer can inspect it. That can slow a busy program down many times over, so use it briefly. For production, eBPF-based tools (bpftrace, perf trace, the BCC tools such as opensnoop and execsnoop) trace the same events inside the kernel with far lower overhead. In a container, tracing needs CAP_SYS_PTRACE (docker run --cap-add SYS_PTRACE).

On macOS

macOS has a different kernel (XNU) with its own system calls. dtruss and dtrace can trace them, but only with System Integrity Protection partly disabled, so the practical way to learn Linux system calls on a Mac is strace inside a Linux container.

2

Check yourself

3 questions. Get them all right to finish this topic.

  1. 1.An app fails with a vague 'config error'. How can strace help without reading its code?
  2. 2.What does strace -c print?
  3. 3.Why is strace not suitable for leaving on in production?

0 of 3 right so far. Get all three to finish this topic.

Go deeper