sdlc.dev

Linux path · Processes · Topic 2 of 12

File descriptors, redirection and pipes

Not started

When a process opens a file, the kernel gives it back a file descriptor: a small integer that indexes the process's table of open files. The same table holds pipes, sockets, terminals and devices, so reading and writing look the same for all of them. By convention a program starts with three already open: 0 (standard input), 1 (standard output) and 2 (standard error). The shell's redirections and pipes simply decide what those three numbers point at before it execs the program.

Why it matters: Logs that vanish, 2>&1 in the wrong place, a pipeline that hangs: all of it is about which descriptor points where.

  1. 1A descriptor is a small number

    Each process has a table: 0 is stdin, 1 is stdout, 2 is stderr, and every file, pipe or socket it opens gets the next free number.

  2. 2The shell sets them up before exec

    Redirection (>, 2>, <) opens files and puts them at the right numbers in the child; the program just reads 0 and writes 1 and 2.

  3. 3Order matters

    Redirections are applied left to right, so > file 2>&1 sends both to the file, while 2>&1 > file does not.

  4. 4A pipe is a kernel buffer

    It connects one process's stdout to the next one's stdin. A full pipe blocks the writer; a pipe with no reader kills it with SIGPIPE.

1

How it works

The explanation, with examples you can run.

See a process's descriptors

/proc/self/fd lists the descriptors of whichever process reads it, here ls:

ls -l /proc/self/fdls -l /proc/self/fd | catls -l /proc/self/fd 2>/dev/null
lrwx------ 1 root root 64 Oct 16 10:23 0 -> /dev/pts/0lrwx------ 1 root root 64 Oct 16 10:23 1 -> /dev/pts/0lrwx------ 1 root root 64 Oct 16 10:23 2 -> /dev/pts/0lr-x------ 1 root root 64 Oct 16 10:23 3 -> /proc/3557/fd lrwx------ 1 root root 64 Oct 16 10:23 0 -> /dev/pts/0l-wx------ 1 root root 64 Oct 16 10:23 1 -> pipe:[17960]lrwx------ 1 root root 64 Oct 16 10:23 2 -> /dev/pts/0lr-x------ 1 root root 64 Oct 16 10:23 3 -> /proc/3558/fd lrwx------ 1 root root 64 Oct 16 10:23 0 -> /dev/pts/0lrwx------ 1 root root 64 Oct 16 10:23 1 -> /dev/pts/0l-wx------ 1 root root 64 Oct 16 10:23 2 -> /dev/nulllr-x------ 1 root root 64 Oct 16 10:23 3 -> /proc/3560/fd

(The total 0 lines are left out and blank lines added between runs.) At a terminal all three point at the terminal device. Piping into cat replaces only 1 with a pipe; 2>/dev/null replaces only 2. Descriptor 3 is the directory ls opened to read the listing. ls itself has no idea any of this happened.

Redirection, left to right

ls /etc/hostname /nope > out.txt 2> err.txt; echo "exit=$?"cat out.txt err.txtls /etc/hostname /nope > all.txt 2>&1; cat all.txtls /etc/hostname /nope 2>&1 > only-out.txt
exit=2/etc/hostnamels: cannot access '/nope': No such file or directoryls: cannot access '/nope': No such file or directory/etc/hostnamels: cannot access '/nope': No such file or directory

2>&1 means "make descriptor 2 a copy of whatever 1 is right now" (the shell calls dup2). In the last line 2 copies the terminal before 1 is pointed at the file, so the error still reaches the screen and only the listing goes to only-out.txt. >> appends instead of truncating, < feeds a file to stdin, and &> is bash shorthand for "both".

Pipes

a | b asks the kernel for a pipe, a buffer with a write end and a read end, then forks both processes with a's descriptor 1 on the write end and b's descriptor 0 on the read end. Both run at the same time. On Linux a pipe holds 64 KiB by default; when it is full the writer blocks until the reader catches up, which keeps a fast producer from flooding memory. If the reader exits, the next write gets SIGPIPE (or the error EPIPE), which is why yes | head -1 ends straight away instead of running forever. A pipeline's exit status is the last command's; set -o pipefail makes a failure anywhere count.

Leaks and limits

Every open descriptor counts against a per-process limit (ulimit -n, covered with resource limits). A server that opens files or sockets and forgets to close them eventually fails with EMFILE: Too many open files. ls /proc/<pid>/fd | wc -l or lsof -p <pid> shows how many it holds.

On macOS

Descriptors, redirection and pipes work the same way in zsh and bash on macOS. There is no /proc; use lsof -p <pid> to list a process's open files, or ls /dev/fd to see the current process's descriptors.

2

Check yourself

3 questions. Get them all right to finish this topic.

  1. 1.Which command sends both stdout and stderr of ls to all.txt?
  2. 2.Why does a program's error message still appear on screen in ls /nope | grep x?
  3. 3.What happens to a process writing into a pipe when the pipe's buffer is full?

0 of 3 right so far. Get all three to finish this topic.

Go deeper