Alternative I/O Models Signal-driven I/O and the epoll API

 

Chapter 63: Alternative I/O Models
Part 4 of 5 โ€” Signal-driven I/O and the epoll API
๐Ÿ“– TLPI Chapter 63
โฑ ~25 min read
๐Ÿ’ป Linux Systems

When you have thousands of file descriptors, select() and poll() become too slow because they scan every FD on every call. Two better alternatives solve this: Signal-driven I/O where the kernel sends your process a signal when an FD is ready, and epoll which is Linux’s highly efficient event notification system used by every modern high-performance server.

Keywords:

SIGIO SIGPOLL F_SETOWN F_SETSIG SA_SIGINFO epoll_create() epoll_ctl() epoll_wait() EPOLLIN EPOLLOUT EPOLLET Level-triggered Edge-triggered EPOLL_CTL_ADD

Part A: Signal-driven I/O

In signal-driven I/O, you tell the kernel: “Watch this file descriptor for me. When it becomes ready, send me a signal.” Your process then goes off and does other work freely. When the kernel sends the signal, your signal handler runs and you perform the I/O.

Signal-driven I/O Flow
1
Setup: Enable O_ASYNC on the FD. Set the process as the signal target using fcntl(F_SETOWN). Optionally change signal from SIGIO to a real-time signal.
2
Your process does other work: You are not blocked in any system call. You process other data, handle timers, or just run your main logic.
3
Data arrives on FD: The kernel detects I/O is ready and sends a signal (default: SIGIO) to your process.
4
Signal handler runs: Your SIGIO handler calls read()/write() to do the actual I/O. The FD is ready so it will not block.

Setting Up Signal-driven I/O: Step by Step
Step What to Do System Call / Flag
Step 1 Install a signal handler for SIGIO (or your chosen real-time signal) sigaction(SIGIO, …)
Step 2 Set the process (or process group) that will receive the signal fcntl(fd, F_SETOWN, getpid())
Step 3 (optional) Change from SIGIO to a real-time signal for better queuing and FD info in siginfo_t fcntl(fd, F_SETSIG, SIGRTMIN)
Step 4 Enable async (signal-driven) notification on the FD fcntl(fd, F_SETFL, O_ASYNC | existing_flags)

#include <stdio.h>
#include <stdlib.h>
#include <signal.h>
#include <fcntl.h>
#include <unistd.h>

/*
 * Signal handler: called when stdin has data ready.
 * Keep signal handlers short and async-signal-safe.
 * Write a byte to a pipe or set a flag for the main loop.
 */
static volatile sig_atomic_t io_ready = 0;

static void sigio_handler(int sig) {
    /*
     * Do not call printf() or read() directly here in production!
     * Only async-signal-safe functions are allowed in signal handlers.
     * Set a flag and let the main loop do the actual I/O.
     */
    io_ready = 1;
}

int main() {
    struct sigaction sa;
    int flags;
    char buf[256];
    ssize_t n;

    /* Step 1: Install handler for SIGIO */
    sa.sa_handler = sigio_handler;
    sa.sa_flags   = SA_RESTART;
    sigemptyset(&sa.sa_mask);
    if (sigaction(SIGIO, &sa, NULL) == -1) {
        perror("sigaction");
        exit(EXIT_FAILURE);
    }

    /* Step 2: Set this process as the owner (receiver) of SIGIO for stdin */
    if (fcntl(STDIN_FILENO, F_SETOWN, getpid()) == -1) {
        perror("fcntl F_SETOWN");
        exit(EXIT_FAILURE);
    }

    /* Step 4: Enable O_ASYNC on stdin */
    flags = fcntl(STDIN_FILENO, F_GETFL);
    if (fcntl(STDIN_FILENO, F_SETFL, flags | O_ASYNC) == -1) {
        perror("fcntl F_SETFL O_ASYNC");
        exit(EXIT_FAILURE);
    }

    printf("Signal-driven I/O enabled on stdin. Type something!\n");
    printf("Main loop doing other work...\n");

    /* Main loop: do other work, check io_ready flag */
    for (int i = 0; i < 20; i++) {
        if (io_ready) {
            io_ready = 0;
            /* Now safe to read since kernel told us data is ready */
            n = read(STDIN_FILENO, buf, sizeof(buf) - 1);
            if (n > 0) {
                buf[n] = '\0';
                printf("Got from stdin: '%s'\n", buf);
                break;
            }
        }
        printf("Working... iteration %d\n", i + 1);
        sleep(1);
    }

    return 0;
}
    
โš  Important limitations of SIGIO:

  • SIGIO is a standard signal. If multiple FDs become ready before your handler runs, only one signal is delivered. You may miss events. This is why using a real-time signal (F_SETSIG with SIGRTMIN) is recommended โ€” real-time signals are queued.
  • With plain SIGIO you cannot know which FD caused the signal without checking all of them.
  • Signal handlers have strict async-signal-safe rules. You cannot call most library functions inside them safely.

Problems with Signal-driven I/O
โŒ Complexity

Signal handling code is hard to write correctly. You must use only async-signal-safe functions in signal handlers. Many standard C library functions are NOT async-signal-safe.

โŒ Signal Loss (SIGIO)

Standard SIGIO is not queued. If 10 FDs become ready at once, you may only receive 1 signal. Real-time signals fix this but add more complexity.

โŒ Hard to tell which FD

With SIGIO you do not know which FD caused the signal without scanning. With SA_SIGINFO and real-time signals you can get more info, but the setup is complex.

โŒ Linux-specific features needed

To use signal-driven I/O effectively (F_SETSIG, real-time signals) you need Linux-specific extensions, giving up portability with no advantage over epoll.

Bottom line: Signal-driven I/O was an improvement over poll/select for large FD counts, but epoll is almost always the better choice on Linux. epoll gives the same performance without signal handling complexity.

Part B: The epoll API

epoll (event poll) is a Linux-specific API introduced in kernel 2.6. It is the backbone of every major high-performance Linux server: Nginx, Node.js, Redis, and many more all use epoll internally.

The key insight of epoll is: instead of passing your FD list on every call (like select/poll do), you register FDs with a kernel-side data structure once. The kernel then tracks events internally. When you call epoll_wait(), the kernel gives you only the FDs that are actually ready โ€” no scanning through inactive ones.

epoll vs poll: Key Difference
poll() – Every Call
1. Copy all 10,000 FDs to kernel
2. Kernel scans all 10,000 FDs
3. Kernel copies result back
4. App scans all 10,000 results
Repeat for every call!
epoll – One-time Setup + Efficient Wait
Once only: Register FDs with epoll_ctl()
Kernel maintains interest list internally
Each call: epoll_wait() sleeps
Only ready FDs returned (e.g. 3 out of 10,000)
O(ready) not O(total)!

The Three epoll System Calls

1. epoll_create() โ€” Create an epoll instance

#include <sys/epoll.h>

int epoll_create(int size);  /* size is ignored since Linux 2.6.8, but must be > 0 */
int epoll_create1(int flags); /* newer version; pass 0 or EPOLL_CLOEXEC */

/*
 * Returns an "epoll file descriptor" - a handle to the kernel's
 * event interest list. Think of it as a container for the FDs you want to watch.
 * You must close() it when done.
 */
        

This creates a kernel data structure that holds the list of FDs you want to monitor and tracks which ones have events. The returned epfd is itself a file descriptor โ€” you can even add it to another epoll instance.

2. epoll_ctl() โ€” Add, modify, or remove FDs

int epoll_ctl(int epfd, int op, int fd, struct epoll_event *event);

/*
 * epfd   : epoll FD returned by epoll_create()
 * op     : what to do - EPOLL_CTL_ADD, EPOLL_CTL_MOD, EPOLL_CTL_DEL
 * fd     : the actual FD you want to watch/change/remove
 * event  : what events to watch for (NULL when deleting)
 */

struct epoll_event {
    uint32_t events;   /* Bitmask: EPOLLIN, EPOLLOUT, EPOLLERR, EPOLLET, etc. */
    epoll_data_t data; /* User data: store fd, pointer, or any 64-bit value */
};

/* epoll_data_t is a union: */
typedef union epoll_data {
    void    *ptr;
    int      fd;
    uint32_t u32;
    uint64_t u64;
} epoll_data_t;
        
op value Meaning
EPOLL_CTL_ADD Add fd to the interest list with the given events
EPOLL_CTL_MOD Change the events being monitored for an already-registered fd
EPOLL_CTL_DEL Remove fd from the interest list (event can be NULL)

3. epoll_wait() โ€” Wait for events

int epoll_wait(int epfd,
               struct epoll_event *events,
               int maxevents,
               int timeout);

/*
 * epfd      : epoll FD
 * events    : array to receive ready events (you allocate this)
 * maxevents : max events to return in one call
 * timeout   : -1=block forever, 0=return immediately, N=wait N milliseconds
 *
 * Returns: number of ready events placed in events[] array
 *          0 if timeout expired
 *         -1 on error
 */
        

This is the call that blocks until events occur. When it returns, events[] contains only the FDs that are ready. You iterate only over those. No scanning of inactive FDs needed.

epoll Complete Working Example

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <sys/epoll.h>

#define MAX_EVENTS 10

/*
 * epoll example: monitor stdin and a pipe simultaneously.
 * Shows the typical epoll usage pattern used in real servers.
 */
int main() {
    int epfd;
    int pipefd[2];
    char buf[256];
    ssize_t n;
    int ret;
    struct epoll_event ev, events[MAX_EVENTS];

    /* Create a pipe and pre-fill it with data */
    if (pipe(pipefd) == -1) {
        perror("pipe");
        exit(EXIT_FAILURE);
    }
    write(pipefd[1], "hello from pipe", 15);

    /* Step 1: Create epoll instance */
    epfd = epoll_create1(0);
    if (epfd == -1) {
        perror("epoll_create1");
        exit(EXIT_FAILURE);
    }

    /* Step 2a: Register stdin with epoll */
    ev.events  = EPOLLIN;          /* Watch for data to read */
    ev.data.fd = STDIN_FILENO;     /* Store fd in data so we know which FD later */
    if (epoll_ctl(epfd, EPOLL_CTL_ADD, STDIN_FILENO, &ev) == -1) {
        perror("epoll_ctl add stdin");
        exit(EXIT_FAILURE);
    }

    /* Step 2b: Register pipe read end with epoll */
    ev.events  = EPOLLIN;
    ev.data.fd = pipefd[0];
    if (epoll_ctl(epfd, EPOLL_CTL_ADD, pipefd[0], &ev) == -1) {
        perror("epoll_ctl add pipe");
        exit(EXIT_FAILURE);
    }

    printf("epoll watching stdin and pipe. Type something or wait...\n");

    /* Step 3: Event loop */
    while (1) {
        /*
         * epoll_wait: sleep until at least one FD is ready.
         * events[] will only contain the READY FDs.
         * Timeout 5000ms = 5 seconds.
         */
        ret = epoll_wait(epfd, events, MAX_EVENTS, 5000);

        if (ret == -1) {
            perror("epoll_wait");
            break;
        }
        if (ret == 0) {
            printf("Timeout: no events in 5 seconds\n");
            break;
        }

        /*
         * Loop over ONLY the ready FDs (ret of them).
         * Not over all registered FDs. This is what makes epoll fast!
         */
        for (int i = 0; i < ret; i++) {

            if (events[i].data.fd == STDIN_FILENO) {
                n = read(STDIN_FILENO, buf, sizeof(buf) - 1);
                if (n > 0) {
                    buf[n] = '\0';
                    printf("stdin: '%s'\n", buf);
                } else if (n == 0) {
                    printf("stdin: EOF\n");
                    /* Remove from epoll when done */
                    epoll_ctl(epfd, EPOLL_CTL_DEL, STDIN_FILENO, NULL);
                }
            }

            else if (events[i].data.fd == pipefd[0]) {
                n = read(pipefd[0], buf, sizeof(buf) - 1);
                if (n > 0) {
                    buf[n] = '\0';
                    printf("pipe: '%s'\n", buf);
                    /* Remove pipe from epoll after reading */
                    epoll_ctl(epfd, EPOLL_CTL_DEL, pipefd[0], NULL);
                    goto done;
                }
            }

            /* Handle error or hangup */
            if (events[i].events & (EPOLLERR | EPOLLHUP)) {
                printf("Error or hangup on fd %d\n", events[i].data.fd);
                epoll_ctl(epfd, EPOLL_CTL_DEL, events[i].data.fd, NULL);
                close(events[i].data.fd);
            }
        }
    }

done:
    close(epfd);
    close(pipefd[0]);
    close(pipefd[1]);
    return 0;
}
    

Compile: gcc -o epoll_demo epoll_demo.c && ./epoll_demo

Level-triggered vs Edge-triggered Notification

epoll supports two notification modes. This is one of its most important features and also a common interview topic.

Level-triggered (LT) โ€” Default

When notified: epoll_wait() returns as long as the FD is in a ready state.

Analogy: A light bulb that stays ON as long as there is current. Every time you check, it tells you “still ready”.

If you read only part of the data: epoll_wait() will notify you again on the next call because data is still there.

Safer: You cannot miss events. If you don’t process all data in one go, you will be notified again.

Same behavior as: select() and poll()

Edge-triggered (ET) โ€” EPOLLET flag

When notified: epoll_wait() returns only when the FD transitions from not-ready to ready (the “edge”).

Analogy: A button that fires once when you press it. If you ignore it, it won’t fire again until the next press.

If you read only part of the data: epoll_wait() will NOT notify you again for the remaining data.

Faster: Fewer notifications, but you must drain the FD completely when notified (loop read until EAGAIN).

Requires: FD in nonblocking mode. Loop read/write until EAGAIN.

/* Level-triggered vs Edge-triggered: which flag to use */

/* Level-triggered (default) - no extra flag needed */
ev.events = EPOLLIN;
epoll_ctl(epfd, EPOLL_CTL_ADD, fd, &ev);

/* Edge-triggered - add EPOLLET flag */
ev.events = EPOLLIN | EPOLLET;
epoll_ctl(epfd, EPOLL_CTL_ADD, fd, &ev);

/*
 * In edge-triggered mode: when you get an EPOLLIN event,
 * you MUST read until EAGAIN, otherwise you will miss data:
 */
if (events[i].events & EPOLLIN) {
    while (1) {
        n = read(fd, buf, sizeof(buf));
        if (n == -1) {
            if (errno == EAGAIN || errno == EWOULDBLOCK)
                break;  /* All data consumed. Stop looping. */
            perror("read");
            break;
        }
        if (n == 0) {
            printf("Connection closed\n");
            break;
        }
        /* process buf[0..n-1] */
        process_data(buf, n);
    }
}
      

Aspect Level-triggered (LT) Edge-triggered (ET)
When notified FD is in ready state FD transitions to ready
Partial read safe? โœ” Yes, notified again โœ– Must drain completely
Requires nonblocking FD? Optional (recommended) Mandatory
Notification overhead More (repeated) Less (once per transition)
Ease of use Simpler Requires more care

Interview Questions
Q1. What are the three system calls of the epoll API and what does each do?

epoll_create1() creates a new epoll instance and returns a file descriptor that represents the kernel’s interest list. epoll_ctl() adds, modifies, or removes file descriptors from the interest list. epoll_wait() blocks until one or more monitored FDs become ready, then returns only the ready ones in an array.

Q2. What is the difference between level-triggered and edge-triggered epoll?

Level-triggered (default) notifies you every time epoll_wait() is called while the FD has data available. If you read only part of the data, the next call to epoll_wait() will notify you again. Edge-triggered notifies you only when the FD transitions from not-ready to ready. If you read only part of the data, you will NOT be notified again until new data arrives. Edge-triggered requires the FD to be in nonblocking mode and you must read in a loop until EAGAIN.

Q3. Why does epoll scale better than poll() for large numbers of FDs?

With poll(), your application copies the entire list of FDs to the kernel on every call, the kernel scans all of them, and copies results back. This is O(n) work even if only one FD is ready. With epoll, you register FDs once. The kernel maintains the interest list internally and uses efficient data structures. epoll_wait() returns only the ready FDs, so both the kernel and application work is O(number of ready FDs) not O(total FDs).

Q4. What is stored in epoll_event.data and why is it useful?

epoll_event.data is a union (epoll_data_t) that can store an int fd, a pointer, or a 64-bit integer. You choose what to store when you call epoll_ctl to register the FD. When epoll_wait returns events, you read back this data to identify which FD or context the event belongs to. The most common usage is to store the fd itself (ev.data.fd = fd) so you know which file descriptor is ready when an event fires.

Q5. Why must edge-triggered epoll be used with nonblocking file descriptors?

In edge-triggered mode you must read in a loop until EAGAIN to drain all available data, because you will not get another notification until new data arrives. If your FD is blocking, the final read() call (when there is no more data) will block indefinitely instead of returning EAGAIN. This would hang your process. Nonblocking mode ensures the last read() returns immediately with EAGAIN, safely signaling that all data has been consumed.

Q6. How do you set up signal-driven I/O on a file descriptor?

First install a SIGIO signal handler with sigaction(). Then call fcntl(fd, F_SETOWN, getpid()) to set your process as the signal recipient. Optionally call fcntl(fd, F_SETSIG, SIGRTMIN) to use a real-time signal instead of SIGIO for better queuing. Finally enable O_ASYNC on the FD with fcntl(fd, F_SETFL, flags | O_ASYNC). After this the kernel will send the signal when the FD becomes ready.

Q7. What are the advantages of epoll over signal-driven I/O?

epoll is simpler because you avoid the complexity of signal handlers and async-signal-safe restrictions. With epoll you can specify precisely what events to watch (read, write, etc.) and choose between level-triggered and edge-triggered modes. Signal-driven I/O with SIGIO does not queue signals so events can be lost. To fix signal loss you need Linux-specific real-time signals (F_SETSIG), at which point signal-driven I/O is no more portable than epoll anyway.

Q8. What happens to an epoll interest list entry when the monitored FD is closed?

When you close() a file descriptor, the kernel automatically removes it from all epoll interest lists it was registered with. You do not need to call epoll_ctl(EPOLL_CTL_DEL) before close(). However, if you have duplicate FDs (dup/dup2) pointing to the same underlying file description, the entry is only removed when all duplicates are closed, because the underlying file description is still open.

Next: Comparison and libevent
Part 5 brings everything together: a full comparison of all I/O models, when to choose each one, and how the libevent library abstracts all of them for portable high-performance code.

โ† Part 3: select() and poll() Part 5: Comparison & libevent โ†’

Leave a Reply

Your email address will not be published. Required fields are marked *