Chapter 61 · Part 3 of 7
recv() and send() System Calls
Socket-specific I/O with flags for peeking, non-blocking, OOB data, and more
Why recv() and send() Exist
You can use read() and write() on sockets, and they work perfectly fine for basic I/O. But sockets have capabilities that file descriptors don’t — like peeking at incoming data without consuming it, or suppressing the SIGPIPE signal on a broken connection.
recv() and send() are the socket-specific equivalents of read() and write(). They take an extra flags argument that unlocks these socket-specific behaviours.
The API
#include <sys/socket.h>
/*
* Receive data from a socket.
* Returns: bytes received, 0 on EOF, -1 on error.
*/
ssize_t recv(int sockfd, void *buffer, size_t length, int flags);
/*
* Send data on a socket.
* Returns: bytes sent, -1 on error.
*/
ssize_t send(int sockfd, const void *buffer, size_t length, int flags);
When flags is 0, recv() behaves identically to read(), and send() behaves identically to write().
recv() and send() can only be used on sockets, not on regular files or pipes. read() and write() can be used on both.
recv() Flags
Flag
What it does
MSG_PEEK
Read data from the receive buffer without removing it. The same data will be returned on the next recv() or read() call.
MSG_WAITALL
Block until the full requested number of bytes arrive. Similar to readn() but as a single call. Can still return early on signal, error, or EOF.
MSG_DONTWAIT
Non-blocking receive for this call only (without setting O_NONBLOCK on the fd). Returns EAGAIN if no data is available.
MSG_OOB
Receive out-of-band (urgent) data. Used with TCP urgent data (rarely used in modern applications).
MSG_TRUNC
(Datagram sockets) Return the actual datagram size even if it exceeded the buffer. Useful to know if data was truncated.
send() Flags
Flag
What it does
MSG_DONTWAIT
Non-blocking send for this call only. Returns EAGAIN if the send buffer is full.
MSG_NOSIGNAL
If the peer has closed the connection, do NOT generate SIGPIPE. Instead, return -1 with errno set to EPIPE. Essential for multi-threaded servers.
MSG_OOB
Send out-of-band (urgent) data byte. Only the last byte of the passed data is actually sent as urgent data.
MSG_MORE
Hint to the kernel that more data is coming soon. Delays sending to allow TCP to batch data (similar to TCP_CORK socket option).
Example 1 — MSG_PEEK: Peeking at Incoming Data
MSG_PEEK lets you look at what’s in the receive buffer without consuming it. The data remains available for the next recv() call. This is useful for protocol parsers that need to inspect the first few bytes to decide how to read the rest.
#include <stdio.h>
#include <sys/socket.h>
#include <string.h>
/*
* Peek at the first 4 bytes to detect protocol version
* without consuming the data.
*/
void handle_connection(int connfd)
{
unsigned char header[4];
ssize_t n;
/*
* MSG_PEEK: reads up to 4 bytes but leaves them in buffer.
* The next recv() will see the same bytes again.
*/
n = recv(connfd, header, sizeof(header), MSG_PEEK);
if (n <= 0) {
perror("recv peek"); return;
}
printf("Peeked header bytes: %02x %02x %02x %02x\n",
header[0], header[1], header[2], header[3]);
/* Decide protocol based on peeked magic bytes */
if (header[0] == 0x16 && header[1] == 0x03) {
printf("Detected TLS handshake — routing to TLS handler\n");
/* handle_tls(connfd); */
} else {
printf("Detected plain text — routing to HTTP handler\n");
/* handle_http(connfd); */
}
/*
* Now do the real read — data is still in buffer
* because we used MSG_PEEK above.
*/
char buf[256];
n = recv(connfd, buf, sizeof(buf), 0);
printf("Consumed %zd bytes from buffer\n", n);
}
Example 2 — MSG_WAITALL: Wait for All Bytes
MSG_WAITALL asks the kernel to block until the full requested number of bytes is available. It’s like a built-in readn() for a single call. However, it can still return fewer bytes on signal delivery, EOF, or error — so you should still check the return value.
#include <stdio.h>
#include <stdlib.h>
#include <sys/socket.h>
#define HEADER_SIZE 8 /* fixed-size protocol header */
typedef struct {
uint32_t magic;
uint32_t payload_len;
} ProtoHeader;
int read_message(int sockfd)
{
ProtoHeader hdr;
char *payload;
ssize_t n;
/*
* Read exactly 8 bytes for the header.
* MSG_WAITALL blocks until all 8 bytes arrive.
* Still check n == HEADER_SIZE in case of early EOF/signal.
*/
n = recv(sockfd, &hdr, sizeof(hdr), MSG_WAITALL);
if (n != sizeof(hdr)) {
fprintf(stderr, "Short header read: got %zd, expected %zu\n",
n, sizeof(hdr));
return -1;
}
uint32_t plen = ntohl(hdr.payload_len);
if (plen == 0 || plen > 1024*1024) {
fprintf(stderr, "Invalid payload length: %u\n", plen);
return -1;
}
payload = malloc(plen);
if (!payload) { perror("malloc"); return -1; }
/* Now read the variable-length payload */
n = recv(sockfd, payload, plen, MSG_WAITALL);
if ((size_t)n != plen) {
fprintf(stderr, "Short payload read: got %zd, expected %u\n", n, plen);
free(payload);
return -1;
}
printf("Received message: magic=0x%08x, payload=%u bytes\n",
ntohl(hdr.magic), plen);
free(payload);
return 0;
}
Example 3 — MSG_DONTWAIT: Per-Call Non-Blocking
Instead of setting O_NONBLOCK on the socket (which affects all operations), MSG_DONTWAIT makes just this one recv() call non-blocking. This is useful in event loops where you want to drain a socket without blocking.
#include <stdio.h>
#include <errno.h>
#include <sys/socket.h>
/*
* Drain all currently available data from sockfd without blocking.
* Returns total bytes read, or -1 on error.
*/
ssize_t drain_socket(int sockfd)
{
char buf[4096];
ssize_t n;
ssize_t total = 0;
for (;;) {
n = recv(sockfd, buf, sizeof(buf), MSG_DONTWAIT);
if (n > 0) {
total += n;
/* process buf[0..n-1] */
printf("Drained %zd bytes\n", n);
continue;
}
if (n == 0) {
printf("Peer closed connection\n");
break;
}
/* n == -1 */
if (errno == EAGAIN || errno == EWOULDBLOCK) {
/* No more data right now — come back later */
printf("No more data available, total drained: %zd\n", total);
break;
}
perror("recv");
return -1;
}
return total;
}
Example 4 — MSG_NOSIGNAL: Avoiding SIGPIPE
By default, writing to a socket whose peer has closed the connection sends SIGPIPE to your process. The default action for SIGPIPE is to terminate the process. In a server, this is dangerous — one client disconnecting could kill the whole server.
MSG_NOSIGNAL suppresses SIGPIPE and returns EPIPE as errno instead, which you can handle cleanly.
#include <stdio.h>
#include <errno.h>
#include <string.h>
#include <sys/socket.h>
/*
* Safe send that won't kill the process via SIGPIPE
* even if the peer has closed the connection.
*/
ssize_t safe_send(int sockfd, const void *buf, size_t len)
{
ssize_t n;
/*
* MSG_NOSIGNAL: if peer closed connection, return -1
* with errno == EPIPE instead of raising SIGPIPE.
*/
n = send(sockfd, buf, len, MSG_NOSIGNAL);
if (n == -1) {
if (errno == EPIPE) {
/* Peer closed — connection is gone */
printf("Connection broken: peer closed socket\n");
} else {
perror("send");
}
}
return n;
}
/*
* Alternative: install a SIG_IGN handler for SIGPIPE globally.
* Then use regular write() and check errno for EPIPE.
*/
#include <signal.h>
void ignore_sigpipe(void)
{
struct sigaction sa;
memset(&sa, 0, sizeof(sa));
sa.sa_handler = SIG_IGN;
sigaction(SIGPIPE, &sa, NULL);
/* Now write() returns -1 with errno=EPIPE instead of raising signal */
}
recvmsg() and sendmsg() — Advanced I/O
For the most advanced use cases — scatter/gather I/O, sending ancillary (control) data, passing file descriptors over UNIX sockets — Linux provides recvmsg() and sendmsg().
#include <sys/socket.h>
ssize_t recvmsg(int sockfd, struct msghdr *msg, int flags);
ssize_t sendmsg(int sockfd, const struct msghdr *msg, int flags);
/*
* The msghdr structure allows:
* - Scatter I/O: receive into multiple non-contiguous buffers (iovec)
* - Ancillary data: fd passing, credentials, timestamps
* - Peer address (for datagram sockets)
*/
struct msghdr {
void *msg_name; /* peer address (UDP), NULL for TCP */
socklen_t msg_namelen; /* address length */
struct iovec *msg_iov; /* scatter/gather I/O buffers */
size_t msg_iovlen; /* number of iovec elements */
void *msg_control; /* ancillary data buffer */
size_t msg_controllen; /* ancillary data buffer size */
int msg_flags; /* flags on received message (output) */
};
Example: Passing a file descriptor over a UNIX socket
#include <sys/socket.h>
#include <sys/un.h>
#include <stdio.h>
#include <string.h>
/*
* Send a file descriptor 'fd_to_send' to a peer over unix socket 'sockfd'.
* This is used by privilege-separation daemons and systemd activation.
*/
int send_fd(int sockfd, int fd_to_send)
{
struct msghdr msg;
struct iovec iov;
char iobuf[1]; /* must send at least 1 data byte */
char cmsgbuf[CMSG_SPACE(sizeof(int))];
struct cmsghdr *cmsg;
memset(&msg, 0, sizeof(msg));
/* We must send at least 1 byte of regular data */
iobuf[0] = '\0';
iov.iov_base = iobuf;
iov.iov_len = 1;
msg.msg_iov = &iov;
msg.msg_iovlen = 1;
/* Ancillary data: the file descriptor */
msg.msg_control = cmsgbuf;
msg.msg_controllen = sizeof(cmsgbuf);
cmsg = CMSG_FIRSTHDR(&msg);
cmsg->cmsg_level = SOL_SOCKET;
cmsg->cmsg_type = SCM_RIGHTS; /* "rights" = file descriptors */
cmsg->cmsg_len = CMSG_LEN(sizeof(int));
memcpy(CMSG_DATA(cmsg), &fd_to_send, sizeof(int));
if (sendmsg(sockfd, &msg, 0) == -1) {
perror("sendmsg");
return -1;
}
printf("Sent fd %d to peer\n", fd_to_send);
return 0;
}
Interview Questions & Answers
Q1. What is the difference between recv() and read() on a socket?
They are functionally identical when flags=0. recv() provides the extra flags argument (MSG_PEEK, MSG_WAITALL, MSG_DONTWAIT, etc.) that enable socket-specific behaviours. read() has no such parameter. recv() can only be used on sockets; read() works on any file descriptor.
Q2. What does MSG_PEEK do and when would you use it?
MSG_PEEK reads data from the receive buffer without removing it. The same data is returned on the next read. Use it when you need to inspect the first few bytes (e.g., a magic number or protocol type) before deciding how to parse the rest of the message.
Q3. Is MSG_WAITALL reliable for getting exactly n bytes?
Not completely. MSG_WAITALL is a hint to the kernel to wait until n bytes arrive, but it can still return fewer bytes if a signal is delivered, EOF is received, or an error occurs. You must always check the return value and handle short reads.
Q4. Why is SIGPIPE dangerous in a multi-threaded server, and how do you prevent it?
SIGPIPE’s default action is to terminate the process. If any thread calls write/send on a closed peer’s socket, the entire server process dies, taking down all other connections. Prevention options: (1) use MSG_NOSIGNAL flag on send(), (2) globally ignore SIGPIPE with signal(SIGPIPE, SIG_IGN), or (3) install a no-op signal handler. Then check errno for EPIPE to detect broken connections.
Q5. What is the difference between MSG_DONTWAIT and O_NONBLOCK?
O_NONBLOCK makes ALL operations on the socket non-blocking permanently. MSG_DONTWAIT makes only that single recv()/send() call non-blocking, without changing the socket’s mode. MSG_DONTWAIT is more surgical and avoids the overhead of fcntl() calls.
Q6. What is ancillary data (control messages) in recvmsg/sendmsg?
Ancillary data travels alongside the regular socket data but in a separate channel. Common uses: passing open file descriptors between processes (SCM_RIGHTS), passing process credentials (SCM_CREDENTIALS), receiving packet timestamps. It’s the mechanism that lets you “pass a file descriptor” to another process over a UNIX socket.
Q7. What does MSG_MORE do and how is it related to Nagle’s algorithm?
MSG_MORE tells the kernel “I have more data coming soon — hold off on sending this segment immediately”. It’s similar to the TCP_CORK socket option. This allows the kernel to batch small writes into larger TCP segments, reducing per-packet overhead. Unlike Nagle’s algorithm (which the kernel applies automatically), MSG_MORE gives the application explicit control over when to flush.