Overview
The sendmsg() and recvmsg() system calls are the most powerful and general-purpose socket I/O functions in Linux. Everything that write(), send(), sendto(), read(), recv(), and recvfrom() can do, sendmsg()/recvmsg() can do — plus more.
Their two unique superpowers are: (1) scatter-gather I/O on sockets, and (2) sending/receiving ancillary data — domain-specific control information that can even carry open file descriptors or sender identity credentials between processes.
Key Terms
1. The msghdr Structure — The Heart of sendmsg/recvmsg
Both sendmsg() and recvmsg() take a single struct msghdr argument that packs together everything: the address, the data buffers, and ancillary data.
#include <sys/socket.h>
struct msghdr {
void *msg_name; /* Optional destination/source address */
socklen_t msg_namelen; /* Size of msg_name */
struct iovec *msg_iov; /* Array of data buffers (scatter-gather) */
size_t msg_iovlen; /* Number of buffers in msg_iov */
void *msg_control; /* Pointer to ancillary data buffer */
size_t msg_controllen; /* Size of ancillary data buffer */
int msg_flags; /* Flags (output only for recvmsg) */
};
/*
* struct iovec describes one buffer in the scatter-gather array.
* Same structure used by readv()/writev().
*/
struct iovec {
void *iov_base; /* Start of buffer */
size_t iov_len; /* Length of buffer */
};
/* System call signatures */
ssize_t sendmsg(int sockfd, const struct msghdr *msg, int flags);
ssize_t recvmsg(int sockfd, struct msghdr *msg, int flags);(for UDP / unconnected sockets)
for scatter-gather I/O
(file descriptors, credentials)
(MSG_TRUNC, MSG_CTRUNC, etc.)
2. Scatter-Gather I/O with Sockets
Scatter-gather I/O means you can read from (or write to) multiple separate memory buffers in a single system call, without having to combine them into one big buffer first. With sendmsg(), multiple buffers are gathered into one outgoing message. With recvmsg(), one incoming message is scattered into multiple buffers.
on the wire
Why is this useful? A common pattern in protocol implementations is to have a fixed header structure and a variable data payload stored in separate buffers. Without scatter-gather, you would need to memcpy() them together into a temporary buffer before sending. With sendmsg(), you just point to both and send.
#include <sys/socket.h>
#include <sys/uio.h>
#include <string.h>
#include <stdio.h>
/* Example: send a protocol header + payload in one call */
typedef struct {
uint32_t msg_len;
uint16_t msg_type;
uint16_t msg_flags;
} ProtoHeader;
ssize_t send_with_header(int sockfd, const void *data, size_t datalen,
uint16_t msg_type)
{
ProtoHeader hdr;
struct iovec iov[2];
struct msghdr msg;
/* Fill in the header */
hdr.msg_len = htonl((uint32_t)(sizeof(hdr) + datalen));
hdr.msg_type = htons(msg_type);
hdr.msg_flags = 0;
/* iov[0]: header buffer */
iov[0].iov_base = &hdr;
iov[0].iov_len = sizeof(hdr);
/* iov[1]: payload buffer */
iov[1].iov_base = (void *)data;
iov[1].iov_len = datalen;
/* Build the message */
memset(&msg, 0, sizeof(msg));
msg.msg_iov = iov;
msg.msg_iovlen = 2; /* Two buffers */
/* One system call sends both buffers as a single message */
return sendmsg(sockfd, &msg, 0);
}
/* Example: scatter incoming data into header + payload buffers */
ssize_t recv_with_header(int sockfd, ProtoHeader *hdr,
void *data, size_t datalen)
{
struct iovec iov[2];
struct msghdr msg;
iov[0].iov_base = hdr;
iov[0].iov_len = sizeof(*hdr);
iov[1].iov_base = data;
iov[1].iov_len = datalen;
memset(&msg, 0, sizeof(msg));
msg.msg_iov = iov;
msg.msg_iovlen = 2;
/* Single call fills both buffers from one datagram */
return recvmsg(sockfd, &msg, 0);
}sendmsg() to gather multiple buffers on a datagram socket, all the buffers are combined into a single datagram. On the receive side, recvmsg() on a datagram socket scatters the bytes of one datagram into multiple buffers.3. Ancillary Data (Control Messages)
Ancillary data (also called control information) is extra metadata that travels alongside the regular socket data. It is set via msg_control and msg_controllen in struct msghdr.
Ancillary data is structured as one or more struct cmsghdr control message headers followed by their data:
/* Ancillary data header */
struct cmsghdr {
size_t cmsg_len; /* Total length: sizeof(cmsghdr) + data length */
int cmsg_level; /* Protocol level (e.g., SOL_SOCKET) */
int cmsg_type; /* Type of control message:
SCM_RIGHTS — pass file descriptors
SCM_CREDENTIALS — pass sender credentials */
};
/*
* IMPORTANT: Never access ancillary data directly.
* Always use the provided macros to ensure correct alignment.
*/
/* Navigate through control messages in a received msghdr */
struct cmsghdr *cmsg;
for (cmsg = CMSG_FIRSTHDR(&msg);
cmsg != NULL;
cmsg = CMSG_NXTHDR(&msg, cmsg)) {
/* CMSG_DATA(cmsg) points to the data after the header */
void *data = CMSG_DATA(cmsg);
}
/* Calculate the space needed for ancillary data of size n */
size_t space = CMSG_SPACE(n); /* includes header + padding */
size_t len = CMSG_LEN(n); /* cmsg_len value to put in header */CMSG_LEN(data)
SOL_SOCKET
SCM_RIGHTS
CMSG_DATA() points here
for alignment
4. Passing File Descriptors Between Processes (SCM_RIGHTS)
One of the most powerful features of UNIX domain sockets is the ability to send an open file descriptor from one process to another. The receiving process gets its own file descriptor referring to the same open file description.
This works for any file descriptor: files opened with open(), pipes, sockets, device nodes, etc.
“my_log.txt”
sends fd=5
UNIX socket
→ same open file
“my_log.txt”
the file descriptor
Practical Use Case: Master Server + Worker Pool
A master server process accepts new TCP client connections on a listening socket. It then passes the connected socket file descriptor to one of a pool of worker processes (or threads) via a UNIX domain socket. The worker handles the client without needing a separate accept() call.
#include <sys/socket.h>
#include <sys/un.h>
#include <string.h>
#include <stdio.h>
/*
* Send a file descriptor 'fd_to_send' over a UNIX domain socket.
* The receiver must call recv_fd() on the other end.
*/
int send_fd(int unix_sock, int fd_to_send)
{
struct msghdr msg;
struct cmsghdr *cmsg;
struct iovec iov;
char buf[CMSG_SPACE(sizeof(int))]; /* ancillary buffer */
char dummy = 'F'; /* must send at least 1 byte of regular data */
memset(&msg, 0, sizeof(msg));
memset(buf, 0, sizeof(buf));
/* We must send at least 1 byte of normal data along with the ancillary data */
iov.iov_base = &dummy;
iov.iov_len = 1;
msg.msg_iov = &iov;
msg.msg_iovlen = 1;
/* Set up the ancillary data control message */
msg.msg_control = buf;
msg.msg_controllen = sizeof(buf);
cmsg = CMSG_FIRSTHDR(&msg);
cmsg->cmsg_level = SOL_SOCKET;
cmsg->cmsg_type = SCM_RIGHTS; /* "Send rights" — pass an fd */
cmsg->cmsg_len = CMSG_LEN(sizeof(int));
/* Copy the file descriptor into the control message data area */
memcpy(CMSG_DATA(cmsg), &fd_to_send, sizeof(int));
return sendmsg(unix_sock, &msg, 0);
}
/*
* Receive a file descriptor from a UNIX domain socket.
* Returns the received fd, or -1 on error.
*/
int recv_fd(int unix_sock)
{
struct msghdr msg;
struct cmsghdr *cmsg;
struct iovec iov;
char buf[CMSG_SPACE(sizeof(int))];
char dummy;
int received_fd = -1;
memset(&msg, 0, sizeof(msg));
memset(buf, 0, sizeof(buf));
iov.iov_base = &dummy;
iov.iov_len = 1;
msg.msg_iov = &iov;
msg.msg_iovlen = 1;
msg.msg_control = buf;
msg.msg_controllen = sizeof(buf);
if (recvmsg(unix_sock, &msg, 0) <= 0) {
perror("recvmsg");
return -1;
}
/* Walk the control messages looking for SCM_RIGHTS */
for (cmsg = CMSG_FIRSTHDR(&msg);
cmsg != NULL;
cmsg = CMSG_NXTHDR(&msg, cmsg)) {
if (cmsg->cmsg_level == SOL_SOCKET &&
cmsg->cmsg_type == SCM_RIGHTS) {
/* Extract the received file descriptor */
memcpy(&received_fd, CMSG_DATA(cmsg), sizeof(int));
printf("Received fd = %d\n", received_fd);
break;
}
}
return received_fd;
}
/*
* Usage in a master/worker architecture:
*
* Master process:
* client_fd = accept(listen_fd, ...);
* send_fd(unix_socket_to_worker, client_fd);
* close(client_fd); // master closes its copy
*
* Worker process:
* int client_fd = recv_fd(unix_socket_from_master);
* // Now handle the client directly
* read(client_fd, buf, ...);
* write(client_fd, response, ...);
* close(client_fd);
*/5. Receiving Sender Credentials (SCM_CREDENTIALS)
Another use of ancillary data is to receive the credentials of the sending process: its user ID (UID), group ID (GID), and process ID (PID). This allows the receiving process to authenticate who is connecting when using UNIX domain sockets.
This is particularly useful for privileged server processes that need to know the identity of a client before granting access to a resource.
#include <sys/socket.h>
#include <stdio.h>
#include <string.h>
/*
* The ucred structure holds sender credentials (Linux-specific).
* Defined in <sys/socket.h> on Linux.
*/
struct ucred {
pid_t pid; /* Process ID of sender */
uid_t uid; /* User ID of sender */
gid_t gid; /* Group ID of sender */
};
/*
* Receive credentials from a UNIX domain socket peer.
* The sender must pass their credentials using SCM_CREDENTIALS ancillary data.
*/
int recv_credentials(int unix_sock, struct ucred *cred_out)
{
struct msghdr msg;
struct cmsghdr *cmsg;
struct iovec iov;
char cmsg_buf[CMSG_SPACE(sizeof(struct ucred))];
char dummy;
memset(&msg, 0, sizeof(msg));
iov.iov_base = &dummy;
iov.iov_len = 1;
msg.msg_iov = &iov;
msg.msg_iovlen = 1;
msg.msg_control = cmsg_buf;
msg.msg_controllen = sizeof(cmsg_buf);
if (recvmsg(unix_sock, &msg, 0) <= 0) {
perror("recvmsg");
return -1;
}
for (cmsg = CMSG_FIRSTHDR(&msg);
cmsg != NULL;
cmsg = CMSG_NXTHDR(&msg, cmsg)) {
if (cmsg->cmsg_level == SOL_SOCKET &&
cmsg->cmsg_type == SCM_CREDENTIALS) {
memcpy(cred_out, CMSG_DATA(cmsg), sizeof(struct ucred));
printf("Sender PID: %d\n", cred_out->pid);
printf("Sender UID: %d\n", cred_out->uid);
printf("Sender GID: %d\n", cred_out->gid);
return 0;
}
}
return -1; /* No credentials found */
}
/*
* IMPORTANT NOTES on SCM_CREDENTIALS:
*
* 1. The server must enable SO_PASSCRED on the socket:
* setsockopt(sock, SOL_SOCKET, SO_PASSCRED, &(int){1}, sizeof(int));
*
* 2. The sender specifies its own credentials.
* It can use its real, effective, or saved set UID/GID.
* It cannot spoof another process's credentials.
*
* 3. This is Linux-specific and NOT specified by SUSv3 (POSIX).
* Portable alternatives: use getpeercred() on some BSD systems.
*
* 4. Only works for UNIX domain sockets — not TCP/UDP.
*/SCM_CREDENTIALS to verify the caller’s UID/PID before allowing access. This is more reliable than having the client send its own PID in normal data, because the kernel enforces the credential values.6. recvmmsg() — Receiving Multiple Datagrams at Once
Linux 2.6.33 added recvmmsg(), which can receive multiple datagrams in a single system call. This reduces the per-datagram system-call overhead for high-throughput UDP servers (like DNS servers or network monitoring tools).
#include <sys/socket.h>
/*
* recvmmsg() — receive multiple datagrams in one system call.
*
* struct mmsghdr {
* struct msghdr msg_hdr; // same as recvmsg
* unsigned int msg_len; // filled in by kernel: bytes received
* };
*
* int recvmmsg(int sockfd, struct mmsghdr *msgvec, unsigned int vlen,
* int flags, struct timespec *timeout);
*
* Returns number of datagrams received, or -1 on error.
*
* Why use it:
* - A busy DNS server receiving 100,000 queries/sec normally
* makes 100,000 recvmsg() system calls per second.
* - With recvmmsg(vlen=32), it can receive 32 datagrams per call,
* reducing system-call overhead by up to 32x.
*
* An analogous sendmmsg() call also exists for sending multiple
* datagrams in one system call.
*/recvmsg() is perfectly sufficient.7. sendmsg/recvmsg Capability Summary
| Capability | write/send | sendmsg |
|---|---|---|
| Send data from one buffer | ✅ | ✅ |
| Specify destination address (UDP) | ✅ sendto only | ✅ |
| Scatter-gather I/O (multiple buffers) | ❌ | ✅ |
| Send ancillary data | ❌ | ✅ |
| Pass file descriptors (SCM_RIGHTS) | ❌ | ✅ |
| Pass sender credentials (SCM_CREDENTIALS) | ❌ | ✅ |
Interview Questions
sendmsg() can do two things send() cannot: (1) scatter-gather I/O — sending from multiple buffers in a single call; and (2) sending ancillary data (control messages) alongside the regular data. Ancillary data is used for things like passing open file descriptors or sender credentials to another process.
Scatter-gather I/O allows a single system call to operate on multiple non-contiguous memory buffers. With sendmsg(), an array of struct iovec buffers in msg_iov are gathered into one outgoing message. With recvmsg(), one incoming message is scattered into multiple iovec buffers. This avoids the need to copy data into a single temporary buffer before sending.
Ancillary data (also called control messages) is extra domain-specific information transmitted alongside the regular socket data. It is passed through the msg_control field of struct msghdr and structured as one or more struct cmsghdr headers. Examples include SCM_RIGHTS for passing file descriptors and SCM_CREDENTIALS for passing process credentials, both over UNIX domain sockets.
Using sendmsg() with SCM_RIGHTS ancillary data over a UNIX domain socket, one process can send an open file descriptor to another process on the same host. What is actually passed is a reference to the same open file description (the kernel’s internal file table entry) — not just the integer fd number. The receiving process gets a new fd number in its own fd table, but it points to the same underlying file object. Both processes share the same file offset, open flags, and access mode.
A master server process listens on a TCP port and accepts incoming client connections. Instead of handling each client itself, it passes the accepted connected socket fd to one of a pool of pre-forked worker processes via a UNIX domain socket using SCM_RIGHTS. The worker then handles the client directly. This separates the connection-acceptance role from the request-handling role, enabling clean load distribution and privilege separation.
SCM_CREDENTIALS is a type of ancillary data for UNIX domain sockets that carries the sending process’s credentials: its PID (process ID), UID (user ID), and GID (group ID), packaged in a struct ucred. The receiving process can use these to authenticate the sender. The sender can specify its real, effective, or saved set IDs, but cannot spoof another process’s credentials. This is Linux-specific and not in the POSIX standard.
recvmmsg() (added in Linux 2.6.33) receives multiple datagrams in a single system call, reducing the per-datagram system-call overhead. It is useful for high-throughput UDP applications (DNS servers, network monitoring) where the cost of making one recvmsg() call per datagram becomes a bottleneck. For most applications, regular recvmsg() is sufficient. An analogous sendmmsg() exists for sending.
You must use CMSG_FIRSTHDR(), CMSG_NXTHDR(), CMSG_DATA(), CMSG_LEN(), and CMSG_SPACE(). These macros handle the correct alignment of the control message headers and data for the platform’s ABI requirements. Directly casting pointers into the msg_control buffer without these macros can cause alignment faults and undefined behavior on architectures with strict alignment requirements.