Kernel Limits · Why Avoid · POSIX Queues · Final Review

 

Chapter 46 · File 6 of 6

Limitations & Alternatives

Section 46.9 · Kernel Limits · Why Avoid · POSIX Queues · Final Review

TLPI Section 46.9 discusses why System V message queues are considered obsolete for new code. Understanding their limitations is important both for maintaining old code and for making the right design choices in new projects.

Section 46.9 — Limitations of System V Message Queues

🚫
No file descriptor — cannot use with select/poll/epoll
System V message queues use a msqid (integer), not an fd. You cannot multiplex them with other I/O sources. You cannot say “wait until either a socket receives data OR a message arrives in the queue.” This is a severe limitation for event-driven programs.
🕰️
Kernel persistence — queues survive process exit
Unlike pipes (which vanish when the last fd is closed), System V queues remain in the kernel until explicitly removed or the system reboots. A crashed program that didn’t call msgctl(IPC_RMID) leaves a permanent queue consuming kernel resources. Over time, a server that creates queues can exhaust the kernel limit.
🔢
System-wide limits — can run out of resources
The kernel enforces hard limits on the total number of queues, total messages, and total bytes across all queues. These are tuneable via /proc/sys/kernel/msg* but still finite. On a busy server, leaked queues can exhaust these limits and cause new msgget() calls to fail.
🔑
Key management is error-prone
ftok() can generate collisions (two different file paths producing the same key if they happen to share inode + device). IPC_PRIVATE is safe but only works for related processes. Hard-coded keys are fragile. There’s no namespace isolation.
🔧
Non-standard API design
The System V IPC API (msgget, msgsnd, msgrcv, msgctl) doesn’t follow the standard file-based UNIX I/O model. It uses its own flags, its own error codes, and its own identifier type. Programs using it are harder to reason about and maintain.
📏
Fixed maximum message size
There’s a kernel-enforced per-message size limit (MSGMAX, typically 8192 bytes). You cannot send large data blobs in a single message. For large transfers, you’d need fragmentation — complex code that pipes handle automatically.

Kernel Limits — Reading and Tuning

## View current kernel limits for message queues
$ cat /proc/sys/kernel/msgmax   # Max size of a single message (bytes)
8192

$ cat /proc/sys/kernel/msgmnb   # Max bytes in a single queue
16384

$ cat /proc/sys/kernel/msgmni   # Max number of queues system-wide
32000

## Temporarily increase (lost on reboot)
$ sudo sysctl -w kernel.msgmnb=65536
$ sudo sysctl -w kernel.msgmni=1024

## Permanently increase (survives reboot) — add to /etc/sysctl.conf:
kernel.msgmnb = 65536
kernel.msgmni = 1024
kernel.msgmax = 65536
/* limits_demo.c — Show queue limits from inside C */
#include <stdio.h>
#include <stdlib.h>
#include <sys/types.h>
#include <sys/ipc.h>
#include <sys/msg.h>

int main(void)
{
    /* msginfo holds system-wide limits */
    struct msginfo info;
    if (msgctl(0, IPC_INFO, (struct msqid_ds *)&info) == -1) {
        perror("msgctl IPC_INFO");
        return 1;
    }
    printf("MSGMAX (max msg bytes) : %d\n", info.msgmax);
    printf("MSGMNB (max queue bytes): %d\n", info.msgmnb);
    printf("MSGMNI (max queues)     : %d\n", info.msgmni);
    printf("MSGTQL (max messages)   : %d\n", info.msgtql);
    printf("MSGPOOL (msg pool size) : %d KB\n", info.msgpool);
    return 0;
}

System V vs POSIX Message Queues

POSIX message queues (mq_open) were designed to fix the limitations of System V. Here’s a direct comparison:

Feature System V (msgget) POSIX (mq_open)
Identifier type Integer (msqid) File descriptor (mqd_t)
Works with select/poll? No Yes!
Async notification? No Yes — mq_notify()
Naming ftok() or IPC_PRIVATE /name path (like filesystem)
Cleanup Manual (msgctl IPC_RMID) mq_close() + mq_unlink()
Priority via message type tricks Built-in priority field
POSIX standard? XSI / legacy Yes — POSIX.1-2001
Header sys/msg.h mqueue.h
Link flags (none needed) -lrt (on some systems)

Quick POSIX Message Queue Example

/* posix_mq_demo.c — POSIX alternative to System V */
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <mqueue.h>
#include <fcntl.h>

#define QUEUE_NAME  "/demo_queue"
#define MAX_MSG     128
#define MAX_MSGS    10

int main(void)
{
    struct mq_attr attr = {
        .mq_flags   = 0,
        .mq_maxmsg  = MAX_MSGS,
        .mq_msgsize = MAX_MSG,
        .mq_curmsgs = 0
    };

    /* Create queue — note the /name style, like a file */
    mqd_t mq = mq_open(QUEUE_NAME, O_CREAT | O_RDWR, 0666, &attr);
    if (mq == (mqd_t)-1) { perror("mq_open"); return 1; }

    /* Send — has a built-in priority argument (0=lowest) */
    const char *msg = "Hello from POSIX MQ";
    if (mq_send(mq, msg, strlen(msg)+1, 0) == -1) {
        perror("mq_send"); return 1;
    }
    printf("Sent: '%s'\n", msg);

    /* Receive */
    char buf[MAX_MSG];
    unsigned int prio;
    ssize_t n = mq_receive(mq, buf, MAX_MSG, &prio);
    if (n == -1) { perror("mq_receive"); return 1; }
    printf("Received (priority=%u): '%s'\n", prio, buf);

    /* Close and unlink — like close() + unlink() for files */
    mq_close(mq);
    mq_unlink(QUEUE_NAME);  /* Remove the named queue */
    return 0;
}
/* Compile: gcc posix_mq_demo.c -o posix_mq_demo -lrt */

Decision Guide: Which IPC Mechanism to Choose

IPC Selection Guide
Need Best Choice Why
Simple stream between parent and child Pipe Simplest, auto-cleanup, FD-based
Stream between unrelated processes FIFO (named pipe) Simple, uses filename, FD-based
Typed messages + select/poll support POSIX message queue Modern, FD-based, built-in priority
Full duplex, network-capable Unix domain socket FD-based, full duplex, bidirectional
Large data shared (no copy) Shared memory Zero-copy, fastest IPC
Maintaining old code System V message queue Legacy compatibility only

Chapter 46 — Complete Summary

msgget(key, flags)

Creates or opens a queue. Returns msqid. Use IPC_PRIVATE for related processes, ftok() for unrelated. IPC_CREAT | IPC_EXCL for guaranteed fresh creation.

msgsnd(qid, msg, size, flags)

Sends a message. size = payload only (not full struct). Blocks if queue full unless IPC_NOWAIT. EINTR on signal interrupt.

msgrcv(qid, buf, max, type, flags)

Receives a message. type=0 (FIFO), type>0 (specific type), type<0 (lowest type ≤ abs). MSG_NOERROR truncates. IPC_NOWAIT is non-blocking.

msgctl(qid, cmd, buf)

IPC_STAT reads info. IPC_SET changes msg_qbytes/permissions. IPC_RMID deletes queue — always call this to avoid leaks.

Quick Reference Card

/* === System V Message Queue Quick Reference === */

/* 1. Create/Open */
int qid = msgget(IPC_PRIVATE,  IPC_CREAT | 0666);        // private
int qid = msgget(ftok("/f",'A'), IPC_CREAT | 0666);       // shared
int qid = msgget(key,  IPC_CREAT | IPC_EXCL | 0666);     // fail if exists
int qid = msgget(key,  0666);                              // open only

/* 2. Message structure */
struct Msg { long mtype; /* your data */ };

/* 3. Send */
msgsnd(qid, &msg, sizeof(msg.payload_only), 0);           // blocking
msgsnd(qid, &msg, sizeof(msg.payload_only), IPC_NOWAIT);  // non-blocking

/* 4. Receive */
msgrcv(qid, &buf, max_payload, 0,   0);        // first any type
msgrcv(qid, &buf, max_payload, 5,   0);        // type=5 only
msgrcv(qid, &buf, max_payload, -5,  0);        // lowest type <=5
msgrcv(qid, &buf, max_payload, 0,   IPC_NOWAIT);  // non-blocking

/* 5. Control */
msgctl(qid, IPC_STAT, &ds);   // read info
msgctl(qid, IPC_SET,  &ds);   // change settings
msgctl(qid, IPC_RMID, NULL);  // delete — always do this!

/* 6. Shell */
// ipcs -q          list queues
// ipcrm -q msqid   remove by ID

Final Interview Questions — Limitations & Full Chapter Review

Q1. List 4 limitations of System V message queues. (1) No file descriptor — cannot use with select/poll/epoll. (2) Kernel persistence — survive process exit, require manual deletion. (3) System-wide limits on number of queues and total bytes. (4) Key management with ftok() is error-prone with potential collisions.
Q2. What are the three kernel tunable parameters for message queues? MSGMAX (max bytes in a single message), MSGMNB (max bytes in a single queue), and MSGMNI (max number of queues system-wide). Readable via /proc/sys/kernel/msg*.
Q3. How does a POSIX message queue differ from System V? POSIX mq_open() returns an mqd_t which IS a file descriptor — works with select/poll. It has a built-in message priority. Named with /name style. Auto-cleanup semantics similar to files. Designed to the POSIX standard.
Q4. Write the complete sequence of calls to send one message and receive it. qid = msgget(IPC_PRIVATE, IPC_CREAT|0666); → struct Msg m = {.mtype=1}; → msgsnd(qid, &m, sizeof(m.data), 0); → msgrcv(qid, &r, sizeof(r.data), 0, 0); → msgctl(qid, IPC_RMID, NULL);
Q5. A server creates a System V queue and crashes without deleting it. What do you do? Run ipcs -q to find the stale queue, then ipcrm -q msqid to remove it. Prevent future crashes from leaving queues by installing signal handlers (SIGTERM/SIGINT) that call msgctl(IPC_RMID).
Q6. What is MSGMAX and what error does msgsnd() return when exceeded? MSGMAX is the kernel limit for the size of a single message payload (typically 8192 bytes). msgsnd() returns -1 with errno = EINVAL when the message exceeds this limit.
Q7. Why can’t you use epoll_wait() to wait for a message to arrive? epoll_wait() requires a file descriptor. System V message queues use an integer msqid, not an fd. The only way to wait for a message is a blocking msgrcv() call or polling with IPC_NOWAIT + sleep — both are inferior to epoll.
Q8. In the client-server pattern, why does each client create its own reply queue? So the server can route replies to the correct client using that client’s queue ID. If all clients shared one queue, replies meant for one client could be read by another.
Q9. What is the difference between ENOENT and EIDRM in message queue context? ENOENT = msgget() was called without IPC_CREAT but no queue with that key exists. EIDRM = a process was blocked in msgsnd/msgrcv when another process deleted the queue — “identifier removed.”
Q10. What does msg_qnum tell you in msqid_ds? The current number of messages in the queue. Combined with msg_cbytes (current bytes), it helps diagnose whether a queue is backing up because the consumer is too slow.

Leave a Reply

Your email address will not be published. Required fields are marked *