Why Per-Client Queues?
In the single-queue design, all traffic — requests AND responses — shares one queue. This creates deadlock and denial-of-service risks. The per-client queue design solves the deadlock problem by giving each client its own private message queue for receiving responses. The server has a well-known queue for requests, and each client creates an IPC_PRIVATE queue for receiving its own responses.
This is the design used by the file server application in Section 46.8 of TLPI and is generally preferred for production-grade applications.
The design involves three types of participants:
- Server’s queue — Created once at startup with a well-known IPC key. All clients know this key and use it to locate the server.
- Client’s private queue — Each client creates this with
IPC_PRIVATEbefore sending any request. The queue ID is passed to the server inside the request message. - Server child process — In a forking server, the main process creates a child for each request. The child handles the work and sends the response to the client’s private queue.
2. Sends request
6. Reads response
→
(well-known key)
(IPC_PRIVATE)
←
←
1. Client calls
msgget(IPC_PRIVATE, 0600) to create its private queue2. Client sends request to server’s well-known queue, embedding its private queue ID
3. Server’s main loop reads the request from its queue
4. Server forks a child to handle the request
5. Child opens the requested resource and sends responses to client’s private queue
6. Client reads all responses from its own private queue
7. Client calls
msgctl(clientQid, IPC_RMID, NULL) to delete its queueIPC_PRIVATE is a special key value (equal to 0) that instructs msgget() to always create a new, unique queue regardless of any existing queues. It is guaranteed not to match any existing IPC key. The resulting queue ID is unique and can only be used by processes that are told this ID explicitly (e.g., via a message, a pipe, or shared memory).
/* Creating a private queue — IPC_PRIVATE guarantees uniqueness */
int clientQid = msgget(IPC_PRIVATE, S_IRUSR | S_IWUSR);
/* ^^^^^^^^^^^
* IPC_PRIVATE = 0: always creates a brand new queue,
* never fails due to "queue already exists"
*/
if (clientQid == -1) {
perror("msgget IPC_PRIVATE");
exit(1);
}
printf("Private queue created: ID = %d\n", clientQid);
/* This ID must be communicated to the server via the request message */
With a named key (e.g., ftok() or a hardcoded value), multiple calls to
msgget() with the same key return the same queue. With IPC_PRIVATE, every call creates a new queue — exactly what clients need for a private response channel.Every IPC_PRIVATE queue a client creates counts toward the system-wide limit MSGMNI (max number of message queues). On older systems this default is very low (e.g., 16 or 32). If many clients connect simultaneously, you may hit this limit, causing msgget() to fail with ENOSPC. Modern Linux systems default to 32000, but embedded systems may have much lower limits.
Check/set with: cat /proc/sys/kernel/msgmni or sysctl kernel.msgmni=1024
If a client crashes after sending a request but before the server child sends the response, the server child will attempt to write to the client’s (now deleted) queue. This fails with EINVAL from msgsnd(). The server must handle this gracefully — check for EINVAL, log it, and exit the child cleanly without crashing the parent server.
IPC resources are NOT automatically cleaned up when a process exits. If a client crashes without calling msgctl(IPC_RMID), its private queue leaks. Over time, leaked queues accumulate and exhaust MSGMNI. Clients must register a cleanup handler for normal exits AND signals. Use atexit() + signal handlers to ensure cleanup.
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <signal.h>
#include <sys/types.h>
#include <sys/msg.h>
#define SERVER_KEY 0x1aaaaaa1 /* Well-known server queue key */
#define RESP_MT_DATA 2
#define RESP_MT_END 3
#define RESP_MT_FAILURE 1
/* Request message format */
struct requestMsg {
long mtype; /* Unused by server (set to 1) */
int clientId; /* ID of client's private queue */
char pathname[256]; /* File or resource requested */
};
/* Response message format */
struct responseMsg {
long mtype; /* RESP_MT_DATA, RESP_MT_END, or RESP_MT_FAILURE */
char data[4096];
};
static int clientQid = -1; /* Global so cleanup handler can access it */
/* Cleanup handler — called on exit and signals */
static void cleanup(void) {
if (clientQid != -1) {
msgctl(clientQid, IPC_RMID, NULL);
printf("Client queue deleted.\n");
}
}
static void sigHandler(int sig) {
cleanup();
_exit(EXIT_FAILURE);
}
int main(int argc, char *argv[]) {
int serverQid;
struct requestMsg req;
struct responseMsg resp;
if (argc != 2) {
fprintf(stderr, "Usage: %s <pathname>\n", argv[0]);
exit(1);
}
/* Register cleanup on normal exit and common signals */
atexit(cleanup);
signal(SIGINT, sigHandler);
signal(SIGTERM, sigHandler);
/* Step 1: Create private queue for receiving responses */
clientQid = msgget(IPC_PRIVATE, 0600);
if (clientQid == -1) {
perror("msgget IPC_PRIVATE");
exit(1);
}
printf("Client private queue: ID = %d\n", clientQid);
/* Step 2: Open the server's well-known queue */
serverQid = msgget(SERVER_KEY, 0);
if (serverQid == -1) {
perror("msgget server queue");
exit(1);
}
/* Step 3: Build and send request to server */
req.mtype = 1; /* Convention: all requests use mtype=1 */
req.clientId = clientQid; /* Tell server where to send responses */
strncpy(req.pathname, argv[1], sizeof(req.pathname) - 1);
if (msgsnd(serverQid, &req, sizeof(req) - sizeof(long), 0) == -1) {
perror("msgsnd request");
exit(1);
}
/* Step 4: Read responses from our private queue */
printf("Waiting for response...\n");
while (1) {
/* Receive ANY message from our private queue (msgtyp = 0) */
if (msgrcv(clientQid, &resp,
sizeof(resp) - sizeof(long),
0, /* Accept any mtype from this queue */
0) == -1) {
perror("msgrcv response");
break;
}
if (resp.mtype == RESP_MT_DATA) {
printf("DATA: %s", resp.data);
} else if (resp.mtype == RESP_MT_END) {
printf("\n[Transfer complete]\n");
break;
} else if (resp.mtype == RESP_MT_FAILURE) {
fprintf(stderr, "Server error: %s\n", resp.data);
break;
}
}
/* cleanup() called automatically via atexit() */
return 0;
}
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <fcntl.h>
#include <unistd.h>
#include <errno.h>
#include <sys/types.h>
#include <sys/msg.h>
#include <sys/wait.h>
#define SERVER_KEY 0x1aaaaaa1
#define RESP_BUF 4096
#define RESP_MT_DATA 2
#define RESP_MT_END 3
#define RESP_MT_FAILURE 1
struct requestMsg { long mtype; int clientId; char pathname[256]; };
struct responseMsg { long mtype; char data[RESP_BUF]; };
/* Child process: serve one client request */
static void serveRequest(const struct requestMsg *req) {
int fd, numRead;
struct responseMsg resp;
/* Try to open the requested file */
fd = open(req->pathname, O_RDONLY);
if (fd == -1) {
/* Send failure notification to client */
resp.mtype = RESP_MT_FAILURE;
snprintf(resp.data, RESP_BUF, "Cannot open '%s': %s",
req->pathname, strerror(errno));
msgsnd(req->clientId, &resp, sizeof(resp) - sizeof(long), 0);
return;
}
/* Stream file contents as DATA messages */
resp.mtype = RESP_MT_DATA;
while ((numRead = read(fd, resp.data, RESP_BUF)) > 0) {
if (msgsnd(req->clientId, &resp, numRead, 0) == -1) {
/* Client queue may be gone (client crashed) */
if (errno == EINVAL)
fprintf(stderr, "Client queue gone — client may have crashed\n");
else
perror("msgsnd data");
break;
}
}
close(fd);
/* Send end-of-transfer marker */
resp.mtype = RESP_MT_END;
resp.data[0] = '\0';
msgsnd(req->clientId, &resp, 1, 0);
}
int main(void) {
int serverQid;
struct requestMsg req;
pid_t pid;
/* Create server's well-known queue */
serverQid = msgget(SERVER_KEY, IPC_CREAT | IPC_EXCL | 0600);
if (serverQid == -1) {
if (errno == EEXIST) {
/* Queue already exists from a previous run — reopen it */
serverQid = msgget(SERVER_KEY, 0600);
}
if (serverQid == -1) { perror("msgget server"); exit(1); }
}
printf("File server started. Queue ID = %d\n", serverQid);
/* SIGCHLD: reap children automatically to prevent zombies */
signal(SIGCHLD, SIG_DFL); /* Or use SA_NOCLDWAIT */
while (1) {
/* Block until a client request arrives (mtype = 1) */
if (msgrcv(serverQid, &req,
sizeof(req) - sizeof(long),
1, 0) == -1) {
if (errno == EINTR) continue;
perror("msgrcv");
break;
}
printf("Handling request from client queue %d: '%s'\n",
req.clientId, req.pathname);
/* Fork a child to handle this request */
pid = fork();
if (pid == -1) {
perror("fork");
continue;
}
if (pid == 0) { /* Child */
serveRequest(&req);
exit(0);
}
/* Parent continues accepting more requests immediately */
/* (Children are reaped asynchronously via SIGCHLD) */
}
msgctl(serverQid, IPC_RMID, NULL);
return 0;
}
#include <stdio.h>
#include <stdlib.h>
#include <sys/types.h>
#include <sys/msg.h>
int main(void) {
/* Method 1: Read from /proc */
FILE *fp = fopen("/proc/sys/kernel/msgmni", "r");
if (fp) {
int limit;
fscanf(fp, "%d", &limit);
fclose(fp);
printf("Current MSGMNI (from /proc): %d\n", limit);
}
/* Method 2: Use MSG_INFO via msgctl */
struct msginfo info;
if (msgctl(0, MSG_INFO, (struct msqid_ds *)&info) != -1) {
printf("Current MSGMNI (from MSG_INFO): %d\n", info.msgmni);
}
/* Raising the limit requires root — done via sysctl or /proc:
*
* echo 1024 > /proc/sys/kernel/msgmni
* sysctl -w kernel.msgmni=1024
*
* For permanent change, add to /etc/sysctl.conf:
* kernel.msgmni = 1024
*/
/* Simulate what happens when limit is exceeded */
printf("\nAttempting to create many queues...\n");
int ids[100], count = 0;
while (count < 100) {
ids[count] = msgget(IPC_PRIVATE, 0600);
if (ids[count] == -1) {
perror("msgget (limit reached)");
printf("Could only create %d queues before hitting limit.\n", count);
break;
}
count++;
}
/* Cleanup: delete all created queues */
for (int i = 0; i < count; i++)
msgctl(ids[i], IPC_RMID, NULL);
return 0;
}
| Aspect | Single Queue | Per-Client Queue |
|---|---|---|
| Number of queues | 1 shared | 1 server + 1 per active client |
| Deadlock risk | Yes — queue overflow | No — each queue is independent |
| Malicious client risk | High — can clog shared queue | Low — only affects own queue |
| Routing mechanism | mtype = client PID | Client sends its queue ID |
| Message size constraint | Strict — shared capacity | More flexible |
| System resource usage | Low (1 queue) | Higher (N+1 queues) |
| MSGMNI concern | No | Yes — each client uses one slot |
| Cleanup needed? | No (one queue, server owns it) | Yes — clients must delete own queues |
| Complexity | Simple | More complex |
| Best for | Simple, low-traffic, trusted clients | Production, large messages, many clients |
Interview Questions — Per-Client Queue Design
IPC_PRIVATE is a special key value (equal to 0) passed to msgget() that guarantees creation of a brand new, unique message queue. Unlike a named key (e.g., from ftok()), using IPC_PRIVATE never matches an existing queue, so it always creates a fresh one. Clients use it for their private response queues because: (1) each client needs its own unique queue, (2) no coordination with other clients is needed to pick a key, and (3) there is no risk of accidentally sharing a queue with another process.clientQid) as a field in the request message body. When the server reads the request, it extracts this ID and uses it in msgsnd(clientId, &resp, ...) to send the response directly to that client’s queue. The client’s queue ID acts as a return address — similar to including a return address on an envelope.msgctl(IPC_RMID), its private queue becomes a zombie IPC object — it persists in the kernel until explicitly deleted or the system reboots. Over time, leaked queues accumulate and consume slots in the MSGMNI limit, eventually causing new queue creation to fail.
When the server child tries to send a response to a deleted queue, msgsnd() fails with EINVAL. The server should: check for EINVAL on msgsnd(), treat it as “client gone”, log the event, and exit the child cleanly. The server parent should never crash due to a misbehaving or crashed client.
msgsnd() blocks, but clients also can’t send more requests, creating a circular wait (deadlock).
In the per-client design, requests go to the server’s queue and responses go to each client’s own queue. The server’s queue only ever receives requests, and each client queue only ever receives that one client’s responses. A slow client can only fill its own queue — it cannot block the server from writing to other clients’ queues or prevent other clients from sending requests. This isolation eliminates the circular dependency that causes deadlock.
MSGMNI is the maximum number of System V message queues that can exist simultaneously on the system. When this limit is reached, any further call to msgget() to create a new queue fails with ENOSPC (no space for new IPC identifier). In a per-client queue design, each active client holds one queue, so MSGMNI directly limits the maximum number of concurrent clients. The limit can be read from /proc/sys/kernel/msgmni and changed with sysctl kernel.msgmni=N (requires root).msgrcv() loop and can accept the next request while the child handles the current one concurrently. This makes the server handle multiple simultaneous clients efficiently.msgtyp = 0 to msgrcv(). This means “receive the first available message regardless of its mtype”. This is correct and safe because the client’s private queue only ever receives messages intended for that client — no routing by mtype is needed since the queue itself provides isolation. The server uses the mtype field in this design to indicate the message purpose (DATA, END, or FAILURE), not for routing. The client reads these in FIFO order using msgtyp=0.(1) atexit() — registers a cleanup function that runs on normal program exit via
exit() or returning from main.(2) Signal handlers — catch SIGINT, SIGTERM, and SIGHUP to call
msgctl(IPC_RMID) before exiting.(3) Store the queue ID in a global variable accessible to the cleanup function.
Note: This cannot protect against SIGKILL (which cannot be caught), crashes due to hardware faults, or _exit() calls. For robustness, the server should also periodically scan for “orphaned” client queues (using MSG_STAT) and delete those belonging to dead processes.
