Per-Client Queue Design IPC_PRIVATE Queues & Robust IPC

 

Per-Client Queue Design
Chapter 46 – Section 46.7 (Part B)  |  IPC_PRIVATE Queues & Robust IPC
IPC_PRIVATE
Client Queue Key
MSGMNI
System Queue Limit
No Deadlock
Main Advantage

Why Per-Client Queues?

In the single-queue design, all traffic — requests AND responses — shares one queue. This creates deadlock and denial-of-service risks. The per-client queue design solves the deadlock problem by giving each client its own private message queue for receiving responses. The server has a well-known queue for requests, and each client creates an IPC_PRIVATE queue for receiving its own responses.

This is the design used by the file server application in Section 46.8 of TLPI and is generally preferred for production-grade applications.

Key Concepts

IPC_PRIVATE Per-client queue MSGMNI limit Queue ID in request IPC_RMID cleanup Stale queue handling Fork-based server Robust IPC design

1. Architecture: How Per-Client Queues Work

The design involves three types of participants:

  1. Server’s queue — Created once at startup with a well-known IPC key. All clients know this key and use it to locate the server.
  2. Client’s private queue — Each client creates this with IPC_PRIVATE before sending any request. The queue ID is passed to the server inside the request message.
  3. Server child process — In a forking server, the main process creates a child for each request. The child handles the work and sends the response to the client’s private queue.

Per-Client Queue Architecture

CLIENT
1. Creates private MQ
2. Sends request
6. Reads response

request msg: {clientQueueId, pathname}

SERVER MQ
(well-known key)
CLIENT PRIVATE MQ
(IPC_PRIVATE)

response msg(s): {data / end / error}

SERVER CHILD

Step-by-step flow:
1. Client calls msgget(IPC_PRIVATE, 0600) to create its private queue
2. Client sends request to server’s well-known queue, embedding its private queue ID
3. Server’s main loop reads the request from its queue
4. Server forks a child to handle the request
5. Child opens the requested resource and sends responses to client’s private queue
6. Client reads all responses from its own private queue
7. Client calls msgctl(clientQid, IPC_RMID, NULL) to delete its queue

2. IPC_PRIVATE — Creating a Private Queue

IPC_PRIVATE is a special key value (equal to 0) that instructs msgget() to always create a new, unique queue regardless of any existing queues. It is guaranteed not to match any existing IPC key. The resulting queue ID is unique and can only be used by processes that are told this ID explicitly (e.g., via a message, a pipe, or shared memory).

/* Creating a private queue — IPC_PRIVATE guarantees uniqueness */
int clientQid = msgget(IPC_PRIVATE, S_IRUSR | S_IWUSR);
/*                       ^^^^^^^^^^^
 *  IPC_PRIVATE = 0: always creates a brand new queue,
 *  never fails due to "queue already exists"
 */

if (clientQid == -1) {
    perror("msgget IPC_PRIVATE");
    exit(1);
}

printf("Private queue created: ID = %d\n", clientQid);

/* This ID must be communicated to the server via the request message */
IPC_PRIVATE vs a named key:
With a named key (e.g., ftok() or a hardcoded value), multiple calls to msgget() with the same key return the same queue. With IPC_PRIVATE, every call creates a new queue — exactly what clients need for a private response channel.

3. Key Considerations and Potential Problems
⚠ MSGMNI: System-wide queue limit

Every IPC_PRIVATE queue a client creates counts toward the system-wide limit MSGMNI (max number of message queues). On older systems this default is very low (e.g., 16 or 32). If many clients connect simultaneously, you may hit this limit, causing msgget() to fail with ENOSPC. Modern Linux systems default to 32000, but embedded systems may have much lower limits.
Check/set with: cat /proc/sys/kernel/msgmni or sysctl kernel.msgmni=1024

⚠ Stale client queues (client crashed mid-request)

If a client crashes after sending a request but before the server child sends the response, the server child will attempt to write to the client’s (now deleted) queue. This fails with EINVAL from msgsnd(). The server must handle this gracefully — check for EINVAL, log it, and exit the child cleanly without crashing the parent server.

⛔ Queue cleanup responsibility

IPC resources are NOT automatically cleaned up when a process exits. If a client crashes without calling msgctl(IPC_RMID), its private queue leaks. Over time, leaked queues accumulate and exhaust MSGMNI. Clients must register a cleanup handler for normal exits AND signals. Use atexit() + signal handlers to ensure cleanup.

Code Example 1: Client — Creating Private Queue and Sending Request
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <signal.h>
#include <sys/types.h>
#include <sys/msg.h>

#define SERVER_KEY  0x1aaaaaa1  /* Well-known server queue key */
#define RESP_MT_DATA    2
#define RESP_MT_END     3
#define RESP_MT_FAILURE 1

/* Request message format */
struct requestMsg {
    long mtype;          /* Unused by server (set to 1) */
    int  clientId;       /* ID of client's private queue */
    char pathname[256];  /* File or resource requested */
};

/* Response message format */
struct responseMsg {
    long mtype;          /* RESP_MT_DATA, RESP_MT_END, or RESP_MT_FAILURE */
    char data[4096];
};

static int clientQid = -1; /* Global so cleanup handler can access it */

/* Cleanup handler — called on exit and signals */
static void cleanup(void) {
    if (clientQid != -1) {
        msgctl(clientQid, IPC_RMID, NULL);
        printf("Client queue deleted.\n");
    }
}

static void sigHandler(int sig) {
    cleanup();
    _exit(EXIT_FAILURE);
}

int main(int argc, char *argv[]) {
    int serverQid;
    struct requestMsg  req;
    struct responseMsg resp;

    if (argc != 2) {
        fprintf(stderr, "Usage: %s <pathname>\n", argv[0]);
        exit(1);
    }

    /* Register cleanup on normal exit and common signals */
    atexit(cleanup);
    signal(SIGINT,  sigHandler);
    signal(SIGTERM, sigHandler);

    /* Step 1: Create private queue for receiving responses */
    clientQid = msgget(IPC_PRIVATE, 0600);
    if (clientQid == -1) {
        perror("msgget IPC_PRIVATE");
        exit(1);
    }
    printf("Client private queue: ID = %d\n", clientQid);

    /* Step 2: Open the server's well-known queue */
    serverQid = msgget(SERVER_KEY, 0);
    if (serverQid == -1) {
        perror("msgget server queue");
        exit(1);
    }

    /* Step 3: Build and send request to server */
    req.mtype    = 1;           /* Convention: all requests use mtype=1 */
    req.clientId = clientQid;   /* Tell server where to send responses */
    strncpy(req.pathname, argv[1], sizeof(req.pathname) - 1);

    if (msgsnd(serverQid, &req, sizeof(req) - sizeof(long), 0) == -1) {
        perror("msgsnd request");
        exit(1);
    }

    /* Step 4: Read responses from our private queue */
    printf("Waiting for response...\n");
    while (1) {
        /* Receive ANY message from our private queue (msgtyp = 0) */
        if (msgrcv(clientQid, &resp,
                   sizeof(resp) - sizeof(long),
                   0,    /* Accept any mtype from this queue */
                   0) == -1) {
            perror("msgrcv response");
            break;
        }

        if (resp.mtype == RESP_MT_DATA) {
            printf("DATA: %s", resp.data);
        } else if (resp.mtype == RESP_MT_END) {
            printf("\n[Transfer complete]\n");
            break;
        } else if (resp.mtype == RESP_MT_FAILURE) {
            fprintf(stderr, "Server error: %s\n", resp.data);
            break;
        }
    }
    /* cleanup() called automatically via atexit() */
    return 0;
}

Code Example 2: Server — Receiving Requests and Forking a Child
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <fcntl.h>
#include <unistd.h>
#include <errno.h>
#include <sys/types.h>
#include <sys/msg.h>
#include <sys/wait.h>

#define SERVER_KEY  0x1aaaaaa1
#define RESP_BUF    4096
#define RESP_MT_DATA    2
#define RESP_MT_END     3
#define RESP_MT_FAILURE 1

struct requestMsg  { long mtype; int clientId; char pathname[256]; };
struct responseMsg { long mtype; char data[RESP_BUF]; };

/* Child process: serve one client request */
static void serveRequest(const struct requestMsg *req) {
    int fd, numRead;
    struct responseMsg resp;

    /* Try to open the requested file */
    fd = open(req->pathname, O_RDONLY);
    if (fd == -1) {
        /* Send failure notification to client */
        resp.mtype = RESP_MT_FAILURE;
        snprintf(resp.data, RESP_BUF, "Cannot open '%s': %s",
                 req->pathname, strerror(errno));
        msgsnd(req->clientId, &resp, sizeof(resp) - sizeof(long), 0);
        return;
    }

    /* Stream file contents as DATA messages */
    resp.mtype = RESP_MT_DATA;
    while ((numRead = read(fd, resp.data, RESP_BUF)) > 0) {
        if (msgsnd(req->clientId, &resp, numRead, 0) == -1) {
            /* Client queue may be gone (client crashed) */
            if (errno == EINVAL)
                fprintf(stderr, "Client queue gone — client may have crashed\n");
            else
                perror("msgsnd data");
            break;
        }
    }
    close(fd);

    /* Send end-of-transfer marker */
    resp.mtype = RESP_MT_END;
    resp.data[0] = '\0';
    msgsnd(req->clientId, &resp, 1, 0);
}

int main(void) {
    int serverQid;
    struct requestMsg req;
    pid_t pid;

    /* Create server's well-known queue */
    serverQid = msgget(SERVER_KEY, IPC_CREAT | IPC_EXCL | 0600);
    if (serverQid == -1) {
        if (errno == EEXIST) {
            /* Queue already exists from a previous run — reopen it */
            serverQid = msgget(SERVER_KEY, 0600);
        }
        if (serverQid == -1) { perror("msgget server"); exit(1); }
    }

    printf("File server started. Queue ID = %d\n", serverQid);

    /* SIGCHLD: reap children automatically to prevent zombies */
    signal(SIGCHLD, SIG_DFL); /* Or use SA_NOCLDWAIT */

    while (1) {
        /* Block until a client request arrives (mtype = 1) */
        if (msgrcv(serverQid, &req,
                   sizeof(req) - sizeof(long),
                   1, 0) == -1) {
            if (errno == EINTR) continue;
            perror("msgrcv");
            break;
        }

        printf("Handling request from client queue %d: '%s'\n",
               req.clientId, req.pathname);

        /* Fork a child to handle this request */
        pid = fork();
        if (pid == -1) {
            perror("fork");
            continue;
        }

        if (pid == 0) {          /* Child */
            serveRequest(&req);
            exit(0);
        }
        /* Parent continues accepting more requests immediately */
        /* (Children are reaped asynchronously via SIGCHLD) */
    }

    msgctl(serverQid, IPC_RMID, NULL);
    return 0;
}
Why fork()? The server forks a child for each request so it can immediately return to accepting new requests without blocking on file I/O. The child handles potentially slow file reads while the parent stays responsive. This is a classic Unix design pattern.

Code Example 3: Checking and Raising the MSGMNI Limit
#include <stdio.h>
#include <stdlib.h>
#include <sys/types.h>
#include <sys/msg.h>

int main(void) {
    /* Method 1: Read from /proc */
    FILE *fp = fopen("/proc/sys/kernel/msgmni", "r");
    if (fp) {
        int limit;
        fscanf(fp, "%d", &limit);
        fclose(fp);
        printf("Current MSGMNI (from /proc): %d\n", limit);
    }

    /* Method 2: Use MSG_INFO via msgctl */
    struct msginfo info;
    if (msgctl(0, MSG_INFO, (struct msqid_ds *)&info) != -1) {
        printf("Current MSGMNI (from MSG_INFO): %d\n", info.msgmni);
    }

    /* Raising the limit requires root — done via sysctl or /proc:
     *
     *   echo 1024 > /proc/sys/kernel/msgmni
     *   sysctl -w kernel.msgmni=1024
     *
     * For permanent change, add to /etc/sysctl.conf:
     *   kernel.msgmni = 1024
     */

    /* Simulate what happens when limit is exceeded */
    printf("\nAttempting to create many queues...\n");
    int ids[100], count = 0;
    while (count < 100) {
        ids[count] = msgget(IPC_PRIVATE, 0600);
        if (ids[count] == -1) {
            perror("msgget (limit reached)");
            printf("Could only create %d queues before hitting limit.\n", count);
            break;
        }
        count++;
    }

    /* Cleanup: delete all created queues */
    for (int i = 0; i < count; i++)
        msgctl(ids[i], IPC_RMID, NULL);

    return 0;
}

4. Single Queue vs Per-Client Queue — Full Comparison
Aspect Single Queue Per-Client Queue
Number of queues 1 shared 1 server + 1 per active client
Deadlock risk Yes — queue overflow No — each queue is independent
Malicious client risk High — can clog shared queue Low — only affects own queue
Routing mechanism mtype = client PID Client sends its queue ID
Message size constraint Strict — shared capacity More flexible
System resource usage Low (1 queue) Higher (N+1 queues)
MSGMNI concern No Yes — each client uses one slot
Cleanup needed? No (one queue, server owns it) Yes — clients must delete own queues
Complexity Simple More complex
Best for Simple, low-traffic, trusted clients Production, large messages, many clients

Interview Questions — Per-Client Queue Design

Q1. What is IPC_PRIVATE and why is it used for client queues?
IPC_PRIVATE is a special key value (equal to 0) passed to msgget() that guarantees creation of a brand new, unique message queue. Unlike a named key (e.g., from ftok()), using IPC_PRIVATE never matches an existing queue, so it always creates a fresh one. Clients use it for their private response queues because: (1) each client needs its own unique queue, (2) no coordination with other clients is needed to pick a key, and (3) there is no risk of accidentally sharing a queue with another process.
Q2. How does the server know where to send the response in the per-client queue design?
The client includes the ID of its private queue (clientQid) as a field in the request message body. When the server reads the request, it extracts this ID and uses it in msgsnd(clientId, &resp, ...) to send the response directly to that client’s queue. The client’s queue ID acts as a return address — similar to including a return address on an envelope.
Q3. What happens if a client crashes without deleting its private queue? How should the server handle this?
If a client crashes without calling msgctl(IPC_RMID), its private queue becomes a zombie IPC object — it persists in the kernel until explicitly deleted or the system reboots. Over time, leaked queues accumulate and consume slots in the MSGMNI limit, eventually causing new queue creation to fail.

When the server child tries to send a response to a deleted queue, msgsnd() fails with EINVAL. The server should: check for EINVAL on msgsnd(), treat it as “client gone”, log the event, and exit the child cleanly. The server parent should never crash due to a misbehaving or crashed client.

Q4. Why does the per-client design prevent deadlocks that affect the single-queue design?
In the single-queue design, requests and responses share capacity. If the shared queue fills up with responses that clients haven’t read, the server’s msgsnd() blocks, but clients also can’t send more requests, creating a circular wait (deadlock).

In the per-client design, requests go to the server’s queue and responses go to each client’s own queue. The server’s queue only ever receives requests, and each client queue only ever receives that one client’s responses. A slow client can only fill its own queue — it cannot block the server from writing to other clients’ queues or prevent other clients from sending requests. This isolation eliminates the circular dependency that causes deadlock.

Q5. What is the MSGMNI system parameter, and what error does exceeding it cause?
MSGMNI is the maximum number of System V message queues that can exist simultaneously on the system. When this limit is reached, any further call to msgget() to create a new queue fails with ENOSPC (no space for new IPC identifier). In a per-client queue design, each active client holds one queue, so MSGMNI directly limits the maximum number of concurrent clients. The limit can be read from /proc/sys/kernel/msgmni and changed with sysctl kernel.msgmni=N (requires root).
Q6. Why does the server fork a child to handle each request instead of handling it in the main process?
Handling a request (e.g., reading a file and sending its contents) may take significant time due to disk I/O. If the server’s main process performed this work directly, it would be blocked during that time and unable to accept new requests from other clients — effectively becoming a serial server. By forking a child for each request, the main process returns immediately to its msgrcv() loop and can accept the next request while the child handles the current one concurrently. This makes the server handle multiple simultaneous clients efficiently.
Q7. In the per-client design, what msgtyp value does the client pass to msgrcv() and why?
The client passes msgtyp = 0 to msgrcv(). This means “receive the first available message regardless of its mtype”. This is correct and safe because the client’s private queue only ever receives messages intended for that client — no routing by mtype is needed since the queue itself provides isolation. The server uses the mtype field in this design to indicate the message purpose (DATA, END, or FAILURE), not for routing. The client reads these in FIFO order using msgtyp=0.
Q8. How do you ensure a client’s private queue is always deleted, even if the client crashes?
Use a combination of:
(1) atexit() — registers a cleanup function that runs on normal program exit via exit() or returning from main.
(2) Signal handlers — catch SIGINT, SIGTERM, and SIGHUP to call msgctl(IPC_RMID) before exiting.
(3) Store the queue ID in a global variable accessible to the cleanup function.

Note: This cannot protect against SIGKILL (which cannot be caught), crashes due to hardware faults, or _exit() calls. For robustness, the server should also periodically scan for “orphaned” client queues (using MSG_STAT) and delete those belonging to dead processes.

Leave a Reply

Your email address will not be published. Required fields are marked *