Pseudoterminals Opening a PTY Master: posix_openpt, grantpt, unlockpt, ptsname & ptyMasterOpen()

 

Chapter 64: Pseudoterminals
Part 2 of 3 โ€” Opening a PTY Master: posix_openpt, grantpt, unlockpt, ptsname & ptyMasterOpen()
4
Key Functions
UNIX98
PTY Style
TLPI
ยง64.2โ€“64.3

๐Ÿ“š Chapter Navigation

The 4-Step Process to Open a PTY Master

Opening a UNIX 98 pseudoterminal master is a well-defined sequence of four function calls. Each step has a specific purpose, and skipping any one of them will either cause failure or create a security hole. The TLPI function ptyMasterOpen() wraps all four steps into a single, safe, reusable function.

All four functions are declared in <stdlib.h> and require _XOPEN_SOURCE 600 (or _GNU_SOURCE) to be defined.

Key Terms in This Part

posix_openpt() grantpt() unlockpt() ptsname() ptsname_r() ptyMasterOpen() /dev/ptmx EOVERFLOW savedErrno

๐Ÿ”ข The 4-Step PTY Master Open Sequence

1
posix_openpt(flags)
Opens /dev/ptmx and returns a master fd. Like open("/dev/ptmx", flags) but portable.
โ†“

2
grantpt(masterFd)
Changes ownership of the slave PTY to the calling process’s UID and sets permissions to 620 (owner rw, group w). Prevents other users from accessing this slave.
โ†“

3
unlockpt(masterFd)
Removes an internal kernel lock on the slave. The slave cannot be opened until this lock is cleared. This prevents race conditions during setup.
โ†“

4
ptsname(masterFd)
Returns the name of the slave PTY device (e.g., /dev/pts/3). Now you can open() the slave.

๐Ÿ“– Function Details

1. posix_openpt()

Opens the PTY master multiplexer device and returns a file descriptor.

#include <stdlib.h>
#include <fcntl.h>

int posix_openpt(int flags);
/*
 * flags: O_RDWR  โ€” open for reading and writing (always required)
 *        O_NOCTTY โ€” do NOT make this terminal the controlling terminal
 *                   of the calling process (almost always used)
 *
 * Returns: master fd on success, -1 on error
 *
 * Internally: equivalent to open("/dev/ptmx", flags)
 */

int masterFd = posix_openpt(O_RDWR | O_NOCTTY);
if (masterFd == -1) {
    perror("posix_openpt");
    exit(EXIT_FAILURE);
}

2. grantpt()

Grants the calling process permission to access the slave PTY. On older systems this forks a setuid helper; on modern Linux it is a simple syscall.

int grantpt(int masterFd);
/*
 * Changes slave PTY device:
 *   - Owner: set to the UID of the calling process
 *   - Group: set to 'tty' group
 *   - Permissions: 0620 (owner rw, group write only)
 *
 * IMPORTANT: Do NOT have a SIGCHLD handler that calls waitpid() 
 * on all children when calling grantpt() on older systems,
 * as it may internally fork a helper process.
 *
 * Returns: 0 on success, -1 on error
 */

if (grantpt(masterFd) == -1) {
    perror("grantpt");
    close(masterFd);
    exit(EXIT_FAILURE);
}

3. unlockpt()

Clears the internal lock on the slave PTY. Until this is called, any attempt to open() the slave will fail with EIO.

int unlockpt(int masterFd);
/*
 * Clears the kernel-internal lock on the slave PTY associated
 * with this master fd. Must be called AFTER grantpt().
 *
 * Returns: 0 on success, -1 on error
 */

if (unlockpt(masterFd) == -1) {
    perror("unlockpt");
    close(masterFd);
    exit(EXIT_FAILURE);
}

4. ptsname() and ptsname_r()

Returns the name of the slave PTY device corresponding to the master fd.

char *ptsname(int masterFd);
/*
 * Returns: pointer to a static string like "/dev/pts/3"
 *          NULL on error
 *
 * WARNING: Returns pointer to static storage โ€” NOT thread-safe.
 *          Subsequent calls overwrite the same buffer.
 */

/* Thread-safe version (GNU extension): */
int ptsname_r(int masterFd, char *buf, size_t buflen);
/*
 * Writes the slave name into 'buf' (size buflen).
 * Requires: #define _GNU_SOURCE
 * Returns: 0 on success, nonzero on error
 */

char slaveName[256];
char *p = ptsname(masterFd);
if (p == NULL) {
    perror("ptsname");
    close(masterFd);
    exit(EXIT_FAILURE);
}
strncpy(slaveName, p, sizeof(slaveName));
printf("Slave PTY: %s\n", slaveName);   /* e.g., /dev/pts/5 */

โš ๏ธ The savedErrno Pattern โ€“ Why It Matters

When an error occurs and you need to close the master fd before returning -1, the close() call itself may modify errno. This would overwrite the original error code. TLPI’s ptyMasterOpen() solves this with a savedErrno pattern:

/* BAD: close() may overwrite errno */
if (grantpt(masterFd) == -1) {
    close(masterFd);        /* errno from grantpt() may be lost here! */
    return -1;
}

/* GOOD: Save errno before close() */
if (grantpt(masterFd) == -1) {
    int savedErrno = errno;     /* save the real error */
    close(masterFd);            /* errno might change here */
    errno = savedErrno;         /* restore the real error */
    return -1;
}
/* Now the caller's perror() / strerror(errno) shows the right message */

This pattern appears in all three error paths inside ptyMasterOpen().

๐Ÿ’ป ptyMasterOpen() โ€“ Full Implementation (Listing 64-1)

This is the complete implementation of ptyMasterOpen() as presented in TLPI. It combines all four steps with proper error handling.

/* File: pty_master_open.c */

#define _XOPEN_SOURCE 600   /* Required for posix_openpt(), grantpt(),
                               unlockpt(), ptsname() */
#include <stdlib.h>
#include <fcntl.h>
#include <string.h>
#include <errno.h>
#include <unistd.h>

/*
 * ptyMasterOpen() - Open a PTY master and get the slave name.
 *
 * slaveName : buffer to hold the slave device name (e.g., "/dev/pts/3")
 * snLen     : size of slaveName buffer
 *
 * Returns   : master fd on success, -1 on error (errno set)
 */
int ptyMasterOpen(char *slaveName, size_t snLen)
{
    int masterFd, savedErrno;
    char *p;

    /* Step 1: Open the PTY master via /dev/ptmx */
    masterFd = posix_openpt(O_RDWR | O_NOCTTY);
    if (masterFd == -1)
        return -1;      /* errno already set by posix_openpt */

    /* Step 2: Grant access rights on slave PTY */
    if (grantpt(masterFd) == -1) {
        savedErrno = errno;
        close(masterFd);        /* might change errno */
        errno = savedErrno;
        return -1;
    }

    /* Step 3: Unlock the slave PTY (remove kernel lock) */
    if (unlockpt(masterFd) == -1) {
        savedErrno = errno;
        close(masterFd);
        errno = savedErrno;
        return -1;
    }

    /* Step 4: Get the name of the slave PTY device */
    p = ptsname(masterFd);
    if (p == NULL) {
        savedErrno = errno;
        close(masterFd);
        errno = savedErrno;
        return -1;
    }

    /* Copy slave name to caller's buffer */
    if (strlen(p) < snLen) {
        strncpy(slaveName, p, snLen);
    } else {
        /* Buffer too small โ€” return EOVERFLOW */
        close(masterFd);
        errno = EOVERFLOW;
        return -1;
    }

    return masterFd;    /* Caller now owns this fd */
}

Error Handling Flow Inside ptyMasterOpen()
posix_openpt()
โ†’ fail: return -1 directly (no fd to close)
โ†“ success
grantpt()
โ†’ fail: save errno, close(masterFd), restore errno, return -1
โ†“ success
unlockpt()
โ†’ fail: save errno, close(masterFd), restore errno, return -1
โ†“ success
ptsname()
โ†’ fail: save errno, close(masterFd), restore errno, return -1
โ†“ success
strlen check
โ†’ fail: close(masterFd), set EOVERFLOW, return -1
โ†“ success
return masterFd โœ…

๐Ÿ’ป Complete Example: Open a PTY Pair and Exchange Data

This example opens a PTY master+slave pair and demonstrates writing from master to slave and reading back from slave.

/* File: pty_pair_demo.c
 * Compile: gcc -o pty_pair_demo pty_pair_demo.c
 * Run:     ./pty_pair_demo
 */

#define _XOPEN_SOURCE 600
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <fcntl.h>
#include <errno.h>

#define SLAVE_NAME_MAX 256

int ptyMasterOpen(char *slaveName, size_t snLen);  /* from above */

int main(void)
{
    int masterFd, slaveFd;
    char slaveName[SLAVE_NAME_MAX];
    char wbuf[] = "Hello PTY!\n";
    char rbuf[64];
    ssize_t n;

    /* Open the PTY master and get slave name */
    masterFd = ptyMasterOpen(slaveName, sizeof(slaveName));
    if (masterFd == -1) {
        perror("ptyMasterOpen");
        exit(EXIT_FAILURE);
    }
    printf("Master fd = %d, Slave = %s\n", masterFd, slaveName);

    /* Open the PTY slave */
    slaveFd = open(slaveName, O_RDWR);
    if (slaveFd == -1) {
        perror("open slave");
        close(masterFd);
        exit(EXIT_FAILURE);
    }
    printf("Slave fd  = %d\n", slaveFd);

    /*
     * In canonical (cooked) mode, the slave's line discipline 
     * echoes writes from master back to the master.
     * Write to master โ†’ slave's line discipline echoes โ†’ read back from master.
     */
    if (write(masterFd, wbuf, strlen(wbuf)) == -1) {
        perror("write master");
        exit(EXIT_FAILURE);
    }

    /* Read the echo back from the master side */
    n = read(masterFd, rbuf, sizeof(rbuf) - 1);
    if (n == -1) {
        perror("read master");
        exit(EXIT_FAILURE);
    }
    rbuf[n] = '\0';
    printf("Read back from master: [%s]\n", rbuf);

    /* Also read from slave side to drain the line-buffered input */
    n = read(slaveFd, rbuf, sizeof(rbuf) - 1);
    if (n > 0) {
        rbuf[n] = '\0';
        printf("Read from slave:       [%s]\n", rbuf);
    }

    close(slaveFd);
    close(masterFd);
    return 0;
}

/*
 * Expected output:
 * Master fd = 5, Slave = /dev/pts/7
 * Slave fd  = 6
 * Read back from master: [Hello PTY!
 * ]
 * Read from slave:       [Hello PTY!
 * ]
 *
 * Note: The line discipline's echo causes data written to master to 
 *       be echoed back to master AND available on the slave.
 */

๐Ÿ” Why slaveName + snLen Arguments?

You might ask: why not just call ptsname() yourself after calling ptyMasterOpen()? TLPI’s design choice is explained clearly:

โŒ Without slaveName argument
  • Caller must call ptsname() separately
  • Caller handles thread-safety concern
  • BSD reimplementation would break API
โœ… With slaveName + snLen
  • Self-contained: one call gives both master fd and slave name
  • BSD reimplementation can use its own method internally
  • Caller code stays identical regardless of PTY style

The snLen argument protects against buffer overflow. If the slave name (e.g. /dev/pts/12345) is longer than snLen bytes, ptyMasterOpen() returns -1 with errno = EOVERFLOW.

๐ŸŽฏ Interview Questions โ€“ Opening PTY Master
Q1. What are the four steps to open a UNIX 98 PTY master? Why is the order important?

(1) posix_openpt() โ€” obtain master fd. (2) grantpt() โ€” set ownership/permissions on slave. (3) unlockpt() โ€” remove slave’s kernel lock. (4) ptsname() โ€” get slave device name. Order matters: the slave is locked until unlockpt(), so it can’t be opened before step 3. grantpt() must precede unlockpt() per POSIX.

Q2. What permissions does grantpt() set on the slave PTY? Why?

grantpt() sets owner to the calling process’s UID, group to tty, and permissions to 0620 (owner rw, group w only). This ensures only the user who opened the master can read from the slave, preventing other users from eavesdropping on the terminal session.

Q3. Why does ptyMasterOpen() save and restore errno around close()?

close() can fail (e.g., EINTR, EIO) and overwrite errno. If you called grantpt(), got an error, then called close(), the caller’s perror() would print the wrong error. Saving errno before and restoring it after close() ensures the original error is preserved.

Q4. What is the difference between ptsname() and ptsname_r()?

ptsname() returns a pointer to a static internal buffer โ€” it is not thread-safe because a second concurrent call will overwrite the same buffer. ptsname_r() is a GNU extension (requires _GNU_SOURCE) that writes into a caller-provided buffer, making it thread-safe.

Q5. What flags should be passed to posix_openpt()? Why O_NOCTTY?

Always pass O_RDWR (must read and write). O_NOCTTY prevents the master from becoming the controlling terminal of the calling process. The master is a control channel, not a terminal โ€” the slave should be the controlling terminal (for the child process), not the master.

Q6. What error does ptyMasterOpen() return if the slaveName buffer is too small?

It sets errno = EOVERFLOW and returns -1. This is because the slave name (like /dev/pts/N) might be longer than the provided buffer. The function uses strlen(p) < snLen to check โ€” note it uses strict less-than so there is room for the null terminator.

Q7. Why does ptyMasterOpen() abstract away the BSD vs UNIX 98 difference?

BSD PTY does not have ptsname() โ€” the slave name must be derived differently (by searching /dev/ttyXY pairs). By encapsulating the slave name retrieval inside ptyMasterOpen(), all calling code in the rest of Chapter 64 can work unchanged with either PTY style โ€” only the implementation of ptyMasterOpen() changes.

Next: ptyFork() โ€“ Forking with a PTY Pair
Learn how ptyFork() sets up a child process connected to the parent via a PTY, including setsid(), TIOCSCTTY, and dup2()

Part 3: ptyFork() โ†’ โ† Part 1: Introduction

Leave a Reply

Your email address will not be published. Required fields are marked *