Opening a PTY Master with posix_openpt() and PTY Limits

 

Chapter 64 – Pseudoterminals (Part 2)
Opening a PTY Master: posix_openpt() & PTY Limits
/dev/ptmx
PTY master clone device
4096
Default PTY limit
SUSv3
POSIX standard

UNIX 98 PTY API Overview

To work with a UNIX 98 pseudoterminal, you follow a fixed sequence of four steps. Each step uses one library function. The functions are:

Step 1
posix_openpt()
Open an unused master
Step 2
grantpt()
Fix slave ownership
Step 3
unlockpt()
Unlock the slave
Step 4
ptsname() + open()
Get slave name & open it

In this part we cover Step 1 in detail: posix_openpt(), how it works internally, and the limits the kernel places on total PTY count.

64.2.1 Opening an Unused Master: posix_openpt()

posix_openpt() finds the next unused PTY master device, opens it, and returns a file descriptor. This is always your first step when creating a PTY pair.

Function Signature

#define _XOPEN_SOURCE 600
#include <stdlib.h>
#include <fcntl.h>

int posix_openpt(int flags);
/* Returns file descriptor on success, or -1 on error */

The flags Argument

You pass one or more of these flags ORed together:

Flag Meaning
O_RDWR Open master for both reading and writing. Always include this. You need to both read data coming from the slave and write data going to the slave.
O_NOCTTY Do not make this the process’s controlling terminal. On Linux this flag has no effect for the master (the master can never become a controlling terminal anyway), but on other systems it may be required. Include it for portability.
Why can’t the master become a controlling terminal?
The master is the driver side of the PTY — it is used by programs like sshd or xterm to control the terminal. It is not a terminal itself; it just looks like one from the slave’s perspective. Controlling terminals only make sense for the slave side.

What happens when posix_openpt() is called?

Two things happen automatically:

posix_openpt(O_RDWR)
✓ Kernel finds the next free PTY master slot
✓ Returns the lowest available file descriptor (same behavior as open())
✓ Creates the slave device file at /dev/pts/N automatically
Returns int mfd (master file descriptor)

How posix_openpt() Works Internally on Linux

posix_openpt() is a POSIX-standard wrapper. On Linux it is implemented very simply — it just opens the special clone device /dev/ptmx:

/* This is how posix_openpt() is implemented on Linux */
int
posix_openpt(int flags)
{
    return open("/dev/ptmx", flags);
}

/dev/ptmx is the PTY master multiplexer. Every time you open it, the kernel automatically:

Action on open(“/dev/ptmx”, flags)
1. Picks the next available PTY number (e.g., 5)
2. Creates /dev/pts/5 in the devpts filesystem
3. Locks the slave (must call unlockpt() before opening slave)
4. Returns a file descriptor for the master side

Before POSIX standardized posix_openpt(), developers had to manually open /dev/ptmx themselves. Using posix_openpt() is the portable way to do this in modern code.

Complete Code Example: Opening a PTY Master

This example shows all four steps together. We open the master, prepare the slave, and then print the slave’s device path. This is the correct standard sequence every PTY-using program must follow.

#define _XOPEN_SOURCE 600
#include <stdio.h>
#include <stdlib.h>
#include <fcntl.h>
#include <string.h>
#include <errno.h>

int main(void)
{
    int mfd;        /* master file descriptor */
    char *slaveName;

    /* Step 1: Open an unused PTY master device */
    mfd = posix_openpt(O_RDWR | O_NOCTTY);
    if (mfd == -1) {
        perror("posix_openpt");
        exit(EXIT_FAILURE);
    }
    printf("PTY master fd = %d\n", mfd);

    /* Step 2: Change slave ownership and permissions */
    if (grantpt(mfd) == -1) {
        perror("grantpt");
        close(mfd);
        exit(EXIT_FAILURE);
    }

    /* Step 3: Unlock the slave so it can be opened */
    if (unlockpt(mfd) == -1) {
        perror("unlockpt");
        close(mfd);
        exit(EXIT_FAILURE);
    }

    /* Step 4: Get the slave device name */
    slaveName = ptsname(mfd);
    if (slaveName == NULL) {
        perror("ptsname");
        close(mfd);
        exit(EXIT_FAILURE);
    }

    printf("PTY slave device: %s\n", slaveName);

    /* Now you can open the slave */
    int sfd = open(slaveName, O_RDWR | O_NOCTTY);
    if (sfd == -1) {
        perror("open slave");
        close(mfd);
        exit(EXIT_FAILURE);
    }
    printf("PTY slave fd = %d\n", sfd);

    /* At this point:
       - Write to mfd  -> readable from sfd (master writes, slave reads)
       - Write to sfd  -> readable from mfd (slave writes, master reads)
    */

    close(sfd);
    close(mfd);
    return 0;
}

Compile and run:

gcc -o pty_open pty_open.c
./pty_open
# Output example:
# PTY master fd = 3
# PTY slave device: /dev/pts/7
# PTY slave fd = 4

You can verify the slave device was created by running ls /dev/pts/ before and after.

Limits on the Number of UNIX 98 Pseudoterminals

Every PTY pair in use consumes a small amount of non-swappable kernel memory. Because of this, the kernel enforces a limit on how many PTY pairs can exist at once.

Kernel Version How Limit Is Set Default Maximum
Up to 2.6.3 CONFIG_UNIX98_PTYS kernel build option 256 2048
2.6.4 and later /proc/sys/kernel/pty/max (runtime tunable) 4096 1,048,576

Checking and Changing PTY Limits at Runtime

# View the current PTY limit
cat /proc/sys/kernel/pty/max

# View how many PTYs are currently in use
cat /proc/sys/kernel/pty/nr

# Increase the limit to 8192 (until next reboot)
echo 8192 | sudo tee /proc/sys/kernel/pty/max

# Make the change permanent (survives reboots)
echo "kernel.pty.max = 8192" | sudo tee -a /etc/sysctl.conf
sudo sysctl -p

/proc/sys/kernel/pty/
max (read/write) Maximum number of PTY pairs allowed system-wide. Default: 4096.
nr (read-only) Number of PTY pairs currently in use right now.
Practical tip: On servers running many SSH sessions, terminal multiplexers, or containers, you may hit the PTY limit. Running cat /proc/sys/kernel/pty/nr helps you monitor usage. Each SSH session typically consumes one PTY pair.

The /dev/pts Filesystem

When posix_openpt() is called, the kernel automatically creates a device node in /dev/pts/. This directory is a special devpts filesystem — it is not a regular directory on disk. The files in it are dynamically created and destroyed as PTY pairs are opened and closed.

# List current PTY slave device files
ls -l /dev/pts/

# Typical output:
# crw--w---- 1 ravi tty 136, 0 Jun 18 10:00 0
# crw--w---- 1 ravi tty 136, 1 Jun 18 10:05 1
# crw-rw-rw- 1 root tty   5, 2 Jun 18 09:58 ptmx

# /dev/pts/ptmx is the same as /dev/ptmx — clone device inside devpts

The number in each filename (0, 1, 2, …) is the PTY number. ptsname(mfd) returns the full path such as /dev/pts/0 for this slave.

Key Terms

posix_openpt() O_RDWR O_NOCTTY /dev/ptmx /dev/pts/N devpts filesystem CONFIG_UNIX98_PTYS /proc/sys/kernel/pty/max /proc/sys/kernel/pty/nr PTY clone device non-swappable kernel memory

Interview Questions & Answers

Q1. What does posix_openpt() do? What does it return?

It opens the next available PTY master device and returns a file descriptor for it (the master fd). On error it returns -1 and sets errno. It also causes the kernel to create the corresponding slave device file under /dev/pts/.

Q2. What is /dev/ptmx? How is it related to posix_openpt()?

/dev/ptmx is the PTY master multiplexer — a special kernel device. Each time you open it, the kernel allocates a new PTY pair and returns a master fd. On Linux, posix_openpt(flags) is literally implemented as open("/dev/ptmx", flags). Using posix_openpt() is the portable POSIX way to do this.

Q3. Why should O_NOCTTY be passed to posix_openpt() even though it has no effect on Linux?

For portability. On some other UNIX systems, opening a master device can accidentally make it the calling process’s controlling terminal. Passing O_NOCTTY prevents this on those systems. On Linux the master can never become a controlling terminal regardless, but the flag should still be included for portable code.

Q4. How do you check how many PTY pairs are currently in use on a Linux system?

Read the file /proc/sys/kernel/pty/nr. It is a read-only file that shows the current count. The maximum allowed is in /proc/sys/kernel/pty/max.

Q5. What happens if you try to open too many PTY pairs beyond the kernel limit?

posix_openpt() (or equivalently open("/dev/ptmx", ...)) will fail with errno set to ENOSPC or EIO depending on the implementation. You can increase the limit via /proc/sys/kernel/pty/max if you have root access.

Q6. What is the correct 4-step sequence to set up a UNIX 98 PTY pair?

1. posix_openpt(O_RDWR | O_NOCTTY) — open the master.
2. grantpt(mfd) — fix slave device ownership and permissions.
3. unlockpt(mfd) — remove the internal lock so slave can be opened.
4. ptsname(mfd)open(slaveName, O_RDWR | O_NOCTTY) — get slave name and open it.

Q7. Why does each PTY pair consume non-swappable kernel memory?

The kernel must maintain internal data structures for each PTY pair (buffers, state, line discipline). These structures must remain in physical RAM at all times because the kernel code that manages PTYs runs in kernel mode and cannot safely page-fault. This is why the kernel limits the total number of PTY pairs.

Leave a Reply

Your email address will not be published. Required fields are marked *

© 2026 embeddedpathashala.com - WordPress Video Theme by WPEnjoy