ptyFork() Implementation

Linux system programming — June 20, 2026 · 0 Comment

 

Chapter 64: Pseudoterminals
Part 2 of 4 — ptyFork() Implementation
fork()
+ setsid()
dup2()
stdin/out/err
TIOCSCTTY
BSD ioctl

📂 Tutorial Series Navigation
Part 1: Intro & PTY I/O ▶ Part 2: ptyFork() (this file) Part 3: Packet Mode Part 4: script(1) Program

🏷 Key Terms in This Part
ptyFork() fork() setsid() dup2() TIOCSCTTY controlling terminal session leader tcsetattr() TIOCSWINSZ slaveName

What is ptyFork()?

ptyFork() is a utility function that combines two operations into one:

  1. Open a PTY pair (master + slave)
  2. Fork a child process and give the child the slave side as its controlling terminal (stdin, stdout, stderr)

The parent keeps the master fd. The child gets the slave as its standard streams. After ptyFork() returns, the parent and child can communicate through the PTY pair just as a terminal emulator communicates with a shell.

This function is used internally by programs like SSH, screen, tmux, expect, and script.

📄 ptyFork() Function Signature
#include "pty_fork.h"   /* from TLPI pty/ directory */

pid_t ptyFork(int *masterFd,
              char *slaveName,
              size_t snLen,
              const struct termios *slaveTermios,
              const struct winsize  *slaveWS);
Parameter Direction Purpose
masterFd Output Filled with the PTY master file descriptor (parent only)
slaveName Output Buffer to receive the slave device name (e.g. /dev/pts/7)
snLen Input Size of the slaveName buffer
slaveTermios Input Terminal attributes to apply to slave (NULL = defaults)
slaveWS Input Window size to set on slave (NULL = defaults)

Return value: Behaves like fork(). Returns child PID to parent, 0 to child, -1 on error.

⚙ What ptyFork() Does — Step by Step

1 posix_openpt() — Open the PTY master device. Get back mfd.
2 grantpt() + unlockpt() — Set permissions and unlock the slave device.
3 ptsname() — Get the slave device name string (e.g. /dev/pts/5). Copy to slaveName buffer.
4 fork() — Create a child process. Parent gets child PID. Child continues below.
5 [CHILD] setsid() — Child becomes a new session leader with no controlling terminal.
6 [CHILD] close(mfd) — Child does not need the master fd.
7 [CHILD] open(slaveName) — Opening the slave makes it the controlling terminal of the new session (on Linux). On BSD, needs ioctl(TIOCSCTTY) as well.
8 [CHILD] tcsetattr() / ioctl(TIOCSWINSZ) — Optionally set terminal attributes and window size.
9 [CHILD] dup2(slaveFd, 0/1/2) — Make the slave stdin, stdout, and stderr of the child.

📈 Parent vs Child After ptyFork()

👤 PARENT PROCESS
  • Has masterFd (PTY master)
  • Does NOT have slaveFd
  • Returns child PID from ptyFork()
  • Reads/writes masterFd to talk to shell
  • Responsible for waiting (waitpid)

🐠 CHILD PROCESS
  • masterFd is CLOSED
  • stdin (fd 0) = PTY slave
  • stdout (fd 1) = PTY slave
  • stderr (fd 2) = PTY slave
  • New session leader (setsid done)
  • PTY slave is controlling terminal
  • Returns 0 from ptyFork()

🌎 Why setsid() is Critical Before Opening the Slave

The child calls setsid() before opening the slave device. Here is why this order matters:

Without setsid() With setsid() first
Child is still in parent’s session Child starts a brand new session
Child still has parent’s controlling terminal Child has NO controlling terminal yet
Opening slave does NOT make it controlling terminal Opening slave automatically becomes controlling terminal on Linux
Signals go to wrong process group Signals (SIGINT, SIGHUP) go to correct child session
💡 Rule: Always call setsid() in the child before opening the PTY slave. This is the correct and portable way to set up a new terminal session.

⚙ TIOCSCTTY — BSD Portability

On Linux, simply opening the slave device as a session leader (after setsid) automatically makes it the controlling terminal.

On BSD (FreeBSD, macOS), you also need to explicitly call:

#ifdef TIOCSCTTY
    if (ioctl(slaveFd, TIOCSCTTY, 0) == -1)
        err_exit("ptyFork: ioctl TIOCSCTTY");
#endif

The #ifdef guard means this code compiles on both Linux (where TIOCSCTTY may still be defined but is optional) and BSD (where it is required). This is standard practice in portable system-programming code.

📄 Full ptyFork() Implementation (from TLPI Chapter 64)
#include <fcntl.h>
#include <termios.h>
#include <sys/ioctl.h>
#include <string.h>
#include <errno.h>
#include <unistd.h>
#include "pty_fork.h"   /* declares ptyFork */
#include "tlpi_hdr.h"   /* err_exit macro */

#define MAX_SNAME 1000

pid_t ptyFork(int *masterFd, char *slaveName, size_t snLen,
              const struct termios *slaveTermios,
              const struct winsize  *slaveWS)
{
    int    mfd, slaveFd, savedErrno;
    pid_t  childPid;
    char   slname[MAX_SNAME];

    /* --- Open PTY master --- */
    mfd = posix_openpt(O_RDWR | O_NOCTTY);
    if (mfd == -1)
        return -1;

    if (grantpt(mfd) == -1) {
        savedErrno = errno;
        close(mfd);
        errno = savedErrno;
        return -1;
    }

    if (unlockpt(mfd) == -1) {
        savedErrno = errno;
        close(mfd);
        errno = savedErrno;
        return -1;
    }

    /* Get slave name */
    if (ptsname_r(mfd, slname, MAX_SNAME) != 0) {
        savedErrno = errno;
        close(mfd);
        errno = savedErrno;
        return -1;
    }

    /* Copy slave name to caller's buffer if requested */
    if (slaveName != NULL) {
        if (strlen(slname) < snLen) {
            strncpy(slaveName, slname, snLen);
        } else {
            close(mfd);
            errno = EOVERFLOW;
            return -1;
        }
    }

    /* --- Fork --- */
    childPid = fork();
    if (childPid == -1) {
        savedErrno = errno;
        close(mfd);
        errno = savedErrno;
        return -1;
    }

    /* --- PARENT returns here --- */
    if (childPid != 0) {
        *masterFd = mfd;
        return childPid;        /* parent gets child PID */
    }

    /* === CHILD from here === */

    /* Start a new session - child has no controlling terminal yet */
    if (setsid() == -1)
        err_exit("ptyFork: setsid");

    /* Child does not need master fd */
    close(mfd);

    /* Open slave - on Linux this also sets it as controlling terminal */
    slaveFd = open(slname, O_RDWR);
    if (slaveFd == -1)
        err_exit("ptyFork: open slave");

    /* BSD portability: explicitly acquire controlling terminal */
#ifdef TIOCSCTTY
    if (ioctl(slaveFd, TIOCSCTTY, 0) == -1)
        err_exit("ptyFork: ioctl TIOCSCTTY");
#endif

    /* Set terminal attributes on slave if caller provided them */
    if (slaveTermios != NULL)
        if (tcsetattr(slaveFd, TCSANOW, slaveTermios) == -1)
            err_exit("ptyFork: tcsetattr");

    /* Set window size on slave if caller provided it */
    if (slaveWS != NULL)
        if (ioctl(slaveFd, TIOCSWINSZ, slaveWS) == -1)
            err_exit("ptyFork: ioctl TIOCSWINSZ");

    /* Make slave stdin, stdout, stderr of child */
    if (dup2(slaveFd, STDIN_FILENO)  != STDIN_FILENO)
        err_exit("ptyFork: dup2 STDIN");
    if (dup2(slaveFd, STDOUT_FILENO) != STDOUT_FILENO)
        err_exit("ptyFork: dup2 STDOUT");
    if (dup2(slaveFd, STDERR_FILENO) != STDERR_FILENO)
        err_exit("ptyFork: dup2 STDERR");

    /* Close original slaveFd - no longer needed (stdin/out/err cover it) */
    if (slaveFd > STDERR_FILENO)
        close(slaveFd);

    return 0;   /* child returns 0 like fork() */
}

🔄 Why dup2() is Used for stdin/stdout/stderr

After opening the slave device, slaveFd might be file descriptor 5 or 7 or any number. The child’s shell will read from fd 0 (stdin), write to fd 1 (stdout), and print errors to fd 2 (stderr).

dup2(slaveFd, 0) means: “make fd 0 point to the same file as slaveFd“. It closes fd 0 first (the old stdin) and replaces it.

/* Before dup2: */
/* fd 0 = pipe or /dev/null or parent's terminal */
/* fd 5 = slaveFd (PTY slave) */

dup2(slaveFd, STDIN_FILENO);    /* fd 0 now = PTY slave */
dup2(slaveFd, STDOUT_FILENO);   /* fd 1 now = PTY slave */
dup2(slaveFd, STDERR_FILENO);   /* fd 2 now = PTY slave */

if (slaveFd > STDERR_FILENO)    /* slaveFd is now redundant */
    close(slaveFd);             /* close fd 5 - no leak */

/* After dup2: */
/* fd 0 = PTY slave (stdin) */
/* fd 1 = PTY slave (stdout) */
/* fd 2 = PTY slave (stderr) */
⚠ Safety check: The if (slaveFd > STDERR_FILENO) guard prevents accidentally closing fd 0, 1, or 2 if slaveFd happened to be one of those (very rare but possible).

📐 Setting Terminal Attributes and Window Size

ptyFork() optionally accepts terminal attributes (struct termios) and window size (struct winsize) to apply to the slave. This lets the caller match the slave to the actual user terminal.

#include <stdio.h>
#include <termios.h>
#include <sys/ioctl.h>
#include "pty_fork.h"

int main(void)
{
    int masterFd;
    pid_t childPid;
    struct termios tios;
    struct winsize ws;
    char slaveName[100];

    /* Read the current terminal settings from the real terminal */
    if (tcgetattr(STDIN_FILENO, &tios) == -1) {
        perror("tcgetattr");
        return 1;
    }

    /* Read window size from the real terminal */
    if (ioctl(STDIN_FILENO, TIOCGWINSZ, &ws) == -1) {
        perror("ioctl TIOCGWINSZ");
        return 1;
    }

    printf("Terminal: %d rows x %d cols\n", ws.ws_row, ws.ws_col);

    /* Fork with matching terminal config for the slave */
    childPid = ptyFork(&masterFd, slaveName, sizeof(slaveName),
                       &tios, &ws);

    if (childPid == -1) {
        perror("ptyFork");
        return 1;
    }

    if (childPid == 0) {
        /* Child: exec a shell */
        execlp("bash", "bash", NULL);
        perror("execlp");
        return 1;
    }

    /* Parent: masterFd talks to the shell */
    printf("Parent: child PID=%d, slave=%s\n", childPid, slaveName);
    /* ... read/write masterFd ... */

    return 0;
}

⚠ Common Mistakes with ptyFork()
Mistake 1: Not closing masterFd in the child
If the child forgets to close the master fd, then the child holds both ends of the PTY. When the parent closes its master fd, the child still has it open. This means SIGHUP is never sent to the child and the PTY never properly closes.
Mistake 2: Calling open(slave) before setsid()
If you open the slave before setsid(), the kernel may not make it the controlling terminal (or may assign the wrong terminal). Always call setsid() first.
Mistake 3: Not closing original slaveFd after dup2()
After dup2() makes fd 0/1/2 point to the slave, the original slaveFd is still open. It must be closed to avoid leaking a file descriptor. The guard if (slaveFd > STDERR_FILENO) handles this safely.
Mistake 4: Ignoring errno on fork() failure
If fork() fails, you must close masterFd before returning -1, otherwise the master fd leaks. The TLPI implementation saves errno before close() because close() could change errno.

🎓 Interview Questions — ptyFork() Implementation
Q1. What is the purpose of ptyFork() and what does it return to parent vs child?
ptyFork() opens a PTY pair, forks a child, and sets up the child with the slave as its stdin/stdout/stderr and controlling terminal. To the parent it returns the child PID and fills *masterFd. To the child it returns 0.
Q2. Why must the child call setsid() before opening the PTY slave?
setsid() creates a new session with the child as leader and removes any existing controlling terminal. On Linux, the first terminal opened by a session leader automatically becomes its controlling terminal. Without setsid(), the slave will not become the controlling terminal and signals will not work correctly.
Q3. What is TIOCSCTTY and on which platforms is it needed?
TIOCSCTTY is an ioctl that explicitly assigns a terminal as the controlling terminal of a session. It is required on BSD systems (FreeBSD, macOS). On Linux, opening the slave as a session leader is enough, but TIOCSCTTY is still defined and calling it is harmless.
Q4. Why does ptyFork() close masterFd in the child?
The child only needs the slave side. If the child also holds masterFd, then even after the parent closes the master, the PTY stays open (because the child still has it). SIGHUP and EIO behavior would break. Closing masterFd in the child ensures each process owns exactly one side of the PTY.
Q5. Explain the purpose of dup2() in ptyFork() and why a guard is needed after it.
dup2(slaveFd, 0) makes fd 0 (stdin) point to the PTY slave. After three dup2() calls, fd 0, 1, 2 all refer to the slave. The original slaveFd is now a fourth descriptor pointing to the same device and must be closed to prevent a leak. The guard if (slaveFd > STDERR_FILENO) prevents accidentally closing fd 0, 1, or 2 if slaveFd happened to be one of those numbers.
Q6. How does ptyFork() handle fork() failure cleanly?
It saves errno before calling close(masterFd), because close() may overwrite errno. After closing the master, it restores errno and returns -1. This ensures no fd is leaked and errno correctly reflects the fork failure reason.
Q7. What is the purpose of passing slaveTermios and slaveWS to ptyFork()?
They allow the caller to configure the slave’s terminal attributes and window size to match the user’s real terminal. For example, the script program reads the real terminal’s settings and window size, then passes them to ptyFork() so the shell inside the slave behaves exactly as it would on the real terminal (correct line discipline, correct terminal size for ncurses apps, etc.).
Q8. What is TCSANOW in the tcsetattr() call and why is it used?
TCSANOW means “apply the terminal attribute changes immediately”, without waiting for output to drain or input to flush. It is appropriate here because we are configuring a brand-new PTY slave before any I/O has occurred on it.

Next: Packet Mode — Flow Control Events on the PTY
Learn how TIOCPKT lets the master side detect flow control changes on the slave, and how select/poll integrate with packet mode.

Part 3: Packet Mode → ← Part 1: Intro & I/O

Leave a Reply

Your email address will not be published. Required fields are marked *

© 2026 embeddedpathashala.com - WordPress Video Theme by WPEnjoy