mprotect() – Changing Memory Protection Virtual Memory Operations

 

mprotect() – Changing Memory Protection
Chapter 50 – Virtual Memory Operations | Topic 1 of 4

← Chapter Index  |  Topic 1: mprotect()  |  Topic 2: mlock() →

What is mprotect()?

Every region of a process’s virtual memory has protection bits that control what operations are allowed on it. These are enforced by the hardware Memory Management Unit (MMU). The mprotect() system call lets you change these protection bits at runtime for any page-aligned region of virtual memory.

By default, when you map memory or allocate it with malloc(), the protection matches the intended use — data pages are readable and writable, code pages are readable and executable. But sometimes you need to change these dynamically. For example, a JIT compiler writes machine code into a buffer (needs WRITE), then makes it executable (needs EXEC).

Function Signature

#include <sys/mman.h>

int mprotect(void *addr, size_t len, int prot);

/* Returns: 0 on success, -1 on error (errno set) */
Parameter Type Description
addr void * Start of region — must be page-aligned
len size_t Length in bytes (rounded up to page boundary)
prot int New protection flags (OR’d together)

Protection Flags (prot)

Flag Value Meaning
PROT_NONE 0 No access at all
PROT_READ 1 Pages may be read
PROT_WRITE 2 Pages may be written
PROT_EXEC 4 Pages may be executed as code

These flags can be combined with bitwise OR: for example, PROT_READ | PROT_WRITE makes a region both readable and writable. If a process violates the protection (e.g., writes to a read-only page), the kernel delivers SIGSEGV (segmentation fault).

How the MMU Enforces Protection

Process attempts memory access (read/write/exec)
CPU MMU checks Page Table Entry (PTE) protection bits
Access Allowed
Operation proceeds normally
Access Denied
MMU raises hardware exception → Kernel sends SIGSEGV

Important Rules

⚠ addr must be page-aligned: On x86/ARM Linux, the page size is usually 4096 bytes. If addr is not aligned to a page boundary, mprotect() returns EINVAL. Always use sysconf(_SC_PAGESIZE) to get the runtime page size.
✅ len is rounded up: The len parameter is automatically rounded up to the nearest page boundary by the kernel. So if you pass 1 byte, the entire page containing that byte gets the new protection.
ℹ Region must be mapped: mprotect() can only change protection on memory that is already mapped (via mmap(), malloc(), or loaded segments). Calling it on unmapped memory returns ENOMEM.

Example 1: Make a Region Read-Only Then Write to It

This example shows what happens when you violate memory protection — the classic way to learn SIGSEGV:

#include <stdio.h>
#include <stdlib.h>
#include <sys/mman.h>
#include <string.h>
#include <unistd.h>

int main(void)
{
    long pagesize = sysconf(_SC_PAGESIZE);

    /* Allocate one page of memory using mmap */
    char *buf = mmap(NULL, pagesize,
                     PROT_READ | PROT_WRITE,
                     MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
    if (buf == MAP_FAILED) {
        perror("mmap");
        exit(EXIT_FAILURE);
    }

    /* Write some data while it's writable */
    strcpy(buf, "Hello, Virtual Memory!");
    printf("Before mprotect: %s\n", buf);

    /* Now make the region READ-ONLY */
    if (mprotect(buf, pagesize, PROT_READ) == -1) {
        perror("mprotect");
        exit(EXIT_FAILURE);
    }

    /* This read is fine */
    printf("After mprotect (read): %s\n", buf);

    /* This WRITE will cause SIGSEGV! */
    /* buf[0] = 'X';   <-- uncomment to see the crash */

    munmap(buf, pagesize);
    return 0;
}
Output:
Before mprotect: Hello, Virtual Memory!
After mprotect (read): Hello, Virtual Memory!

Example 2: JIT Compiler Pattern – Write Code, Then Execute It

JIT (Just-In-Time) compilers generate machine code at runtime. They need to write bytes first (WRITE permission), then switch the region to executable (EXEC permission). This is a classic W^X (Write XOR Execute) security pattern.

#include <stdio.h>
#include <stdlib.h>
#include <sys/mman.h>
#include <string.h>
#include <unistd.h>

int main(void)
{
    long pagesize = sysconf(_SC_PAGESIZE);

    /*
     * Step 1: Allocate page with WRITE permission to write machine code.
     * Note: We do NOT give EXEC yet (W^X principle).
     */
    unsigned char *code = mmap(NULL, pagesize,
                                PROT_READ | PROT_WRITE,
                                MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
    if (code == MAP_FAILED) {
        perror("mmap");
        exit(EXIT_FAILURE);
    }

    /*
     * Step 2: Write a simple x86-64 function that returns 42.
     *   mov eax, 42    (B8 2A 00 00 00)
     *   ret            (C3)
     */
    unsigned char machine_code[] = {
        0xB8, 0x2A, 0x00, 0x00, 0x00,  /* mov eax, 42 */
        0xC3                            /* ret         */
    };
    memcpy(code, machine_code, sizeof(machine_code));

    /*
     * Step 3: Remove WRITE, add EXEC.
     * This is the W^X switch — never have both W and X at the same time.
     */
    if (mprotect(code, pagesize, PROT_READ | PROT_EXEC) == -1) {
        perror("mprotect (EXEC)");
        exit(EXIT_FAILURE);
    }

    /* Step 4: Call the generated code as a function */
    int (*func)(void) = (int (*)(void)) code;
    int result = func();
    printf("JIT function returned: %d\n", result);  /* Prints: 42 */

    munmap(code, pagesize);
    return 0;
}
Output: JIT function returned: 42

Example 3: Catching Protection Violations with a Signal Handler

Instead of crashing, you can install a SIGSEGV handler to react gracefully when a protection violation occurs. This is used in advanced techniques like read barriers in garbage collectors.

#include <stdio.h>
#include <stdlib.h>
#include <signal.h>
#include <sys/mman.h>
#include <setjmp.h>
#include <unistd.h>

static sigjmp_buf jump_buffer;

/* Signal handler for SIGSEGV */
static void segv_handler(int signum)
{
    printf("Caught SIGSEGV (signal %d) – protection violation!\n", signum);
    siglongjmp(jump_buffer, 1);  /* Jump back to safe point */
}

int main(void)
{
    long pagesize = sysconf(_SC_PAGESIZE);

    /* Install SIGSEGV handler */
    struct sigaction sa;
    sa.sa_handler = segv_handler;
    sigemptyset(&sa.sa_mask);
    sa.sa_flags = SA_RESETHAND;  /* Reset to default after first signal */
    sigaction(SIGSEGV, &sa, NULL);

    /* Map a page */
    char *buf = mmap(NULL, pagesize,
                     PROT_READ | PROT_WRITE,
                     MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
    if (buf == MAP_FAILED) { perror("mmap"); exit(1); }

    /* Make it read-only */
    mprotect(buf, pagesize, PROT_READ);

    /* Set safe return point */
    if (sigsetjmp(jump_buffer, 1) == 0) {
        printf("Attempting write to read-only page...\n");
        buf[0] = 'X';  /* This will trigger SIGSEGV */
    } else {
        printf("Recovered from violation. Program continues.\n");
    }

    munmap(buf, pagesize);
    return 0;
}

Example 4: Guard Pages for Buffer Overflow Detection

Guard pages are a classic use of PROT_NONE. Place an inaccessible page just after a buffer. Any overflow will immediately fault instead of silently corrupting memory.

#include <stdio.h>
#include <stdlib.h>
#include <sys/mman.h>
#include <string.h>
#include <unistd.h>

int main(void)
{
    long pagesize = sysconf(_SC_PAGESIZE);

    /*
     * Allocate 2 pages:
     *   Page 0: actual buffer (PROT_READ | PROT_WRITE)
     *   Page 1: guard page    (PROT_NONE — no access allowed)
     */
    char *region = mmap(NULL, 2 * pagesize,
                        PROT_READ | PROT_WRITE,
                        MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
    if (region == MAP_FAILED) { perror("mmap"); exit(1); }

    char *buffer    = region;               /* Page 0 – usable buffer */
    char *guardpage = region + pagesize;    /* Page 1 – inaccessible  */

    /* Make the guard page completely inaccessible */
    if (mprotect(guardpage, pagesize, PROT_NONE) == -1) {
        perror("mprotect guard");
        exit(1);
    }

    printf("Buffer at: %p\n", buffer);
    printf("Guard  at: %p\n", guardpage);

    /* Normal write – fine */
    memset(buffer, 'A', 10);
    printf("Wrote 10 bytes to buffer: OK\n");

    /* Overflow past the buffer into the guard page → SIGSEGV */
    /* memset(buffer, 'A', pagesize + 1); */
    /* Uncomment above to see immediate crash instead of silent corruption */

    munmap(region, 2 * pagesize);
    return 0;
}

Common Errors

errno Cause
EINVAL addr is not page-aligned, or invalid prot flags
ENOMEM Address range is not mapped, or kernel ran out of memory
EACCES Trying to set PROT_WRITE on a file opened read-only via mmap

Interview Questions & Answers

Q1. What does mprotect() do and why would you use it?

mprotect() changes the protection (read/write/execute permissions) of a region of virtual memory at runtime. You would use it when you need to dynamically change memory access rights — for example, in a JIT compiler that writes machine code (needs WRITE) and then makes it executable (needs EXEC), or to implement guard pages that detect buffer overflows.

Q2. What happens if you violate memory protection?

The hardware MMU detects the violation during address translation and raises a hardware exception. The kernel catches this and sends SIGSEGV (signal 11) to the process. The default action for SIGSEGV is to terminate the process and optionally generate a core dump. A program can install a signal handler for SIGSEGV to recover, but this is complex and usually only done in specialized runtime systems.

Q3. Why must addr be page-aligned in mprotect()?

Because memory protection is enforced at the page granularity by the MMU hardware. The MMU stores one set of permission bits per page in the page table — it cannot protect individual bytes within a page differently. The system page size is typically 4096 bytes on x86/ARM. If addr is not page-aligned, mprotect() fails with EINVAL. Use sysconf(_SC_PAGESIZE) to find the runtime page size.

Q4. What is the W^X (Write XOR Execute) principle?

W^X is a security policy that says a memory region should never be both writable AND executable at the same time. If a page is writable, an attacker can inject code into it. If that same page is also executable, the injected code can run. By never allowing both simultaneously, you prevent code injection attacks. JIT compilers implement this by first writing code (PROT_WRITE, not EXEC), then using mprotect() to switch to (PROT_EXEC, not WRITE) before executing it.

Q5. What is a guard page and how is it implemented?

A guard page is a virtual memory page set to PROT_NONE (no access) placed next to a buffer. If the program overflows the buffer and accesses the guard page, it immediately gets a SIGSEGV instead of silently corrupting adjacent memory. This is widely used in stack overflow detection (the kernel maps a PROT_NONE page at the bottom of each thread’s stack), and can be added manually with mprotect().

Q6. Can mprotect() change the protection of the stack or text segment?

Technically yes — mprotect() works on any mapped region. However, changing the text (code) segment to writable and the stack to non-executable is dangerous and typically blocked or restricted by security policies (SELinux, seccomp). Making the stack executable to run shellcode is a classic exploit technique, which is why modern OSes enforce NX (No-Execute) bits on stack pages by default.

Q7. What is the difference between PROT_NONE and unmapping memory?

PROT_NONE keeps the virtual address range reserved and mapped — any access causes SIGSEGV, but the addresses are still “occupied” in the process’s address space. munmap() completely releases the mapping, returning the address range to the kernel. After munmap(), the addresses may be reused by future mmap() calls. Guard pages use PROT_NONE (not unmap) so the address space remains reserved.

Topic Summary

  • mprotect(addr, len, prot) changes page permissions at runtime.
  • prot is a bitwise OR of PROT_NONE, PROT_READ, PROT_WRITE, PROT_EXEC.
  • addr must be page-aligned; len is rounded up to page size.
  • Violating protection → kernel sends SIGSEGV to the process.
  • Used in JIT compilers (W^X), guard pages, memory-mapped file protection.
  • Returns 0 on success, -1 on error (check errno).

Leave a Reply

Your email address will not be published. Required fields are marked *