Why malloc() Doesn’t Hand You RAM

When a process calls malloc(1024 * 1024) and gets back a non-NULL pointer, it is tempting to assume one megabyte of physical memory is now reserved for it. That assumption is wrong, and understanding why is central to this free embedded systems course. Linux uses a strategy called lazy allocation (also called demand paging): the kernel reserves a range of virtual addresses and creates page table entries for them, but those entries are marked “not present.” No physical page frame is assigned yet. The kernel is deliberately deferring the expensive work — finding a free physical page and wiring it into the page table — until the process actually touches that memory.

This matters because reserving virtual address space is cheap (it is just bookkeeping in the kernel’s VMA — virtual memory area — structures), while assigning physical RAM is comparatively expensive. A process that allocates a large buffer “just in case” but only ever uses a fraction of it costs the system almost nothing extra for the unused portion, because that portion never receives a physical page at all.

Minor Faults vs Major Faults

A page fault is simply the CPU trapping into the kernel because it hit a page table entry that isn’t marked present. There are two kinds, and the difference matters a great deal for performance on an embedded target:

A newly malloc()‘d and zero-filled anonymous page is the classic minor-fault case. A memory-mapped file whose contents have never been read into the page cache is the classic major-fault case, since the kernel must issue a block read before the access can complete.

Hands-On: Counting Your Own Page Faults

Rather than reusing a textbook example, this lecture uses mmap() directly to request anonymous memory — the same mechanism malloc() uses internally for large requests — so you can see the page-fault behaviour without a libc allocator in the way. The demo below, ep_faultwatch.c, requests a 2 MiB anonymous mapping, then touches it in two separate passes while printing the fault counters between each step.

// ep_faultwatch.c — original demo, EmbeddedPathashala
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/mman.h>
#include <sys/resource.h>

#define REGION_SIZE (2 * 1024 * 1024)   /* 2 MiB */

static void show_faults(const char *label)
{
    struct rusage ru;

    if (getrusage(RUSAGE_SELF, &ru) == -1) {
        perror("getrusage");
        return;
    }
    printf("%-22s minor=%-6ld major=%-6ld\n",
           label, ru.ru_minflt, ru.ru_majflt);
}

int main(void)
{
    unsigned char *region;

    show_faults("baseline");

    region = mmap(NULL, REGION_SIZE, PROT_READ | PROT_WRITE,
                  MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
    if (region == MAP_FAILED) {
        perror("mmap");
        return 1;
    }
    show_faults("after mmap()");

    /* First touch: every page here is genuinely new to the process */
    memset(region, 0xAB, REGION_SIZE);
    show_faults("after first touch");

    /* Second touch: pages are already resident, no new faults expected */
    memset(region, 0xCD, REGION_SIZE);
    show_faults("after second touch");

    munmap(region, REGION_SIZE);
    return 0;
}

Build and run it on any standard Linux host or embedded target with a libc toolchain:

$ gcc -Wall -o ep_faultwatch ep_faultwatch.c
$ ./ep_faultwatch

Expected output (exact minor-fault counts vary slightly by libc and architecture, but the shape is consistent):

baseline               minor=178    major=0
after mmap()           minor=178    major=0
after first touch      minor=690    major=0
after second touch     minor=690    major=0

Reading the Fault Counters

Notice that the mmap() call itself adds zero new faults — it only reserves virtual address space and creates not-present page table entries. All the real work happens during “after first touch,” where memset() writes to every byte of the 2 MiB region for the first time: 2 MiB is 512 pages on a 4 KiB-page system, and the jump in the minor counter (690 – 178 = 512) matches exactly. The second memset() pass adds no new faults at all, because every page is already resident and mapped from the first pass — this is the payoff of demand paging in action.

The major-fault counter stays at zero throughout, which is expected: MAP_ANONYMOUS memory has no backing file, so there is nothing for the kernel to read from storage. You would only see major faults climb if you mapped a file with mmap() and read from parts of it that hadn’t yet been pulled into the page cache.

Why Embedded Developers Should Care

On a desktop, a handful of page faults are invisible. On an embedded target with a slow eMMC or NAND flash backing store, a burst of major faults during startup — for example, when a large executable or shared library is first mapped and executed — can visibly stall boot time or cause a UI to stutter on first launch. Understanding lazy allocation also explains why free and top can show surprisingly little RSS growth right after a large malloc(): the memory has been reserved, not yet paid for in physical pages.

Common Mistakes and Troubleshooting

• Assuming a successful malloc() proves physical memory is available — the real test only happens when the pages are touched, which is why systems can still hit out-of-memory conditions well after allocation succeeds.
• Benchmarking allocation speed by timing malloc() alone and ignoring the first-touch cost, which produces misleadingly optimistic numbers.
• Confusing major faults with swapping — on many embedded targets swap is disabled entirely, so major faults there are almost always file-backed mappings, not swapped-out anonymous pages.

Best Practices

• Pre-touch latency-critical buffers during application startup (a deliberate first-touch pass) rather than letting the fault happen on the hot path.
• Prefer mlock() or mlockall() for real-time paths where a page fault’s latency spike is unacceptable, keeping in mind the memory-locking limits set by ulimit.
• When profiling on constrained hardware, watch ru_majflt specifically — a rising major-fault count during steady-state operation is a strong signal of either swap thrashing or a poorly cached file-backed mapping.

Interview Questions

• What is the difference between a minor and a major page fault, and give one real example of each.
• Why doesn’t a successful malloc() call guarantee physical memory has been allocated?
• Which system call lets a user-space program read its own minor and major fault counts?