Free Linux Kernel Development Course — Understanding Kernel Memory Layout on ARM and x86-64
What You Will Learn in This Free Linux Kernel Development Course
In this tutorial from our free Linux kernel development course, you will learn exactly how the Linux kernel organises its own memory — what regions exist inside the kernel segment, why those regions exist, and how they differ across 32-bit and 64-bit architectures. This knowledge is foundational for anyone writing Linux device drivers or kernel modules.
- What the kernel virtual address space (kernel VAS) is and why it matters
- The VM split concept: how user space and kernel space share the address space on 32-bit systems
- Key kernel segment regions: lowmem, vmalloc, modules, fixmap, highmem
- How the layout differs between 32-bit ARM and 64-bit x86 (and ARM64)
- How to inspect your own kernel’s memory layout using
/procand kernel modules - Modern Linux kernel (v6.x) memory layout differences
Before reading this tutorial, you should be comfortable with the following from our free embedded systems course and free Linux device drivers course series:
What Is the Linux Kernel Virtual Address Space?
Every process running on Linux sees two regions in its virtual address space: the user space and the kernel space. The user space is where your program’s code, data, heap, and stack live. The kernel space is where the Linux kernel itself resides — its code, data structures, driver memory, and mapped hardware registers.
The kernel virtual address space (kernel VAS), sometimes called the kernel segment, is the portion of the address space reserved exclusively for the kernel. On a 32-bit system, user space and kernel space together must fit within 4 GB. On a 64-bit system this constraint effectively disappears because the address space is astronomically large (128 TB per side on x86-64).
Understanding the kernel VAS is critical if you are writing Linux device drivers or kernel modules as part of your free Linux kernel development journey, because any memory your driver allocates or maps comes from inside this kernel segment.
3:1 GB split (most common)
No practical split constraint
The VM Split on 32-bit Systems — Why It Matters for Linux Kernel Development
On a 32-bit processor, the entire virtual address space is only 4 GB. The Linux kernel must share this limited space with user processes. This sharing is controlled by a compile-time constant called PAGE_OFFSET (also referred to as CONFIG_PAGE_OFFSET). Everything at or above PAGE_OFFSET is kernel space; everything below is user space.
The classic split on 32-bit x86 and ARM systems is 3 GB user / 1 GB kernel, where PAGE_OFFSET = 0xC000 0000. Some embedded boards (Raspberry Pi 3B+ running 32-bit Linux, for example) use a 2 GB / 2 GB split because they need more kernel-addressable RAM, setting PAGE_OFFSET = 0x8000 0000. The split you use affects how much physical RAM the kernel can directly address without tricks like highmem.
| Config | PAGE_OFFSET | User Space | Kernel Space | Typical Use Case |
|---|---|---|---|---|
| 3:1 | 0xC000 0000 |
3 GB | 1 GB | Standard desktop Linux (x86) |
| 2:2 | 0x8000 0000 |
2 GB | 2 GB | Raspberry Pi 3B+ 32-bit OS |
| 1:3 | 0x4000 0000 |
1 GB | 3 GB | Server workloads with large kernel data |
| N/A | 0xFFFF 8000 0000 0000 |
128 TB | 128 TB | 64-bit (x86-64) — no split problem |
On 64-bit systems (which is the default today), this VM split problem simply does not exist. Both kernel and user space get a full 128 TB each on standard x86-64 kernels. This is one of the biggest reasons the industry moved away from 32-bit Linux for production embedded systems wherever possible.
Inside the Kernel Segment — Key Memory Regions
The kernel segment is not one flat blob of memory. It is divided into distinct regions, each serving a different purpose. Knowing these regions is essential for anyone studying our free Linux device drivers course, because different kernel APIs for memory allocation come from different regions.
Exception/interrupt handlers — 4 KB, fixed at top
Fixed virtual address mappings (boot-time, kmap, etc.) — ~3 MB
DMA bounce buffers, reserved regions (platform-specific)
vmalloc(), vmap(), ioremap() — kernel virtual addresses (KVAs). Size varies (e.g., 1088 MB in a 2:2 split)
Separates lowmem from vmalloc area — catches overflows
Physical RAM directly mapped to kernel logical addresses. kmalloc(), kzalloc() come from here. The most precious region on 32-bit.
Just below PAGE_OFFSET — where insmod loads your .ko files. Typically 16 MB on ARM-32.
Everything below TASK_SIZE is user virtual address space
Deep Dive — Each Kernel Segment Region Explained
1. Lowmem — The Direct-Mapped RAM Region
The lowmem region is the most important part of the kernel segment. It is a direct, linear mapping of physical RAM into the kernel’s virtual address space. Every physical page of RAM in lowmem has a corresponding kernel virtual address (called a kernel logical address) that is simply PAGE_OFFSET + physical_address.
Because the mapping is fixed and simple, the kernel can instantly convert between a physical address and a kernel logical address. This is why kmalloc() and kzalloc() — the most common kernel memory allocators you will use in your Linux device drivers — allocate from this region. The memory they give you is physically contiguous, which is required for DMA on many platforms.
/* Converting between kernel logical address and physical address */ /* These macros work ONLY in the lowmem region */ virt_to_phys(kva) /* kernel logical address → physical address */ phys_to_virt(pa) /* physical address → kernel logical address */ /* Example */ void *ptr = kmalloc(64, GFP_KERNEL); phys_addr_t pa = virt_to_phys(ptr); /* get physical address of your allocation */
On a 32-bit system with a 3:1 split (PAGE_OFFSET = 0xC000 0000), only 1 GB of kernel space exists. If you have 4 GB of physical RAM, the kernel can only directly map 1 GB of it in lowmem. The rest is handled via highmem (see below). This is why 32-bit Linux systems were limited for servers. On 64-bit, all RAM is directly mapped and highmem does not exist.
2. Highmem Region — Only on 32-bit
On 32-bit systems where physical RAM exceeds the lowmem size (e.g., more than 896 MB of RAM on a 3:1 split kernel), the excess physical pages cannot be permanently mapped into the kernel’s limited virtual address space. This overflow zone is called highmem.
To access highmem pages, the kernel uses temporary mappings (via kmap() and kmap_atomic()) that borrow fixmap or pkmap slots. This is slow and complex. Modern 64-bit kernels have no highmem at all — all physical RAM is directly mapped.
3. Vmalloc Region — Virtually Contiguous, Not Physically Contiguous
The vmalloc region provides virtually contiguous memory that may be scattered across many non-contiguous physical pages. When the kernel (or a driver) needs a large allocation that does not need to be physically contiguous, it uses vmalloc() to draw from this region.
This region is also used by ioremap() — one of the most commonly used functions in Linux device driver development. When you write a driver for a memory-mapped peripheral (like a UART, SPI controller, or custom FPGA), you call ioremap() to map the hardware’s physical register base address into the kernel’s vmalloc region so your driver code can access it.
/* ioremap — mapping hardware registers in a device driver */ /* This is from the vmalloc region, NOT lowmem */ #define MY_UART_BASE 0x48020000UL /* physical base of UART registers */ #define MY_UART_SIZE 0x1000 /* 4 KB register block */ void __iomem *base; base = ioremap(MY_UART_BASE, MY_UART_SIZE); if (!base) { pr_err("ioremap failed\n"); return -ENOMEM; } /* Read/write registers using readl/writel, NOT direct pointer deref */ u32 val = readl(base + 0x00); writel(0x01, base + 0x04); iounmap(base); /* always unmap in driver remove/exit */
4. Kernel Modules Region
When you run insmod mydriver.ko, the kernel loads your module’s code and data into the kernel modules region. On ARM-32, this region sits just below PAGE_OFFSET and is typically 16 MB in size. On x86-64, modules have their own dedicated region near the kernel text.
The reason modules get their own region (rather than just loading into vmalloc) is that on 32-bit ARM, the branch instruction’s reach is limited — a module too far from the core kernel text cannot branch to kernel functions. Keeping modules close to the kernel text, just below PAGE_OFFSET, ensures all branch distances stay within range.
5. Fixmap Region
The fixmap region holds a set of fixed virtual addresses — virtual addresses that the kernel reserves at compile time for special purposes. These include boot-time console output mappings, kmap slots for highmem page access, and early hardware mappings needed before the full memory subsystem is initialised. As a driver developer you will rarely interact with the fixmap region directly.
Kernel VAS on Modern 64-bit Linux (Kernel v6.x)
Today almost all production Linux systems run 64-bit kernels. The memory layout is fundamentally more generous and simpler. Here is how the kernel segment looks on a modern x86-64 system running Linux 6.x:
All physical RAM directly mapped here (replaces lowmem concept; no highmem!)
Process memory (mmap, heap, stack, text, libs)
Inspecting the Kernel VAS on Your Own Linux System
You do not need to write a kernel module to get a first look at your kernel’s memory layout. The Linux kernel exposes some information through /proc on most configurations.
Using /proc/iomem
/proc/iomem shows the physical address ranges that the system has reserved for RAM, device registers, BIOS, and so on. This is not the virtual layout, but it tells you how much physical RAM and which I/O regions exist.
# View physical memory and IO regions $ cat /proc/iomem # Sample output on x86-64 (simplified): 00000000-00000fff : Reserved 00001000-0009fbff : System RAM 00100000-bfffffff : System RAM 100000000-23fffffff : System RAM ← RAM above 4 GB fee00000-fee00fff : Local APIC
Using /proc/vmallocinfo
/proc/vmallocinfo shows all current allocations in the vmalloc region, including ioremap mappings made by drivers. This is extremely useful when debugging a driver — you can verify that your ioremap() call actually created a mapping and see its virtual address range.
# Show all vmalloc/ioremap allocations $ sudo cat /proc/vmallocinfo # Each line: virtual_start-virtual_end size caller flags 0xffffa12300000000-0xffffa12300002000 8192 ioremap+0x... phys=0xfe000000 ioremap
Using /proc/buddyinfo and /proc/meminfo
/proc/meminfo gives you a high-level picture of overall memory usage including MemTotal, MemFree, Cached, and importantly VmallocTotal, VmallocUsed, and VmallocChunk.
$ grep -i vmalloc /proc/meminfo
VmallocTotal: 34359738367 kB ← Total vmalloc region size (huge on 64-bit)
VmallocUsed: 45312 kB ← Currently used by drivers/kernel
VmallocChunk: 0 kB ← Largest free contiguous chunk
Writing a Kernel Module to Display Memory Layout — Free Linux Kernel Development Exercise
The most educational exercise in our free Linux kernel development course is writing a small LKM (Loadable Kernel Module) that prints the kernel’s own segment boundaries. Below is a complete, working example that compiles on modern kernels (tested on 5.15 LTS and 6.x). This uses only standard kernel headers — no dependency on any book-specific code.
// vas_info.c — Print kernel virtual address space boundaries // Works on Linux kernel 5.15+ (LTS) and 6.x // Free Linux Kernel Development Course — EmbeddedPathashala #include <linux/module.h> #include <linux/kernel.h> #include <linux/init.h> #include <linux/mm.h> #include <linux/vmalloc.h> #include <asm/pgtable.h> MODULE_LICENSE("GPL"); MODULE_AUTHOR("EmbeddedPathashala"); MODULE_DESCRIPTION("Print Kernel VAS boundaries"); MODULE_VERSION("1.0"); static int __init vas_info_init(void) { pr_info("=== Kernel VAS Info (EmbeddedPathashala) ===\n"); /* PAGE_OFFSET: start of kernel direct-mapped RAM (lowmem start) */ pr_info("PAGE_OFFSET (lowmem start) : 0x%016lx\n", PAGE_OFFSET); /* TASK_SIZE: top of user virtual address space */ pr_info("TASK_SIZE (user VAS top) : 0x%016lx\n", TASK_SIZE); #ifdef CONFIG_X86_64 pr_info("VMALLOC_START : 0x%016lx\n", VMALLOC_START); pr_info("VMALLOC_END : 0x%016lx\n", VMALLOC_END); #endif pr_info("MODULES_VADDR : 0x%016lx\n", MODULES_VADDR); pr_info("MODULES_END : 0x%016lx\n", MODULES_END); pr_info("Kernel text start : 0x%016lx\n", (unsigned long)_text); pr_info("Kernel text end : 0x%016lx\n", (unsigned long)_etext); pr_info("Kernel data end (BSS end) : 0x%016lx\n", (unsigned long)_end); pr_info("=== End of VAS Info ===\n"); return 0; } static void __exit vas_info_exit(void) { pr_info("vas_info: module removed\n"); } module_init(vas_info_init); module_exit(vas_info_exit);
Makefile for the Module
# Makefile — for vas_info.c
obj-m += vas_info.o
KDIR := /lib/modules/$(shell uname -r)/build
all:
make -C $(KDIR) M=$(PWD) modules
clean:
make -C $(KDIR) M=$(PWD) clean
Build and Test
# Install kernel headers first $ sudo apt install linux-headers-$(uname -r) # Build the module $ make # Insert the module $ sudo insmod vas_info.ko # Read the output $ dmesg | tail -20 # Remove the module $ sudo rmmod vas_info
Common Mistakes When Working with Kernel Memory
The virt_to_phys() macro only works correctly on lowmem (kernel logical) addresses. If you call it on a vmalloc address, you get garbage. Always check your allocation source before converting.
void *vp = vmalloc(4096); phys_addr_t pa = virt_to_phys(vp); /* WRONG — undefined result */ void *kp = kmalloc(4096, GFP_KERNEL); phys_addr_t pa = virt_to_phys(kp); /* CORRECT — kmalloc gives lowmem addr */
After ioremap(), you must use readl()/writel() (or readb()/writeb() etc.) to access the mapped registers. Directly dereferencing the pointer with *ptr = val may work on some architectures but is wrong and non-portable — it bypasses memory barriers that prevent compiler/CPU reordering of register accesses.
Each ioremap() consumes a virtual address range in the vmalloc region. If you forget iounmap() in your driver’s remove() or module exit function, you leak virtual address space. On a system where the driver is repeatedly loaded and unloaded, you can exhaust the vmalloc region.
Key Takeaways — Linux Kernel Virtual Address Space
- The kernel segment (kernel VAS) is the portion of virtual address space reserved for the Linux kernel, above
PAGE_OFFSETon 32-bit systems. - On 32-bit systems the VM split (e.g., 3:1 or 2:2 GB) determines how much of the 4 GB address space each side gets. On 64-bit, there is no practical constraint.
- The lowmem region is a direct linear mapping of physical RAM —
kmalloc()allocates from here. It provides physically contiguous memory needed for DMA. - The vmalloc region provides virtually contiguous (but physically scattered) memory —
vmalloc()andioremap()use this region. - The kernel modules region is where
insmodplaces your.kofiles. On ARM-32 it is just belowPAGE_OFFSETto keep branch distances manageable. - Highmem exists only on 32-bit systems where physical RAM exceeds the lowmem capacity. 64-bit kernels map all RAM directly.
- On modern Linux 6.x x86-64, the direct physical memory map can hold up to 64 TB of RAM without any highmem complexity.
- Use
/proc/vmallocinfoand/proc/meminfoto inspect kernel memory usage on a running system.
Frequently Asked Questions — Linux Kernel Memory Layout
PAGE_OFFSET is the virtual address that marks the boundary between user space and kernel space on 32-bit Linux systems. Every virtual address at or above PAGE_OFFSET belongs to the kernel. Its value is set at kernel build time (e.g., 0xC000 0000 for a 3:1 split or 0x8000 0000 for a 2:2 split). On 64-bit kernels, PAGE_OFFSET marks the start of the direct physical memory map.
kmalloc() allocates physically contiguous memory from the lowmem region. The allocated buffer has a single contiguous physical address range, making it suitable for DMA. vmalloc() allocates virtually contiguous memory from the vmalloc region — the physical pages may be scattered, so it is not directly usable for DMA but can handle larger allocations without requiring physically contiguous free pages.
Highmem is a 32-bit Linux mechanism to access physical RAM that exceeds the lowmem capacity. Since the kernel’s direct mapping is limited by the kernel VA space size, extra RAM cannot be permanently mapped. 64-bit kernels do not have this problem and do not use highmem at all. If you are writing drivers for 64-bit targets (which most modern embedded SoCs are), you do not need to worry about highmem.
Hardware register blocks have physical addresses assigned by the SoC design, not by the OS. These physical addresses are not part of RAM, so they cannot appear in the lowmem (direct RAM mapping). The vmalloc region is the right place to put arbitrary physical-to-virtual mappings, because it provides available virtual address space with flexible page table entries — exactly what ioremap() needs to create a mapping to hardware registers.
After inserting your module with insmod, check /proc/modules or run sudo cat /sys/module/<module_name>/sections/.text to see the virtual address of your module’s text section. On modern kernels with KASLR, this address is randomised at load time.
Yes. The kernel segment is shared across all processes — every process’s page tables map the same kernel virtual addresses to the same kernel physical pages. This is what allows a system call to seamlessly switch from a process’s user-space page tables into the kernel without needing to switch the address space entirely. (With KPTI enabled on x86 to mitigate Meltdown, a minimal kernel mapping is used while in user mode, but the full kernel mapping is restored on entry to the kernel.)
KASLR (Kernel Address Space Layout Randomisation) randomises the virtual addresses at which the kernel text, data, vmalloc, and modules regions are loaded at boot time. The overall structure and relative ordering of regions remains the same, but absolute addresses are different each boot. This makes it harder for an attacker who has a memory-read vulnerability to locate kernel functions. KASLR is enabled by default on most distributions since Linux 3.14.
No. The kernel segment is mapped with page table entries that set the supervisor-only protection bit (U/S bit on x86 and equivalent on ARM). Any user-space attempt to access a kernel virtual address triggers a page fault and results in a SIGSEGV or SIGBUS. This hardware-enforced boundary is the foundation of OS security.
Conclusion
Understanding the Linux kernel virtual address space is not just academic — it directly shapes every memory-related decision you make when writing Linux device drivers or kernel modules. Knowing that kmalloc() gives you physically contiguous lowmem (great for DMA), while vmalloc() and ioremap() live in the vmalloc region (great for large or hardware-register mappings), is the difference between a correct driver and a subtly broken one.
In our next tutorial from this free Linux kernel development course, we will go deeper into the user-space side of the VAS — Virtual Memory Areas (VMAs), how the kernel tracks them in the mm_struct, and how to walk a process’s memory map from inside a kernel module. If you found this tutorial helpful, share it with others who are learning Linux kernel programming or taking our free Linux device drivers course on EmbeddedPathashala.
Continue Your Free Linux Kernel Development Journey
EmbeddedPathashala — Free Linux Kernel, Device Drivers & Embedded Systems Courses
← Previous Lecture Next Lecture →