🚀 Learn BLE on Linux (BlueZ) – Programming Part 1
Hands-on Bluetooth Low Energy course. Special price: ₹399
Enroll Now on Udemy →Intermediate
Kernel Memory
Kernel Internals
Welcome to this lecture in our free Linux kernel development course. In this tutorial we are going to understand one of the most important concepts in Linux memory management: the Virtual Address Space (VAS) and the VM split. If you are preparing for embedded Linux roles or working on Linux device drivers, understanding how the kernel divides virtual memory between user space and kernel space is absolutely essential.
In this lesson we focus specifically on how the VM split works on 64-bit x86_64 Linux systems running modern kernels (Linux 5.x and 6.x). We also look at how virtual addresses are structured as bitmasks, what UVA and KVA mean, and how address translation actually works under the hood.
What You Will Learn
- What Virtual Address Space (VAS) means and why every process gets its own
- The concept of VM split: dividing VAS between user space and kernel space
- Why 64-bit Linux only uses 48 bits (or 57 bits with 5-level paging) for addressing
- How a virtual address is a bitmask broken into PGD, PUD, PMD, PTE, and offset fields
- The difference between a User Virtual Address (UVA) and a Kernel Virtual Address (KVA)
- How to identify whether an address belongs to user space or kernel space just by looking at it
- What canonical addresses are and why the 64-bit VAS has a “hole” in the middle
- How the MMU uses page tables to translate virtual addresses to physical RAM addresses
Prerequisites
- Basic C programming (pointers, memory addresses)
- What a Linux process is and what a kernel module is
- Binary and hexadecimal number systems
- The general idea that modern CPUs use paging for memory management
What Is the Virtual Address Space?
Every process running on Linux lives inside its own Virtual Address Space (VAS). Think of the VAS as a private map of memory addresses that belongs exclusively to that process. Two processes can have variables sitting at the same virtual address, say 0x7fff1234, yet those two variables are completely independent. The kernel makes sure each process sees its own private copy of that address map.
The VAS is divided into two distinct halves:
- User Space (lower half): Where the process’s own code, stack, heap, and libraries live.
- Kernel Space (upper half): Where the Linux kernel, its data structures, device driver memory, and kernel stacks reside.
This split is what kernel developers call the VM split. The dividing line between user space and kernel space is one of the most fundamental design choices in any operating system. In this free Linux kernel development course we will look at exactly where that line sits on modern 64-bit systems.
Kernel code, data, drivers, page tables
Process code, stack, heap, libraries
Every Linux process sees this same layout. The kernel portion is shared; the user portion is private per process.
Why 64 Bits Are Not Fully Used on x86_64
Here is a surprising fact that confuses many beginners. Even though a 64-bit processor has 64-bit registers, Linux on x86_64 does not use all 64 bits for virtual addresses. Let us understand why.
If we were to use all 64 bits for addressing, the total virtual address space would be 264 bytes. Do the math and that works out to approximately 16 Exabytes (EB). That is 16,384 Petabytes of virtual memory. No computer today, or likely for decades to come, will ever need that much. Allocating hardware resources in the CPU (TLB entries, page table walker logic) to support the full 64-bit range would be enormously wasteful.
So the x86_64 architecture (and also AArch64/ARM64) made a practical engineering decision: use only the lower 48 bits of a 64-bit address for actual addressing, treating this as the Least Significant Bits (LSB) portion.
With 48-bit addressing the total VAS is 248 bytes, which equals 256 Terabytes. That is more than enough for any realistic workload today. The remaining upper 16 bits (bits 63 down to 48) are not used for addressing — they are reserved and treated as a sign extension.
Starting from Linux kernel 4.14 (and widely used from kernel 5.x), Intel and AMD CPUs support an optional feature called 5-level paging. When enabled, the virtual address space expands to use 57 bits instead of 48, giving a VAS of 257 bytes = 128 PB per process. This is enabled via the
CONFIG_X86_5LEVEL kernel config option. Most general purpose systems today still run 4-level paging (48-bit addressing) as 128 TB per process is more than sufficient. This tutorial focuses on the 4-level paging / 48-bit model, which remains the most common and is the foundation you need to understand 5-level paging later.
| Bits 63–48 (16 bits) |
Bits 47–39 (9 bits) |
Bits 38–30 (9 bits) |
Bits 29–21 (9 bits) |
Bits 20–12 (9 bits) |
Bits 11–0 (12 bits) |
|---|---|---|---|---|---|
| Sign Extension Unused (must be copies of bit 47) |
PGD Index Page Global Directory |
PUD Index Page Upper Directory |
PMD Index Page Middle Directory |
PTE Index Page Table Entry |
Page Offset Offset within 4 KB page |
| Kernel: all 1s User: all 0s |
Index into L4 page table (512 entries) |
Index into L3 page table (512 entries) |
Index into L2 page table (512 entries) |
Index into L1 page table (512 entries) |
Byte offset within the page |
A 64-bit virtual address on x86_64 uses 4 groups of 9 bits as page table indices plus 12 bits as the page offset. The top 16 bits are a sign extension and must match bit 47.
Canonical Addresses — The “Hole” in 64-bit VAS
Now here is something very important. On x86_64, the top 16 bits of a 64-bit address (bits 63 to 48) are not free to take any value. The CPU hardware enforces a rule: bits 63 to 48 must all be identical copies of bit 47. This is called sign extension. Any address that follows this rule is called a canonical address. An address that violates this rule is a non-canonical address, and the CPU will raise a General Protection Fault (#GP) if you try to access it.
This rule splits the 64-bit address space into three zones:
0xffff8000 00000000 → 0xffffffff ffffffff
128 TB — Bits 63:48 = all 1s
0x0000800000000000 → 0xffff7fffffffffff
~16 million TB unusable — accessing causes #GP fault
0x0000000000000000 → 0x00007fffffffffff
128 TB — Bits 63:48 = all 0s
On x86_64 with 4-level paging, the usable VAS is 256 TB total: 128 TB for user space and 128 TB for kernel space, separated by a huge non-canonical hole.
This design has a practical benefit: by simply examining bits 63 to 48 of any address, you can tell immediately whether that address belongs to user space or kernel space:
| Address Zone | Bits 63–48 Value | Example Address Prefix | What It Is |
|---|---|---|---|
| User VAS | All zeros (0x0000) | 0x0000xxxx xxxxxxxx | User Virtual Address (UVA) |
| Non-Canonical Hole | Mixed (0x0001 to 0xfffe) | — | Invalid — causes CPU fault |
| Kernel VAS | All ones (0xffff) | 0xffffxxxx xxxxxxxx | Kernel Virtual Address (KVA) |
User Virtual Addresses (UVA) vs Kernel Virtual Addresses (KVA)
This distinction shows up constantly in Linux kernel and device driver development, so let us make it crystal clear.
Consider a simple C variable in a user space program:
/* Running as a normal user space process */
int sensor_reading = 42;
printf("Address: %p\n", (void *)&sensor_reading);
/* Output might be: 0x00007ffe4b3c2d10 */
That printed address is a User Virtual Address (UVA). Notice the leading 0x0000 bytes — confirming it is in the user VAS. User space code can only read and write addresses within its own user VAS region.
Now consider the same variable declaration inside a Linux kernel module:
/* Inside a kernel module */
#include <linux/kernel.h>
#include <linux/module.h>
static int sensor_reading = 42;
static int __init my_driver_init(void)
{
/* printk is the kernel's equivalent of printf */
printk(KERN_INFO "KVA of sensor_reading: %px\n", &sensor_reading);
/* Output might be: ffffffffc08a3010 */
return 0;
}
That address is a Kernel Virtual Address (KVA). Notice the leading 0xffff bytes. Kernel code lives in the kernel VAS (upper half) and can access both the user VAS (with special accessor functions like copy_from_user()) and the kernel VAS directly.
A kernel module must never directly dereference a User Virtual Address (UVA) pointer received from user space. Always use the kernel-provided safe accessor APIs:
copy_from_user(), copy_to_user(), get_user(), and put_user(). Directly dereferencing a UVA from kernel context can cause kernel panics, security vulnerabilities, and subtle data corruption bugs. This is one of the first things checked during Linux kernel code reviews.
A Virtual Address Is a Bitmask, Not a Counter
One of the most important conceptual shifts when learning Linux kernel memory management in this free Linux kernel development course is understanding that a virtual address is not simply a sequential number counting bytes from zero. It is a bitmask — a structured bit field where different groups of bits serve different purposes.
As shown in Figure 2, on x86_64 with 4 KB pages, a 64-bit virtual address is broken down as follows:
| Field Name | Bit Range | Width | Purpose | Max Index |
|---|---|---|---|---|
| Sign Extension | 63 – 48 | 16 bits | Must mirror bit 47 (canonical rule) | N/A |
| PGD Index | 47 – 39 | 9 bits | Index into Page Global Directory (L4 table) | 511 |
| PUD Index | 38 – 30 | 9 bits | Index into Page Upper Directory (L3 table) | 511 |
| PMD Index | 29 – 21 | 9 bits | Index into Page Middle Directory (L2 table) | 511 |
| PTE Index | 20 – 12 | 9 bits | Index into Page Table Entry (L1 table) | 511 |
| Page Offset | 11 – 0 | 12 bits | Byte offset within the physical 4 KB page | 4095 |
Why 9 bits for each table index? Because 29 = 512. Each page table level holds exactly 512 entries (each entry is 8 bytes on a 64-bit system, so one page table fits neatly in a single 4 KB page). This elegant design means page tables themselves are always page-aligned and page-sized.
Why 12 bits for the page offset? Because 212 = 4096, exactly matching the standard 4 KB page size used by Linux on x86_64.
You can verify this breakdown yourself on any running Linux system. Run
cat /proc/self/maps in a terminal. You will see lines like 7f8a2c000000-7f8a2c021000. These are the virtual address ranges of your shell process. Notice how all user-space addresses start with 0x00007f... — the leading 0x0000 tells you immediately these are User Virtual Addresses in the lower canonical half.
The MMU: Who Actually Translates Virtual to Physical?
The Linux kernel maintains a set of page tables for every process. These are data structures stored in kernel memory that map each virtual page to a physical page frame in RAM. But maintaining the tables is only half the story — someone has to actually perform the translation at runtime, for every single memory access made by running code.
That job belongs to the Memory Management Unit (MMU), which is a dedicated hardware unit built directly into the CPU silicon. Here is how the three players divide the work:
The MMU’s page table walk for a 4-level paging system on x86_64 works like this:
- The CPU register
CR3holds the physical address of the current process’s PGD (L4 table). - The MMU extracts bits 47–39 from the virtual address and uses them as an index into the PGD to find the PUD base address.
- It then extracts bits 38–30 and indexes into the PUD to find the PMD base address.
- It extracts bits 29–21 and indexes into the PMD to find the PTE base address.
- It extracts bits 20–12 and indexes into the PTE to find the physical page frame number (PFN).
- Finally, it adds bits 11–0 (the page offset) to the physical page base address, producing the final physical RAM address.
PGD phys addr
[bits 47:39]
[bits 38:30]
[bits 29:21]
[bits 20:12]
+ offset[11:0]
The MMU hardware walks four levels of page tables using the index fields in the virtual address, then adds the page offset to get the final physical RAM address.
Walking four levels of page tables for every single memory access would be extremely slow. The CPU includes a hardware cache called the Translation Lookaside Buffer (TLB) that stores recently used virtual-to-physical address translations. On most accesses, the TLB gives the answer instantly without a full page table walk. Context switches (switching between processes) typically require flushing TLB entries, which is one reason context switching has a non-zero cost. Understanding TLB behavior is important when writing high-performance Linux kernel code.
Exploring the VM Split in the Linux Kernel Source
Modern Linux kernel source code (6.x) defines the VAS layout through a set of macros and constants. Here are some of the most important ones you will encounter when reading kernel source or writing device drivers:
/* From arch/x86/include/asm/page_64_types.h (Linux 6.x) */
/* Task size: the boundary between user VAS and kernel VAS */
#define TASK_SIZE_MAX ((_AC(1,UL) << 47) - PAGE_SIZE)
/* Start of kernel virtual address space */
#define __START_KERNEL_map _AC(0xffffffff80000000, UL)
/* From include/linux/mm.h */
/* Page size is 4096 bytes = 2^12 */
#define PAGE_SHIFT 12
#define PAGE_SIZE (_AC(1, UL) << PAGE_SHIFT) /* 4096 */
#define PAGE_MASK (~(PAGE_SIZE - 1))
/* From arch/x86/include/asm/pgtable_64_types.h */
/* Number of entries per page table level */
#define PTRS_PER_PGD 512
#define PTRS_PER_PUD 512
#define PTRS_PER_PMD 512
#define PTRS_PER_PTE 512
The macro TASK_SIZE_MAX is particularly important. It defines the maximum virtual address a user space process can use. Any address above this value is kernel VAS. You will see TASK_SIZE checked inside the kernel’s memory access functions like access_ok(), which validates that a user-supplied pointer actually points into user space and not into kernel space.
/* Example: How the kernel validates a user-space pointer */
/* From include/asm-generic/access_ok.h (simplified concept) */
static inline int access_ok(const void __user *addr, unsigned long size)
{
/*
* Check that the address range [addr, addr+size) is within
* the user virtual address space (below TASK_SIZE).
* This prevents a malicious user process from passing a
* kernel address to a syscall, tricking the kernel into
* reading/writing kernel memory on its behalf.
*/
unsigned long limit = TASK_SIZE;
return (unsigned long)addr < limit &&
size <= limit - (unsigned long)addr;
}
Common Mistakes and How to Avoid Them
A very common beginner mistake is assuming that a virtual address printed by printf() or printk() refers to actual RAM. It does not. Virtual addresses go through MMU translation before accessing RAM. Two different processes can have the same virtual address mapped to completely different physical memory.
Fix: Always distinguish between virtual and physical addresses in your code and documentation. In kernel drivers, use virt_to_phys() only for kernel logical addresses (vmalloc addresses cannot be converted this way).
The kernel VAS contains several different regions: direct-mapped memory (lowmem), vmalloc area, module space, and the kernel image itself. Only the direct-mapped region has a simple fixed offset relationship with physical addresses.
Fix: Use kernel APIs (kmalloc, vmalloc, __get_free_pages) appropriately for each use case. Understand which region a pointer comes from before performing address arithmetic.
The 128 TB user VAS and 128 TB kernel VAS are virtual sizes. Your machine might only have 16 GB of physical RAM. Virtual address space is backed by physical RAM on demand through the page fault mechanism. You can have a 128 TB virtual address space on a machine with 8 GB RAM.
Fix: Always think of virtual and physical memory as separate dimensions. Physical RAM is a scarce resource; virtual address space is relatively plentiful on 64-bit systems.
Key Takeaways
- Every Linux process has its own Virtual Address Space (VAS) divided into user space (lower half) and kernel space (upper half) by the VM split.
- On x86_64 with 4-level paging, only 48 bits of the 64-bit address are used. The upper 16 bits are a sign extension (canonical addressing rule).
- A virtual address is a bitmask: four 9-bit page table indices (PGD, PUD, PMD, PTE) plus a 12-bit page offset. Each level indexes a 512-entry page table.
- The upper 128 TB is Kernel VAS (addresses start with
0xffff). The lower 128 TB is User VAS (addresses start with0x0000). The huge gap between them is the non-canonical hole. - Address translation is performed at runtime by the MMU hardware, which walks the page tables maintained by the Linux kernel.
- In kernel modules and device drivers, always use
copy_from_user()/copy_to_user()to safely cross the user/kernel VAS boundary. - Linux 6.x optionally supports 5-level paging (57-bit addressing, 128 PB per process) for future large-scale workloads.
FAQ — Virtual Address Space and VM Split
Supporting the full 64-bit range would require enormous TLB and MMU hardware for 264 bytes (16 EB) of addressable space, which no real-world system needs. Using 48 bits gives 256 TB of VAS, which is more than sufficient, while keeping CPU design practical and cost-effective. Modern kernels can expand to 57 bits using 5-level paging when needed.
ARM64 Linux also uses 48-bit addressing by default (with optional 52-bit or 56-bit extensions on newer hardware). The VM split on ARM64 is symmetric: 256 TB for user VAS (addresses start with many zero bits) and 256 TB for kernel VAS (addresses start with many one bits), with a non-canonical hole in between, very similar to x86_64.
No. The page table entries for kernel VAS are marked with the Supervisor (U/S) bit cleared, meaning the CPU hardware will generate a Page Fault if user-mode code attempts to access any kernel address. This hardware-enforced protection is one of the core security mechanisms of Linux. Additionally, kernel address randomization (KASLR) and features like KPTI (Kernel Page Table Isolation) provide additional layers of protection.
A page fault occurs when a process accesses a virtual address that does not currently have a valid mapping in the page tables. The MMU triggers a hardware exception, the Linux kernel’s page fault handler runs, and the kernel either maps a physical page to that virtual address (if the access is legitimate) or sends a SIGSEGV signal to the process (if the access is invalid, e.g., accessing NULL or unmapped memory). Page faults are also how demand paging and copy-on-write work in Linux.
Run cat /proc/1/maps (with root) to see the memory map of the init process. For a kernel-level view, look at /proc/iomem and /proc/vmallocinfo. To inspect the actual kernel address layout on your system, run cat /proc/kallsyms | head -20 to see kernel symbol addresses (which will start with 0xffff... on x86_64). You can also check dmesg | grep -i "virtual kernel memory layout" on older kernels.
KASLR (Kernel Address Space Layout Randomization) randomizes the base address at which the kernel loads within the kernel VAS on each boot. This makes it harder for attackers to exploit kernel vulnerabilities because they cannot predict the address of kernel functions and data structures. KASLR changes where inside the kernel VAS the kernel sits, but it does not change the overall VM split boundary between user VAS and kernel VAS.
You should never dereference a user pointer directly in kernel code. The correct approach is: (1) call access_ok() to verify the address is actually within user VAS, then (2) use copy_from_user(dst, src, size) to safely copy data from user space into a kernel buffer. The kernel buffer has a KVA and is safe to use directly. For the reverse direction, use copy_to_user(). These functions handle page faults gracefully and prevent security issues.
kmalloc() allocates physically contiguous memory from the direct-mapped kernel region (lowmem). The resulting virtual address has a fixed offset from the physical address. vmalloc() allocates virtually contiguous memory from the vmalloc area; the physical pages backing it may be scattered across RAM. Device drivers that need DMA-capable memory use kmalloc or dma_alloc_coherent(). Large allocations that do not need physical contiguity use vmalloc().
References and Further Reading
- Linux Kernel Documentation: x86_64 Memory Map — Official kernel docs showing the actual kernel VAS layout regions.
- Linux Kernel Documentation: Memory Management Concepts — Official overview of Linux memory management from the kernel team.
- Intel x86_64 Software Developer Manuals — The authoritative reference for paging, virtual addressing, and MMU behavior on Intel processors.
- Linux Kernel Documentation: AArch64 Memory Layout — ARM64 equivalent of the x86_64 memory map documentation.
Continue Your Free Linux Kernel Development Journey
This tutorial is part of EmbeddedPathashala’s completely free Linux kernel programming course. No fees, no paywalls — ever.
🏠 EmbeddedPathashala Home 📚 Full Course Index