🚀 Learn BLE on Linux (BlueZ) – Programming Part 1
Hands-on Bluetooth Low Energy course. Special price: ₹399
Enroll Now on Udemy →Intermediate
Address Translation
Kernel Memory
Welcome back to EmbeddedPathashala’s free Linux kernel development course. In the previous lecture we understood the VM split and how the 64-bit VAS is divided between user space and kernel space on x86_64. In this part we go deeper into what actually happens when a running program accesses a virtual address, how the Linux kernel and the CPU MMU collaborate to make that work, and what this means for you as a Linux device driver developer or kernel programmer.
If you are preparing for a role at companies like TI, Qualcomm, STMicroelectronics, or any company doing embedded Linux and Linux device driver work, the concepts in this free Linux kernel development course tutorial are exactly what interviewers test.
What You Will Learn in This Part
- The complete lifecycle of a memory access: from virtual address to physical RAM
- What a Page Table Entry (PTE) contains beyond just the physical frame number
- The role of the TLB and why TLB flushes matter for driver performance
- How Linux manages the kernel VAS: direct map, vmalloc, module space
- Practical kernel APIs for working with addresses in device drivers
- How to decode a virtual address manually using the bit fields
- Understanding
/proc/pid/pagemapfor virtual-to-physical mapping inspection - Common interview questions and answers on Linux memory management
Step by Step: What Happens When a Program Accesses Memory
Let us trace exactly what happens from the moment a line of C code reads a variable to the moment the data arrives from RAM. This is the complete picture that a free Linux kernel development course must cover to prepare you for real-world kernel work.
Assume your user-space program does this:
#include <stdio.h>
int main(void)
{
int value = 100; /* compiler places this on the stack */
int result = value + 1; /* this triggers a memory read */
printf("%d\n", result);
return 0;
}
When the processor executes int result = value + 1, it needs to read value from memory. Here is the complete step-by-step sequence:
The CPU issues a read for virtual address
0x00007ffe4b3c2d10 (stack variable). This goes first to the TLB.The TLB checks if it has a cached translation for this virtual page. If yes (TLB hit) → go directly to step 6. If no (TLB miss) → the MMU hardware walker runs steps 3–5.
The MMU reads
CR3 to get the physical address of the PGD. It then walks all four levels: PGD → PUD → PMD → PTE, using the bit fields of the virtual address as indices at each level.The MMU reads the final PTE. It checks the Present (P) bit, Read/Write (R/W) bit, User/Supervisor (U/S) bit, and other permission flags. If permissions fail → CPU raises a fault. If Present bit is 0 → Page Fault. If all OK → extract physical frame number (PFN).
The MMU stores this new virtual→physical translation in the TLB so future accesses to the same page are fast.
Physical address = (PFN << 12) | (virtual_address & 0xFFF). The CPU sends this physical address to the memory subsystem (L1/L2/L3 cache or RAM). Data is returned to the CPU register.
This entire process happens in hardware for every memory instruction. The TLB makes most accesses a single-cycle lookup.
Inside a Page Table Entry (PTE)
A Page Table Entry on x86_64 is 8 bytes (64 bits). It stores the physical frame number of the mapped page plus a set of hardware-managed permission and status flags. Understanding what these flags mean is essential for Linux kernel and device driver development.
| Bits 63–52 (Reserved / NX) |
Bits 51–12 (40 bits) |
Bit 9–8 (2 bits) |
Bit 7 | Bit 6 | Bits 5–0 |
|---|---|---|---|---|---|
| NX bit (bit 63) No-Execute protection |
Physical Frame Number (PFN) Physical page base address >> 12 |
SW bits Available for OS use |
PS Page Size (huge page) |
D Dirty (written) |
Flags A, PCD, PWT, U/S, R/W, P |
The most important PTE flag bits for kernel developers are:
| Flag | Bit | Meaning When Set | Kernel/Driver Relevance |
|---|---|---|---|
| P (Present) | 0 | Page is in RAM and valid | If 0, accessing the address triggers a page fault |
| R/W (Read/Write) | 1 | Page is writable | Read-only mappings have this cleared; write attempt causes fault |
| U/S (User/Supervisor) | 2 | User mode can access this page | Kernel pages have this cleared; user code cannot access them |
| A (Accessed) | 5 | CPU has read this page | Used by kernel’s page reclaim (LRU) algorithm |
| D (Dirty) | 6 | CPU has written to this page | Used to track which pages need to be written back to swap/file |
| NX (No-Execute) | 63 | Instruction fetch forbidden | Data pages should always have NX set to prevent code injection attacks |
| PCD (Cache Disable) | 4 | CPU cache bypassed for this page | Device driver memory-mapped I/O (MMIO) regions must set this |
When a device driver maps hardware registers using
ioremap(), Linux marks those pages with PCD (Cache Disable) and PWT (Page Write Through) in the PTE. If the CPU cached reads/writes to hardware registers, the driver might read a stale cached value instead of the actual hardware register value, causing completely wrong behavior. This is why you must always access hardware MMIO through readl()/writel() accessors and never cast an ioremap() address to a plain C pointer and dereference it directly.
Inside the Linux Kernel VAS — What Lives Where
The 128 TB of kernel virtual address space on x86_64 is not a flat, undifferentiated block. The Linux kernel divides it into several well-defined regions. Understanding these regions is critical for free Linux kernel development course graduates who want to write efficient drivers.
The Linux kernel VAS contains several distinct regions each serving a different purpose. This layout is defined in arch/x86/include/asm/pgtable_64_types.h.
Here is what each region is used for:
This is the largest and most important region of the kernel VAS. The Linux kernel maps all available physical RAM here with a fixed, constant offset. On x86_64, this region starts at 0xffff888000000000. Every byte of physical RAM has a corresponding kernel virtual address in this region. You can convert between them using phys_to_virt() and virt_to_phys(). Memory allocated with kmalloc() or __get_free_pages() comes from this region. Because the physical addresses are contiguous, DMA operations work correctly on this memory.
The vmalloc area provides virtually contiguous address ranges that may be backed by physically scattered pages. When you call vmalloc(size) in a kernel module, the kernel allocates individual pages from wherever they are available in physical RAM and maps them into a contiguous chunk of vmalloc VAS. This is how the kernel can provide large contiguous virtual allocations even when physical RAM is fragmented. The ioremap() function — used by device drivers to map hardware registers into the kernel VAS — also carves mappings from this region.
/* Driver example: mapping hardware peripheral registers */
#include <linux/io.h>
#define MY_DEVICE_BASE_ADDR 0xFE201000UL /* physical address of device */
#define MY_DEVICE_SIZE 0x1000 /* 4 KB register space */
static void __iomem *reg_base;
static int my_driver_probe(struct platform_device *pdev)
{
/* ioremap() creates a KVA in the vmalloc area pointing to
* the physical hardware registers. The resulting pointer
* is a Kernel Virtual Address (KVA). */
reg_base = ioremap(MY_DEVICE_BASE_ADDR, MY_DEVICE_SIZE);
if (!reg_base) {
pr_err("ioremap failed for device registers\n");
return -ENOMEM;
}
/* Use readl/writel, NEVER plain pointer dereference */
u32 chip_id = readl(reg_base + 0x00);
pr_info("Device chip ID: 0x%08x\n", chip_id);
return 0;
}
static int my_driver_remove(struct platform_device *pdev)
{
/* Always unmap when done */
iounmap(reg_base);
return 0;
}
When you load a kernel module with insmod or modprobe, the kernel allocates space for the module’s text (code) and data in the module space region. On x86_64, this region is placed close to the core kernel image. The reason for placing modules nearby is that x86_64 call instructions use 32-bit relative offsets. If modules were too far from the kernel in VAS, direct calls between kernel code and module code would not fit in a 32-bit relative offset and would require expensive indirect jumps. Linux kernel 6.x handles modules larger than the traditional 1.5 GB module space using a mechanism based on memory map attributes.
/* Check where a module function lives vs core kernel */
/* In a kernel module: */
#include <linux/kernel.h>
#include <linux/module.h>
static int __init my_mod_init(void)
{
pr_info("Module init function KVA: %px\n", my_mod_init);
/* Typical output: ffffffffc0a12000 -- in module space */
/* Compare to kernel functions which print ffffffff8xxxxxxx */
pr_info("printk KVA: %px\n", printk);
return 0;
}
Hands-On: Decoding a Virtual Address Manually
Let us practice decoding a virtual address into its component bit fields. This is a classic Linux kernel interview question and also genuinely useful when debugging memory issues in kernel development.
Take the virtual address 0xffff888012345678:
/* Virtual address to decode: 0xffff888012345678
*
* Binary representation of bits 47:0 (ignoring the 0xffff sign extension):
* 0x888012345678
*
* Split according to x86_64 4-level paging, 4 KB pages:
*
* Bits 47:39 = PGD index
* Bits 38:30 = PUD index
* Bits 29:21 = PMD index
* Bits 20:12 = PTE index
* Bits 11:0 = Page offset
*
* 0x 8880 1234 5678
* Binary (48 bits):
* 1000 1000 1000 0000 0001 0010 0011 0100 0101 0110 0111 1000
*
* PGD index [47:39] = 1 0001 0001 = 0x111 = 273
* PUD index [38:30] = 0 0000 0000 = 0x000 = 0
* PMD index [29:21] = 0 1001 0001 = 0x091 = 145
* PTE index [20:12] = 1 0100 0101 = 0x145 = 325
* Page offset[11:0] = 0110 0111 1000 = 0x678 = 1656
*
* Since bits 63:48 = 0xffff (all ones), this is a KVA.
* The PGD index 273 falls in the direct-map range,
* suggesting this address is in the kernel's direct
* physical memory map region.
*/
The Linux kernel provides macros to extract each component from a virtual address. These are defined in
arch/x86/include/asm/pgtable.h and related headers:
pgd_index(addr), pud_index(addr), pmd_index(addr), pte_index(addr). These are used internally by the kernel’s own page table management code and are useful to understand when reading kernel source.
Inspecting Address Translations with /proc/pid/pagemap
Linux exposes a powerful tool for examining virtual-to-physical address translations for any process: the /proc/pid/pagemap interface. This is incredibly useful when debugging DMA issues in device drivers or verifying memory layout in kernel modules.
/* Tool to read physical address from /proc/self/pagemap
* for a given virtual address.
* Compile: gcc -o pagemap_reader pagemap_reader.c
* Run as root for physical address information.
*/
#include <stdio.h>
#include <stdlib.h>
#include <stdint.h>
#include <fcntl.h>
#include <unistd.h>
#define PAGE_SHIFT 12
#define PAGE_SIZE (1UL << PAGE_SHIFT)
#define PFN_MASK ((1ULL << 55) - 1) /* bits 54:0 are PFN */
#define PAGE_PRESENT (1ULL << 63)
static uint64_t vaddr_to_paddr(pid_t pid, uint64_t vaddr)
{
char path[64];
snprintf(path, sizeof(path), "/proc/%d/pagemap", pid);
int fd = open(path, O_RDONLY);
if (fd < 0) {
perror("open pagemap");
return 0;
}
/* Each 8-byte entry corresponds to one virtual page */
uint64_t entry;
off_t offset = (vaddr / PAGE_SIZE) * sizeof(uint64_t);
if (pread(fd, &entry, sizeof(entry), offset) != sizeof(entry)) {
perror("pread pagemap");
close(fd);
return 0;
}
close(fd);
if (!(entry & PAGE_PRESENT)) {
printf("Page not present in RAM (may be swapped out)\n");
return 0;
}
uint64_t pfn = entry & PFN_MASK;
uint64_t paddr = (pfn << PAGE_SHIFT) | (vaddr & (PAGE_SIZE - 1));
return paddr;
}
int main(void)
{
int my_var = 42;
uint64_t vaddr = (uint64_t)(uintptr_t)&my_var;
uint64_t paddr = vaddr_to_paddr(getpid(), vaddr);
printf("Variable value: %d\n", my_var);
printf("Virtual address: 0x%016lx\n", vaddr);
printf("Physical address: 0x%016lx\n", paddr);
return 0;
}
Since Linux kernel 4.0, reading the physical address from
/proc/pid/pagemap requires CAP_SYS_ADMIN capability (run as root). This restriction was added to prevent side-channel attacks where unprivileged processes could determine the physical layout of memory and exploit DRAM row-hammer vulnerabilities. Always run this as root or with appropriate capabilities.
Best Practices for Address Handling in Linux Kernel Modules
The Linux kernel uses GCC’s sparse checker tool and compiler attributes to annotate pointer types with their expected address space. Use __user for user virtual addresses, __iomem for memory-mapped I/O pointers, and __percpu for per-CPU data pointers. These annotations catch misuse at compile time when running with sparse (make C=2).
/* Correct pointer annotation in a character driver */
static ssize_t my_read(struct file *filp, char __user *buf,
size_t count, loff_t *pos)
{
char kernel_buffer[256];
/* ... fill kernel_buffer ... */
/* copy_to_user: kernel_buffer (KVA) -> buf (UVA) */
if (copy_to_user(buf, kernel_buffer, count))
return -EFAULT;
return count;
}
| Use Case | API | Address Space | Physical Contiguous? |
|---|---|---|---|
| Small kernel allocations (< 4 MB) | kmalloc() | Direct map (KVA) | Yes |
| Large allocations (no DMA) | vmalloc() | vmalloc area (KVA) | No |
| DMA buffers (hardware access) | dma_alloc_coherent() | Direct map (KVA) | Yes |
| Map hardware registers | ioremap() | vmalloc area (KVA) | N/A (MMIO) |
| User-space mmap to driver | remap_pfn_range() | User VAS | Configurable |
Dereferencing an invalid KVA in kernel context does not simply return garbage — it causes a kernel oops or panic. Use virt_addr_valid(addr) to check if a kernel virtual address is in the direct-map region before converting it to a physical address. For vmalloc addresses, use is_vmalloc_addr(addr).
#include <linux/mm.h>
#include <linux/vmalloc.h>
void inspect_kva(void *addr)
{
if (virt_addr_valid(addr)) {
pr_info("Direct-map KVA: %px, phys: %lx\n",
addr, virt_to_phys(addr));
} else if (is_vmalloc_addr(addr)) {
pr_info("vmalloc KVA: %px\n", addr);
/* Cannot use virt_to_phys() here! */
} else {
pr_warn("Unknown address space: %px\n", addr);
}
}
Common Interview Questions on Linux Memory Management
These questions come up repeatedly in Linux kernel and embedded Linux developer interviews at companies working on Linux device drivers and embedded systems:
A virtual address is what software sees and uses. It is a bitmask that the MMU translates to a physical address (actual RAM location) using page tables. Two processes can have the same virtual address but it maps to different physical memory. The Linux kernel maintains per-process page tables; the MMU hardware does the runtime translation.
User-space pointers are UVAs valid in the user process’s address space. In kernel context, the page tables are different (kernel’s own page tables are in CR3 during syscall handling on older kernels; KPTI may modify this). More critically, the user page might be swapped out, not present, or the address might be malicious. copy_from_user() handles all these cases safely, including recovering from page faults that occur during the copy.
ioremap(phys_addr, size) creates a kernel virtual address mapping in the vmalloc area that points to the physical address range of your hardware device’s registers or memory-mapped I/O region. It sets up PTE entries with cache-disabling flags (PCD) so CPU caches do not interfere with hardware register access. The returned KVA should be accessed only through readb/readw/readl/readq and writeb/writew/writel/writeq accessor functions.
kmalloc() allocates physically contiguous memory from the direct-mapped region. It is suitable for DMA and for sizes up to a few megabytes. vmalloc() allocates virtually contiguous but physically scattered pages from the vmalloc area. It handles larger allocations but cannot be used for DMA since the physical pages are non-contiguous. kmalloc() is generally faster due to simpler page table setup.
A TLB flush invalidates cached virtual-to-physical address translations in the CPU’s Translation Lookaside Buffer. The kernel must flush TLBs when it changes page table entries, for example when unmapping a page, changing permissions, or during a context switch between processes. On multi-core systems, TLB flushes must be sent to all CPUs that might have the stale entries cached, through a mechanism called TLB shootdown (using IPIs — Inter-Processor Interrupts). Unnecessary TLB flushes are a significant source of performance bottlenecks in high-performance kernel code.
KPTI (Kernel Page Table Isolation) was introduced in Linux 4.15 as a mitigation for the Meltdown vulnerability (CVE-2017-5754). Meltdown allowed user-space programs to read kernel memory by exploiting speculative execution in the CPU. KPTI fixes this by maintaining two separate sets of page tables: one for user mode (which does not include kernel VAS mappings except for the minimum needed for syscall entry) and one for kernel mode. Switching between user and kernel mode now involves a CR3 switch, which is more expensive but prevents the speculative access to kernel memory.
Summary: Key Takeaways from Part 2
- Every memory access goes: CPU → TLB lookup → (on miss) MMU hardware page table walk → physical RAM. The TLB makes most accesses fast by caching recent translations.
- Each Page Table Entry contains the physical frame number plus hardware-managed flags (Present, R/W, U/S, Dirty, Accessed, NX, Cache-Disable).
- The kernel VAS has distinct regions: direct-mapped lowmem (all physical RAM), vmalloc/ioremap area (virtually contiguous allocs and hardware mappings), and module space (loadable kernel modules).
- In device drivers, hardware registers must be accessed via
ioremap()+readl()/writel()— never via plain pointer dereference. - Choose memory allocation API carefully:
kmallocfor small DMA-capable allocations,vmallocfor large non-DMA allocations,dma_alloc_coherentfor DMA-capable large buffers. - Use
__user,__iomemannotations on pointers. Always usecopy_from_user()andcopy_to_user()to cross the user/kernel boundary. - The
/proc/pid/pagemapinterface lets you inspect virtual-to-physical mappings from user space (requires root on Linux 6.x).
References and Further Reading
- Linux Kernel Docs: x86_64 Memory Map — Definitive reference for kernel VAS region boundaries.
- Linux Kernel Docs: Memory Management APIs — Official documentation for
kmalloc,vmalloc,ioremap, and related APIs. - Linux Kernel Docs: Accessing I/O Memory — How to correctly access MMIO in device drivers.
- Intel x86_64 Software Developer Manuals (Vol 3, Chapter 4) — Complete hardware-level specification of paging and address translation.
Join Thousands of Engineers on EmbeddedPathashala
100% free Linux kernel development course, Linux device drivers course, and embedded systems course. No registration fees, no paywalls.
🏠 Visit EmbeddedPathashala 📚 Full Course Index
1 Comment