PREV_LEC NEXT_LEC

What You Will Learn

Prerequisites

This lecture assumes you have already gone through the earlier lectures in this free linux device drivers course on virtual memory, page tables, and remap_pfn_range()/mmap() file operations. You should be comfortable building and loading a kernel module (insmod/rmmod) and reading dmesg output. A working Linux 6.x kernel build environment (kernel headers installed, or a full kernel source tree for cross-compiling on embedded boards) is required to run the demo module.

Why Caching Exists

Every computer system has a huge gap between how fast the CPU can compute and how fast storage can respond. A modern CPU core can execute an instruction in under a nanosecond, while a spinning disk can take milliseconds to seek and return data — a difference of six orders of magnitude. Caching exists to hide this gap: keep a copy of the data you are likely to reuse in something faster and closer to the CPU, so that most accesses never have to wait on the slow device.

This idea repeats at every layer of the system. The CPU itself has a small, extremely fast L1 cache next to each core, a larger but slightly slower L2 cache, and a shared L3 cache that all cores on the chip can use. As you move from L1 to L3, capacity grows and speed drops, but even L3 is still far faster than main memory (RAM). The operating system applies exactly the same principle one level down: it treats RAM as a cache in front of persistent storage. That RAM-as-cache-for-disk layer is what the kernel calls the page cache.

The Linux Page Cache

Every regular file, block device, and memory-mapped file in Linux is backed by an in-kernel structure called struct address_space, which indexes the file’s data in units of pages. When a process calls read(), the kernel first checks whether the requested page already exists in this address space’s page cache. If it does, the data is copied straight back to user space — no disk access at all. If it doesn’t, the kernel issues a read from the underlying block device, stores the result as a new page in the cache, and then serves the request. The next read of the same offset is effectively free.

Writes work the other way around but follow the same principle. A buffered write() does not touch the disk immediately. It copies the data into the corresponding page in the page cache and marks that page as dirty — meaning the in-memory copy is now newer than what is stored on disk. A page whose in-memory content still matches the on-disk content is called clean. This distinction, clean vs dirty, is the single most important piece of bookkeeping the page cache performs, because it is what tells the kernel which pages must eventually be written back before they can be safely discarded or before the system can be considered “in sync.”

Writeback: How Dirty Pages Reach Disk

Delaying writes is deliberate, not a shortcut. It buys two real benefits: the application resumes immediately after write() returns instead of blocking on the disk, and the kernel gets to batch and reorder many small writes into fewer, larger, more disk-friendly I/O operations. But dirty pages cannot be left in memory forever — they need to be flushed back to the backing store, a process the kernel calls writeback.

On modern Linux kernels, writeback is handled per backing device by a pool of kernel worker threads built on the bdi_writeback infrastructure (the old single global pdflush daemon from very old kernels is long gone). These flusher threads wake up under two conditions:

You can observe these tunables directly on any Linux 6.x system:

$ cat /proc/sys/vm/dirty_background_ratio
10
$ cat /proc/sys/vm/dirty_ratio
20
$ cat /proc/sys/vm/dirty_writeback_centisecs
500
$ cat /proc/sys/vm/dirty_expire_centisecs
3000

The dd command is the classic example that trips people up: dd finishing does not mean your data is on disk, it only means the data has reached the page cache. This is exactly why dd is so often chained with an explicit sync afterwards when writing to removable media or embedded flash storage.

Forcing Writeback: O_SYNC, O_DIRECT, fsync() and fdatasync()

Sometimes buffered, delayed writeback is exactly what you don’t want — for example, a database committing a transaction, or a bootloader writing critical configuration to an eMMC device before a power cut. Linux gives you several explicit ways to control this:

O_DIRECT is worth a specific warning: its exact semantics depend heavily on the underlying filesystem, alignment requirements for buffers and offsets are strict, and the kernel documentation itself notes it should be treated as a specialized tool, not a default performance switch.

Inspecting Cache and Dirty State from the Kernel

From kernel space, the page cache and dirty accounting are exposed through the page/folio state counters. On a modern kernel these are read with global_node_page_state() using node stat items such as NR_FILE_DIRTY and NR_WRITEBACK, and the same numbers are what populate /proc/meminfo in user space:

$ grep -E "Dirty|Writeback|Cached" /proc/meminfo
Cached:          412356 kB
Dirty:              1024 kB
Writeback:             0 kB

Cached is the current size of the page cache, Dirty is memory waiting to be written back, and Writeback is memory currently in the process of being written. These three numbers are exactly what our demo module below will read from inside the kernel.

Demo: ep_pagecache_stats Kernel Module

The following original module creates a /proc/ep_pagecache_stats entry that reports live dirty and writeback page counts using the same node page-state counters the kernel itself uses for /proc/meminfo. It is deliberately read-only and safe to load on any 6.x kernel.

// ep_pagecache_stats.c
#include <linux/module.h>
#include <linux/kernel.h>
#include <linux/proc_fs.h>
#include <linux/seq_file.h>
#include <linux/mm.h>
#include <linux/vmstat.h>

static int ep_pagecache_show(struct seq_file *m, void *v)
{
    unsigned long dirty_pages     = global_node_page_state(NR_FILE_DIRTY);
    unsigned long writeback_pages = global_node_page_state(NR_WRITEBACK);

    seq_printf(m, "ep_pagecache_stats: dirty=%lu pages (%lu KB)\n",
               dirty_pages, dirty_pages * (PAGE_SIZE / 1024));
    seq_printf(m, "ep_pagecache_stats: writeback=%lu pages (%lu KB)\n",
               writeback_pages, writeback_pages * (PAGE_SIZE / 1024));
    return 0;
}

static int ep_pagecache_open(struct inode *inode, struct file *file)
{
    return single_open(file, ep_pagecache_show, NULL);
}

static const struct proc_ops ep_pagecache_fops = {
    .proc_open    = ep_pagecache_open,
    .proc_read    = seq_read,
    .proc_lseek   = seq_lseek,
    .proc_release = single_release,
};

static int __init ep_pagecache_init(void)
{
    proc_create("ep_pagecache_stats", 0444, NULL, &ep_pagecache_fops);
    pr_info("ep_pagecache_stats: module loaded, read /proc/ep_pagecache_stats\n");
    return 0;
}

static void __exit ep_pagecache_exit(void)
{
    remove_proc_entry("ep_pagecache_stats", NULL);
    pr_info("ep_pagecache_stats: module unloaded\n");
}

module_init(ep_pagecache_init);
module_exit(ep_pagecache_exit);
MODULE_LICENSE("GPL");
MODULE_AUTHOR("EmbeddedPathashala");
MODULE_DESCRIPTION("Reads live page cache dirty/writeback counters");

A minimal Makefile to build it against your running kernel headers:

obj-m += ep_pagecache_stats.o

all:
	make -C /lib/modules/$(shell uname -r)/build M=$(PWD) modules

clean:
	make -C /lib/modules/$(shell uname -r)/build M=$(PWD) clean

Build, load, and generate some dirty pages to watch the counters move:

$ make
$ sudo insmod ep_pagecache_stats.ko
$ dd if=/dev/zero of=testfile bs=1M count=200 status=none
$ cat /proc/ep_pagecache_stats
ep_pagecache_stats: dirty=51200 pages (204800 KB)
ep_pagecache_stats: writeback=0 pages (0 KB)
$ sync
$ cat /proc/ep_pagecache_stats
ep_pagecache_stats: dirty=0 pages (0 KB)
ep_pagecache_stats: writeback=0 pages (0 KB)

Expected dmesg output when loading and unloading:

$ dmesg | tail -4
[  812.441023] ep_pagecache_stats: module loaded, read /proc/ep_pagecache_stats
[  901.552211] ep_pagecache_stats: module unloaded

Notice the dirty count jumps immediately after the buffered dd write (data only reached RAM) and drops back to zero only after the explicit sync forces writeback — a direct, hands-on demonstration of everything explained above.

CPU Cache vs Page Cache: Not the Same Layer

It’s worth being precise about terminology here, since both use the word “cache.” The CPU’s L1/L2/L3 caches are hardware structures that cache raw memory words for the CPU core and are managed almost entirely by silicon, not by your driver code. The Linux page cache is a software structure, entirely managed by the kernel’s memory management subsystem, that caches file-backed data in ordinary RAM. A page sitting in the page cache still has to pass through the CPU cache hierarchy every time it’s actually touched by a core — the two layers are stacked, not interchangeable.

Common Mistakes

• Assuming a successful write() or a finished dd means data is safely on disk — it only means the data reached the page cache.
• Reaching for O_DIRECT as a general “make writes faster” switch — it removes the kernel’s read-ahead and write-coalescing benefits and has strict, filesystem-specific alignment rules.
• Calling fsync() after every single small write in a hot loop, which defeats the entire purpose of write-back batching and can severely hurt throughput on embedded flash storage.
• Forgetting that close() does not imply the data is flushed to disk — only an explicit fsync()/fdatasync()/sync() does.
• Not accounting for page cache growth on memory-constrained embedded boards, where a large sequential write can quickly push the system toward its dirty_ratio throttling point.

Best Practices

• Prefer normal buffered I/O by default; only reach for O_SYNC/O_DIRECT when you have a specific durability or bypass requirement you can justify.
• Batch related writes and call fsync() once at a natural transaction boundary rather than after every write.
• On embedded systems with small RAM budgets, consider tuning dirty_background_ratio/dirty_ratio lower so writeback happens more proactively instead of in large bursts.
• Use /proc/meminfo and /proc/vmstat (or a module like ep_pagecache_stats above) during development to actually observe cache/dirty behaviour instead of guessing.
• Always call sync (or the equivalent fsync()/fdatasync()) before removing external storage such as an SD card or USB drive.

Interview Questions

Summary

The Linux page cache is what makes ordinary file I/O fast: reads are served from RAM whenever possible, and writes land in RAM first and are written back to disk on a schedule the kernel controls through time and threshold-based triggers. Understanding clean vs dirty pages, the writeback flusher infrastructure, and the explicit sync primitives (O_SYNC, O_DIRECT, fsync(), fdatasync(), sync()) gives you real control over the durability-vs-performance trade-off — a skill that matters directly in embedded storage, databases, and any system where power can be lost mid-write. This wraps up the Kernel Memory Management chapter of this free linux kernel development course; the next chapter moves into DMA and hardware-driven memory transfers.

FAQ

PREV_LEC  NEXT_LEC