What You Will Learn

• How CPU cache lines work and why they matter for kernel and driver code
• What false sharing is and why it can silently wreck scalability
• Practical techniques the kernel uses to prevent it: padding, alignment, and per-CPU data
• How to detect false sharing on a running 6.12+ kernel using modern tooling

Prerequisites

This lecture assumes you’re already familiar with basic locking (spinlocks, mutexes) and with the general idea of concurrent access from multiple CPU cores, covered earlier in this free Linux kernel development course.

Why CPUs Don’t Read RAM One Byte at a Time

Modern processors never fetch a single byte from main memory in isolation. Whenever the CPU needs a byte, it pulls in an entire block around it — the cache line, almost always 64 bytes on current x86-64 and ARM64 hardware — into its L1, L2, and (usually shared) L3 caches. This exists purely for speed: an L1 cache hit costs roughly one to a few nanoseconds, while a trip out to main memory can cost fifty to well over a hundred nanoseconds depending on the platform. Sequential memory access benefits automatically, because once one byte of a cache line is fetched, the rest of that line is already sitting in cache for free.

Cache Coherency: The Hidden Cost of Multi-Core Access

On a single core, this is purely a speed win. On a multi-core system, it introduces a new problem: if two different cores each cache the same line, and one of them writes to it, the hardware must guarantee the other core doesn’t keep reading a stale copy. Processors solve this with cache-coherency protocols (commonly a variant of MESI — Modified, Exclusive, Shared, Invalid). When core A writes to a cache line that core B also has cached, the protocol invalidates B’s copy, and B has to re-fetch the line the next time it touches that data. That re-fetch is not free, and if it happens repeatedly between cores in quick succession, the cache line effectively bounces back and forth — a pattern the kernel community usually calls cache ping-pong.

False Sharing: When Unrelated Data Fights Over One Cache Line

False sharing happens when two variables that are logically unrelated — used by different CPUs, protected by different locks, or not related at all — happen to land in the same 64-byte cache line purely because of how a structure was laid out in memory. Even though the two CPUs are never touching the same logical data, the hardware still has to treat the whole cache line as shared, so writes from one CPU invalidate the other CPU’s cached copy of a variable it never even looks at.

This is a purely mechanical, layout-driven problem. There’s no logical race, no incorrect lock, and no bug that a correctness checker like lockdep or KCSAN would flag — the code is functionally correct. The only symptom is that throughput drops as more cores are added, sometimes dramatically, which makes false sharing one of the more frustrating classes of performance bug to diagnose by inspection alone.

How the Kernel Avoids False Sharing

1. Cache-Line Alignment and Padding

The most direct fix is to make sure hot, independently-accessed fields don’t share a cache line in the first place. The kernel provides the ____cacheline_aligned (and related __cacheline_aligned_in_smp) annotations for exactly this purpose, forcing a structure or field onto its own cache-line boundary.

struct per_cpu_stat {
    long hits;
} ____cacheline_aligned;

struct per_cpu_stat cpu_stats[NR_CPUS];

Because each element is padded out to a full cache line, no two CPUs’ counters can ever land in the same line, even though the array itself is contiguous in memory.

2. Grouping Hot Fields Together, Deliberately

The flip side of avoiding false sharing is embracing true sharing where it helps: fields that are always read together should be placed together, ideally within one cache line, so a single fetch satisfies the whole access instead of pulling in multiple lines. This is why performance-sensitive kernel structures are often organized with frequently accessed “hot” fields clustered near the top of the structure and rarely touched “cold” fields pushed toward the end.

3. Per-CPU Data

For counters and statistics that every core updates constantly, the kernel’s per-CPU data mechanism (DEFINE_PER_CPU, this_cpu_inc(), and friends) sidesteps the whole problem: each CPU gets its own private copy of the variable, allocated so that different CPUs’ copies don’t share cache lines. There’s no cross-core invalidation at all during normal operation, because no core ever writes to another core’s copy.

DEFINE_PER_CPU(long, irq_count);

/* Safe, contention-free increment from any CPU */
this_cpu_inc(irq_count);

Detecting False Sharing on a Running Kernel

You rarely need to guess where false sharing is happening. On current kernels, perf c2c (“cache-to-cache”) is the standard tool: it records memory accesses along with which CPU touched which cache line, and highlights lines with heavy cross-CPU contention.

# Record system-wide cache-to-cache access data
perf c2c record -a -- sleep 20

# Analyze and display the contended cache lines
perf c2c report

The report ranks cache lines by the amount of cross-CPU load/store contention observed, which is usually enough to point directly at the offending structure. Alongside perf c2c, general-purpose profiling with perf stat, watching counters like cache-misses and LLC-load-misses, can hint that something cache-related is worth investigating further before you reach for the more targeted tool.

Common Mistakes

• Packing unrelated per-CPU fields into one struct without padding. This is the single most common source of false sharing in driver code.
• Assuming a correctness pass rules out this class of bug. False sharing produces no incorrect results, only a scalability cliff, so it’s invisible to functional testing.
• Over-padding everything “just in case.” Cache-line padding trades memory footprint for cache friendliness; applying it indiscriminately bloats structures and can hurt overall cache utilization elsewhere.
• Ignoring struct field order. The compiler won’t reorder your fields for you; layout is exactly what you wrote.

Best Practices

• Group fields by access pattern first, then by logical relationship — hot fields together, cold fields separated.
• Use per-CPU variables for counters and statistics that every core updates independently.
• Apply ____cacheline_aligned deliberately, on fields you’ve actually identified as contended, not everywhere by default.
• When chasing a multi-core scalability problem that correctness tools can’t explain, reach for perf c2c before assuming it’s a locking issue.

Performance and Security Considerations

Cache-line padding is a pure performance technique, not a correctness one, so it never changes program behavior — only throughput and latency under concurrent load. There isn’t a direct security angle to false sharing itself, but timing side-channels in general do exploit cache behavior, so security-sensitive kernel code (cryptographic routines in particular) sometimes has additional constant-time and cache-behavior requirements that go beyond ordinary performance tuning; that’s a distinct topic from the false-sharing avoidance covered here.

Summary & Key Takeaways

• CPUs move memory in fixed-size cache lines (64 bytes on most current hardware), not individual bytes.
• Cache-coherency protocols keep multiple cores’ cached copies consistent, at a real performance cost when lines are shared across cores.
• False sharing occurs when unrelated variables land in the same cache line, causing unnecessary cross-core invalidation.
• The kernel avoids it with cache-line alignment/padding, deliberate field grouping, and per-CPU data structures.
• perf c2c is the standard modern tool for finding real false-sharing hotspots on a running system.

Conclusion

False sharing is a good reminder that “correct” and “fast” are separate goals in kernel and systems programming. The data can be perfectly race-free and still scale badly across cores purely because of memory layout. Once you know to look for it, the fix is usually small — padding, regrouping fields, or switching to per-CPU storage — but finding it without the right tooling can be genuinely difficult. That wraps up this part of the free Linux kernel development course; from here, the synchronization series continues into how these locking and cache-awareness principles show up together in real driver code.

Frequently Asked Questions