False Sharing and CPU Cache Coherency in the Linux Kernel-Linux Device Driver Training Online

False Sharing and CPU Cache Coherency in the Linux Kernel
A free Linux kernel development course lecture — understand why two unrelated variables sitting next to each other in memory can quietly wreck your multicore performance

If you are following this free Linux kernel development course, you already know that multithreaded code on a multicore machine is fast only when the CPU cores can actually work independently. False sharing is one of the sneakiest performance bugs in systems programming, embedded C, and Linux kernel/driver code, because the code looks completely correct — there is no data race, no missing lock, nothing a compiler or a sanitizer will ever flag. Yet throughput can drop by 5x or more on a busy multicore board. This lecture builds the mental model of CPU caches and cache coherency from the ground up, in plain language, and then shows you exactly how to detect and fix false sharing on a modern Linux system running kernel 6.12 LTS or newer, up through the current 7.1 stable series.

What You Will Learn
CPU cache basics (L1/L2/L3) What a cacheline is Cache coherency (MESI-style) False sharing, step by step Detecting it with perf c2c Fixing it with padding/alignment Kernel-level examples
Prerequisites
Basic C programming POSIX threads basics Comfortable with a Linux terminal Idea of what a mutex/lock does

You don’t need prior kernel experience for this lecture. If you’re completely new to concurrency, skim through the “Critical Sections and Locking” lecture in this free embedded systems course first.

Why Should You Care About False Sharing?

Every performance-sensitive piece of software you will write in this free Linux kernel development course — a kernel module, a network driver’s per-packet counters, a real-time embedded control loop — eventually touches shared memory from more than one CPU core. If two threads on two different cores keep stepping on each other’s cachelines, your carefully lock-free, race-free code can end up slower than a single-threaded version. Understanding false sharing is what separates “it compiles and runs” from “it actually scales.”

A Quick Refresher: What Is a CPU Cacheline?

Modern CPUs never move a single byte between RAM and the CPU. Instead, memory is transferred in fixed-size chunks called cachelines, almost always 64 bytes on current x86-64 and ARM64 cores. When your code reads one byte of a variable, the hardware silently pulls in the entire 64-byte block that byte belongs to, and stores it in the L1 cache of that core. This is great for performance when your code accesses nearby data together (this is called spatial locality) — but it becomes a problem when two variables that are logically unrelated happen to land in the same 64-byte block.

Cacheline Layout — 64 Bytes on a Typical x86-64 / ARM64 Core
counter_a
(4 bytes)
counter_b
(4 bytes)
remaining 56 bytes of the same cacheline

counter_a and counter_b are two independent variables, but the CPU can only fetch, invalidate, and update this cacheline as one 64-byte unit.

Cache Coherency: How Multiple Cores Stay in Sync

On a multicore chip, every core has its own private L1 (and usually L2) cache, plus a shared L3 cache. If Core 0 and Core 1 both keep a copy of the same cacheline, the hardware must guarantee that neither core ever reads a stale value. This guarantee is called cache coherency, and it is implemented through a hardware protocol (commonly a MESI-family protocol — Modified, Exclusive, Shared, Invalid) that tracks the state of every cacheline in every core’s cache.

The rule the hardware enforces is simple: the instant one core modifies a cacheline, every other core’s copy of that same cacheline must be invalidated and re-fetched before it can be used again. This is completely automatic and completely correct — you never see a torn or stale value. The catch is that this invalidate-and-refetch dance is not free. It costs cycles, and it generates traffic on the interconnect between cores.

Cache Coherency Traffic Between Two Cores
Core 0
writes counter_a
⇆
Core 1
writes counter_b

Both cores keep invalidating and re-fetching the shared cacheline on every write, even though they never touch each other’s variable directly. This bouncing back and forth is called cache ping-pong.

What Exactly Is False Sharing?

False sharing happens when two (or more) independent, unrelated variables that are never accessed by the same thread still end up sharing one cacheline purely due to memory layout, and are then written to concurrently by different cores. There is no logical data race — each thread owns its own variable — but the hardware still treats the whole cacheline as one unit, so every write from one core forces an invalidation that the other core pays for. The word “false” refers to the fact that the sharing is an artifact of layout, not of the program’s actual data dependencies.

This is different from true sharing, where two threads genuinely operate on the same variable and need a lock. False sharing needs no lock to fix — it needs better memory layout.

A Minimal Example

Here is a small, self-contained example you can compile and measure yourself. Two threads each increment their own private counter a huge number of times, with no shared state in the logical sense.

struct counters {
    long a;   /* touched only by thread 0 */
    long b;   /* touched only by thread 1 */
};

struct counters shared = {0, 0};

void *worker_a(void *arg) {
    for (long i = 0; i < 500000000L; i++)
        shared.a++;
    return NULL;
}

void *worker_b(void *arg) {
    for (long i = 0; i < 500000000L; i++)
        shared.b++;
    return NULL;
}

Because a and b sit only 8 bytes apart inside the same struct, they almost certainly fall in the same 64-byte cacheline, and each thread’s increments keep evicting the other core’s copy. Measured on a typical multicore Linux box, this version can run several times slower than the fixed version below, even though nothing about the algorithm changed.

Detecting False Sharing on a Modern Kernel

On kernel 6.12 and later, the standard way to actually see false sharing (instead of guessing) is the perf c2c (“cache-to-cache”) subcommand of the Linux perf tool, which is built specifically to surface cacheline contention between cores using hardware performance counters.

# record cacheline contention events while your program runs
sudo perf c2c record -- ./false_sharing_demo

# generate a report sorted by contention
sudo perf c2c report

The report highlights “hot” cachelines with high HITM (Hit-Modified) counts — that is the direct fingerprint of false sharing. This is far more reliable than eyeballing source code, especially in a large kernel module or driver where the offending struct might be defined far away from where it’s actually used concurrently.

Fixing False Sharing: Padding and Alignment

The fix is always the same idea: force each hot, independently-written variable onto its own cacheline so cores stop invalidating each other’s data. In modern C (C11 and later) you do this with alignas, or with compiler attributes.

#include <stdalign.h>

struct counters {
    alignas(64) long a;   /* now on its own cacheline */
    alignas(64) long b;   /* now on its own cacheline */
};

Inside the Linux kernel source tree itself, the same technique is used constantly, but through kernel-provided macros rather than raw alignas, so the layout stays portable across architectures with different cacheline sizes:

struct my_driver_stats {
    atomic_t rx_packets;
} ____cacheline_aligned_in_smp;

The ____cacheline_aligned_in_smp attribute (defined in include/linux/cache.h) pads and aligns a structure to SMP_CACHE_BYTES, which the kernel derives per-architecture at build time. You’ll find this pattern throughout core kernel data structures wherever a field is updated frequently by one CPU while a neighboring field is updated frequently by another — for example around per-CPU run-queue statistics in the scheduler and around hot spinlocks in networking and memory-management code.

Technique Where to Use It Trade-off
alignas(64) / alignas(CACHE_LINE) User-space C/C++ code, RTOS application code Wastes up to 63 bytes per padded field
____cacheline_aligned_in_smp Linux kernel modules and drivers Same memory cost, but architecture-portable
Per-CPU variables Kernel counters that scale with CPU count Best fix when the variable is truly per-CPU (covered in the next lecture)

Common Mistakes and Troubleshooting

Mistake: Padding Everything “Just in Case”

Adding alignas(64) to every field of every struct bloats memory usage and hurts cache efficiency for the common case where fields are accessed together, not concurrently by different cores. Pad only fields that are (a) hot and (b) written by different cores.

Mistake: Assuming a Fixed 64-Byte Cacheline

Some ARM cores and some server-class chips use different cacheline sizes. In kernel code always use SMP_CACHE_BYTES / ____cacheline_aligned_in_smp instead of hardcoding 64, so your driver stays portable.

Mistake: Chasing False Sharing Before Profiling

Restructuring structs is a real engineering cost. Always confirm the hot cacheline with perf c2c first; don’t guess.

Best Practices

  • Group fields that are read together on the same cacheline (this helps performance).
  • Separate fields that are written concurrently by different cores onto different cachelines.
  • Prefer per-CPU data over shared, padded counters when the value genuinely differs per core.
  • Always measure with perf c2c or similar hardware-counter tooling before and after any layout change.
  • Re-check cacheline layout after adding new fields to a hot struct — false sharing can be introduced by an innocent-looking new field months later.

Performance Considerations

False sharing gets dramatically worse as core count increases, because more cores means more copies of the same cacheline bouncing around the interconnect. On an embedded dual-core SoC the effect might be a modest slowdown; on a 32-core or 64-core server-class part running kernel networking or scheduler code, an unpadded hot struct can become a scalability wall that limits throughput no matter how many cores you add. This is exactly why the Linux kernel’s cache-line-aware padding matters more, not less, as core counts keep climbing.

Real-World Use Case

A very common place this bites people in driver development is a small “statistics” struct with adjacent packet counters, one incremented by the receive interrupt handler on one CPU and another incremented by the transmit path pinned to a different CPU. Nothing about the code is wrong, yet under load the driver’s throughput plateaus well below the link’s theoretical bandwidth. Cache-line-aware layout (or switching to true per-CPU counters) is the standard fix applied in real kernel drivers.

Key Takeaways
  • CPUs move data in fixed cacheline-sized chunks (typically 64 bytes), not individual variables.
  • Cache coherency hardware keeps all cores’ copies of a cacheline correct, but at a real performance cost when that cacheline is contended.
  • False sharing is unrelated variables on the same cacheline being written by different cores — no logical race, but real slowdown.
  • Detect it with perf c2c; fix it with cacheline alignment/padding or, better, per-CPU variables.

Conclusion

False sharing is one of those topics that separates intermediate systems programmers from people who can genuinely make multicore Linux kernel code scale. The good news is that once you understand cachelines and cache coherency, spotting and fixing false sharing becomes a routine part of writing performance-sensitive kernel modules, drivers, and embedded C code. In the next lecture of this free Linux kernel development course, we take the natural next step: per-CPU variables, which sidestep false sharing entirely by giving each core its own private copy of hot data.

Frequently Asked Questions

Is false sharing a bug or just a performance issue?

Purely a performance issue. The program produces correct results; it just runs slower than it should because of hardware cache-coherency overhead.

Does false sharing happen in single-core systems?

No. False sharing requires two different CPU cores writing to the same cacheline concurrently. On a single core there’s only one copy of the cache, so there’s nothing to invalidate.

How big is a cacheline on Linux?

Usually 64 bytes on x86-64 and most ARM64 cores, but always confirm on your specific target via getconf LEVEL1_DCACHE_LINESIZE in user space, or SMP_CACHE_BYTES inside kernel code.

Can a lock fix false sharing?

No, and it usually makes it worse. Locking adds its own cacheline (the lock word itself) and serializes access. The correct fix is layout, not locking.

Does false sharing matter on a single-core embedded microcontroller?

Generally no, since there’s only one core and one cache hierarchy. It becomes relevant the moment you move to a multicore SoC, which is increasingly common even in embedded designs.

What is the difference between true sharing and false sharing?

True sharing is when threads genuinely operate on the same logical variable and need synchronization. False sharing is when threads operate on different variables that merely happen to share a cacheline; no synchronization is logically needed, only better memory layout.

Is perf c2c available on every Linux distribution?

It ships as part of the linux-tools/perf package on most modern distributions and requires a kernel with performance monitoring unit (PMU) support enabled, which is standard on virtually all current x86-64 and ARM64 systems running kernel 6.12 or newer.

Continue This Free Linux Kernel Development Course

This lecture is part of EmbeddedPathashala’s free Linux kernel development course, free Linux device drivers course, and free embedded systems course.

Explore All Free Courses

2 Comments

Leave a Reply

Your email address will not be published. Required fields are marked *