• The real difference between a spinlock and a mutex, not just the textbook definition
• Why the spinlock-vs-mutex choice depends entirely on your critical section’s behaviour, not on “which is faster”
• Which kernel APIs are safe to call inside a spinlock, and which ones will crash your system
• How to deliberately trigger and read a “scheduling while atomic” bug report
• How lockdep, CONFIG_DEBUG_ATOMIC_SLEEP, and preempt_count() cooperate to catch this class of bug
• How PREEMPT_RT changes spinlock behaviour on kernel 6.12 and later

• Comfortable building and inserting a basic loadable kernel module (insmod/rmmod)
• A Linux VM running a reasonably recent kernel (6.6 LTS or 6.12 LTS recommended) with kernel headers installed
• Basic understanding of process context vs interrupt/atomic context
• Root access on a disposable VM — we are going to crash things on purpose

What Is a Spinlock in the Linux Kernel?

A spinlock is the simplest mutual-exclusion primitive the kernel offers. When a CPU tries to acquire a spinlock that is already held, it does not go to sleep — it “spins,” repeatedly polling the lock in a tight loop, burning CPU cycles until the lock becomes free. That busy-wait behaviour is exactly why a spinlock critical section must be short and must never block.

Because a spinning CPU cannot be doing anything else, spinlocks are meant for very brief critical sections — a handful of instructions — often used to protect data shared between process context and interrupt handlers.

What Is a Mutex in the Linux Kernel?

A mutex behaves very differently. When a task cannot acquire a mutex, it is put to sleep and taken off the CPU entirely; the scheduler picks another task to run, and the sleeping task is woken up once the mutex is released. This makes a mutex safe to hold across code that might block — file I/O, memory allocation with GFP_KERNEL, or copying data to and from user space.

The trade-off is cost: putting a task to sleep and waking it back up involves the scheduler, and that is measurably more expensive than a short spin. This is the heart of the spinlock vs mutex trade-off — busy-waiting cost versus context-switch cost.

Spinlock vs Mutex: The One Rule That Matters

Here is the rule that this entire lecture is built around:

Never call anything that can sleep while holding a spinlock. If your critical section might block — even occasionally — you need a mutex (or a different design entirely), not a spinlock.

Why copy_to_user() Is the Classic Trap

copy_to_user() looks harmless — it just moves bytes from kernel space to a user buffer. But the destination is user-space memory, which can be paged out. If a page fault occurs while copying, the kernel may need to block to bring the page back in, and blocking while holding a spinlock is exactly the violation we are talking about. This single function is responsible for more spinlock-vs-mutex bugs in real driver code than almost anything else.

Property	Spinlock	Mutex
Behaviour when contended	Busy-waits (spins)	Sleeps, scheduler picks another task
Safe to hold across a sleep?	No — must never sleep while held	Yes, that is exactly its purpose
Usable in interrupt context?	Yes (with _irqsave variant)	No, never
Ideal critical section length	Very short (microseconds)	Can be longer, including I/O
Cost when uncontended	Very low	Slightly higher (atomic ops + bookkeeping)

A Practical Example: Building a Deliberately Buggy Driver

The fastest way to internalize the spinlock vs mutex rule is to break it on purpose and watch the kernel complain. Below is an original, minimal misc character driver that guards a small in-kernel buffer with a spinlock, plus a module parameter that lets us intentionally sleep inside the critical section.

// guarded_secret_drv.c — minimal misc char driver for demonstration
#include <linux/module.h>
#include <linux/miscdevice.h>
#include <linux/fs.h>
#include <linux/spinlock.h>
#include <linux/uaccess.h>
#include <linux/delay.h>

#define MAX_SECRET_LEN 64

static char secret_buf[MAX_SECRET_LEN] = "hello-secret";
static DEFINE_SPINLOCK(guard_lock);

static int trigger_bug;
module_param(trigger_bug, int, 0644);
MODULE_PARM_DESC(trigger_bug, "Set to 1 to sleep inside the spinlock on purpose");

static ssize_t secret_write(struct file *filp, const char __user *ubuf,
                             size_t count, loff_t *off)
{
    if (count >= MAX_SECRET_LEN)
        count = MAX_SECRET_LEN - 1;

    spin_lock(&guard_lock);

    /* copying kernel-local data only here — this part is fine */
    memset(secret_buf, 0, MAX_SECRET_LEN);

    if (trigger_bug) {
        /* THIS is the violation: sleeping while holding a spinlock */
        set_current_state(TASK_INTERRUPTIBLE);
        schedule_timeout(HZ / 2);
    }

    spin_unlock(&guard_lock);

    if (copy_from_user(secret_buf, ubuf, count))
        return -EFAULT;

    return count;
}

static const struct file_operations secret_fops = {
    .owner = THIS_MODULE,
    .write = secret_write,
};

static struct miscdevice secret_dev = {
    .minor = MISC_DYNAMIC_MINOR,
    .name  = "guarded_secret",
    .fops  = &secret_fops,
};

static int __init secret_init(void)
{
    return misc_register(&secret_dev);
}

static void __exit secret_exit(void)
{
    misc_deregister(&secret_dev);
}

module_init(secret_init);
module_exit(secret_exit);
MODULE_LICENSE("GPL");

With trigger_bug=0, this driver behaves normally. Set the module parameter to 1 and write to the device node, and you will deliberately hit the exact violation this lecture is about.

Triggering the Bug

$ sudo insmod guarded_secret_drv.ko trigger_bug=1
$ sudo dmesg -C
$ echo "test" | sudo tee /dev/guarded_secret
$ dmesg | tail -n 30

How the Kernel Detects Scheduling While Atomic

Every task carries a preempt_count. Acquiring a spinlock increments it; releasing decrements it. When the scheduler is invoked with a non-zero preempt_count, it knows something is wrong — code is trying to give up the CPU while it is supposed to be in an atomic section. The scheduler’s internal bug handler prints a report and, on a debug-enabled kernel, a full stack trace pointing straight at the offending function.

On a kernel built with CONFIG_DEBUG_ATOMIC_SLEEP enabled, functions that can block — such as might_sleep() internally used by memory allocators and locking primitives — actively check the atomic-context state before proceeding, so you often get the warning even earlier, right at the point of the offending call rather than deep inside the scheduler.

Testing on a Modern 6.12+ Kernel

The core mechanism has not changed since older kernels, but the tooling around it has matured. On kernel 6.12 and later:

• CONFIG_DEBUG_ATOMIC_SLEEP is commonly enabled in distro debug kernels and catches the violation earlier, with a clearer message.
• Lockdep (CONFIG_PROVE_LOCKING) tracks lock-ordering and context rules across the whole system, not just this one lock, so it can catch related mistakes lockdep-style tools missed on older kernels.
• PREEMPT_RT, merged into the mainline kernel starting with 6.12, changes spinlock semantics on real-time-enabled kernels: a regular spinlock_t becomes a sleeping lock internally (backed by an rt_mutex), while only raw_spinlock_t retains true non-sleeping, disable-preemption behaviour. If you are building drivers that must run correctly under PREEMPT_RT, this distinction matters — code that assumes a plain spinlock never sleeps may need to move to raw_spinlock_t.

$ uname -r
6.12.0-generic
$ zcat /proc/config.gz | grep -E "DEBUG_ATOMIC_SLEEP|PROVE_LOCKING|PREEMPT_RT"

Common Mistakes and Troubleshooting

• Mistake: Calling copy_to_user()/copy_from_user() inside a spinlock. Fix: Copy into a kernel-local stack or heap buffer first, release the lock, then copy to/from user space.
• Mistake: Allocating memory with GFP_KERNEL inside a spinlock. Fix: Use GFP_ATOMIC if allocation truly cannot be moved outside the lock, or restructure to allocate before locking.
• Mistake: Nesting a mutex_lock() inside a spinlock’s critical section. Fix: Never combine them this way — redesign so the mutex-protected work happens outside the spinlock entirely.
• Mistake: Forgetting the _irqsave variant when a spinlock is also touched from an interrupt handler, causing a self-deadlock. Fix: Use spin_lock_irqsave()/spin_unlock_irqrestore() consistently wherever the lock might be taken from IRQ context.

Best Practices for Choosing Between Spinlock and Mutex

• Default to a mutex unless you have a specific, measured reason to use a spinlock.
• Keep spinlock critical sections as short as physically possible — ideally just a handful of instructions.
• If the critical section is or ever could become long, or needs to call anything that might block, that is your signal to use a mutex instead.
• Use spinlocks when the same data is shared between process context and an interrupt handler, since mutexes cannot be used in interrupt context at all.
• Always build and test with a debug kernel (CONFIG_DEBUG_ATOMIC_SLEEP, CONFIG_PROVE_LOCKING) during development, even if your production kernel disables these checks for performance.

Performance Considerations

On a single-core system, a spinlock that spins is almost pure waste — there is no other CPU to release the lock, so the kernel typically just disables preemption instead of truly spinning. On multi-core systems, spinlocks shine for very short sections because they avoid the cost of a context switch entirely. Mutexes cost more per acquisition due to scheduler involvement, but that cost is usually irrelevant next to the cost of whatever blocking operation justified using a mutex in the first place.

Security Considerations

Getting the spinlock vs mutex choice wrong is not just a stability bug — a scheduling-while-atomic crash can be triggered by an unprivileged user simply exercising a code path in your driver, turning a locking mistake into a local denial-of-service vector. Always validate untrusted input sizes before entering a critical section, and never trust that a “rare” code path (like an error branch) is exempt from the same locking discipline as the common path.

Real-World Use Cases

• Network drivers use spinlocks to protect packet queues shared between the driver’s interrupt handler and its transmit path.
• Character and block device drivers commonly use a mutex to serialize file operations like read/write/ioctl, since these routinely call copy_to_user()/copy_from_user().
• Filesystem code uses mutexes extensively for operations that touch disk I/O, which is inherently blocking.

• A spinlock busy-waits; a mutex sleeps. That single difference decides the spinlock-vs-mutex choice in almost every case.
• Never call a blocking API — copy_to_user(), GFP_KERNEL allocation, mutex_lock(), msleep() — while holding a spinlock.
• The kernel detects violations through preempt_count tracking, surfaced as a “scheduling while atomic” bug report.
• CONFIG_DEBUG_ATOMIC_SLEEP and lockdep make these bugs easier to catch during development on modern kernels.
• Under PREEMPT_RT on kernel 6.12+, plain spinlocks behave like sleeping locks; use raw_spinlock_t where true atomicity is required.

Conclusion

The spinlock vs mutex decision is one of the first real synchronization judgment calls every kernel and driver developer has to make, and it is also one of the easiest to get subtly wrong. The good news is that the kernel is not silent about it — with debug options enabled, it will tell you exactly where you went wrong, right down to the function and line. Build the buggy driver above, trigger it yourself, read the report, and this rule will never catch you off guard in production code.

Frequently Asked Questions

1. Can I use a spinlock inside an interrupt handler?
Yes — in fact this is one of the main reasons spinlocks exist. Just remember that a mutex can never be used in interrupt context at all.

2. Why not just always use a mutex to be safe?
Mutexes cannot be used in interrupt or other atomic contexts, and their overhead is unnecessary for very short critical sections where a spinlock is both simpler and faster.

3. What exactly does “atomic context” mean?
It means a context where the CPU cannot be preempted or put to sleep — for example, while holding a spinlock, inside an interrupt handler, or with preemption explicitly disabled.

4. Is copy_to_user() always unsafe inside a spinlock?
Yes, treat it as always unsafe. It can fault on user-space memory and potentially block, which violates the spinlock’s atomic-context requirement.

5. What is the difference between spin_lock() and spin_lock_irqsave()?
spin_lock_irqsave() additionally disables local interrupts and saves their previous state, which is required when the same lock might also be acquired from an interrupt handler.

6. Does PREEMPT_RT remove the need for spinlocks entirely?
No. It changes what a regular spinlock_t does internally (making it sleep-capable), but raw_spinlock_t still provides true non-sleeping, atomic-context locking where genuinely needed.

7. How do I enable CONFIG_DEBUG_ATOMIC_SLEEP on my own kernel build?
Enable it under the “Kernel hacking” menu in make menuconfig, alongside CONFIG_PROVE_LOCKING for full lockdep coverage, then rebuild and boot into that kernel for testing.

8. Can a scheduling-while-atomic bug crash a production system?
It can hang or crash the system, especially if it happens inside a lock that many code paths depend on, which is why this must always be tested against before shipping a driver.