Debugging “Scheduling While Atomic” Bugs in Linux Kernel Modules

Free Linux Kernel Programming Course — Kernel Synchronization Series (Kernel 6.x)

Kernel 6.x Updated

Original Debug Demo

Free Linux Device Drivers Course

In the previous lecture of this free Linux kernel development course you learned the rule: never sleep while holding a spinlock. In this lecture we deliberately break that rule in an original demo driver so you can see, first-hand, what sleeping in atomic context in a Linux kernel module actually looks like — the dmesg output, why a debug kernel catches it and a plain distro kernel might not, and how to fix it. Seeing the failure once makes the rule impossible to forget.

scheduling while atomic linux kernel sleep in atomic context linux kernel free linux kernel development course free linux device drivers course free embedded systems course

What Does “Atomic Context” Actually Mean?

A section of kernel code is running in atomic context whenever it must run to completion without ever calling schedule(). This includes: interrupt handlers, softirqs/tasklets, and any region of process context where you are holding a spinlock or have preemption/interrupts explicitly disabled. The scheduler simply refuses to run inside such a region — if something inside that region tries to sleep anyway, the kernel’s own consistency checks (when enabled) will flag it as a bug.

An Original Demo: Breaking the Rule on Purpose

The following module (written for this course, not copied from any book) exposes a module parameter, trigger_bug. When set to 1, the write path deliberately calls a sleeping API while still holding a spinlock, so you can observe the failure safely on a test machine.

static spinlock_t demo_lock;
static int trigger_bug;
module_param(trigger_bug, int, 0644);
MODULE_PARM_DESC(trigger_bug,
    "If 1, sleep while holding a spinlock to demonstrate the bug (test only!)");

static ssize_t ep_write(struct file *filp, const char __user *ubuf,
                         size_t count, loff_t *off)
{
    spin_lock(&demo_lock);

    /* ... normal fast, non-blocking updates would go here ... */

    if (trigger_bug) {
        /* Deliberately broken: this call can sleep, but we are
         * still holding demo_lock, which is atomic context.
         */
        set_current_state(TASK_INTERRUPTIBLE);
        schedule_timeout(HZ / 10);
    }

    spin_unlock(&demo_lock);
    return count;
}

static int __init ep_demo_init(void)
{
    spin_lock_init(&demo_lock);
    pr_info("ep_atomic_bug_demo loaded (trigger_bug=%d)\n", trigger_bug);
    return 0;
}

Load the module, then write to its device node with trigger_bug=1 set, and watch dmesg.

$ sudo insmod ep_atomic_bug_demo.ko trigger_bug=1
$ echo test > /dev/ep_atomic_bug_demo
$ dmesg | tail -n 20

Reading the Failure Report

On a kernel built with the relevant “Kernel Hacking” debug options enabled, this will produce a report that begins with a line similar in spirit to BUG: sleeping function called from invalid context, followed by the process name/PID, the file and line that triggered it, and a full call stack. The report also tells you the current preempt count, which will be non-zero because the spinlock bumped it.

The key function here is might_sleep(), an internal debug check compiled into sleeping APIs when CONFIG_DEBUG_ATOMIC_SLEEP is enabled. It inspects the current preemption count and, if it is non-zero (meaning some lock or explicit preempt-disable is active), prints the warning above before continuing. This option is still present and fully supported on modern kernel 6.x trees, alongside the broader lockdep infrastructure.

Debug Kernel vs. Generic Distro Kernel

Kernel Type	CONFIG_DEBUG_ATOMIC_SLEEP	Observed Behaviour
Custom debug kernel (Kernel Hacking options on)	Enabled	Clear “sleeping function called from invalid context” report with call stack, no silent corruption
Generic production/distro kernel	Usually disabled	No warning printed; the bug can still cause a hang, latency spike, or corruption — it is simply invisible until it does

This is precisely why kernel developers build and test against a debug kernel during development: the same bug that is loudly reported on a debug build can be a silent, intermittent production hang on a generic kernel, discovered only much later and far harder to diagnose.

Fixing the Bug

The fix is always the same shape: move anything that can sleep outside of the spinlock-protected region, or replace the spinlock with a mutex if the whole section genuinely needs to be able to sleep.

/* Fixed version: sleeping call moved out of the atomic region */
static ssize_t ep_write_fixed(struct file *filp, const char __user *ubuf,
                               size_t count, loff_t *off)
{
    spin_lock(&demo_lock);
    /* ... fast, non-blocking updates only ... */
    spin_unlock(&demo_lock);

    if (trigger_bug) {
        set_current_state(TASK_INTERRUPTIBLE);
        schedule_timeout(HZ / 10);   /* now safe: no lock held */
    }

    return count;
}

Common Mistakes

• Testing only on a generic distro kernel and assuming “no warning” means “no bug”.
• Calling kmalloc(..., GFP_KERNEL) inside a spinlock — GFP_KERNEL allocations can sleep; use GFP_ATOMIC if allocation inside a spinlock is unavoidable.
• Taking a mutex while already holding a spinlock — the mutex lock call itself can sleep.
• Assuming a short sleep is “safe enough” — there is no safe amount of sleeping in atomic context.

Best Practices

• Always develop and test against a debug kernel with CONFIG_DEBUG_ATOMIC_SLEEP and lockdep enabled before shipping.
• Keep spinlock-protected regions minimal and audit every function call inside them for hidden sleeping behaviour.
• Use might_sleep() yourself in helper functions that assume a sleepable context, so misuse is caught early.

Performance Considerations

Debug options like CONFIG_DEBUG_ATOMIC_SLEEP and lockdep add measurable overhead, which is why production kernels usually ship without them. The right workflow is to develop and stress-test with these options on, then build the final production kernel with them off — not to skip debug testing entirely.

Security Considerations

An undetected sleep-in-atomic bug is a reliability and availability risk: on a busy system it can escalate from a rare warning into a full CPU stall, which in an embedded or server context is effectively a denial-of-service condition triggered by ordinary use of the driver.

Summary / Key Takeaways

• Atomic context means: no schedule() allowed, ever.
• might_sleep(), enabled via CONFIG_DEBUG_ATOMIC_SLEEP, is what actually catches these bugs at runtime, by checking the preemption count.
• A generic distro kernel without these debug options will not warn you — the bug still exists, it is just invisible until it causes real trouble.
• The fix is always to move sleeping calls outside the atomic region, or switch to a lock type that permits sleeping.

Conclusion

Seeing a “sleeping function called from invalid context” report once, on your own test machine, against your own code, is worth more than reading the rule a dozen times. Build a debug kernel, run the demo above with trigger_bug=1, and study the report it produces — then compare it against what (doesn’t) happen on a plain distro kernel. That contrast is the real lesson of this lecture in the free Linux kernel development course.

Frequently Asked Questions

Continue the Free Linux Kernel Development Course

More free Linux device driver and embedded systems lectures are on the way on EmbeddedPathashala.

Browse the Full Course