← Previous Lecture  |  Next Lecture →

What You Will Learn

• Why lockdep is able to mathematically prove a lock sequence is deadlock-free
• The difference between a lock class and a lock instance, and why it matters for performance
• How to build and boot a debug kernel with lockdep enabled and verify it is active
• How to write an original demo driver that triggers a genuine self-deadlock
• How to read the warning lockdep produces and what each part of it means
• User-space tools (ps, strace, ltrace, journalctl, kdump) for chasing a hung kernel thread when lockdep hasn’t caught the bug yet
• Best practices, common mistakes, and performance/security trade-offs around lock debugging

Prerequisites

• Comfort building and loading kernel modules on a Linux 6.x kernel (preferably in a VM)
• Basic understanding of spinlocks and mutexes (covered in earlier lectures in this series)
• A debug kernel build, or the willingness to configure one (we show you how below)

How Lockdep Proves Locking Correctness

Most bugs in software are found by testing: you run the code enough times, in enough conditions, and hope you exercise the buggy path. Deadlocks are notoriously bad at showing up this way — a locking bug might work perfectly for months and then hang production hardware the one time two code paths interleave in just the wrong order. This is exactly the gap that catching deadlock bugs with lockdep is designed to close.

Lockdep does not wait for the unlucky interleaving to actually occur. Instead, every time your code acquires a lock, lockdep records the acquisition order relative to every other lock that was already held at that moment. Over the lifetime of the running kernel, it builds up a graph of “lock A was taken while lock B was held.” From this graph it can derive, using ordinary graph-cycle analysis, whether any two locks could ever be acquired in conflicting orders on different code paths. If a cycle is possible, a deadlock is possible — regardless of whether the unlucky timing has ever actually happened on your machine. That is what makes lock validation with lockdep so much stronger than manual stress testing.

Lockdep watches for several distinct classes of bugs, not only plain deadlocks:

Lock Classes vs Lock Instances

Understanding lock classes is a core part of catching deadlock bugs with lockdep efficiently. A running kernel can easily have tens of thousands of live lock instances — every open file, every socket, every task_struct carries its own private lock. If lockdep tracked each instance separately, the validation cost would grow quadratically with the number of instances, which would make a debug kernel unusably slow. Lockdep sidesteps this by tracking a lock class instead of a lock instance: all mutexes created from the same line of driver code, or all spinlocks embedded in the same structure definition, are treated as one logical lock for validation purposes. A structure such as the kernel’s open-file object typically embeds more than one lock field, and each field is its own class — but every file descriptor in the system shares that same small set of classes. Lockdep computes a hash for each unique acquisition chain and only re-validates a chain the first time it is seen, which is what keeps the overhead manageable even under heavy load. You can inspect every chain lockdep has recorded through /proc/lockdep_chains on a debug kernel.

Case Study: Catching a Self-Deadlock Bug with Lockdep

Theory is useful, but the real value of catching deadlock bugs with lockdep shows up when you watch it happen. Below is an original demo driver, written for kernel 6.x, that recreates a very common real-world mistake: calling a helper function that internally takes a lock, while you are already holding that same lock yourself.

Step 1: Confirm Your Debug Kernel Has Lockdep Enabled

Before testing, verify you are running a kernel built with CONFIG_PROVE_LOCKING enabled (covered in detail in the previous lecture on configuring a debug kernel):

$ uname -r
6.8.0-ep-dbg
$ grep PROVE_LOCKING /boot/config-$(uname -r)
CONFIG_PROVE_LOCKING=y

Step 2: An Original Driver That Self-Deadlocks

This demo module maintains a small status record protected by a spinlock. The catch: the “report” helper takes the very same spinlock internally to safely read the record — a completely normal and safe pattern on its own. The bug appears only when a second function takes the lock first, and then calls the helper while still holding it.

#include <linux/module.h>
#include <linux/spinlock.h>
#include <linux/miscdevice.h>
#include <linux/fs.h>

struct ep_status_record {
    spinlock_t lock;
    int        health_code;
    int        last_updated_by;
};

static struct ep_status_record ep_rec;

/* Safe on its own: takes ep_rec.lock, reads the fields, releases it */
static void ep_report_status(int caller_id)
{
    unsigned long flags;

    spin_lock_irqsave(&ep_rec.lock, flags);
    pr_info("ep_lockdep_demo: health=%d last_updated_by=%d (queried by %d)\n",
            ep_rec.health_code, ep_rec.last_updated_by, caller_id);
    spin_unlock_irqrestore(&ep_rec.lock, flags);
}

/* BUGGY: takes the lock, then calls a helper that takes it again */
static void ep_update_status_buggy(int new_code, int caller_id)
{
    unsigned long flags;

    spin_lock_irqsave(&ep_rec.lock, flags);
    ep_rec.health_code      = new_code;
    ep_rec.last_updated_by  = caller_id;

    /* Self-deadlock: ep_report_status() tries to re-acquire ep_rec.lock
     * while this function is still holding it. */
    ep_report_status(caller_id);

    spin_unlock_irqrestore(&ep_rec.lock, flags);
}

static int ep_demo_open(struct inode *inode, struct file *filp)
{
    ep_update_status_buggy(1, current->pid);
    return 0;
}

static const struct file_operations ep_demo_fops = {
    .owner = THIS_MODULE,
    .open  = ep_demo_open,
};

static struct miscdevice ep_demo_dev = {
    .minor = MISC_DYNAMIC_MINOR,
    .name  = "ep_lockdep_demo",
    .fops  = &ep_demo_fops,
};

static int __init ep_lockdep_demo_init(void)
{
    spin_lock_init(&ep_rec.lock);
    return misc_register(&ep_demo_dev);
}

static void __exit ep_lockdep_demo_exit(void)
{
    misc_deregister(&ep_demo_dev);
}

module_init(ep_lockdep_demo_init);
module_exit(ep_lockdep_demo_exit);
MODULE_LICENSE("GPL");
MODULE_DESCRIPTION("EmbeddedPathashala original self-deadlock demo for lockdep");

Step 3: Trigger It and Read the Lockdep Warning

Load the module and open the device node once to trigger the buggy path:

$ sudo insmod ep_lockdep_demo.ko
$ cat /dev/ep_lockdep_demo
$ sudo dmesg | tail -40

On a lockdep-enabled kernel, this never actually hangs the machine the way it would on a production kernel — lockdep recognizes the same lock class being re-acquired by the same task and immediately prints a warning instead of letting the CPU spin forever. A typical report looks roughly like this (paraphrased structure, not literal kernel output):

WARNING: possible recursive locking detected
--------------------------------------------
cat/1842 is trying to acquire lock:
 ep_rec.lock, at: ep_report_status+0x1a/0x60 [ep_lockdep_demo]

but task is already holding lock:
 ep_rec.lock, at: ep_update_status_buggy+0x22/0x70 [ep_lockdep_demo]

other info that might help you debug this:
 Possible unsafe locking scenario:

       CPU0
       ----
  lock(ep_rec.lock);
  lock(ep_rec.lock);

 *** DEADLOCK ***

Notice that lockdep names the exact function and offset where each acquisition happened, so you can walk straight to the offending call site instead of guessing from a frozen console.

The fix is straightforward once lockdep points it out: split the helper into a locking wrapper and a lock-free inner function, and call the inner, non-locking version from any code path that already holds the lock.

/* Fixed: inner function assumes the caller already holds the lock */
static void ep_report_status_locked(int caller_id)
{
    pr_info("ep_lockdep_demo: health=%d last_updated_by=%d (queried by %d)\n",
            ep_rec.health_code, ep_rec.last_updated_by, caller_id);
}

static void ep_report_status(int caller_id)
{
    unsigned long flags;

    spin_lock_irqsave(&ep_rec.lock, flags);
    ep_report_status_locked(caller_id);
    spin_unlock_irqrestore(&ep_rec.lock, flags);
}

static void ep_update_status_fixed(int new_code, int caller_id)
{
    unsigned long flags;

    spin_lock_irqsave(&ep_rec.lock, flags);
    ep_rec.health_code     = new_code;
    ep_rec.last_updated_by = caller_id;
    ep_report_status_locked(caller_id);   /* no re-lock */
    spin_unlock_irqrestore(&ep_rec.lock, flags);
}

A note on modern kernels: this exact “lock inside a helper, called again from a path already holding it” mistake is a well-known historical trap around task-name accessor functions in the process-management code. Recent kernel development (2024 onward) has actually been moving to strip unnecessary internal locking out of some of those read-side accessors, since it was found to add overhead without adding real protection for readers. The lesson for you as a driver author does not change either way: never assume a helper function is lock-free just because it looks like a simple getter — always check whether it takes a lock internally before calling it from inside your own critical section.

Real-World Use Case: Debugging a Hung Kernel Module

Not every deadlock will be conveniently caught by lockdep, especially if you are testing on a production kernel without CONFIG_PROVE_LOCKING. When a system hangs and you suspect a stuck kernel thread, these user-space tools help you narrow it down:

$ ps -LA -o state,pid,cmd | grep "^D"

Any thread reported in state D (uninterruptible sleep) has been stuck for a while, which is consistent with — though not proof of — a deadlock. If it persists for a long time, suspicion should rise considerably. Note this only works with GNU ps; lightweight variants such as BusyBox’s ps do not support this option set.

$ sudo strace -p <PID>
$ sudo ltrace -p <PID>

strace and ltrace show you the last system call or library call a process made before it froze, which often points directly at the blocking operation. If the whole VM becomes unresponsive before you can capture anything, reboot and check the saved kernel log with journalctl:

$ journalctl --since="1 hour ago"

For the toughest hangs, configuring kdump ahead of time so you can capture a full crash image and analyze it afterward with the crash utility is the most reliable approach — postmortem analysis beats guessing from a frozen terminal every time.

Best Practices for Lockdep and Lock Debugging

These habits make catching deadlock bugs with lockdep a routine part of your development workflow rather than a last resort:

• Always develop and test new locking code on a lockdep-enabled debug kernel before it ever reaches a production build
• Never call a function you did not write from inside a critical section without first checking whether it takes a lock internally
• Split “locking wrapper” and “assumes lock is held” logic into two clearly named functions, as shown in the fix above
• Treat every lockdep warning as a real bug until proven otherwise — false positives exist but are rare
• Keep debug-kernel test runs in your CI pipeline so lockdep exercises your driver’s code paths automatically on every change

Performance and Security Considerations

Lockdep’s per-class, hash-based chain caching is what keeps the overhead of catching deadlock bugs with lockdep acceptable on a debug kernel — without it, the validation cost would scale quadratically with the number of live lock instances, which is not viable on a busy system. Even so, lockdep instrumentation is not free: it adds noticeable CPU and memory overhead, which is exactly why production kernels normally ship with CONFIG_PROVE_LOCKING disabled. From a security and reliability standpoint, an undetected deadlock is effectively a denial-of-service bug — a hung kernel thread can wedge an entire subsystem or the whole machine. Running your test and staging environments on a debug kernel is a cheap way to catch this class of bug long before it becomes a field outage.

Common Mistakes

Summary and Key Takeaways

• Lockdep proves locking correctness by graph analysis of recorded acquisition order, not by waiting for the bad timing to occur
• It tracks lock classes, not lock instances, which is what keeps validation overhead manageable at scale
• A self-deadlock often hides inside an innocent-looking helper function that locks internally
• When lockdep is not available, ps/strace/ltrace/journalctl/kdump are your fallback toolkit for a hung kernel
• Fix self-deadlocks by separating the “acquires the lock” function from the “assumes the lock is held” function

Conclusion

Catching deadlock bugs with lockdep turns one of the hardest classes of kernel bugs — the kind that hides for months and then wedges a device in the field — into something you can find on your very first test run. By understanding lock classes, building a debug kernel, and knowing how to read a lockdep warning, you can catch self-deadlocks, lock inversions, and IRQ-unsafe locking long before they ever reach production. In the next lecture in this Kernel Synchronization series, we continue with more advanced lockdep scenarios and additional locking primitives.

Frequently Asked Questions

← Previous Lecture  |  Next Lecture →