← Previous Lecture | Next Lecture →

Understanding Linux Kernel Lockdep Issues and Lock Statistics

Why Lockdep Itself Can Misbehave

Lockdep is a static analysis engine built into the kernel, and like any analysis engine it has resource limits. The most common of the linux kernel lockdep issues developers run into during driver development is running out of lock classes. Every time you insmod a kernel module, the kernel allocates a fresh set of lock classes for every lock defined inside that module. Unloading the module with rmmod does not free those classes — the kernel keeps them around so it can reuse the tracking data if the same module is loaded again later. Reload the same module a few dozen times during a debugging session, and you can quietly exhaust lockdep’s internal lock class table.

The practical fix is simple: avoid tight insmod/rmmod cycles once you notice lockdep behaving oddly, and reboot the test machine between long debugging sessions instead of repeatedly reloading. This is especially relevant for driver code with large arrays of embedded locks, such as a per-descriptor spinlock inside a ring buffer structure — a single array-heavy driver can consume a large chunk of the class table on one load.

The “Lock Debugging Disabled” Warning

A second common source of confusion is this message appearing in dmesg:

WARNING: lock debugging disabled!! - possibly due to a lockdep warning

This is not a new bug — it is lockdep telling you it already gave up earlier. Internally the kernel keeps a flag, debug_locks, which gets forced to zero the moment lockdep detects something it considers unrecoverable, even on a fully debug-enabled kernel. Once that flag flips, all further lock debugging output is suppressed, which means a completely unrelated warning later in your session may really be a symptom of an earlier lockdep failure you scrolled past. The correct response is to scroll back through dmesg for the first lockdep warning, fix that, and reboot before trusting any subsequent lock debugging output.

KCSAN as a Complementary Data-Race Detector

Lockdep proves things about lock ordering, but it does not directly catch plain data races on unprotected memory. The Kernel Concurrency Sanitizer (KCSAN) fills that gap using compile-time instrumentation, and by kernel 6.x it has matured into a standard, well-supported option rather than an experimental patch. If you are chasing intermittent memory corruption that lockdep says nothing about, building a kernel with KCSAN enabled is usually the next step, since it can catch races that never manifest as a classic self-deadlock or AB-BA lock ordering violation.

Enabling Kernel Lock Statistics

Once your locking is passing lockdep cleanly, the next question is usually a performance one: which lock is actually slowing my driver down under load? This is where kernel lock statistics come in. Lockdep already inserts hooks into every lock acquire and release path, and the lock statistics subsystem reuses those same hooks to record timing data instead of just correctness data.

To use this feature your kernel must be built with CONFIG_LOCK_STAT=y. Without it, /proc/lock_stat simply will not exist, which is the case on most stock distribution kernels — you generally need a custom debug build for this, the same kind of build most of this course’s synchronization lectures assume.

Reading /proc/lock_stat Fields

The output columns look intimidating the first time you see them, but each one maps to a specific moment in a lock’s life cycle. Understanding these fields is the real payoff of this section on linux kernel lock statistics.

A lock with high contentions and a large holdtime-avg is your prime suspect for a scalability bottleneck — it is both popular and slow, which is the worst combination for a multi-core system. A lock with high con-bounces specifically points at cacheline ping-pong between CPU cores, a topic we covered in more depth in this course’s false-sharing lecture.

Hands-On: An Original Lock Statistics Demo

Below is an original demo driver, ep_lockstat_demo, written for this course. It deliberately creates contention on a single spinlock by running two kernel threads that both hammer a shared counter, so you have something worth measuring in /proc/lock_stat.

#include <linux/module.h>
#include <linux/kthread.h>
#include <linux/spinlock.h>
#include <linux/delay.h>

static DEFINE_SPINLOCK(ep_counter_lock);
static unsigned long ep_shared_counter;
static struct task_struct *ep_worker_a, *ep_worker_b;

static int ep_worker_fn(void *data)
{
	while (!kthread_should_stop()) {
		spin_lock(&ep_counter_lock);
		ep_shared_counter++;
		udelay(50); /* hold the lock briefly to force contention */
		spin_unlock(&ep_counter_lock);
		usleep_range(100, 200);
	}
	return 0;
}

static int __init ep_lockstat_demo_init(void)
{
	ep_worker_a = kthread_run(ep_worker_fn, NULL, "ep_lockstat_a");
	ep_worker_b = kthread_run(ep_worker_fn, NULL, "ep_lockstat_b");
	pr_info("ep_lockstat_demo: two workers contending on ep_counter_lock\n");
	return 0;
}

static void __exit ep_lockstat_demo_exit(void)
{
	kthread_stop(ep_worker_a);
	kthread_stop(ep_worker_b);
	pr_info("ep_lockstat_demo: stopped\n");
}

module_init(ep_lockstat_demo_init);
module_exit(ep_lockstat_demo_exit);
MODULE_LICENSE("GPL");
MODULE_DESCRIPTION("EmbeddedPathashala lock statistics demo");

With CONFIG_LOCK_STAT enabled, load the module, let it run for a few seconds, then filter the output for just this lock:

sudo insmod ep_lockstat_demo.ko
sleep 5
sudo grep -A2 "ep_counter_lock" /proc/lock_stat
sudo rmmod ep_lockstat_demo

Because both worker threads hold the spinlock for roughly 50 microseconds on every iteration, you should see non-zero contentions and a measurable holdtime-avg close to that value — a concrete, self-generated example instead of an abstract one.

Beyond Lockdep: Other Concurrency Debugging Tools

Kernel-space lockdep and lock statistics are powerful, but they are not the only tools worth knowing. Userspace equivalents include Helgrind from the Valgrind suite and ThreadSanitizer (TSan), both of which instrument multithreaded applications for data races the same way KCSAN does for the kernel. Lockdep itself even has a userspace port for library developers who want the same deadlock-ordering proofs outside the kernel. On the tracing side, the modern eBPF-based deadlock-bpfcc tool from the BCC toolkit can attach to a running process or thread and flag potential lock order inversions without needing a special debug kernel at all — a handy option when you cannot rebuild the kernel on a production-like system.

Best Practices for Lock Debugging

• Keep a dedicated debug kernel build with CONFIG_PROVE_LOCKING and CONFIG_LOCK_STAT for development, separate from anything resembling production.
• Reboot between heavy insmod/rmmod debugging sessions to avoid silent lockdep lock class exhaustion.
• Treat the first lockdep warning in dmesg as authoritative; ignore or disbelieve anything that appears after debug_locks has already flipped to zero.
• Clear /proc/lock_stat before each measurement run so old numbers do not skew your analysis.
• Correlate high con-bounces with your system’s actual CPU topology before assuming it is a software bug.

Common Mistakes and Troubleshooting

Summary and Key Takeaways

• Lockdep allocates lock classes per module load and never frees them on unload, which can exhaust its internal table during heavy debugging sessions.
• The “lock debugging disabled” warning means lockdep already gave up earlier — trust the first warning in dmesg, not the latest one.
• KCSAN is the modern complement to lockdep, catching plain data races that lock-ordering analysis alone cannot see.
• CONFIG_LOCK_STAT exposes real per-lock timing data through /proc/lock_stat, letting you find genuinely contended locks instead of guessing.
• con-bounces, waittime and holdtime together tell you whether a lock is a correctness problem, a performance problem, or both.

Conclusion

Lockdep and kernel lock statistics are two sides of the same debugging hook: one proves your locking is ordered correctly, the other tells you whether it is fast enough. Knowing the common linux kernel lockdep issues ahead of time — lock class exhaustion, the disabled-debugging warning — means you will spend less time debugging the debugger, and more time using /proc/lock_stat for what it is genuinely good at: finding the one contended lock that is quietly capping your driver’s throughput on a multi-core system.