What You Will Learn

• Why locking bugs are dangerous and why kernel lock debugging tools exist
• The core Kconfig options that make up a lock-debugging-ready kernel on 6.12 and newer
• How lockdep proves locking correctness without needing a deadlock to actually happen
• Step-by-step instructions to configure and build a debug kernel for lock debugging
• Common configuration mistakes and how to avoid them
• Performance trade-offs of running a lock-debug kernel

Prerequisites

Before starting this lecture, you should be comfortable with:

• Building a mainline Linux kernel from source (make menuconfig, make, make modules_install)
• Basic kernel synchronization primitives: spinlocks, mutexes, and reader-writer semaphores
• Working inside a disposable virtual machine, since a debug kernel should never be your first stop on production hardware

Why Kernel Lock Debugging Matters in Modern Development

Every time a driver or subsystem takes more than one lock, there is a chance of introducing a deadlock: two code paths each holding one lock while waiting on the other. These bugs rarely show up during quick testing because they depend on timing, CPU count, and interrupt patterns that only line up occasionally. A bug like this can sit in a driver for months before a customer hits it on a sixteen-core server under heavy load.

The mainline kernel solves this with a runtime lock validator commonly known as lockdep. Rather than waiting for an actual deadlock, lockdep tracks the order in which every lock in the system is acquired and builds a dependency graph as your code runs. The moment it spots an ordering that could theoretically deadlock, it prints a detailed warning, even if the unlucky timing that would trigger the real deadlock never occurs. That is the heart of kernel lock debugging, and it is one of the most valuable tools available to anyone building drivers or kernel modules.

Kernel Lock Debugging Options You Need on 6.12 and Later

The debug options live under Kernel hacking > Lock Debugging (spinlocks, mutexes, etc...) in menuconfig, the same location the option has occupied for many kernel generations. The table below lists what to enable and why, with notes on how things have shifted now that PREEMPT_RT is merged into mainline as of kernel 6.12.

Understanding How Lockdep Proves Locking Correctness

Lockdep does not simulate your entire system. Instead, it watches every lock acquisition and release that actually happens while your kernel runs, and it groups locks into “lock classes” based on where they are initialized in source code, not by their individual instance in memory. Every time two lock classes are taken back to back, lockdep records that ordering as an edge in a dependency graph. If it ever observes an edge that would create a cycle, such as class A being taken before class B in one code path and class B before class A in another, it immediately reports a possible deadlock.

This is why lockdep is so valuable during development: it can flag a possible deadlock the very first time both code paths run in the same boot session, long before a real system would ever hang. It also tracks interrupt context, so it can catch a lock taken safely in process context but unsafely from an interrupt handler.

Step-by-Step: Building a Debug Kernel for Lock Debugging

Follow these steps inside a kernel source tree you have already configured for your target architecture.

# 1. Start from your existing config or a fresh defconfig
make defconfig

# 2. Open the interactive configuration menu
make menuconfig
# Navigate to: Kernel hacking -> Lock Debugging (spinlocks, mutexes, etc...)
# Enable: PROVE_LOCKING, LOCK_STAT, DEBUG_SPINLOCK,
#         DEBUG_MUTEXES, DEBUG_RWSEMS, DEBUG_RT_MUTEXES

# 3. Confirm the options landed in your .config
grep -E "PROVE_LOCKING|LOCK_STAT|DEBUG_SPINLOCK|DEBUG_MUTEXES" .config

# 4. Build the kernel and modules
make -j$(nproc)
make modules

# 5. Install and boot into the debug kernel inside your test VM
sudo make modules_install
sudo make install

After rebooting, confirm lockdep is active by checking the proc entry it exposes:

cat /proc/lockdep_stats | head -n 5

A populated output with non-zero counts of lock classes confirms lockdep is tracking your system’s locks correctly.

Common Mistakes When Configuring a Lock Debug Kernel

• Forgetting CONFIG_DEBUG_KERNEL. Several of these options are hidden until this parent option is enabled first.
• Testing on production hardware. A lock-debug kernel is significantly slower and should only run in a VM or dedicated test rig.
• Enabling only CONFIG_PROVE_LOCKING and skipping CONFIG_DEBUG_SPINLOCK. Without the basic checks, some categories of bugs such as use of an uninitialized lock go unnoticed.
• Ignoring the first lockdep warning in the log. Once lockdep hits an internal inconsistency, it disables itself for the rest of the boot, so later locking bugs go unreported until reboot.

Best Practices for Lock Debug Kernel Builds

• Keep a dedicated debug .config file in your driver repository so every contributor tests against the same lock debugging setup
• Pair lock debugging with a serial console or netconsole so you never lose a splat when the screen scrolls past it
• Re-run your full driver test suite after every merge with lock debugging enabled, not just before a release
• Combine lock debugging with KASAN in CI pipelines, since memory corruption often manifests as impossible lock states

Performance Considerations

CONFIG_PROVE_LOCKING adds meaningful overhead because every single lock acquisition and release now updates an in-memory dependency graph. Expect noticeably lower throughput and higher latency on a lock-debug kernel, which is completely expected and by design. Never ship a lock-debug kernel to production or use it for performance benchmarking; its entire purpose is catching bugs during development, not speed.

Summary and Key Takeaways

• Kernel lock debugging catches deadlocks and locking misuse before they cause real hangs
• Lockdep proves locking correctness by tracking lock ordering across every code path that executes
• CONFIG_PROVE_LOCKING, CONFIG_DEBUG_SPINLOCK, CONFIG_DEBUG_MUTEXES, CONFIG_DEBUG_RWSEMS, and CONFIG_DEBUG_RT_MUTEXES together form a solid lock-debug configuration on kernel 6.12 and newer
• RT-mutex debugging is more important than ever now that mainline PREEMPT_RT is part of the kernel
• A lock-debug kernel belongs in a test VM, never in production

Conclusion

Setting up a lock-debugging kernel takes only a few minutes, but the bugs it will catch could otherwise take weeks to reproduce and diagnose. As part of this free Linux kernel development course, treat lock debugging as a standard part of your driver development loop rather than something you reach for only after a hang. In the next lecture of this free Linux device drivers course, we will look at how to actually read a lockdep warning line by line, dig into lock statistics, and pair lockdep with KASAN for a complete debugging workflow.

Frequently Asked Questions

Q1. Does lockdep only detect deadlocks that have already happened?
No. Lockdep proves that a deadlock is possible based on lock ordering it has observed, and reports it immediately, even if the exact interleaving needed to trigger the real deadlock never occurs during that boot.

Q2. Will a lock-debug kernel work on my production server?
Technically yes, but it should not run there. The overhead is too high for production workloads, and it is intended purely as a development and testing tool.

Q3. Do I need CONFIG_PROVE_LOCKING and CONFIG_DEBUG_SPINLOCK together?
Yes, for the most complete coverage. CONFIG_PROVE_LOCKING catches ordering issues, while CONFIG_DEBUG_SPINLOCK catches basic misuse such as uninitialized locks.

Q4. Is rt-mutex debugging relevant if I am not using PREEMPT_RT?
It is still useful for real-time mutex APIs used elsewhere in the kernel, but it becomes essential once you build a PREEMPT_RT-enabled kernel, since ordinary spinlocks turn into sleeping rt-mutexes under the hood.

Q5. What happens if lockdep detects a problem?
It prints a detailed warning to the kernel log describing the conflicting lock classes and the code paths involved, which we will learn to read in detail in the next lecture.

Q6. Can I enable lock debugging on an already-built kernel without recompiling?
No. These are compile-time Kconfig options, so you must reconfigure and rebuild the kernel to enable them.

Q7. Does lock debugging work with kernel modules I build separately?
Yes. As long as the running kernel has lockdep enabled, any locks initialized and used inside your out-of-tree module are tracked the same way as in-tree locks.