What You Will Learn

• What “lock contention” actually means and why it matters for performance
• How to enable, clear, and read the kernel’s /proc/lock_stat facility
• What each column in a lock stats report tells you
• How to reproduce a simple contention scenario for practice
• Modern alternatives available on 6.12+ kernels using eBPF-based tracing

Prerequisites

You should already be comfortable with basic spinlocks and mutexes, and it helps to have gone through the earlier lecture in this free linux device drivers course on lockdep annotations, since lock statistics reuse the same underlying lockdep hooks.

What Is Lock Contention?

A lock is contended the moment a thread or interrupt context wants to acquire it but finds it already held by someone else, forcing it to wait. Occasional contention is normal and harmless. Heavy, sustained contention on a specific lock is a red flag — it means multiple CPUs are effectively serializing on that one piece of code, which caps how much your workload can scale no matter how many cores you throw at it.

The Linux kernel ships a dedicated facility for measuring exactly this, built on top of the same lockdep infrastructure used for correctness checking.

Enabling Lock Statistics

Lock statistics live behind the CONFIG_LOCK_STAT kernel configuration option. Without it enabled in your kernel build, the /proc/lock_stat file simply won’t exist — which is the normal state on most distribution kernels, since this is a debug feature, not something shipped for general use.

Once you have a kernel built with CONFIG_LOCK_STAT=y, three simple commands control the facility:

# Clear existing lock statistics
sudo sh -c "echo 0 > /proc/lock_stat"

# Enable lock statistics collection
sudo sh -c "echo 1 > /proc/sys/kernel/lock_stat"

# Disable lock statistics collection
sudo sh -c "echo 0 > /proc/sys/kernel/lock_stat"

A Simple Reproducible Demo

Here’s a minimal shell workflow you can run yourself to see the facility in action, using nothing more exotic than cat to trigger some kernel-internal locking:

#!/bin/bash
# lock_stats_walkthrough.sh — simple lock stats demo

echo 0 | sudo tee /proc/lock_stat > /dev/null       # clear stats
echo 1 | sudo tee /proc/sys/kernel/lock_stat > /dev/null   # enable

# Trigger some kernel locking activity
cat /proc/self/cmdline > /dev/null

echo 0 | sudo tee /proc/sys/kernel/lock_stat > /dev/null   # disable

# Filter for a few interesting lock classes
grep -E "alloc_lock|task|mm" /proc/lock_stat

Running cat /proc/self/cmdline quietly walks through several shared kernel data structures under the hood — inside the process filesystem code — and each of those touch-points acquires and releases locks, which is exactly what gets recorded.

Reading the Lock Stats Output

/proc/lock_stat reports several columns per lock class. Here’s what each one means:

As a rule of thumb: a lock class with a high contentions count relative to its acquisitions, combined with a large waittime-total, is your prime candidate for a real scalability bottleneck worth investigating further.

Lock Statistics on Modern 6.12+ Kernels

The CONFIG_LOCK_STAT facility and its /proc/lock_stat interface still work exactly the same way on current LTS kernels — nothing about the mechanism itself has changed. What has expanded is the toolbox around it:

• perf lock — the perf lock record / perf lock report subcommands wrap much of this same lockdep instrumentation into a more ergonomic, sortable report, and are the more commonly reached-for option on modern distros.
• BPF-based tracing — tools built on bpftrace or BCC can hook lock-related tracepoints and kprobes to build custom, live contention dashboards without needing a special CONFIG_LOCK_STAT debug kernel at all in some cases.
• Tracepoints — the kernel’s lock tracepoints (where available) let you correlate lock contention with other system events in a single trace, which the old flat /proc/lock_stat table can’t do on its own.

For quick, no-dependency investigation on a debug kernel you already control, /proc/lock_stat remains genuinely useful. For production-adjacent, low-overhead investigation, perf lock or BPF-based tracing is the more modern default.

Real-World Use Case

A storage driver team notices throughput plateaus well below expected disk bandwidth once request queue depth increases. Rather than guessing, they enable CONFIG_LOCK_STAT on a test kernel, run their benchmark, and inspect /proc/lock_stat. One queue lock shows a contention count almost equal to its acquisition count — nearly every acquisition is waiting. That single number redirects the investigation from “maybe it’s the disk” to “it’s actually a single coarse-grained lock serializing all queue submissions,” which is a completely different — and much more fixable — problem.

Best Practices

• Always clear stats (echo 0 > /proc/lock_stat) before a fresh measurement run, so old data doesn’t skew your numbers.
• Measure under realistic load — lock contention that doesn’t show up under light load can be severe under real production traffic.
• Cross-check high-contention locks against waittime-total, not just raw contention count, since a lock contended often but held very briefly may matter less than one contended rarely but held a long time.
• On modern kernels, reach for perf lock report first for a quicker top-down view before diving into raw /proc/lock_stat output.

Performance Considerations

Like lockdep itself, lock statistics collection adds overhead to every lock operation while active, so it should be treated strictly as a debug-time facility — enable it, collect your data, then disable it again rather than leaving it running continuously.

Security Considerations

/proc/lock_stat exposes internal kernel structure names and timing behavior, which is diagnostic information best kept off of any system with untrusted local users, and access should be limited to root on controlled development and test machines.

Common Mistakes & Troubleshooting

Summary / Key Takeaways

• Lock contention happens when a waiter finds a lock already held, and heavy contention caps real-world scalability.
• CONFIG_LOCK_STAT plus /proc/lock_stat gives you a built-in, no-extra-tooling way to measure exactly which locks are contended and by how much.
• Key columns to focus on are contentions, waittime-total, and holdtime-total.
• On modern 6.12+ kernels, perf lock and BPF-based tracing are the more commonly used complements to the classic /proc/lock_stat approach.

Conclusion

Lock statistics turn a vague “the system feels slow under load” complaint into a concrete, measurable answer pointing at a specific lock. Whether you reach for the classic /proc/lock_stat interface or a modern perf lock / BPF-based workflow, the underlying question is the same: which lock is everyone waiting on, and why. That’s a core diagnostic skill for anyone doing serious kernel or driver performance work, and it rounds out this pair of lectures in our free Linux kernel development course on lockdep and locking diagnostics.

Frequently Asked Questions

Q1. What is lock contention in the Linux kernel?
Lock contention happens when a thread or interrupt context tries to acquire a lock that is already held by another context, forcing it to wait until the lock is released.

Q2. How do I enable kernel lock statistics?
Build a kernel with CONFIG_LOCK_STAT=y, then enable collection with echo 1 > /proc/sys/kernel/lock_stat and read results from /proc/lock_stat.

Q3. Why doesn’t /proc/lock_stat exist on my system?
Most distribution kernels don’t enable CONFIG_LOCK_STAT by default since it’s a debug-only feature with runtime overhead, so you’d need a custom or debug kernel build to use it.

Q4. What’s the difference between contentions and acquisitions in the report?
Acquisitions count every successful lock grab, while contentions count only the subset of attempts where the caller had to wait because the lock was already held.

Q5. Is /proc/lock_stat still relevant on modern 6.12+ kernels?
Yes, the mechanism is unchanged, though many developers now prefer perf lock or BPF-based tracing tools for a more ergonomic view of the same underlying data.

Q6. Does enabling lock statistics slow down the system?
Yes, it adds measurable overhead to lock operations while active, so it should only be enabled during a focused debugging session, not left running continuously.

Q7. What should I look at first in a lock stats report?
Start with locks that have both a high contentions count and a large waittime-total, since those combine frequency and real wait-time cost into the strongest signal of an actual bottleneck.