What You Will Learn

• What lock contention is and why it matters for embedded and systems performance
• How to enable the kernel’s lock statistics subsystem safely
• How to read and interpret every column of the /proc/lock_stat report
• How to reset counters and isolate contention during a specific test window
• Modern alternatives — bpftrace and perf lock — for kernels where lock_stat is disabled
• Common mistakes engineers make when profiling locks, and how to avoid them

Prerequisites

• Basic familiarity with spinlocks and mutexes in the Linux kernel
• A kernel you can rebuild, or root access to a distro kernel with debug symbols
• Comfort working at the shell as root (all commands below need sudo)

Why Lock Contention Matters

Every time two CPU cores race to grab the same spinlock or mutex, one of them has to wait. Occasional waiting is normal and harmless. But when a lock is grabbed thousands of times a second from multiple cores, that waiting adds up into real, measurable latency — the kind that shows up as dropped audio frames, missed real-time deadlines, or a network stack that can’t keep up with line rate. The kernel’s lock statistics engine exists precisely to make this invisible cost visible.

Enabling Lock Statistics

Lock statistics ride on top of the kernel’s lock validator (lockdep), so both need to be enabled in your kernel configuration before you can collect anything.

CONFIG_DEBUG_KERNEL=y
CONFIG_LOCKDEP=y
CONFIG_LOCK_STAT=y
CONFIG_DEBUG_LOCK_ALLOC=y

Once you boot a kernel built with these options, a control file appears that lets you turn tracking on and off without rebooting:

echo 1 | sudo tee /proc/sys/kernel/lock_stat

Reading the /proc/lock_stat Report

With statistics enabled, run your workload for a few seconds and then dump the report:

sudo cat /proc/lock_stat | head -40

Each row describes one lock class — not one lock instance, but the type of lock (for example, every instance of a particular spinlock in a driver is grouped together). The table below explains the columns you’ll see, in plain language rather than kernel jargon:

Rule of thumb: a lock with high acquisitions but low waittime is simply busy, not a problem. A lock with even a modest acquisition count but a high average waittime or holdtime is the one worth investigating first.

Resetting Counters Before a Test

Lock statistics accumulate from the moment they’re enabled, so stale numbers from boot-time activity can hide the pattern you actually care about. Clear the table immediately before running your test:

sudo sh -c 'echo 0 > /proc/lock_stat'
# ... run your workload here ...
sudo cat /proc/lock_stat

Modern Alternative: Tracing Lock Contention with bpftrace

On distro kernels shipped without CONFIG_LOCK_STAT — which is common in production 6.12+ builds because lockdep instrumentation carries a measurable performance cost — you can still observe contention using eBPF-based tracing instead of rebuilding your kernel:

sudo bpftrace -e '
kprobe:queued_spin_lock_slowpath
{
    @contended[comm] = count();
}
interval:s:5 { print(@contended); exit(); }
'

This one-liner hooks the slow-path entry point that spinlocks fall into only when contended, so every hit it records is genuine contention — a good first sanity check before you commit to a full lockdep rebuild.

Common Mistakes

• Leaving lockdep enabled in production — lock_stat is a debugging aid; disable it before shipping, as the tracking overhead affects real-world latency.
• Comparing raw acquisition counts across different workloads — always normalize by test duration.
• Ignoring holdtime — engineers often stare only at waittime and miss that a lock held too long is the actual cause.
• Forgetting to reset counters — old boot-time noise skews short test runs.

Best Practices

• Always reset the lock_stat table right before a controlled test run.
• Cross-check suspicious locks with bpftrace or perf lock record to confirm the finding on a production-like kernel.
• Fix contention by shrinking critical sections first — reducing holdtime is usually cheaper than redesigning the locking scheme.

Performance and Security Considerations

Lockdep-based tracking adds real overhead to every lock acquisition on the system, so treat it strictly as a development/debug-kernel feature. From a security standpoint, /proc/lock_stat reveals internal kernel structure names and timing behavior, so it’s gated behind root access by default — keep it that way on any shared or production system.

Summary / Key Takeaways

• Lock contention is invisible until you measure it — /proc/lock_stat makes it visible on debug kernels.
• Watch waittime for latency impact and holdtime for root cause.
• On production kernels without lockdep, bpftrace gives you a lightweight second option.

Conclusion

Lock statistics turn a vague feeling that “something feels slow” into a concrete, per-lock number you can act on. Whether you reach for the kernel’s built-in lockdep-based reporting or a quick bpftrace probe on a production system, the goal is the same: find the lock, find whether it’s contention or hold-time driven, and fix the smallest possible critical section. This kind of measurement-first debugging is exactly the mindset we build throughout this free linux kernel development course.

Frequently Asked Questions