« Previous Lecture | Next Lecture »
A very common question in this free Linux kernel development course is: “the kernel has both a mutex and a semaphore — aren’t they the same thing?” They look similar on the surface (both can block a thread until a resource is available), but they solve different problems. This lecture, continuing our free embedded systems course on kernel synchronization, clears up that confusion and then introduces a scheduling hazard every kernel and RTOS developer should know: priority inversion.
The Linux Kernel Semaphore API
Linux does provide a semaphore object alongside the mutex. Its core operations are:
- Acquire with
down(), or the interruptible formdown_interruptible() - Release with
up()
The semaphore is the older of the two mechanisms in kernel history, and modern kernel guidance is to reach for a mutex whenever a simple binary lock is all you need, reserving the semaphore for the specific signaling role described below.
Mutex and Semaphore Are Solving Different Problems
The two objects are conceptually distinct, not just two names for the same primitive:
Three points are worth calling out individually:
- Generality: a semaphore is a more general primitive than a mutex. A mutex can be locked and later unlocked exactly once at a time, whereas a (counting) semaphore can be acquired and released multiple times, tracking a count rather than a strict binary state.
- Purpose: a mutex exists to protect a critical section from simultaneous access by multiple threads. A semaphore is better thought of as a signaling mechanism — one task (a “producer”) posts to the semaphore once some milestone is reached, and another task (a “consumer”) that is waiting on that semaphore wakes up and continues.
- Ownership: a mutex has a defined owner — only the thread that locked it may unlock it, and the kernel can enforce or at least assume this. A binary semaphore has no ownership concept at all; any thread can call
up()on it, even one that never calleddown().
Original Demo: Producer/Consumer Signaling with a Semaphore
The following short example, written for this course, shows the signaling use case a mutex cannot express cleanly — one thread announcing that data is ready, and another thread waiting for that announcement.
#include <linux/kthread.h>
#include <linux/semaphore.h>
#include <linux/delay.h>
static struct semaphore ep_data_ready_sem;
static int ep_shared_value;
/* Producer: computes a value, then signals that it is ready */
static int ep_producer_thread(void *arg)
{
while (!kthread_should_stop()) {
msleep(1000);
ep_shared_value = 42; /* pretend this is meaningful work */
pr_info("ep_sem_demo: producer posting data-ready signal\n");
up(&ep_data_ready_sem);
}
return 0;
}
/* Consumer: waits for the producer's signal before using the value */
static int ep_consumer_thread(void *arg)
{
while (!kthread_should_stop()) {
if (down_interruptible(&ep_data_ready_sem))
break;
pr_info("ep_sem_demo: consumer read value %d\n", ep_shared_value);
}
return 0;
}
Notice that neither thread “owns” ep_data_ready_sem — the producer only ever posts (up()) and the consumer only ever waits (down_interruptible()). That asymmetric, ownerless usage is exactly what a mutex is not designed to express, and it is why the kernel keeps both primitives available for their respective roles.
| Aspect | Mutex | Semaphore |
|---|---|---|
| Primary role | Mutual exclusion of a critical section | Signaling between tasks |
| Ownership | Yes, only the locker may unlock | None |
| Count | Binary, locked once | Can be counting |
| Modern kernel guidance | Preferred for simple locking | Use only for its signaling role |
Priority Inversion: A Locking Hazard Beyond Deadlocks
Careful lock ordering helps you avoid deadlocks, which we covered earlier in this chapter. But there is a second, subtler hazard that any locking scheme can trigger: priority inversion. This happens when a high-priority thread ends up waiting, indirectly, on a lower-priority thread — effectively inverting the priorities the scheduler was supposed to respect.
The unbounded version of this scenario — where the high-priority thread’s wait time has no upper limit because medium-priority threads keep preempting the lock holder — can be severe. Left unaddressed, it means the product’s most important thread is kept off the CPU for an unpredictable, potentially very long time, even though the scheduler was designed to always favour it.
Priority inversion is not just a textbook concern — it is famous for having affected a real spacecraft: NASA’s Mars Pathfinder mission in 1997 experienced repeated system resets on the Martian surface that were eventually traced back to a priority inversion problem in its onboard software’s locking behaviour. It is a good historical reminder that this class of bug is subtle enough to survive extensive pre-flight testing and only surface under specific real-world timing conditions.
How the Kernel Mitigates Priority Inversion: the RT-Mutex
The classic mitigation technique is priority inheritance: when a high-priority thread blocks on a lock held by a lower-priority thread, the lock holder temporarily “inherits” the waiter’s higher priority for as long as it holds the lock. This prevents medium-priority threads from being able to preempt the lock holder and stall the high-priority waiter indefinitely. Linux implements this through its RT-mutex infrastructure, which underlies the kernel’s real-time locking primitives and is also the basis for the PREEMPT_RT real-time kernel configuration we introduced earlier in this chapter. In practice, as a driver author you rarely call RT-mutex APIs directly — the important takeaway is architectural: understand that priority inversion exists, keep critical sections short, and be aware that the kernel’s own locking primitives are designed with this hazard in mind.
Practical Takeaways
- Use a mutex to protect a critical section; use a semaphore (or, in modern kernel code, a
struct completion) when you need one task to signal another. - Never assume a specific thread “owns” a semaphore — any thread can post to it, so design your signaling logic accordingly.
- Keep every critical section as short as possible; the longer a lock is held, the larger the window for priority inversion to bite.
- Be aware that lockdep, covered in the next chapter, focuses on deadlock detection, while priority inversion is a separate, scheduling-level hazard that the RT-mutex/priority-inheritance mechanism specifically addresses.
Frequently Asked Questions
Not conceptually. While both can block a thread, a mutex protects a critical section and has an owner, whereas a semaphore is a signaling mechanism with no ownership concept and can support a count greater than one.
Yes. A binary semaphore has no ownership rule, so any thread can call up() on it. This is precisely what makes it useful for producer/consumer style signaling.
Prefer the mutex for straightforward mutual exclusion. The semaphore is an older primitive and is best reserved for its signaling role rather than general-purpose locking.
It is a scenario where a high-priority thread ends up waiting indefinitely because a lower-priority thread holding a needed lock keeps getting preempted by medium-priority threads that do not need that lock at all.
The lock-holding thread temporarily inherits the priority of the higher-priority thread waiting on it, so medium-priority threads can no longer preempt it and stall the actual high-priority waiter.
It is the kernel’s priority-inheritance-aware mutex infrastructure, used internally for real-time-sensitive locking and as a foundation for the PREEMPT_RT real-time kernel configuration.
No. A deadlock is a circular wait where none of the involved threads can ever proceed. Priority inversion is a scheduling-priority problem — the threads are not stuck forever, but the high-priority thread is delayed far longer than its priority should allow.
Keep critical sections as short as possible and avoid holding a lock across any lengthy operation. The shorter the time a lock is held, the smaller the window in which a lower-priority holder can be preempted by unrelated threads.
« Previous Lecture | Next Lecture »
This closes out our tour of mutex locking. Next up: the kernel’s lock validator (lockdep) and catching locking issues early.

2 Comments