Why Compare Mainline and Real-Time Preemption at All?

Since kernel 6.12, PREEMPT_RT is not a separate kernel — it’s a preemption model you select inside the same source tree as everything else. That makes the comparison less about “two different kernels” and more about “two different behaviors the same kernel can have,” depending on a single config choice: CONFIG_PREEMPT_RT versus the default CONFIG_PREEMPT or CONFIG_PREEMPT_DYNAMIC. Knowing exactly which subsystems change behavior under that switch helps you decide whether your project actually needs it.

Updated Comparison Table (Kernel 6.12+)

Notice that almost every row now says “same tool” or “same core mechanism” on both sides — that’s the direct result of two decades of gradual merging. The one row that still meaningfully differs is how spinlocks and interrupt handling behave, because that’s the structural change that actually defines the real-time preemption model.

Spinlocks: The Core of the Difference

In the standard model, a spinlock’s critical section is intentionally non-preemptible — nothing else runs on that CPU until the lock is released, which keeps the logic simple and fast for the common case. Under the fully preemptible model, most spinlocks are transformed into a mutex-like primitive with priority inheritance. If a low-priority task holds the lock and a high-priority task needs it, the low-priority task temporarily “borrows” the high-priority task’s priority just long enough to finish and release the lock, preventing unbounded priority inversion.

Threaded Interrupts: Making IRQs Schedulable

Ordinary interrupt handlers historically ran in a context where the scheduler couldn’t intervene. Threaded interrupt handling moves the bulk of that work into a real kernel thread that the scheduler can prioritize, delay, or preempt just like any other task. This is what lets a critical real-time thread genuinely outrank a device interrupt when it needs to.

ps -eo pid,pri,rtprio,cmd | grep irq
# example output
   45  49    50 [irq/34-eth0]
   46  49    50 [irq/35-nvme0q1]

Each threaded IRQ handler shows up as its own schedulable kernel thread with its own priority, which you can adjust with chrt if a particular device’s interrupts need to run above or below your application thread.

Common Mistakes and Troubleshooting

Symptom	Likely Cause & Fix
Interrupt-heavy driver causes RT thread jitter	Lower the IRQ thread’s rtprio below your application thread with chrt -f -p.
Priority inversion suspected	Confirm the lock is an RT-mutex-backed spinlock, not a raw spinlock explicitly marked to stay atomic.
Writer starvation on an RW lock	Check whether the code path uses a fair RW lock primitive or an older, reader-favoring one.

Best Practices

• Never assume “RT kernel” automatically fixes a specific driver’s latency — check whether that driver’s interrupt handler is actually threaded.
• Use chrt and ps -eo pid,rtprio,cmd together to inspect real priorities before tuning blindly.
• Treat raw spinlocks (explicitly marked to stay atomic even under RT) as a red flag in code review — they should be rare and well justified.

Performance and Security Considerations

Performance: priority inheritance and threaded IRQs add a small, bounded overhead per lock acquisition and interrupt, in exchange for predictability. On CPU-bound, non-latency-sensitive workloads this overhead is pure cost with no benefit.

Security: giving a thread real-time priority is effectively giving it the ability to monopolize a CPU core. Only grant SCHED_FIFO/SCHED_RR privileges to trusted, well-tested code, and keep the sched_rt_runtime_us safety valve enabled.

Key Takeaways

• The fully preemptible model and the standard model share most of the kernel’s infrastructure today — they differ mainly in spinlock behavior and interrupt threading.
• Priority inheritance on RT-mutexes is what prevents unbounded priority inversion.
• Threaded IRQs make interrupt processing something the scheduler can reason about and prioritize.
• Most of what used to be “RT-only” — high-resolution timers, lockdep, Ftrace — has been standard in every mainline kernel for years.

Conclusion

The real distance between a standard kernel and a fully preemptible one is smaller today than it has ever been, but the parts that remain different — spinlock behavior and interrupt threading — are exactly the parts that determine whether your application meets its deadlines. In the next lesson, we’ll put this into practice by actually measuring latency on your system with cyclictest and the newer rtla/timerlat tooling.

Frequently Asked Questions