« Previous Lecture | Next Lecture »
Real-Time Linux Kernel (PREEMPT_RT) Explained for Beginners
Part of our free Linux kernel development course — learn what real-time really means, how the real-time Linux kernel works, and why PREEMPT_RT becoming part of mainline Linux changes everything for embedded engineers.
Beginner
Part 1 of 3
100% Free
Welcome to this lecture of our free Linux kernel development course at EmbeddedPathashala. In this tutorial you will understand the real-time Linux kernel from the ground up. For nearly twenty years, turning Linux into a real-time operating system meant downloading a separate patch, applying it to the kernel source, and rebuilding everything. That era is over. Since Linux kernel 6.12, the real-time Linux kernel capability known as PREEMPT_RT ships inside the official mainline kernel itself. If you are preparing for embedded Linux interviews, working on robotics, industrial automation, or automotive projects, this free tutorial gives you the foundation you need — explained in simple language, with no prior real-time experience required.
What You Will Learn
- What “real-time” actually means (and what it does not mean)
- Why the standard Linux kernel cannot guarantee response times
- The four preemption models available in the modern kernel
- How PREEMPT_RT changes kernel internals: sleeping spinlocks, threaded IRQs, priority inheritance
- The history: from out-of-tree patch (2005) to mainline merge (kernel 6.12) to full ARM support (kernel 7.1)
- When you should — and should not — use a real-time Linux kernel
Prerequisites
This lecture from our free embedded systems course assumes only basic familiarity with:
- Linux command line usage
- What a process and an interrupt are (covered earlier in this free Linux kernel development course)
- Basic idea of what the kernel does (scheduling, memory, drivers)
No prior real-time systems knowledge is needed.
What Does “Real-Time” Actually Mean?
The most common misunderstanding among beginners is thinking real-time means fast. It does not. Real-time means predictable. A real-time system guarantees that it will respond to an event within a known, bounded amount of time — every single time, not just on average.
Think of two food delivery services. Service A usually delivers in 10 minutes but occasionally takes 3 hours. Service B always delivers in exactly 30 minutes. Service A is faster on average, but Service B is predictable. A real-time system is like Service B. For a robot arm on a factory line or an airbag controller in a car, an occasional 3-hour delay (or in kernel terms, a multi-millisecond latency spike) is a failure — possibly a dangerous one.
Hard Real-Time vs Soft Real-Time
| Aspect | Hard Real-Time | Soft Real-Time |
|---|---|---|
| Missed deadline means | System failure (unacceptable) | Degraded quality (tolerable) |
| Examples | Airbag deployment, motor control, pacemaker | Video streaming, audio playback, gaming |
| Typical latency target | Tens of microseconds, guaranteed | Milliseconds, usually met |
| Linux with PREEMPT_RT suitable? | Yes, for many firm/hard use cases with careful tuning | Yes, easily |
Why the Standard Linux Kernel Is Not Real-Time
The standard Linux kernel is optimized for throughput — getting the maximum total work done. To achieve this, it deliberately makes choices that hurt predictability:
- Non-preemptible critical sections: When kernel code holds a traditional spinlock, no other task can run on that CPU until the lock is released. If a low-priority task grabs a lock inside the kernel, your critical high-priority task simply waits.
- Interrupt handlers run above everything: A storm of network interrupts can delay your real-time task, because hardware interrupt handlers in the stock kernel preempt all threads — regardless of priority.
- Priority inversion: A high-priority task can end up blocked behind a low-priority task that holds a resource it needs, while a medium-priority task happily runs and starves them both.
The result: on a stock kernel, worst-case scheduling latency can spike from a few microseconds to several milliseconds under load. For a motor controller expecting a response every 250 microseconds, that spike is a crash — sometimes literally.
| Kernel Type | Average Latency | Worst-Case Latency | Predictable? |
|---|---|---|---|
| Standard (throughput-first) | Very low | Unbounded — can reach milliseconds | No |
| PREEMPT_RT (determinism-first) | Low (slightly higher overhead) | Bounded — typically tens of microseconds | Yes |
The Four Preemption Models in the Real-Time Linux Kernel Era
Preemption means the kernel’s ability to interrupt currently running code and switch to something more important. Modern kernels (6.12 and later) offer four preemption models directly in the configuration menu — no external patch required. This is one of the most frequently asked topics in embedded Linux interviews, so study this table carefully:
| Model | Config Option | Kernel Can Be Preempted… | Best For |
|---|---|---|---|
| No Forced Preemption | CONFIG_PREEMPT_NONE |
Only at explicit points (syscall return, etc.) | Servers, batch processing |
| Voluntary Preemption | CONFIG_PREEMPT_VOLUNTARY |
At extra voluntary points sprinkled in kernel code | Desktops (common distro default) |
| Preemptible Kernel | CONFIG_PREEMPT |
Almost anywhere, except spinlock-held sections | Low-latency desktops, multimedia |
| Fully Preemptible (Real-Time) | CONFIG_PREEMPT_RT |
Nearly everywhere, including most former spinlock sections | Robotics, industrial, automotive, audio |
The fourth option is what this tutorial series is about. In the next lecture of this free Linux kernel development course, you will select it yourself and build a real-time Linux kernel step by step.
How PREEMPT_RT Works Internally
PREEMPT_RT follows one simple philosophy: shrink the amount of kernel code that cannot be preempted to the absolute minimum. It achieves this through three core mechanisms.
1. Spinlocks Become Sleeping Locks
In a stock kernel, a spinlock disables preemption — the CPU literally spins in a loop waiting. Under PREEMPT_RT, most spinlocks are transparently converted into rt-mutexes, which can sleep. This means a high-priority task can preempt a lower-priority task even while that task is executing inside the kernel holding a lock. Only a small set of “raw” spinlocks (protecting the scheduler itself, low-level interrupt code) remain truly non-preemptible.
2. Interrupt Handlers Become Kernel Threads
With forced interrupt threading, each hardware interrupt handler runs as a schedulable kernel thread with its own priority. Your critical control task at priority 90 can now preempt the network card’s interrupt thread at priority 50. On a PREEMPT_RT system you can actually see these threads:
$ ps -eLo pid,rtprio,comm | grep irq
45 50 irq/27-eth0
52 50 irq/30-i2c_designware
61 50 irq/33-xhci_hcdEach irq/NN-name entry is an interrupt handler running as a thread whose priority you can change with chrt. This is a superpower for embedded engineers: you decide whether your CAN bus interrupt matters more than your Ethernet interrupt.
3. Priority Inheritance Prevents Priority Inversion
When a high-priority task blocks on a lock held by a low-priority task, the rt-mutex temporarily boosts the low-priority task to the waiter’s priority. The lock holder finishes quickly, releases the lock, and drops back down. This solves the classic priority inversion problem that famously affected NASA’s Mars Pathfinder mission.
| Step | What Happens | Effective Priority of Lock Holder |
|---|---|---|
| 1 | Low-priority task L acquires rt-mutex | Low (e.g. 10) |
| 2 | High-priority task H (90) blocks on same mutex | Boosted to 90 |
| 3 | L finishes critical section fast, releases mutex | Restored to 10 |
| 4 | H acquires mutex and proceeds immediately | 90 — no inversion occurred |
From Out-of-Tree Patch to Mainline: A Short History
Older books and tutorials (including many written around kernel 5.x) describe a workflow where you download the kernel source, download a matching patch-*-rtNN file, apply it with the patch command, and only then does the real-time option appear in the configuration menu. That workflow is now historical for mainstream architectures. Here is the timeline every embedded engineer should know:
- 2005: The PREEMPT_RT patchset begins as an out-of-tree project driven by Ingo Molnár, Thomas Gleixner, and others.
- 2015: The Linux Foundation forms the Real-Time Linux (RTL) collaborative project to fund and coordinate upstreaming.
- 2021: The core real-time locking infrastructure is merged into mainline.
- September 2024: The final blocker (the printk logging rework) is resolved, and kernel 6.12 becomes the first mainline release with built-in PREEMPT_RT for x86, x86_64, ARM64, and RISC-V.
- 2025–2026: LoongArch support lands in 6.13, and with kernel 7.1, classic 32-bit ARM boards also build real-time kernels with zero external patches.
What does this mean for you as a learner? If you use kernel 6.12 or newer on x86, ARM64, or RISC-V, you can enable a real-time Linux kernel purely through configuration. A small external patch queue still exists, carrying the newest optimizations before they get merged — useful for bleeding-edge needs, and still necessary if you are stuck maintaining an older LTS kernel like 5.10 or 6.6. We cover both paths in Part 2.
Real-World Use Cases of the Real-Time Linux Kernel
- Industrial automation: PLCs, CNC machines, and EtherCAT masters need cycle times of 250 µs to 1 ms with tight jitter budgets.
- Robotics: Motor control loops and sensor fusion running at kilohertz rates on ARM64 boards.
- Automotive: Domain controllers where Linux must coexist with safety-critical timing requirements.
- Professional audio: Low-latency audio processing where a buffer underrun means an audible glitch on stage.
- Telecom / 5G: Packet processing in radio access networks with strict deadline scheduling.
- Wireless and Bluetooth stacks: Isochronous audio streams (like LE Audio) benefit from predictable scheduling of the transport threads.
When You Should NOT Use PREEMPT_RT
Honesty matters in engineering. Avoid PREEMPT_RT when:
- Your workload is pure throughput (databases, web servers, build farms) — RT adds overhead and lowers peak throughput.
- You need guaranteed sub-10 microsecond hard deadlines — consider a dedicated microcontroller or RTOS co-processor instead.
- Your application never uses real-time scheduling policies — an RT kernel alone does nothing; your tasks must request
SCHED_FIFO/SCHED_RRpriorities to benefit.
Common Beginner Mistakes
- Believing RT = faster: A real-time kernel can be slightly slower on average. It trades throughput for bounded worst-case latency.
- Booting an RT kernel and expecting magic: Your application must explicitly use real-time priorities and lock its memory; otherwise it is scheduled like any normal task.
- Testing latency on an idle system: Idle-system numbers are meaningless. Worst-case latency only reveals itself under heavy, realistic load over long test runs.
- Following kernel 5.x-era tutorials verbatim: They tell you to hunt for a patch file that, for modern kernels on mainstream architectures, you no longer need.
Key Takeaways
- Real-time means predictable and bounded response time, not raw speed.
- The stock kernel favors throughput; the real-time Linux kernel (PREEMPT_RT) favors determinism.
- PREEMPT_RT works by converting spinlocks to sleeping rt-mutexes, threading interrupt handlers, and applying priority inheritance.
- Since kernel 6.12, PREEMPT_RT is built into mainline Linux for x86, ARM64, and RISC-V — and since kernel 7.1, for 32-bit ARM too. No external patch is required on these architectures.
- Select
CONFIG_PREEMPT_RT(“Fully Preemptible Kernel”) in the Preemption Model menu to enable it — the exact steps are in Part 2 of this free Linux kernel development course.
Frequently Asked Questions
Is the real-time Linux kernel free to use?
Yes. PREEMPT_RT is part of the open-source Linux kernel under the GPL. This entire tutorial is also part of our free Linux kernel development course.
Do I still need to apply the RT patch in 2026?
Not for kernel 6.12+ on x86, x86_64, ARM64, RISC-V (and 32-bit ARM from kernel 7.1). You only need the external patch queue for older LTS kernels or for the very latest not-yet-merged optimizations.
Is PREEMPT_RT hard real-time like a traditional RTOS?
It provides bounded latencies suitable for many firm and hard real-time applications, typically in the tens of microseconds on well-tuned hardware. For guaranteed sub-10 µs deadlines or formal safety certification, a dedicated RTOS or co-processor is still common.
Does enabling PREEMPT_RT slow down my system?
Average throughput drops slightly because of extra locking and threading overhead. In exchange, worst-case latency becomes bounded — the entire point of a real-time kernel.
Which distributions ship a ready-made RT kernel?
Debian provides pre-packaged RT kernels for x86 and ARM64, Ubuntu offers one through Ubuntu Pro’s real-time kernel option, and Yocto has the linux-yocto-rt recipe for embedded builds.
Is this relevant for embedded systems interviews?
Extremely. Preemption models, priority inversion, and threaded IRQs are among the most common Linux kernel interview topics. This free embedded systems course covers all of them.
Continue Your Free Linux Kernel Development Course
In Part 2, you will configure and build your own real-time Linux kernel from source — step by step, on a modern kernel.
Next Lecture: Build the RT Kernel » Course Index
3 Comments