« Previous Lecture | Next Lecture »

What You Will Learn

• What determinism means and why it is the defining property of an RTOS
• The difference between a general-purpose OS and a real-time OS
• What latency and jitter are, in plain language
• The history of PREEMPT_RT and the Real-Time Linux (RTL) project
• Why the old “download the RT patch and apply it” workflow is now obsolete
• The kernel preemption models and where CONFIG_PREEMPT_RT fits in
• The core mechanisms: threaded interrupts, sleeping spinlocks, and priority inheritance

Prerequisites

• Basic Linux command-line skills
• A rough idea of what a process, an interrupt, and the scheduler are
• Having built a kernel once before helps but is not required for this theory lecture

Real-Time Does Not Mean Fast — It Means Predictable

This is the single most important idea in the whole topic, so let us settle it first. A real-time system is not necessarily a fast system. It is a system that responds within a known, bounded time, every single time. A system that answers in 2 milliseconds on average but occasionally takes 200 milliseconds is fast but not real-time. A system that always answers within exactly 5 milliseconds, even under heavy load, is real-time.

Think of a motor controller in a factory. If the emergency-stop signal is normally handled in microseconds, but once a day the handling takes half a second because the CPU was busy writing logs to disk, that one bad case can destroy the machine. Real-time engineering is entirely about controlling that worst case, not the average case.

This property — a stable, even response with minimal variation even under extreme load — is called determinism. The variation itself is called jitter. An RTOS is judged by its worst-case latency and its jitter, never by its throughput.

GPOS vs RTOS: Where Standard Linux Falls Short

Linux was designed as a general-purpose operating system (GPOS). Its scheduler, locking, and I/O paths are tuned to push the maximum amount of total work through the machine. To achieve that, the kernel routinely does things that are poison for determinism:

• It disables preemption inside critical sections, so a high-priority task must wait until the kernel finishes whatever it was doing.
• It holds spinlocks with interrupts disabled, blocking even hardware events for that duration.
• It batches work (disk writeback, network processing) to improve efficiency, creating occasional long pauses.

None of this is a bug. For a web server, finishing 10,000 requests per second matters far more than whether one request took 3 ms instead of 1 ms. But for a robot arm, a hearing aid, or a 5G base station, that one slow response is a failure. A real-time OS makes the opposite trade: it sacrifices a little throughput to guarantee that the highest-priority runnable task gets the CPU almost immediately, no matter what the kernel was doing.

To provide such deterministic behaviour, kernel algorithms must, wherever possible, run in constant time — O(1) complexity — so that response time does not grow with system load. This design goal drives everything you will see in the real-time Linux kernel.

A Short History of PREEMPT_RT and Real-Time Linux

The effort to make Linux behave like an RTOS started around 2005, led by Thomas Gleixner, Ingo Molnár, and a dedicated community. For nearly two decades the work lived as an out-of-tree patch set called PREEMPT_RT: you downloaded a patch matching your kernel version, applied it to the kernel source, and built a real-time kernel yourself. In 2015 the Linux Foundation took over stewardship and formed the Real-Time Linux (RTL) Collaborative Project to fund and coordinate the upstreaming effort. (Do not confuse RTL with co-kernel approaches such as Xenomai or RTAI — those run a separate real-time core alongside Linux, which is a completely different architecture.)

Over the years, large pieces of the patch set were merged piece by piece into mainline: threaded interrupts, priority-inheritance mutexes, the lockdep validator, high-resolution timers, and large parts of the tracing infrastructure. The final blocker for full inclusion was, surprisingly, printk — the kernel’s logging function — because printing to a slow console from atomic context could ruin latency. Once the printk rework landed, the last obstacle was gone.

The milestone: in September 2024, PREEMPT_RT was fully merged and enabled in the mainline kernel, and Linux 6.12 became the first release with built-in real-time capability on x86, x86_64, ARM64, and RISC-V. Since 6.12 is also an LTS kernel, it is an excellent base for real-time projects even today.

Why Wasn’t Real-Time Linux Merged Earlier?

Students often ask this. Two honest reasons:

• Linux is first a GPOS. Linus Torvalds held the position that a kernel designed for general-purpose computing should not carry highly invasive changes that only an RTOS needs. So merging happened slowly and deliberately, one well-reviewed piece at a time.
• The work was enormous. The real effort was not maintaining a patch — it was refactoring the entire kernel so that almost every code path could be safely preempted. Converting interrupt handlers to threads and spinlocks to sleepable locks touches practically every subsystem and every driver.

The upside of that slow path is quality: the real-time Linux kernel you build today is not a bolt-on. Its mechanisms have been reviewed, tested, and shipped in production for years.

Kernel Preemption Models in Modern Linux

When you run make menuconfig on any recent kernel, you will find the preemption model under General setup → Preemption Model. Understanding these options is essential before you build a real-time Linux kernel in the next lecture.

You can check which model your current kernel uses right now:

# Show kernel version string - a real-time kernel prints PREEMPT_RT here
uname -v

# On kernels built with PREEMPT_RT, this file exists and contains 1
cat /sys/kernel/realtime

The Three Core Mechanisms of Real-Time Linux

1. Threaded interrupt handlers

On a standard kernel, hardware interrupt handlers run in atomic context — nothing can interrupt them, not even your most critical task. On a real-time Linux kernel, almost every interrupt handler runs as a normal kernel thread with a priority. If your control task is more important than the network card’s interrupt thread, your task wins. You can literally see these threads:

# Interrupt threads appear as [irq/NN-name] processes
ps -e -o pid,pri,rtprio,comm | grep irq/

2. Sleeping spinlocks (RT-mutexes)

Most kernel spinlocks are converted into mutexes that can sleep. On a stock kernel, a CPU spinning on a lock blocks everything on that core. On PREEMPT_RT, a task waiting for a lock goes to sleep, freeing the CPU for a higher-priority task. This is the change that makes “almost everything preemptible” possible.

3. Priority inheritance

Sleeping locks create a classic danger: priority inversion. Imagine a low-priority task holds a lock that a high-priority task needs, and a medium-priority task keeps the low one from running. The high-priority task is now indirectly blocked by the medium one. Priority inheritance solves this: the low-priority lock holder temporarily inherits the priority of the highest waiter, finishes quickly, and releases the lock. The RT-mutex implementation in the real-time Linux kernel does this automatically.

Hard vs Soft Real-Time: An Honest Assessment

Be careful with vendor claims here. PREEMPT_RT gives Linux excellent deterministic behaviour — typically worst-case latencies in the tens of microseconds on well-tuned hardware — but Linux remains a huge, complex kernel. For applications where a single missed deadline means catastrophe (flight control, anti-lock brakes, grid protection), certified RTOSes or dedicated microcontrollers running an RTOS like Zephyr or FreeRTOS remain the right answer. For industrial automation, robotics, professional audio, telecom edge nodes, and data acquisition, real-time Linux is today a mature, mainstream, production-proven choice.

Real-World Applications of Real-Time Linux

• Industrial automation: PLC-style control loops, CNC machining (LinuxCNC runs on PREEMPT_RT)
• Robotics: ROS 2 control nodes with deterministic scheduling
• Telecom: 5G RAN and edge nodes where packet deadlines are contractual
• Professional audio: low-latency processing where a buffer underrun is an audible glitch
• Automotive and test benches: data acquisition and hardware-in-the-loop rigs

Common Misconceptions (Avoid These Mistakes)

• “RT makes my system faster.” No. Average throughput usually drops slightly. You gain a bounded worst case, not speed.
• “I still need to download RT patches.” Not for kernel 6.12 or newer on x86/ARM64/RISC-V. The old patch-download workflow described in pre-2024 books is obsolete for these architectures. Patches still exist for older LTS kernels and for features not yet upstreamed, but a beginner should simply start from a modern kernel.
• “Enabling PREEMPT_RT is enough.” The config option is step one. Real deployments also need CPU isolation, memory locking, IRQ affinity tuning, and long latency test runs — covered in Part 3.
• “RT Linux equals hard real-time certification.” Determinism is not the same as safety certification. Certification is a separate (and expensive) process.

Key Takeaways

• Real-time means predictable, not fast; determinism and low jitter define an RTOS.
• Standard Linux optimises throughput; real-time Linux optimises worst-case latency.
• PREEMPT_RT was developed out-of-tree from 2005 and fully merged in mainline Linux 6.12 (2024).
• Modern real-time Linux rests on threaded IRQs, sleeping spinlocks, and priority inheritance.
• You enable it via the PREEMPT_RT preemption model — no external patches needed on 6.12+ for x86, ARM64, and RISC-V.

Conclusion

The journey of real-time Linux from a 20-year out-of-tree patch set to a first-class mainline feature is one of the great engineering stories in open source. For you as a learner in this free Linux kernel development course, the practical meaning is simple: the barrier to building and experimenting with a real-time Linux kernel has never been lower. You no longer fight with patch versions and mismatched kernel releases — you pick a modern LTS kernel, flip one option, and build.

In the next lecture we do exactly that: configure, build, and boot a real-time Linux kernel from source, then verify that PREEMPT_RT is really active. After that, Part 3 teaches you to measure scheduling latency with cyclictest and the modern rtla/timerlat tooling, because in real-time engineering, a claim without a measurement is worthless.

Frequently Asked Questions

Is real-time Linux free to use?

Yes. PREEMPT_RT is part of the mainline Linux kernel and is released under the GPL, exactly like the rest of the kernel. This entire course is also free.

Which kernel version should I use for real-time Linux in 2026?

Kernel 6.12 LTS is the first mainline release with built-in PREEMPT_RT and remains a solid long-term choice. Newer LTS kernels (6.18) and current stable releases (7.x) also include it. Pick an LTS kernel for real projects.

Do I still need the RT patch from kernel.org?

Not for x86, x86_64, ARM64, or RISC-V on kernel 6.12 or newer — the support is built in. The RT patch project still maintains patches for older LTS kernels and carries some optimisations not yet merged, but beginners should start from mainline.

Is PREEMPT_RT hard real-time?

It provides strongly deterministic, bounded latencies suitable for many demanding applications, but Linux is not a certified hard real-time OS. For safety-critical deadlines where a miss means disaster, use a certified RTOS or a dedicated microcontroller.

What is the difference between PREEMPT_RT and Xenomai?

PREEMPT_RT makes the Linux kernel itself preemptible. Xenomai (and RTAI) use a co-kernel design: a small real-time core runs beside Linux and handles time-critical work separately. PREEMPT_RT is simpler to develop for because your real-time tasks are ordinary Linux threads.

Does real-time Linux reduce performance?

Throughput drops slightly (the exact amount depends on workload) because the kernel does extra work to stay preemptible. That is the deliberate trade-off for a bounded worst-case latency.

Can I run real-time Linux on a Raspberry Pi?

Yes. The Pi’s ARM64 cores are a supported architecture, and building an RT kernel for the Pi is a popular learning exercise. Latency results depend on the board, firmware, and tuning.

What tools measure real-time latency?

The classic tool is cyclictest from the rt-tests suite. Modern kernels also ship rtla with the timerlat and osnoise tracers built into the kernel tree. Part 3 of this series covers both.

« Previous Lecture | Next Lecture »