What You Will Learn

• What real-time really means: determinism, not speed
• The kernel preemption models and where CONFIG_PREEMPT_RT fits
• The four big internal changes PREEMPT_RT makes to the kernel
• Building and booting a real-time Linux kernel from mainline source
• Writing an RT-friendly application: memory locking, policies, and pitfalls
• Where Linux + PREEMPT_RT is (and is not) an appropriate RTOS choice

Prerequisites

Lectures 1–4 of this series, kernel build experience from earlier in this free linux kernel programming course, and a build machine or VM. Kernel 6.12 LTS or newer is recommended so PREEMPT_RT is available without any external patches.

Real-Time Means Deterministic, Not Fast

A real-time system is one where correctness depends on timing, not only on results. The metric that matters is worst-case latency: the maximum time between an event (interrupt, timer expiry) and the application code responding to it. A throughput-tuned server may answer in 5 µs on average yet occasionally take 20 ms; a real-time system may average 50 µs but never exceed 150 µs. For a motor controller or pacemaker, only the second system is correct. Soft real-time tolerates occasional misses (audio glitch); hard real-time does not (airbag).

Preemption Models in the Mainline Kernel

The question PREEMPT_RT answers is: how quickly can a high-priority task preempt whatever the kernel is currently doing? Mainline offers a spectrum:

Config	Name	Kernel code preemptible?	Typical target
PREEMPT_NONE	Server	No — only at syscall return	Throughput servers
PREEMPT_VOLUNTARY	Desktop	At explicit preemption points	Classic distro default
PREEMPT	Low-latency desktop	Yes, except critical sections	Desktops, soft RT
PREEMPT_RT	Real-time	Yes, almost everywhere	Industrial, audio, robotics

What PREEMPT_RT Actually Changes

In summary, PREEMPT_RT applies four transformations:

• Threaded interrupts everywhere: handlers become kernel threads (visible as irq/N-name in ps) that you can prioritize with the tools from Lecture 3.
• Sleeping spinlocks: most spinlock_t/rwlock_t sections become preemptible mutexes; only raw_spinlock_t remains truly atomic.
• Priority inheritance: built into those locks, defeating classic priority inversion inside the kernel.
• Preemptible almost-everything: softirqs and much other deferred work also run in threads.

The cost is a few percent of throughput. Real-time is a trade: you exchange average-case speed for a bounded worst case.

Building a Real-Time Kernel (No Patches Needed)

# Fetch a mainline source (6.12 LTS or later)
# and configure starting from your distro config
make olddefconfig
make menuconfig
# General setup ---> Preemption Model ---> Fully Preemptible Kernel (Real-Time)

# Or non-interactively:
scripts/config --enable CONFIG_EXPERT
scripts/config --enable CONFIG_PREEMPT_RT

make -j$(nproc)
sudo make modules_install install
sudo reboot

Verify after boot:

uname -v
# ... SMP PREEMPT_RT ...

zcat /proc/config.gz | grep PREEMPT_RT
# CONFIG_PREEMPT_RT=y

Many vendors also ship ready RT kernels (Ubuntu real-time kernel, Debian’s linux-image-rt, Yocto’s linux-yocto-rt) — often the fastest route on embedded projects.

Writing an RT-Friendly Application

An RT kernel bounds kernel-side latency; your application must avoid self-inflicted latency. The standard skeleton:

#define _GNU_SOURCE
#include <sched.h>
#include <sys/mman.h>
#include <time.h>
#include <stdio.h>
#include <stdlib.h>

#define PERIOD_NS 1000000L   /* 1 ms cycle */

int main(void)
{
    struct sched_param sp = { .sched_priority = 80 };
    struct timespec next;

    /* 1. Real-time policy (Lecture 3) */
    if (sched_setscheduler(0, SCHED_FIFO, &sp) == -1) {
        perror("sched_setscheduler"); exit(1);
    }

    /* 2. Lock all memory: a page fault is a latency bomb */
    if (mlockall(MCL_CURRENT | MCL_FUTURE) == -1) {
        perror("mlockall"); exit(1);
    }

    /* 3. Pre-fault and warm up before entering the loop:
          allocate all buffers here, touch every page once. */

    /* 4. Periodic loop with absolute deadlines (no drift) */
    clock_gettime(CLOCK_MONOTONIC, &next);
    for (;;) {
        /* ... deterministic cycle work here ... */

        next.tv_nsec += PERIOD_NS;
        while (next.tv_nsec >= 1000000000L) {
            next.tv_nsec -= 1000000000L;
            next.tv_sec++;
        }
        clock_nanosleep(CLOCK_MONOTONIC, TIMER_ABSTIME, &next, NULL);
    }
}

Golden rules inside the RT loop: no malloc(), no disk I/O, no unbounded locks shared with non-RT threads (use PTHREAD_PRIO_INHERIT mutexes when sharing is unavoidable), and prefer pre-allocated lock-free queues for passing data out.

Is Linux + PREEMPT_RT the Right RTOS for You?

• Great fit: industrial automation, robotics (ROS 2), professional audio, telecom, automotive infotainment side of the ECU, test and measurement — cycle times down to the low hundreds of microseconds on well-tuned hardware.
• Wrong fit: microsecond-hard control loops, safety-certified domains needing tiny certifiable kernels, or MCU-class hardware — dedicated RTOSes (or a hybrid AMP design: RTOS on one core, Linux on others) serve those better.
• Hardware matters: firmware SMIs on x86 and aggressive power management can blow any software guarantee. Tune BIOS/firmware, pin and isolate cores (Lecture 2), and measure (Lecture 6).

Common Mistakes and Troubleshooting

• Expecting RT kernel = fast kernel: throughput drops slightly; only worst-case latency improves.
• Skipping mlockall(): a single major page fault can cost milliseconds — instantly violating your budget.
• Leaving IRQ threads at defaults: your priority-80 task can still be delayed by an unrelated priority-50 IRQ thread; audit and set IRQ thread priorities deliberately.
• Out-of-tree drivers with raw locks or long IRQ-off regions: one bad driver ruins system-wide determinism — trace it (Lecture 1/6).
• Testing idle systems only: worst cases appear under load. Always measure with heavy background stress.

Key Takeaways

• Since kernel 6.12, PREEMPT_RT is a mainline config option — no external patchset required.
• It converts IRQ handlers to threads and most spinlocks to priority-inheriting mutexes, making the kernel preemptible almost everywhere.
• Applications must cooperate: RT policy, locked memory, pre-allocation, absolute-time periodic loops.
• Linux with PREEMPT_RT is a credible soft/firm RTOS down to sub-millisecond cycles; verify with measurement, never assumption.

Conclusion

Mainline Linux can now be an RTOS out of the box — a milestone that reshapes embedded system design and makes this one of the most valuable lectures in our free Linux kernel programming course. Combined with affinity, isolation, policies and bandwidth control from previous lectures, you have the complete recipe practitioners use in industrial and audio systems every day; learners on our free linux device drivers course track should pay special attention to the threaded-IRQ model, since your drivers will live inside it. One question remains: how do you prove your latency numbers? That is Lecture 6.

Frequently Asked Questions (FAQ)

1. Do I still need the out-of-tree PREEMPT_RT patch?

Not on kernel 6.12+ for x86, x86_64, arm64 and riscv — the real-time preemption model is selectable directly in mainline Kconfig.

2. Is PREEMPT_RT Linux “hard real-time”?

It provides strong, measurable worst-case bounds suitable for many industrial uses, but it is not formally certified hard real-time. Safety-critical domains still use certified RTOSes or hybrid architectures.

3. What latency can I expect?

On well-tuned modern hardware with isolated cores, maximum wakeup latencies in the tens-to-low-hundreds of microseconds are typical. Your hardware and firmware dominate — always measure your own system.

4. Why do spinlocks sleep on PREEMPT_RT?

So the sections they protect become preemptible by higher-priority tasks. Priority inheritance in the substituted rt_mutex prevents inversion. Code truly needing atomicity uses raw_spinlock_t.

5. Will my existing drivers work on an RT kernel?

In-tree drivers generally do. Out-of-tree drivers assuming non-preemptible spinlock sections or doing heavy work in hard IRQ context may need fixes.

6. Does PREEMPT_RT change the scheduler policies?

No. SCHED_FIFO/RR/DEADLINE work identically; the RT kernel simply removes most kernel-side delays between wakeup and execution.

7. RT kernel or core isolation — which matters more?

They attack different problems and are best combined: PREEMPT_RT bounds kernel preemption delays, isolation removes competing work from your critical core.