What You Will Learn

Prerequisites

GPOS vs RTOS: Why Stock Linux Is Not Real-Time

Mainline Linux, like Windows and macOS, is a General Purpose Operating System (GPOS). A GPOS optimizes for overall throughput and fairness: finish as much total work as possible and give every application a reasonable share. A Real-Time Operating System (RTOS) optimizes for something different: producing the correct result before a deadline, and doing so predictably. In real-time computing, a correct answer that arrives late is a wrong answer.

Stock Linux actually behaves as a decent soft real-time system: deadlines are met most of the time. That is fine for video playback, where a rare dropped frame annoys nobody. It is not fine for an airbag controller, an industrial robot, a pacemaker, a stock-exchange matching engine, or the audio path of a hearing aid. Those are hard real-time domains where a single missed deadline means failure — sometimes physical damage.

Determinism and Jitter: Speed Is Not the Point

Here is the idea beginners most often get wrong: real-time does not mean fast. A response time of ten milliseconds can be perfectly real-time if the system delivers it every single cycle. What matters is determinism: the same stimulus produces a response in the same, bounded time. The variation of the response time around its expected value is called jitter. An RTOS engineering effort is largely a war on jitter — squeezing the worst case down toward the average case. On a GPOS the worst case is essentially uncontrolled: latency can be a few microseconds now and several milliseconds a moment later, depending on what else the kernel happens to be doing.

Where Latency Comes From: Kernel Preemption Models

Latency for a real-time task is the delay between “the event happened” and “my task is running on a CPU”. On Linux the biggest contributor is time spent where the kernel cannot be preempted: long non-preemptible sections, interrupt handlers, and code holding spinlocks. The kernel offers several preemption models, selectable at build time (and on modern kernels even at boot, via the preempt= parameter):

What PREEMPT_RT Actually Changes

PREEMPT_RT is not a new scheduler bolted on top; it is a set of deep changes that shrink the non-preemptible parts of the kernel to a minimum:

• Sleeping spinlocks: most kernel spinlocks become mutexes that a waiting task can sleep on, so a high-priority task is no longer stuck behind a spinning section. Raw spinlocks remain only where truly unavoidable.
• Threaded interrupt handlers: most interrupt work runs in schedulable kernel threads instead of hard interrupt context, so your real-time task can preempt a lower-priority driver’s interrupt processing.
• Priority inheritance: when a low-priority task holds a lock a high-priority task needs, the holder temporarily inherits the higher priority, defeating the classic priority-inversion problem.
• High-resolution timers and preemptible RCU: timing and read-side kernel synchronization are made compatible with tight deadlines.

The historic part: this work lived as an out-of-tree patch set for about 20 years, maintained by the kernel real-time community and the Linux Foundation’s RT project. Piece by piece it was merged upstream, and with kernel 6.12 the final PREEMPT_RT pieces landed in mainline for x86, x86_64, arm64 and riscv (LoongArch followed in 6.13, and 32-bit ARM support arrived later in the 6.x series). Older tutorials that tell you to download a patch from kernel.org and apply it before building are describing history — on any current kernel, real-time Linux is a checkbox.

Hands-On: Build and Boot a Real-Time Linux Kernel

Because PREEMPT_RT is now mainline, the build is a completely standard kernel build. On Debian/Ubuntu you may not even need to build: apt install linux-image-rt-amd64 (Debian) or the Ubuntu Pro real-time kernel gives you a prebuilt one. For learning, building it yourself is worth the hour:

Step 1 — Fetch a Modern Kernel and Enable Real-Time

$ wget https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.12.95.tar.xz
$ tar xf linux-6.12.95.tar.xz && cd linux-6.12.95
$ make olddefconfig          # start from your running config
$ make menuconfig
# General setup ---> Preemption Model ---> Fully Preemptible Kernel (Real-Time)

That menu entry sets CONFIG_PREEMPT_RT=y. Note it only appears when CONFIG_EXPERT is enabled and the architecture supports RT. Verify:

$ grep -E "PREEMPT_RT|EXPERT" .config
CONFIG_EXPERT=y
CONFIG_PREEMPT_RT=y

Step 2 — Build, Install, Reboot

$ make -j$(nproc)
$ sudo make modules_install
$ sudo make install
$ sudo reboot

$ uname -v
#1 SMP PREEMPT_RT ...

The string PREEMPT_RT in uname -v confirms you are on the real-time kernel. You can also check cat /sys/kernel/realtime, which prints 1 on RT kernels.

Step 3 — Write a Real-Time Task (SCHED_FIFO)

A real-time kernel alone changes nothing for normal tasks; your application must ask for a real-time scheduling class. Here is a minimal, original example of a periodic 1 ms control-loop skeleton:

#include <stdio.h>
#include <time.h>
#include <sched.h>
#include <string.h>
#include <sys/mman.h>

#define PERIOD_NS 1000000L   /* 1 ms cycle */

int main(void)
{
    struct sched_param sp = { .sched_priority = 80 };
    struct timespec next;

    /* 1. Become a real-time task */
    if (sched_setscheduler(0, SCHED_FIFO, &sp)) {
        perror("sched_setscheduler (run as root or set rtprio limits)");
        return 1;
    }

    /* 2. Lock memory: a page fault mid-cycle would ruin determinism */
    mlockall(MCL_CURRENT | MCL_FUTURE);

    clock_gettime(CLOCK_MONOTONIC, &next);
    for (int cycle = 0; cycle < 5000; cycle++) {
        /* ---- your deterministic work goes here ---- */

        /* 3. Sleep until the exact start of the next period */
        next.tv_nsec += PERIOD_NS;
        while (next.tv_nsec >= 1000000000L) {
            next.tv_nsec -= 1000000000L;
            next.tv_sec++;
        }
        clock_nanosleep(CLOCK_MONOTONIC, TIMER_ABSTIME, &next, NULL);
    }
    printf("completed 5000 cycles\n");
    return 0;
}

$ gcc -O2 -o rt_loop rt_loop.c
$ sudo ./rt_loop
completed 5000 cycles

The three habits shown here — SCHED_FIFO priority, mlockall(), and absolute-time clock_nanosleep() — are the core discipline of every real-time Linux application, from CNC controllers to professional audio engines.

Step 4 — Measure Latency with cyclictest

The standard measuring stick for real-time Linux is cyclictest from the rt-tests suite. It arms a timer, sleeps, and records how late it actually woke up:

$ sudo apt install rt-tests
$ sudo cyclictest --mlockall --smp --priority=90 --interval=200 --distance=0
T: 0 ( 2143) P:90 I:200 C: 981234 Min:  2 Act:  4 Avg:  4 Max:  27

Read the last three numbers in microseconds: average wake-up latency 4 µs, worst case 27 µs over nearly a million cycles. On the same machine with a stock kernel under load (compile jobs, network traffic), the Max column commonly explodes into the hundreds or thousands of microseconds. Always run cyclictest while stressing the machine — an idle system flatters any kernel. Pair it with stress-ng for honest numbers.

Real-World Applications of Real-Time Linux

• Industrial automation: PLCs and EtherCAT/PROFINET masters run 250 µs–1 ms control cycles on PREEMPT_RT.
• Robotics: ROS 2 real-time executors and motor servo loops depend on bounded wake-up latency.
• Telecom and 5G: baseband and O-RAN distributed units use RT kernels to hit strict slot timing.
• Professional audio: low-latency audio servers keep buffer sizes tiny without dropouts.
• Automotive and medical: soft ECUs, infusion pumps, patient monitors — domains where late equals unsafe.
• Embedded and Bluetooth devices: isochronous data paths (for example LE Audio streams) benefit from predictable scheduling on embedded Linux gateways.

Common Mistakes and Troubleshooting

Best Practices, Performance and Security Notes

• Isolate CPUs for real-time work with isolcpus/nohz_full boot parameters or cpuset cgroups, and pin the RT thread with CPU affinity. Combined with the cgroup knowledge from the previous lecture, you can fence housekeeping onto other cores.
• Disable deep CPU idle states and aggressive frequency scaling on RT cores; a core waking from deep sleep adds tens of microseconds of latency.
• Keep firmware in mind: SMIs (System Management Interrupts) on x86 bypass the kernel entirely and are a classic source of unexplained latency spikes. Server-grade or well-chosen embedded hardware matters.
• Security: granting unprivileged users real-time priority is a denial-of-service risk — a runaway FIFO task can freeze a CPU. Grant rtprio narrowly, keep the RT throttle enabled during development, and audit which units systemd starts with real-time policies.
• Test worst case, not average: soak the system for hours under representative load and record the Max latency. Hard real-time claims live and die on the tail.

Key Takeaways

Conclusion

Real-time Linux went from a research patch to a mainstream capability: any modern kernel can become fully preemptible with one config option, and the same code base now powers robots, base stations and audio consoles. In this lecture you learned the vocabulary (hard vs soft real-time, determinism, jitter), the mechanism (preemption models and what PREEMPT_RT rewires inside the kernel), and the practice (building an RT kernel, writing a periodic SCHED_FIFO task, and proving latency with cyclictest under load).

This lecture is part of the free Linux kernel development course at EmbeddedPathashala, which connects with our free Linux device drivers course and free embedded systems course. In the previous lecture we limited CPU with the cgroups v2 CPU controller; together the two lessons give you both sides of scheduling control: capping the greedy and guaranteeing the critical.

Frequently Asked Questions (FAQ)