What You Will Learn

• Why standard Linux is not deterministic, and what PREEMPT_RT actually changes
• How real-time Linux kernel latency testing works at the scheduler level
• Installing and running cyclictest on a modern (2024+) kernel
• Reading min/avg/max latency and understanding jitter
• The newer rtla / timerlat tracer that ships inside the kernel itself
• Common mistakes, tuning tips, and best practices for trustworthy results

Prerequisites

• Comfortable with the Linux command line and basic apt/dnf package management
• A Linux machine or single-board computer (x86_64, ARM64, or RISC-V) — a Raspberry Pi 4/5 or any recent PC works fine
• Basic understanding of processes, threads, and interrupts (helpful, not mandatory)
• Root/sudo access to install packages and change scheduler settings

What Is Real-Time Linux Kernel Latency Testing?

Every operating system scheduler makes a promise: “when your task becomes ready to run, I will run it soon.” On a general-purpose Linux kernel, “soon” is usually a few microseconds, but it can occasionally stretch into milliseconds if the kernel is busy handling something else that cannot be interrupted. Real-time Linux kernel latency testing is the practice of measuring exactly how long that gap can get, under realistic system load, so you can decide whether the platform is fit for your timing budget.

The gap being measured is usually called scheduling latency: the time between a timer or interrupt firing and the moment your real-time thread actually starts executing on a CPU core. A device that must sample a sensor every millisecond, or update a motor control loop at a fixed rate, needs this gap to be small and, more importantly, predictable.

Why PREEMPT_RT Matters (and Why It’s a Bigger Deal Now)

For roughly two decades, turning Linux into a real-time-capable kernel meant downloading a separate out-of-tree patch set called PREEMPT_RT and applying it by hand to your kernel source, then rebuilding everything. That changed in September 2024, when PREEMPT_RT was merged into the mainline Linux kernel starting with version 6.12, after the final printk-related blockers were resolved. This means real-time Linux kernel latency testing today can be done on a stock kernel source tree — you simply enable CONFIG_PREEMPT_RT in your kernel configuration instead of hunting for a matching patch file.

PREEMPT_RT works by shrinking the sections of kernel code that cannot be interrupted. It does this mainly by:

• Threading interrupt handlers — most hardware interrupts run as ordinary schedulable kernel threads instead of blocking everything else
• Making spinlocks preemptible — most spinlocks become sleeping locks with priority inheritance, so a low-priority holder cannot silently block a high-priority waiter
• Prioritising the high-resolution timer subsystem so timer-driven wakeups fire as close to on-time as possible

It is important to set expectations correctly: PREEMPT_RT gives you soft real-time behaviour — very low latency, very high probability of meeting your deadline — not the hard, mathematically guaranteed determinism of a microcontroller RTOS like FreeRTOS or Zephyr. For safety-critical hard-real-time work (flight control, braking systems), a dedicated RTOS or MCU is still the right tool. For soft real-time work — industrial data acquisition, robotics control loops, telecom edge nodes, audio/video pipelines — mainline PREEMPT_RT Linux is now a very credible option.

Understanding Latency: Min, Avg, Max, and Jitter

When you run any real-time Linux kernel latency testing tool, it will report three headline numbers, usually in microseconds (µs):

Here is the counter-intuitive part that trips up newcomers doing real-time Linux kernel latency testing for the first time: a standard (non-RT) kernel can have a better average latency than a PREEMPT_RT kernel, because RT threading adds some bookkeeping overhead. But the standard kernel’s worst case can be dramatically worse — occasionally jumping into the tens of milliseconds when the kernel is doing something non-preemptible. PREEMPT_RT trades a little average-case performance for a much tighter worst case. For real-time systems, the worst case is the number you design around, not the average.

Introducing cyclictest: The Standard Latency Measurement Tool

cyclictest is the de-facto community tool for real-time Linux kernel latency testing. It works by creating one or more high-priority threads that repeatedly sleep for a fixed interval using a high-resolution timer, then measure the difference between when they should have woken up and when they actually did. That difference is your scheduling latency sample.

Flag	Purpose
-t	Number of test threads (usually one per CPU core)
-p	Real-time scheduling priority (1–99, SCHED_FIFO)
-i	Wakeup interval in microseconds
-l	Total number of loops/samples to collect
-h	Generate a histogram of latency distribution
-q	Quiet mode — print only the final summary

Step-by-Step: Running Your First Real-Time Linux Kernel Latency Test

These steps assume a Debian/Ubuntu-based distribution on a recent kernel. The same workflow applies on Fedora or a Yocto-based embedded image with minor package-manager differences.

Step 1 — Confirm whether your kernel has real-time support enabled.

uname -r
grep PREEMPT_RT /boot/config-$(uname -r)
cat /sys/kernel/realtime 2>/dev/null

If the last command prints 1, PREEMPT_RT is active. If it fails or shows 0, you are on a standard preemptible kernel — still useful for comparison, but not a true RT kernel.

Step 2 — Install the rt-tests package, which bundles cyclictest.

sudo apt update
sudo apt install rt-tests

Step 3 — Generate some background load so the test reflects real conditions, not an idle system.

sudo apt install stress-ng
stress-ng --cpu 4 --io 2 --vm 2 --vm-bytes 128M --timeout 600s &

Step 4 — Run cyclictest with a realistic priority and interval.

sudo cyclictest -t4 -p95 -i1000 -l100000 -q

This spawns four threads, each running at real-time priority 95, waking up every 1000 microseconds (1 ms), for 100,000 samples. The -q flag keeps the live output quiet and prints only the summary line per thread when finished — typically showing per-thread Min/Avg/Max values in microseconds.

Step 5 — Let it complete, then read the Min/Avg/Max summary for each CPU thread and compare it against your application’s timing budget. If your control loop must respond within 200 µs and your Max value comes back at 40 µs, you have solid headroom. If Max spikes into the thousands of microseconds, you need further tuning (see the Best Practices section below) or a hardware/kernel change.

Interpreting Your Results

The table below shows the kind of pattern you should expect to observe — illustrative values only, since actual numbers depend heavily on your specific hardware, kernel build, and background load:

Notice the pattern: the standard kernel’s average can look fine, but its worst case is orders of magnitude larger than the RT kernel’s worst case. That gap — not the average — is exactly what real-time Linux kernel latency testing exists to expose, and it is the reason a non-deterministic Max value disqualifies a standard kernel from hard timing budgets even when it “usually” performs well.

Modern Alternative: rtla and the timerlat Tracer

cyclictest remains extremely useful, but the mainline kernel now ships its own in-tree tracer for the same job. The timerlat tracer, exposed through the rtla (Real-Time Linux Analysis) command-line tool located under tools/tracing/rtla/ in the kernel source, uses the same cyclic-timer approach as cyclictest but adds the ability to trace exactly *what* caused a latency spike — down to the offending function call.

sudo apt install linux-tools-common linux-tools-$(uname -r)
sudo rtla timerlat top

This prints a live, per-CPU view of timer IRQ latency, kernel-thread latency, and user-thread latency — three numbers that together make up the same “wakeup latency” that cyclictest reports as a single figure, but broken down by stage so you can see exactly where time is being lost.

Real-World Use Cases

• Industrial data acquisition: sampling sensors at a fixed rate for process monitoring
• Robotics control loops: keeping joint controllers or motor drivers on a tight update cycle
• Professional audio: avoiding buffer underruns in low-latency audio interfaces
• Telecom / edge networking: deterministic packet processing for 5G and software-defined networking nodes
• CNC and motion control: step-pulse generation for machine tools (a well-known LinuxCNC use case)

Common Mistakes and Troubleshooting

Mistake	Fix
Testing an idle system	Always generate representative background load (CPU, I/O, memory, network) during the test
Leaving CPU frequency scaling enabled	Set the governor to performance to remove frequency-transition latency spikes
Ignoring competing real-time services	Check ps -eo pid,class,rtprio,cmd for other SCHED_FIFO tasks that could preempt your test threads
Running for too short a duration	Worst-case spikes are rare events — run for hours, not seconds, before trusting a Max value
Not pinning threads to isolated cores	Use isolcpus or cset shield to dedicate cores to the RT workload

Best Practices for Real-Time Linux Kernel Latency Testing

• Always test under realistic, sustained system load — never trust an idle-system result
• Run tests for at least several hours (overnight, if possible) to catch rare worst-case events
• Isolate CPU cores dedicated to real-time work using kernel boot parameters like isolcpus and nohz_full
• Disable unnecessary services, especially anything that might silently run at real-time priority
• Repeat the test after every kernel, firmware, or BIOS update — regressions are common
• Record the exact kernel version, hardware revision, and load profile alongside every result set

Performance Considerations

PREEMPT_RT is not “free” — the additional locking and thread scheduling overhead typically costs some raw throughput compared to a standard kernel, especially on workloads that are not latency-sensitive. This is a deliberate trade-off: you are exchanging a small amount of average-case throughput for a dramatically tighter worst case. Before committing a product to an RT kernel, benchmark your actual application throughput, not just cyclictest numbers, to confirm the trade-off is acceptable for your use case.

Security Considerations

Running processes at real-time priority is powerful, but a runaway real-time task can starve the rest of the system, including administrative and security tooling. The kernel provides a safety valve through /proc/sys/kernel/sched_rt_runtime_us, which reserves a slice of CPU time for non-real-time tasks by default. Avoid disabling this protection in production unless you fully control every real-time task on the system, and always run cyclictest and other RT tooling with the least privilege necessary for the test.

Summary / Key Takeaways

• Real-time Linux kernel latency testing measures the gap between a scheduled wakeup and the actual wakeup time
• PREEMPT_RT is now part of the mainline kernel (since version 6.12, released late 2024), removing the need for out-of-tree patches
• The worst-case (Max) latency, not the average, is what determines real-world determinism
• cyclictest remains the standard measurement tool; rtla timerlat is the newer in-kernel alternative with root-cause tracing
• Meaningful results require realistic background load, long test durations, and proper CPU isolation

Conclusion

Real-time Linux kernel latency testing turns a vague question — “is this system fast enough?” — into a measurable engineering answer backed by data. With PREEMPT_RT now merged into the mainline kernel, building and validating a real-time-capable Linux system is more accessible than it has ever been. Whether you use the battle-tested cyclictest or the newer rtla timerlat tracer, the underlying discipline is the same: apply realistic load, measure honestly, isolate your worst-case number, and design your system around it rather than the average. Master this workflow and you’ll be equipped to validate timing guarantees for embedded Linux, industrial control, robotics, and any other domain where a late response is not an acceptable response — a core skill in our free Linux kernel development course and free embedded systems course tracks.

Frequently Asked Questions (FAQ)

1. What is real-time Linux kernel latency testing used for?
It is used to measure and validate how quickly a Linux system responds to time-critical events, so engineers can confirm a device meets its timing requirements before deployment.

2. Do I need PREEMPT_RT to run cyclictest?
No. cyclictest runs on any Linux kernel. Running it on a standard kernel is actually a useful baseline to compare against a PREEMPT_RT kernel.

3. Is PREEMPT_RT part of the mainline kernel now?
Yes. PREEMPT_RT was merged into mainline Linux starting with kernel version 6.12, released in late 2024, after roughly twenty years of development as an out-of-tree patch set.

4. What is considered a “good” Max latency value?
There is no universal number — it depends entirely on your application’s timing budget. What matters is that the Max value stays comfortably below your required deadline across a long test run under realistic load.

5. Why does the standard kernel sometimes show a better average than PREEMPT_RT?
PREEMPT_RT adds scheduling and locking overhead to guarantee preemptibility, which can slightly raise the average latency while dramatically lowering the worst-case latency — a deliberate and usually worthwhile trade-off for real-time workloads.

6. What is the difference between cyclictest and rtla timerlat?
Both measure the same kind of wakeup latency using a cyclic timer. cyclictest is a long-standing userspace tool; rtla timerlat is a newer, in-kernel tracer that can additionally pinpoint the exact code path responsible for a latency spike.

7. Can I use a Raspberry Pi for real-time Linux kernel latency testing?
Yes. Raspberry Pi boards are commonly used for this kind of testing, particularly for learning and prototyping, though production industrial systems often use hardware with more predictable memory and I/O behaviour.

8. How long should a latency test run?
Worst-case spikes are rare events, so short tests are misleading. Run for at least a few hours, and ideally overnight, before trusting a Max latency figure.

9. Does PREEMPT_RT guarantee hard real-time behaviour?
No. PREEMPT_RT provides soft real-time behaviour with very low, statistically bounded latency. For hard real-time guarantees, a dedicated RTOS or microcontroller is still the appropriate choice.

10. Where does this fit in a free embedded Linux learning path?
Real-time Linux kernel latency testing is typically covered after you understand processes, scheduling, and interrupts — making it a natural module within a free Linux kernel development course or free Linux device drivers course.