Step 1: Check What Your Kernel Already Supports

Before installing anything, check your current kernel version and whether real-time scheduling support is already built in.

uname -r

If the reported version is 6.12 or newer, real-time support may already be compiled into your kernel, or available as a boot-time option depending on your distribution. Check for it directly:

zcat /proc/config.gz 2>/dev/null | grep PREEMPT_RT
cat /sys/kernel/realtime 2>/dev/null

If /sys/kernel/realtime exists and prints 1, your currently running kernel has real-time scheduling active. If it does not exist, your distribution’s default kernel was likely built without CONFIG_PREEMPT_RT enabled, which is still common even on kernels newer than 6.12 — the option exists in mainline, but distributions choose whether to ship it turned on.

Step 2: Install the Measurement Tools

Install rt-tests, which provides cyclictest, along with stress-ng to generate background load while we measure.

sudo apt update
sudo apt install -y rt-tests stress-ng

The modern rtla tool ships as part of the kernel source tree itself under tools/tracing/rtla, and on many distributions it is also packaged separately.

sudo apt install -y linux-tools-common linux-tools-$(uname -r)
rtla --help

If rtla is not available as a package on your distribution, you can build it directly from your kernel source tree, but for a first measurement, cyclictest alone is enough to get started.

Step 3: Run a Baseline cyclictest Measurement

cyclictest works by repeatedly sleeping for a fixed interval and measuring how much extra time passes beyond that interval before the process actually wakes up and runs. That extra time is the scheduling latency we discussed in the previous lecture.

sudo cyclictest -t1 -p 80 -i 1000 -n -l 20000

Flag	Meaning
-t1	Run 1 measurement thread
-p 80	Run the thread at real-time priority 80
-i 1000	Sleep interval of 1000 microseconds between iterations
-n	Use clock_nanosleep() instead of nanosleep()
-l 20000	Run for 20000 loop iterations, then stop

You will see output ending in a summary line similar in shape to this:

T: 0 (  1234) P:80 I:1000 C:  20000 Min:      4 Act:    6 Avg:    7 Max:     46

Here, Min, Avg, and Max are all in microseconds. Min is your best-case wake-up delay, Avg is the typical case, and Max is your worst-case delay across the entire run — the number that matters most for real-time system design.

Step 4: Repeat the Test Under Realistic Load

A latency test on an idle system is close to meaningless for real-world planning. Always test while the system is doing the kind of work it will actually do in production.

sudo stress-ng --cpu 4 --io 2 --vm 2 --vm-bytes 256M --timeout 60s &
sudo cyclictest -t1 -p 80 -i 1000 -n -l 20000

Run both commands together and compare the Max value against your idle-system baseline. It is normal, and expected, for the worst-case latency to increase noticeably under load. What you are looking for is whether it stays within a range your application can tolerate.

Step 5: Try the Modern Alternative — rtla timerlat

cyclictest has been the industry standard for over a decade, but it runs entirely from user space, which means it cannot separate “how long did the interrupt handler take” from “how long did the scheduler take” from “how long did my application thread take.” The timerlat tracer, accessed through rtla, is built into the kernel itself and can break the latency down stage by stage.

sudo rtla timerlat top -d 60s

This command runs for 60 seconds and displays, per CPU, the latency contributed by the timer interrupt handler, the kernel thread, and the user space thread separately. This makes it far easier to identify exactly where a latency spike is coming from, rather than only seeing a single combined number.

Step 6: Practical Tuning Steps

If your measured worst-case latency is higher than your application needs, these steps typically help, roughly in order of impact:

Tuning Step	Why It Helps
Isolate CPU cores with isolcpus / nohz_full boot parameters	Keeps the scheduler from placing unrelated tasks on your latency-critical core
Pin the critical thread with taskset or CPU affinity APIs	Prevents the task from migrating between cores mid-run, which adds latency
Set the CPU frequency governor to performance mode	Avoids delay caused by the CPU ramping up from a low-power state
Move non-critical interrupts away from the isolated core	Reduces interrupt-driven preemption of your real-time thread
Run the critical thread under the SCHED_FIFO real-time scheduling class	Gives the task priority over normal time-sharing tasks on the run queue

Real-World Use Cases

• Industrial data acquisition: Verifying a sensor sampling loop meets its timing budget before deployment.
• Motion control (for example, LinuxCNC-style systems): Confirming step generation timing stays within tolerance under system load.
• Telecom and edge computing: Validating deterministic packet handling on network function virtualization nodes.
• Audio and broadcast equipment: Checking buffer underrun risk under realistic CPU load.

Common Mistakes and Troubleshooting

Mistake	Fix
Testing only on an idle system	Always test with representative background load using a tool like stress-ng
Assuming any distribution kernel has PREEMPT_RT enabled	Check /sys/kernel/realtime directly instead of assuming
Running cyclictest without sudo	Real-time priority scheduling requires elevated privileges
Chasing a zero-latency result	Some latency is unavoidable; define an acceptable threshold for your application instead

Best Practices

• Always record Min, Avg, and Max — never rely on the average alone.
• Run tests long enough to capture rare worst-case spikes; a few seconds is rarely enough.
• Test on the exact hardware you plan to deploy on, not just a development machine.
• Re-test after every significant kernel, driver, or firmware update.

Performance and Security Considerations

Enabling full PREEMPT_RT behavior trades some raw throughput for predictability — this is expected and generally acceptable for control-oriented workloads. On the security side, remember that granting real-time scheduling privileges to a process also gives it the power to starve other tasks of CPU time if misconfigured. Limit real-time priority to the specific processes that genuinely need it, and use /proc/sys/kernel/sched_rt_runtime_us to reserve CPU time for non-real-time tasks so the system does not become unresponsive.

Summary / Key Takeaways

• PREEMPT_RT has been part of mainline Linux since kernel 6.12, so a separate patch set is often no longer needed.
• cyclictest remains a reliable, simple way to measure worst-case wake-up latency from user space.
• rtla timerlat is the modern, kernel-integrated tool that can break latency down by stage.
• Always measure under realistic load, not on an idle system.
• CPU isolation, IRQ affinity, and governor settings are the highest-impact tuning steps.

Conclusion

Measuring scheduling latency turns an abstract kernel concept into a concrete number you can act on. With mainline PREEMPT_RT support now built into the kernel, getting started with real-time measurement is far more accessible than it used to be — no separate patch download required for most modern kernels. Use cyclictest for quick checks, and reach for rtla timerlat when you need a detailed, stage-by-stage breakdown of where your latency is actually coming from.

Frequently Asked Questions

1. Do I still need to download a separate real-time patch for my kernel?
Not for kernel 6.12 and newer — PREEMPT_RT support is part of mainline. Older kernels, or distributions that ship without it enabled, still require either a separate patch or a distribution-provided real-time kernel package.

2. What is a “good” cyclictest Max value?
There is no universal answer — it depends entirely on your application’s timing budget. A hobby project might tolerate a few hundred microseconds; an industrial control loop might require far tighter numbers.

3. Why does rtla timerlat show different numbers than cyclictest?
They measure related but not identical things. cyclictest measures total user-space wake-up latency, while timerlat can separate out interrupt handling, kernel thread, and user thread latency individually.

4. Can I run these tests on a Raspberry Pi?
Yes. Single-board computers are a common and useful platform for this kind of testing, though results will generally show higher latency than a modern desktop-class CPU.

5. Does enabling PREEMPT_RT slow my system down?
It can slightly reduce raw throughput because the kernel does extra work to remain preemptible. For most control and monitoring workloads, this trade-off is worthwhile; for pure computational throughput workloads, it may not be.

6. Is cyclictest still relevant now that rtla exists?
Yes. cyclictest is simple, widely available, and well understood, making it a great first tool. rtla is complementary and useful when you need deeper diagnostic detail.

7. What does a rising Max value under load actually indicate?
It shows how much your worst-case latency degrades under contention — exactly the scenario your production system is likely to face, which is why testing under load matters more than idle testing.