What Do We Actually Mean by “Latency” Here?

In a real-time context, latency usually refers to wake-up latency: the time between when an event should trigger a task (a timer expiring, an interrupt firing) and the moment that task actually starts executing on the CPU. A real-time system isn’t judged by its average latency — it’s judged by its worst-case latency, because a single missed deadline can be a failure even if every other cycle was fast.

Measuring Latency with cyclictest

cyclictest, part of the rt-tests package, has been the de facto standard latency benchmark for real-time Linux for many years. It works by repeatedly arming a timer for a fixed interval and measuring how late the wake-up actually arrives compared to when it was scheduled.

Step 1: Install rt-tests

sudo apt install rt-tests
# or, on distributions without a package:
git clone https://git.kernel.org/pub/scm/utils/rt-tests/rt-tests.git
cd rt-tests && make && sudo make install

Step 2: Run a Basic Test

sudo cyclictest -m -Sp90 -i200 -h400 -q

• -m locks memory, avoiding page-fault-related latency spikes during the test itself
• -S runs one measurement thread per CPU
• -p90 sets the real-time priority of the test threads
• -i200 sets the timer interval to 200 microseconds
• -h400 builds a histogram with a 400-microsecond ceiling
• -q suppresses per-cycle output, printing only a final summary

Step 3: Add System Load While Testing

An idle-system latency number tells you almost nothing about real-world behavior. Always stress the system while measuring:

sudo apt install rt-tests hackbench
sudo hackbench -l 100000 &
sudo cyclictest -m -Sp90 -i200 -h400 -q

Step 4: Read the Summary Output

# T: 0 (  1234) P:90 I:200 C: 500000 Min:      4 Act:    9 Avg:    7 Max:      62

The important field is Max — the worst single latency observed, in microseconds, across the whole run. This is the number you compare against your application’s deadline budget, not the average.

The Modern Alternative: rtla and the timerlat Tracer

Alongside the PREEMPT_RT mainline merge, the kernel gained a built-in Real-Time Linux Analysis tool called rtla, located under tools/tracing/rtla/ in the kernel source tree. It talks directly to a tracer named timerlat, which is compiled into the mainline kernel itself rather than being an external benchmark process.

cd linux/tools/tracing/rtla
make
sudo ./rtla timerlat top

Unlike cyclictest, timerlat runs as a kernel-native tracer, so it can report not just the final wake-up latency but also break down where the time was spent — inside the IRQ handler, inside the scheduler, or waiting for the thread to actually run — which makes root-causing a bad number much faster.

Older-Style Kernel Tracing with Ftrace

Before dedicated tools like timerlat existed, engineers relied directly on Ftrace’s latency tracers to catch the exact code path responsible for a spike:

cd /sys/kernel/debug/tracing
echo wakeup_rt > current_tracer
echo 1 > tracing_on
sleep 5
echo 0 > tracing_on
cat trace > /tmp/wakeup_trace.txt

The wakeup_rt tracer specifically records the highest-latency wake-up of a real-time task during the tracing window, along with the full call stack that led to it — invaluable when timerlat or cyclictest tells you that a spike happened but not why.

Common Mistakes and Troubleshooting

Symptom	Likely Cause & Fix
Great numbers idle, terrible numbers under load	Expected — always report load-tested numbers, never idle-only numbers.
Very high, irregular latency spikes (100s of microseconds)	Often firmware-level System Management Interrupts (SMI); check with turbostat or update firmware/BIOS settings.
Latency worsens over a long test run	CPU frequency scaling or thermal throttling — pin frequency governors to “performance” during testing.
Results vary wildly between runs	Background services with elevated priority (like some NTP daemons) competing for the CPU — check with ps -eo pid,rtprio,cmd.

Best Practices

• Always test under representative system load, not an idle machine.
• Run tests long enough to catch rare outliers — minutes, not seconds.
• Isolate the CPU cores your real-time threads use with isolcpus and irqaffinity before measuring.
• Cross-check a suspicious cyclictest result with the timerlat tracer or Ftrace’s wakeup_rt tracer to find the root cause, not just the symptom.

Performance and Security Considerations

Performance: tracing itself adds a small overhead; use it to diagnose, then disable tracers before final production benchmarking.

Security: both cyclictest and rtla require root or elevated capabilities to set real-time priorities and access tracing infrastructure — treat any script that automates these tools with the same care as other privileged tooling.

Key Takeaways

• Real-time systems are judged by worst-case latency, not average latency.
• cyclictest remains the standard, portable benchmark for measuring wake-up latency under load.
• rtla timerlat is the newer, kernel-native alternative that also helps explain where latency came from.
• Ftrace’s wakeup_rt tracer is still the go-to tool for root-causing a specific bad wake-up event.

Conclusion

Measuring latency correctly — with realistic load, long enough test windows, and the right tool for the question you’re asking — is what turns “I built an RT kernel” into “I can prove my system meets its deadline.” This wraps up the three-part series on real-time Linux in this free Linux kernel programming and device drivers course; from here, the natural next step is applying these same techniques to a real driver or application you’re building.

Frequently Asked Questions