← Previous Lecture | Next Lecture →
- What cyclictest measures, and why latency numbers matter more than averages
- How to install and run cyclictest correctly on a Raspberry Pi
- How to stress a system realistically while measuring latency
- How to read minimum, average, and maximum latency values, and build a histogram
- How to compare a standard kernel against a PREEMPT_RT kernel with real numbers
In the previous lecture of this free Linux kernel development course, you built and booted a PREEMPT_RT kernel on a Raspberry Pi. A kernel that boots with PREEMPT_RT active is not proof that it behaves well under load — you need to measure it. This lecture covers cyclictest, the standard tool the real-time Linux community uses to quantify scheduling latency.
What Does cyclictest Actually Measure?
cyclictest repeatedly asks the kernel to wake it up after a fixed time interval, then measures the difference between the requested wake-up time and the actual wake-up time. That difference is the scheduling latency — the delay between “the kernel should have run this task” and “the kernel actually ran this task.” Doing this thousands of times per second, across all CPU cores, while the system is under artificial load, builds a statistically meaningful picture of your system’s worst-case behavior.
Step 2: Kernel scheduler actually wakes the thread at time T + interval + delay
Step 3: cyclictest records “delay” as one latency sample
Step 4: Repeat thousands of times per core, then report Min / Avg / Max delay
Prerequisites
- A Raspberry Pi with the PREEMPT_RT kernel from the previous lecture (or any Linux system, for comparison purposes)
- Root or sudo access, since cyclictest needs real-time scheduling privileges
- A basic understanding of what “latency spike” and “worst case” mean, covered in Lecture 1 of this series
Step 1: Install cyclictest and Load-Generation Tools
cyclictest ships as part of the rt-tests package, and stress-ng generates
realistic CPU, memory, and I/O load while the test runs:
sudo apt update
sudo apt install -y rt-tests stress-ng gnuplot
gnuplot is optional and only needed if you want to render a latency histogram graph from the
results instead of just reading the raw numbers.
Step 2: Run a Quick Baseline Test
Start with a short, simple run to confirm the tool works and to get a first impression of your system:
sudo cyclictest -t 4 -p 80 -i 1000 -n -l 30000| Flag | Meaning |
|---|---|
-t 4 |
Run 4 test threads, one per CPU core on a quad-core Pi |
-p 80 |
Run threads at real-time priority 80 |
-i 1000 |
Request a wake-up every 1000 microseconds (1 ms) |
-n |
Use POSIX clock_nanosleep for more accurate timing |
-l 30000 |
Run for 30,000 loop iterations, then stop |
Step 3: Test Under Realistic System Load
A latency test with an idle system is not meaningful — real-world systems are busy. Run
stress-ng in one terminal to load the CPU, memory, and disk I/O, while cyclictest measures
latency in another:
# Terminal 1: generate load
stress-ng --cpu 4 --io 2 --vm 2 --vm-bytes 128M --timeout 300s
# Terminal 2: measure latency while under load
sudo cyclictest -t 4 -p 90 -i 200 -h 400 -m -q > cyclictest_output.txt| Flag | Meaning |
|---|---|
-h 400 |
Build a histogram with 400 buckets, one per microsecond of latency |
-m |
Lock memory to prevent page faults from distorting the results |
-q |
Suppress live console output, keeping only the final summary |
Step 4: Read the Results Correctly
A cyclictest summary line reports three numbers per thread: Min, Avg, and Max latency, all in microseconds. The number that matters most for real-time work is Max — a single worst-case spike is what breaks a hard deadline, even if the average looks excellent.
| Metric | What It Tells You | Priority When Judging a System |
|---|---|---|
| Min | Best-case scheduling delay | Low |
| Avg | Typical scheduling delay | Medium |
| Max | Worst observed scheduling delay | Highest — this is what breaks deadlines |
Step 5: Compare a Standard Kernel Against a PREEMPT_RT Kernel
The most convincing way to understand PREEMPT_RT’s value is a side-by-side comparison. Run the identical stress-ng and cyclictest commands once on your standard Raspberry Pi OS kernel, and once after booting into the PREEMPT_RT kernel from the previous lecture, then compare the Max column:
| Kernel | Typical Behavior Under Load |
|---|---|
| Standard kernel | Low average latency, but occasional large spikes measured in milliseconds |
| PREEMPT_RT kernel | Slightly higher average, but spikes stay tightly bounded, typically in the tens to low hundreds of microseconds |
Actual numbers vary significantly by hardware, firmware, and connected peripherals — always measure your own target device rather than relying on numbers from a different board.
Step 6: Generate a Latency Histogram (Optional)
With the -h flag, cyclictest writes bucketed latency-frequency data alongside the summary.
Feed that output into gnuplot to render a visual histogram, which makes it far easier to spot
a long tail of rare, large spikes that a plain average would hide.
Common Mistakes When Running cyclictest
- Testing an idle system: always apply realistic load with stress-ng, otherwise the numbers are meaningless
- Forgetting
sudo: cyclictest needs real-time scheduling privileges to set thread priority - Running too short a test: rare latency spikes may only appear after several minutes or hours of continuous testing
- Comparing across different hardware: latency numbers from one board are not directly comparable to another board with different firmware and peripherals
- Ignoring the Max column: focusing only on the average hides the worst-case behavior that actually matters for deadlines
Best Practices for Reliable Latency Testing
- Run tests for at least several hours for production-grade confidence, not just a few seconds
- Always test under representative CPU, memory, and I/O load, not an idle system
- Disable CPU frequency scaling to the performance governor during tests, since frequency transitions add jitter
- Repeat the test multiple times to confirm results are consistent, not a one-off fluke
- Record the exact kernel version, board revision, and firmware version alongside every result set
Performance and Security Considerations
Running cyclictest at high real-time priority for extended periods can, in rare cases, starve other system
tasks if sched_rt_runtime_us is not configured with a sane limit — always leave a portion of
each second reserved for non-real-time tasks during testing. From a security standpoint, treat latency test
results as internal diagnostic data; publishing exact production timing characteristics publicly can, in
sensitive industrial contexts, reveal information about system design that is best kept internal.
Summary: Key Takeaways
- cyclictest measures the gap between requested and actual thread wake-up time — the real definition of scheduling latency
- Always test under realistic load using stress-ng, never on an idle system
- The Max latency value matters more than the average for real-time deadline analysis
- Comparing a standard kernel against a PREEMPT_RT kernel with identical test parameters is the clearest way to see the benefit
Conclusion
You now have a complete, practical workflow across this three-part free Linux kernel development course: understanding what real-time Linux means, building a verified PREEMPT_RT kernel for Raspberry Pi, and objectively measuring its latency behavior with cyclictest. This same workflow scales directly to production embedded Linux projects in robotics, industrial control, and audio, where proving your latency numbers — not just assuming them — is what separates a real-time system from a system that merely feels fast.
Frequently Asked Questions
1. What unit does cyclictest report latency in?
Microseconds, by default, shown as Min, Avg, and Max values per test thread.
2. Why does cyclictest require sudo?
It needs permission to assign real-time scheduling priorities to its threads, which is a privileged operation on Linux.
3. What is a “good” maximum latency number?
It depends entirely on your application’s deadline requirements; there is no universal target, but well-tuned PREEMPT_RT embedded systems commonly stay in the tens to low hundreds of microseconds under load.
4. Should I trust results from a test that ran for only a few seconds?
No. Rare worst-case spikes often only appear over longer runs, so production-grade testing should run for hours, not seconds.
5. Can I run cyclictest on a standard, non-PREEMPT_RT kernel?
Yes, and doing so is a useful baseline for comparison against your PREEMPT_RT results.
6. What does the -m flag do in cyclictest?
It locks the process’s memory, preventing page faults from adding unrelated latency into the measurement.
7. Is stress-ng required to get valid results?
Some form of realistic system load is required; stress-ng is simply a convenient, well-maintained tool for generating it.
You’ve completed the Real-Time Linux mini-series. Explore more free courses on Linux kernel programming and embedded systems.
Browse All Free Courses Previous Lecture
2 Comments