« Previous Lecture | Next Lecture »

What You Will Learn

• What scheduling latency is and where it comes from
• Running cyclictest correctly and reading Min/Avg/Max output
• Why you must measure under load, and how to generate load with stress-ng
• Using the kernel’s built-in rtla timerlat tracer to find the cause of a latency spike
• Practical tuning: CPU isolation, IRQ affinity, and interrupt thread priorities
• How to interpret results and set realistic expectations

Prerequisites

• A booted PREEMPT_RT kernel from Part 2 (the tools also run on non-RT kernels for comparison — a great exercise)
• Root access on real hardware; VM numbers are not meaningful
• The kernel source tree from Part 2 (for building rtla)

What Exactly Are We Measuring?

Scheduling latency (often called wakeup latency) is the delay between the moment an event says “your highest-priority task must run now” and the moment that task actually executes its first instruction. Several stages contribute to it:

cyclictest measures the sum of all five stages from user space: it arms a timer, sleeps, and compares the intended wakeup time with the actual one. The difference is your latency sample.

Step 1: Install the Tools

# cyclictest ships in the rt-tests package
sudo apt install -y rt-tests stress-ng

# Confirm
cyclictest --help | head -3

Step 2: A First cyclictest Run

Start with the widely used baseline invocation:

# -m lock memory, -S one thread per CPU with sensible defaults,
# -p90 SCHED_FIFO priority 90, -i1000 wake every 1000 us, -D 5m run 5 minutes
sudo cyclictest -m -Sp90 -i1000 -D 5m

Sample output on a tuned RT system looks like this (numbers are microseconds):

T: 0 (1234) P:90 I:1000 C: 299998 Min:      2 Act:    5 Avg:    4 Max:      38
T: 1 (1235) P:90 I:1500 C: 199998 Min:      2 Act:    6 Avg:    5 Max:      41

How to read it: one line per CPU thread. Min, Avg and Act (the latest sample) are interesting, but in real-time engineering the only number that ultimately matters is Max — the worst case observed. A beautiful average with one 800 µs spike means your deadline can be missed.

Step 3: Linux Kernel Latency Measurement Under Load

Here is the rule every professional follows: latency measured on an idle system is meaningless. Worst cases appear when the kernel is busy — interrupts firing, locks contended, caches thrashing. So we hammer the machine while measuring:

# Terminal 1: generate mixed CPU, memory, and I/O load
sudo stress-ng --cpu $(nproc) --vm 2 --vm-bytes 512M --io 4 --timeout 30m

# Terminal 2: measure with a histogram while the storm rages
sudo cyclictest -m -Sp90 -i1000 -h 400 -q -D 30m > hist.txt

The -h 400 option records a histogram of latencies up to 400 µs and -q keeps output quiet until the end. The tail of hist.txt reports the max per CPU. For a serious qualification, teams run this for 24 hours or more — rare worst cases need time to show themselves.

A healthy modern x86 or ARM64 board with a PREEMPT_RT kernel typically shows worst cases in the tens of microseconds under load. If you repeat the same test on your distro’s standard kernel, expect occasional spikes into the hundreds of microseconds or worse — running that comparison yourself is the single most convincing experiment in this whole course.

Step 4: Find the Culprit with rtla timerlat

cyclictest tells you that a spike happened; the kernel’s own tracing can tell you why. Modern kernels include the timerlat and osnoise tracers, with a friendly userspace front-end called rtla living right inside the kernel source you built in Part 2:

# Build and install rtla from your kernel source tree
cd ~/rtkernel/linux-6.12.95/tools/tracing/rtla
sudo apt install -y libtraceevent-dev libtracefs-dev python3-docutils
make && sudo make install

# Top-style live view of timer latency per CPU
sudo rtla timerlat top -u

rtla timerlat shows, for every CPU, the latency seen at three levels — the timer IRQ, a kernel thread, and a user thread — which immediately tells you at which layer delay is being added. Its killer feature is automatic blame assignment:

# Stop tracing the moment any thread latency exceeds 100 us,
# and print the trace showing exactly what blocked it
sudo rtla timerlat top -u -T 100 -t

When the threshold trips, rtla dumps the events leading up to the spike — you will see the offending interrupt, softirq, or task by name. This replaces hours of manual ftrace detective work. A sibling command, rtla osnoise, quantifies how much “operating system noise” steals time from an isolated CPU, which is exactly what HPC and telecom engineers care about.

Step 5: Tuning Knobs That Actually Matter

CPU isolation

Reserve one or more cores exclusively for your real-time work by adding kernel boot parameters (edit /etc/default/grub, then update-grub):

# Example: isolate CPUs 2 and 3 on a 4-core machine
GRUB_CMDLINE_LINUX_DEFAULT="... isolcpus=2,3 nohz_full=2,3 rcu_nocbs=2,3"

# Pin your RT application to the isolated cores
sudo taskset -c 2 ./my_rt_app

IRQ affinity and thread priorities

Keep noisy device interrupts away from your isolated cores, and raise the priority of the interrupt thread your application depends on:

# Steer IRQ 130 to CPUs 0-1 only (bitmask 3 = 0b0011)
echo 3 | sudo tee /proc/irq/130/smp_affinity

# Interrupt threads default to SCHED_FIFO 50; raise the one you rely on
sudo chrt -f -p 80 <pid_of_irq_thread>

Application-side habits

• Always mlockall(MCL_CURRENT | MCL_FUTURE) and pre-fault your buffers — a page fault in the control loop is a self-inflicted latency spike.
• Pre-allocate memory at startup; never call malloc() in the time-critical path.
• Use clock_nanosleep() with TIMER_ABSTIME for periodic loops to avoid drift.

Common Mistakes in Latency Measurement

• Measuring an idle system and publishing the number. Meaningless — always load the machine.
• Short runs. Five minutes finds common cases; qualification needs many hours.
• Measuring in a VM and blaming the kernel. Host scheduling dominates; use bare metal.
• Ignoring firmware. System Management Interrupts (SMIs) on x86 bypass the kernel completely; a BIOS setting can ruin an otherwise perfect setup. rtla hwnoise helps detect this.
• Running the measurement tool at low priority. cyclictest must run above everything it competes with, hence -p90.
• Comparing Max values across different hardware as if they were kernel properties. Latency is a property of the whole platform: silicon, firmware, kernel, and config together.

Best Practices Checklist

• Define your deadline first, then measure against it — “low latency” is not a requirement, “< 150 µs worst case” is.
• Automate: keep your cyclictest + stress-ng invocation in a script and rerun it after every kernel, firmware, or config change.
• Store histograms with dates and kernel versions; regressions hide in the tail.
• Use rtla timerlat with a threshold in day-to-day debugging; use long cyclictest runs for sign-off.
• Document the boot cmdline, IRQ affinities, and thread priorities alongside your results.

Key Takeaways

• Linux kernel latency measurement is about the worst case (Max), never the average.
• cyclictest under stress-ng load, for a long duration, is the standard sign-off method.
• rtla timerlat and osnoise are built into the kernel tree and can name the exact cause of a spike.
• CPU isolation, IRQ affinity, memory locking, and interrupt thread priorities are the tuning levers with the biggest payoff.
• Latency belongs to the whole platform — hardware, firmware, kernel, and configuration together.

Conclusion

With this lecture you have closed the loop of real-time Linux engineering: theory (Part 1), construction (Part 2), and now proof (Part 3). You can build a PREEMPT_RT kernel, load it realistically, capture latency histograms, chase down individual spikes with rtla, and tune the system until the worst case fits inside your deadline. These are exactly the skills embedded companies interview for, and everything you used is free and open source — fitting for a free Linux kernel development course. From here, natural next steps in our course are kernel tracing with ftrace and trace-cmd (which pairs beautifully with rtla) and writing your first character device driver in the free Linux device drivers course track.

Frequently Asked Questions

What is a good cyclictest result?

It depends entirely on hardware and requirements. On well-tuned modern x86/ARM64 boards with PREEMPT_RT, worst cases of a few tens of microseconds under load are commonly achievable. The right question is always: is Max below your deadline with margin?

How long should I run cyclictest?

Minutes for quick sanity checks, hours for tuning sessions, and 24 hours or more under realistic load for qualifying a production system. Rare worst cases need time to appear.

Can I run cyclictest on a normal (non-RT) kernel?

Yes, and you should — comparing the same test on a stock kernel and your PREEMPT_RT kernel is the most instructive experiment in this series.

What is the difference between cyclictest and rtla timerlat?

cyclictest is a user-space sampler: great for long statistical runs and histograms. rtla timerlat drives an in-kernel tracer: it measures at IRQ, kernel-thread, and user-thread level and can automatically capture a trace showing what caused a spike. Use both.

Why are my numbers terrible in VirtualBox/VMware?

Because the host scheduler sits between your guest and the hardware. Guest latency reflects host behaviour, not your kernel. Measure on bare metal.

What are SMIs and why do they matter?

System Management Interrupts are firmware-level interrupts on x86 that the kernel cannot see or preempt. A single SMI can stall a CPU for hundreds of microseconds. Check BIOS settings and use rtla hwnoise to detect them.

Does CPU isolation waste cores?

You dedicate cores to determinism instead of throughput — that is a design decision, not waste. On a 4-core industrial board, giving 2 cores to the control loop and 2 to housekeeping is a very common split.

« Previous Lecture | Next Lecture »