What You Will Learn

• The anatomy of wakeup latency: from hardware event to your first instruction
• Running and interpreting cyclictest correctly (min/avg/max, histograms, stress)
• Using rtla timerlat to measure IRQ and thread latency with automatic root-cause tracing
• Using rtla osnoise to quantify OS interference on isolated cores
• Why the maximum — not the average — is the only number that matters
• A practical checklist for reducing latency once measured

Prerequisites

Lectures 1–5 of this series. A kernel 5.17+ includes the rtla tools (packaged as rtla); PREEMPT_RT from Lecture 5 is ideal but not required — measuring a standard kernel first makes a great before/after experiment for anyone in our free embedded systems course.

What Exactly Are We Measuring?

When a timer fires for your highest-priority thread, several stages occur before your code runs. Each stage contributes latency:

PREEMPT_RT shrinks stages 2–3; firmware quality governs stage 1; your application design governs everything after stage 4.

The Classic Benchmark: cyclictest

cyclictest (from the rt-tests package) runs measurement threads that sleep for a fixed interval and, on each wakeup, record how late they were. Install and run the canonical smoke test:

sudo apt install rt-tests

# One RT thread per core, SCHED_FIFO prio 80, 200 µs interval,
# locked memory, run for ~1 minute
sudo cyclictest --smp -p80 -i200 -m -D1m

Sample output shape:

T: 0 ( 4321) P:80 I:200 C: 299921 Min: 2 Act: 4 Avg: 5 Max: 38
T: 1 ( 4322) P:80 I:200 C: 299899 Min: 2 Act: 3 Avg: 4 Max: 41

Read it right: Max is the verdict. An average of 5 µs with a max of 4 ms fails a 100 µs requirement, period. Two further essentials:

# 1. ALWAYS measure under load — idle numbers are meaningless.
#    In another terminal:
sudo apt install stress-ng
stress-ng --cpu $(nproc) --io 4 --vm 2 --timeout 10m

# 2. Capture a histogram for the full distribution:
sudo cyclictest --smp -p80 -i200 -m -D10m -h 400 -q > hist.txt

Run long — hours for a serious qualification. Rare worst cases (thermal events, background housekeeping) only appear over time.

The Modern Toolset: rtla

cyclictest tells you how bad; the kernel’s own Real-Time Linux Analysis (rtla) tools also tell you why. They wrap in-kernel tracers, so measurement and cause analysis happen in one place.

rtla timerlat — latency with built-in root cause

# Measure IRQ + thread timer latency on CPUs 2-3,
# stop and dump a trace if any thread latency exceeds 100 µs
sudo rtla timerlat top -c 2-3 -T 100 -t

The live display shows separate IRQ-stage and thread-stage latency per CPU — immediately telling you whether delays come from interrupt handling or from scheduling. When the threshold trips, rtla saves a trace of the exact events preceding the spike: the offending driver, IRQ, or kernel path is right there, no manual trace spelunking needed. This is the workflow that replaced hours of Ftrace detective work.

rtla osnoise — how quiet is my isolated core?

# Quantify all OS interference on the isolated CPU 3 for 5 minutes
sudo rtla osnoise top -c 3 -d 5m

osnoise runs a measurement loop and accounts every stolen microsecond to its source: IRQs, NMIs, softirqs, or other threads. On a properly isolated core (Lecture 2) you expect noise near zero; anything else is a tuning to-do list, itemized by the tool.

Choosing between them

Tool	Question it answers	When to reach for it
cyclictest	What is my wakeup latency distribution?	Qualification runs, historical comparability
rtla timerlat	Where is the spike coming from?	Debugging outliers, splitting IRQ vs scheduling delay
rtla osnoise	Who is disturbing my dedicated core?	Validating isolation, polling-style workloads
KernelShark / Trace Compass	What was the whole system doing?	Deep multi-event investigations (Lecture 1)

A Practical Latency-Reduction Checklist

• Measure a baseline under stress; record kernel version, config, hardware and firmware settings.
• Switch to a PREEMPT_RT kernel (Lecture 5) — usually the single biggest max-latency improvement.
• Isolate and pin: dedicate a core via isolcpus/nohz_full or cpuset partition (Lecture 2), pin your task and its IRQ there.
• Prioritize the path: your task and its IRQ thread above competing IRQ threads (Lecture 3).
• Tame the hardware: disable deep C-states and aggressive frequency scaling for the critical cores; audit BIOS for SMI sources.
• Fix the application: mlockall, pre-allocation, no syscalls with unbounded latency in the loop.
• Re-measure after every single change — latency tuning without measurement is guesswork.

Common Mistakes and Troubleshooting

• Quoting the average: requirements are about the worst case; report Max and the histogram tail.
• Short idle runs: ten quiet seconds prove nothing; hours under stress do.
• Measuring with debug kernels: lockdep and similar debug options add huge latencies; measure on production configs.
• VM numbers as truth: hypervisor scheduling distorts everything; qualify on real target hardware.
• Ignoring thermal effects: throttling mid-run creates mysterious spikes; log temperatures alongside latency.

Key Takeaways

• Latency accumulates across hardware, IRQ, and scheduling stages; each has different owners and fixes.
• cyclictest under prolonged stress remains the standard qualification benchmark; the Max column is the verdict.
• rtla timerlat and osnoise measure and explain — automatic root-cause traces on threshold violations.
• Tuning is a loop: measure, change one thing, measure again.

Conclusion

With measurement mastered, the CPU scheduler module of this free Linux kernel programming course is complete: you can observe the scheduler (Lecture 1), place threads (2), prioritize them (3), budget them (4), bound kernel latency (5), and prove the result (6). These six skills together are precisely what employers test for in embedded Linux and real-time roles — and they underpin everything ahead in our free linux device drivers course, where your own driver code becomes the thing being traced and measured. Keep the tools installed; you will use them for the rest of your career.

Frequently Asked Questions (FAQ)

1. What is a “good” cyclictest result?

Whatever satisfies your requirement with margin. As orientation: well-tuned PREEMPT_RT systems on decent hardware commonly show max latencies below ~100 µs under stress, but every board differs — measure yours.

2. cyclictest or rtla — which should I use?

Both: cyclictest for standardized qualification numbers, rtla timerlat when a number is bad and you need the cause.

3. Why must I run stress alongside the measurement?

Worst-case paths (lock contention, IRQ storms, memory pressure) only trigger under load. Idle measurements systematically understate the maximum.

4. Do I need a PREEMPT_RT kernel to use these tools?

No. They run on any modern kernel — which makes the standard-vs-RT before/after comparison an excellent exercise in this free linux kernel development course.

5. What latency does rtla timerlat’s “IRQ” vs “Thr” distinguish?

IRQ latency is hardware event to interrupt handler start; thread latency extends to the measuring thread actually running — isolating stage 2 vs stage 3 of the anatomy above.

6. My max latency spikes once per hour — how do I catch it?

Run rtla timerlat with a threshold (-T) slightly below the spike and let it trace automatically; the saved trace names the culprit at the moment of violation.

7. Are these microsecond numbers meaningful inside containers or VMs?

Containers on a tuned host: yes, with care. VMs: only with real-time-tuned hypervisors (dedicated pinned vCPUs); otherwise measure on bare metal.