What Is Linux Scheduler Latency?

In simple terms, scheduler latency is the gap between the moment a task becomes runnable (it wakes up, or it is created) and the moment it actually starts executing on a CPU core. During that gap, the task sits in the runqueue, waiting for its turn. On a general-purpose OS (GPOS) like a stock Linux kernel, this wait can stretch from a few microseconds to well over a hundred microseconds under load, because the scheduler is optimized for overall throughput and fairness rather than guaranteed response time. On a system tuned for real-time behavior, this same wait is squeezed down to a small, bounded number, often in the low single-digit microseconds.

This is exactly why Linux scheduler latency measurement is a core skill for embedded engineers: without measuring it, you are only guessing whether your control loop, audio pipeline, or sensor-fusion thread will meet its deadline.

Why Scheduler Latency Matters in Embedded Linux

Accurate Linux scheduler latency measurement is not an academic exercise. Consider these real-world situations:

Use Case	Impact of High Scheduler Latency
Pro-audio production (JACK, PipeWire)	Audible clicks, pops, and buffer underruns (xruns)
Robotics control loops	Jittery motor control, unstable feedback loops
Industrial automation (LinuxCNC, PLC bridges)	Missed machining tolerances, safety margin violations
Telecom edge / 5G RAN processing	Dropped packets, missed transmission windows

eBPF Tools for Linux Scheduler Latency Measurement

Modern Linux scheduler latency measurement leans heavily on eBPF (extended Berkeley Packet Filter), a kernel technology that lets small, verified programs attach to kernel events like context switches without needing a custom kernel module. Two toolkits sit on top of eBPF for this purpose: the BCC (BPF Compiler Collection) tools, and the more lightweight bpftrace scripting language.

Tool	Purpose	Typical Output
runqlat	Summarizes runqueue wait time for every task as a histogram	Latency buckets with counts
runqslower	Reports only tasks that waited longer than a chosen threshold	Per-event line with PID and wait time
runqlen	Samples how many tasks are queued and waiting per CPU	Runqueue length histogram per core

Installing eBPF Tracing Tools

On Debian and Ubuntu-based embedded boards, the BCC toolkit is available directly from the package manager:

sudo apt update
sudo apt install -y bpfcc-tools linux-headers-$(uname -r)

For a lighter footprint and quicker one-liners, install bpftrace instead:

sudo apt install -y bpftrace

Capturing a Scheduler Latency Histogram with runqlat

The single most useful command for Linux scheduler latency measurement is runqlat. It captures how long every task spends waiting before it runs, then buckets the results into a histogram:

sudo runqlat 5 1

Here, 5 is the sampling duration in seconds and 1 means “print once and exit.” The output shows latency ranges in microseconds on the left and a count of how many scheduling events fell into that range, letting you immediately see whether your system’s tail latency is acceptable for your workload.

Filtering for Long Delays with runqslower

Histograms are great for an overall picture, but sometimes you just want to know which specific tasks are experiencing unusually long scheduling delays. That’s exactly what runqslower does — you give it a threshold, and it prints only the events that exceed it:

sudo runqslower 2000

This example only reports scheduling delays above 2000 microseconds, which makes it easy to spot the rare, problematic outliers that a histogram might hide.

Checking Runqueue Occupancy with runqlen

A long runqueue is a leading indicator of upcoming scheduler latency problems. Run runqlen to see, per CPU, how many tasks are typically waiting:

sudo runqlen 5 1

If you consistently see two or more tasks queued on a core that is supposed to be reserved for a real-time workload, that’s a strong signal you need to review your CPU affinity and isolation setup.

A Modern Alternative: bpftrace One-Liners

If BCC feels heavy for a quick check, bpftrace lets you write a compact scheduler latency probe directly on the command line. The following one-liner tracks time spent between a task being woken and being switched onto the CPU:

sudo bpftrace -e '
tracepoint:sched:sched_wakeup { @qtime[args->pid] = nsecs; }
tracepoint:sched:sched_switch {
    if (@qtime[args->next_pid]) {
        @latency_ns = hist(nsecs - @qtime[args->next_pid]);
        delete(@qtime[args->next_pid]);
    }
}'

This kind of scriptable Linux scheduler latency measurement is extremely handy when you need a custom probe that the stock BCC tools don’t provide out of the box.

Detecting Real-Time Tasks in Kernel Code

Beyond user-space tracing, kernel and driver code sometimes needs to know, programmatically, whether the currently running (or a given) task is a real-time task. The Linux scheduler exposes a small set of inline helper functions for exactly this purpose.

Function	What It Checks
rt_prio(prio)	Takes a raw priority value and returns whether it falls in the real-time priority range
rt_task(task)	Takes a task_struct pointer and checks its priority via rt_prio()
task_is_realtime(task)	Checks the task’s scheduling policy (SCHED_FIFO/SCHED_RR/SCHED_DEADLINE) instead of raw priority

Here’s a simple, original example of using these helpers inside a kernel module to log whether the calling task is real-time:

#include <linux/sched.h>
#include <linux/sched/rt.h>

static void log_task_class(struct task_struct *tsk)
{
    if (task_is_realtime(tsk))
        pr_info("ep_demo: task %s (pid %d) is real-time\n",
                tsk->comm, tsk->pid);
    else
        pr_info("ep_demo: task %s (pid %d) is a normal task\n",
                tsk->comm, tsk->pid);
}

Prefer task_is_realtime() over rt_task() in new code, since it checks the actual scheduling policy rather than only the numeric priority range, which makes it accurate for SCHED_DEADLINE tasks as well.

2026 Update: PREEMPT_RT Is Now Part of Mainline Linux

Older material on this subject usually describes the real-time patch set, PREEMPT_RT, as something you had to download separately and apply on top of the kernel source. That changed in September 2024: PREEMPT_RT was merged into mainline Linux starting with kernel 6.12, so the “convert a GPOS into a true RTOS” capability now ships in the stock kernel source for x86, x86_64, ARM64, and RISC-V. You still need to explicitly enable CONFIG_PREEMPT_RT at build time (most distro stock kernels still leave it off by default), but you no longer need to hunt down and apply an out-of-tree patch series just to get started. This matters for Linux scheduler latency measurement because it means the same eBPF tools covered above now work identically whether you’re profiling a standard kernel or a fully preemptible real-time build.

Common Mistakes When Measuring Scheduler Latency

Mistake	Fix
Running only a single short trace and trusting the number	Collect data over a long period with a realistic workload before drawing conclusions
Testing on an idle system	Always reproduce production-like CPU and I/O load while tracing
Forgetting to pin the workload with CPU affinity	Isolate cores for latency-sensitive tasks before measuring
Comparing histograms across different kernel configs without noting them	Always record the kernel version, preemption model, and CONFIG_HZ value alongside results

Best Practices for Linux Scheduler Latency Measurement

Performance and Security Considerations

Performance: eBPF programs run in a sandboxed, JIT-compiled context, so tracing overhead is generally low, but attaching probes to very hot tracepoints like sched_switch on a busy multi-core system does add measurable CPU cost. Keep trace windows short and targeted in production.

Security: Most scheduler tracing tools require root privileges or the CAP_BPF and CAP_PERFMON capabilities, and some hardened systems run with kernel lockdown mode, which restricts what eBPF programs can attach to. Always verify your target device’s security policy before deploying eBPF-based Linux scheduler latency measurement tools on a production fleet.

Summary: Key Takeaways

Conclusion

Linux scheduler latency measurement is the difference between assuming your embedded system meets its timing requirements and actually proving it. With eBPF-based tools like runqlat, runqslower, and runqlen, plus kernel-level helpers like task_is_realtime(), you now have a complete, modern toolkit for diagnosing scheduling delays on both standard and real-time-enabled Linux kernels. Practice these tools on your own board, compare a stock kernel against a PREEMPT_RT build, and you’ll build the kind of hands-on intuition that separates a working embedded Linux engineer from a theoretical one.

Frequently Asked Questions