« Previous Lecture | Next Lecture »

What You Will Learn

Prerequisites

What Exactly Are We Measuring?

Scheduling latency is the time between an event occurring (for example, a timer firing) and the moment your task actually starts running on the CPU. On a real-time Linux kernel, we care about the worst case ever observed, because a single missed deadline can be a system failure.

Real-Time Linux Latency Testing with cyclictest

cyclictest from the rt-tests suite is the industry-standard tool. It launches high-priority threads that sleep for a fixed interval and measures how late each wakeup actually was.

Install rt-tests

sudo apt install rt-tests        # Debian/Ubuntu
sudo dnf install realtime-tests  # Fedora

Run a Meaningful Test

sudo cyclictest --mlockall --smp --priority=90 --interval=1000 \
     --distance=0 --duration=30m --histogram=400 --quiet

What the important flags mean:

• --mlockall: locks memory so page faults do not pollute the measurement
• --smp: one measurement thread per CPU
• --priority=90: run at high SCHED_FIFO priority, like a real control task
• --histogram=400: record a latency distribution up to 400 µs

Reading the Output

T: 0 ( 2211) P:90 I:1000 C: 1799985 Min:      2 Act:    5 Avg:    6 Max:      38
T: 1 ( 2212) P:90 I:1000 C: 1799942 Min:      2 Act:    4 Avg:    6 Max:      41

Values are microseconds. Ignore Min and Avg. Only Max matters — it is your observed worst case. A well-tuned PREEMPT_RT system on decent x86 or ARM64 hardware typically stays below roughly 50–100 µs Max even under stress; a stock kernel under the same load can spike into the thousands.

The Golden Rule: Always Test Under Load

An idle system responds quickly no matter what kernel it runs — idle numbers prove nothing. Generate aggressive load in a second terminal while cyclictest runs:

sudo apt install stress-ng
stress-ng --cpu $(nproc) --io 4 --vm 2 --vm-bytes 512M --timeout 30m

For production sign-off, industry practice is to run cyclictest for 24 hours or more under worst-case load, then document the hardware, kernel version, config, workloads, and duration alongside the histogram. If your Max stays within your deadline budget across that run, you have engineering evidence — not a guess.

Modern Tooling: rtla and the timerlat Tracer

cyclictest tells you how bad latency is; the newer in-kernel tooling tells you why. The rtla (Real-Time Linux Analysis) utility ships inside the kernel source tree under tools/tracing/rtla/ and drives the kernel’s built-in timerlat, osnoise, and hwnoise tracers.

sudo rtla timerlat top -d 5m

For each CPU, timerlat reports latency at three stages — the timer IRQ, the kernel thread, and a user thread — so you can see exactly which layer introduces the delay. Even better, it can catch the culprit red-handed:

sudo rtla timerlat top --auto 100

This stops tracing the moment any latency exceeds 100 µs and prints the kernel functions responsible. In older workflows, finding a latency source required manual ftrace sessions; rtla automates the hunt. This is exactly the kind of tracing skill we build further in the Ftrace lectures of our free linux device drivers course.

Writing a Correct Real-Time Application

An RT kernel does nothing for a task that never asks for real-time treatment. Every real-time application must do three things: request a real-time scheduling policy, lock its memory, and pre-fault its stack. Here is a minimal, correct skeleton:

#define _GNU_SOURCE
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <pthread.h>
#include <sched.h>
#include <sys/mman.h>
#include <time.h>

#define CYCLE_NS  (1 * 1000 * 1000)   /* 1 ms control cycle */
#define STACK_PREFAULT (8 * 1024)

static void *rt_loop(void *arg)
{
    struct timespec next;
    unsigned char dummy[STACK_PREFAULT];

    memset(dummy, 0, sizeof(dummy));      /* pre-fault the stack */
    clock_gettime(CLOCK_MONOTONIC, &next);

    for (;;) {
        next.tv_nsec += CYCLE_NS;
        while (next.tv_nsec >= 1000000000L) {
            next.tv_nsec -= 1000000000L;
            next.tv_sec++;
        }
        clock_nanosleep(CLOCK_MONOTONIC, TIMER_ABSTIME, &next, NULL);

        /* ---- your deterministic work goes here ---- */
    }
    return NULL;
}

int main(void)
{
    pthread_t tid;
    pthread_attr_t attr;
    struct sched_param sp = { .sched_priority = 80 };

    /* Lock all current and future memory: no page faults at runtime */
    if (mlockall(MCL_CURRENT | MCL_FUTURE)) {
        perror("mlockall");
        exit(1);
    }

    pthread_attr_init(&attr);
    pthread_attr_setschedpolicy(&attr, SCHED_FIFO);
    pthread_attr_setschedparam(&attr, &sp);
    pthread_attr_setinheritsched(&attr, PTHREAD_EXPLICIT_SCHED);

    if (pthread_create(&tid, &attr, rt_loop, NULL)) {
        perror("pthread_create (need root or CAP_SYS_NICE)");
        exit(1);
    }
    pthread_join(tid, NULL);
    return 0;
}

Compile and run:

gcc -O2 -o rt_app rt_app.c -lpthread
sudo ./rt_app

Key points to internalize:

• mlockall() prevents page faults, which are latency poison for real-time tasks.
• clock_nanosleep with TIMER_ABSTIME avoids drift: each cycle is anchored to absolute time, so errors do not accumulate.
• SCHED_FIFO priority 80 puts the loop above almost everything — but below priority-90+ items you may reserve for watchdogs and critical IRQ threads.

Tuning a PREEMPT_RT System

1. CPU Isolation

Reserve one or more cores exclusively for your real-time workload via kernel boot parameters (added in your bootloader configuration):

isolcpus=nohz,domain,3 nohz_full=3 rcu_nocbs=3 irqaffinity=0-2

This example dedicates CPU 3 to real-time work: the general scheduler avoids it, the periodic tick is stopped on it, RCU callbacks are moved off it, and device interrupts are steered to CPUs 0–2. Then pin your application there:

sudo taskset -c 3 ./rt_app

2. IRQ Thread Priorities

Because PREEMPT_RT runs interrupt handlers as threads, you can rank devices by importance. Give the interrupt of your critical device a higher priority than everything else:

pgrep -f "irq/.*-can0"            # find the IRQ thread PID
sudo chrt -f -p 85 <PID>          # raise it to SCHED_FIFO 85

Persist such settings with a udev rule or a systemd unit, so they survive reboots.

3. The RT Throttling Safety Valve

By default the kernel reserves about 5% of CPU time per second for non-real-time tasks, so a runaway SCHED_FIFO loop cannot freeze the machine:

cat /proc/sys/kernel/sched_rt_runtime_us
950000

During development, keep this protection on — it has saved countless engineers from hard lockups. Only on a fully validated production system, where your tasks are proven to yield correctly, should you consider disabling it by writing -1 to that file.

4. Hardware-Level Noise

Some latency sources live below the kernel: System Management Interrupts (SMIs) from firmware, aggressive CPU power states, and simultaneous multithreading. For serious deployments: disable deep C-states and dynamic frequency scaling in firmware or via boot parameters, prefer hardware with documented low SMI activity, and validate with rtla hwnoise.

Best Practices Checklist

Security and Safety Considerations

• Granting real-time priorities requires privilege (CAP_SYS_NICE); expose it narrowly, not by running whole applications as root. Configure limits in /etc/security/limits.conf for dedicated users.
• A malicious or buggy high-priority task is a denial-of-service risk to the entire system — another reason to keep the throttling valve on and to audit which binaries receive RT capabilities.
• For safety-certified domains (medical, avionics, industrial safety), bounded latency evidence from PREEMPT_RT can support certification arguments, but formal processes demand far more than a cyclictest histogram — plan for full evidence packages.

Key Takeaways

Frequently Asked Questions

« Previous Lecture | Next Lecture »