« Previous Lecture | Next Lecture »

What You Will Learn

• The scheduling policies available on modern Linux kernels and when each one applies
• How to identify real-time threads on a live system
• Using chrt(1) to query and set a thread’s policy and priority
• The sched_getattr(2) and sched_setattr(2) system calls that power these tools
• Why the SCHED_RESET_ON_FORK flag exists and what security problem it solves
• The role of the CAP_SYS_NICE capability in privileged scheduling operations

Prerequisites

You should be comfortable with basic Linux command-line usage and have a working knowledge of processes and threads. If you are new to these topics, start with the earlier lessons in this free Linux kernel development course, then return here. A Linux machine (physical, virtual, or WSL2) running any recent distribution is enough to practice every example.

Scheduling Policies in Linux CPU Scheduler Internals

The first building block of Linux CPU scheduler internals is the policy. Every thread on a Linux system carries a scheduling policy, which tells the kernel how that thread should compete for CPU time. Modern kernels (6.6 and later) provide the following policies:

Policy	Class	Priority Range	Typical Use
SCHED_OTHER (a.k.a. SCHED_NORMAL)	Fair (EEVDF)	nice −20 to +19	Default for almost all tasks
SCHED_BATCH	Fair (EEVDF)	nice −20 to +19	CPU-heavy background jobs
SCHED_IDLE	Fair (EEVDF)	n/a	Runs only when CPU is otherwise idle
SCHED_FIFO	Real-time	1 to 99	Run-to-completion real-time work
SCHED_RR	Real-time	1 to 99	Round-robin among equal-priority RT tasks
SCHED_DEADLINE	Deadline (highest)	runtime/deadline/period	Periodic tasks with hard timing needs

Important 2020s update: since kernel 6.6, the fair class is implemented by EEVDF (Earliest Eligible Virtual Deadline First), replacing the long-serving CFS. From kernel 6.12, a new extensible class called sched_ext even allows schedulers to be written in BPF. The policy names and the tooling you learn below, however, remain fully compatible.

Spotting Real-Time Threads on a Live System

Before going deeper into Linux CPU scheduler internals, let us look at real threads on a real machine.

A handful of kernel threads on any Linux box run with a real-time policy. Most of them hold SCHED_FIFO, and a few specialized ones (such as migration threads) sit at the maximum real-time priority of 99. You can list every thread’s policy and RT priority with standard tooling:

$ ps -eLo pid,tid,cls,rtprio,comm | awk '$3=="FF" || $3=="RR"'
   PID    TID CLS RTPRIO COMMAND
    15     15  FF     99 migration/0
    16     16  FF     50 idle_inject/0
    20     20  FF     99 migration/1
    82     82  FF     50 irq/25-pciehp

Here CLS shows the class: TS means the normal time-sharing (fair) class, FF means SCHED_FIFO, RR means SCHED_RR, and DLN means SCHED_DEADLINE. Do these priority-99 threads hog the CPU? Not at all — they sleep almost all the time. The real-time policy simply guarantees that when the kernel wakes them, they get the processor immediately and keep it until their short job is done. This is close in spirit to an RTOS, though a stock Linux kernel does not offer the hard determinism an RTOS guarantees (the PREEMPT_RT configuration, merged mainline in kernel 6.12, narrows this gap considerably).

A word of caution about scripting such queries in shell: every external command inside a script (awk, grep, cut, and friends) costs a fork–exec–wait cycle plus context switches, so shell pipelines inside loops are convenient for exploration but poor for performance. The tuna(8) utility is a nicer interactive alternative — it can query and set scheduling policy, priority, CPU affinity, and even IRQ affinity from one place.

Using chrt to Query and Set Policy — Linux CPU Scheduler Internals in Practice

The chrt(1) utility from util-linux is the standard way to work with scheduling attributes. Querying is unprivileged:

$ chrt -p 1
pid 1's current scheduling policy: SCHED_OTHER
pid 1's current scheduling priority: 0

Setting a real-time policy, however, is a sensitive operation. You need either root privileges or, preferably on modern systems, the CAP_SYS_NICE capability granted to the binary or service:

# Start a program under SCHED_FIFO, RT priority 40
$ sudo chrt -f 40 ./my_rt_worker

# Change a running thread (TID 2345) to SCHED_RR, priority 20
$ sudo chrt -r -p 20 2345

# Move it back to the normal fair class
$ sudo chrt -o -p 0 2345

For services, prefer capabilities over full root. With systemd you can simply add AmbientCapabilities=CAP_SYS_NICE to the unit file, or set the capability on the binary:

$ sudo setcap cap_sys_nice+ep ./my_rt_worker
$ getcap ./my_rt_worker
./my_rt_worker cap_sys_nice=ep

SCHED_RESET_ON_FORK: A Security Flag in Linux CPU Scheduler Internals

Imagine a privileged real-time process that forks children. Without safeguards, every child would inherit the real-time policy, and a buggy or malicious child could starve the whole system. The SCHED_RESET_ON_FORK flag closes this hole: when it is ORed into the scheduling policy, any child created via fork(2) is reset to the default SCHED_OTHER policy instead of inheriting the privileged one.

# chrt exposes the flag via -R (must be combined with an RT policy)
$ sudo chrt -R -f 30 ./rt_parent_that_forks

Under the Hood: sched_getattr() and sched_setattr()

How does chrt actually do its job? The scheduling policy and priority live inside each thread’s task structure in kernel virtual address space, so a user-space process cannot touch them directly — it must cross into the kernel through a system call. Modern kernels provide a unified pair: sched_getattr(2) to query and sched_setattr(2) to set every scheduling attribute, including the SCHED_DEADLINE parameters that older calls like sched_setscheduler(2) cannot express. Tracing chrt with strace(1) confirms it:

$ strace -e trace=sched_getattr chrt -p $$
sched_getattr(4321, {size=56, sched_policy=SCHED_OTHER,
    sched_flags=0, sched_nice=0, sched_priority=0,
    sched_runtime=0, sched_deadline=0, sched_period=0}, 56, 0) = 0
pid 4321's current scheduling policy: SCHED_OTHER
pid 4321's current scheduling priority: 0

Notice the struct sched_attr being filled in by the kernel. You can call it yourself; glibc does not wrap these calls, so use syscall(2):

#define _GNU_SOURCE
#include <stdio.h>
#include <string.h>
#include <unistd.h>
#include <sys/syscall.h>
#include <linux/sched.h>
#include <linux/sched/types.h>

int main(void)
{
    struct sched_attr attr;

    memset(&attr, 0, sizeof(attr));
    attr.size = sizeof(attr);
    attr.sched_policy = SCHED_FIFO;
    attr.sched_priority = 25;
    /* reset children to SCHED_OTHER on fork (security) */
    attr.sched_flags = SCHED_FLAG_RESET_ON_FORK;

    if (syscall(SYS_sched_setattr, 0 /* self */, &attr, 0) == -1) {
        perror("sched_setattr");
        return 1;
    }
    printf("Now running under SCHED_FIFO, RT prio 25\n");
    /* ... real-time work here ... */
    return 0;
}

Compile and run it with the CAP_SYS_NICE capability (or via sudo). The man page sched(7) documents the full family of scheduler-related system calls and is worth bookmarking.

Common Mistakes and Troubleshooting

• EPERM when setting an RT policy: you lack root or CAP_SYS_NICE. Also check the RLIMIT_RTPRIO resource limit (ulimit -r).
• An RT task freezes the machine: a spinning SCHED_FIFO task can starve everything below it on that CPU. Keep RT work short, always test on one core first (taskset -c 1 ...), and know that RT throttling (sched_rt_runtime_us) reserves a small slice for non-RT tasks by default.
• Passing a priority with SCHED_OTHER: the fair policies take a nice value, not an RT priority; the RT priority must be 0 for them.
• Wrong struct size to sched_setattr: always set attr.size = sizeof(attr), or the call fails with E2BIG/EINVAL across kernel versions.

Best Practices

• Prefer capabilities (CAP_SYS_NICE) over running as full root.
• Use SCHED_RESET_ON_FORK in any RT program that forks.
• Choose SCHED_DEADLINE for genuinely periodic real-time work; it is admission-controlled and safer than hand-picked FIFO priorities.
• Reserve priorities above 50 for the kernel’s own threads unless you know exactly why you need more.

Key Takeaways

• Linux CPU scheduler internals expose fair (EEVDF), real-time (FIFO/RR), and deadline scheduling policies.
• chrt(1) is the go-to tool; it is built on sched_getattr(2)/sched_setattr(2).
• Changing policy/priority is privileged — root or CAP_SYS_NICE.
• SCHED_RESET_ON_FORK stops children from inheriting privileged policies.
• RT kernel threads at priority 99 sleep most of the time; they simply win the CPU instantly when woken.

Frequently Asked Questions

Conclusion

You can now inspect and control the scheduling behavior of any thread on a modern Linux system — from a quick chrt -p PID to a full programmatic sched_setattr() with security flags. These skills are used daily in embedded Linux, audio, robotics, and telecom work, and they form the foundation for the next lesson in this free Linux kernel development course, where we answer a deceptively simple question: who actually runs the scheduler’s code, and when? This free embedded systems course continues in Part 2.

« Previous Lecture | Next Lecture »