Kernel Thread Scheduling in Linux: sched_set_fifo() and the Modern API

What You Will Learn

• Why kernel threads are ordinary schedulable entities with a policy and priority
• What changed in kernel 5.9 and why the old scheduler APIs were removed from module reach
• The modern API set: sched_set_fifo(), sched_set_fifo_low(), sched_set_normal()
• How threaded interrupt handlers get their famous FIFO priority 50
• A complete, buildable kernel module that creates a real-time kernel thread
• How to verify kernel thread scheduling from user space

Prerequisites

You should know how to build and load a basic kernel module (covered in the Linux kernel module programming lessons of this free embedded systems course) and be familiar with the scheduling policies from the previous lesson. A test virtual machine is strongly recommended — we will be creating real-time kernel threads, and mistakes belong in a VM, not on your laptop.

Kernel Threads Are Schedulable Entities Too

The kernel itself is not a process. It is a library of code that runs in process context (when a thread makes a system call) or in interrupt context. But the kernel does own threads: kthreads such as ksoftirqd/N, kworker pool members, migration/N, rcu_preempt, and the per-IRQ handler threads. Each one has a task_struct, shows up in ps, and is scheduled by exactly the same scheduler classes as your applications.

That means each kernel thread also has a scheduling policy and priority — and sometimes the kernel needs to set those deliberately. The classic example: a driver’s threaded interrupt handler must respond fast, so it runs under SCHED_FIFO.

What Changed: The 5.9 Cleanup

Up to kernel 5.8, in-tree code and out-of-tree modules alike commonly called sched_setscheduler() or sched_setscheduler_nocheck() with an explicit struct sched_param, choosing arbitrary FIFO priorities. The scheduler maintainers considered this a mess: dozens of drivers each inventing magic priority numbers with no system-wide reasoning behind them, since static priorities from unrelated subsystems simply cannot be composed meaningfully.

The result, merged in kernel 5.9, was a deliberate API narrowing:

• New helper functions were added that express intent instead of magic numbers.
• The old functions (sched_setscheduler(), sched_setscheduler_nocheck(), sched_setattr() and friends) were un-exported, so loadable modules can no longer call them. Try it and modpost fails your build with an “undefined symbol” error — a very common surprise when porting old drivers to a new kernel.

The modern API set

API	Effect	When to Use
sched_set_fifo(p)	SCHED_FIFO at mid-range priority (50)	Latency-sensitive kernel threads; the standard choice
sched_set_fifo_low(p)	SCHED_FIFO at priority 1	Must beat all normal tasks, but should yield to every other RT task
sched_set_normal(p, nice)	Back to SCHED_NORMAL with the given nice value	Returning a thread to fair (EEVDF) scheduling

Notice what is missing: there is deliberately no parameter for a priority number in the FIFO helpers. The scheduler maintainers’ position is that driver authors should state intent (“this needs RT” or “low RT is fine”) and let the core own the actual numbers. User space administrators can still retune any kernel thread afterwards with chrt, which is the escape hatch if a particular system needs different relative priorities.

Example: The Threaded Interrupt Handler

The best-known user of this machinery is the threaded IRQ infrastructure. When a driver requests a threaded interrupt with request_threaded_irq(), the kernel spawns a dedicated kthread named irq/<nr>-<name> and immediately promotes it with sched_set_fifo(). That is exactly why, on any modern distro, you see those irq/... threads sitting at FIFO priority 50:

$ ps -eLo pid,cls,rtprio,comm | grep '^ *[0-9]* *FF'
   23  FF     50 idle_inject/0
   88  FF     50 irq/9-acpi
  412  FF     50 irq/125-xhci_hcd
  415  FF     50 irq/126-nvme0q1

Priority 50 is the halfway point of the 1–99 real-time range: important enough to preempt every normal task instantly, while leaving headroom above for a system integrator’s own hard-real-time threads.

Hands-On: A Kernel Module That Creates an RT Kernel Thread

Enough theory. The module below creates its own kernel thread, promotes it to SCHED_FIFO with sched_set_fifo(), does a tiny bit of periodic “work”, and cleans up on unload. This is original teaching code for our free Linux device drivers course — build it only inside a test VM.

// rt_kthread_demo.c - create a kernel thread and give it RT policy
// Tested on 6.x kernels. Build with the usual kernel module Makefile.
#include <linux/module.h>
#include <linux/kthread.h>
#include <linux/sched.h>
#include <linux/delay.h>

MODULE_LICENSE("GPL");
MODULE_AUTHOR("EmbeddedPathashala");
MODULE_DESCRIPTION("Demo: kernel thread with SCHED_FIFO via sched_set_fifo()");

static struct task_struct *ep_task;

static int ep_thread_fn(void *data)
{
    pr_info("ep_rtdemo: thread '%s' pid %d policy %d rt_prio %d\n",
            current->comm, current->pid,
            current->policy, current->rt_priority);

    while (!kthread_should_stop()) {
        /* Real drivers block on events; we just nap politely.
         * An RT thread that never sleeps will lock up a CPU! */
        pr_info_ratelimited("ep_rtdemo: tick from RT kthread\n");
        msleep_interruptible(2000);
    }
    pr_info("ep_rtdemo: thread exiting\n");
    return 0;
}

static int __init ep_rtdemo_init(void)
{
    ep_task = kthread_create(ep_thread_fn, NULL, "ep_rtdemo");
    if (IS_ERR(ep_task))
        return PTR_ERR(ep_task);

    /* Modern, module-safe way to request SCHED_FIFO for a kthread.
     * No priority argument: the kernel picks the standard value. */
    sched_set_fifo(ep_task);

    wake_up_process(ep_task);
    pr_info("ep_rtdemo: loaded, kthread started with SCHED_FIFO\n");
    return 0;
}

static void __exit ep_rtdemo_exit(void)
{
    kthread_stop(ep_task);
    pr_info("ep_rtdemo: unloaded\n");
}

module_init(ep_rtdemo_init);
module_exit(ep_rtdemo_exit);

And the matching Makefile:

obj-m += rt_kthread_demo.o

all:
	make -C /lib/modules/$(shell uname -r)/build M=$(PWD) modules

clean:
	make -C /lib/modules/$(shell uname -r)/build M=$(PWD) clean

Build, load, and verify

$ make
$ sudo insmod rt_kthread_demo.ko
$ sudo dmesg | tail -3
[ 6021.402118] ep_rtdemo: thread 'ep_rtdemo' pid 4321 policy 1 rt_prio 50
[ 6021.402131] ep_rtdemo: loaded, kthread started with SCHED_FIFO

# Verify from user space with chrt - same tool as the last lesson!
$ chrt -p 4321
pid 4321's current scheduling policy: SCHED_FIFO
pid 4321's current scheduling priority: 50

$ sudo rmmod rt_kthread_demo

Policy value 1 in the dmesg line is SCHED_FIFO, and the priority the core chose is 50 — the same value the threaded IRQ infrastructure uses. Two lessons, one consistent picture.

What if I really need a custom priority?

From a module, you have three honest options:

• Accept the defaults from the sched_set_fifo*() helpers — correct for nearly all drivers.
• Let user space retune the thread after boot: sudo chrt -f -p 70 <pid> from a systemd unit or udev rule. This is the intended mechanism for system integrators.
• If you genuinely believe a new in-kernel API or exception is needed, that is a conversation for the upstream mailing list — not a hack. Older books demonstrated looking up unexported symbols and calling them through function pointers; on current kernels that fights modpost, kernel lockdown, and every maintainer’s advice. Treat such tricks as historical trivia, never production technique.

Real-World Use Cases

• Threaded IRQs everywhere: touchscreen, NVMe, network, and USB controllers on your machine are already using sched_set_fifo() indirectly.
• Media and audio drivers: capture/playback kthreads that must drain hardware FIFOs on time.
• Thermal and power management: injection/cooling threads that must run promptly even under full system load.
• Watchdog feeders: a feeding kthread that misses its window causes a hardware reset, so it runs RT.

Common Mistakes and Troubleshooting

Symptom	Cause and Fix
modpost: "sched_setscheduler" undefined!	You are porting pre-5.9 driver code. Replace the call with sched_set_fifo() / sched_set_fifo_low() / sched_set_normal().
System stalls; RCU or soft-lockup warnings in dmesg	Your RT kthread loops without sleeping or blocking. Every iteration must end in a sleep, a wait queue, or an event wait.
Thread runs before you set its policy	You used kthread_run(), which wakes the thread immediately. Use kthread_create(), set the policy, then wake_up_process().
rmmod hangs forever	kthread_stop() waits for the thread to notice. Use interruptible sleeps and check kthread_should_stop() every loop iteration.

Best Practices

• Default to a normal-policy kthread; promote to RT only when a measured latency requirement demands it.
• Use sched_set_fifo_low() when your thread merely needs to outrank normal tasks — it keeps you out of the way of the IRQ threads at 50.
• Name your kthreads clearly (the kthread_create() format string); it makes ps-based debugging and chrt retuning trivial for whoever integrates your driver.
• Document in your driver’s comments why the thread is RT, so future maintainers can re-evaluate.
• Test under load: run a kernel build or stress-ng while your driver operates and confirm nothing starves.

Performance and Security Considerations

Performance: every RT kernel thread you add shrinks the guaranteed CPU time available to fair-class tasks. On small embedded cores (single or dual core ARM SoCs, very common in embedded systems course projects), one chatty FIFO kthread can visibly degrade the whole user experience. Measure with ftrace/trace-cmd (see our tracing lessons) before and after promoting a thread.

Security: the un-exporting of the old APIs is itself a security-adjacent hardening — it stops out-of-tree modules from silently claiming arbitrary system-wide priority. Respect the same spirit in your designs: a driver should never assume it is the most important thing on the machine.

Key Takeaways

• Kernel threads are normal schedulable entities; policy and priority apply to them exactly as to user threads.
• Since kernel 5.9, modules must use sched_set_fifo(), sched_set_fifo_low(), or sched_set_normal() — the old priority-picking APIs are unreachable.
• The helpers encode intent, not numbers; user space (chrt) remains the retuning knob for integrators.
• Threaded IRQ handlers demonstrate the pattern: created, promoted to FIFO 50, and driven purely by events.
• Create-then-set-then-wake is the safe ordering when configuring a new kthread’s policy.

Conclusion

You can now read old driver code (direct sched_setscheduler_nocheck() calls with hand-rolled priorities), understand why it no longer builds, and write the modern replacement confidently. More importantly, you understand the philosophy behind the change: static real-time priorities do not compose across subsystems, so the kernel now centralizes the numbers and drivers merely declare intent. That mindset — express intent, let policy live in one place — will serve you across all of Linux device driver development.

The next lesson tackles the flip side of priority: not who runs first but how much CPU anyone is allowed to consume. That is CPU bandwidth control with cgroups v2, and it has changed even more dramatically since the 5.x-era books were written.

FAQ

Why can’t my kernel module call sched_setscheduler() anymore?

The symbol export was removed in kernel 5.9. Loadable modules must use the intent-based helpers sched_set_fifo(), sched_set_fifo_low(), or sched_set_normal() instead.

What priority does sched_set_fifo() actually assign?

The midpoint of the real-time range — priority 50. It is deliberately not a parameter; if a specific system needs different relative ordering, an administrator retunes the thread from user space with chrt.

Is a kernel thread the same as a process?

A kernel thread has a task_struct and is scheduled like any thread, but it has no user-space address space (its mm pointer is NULL) and it executes only kernel code.

How do threaded interrupt handlers relate to all this?

When a driver uses request_threaded_irq(), the kernel creates an irq/<nr>-<name> kthread and calls sched_set_fifo() on it, which is why they all appear at FIFO priority 50.

Can user space change a kernel thread’s priority?

Yes. With root privileges, chrt -f -p <prio> <pid> works on kernel threads just as it does on regular processes. This is the sanctioned mechanism for per-system tuning.

Should my driver’s worker thread be real-time?

Only if you have a measured latency requirement that normal scheduling misses. Most driver work belongs in workqueues or normal-priority kthreads; RT is for genuinely time-critical paths.

What happens if my FIFO kthread never sleeps?

It monopolizes its CPU, starving fair-class tasks and eventually triggering soft-lockup or RCU stall warnings. RT kthreads must block between units of work.

Does the EEVDF scheduler change anything in this lesson?

No. EEVDF replaced the algorithm inside the fair class only. Real-time classes, the 5.9 API changes, and the sched_set_fifo() family are unaffected.