« Previous Lecture | Next Lecture »

What You Will Learn

• The full Linux scheduling priority order, from hardirqs down to normal threads
• Why a hardware interrupt always preempts even your highest-priority real-time thread
• How to read and change an IRQ thread’s real-time priority
• How PREEMPT_RT on kernel 6.x changes this picture
• Practical guidance for setting deadlines on real-time embedded applications

Prerequisites

This lecture assumes you have completed the previous lecture on threaded interrupt handlers, and that you understand the basic difference between the SCHED_FIFO/SCHED_RR real-time scheduling policies and the default SCHED_OTHER policy used by regular threads.

The Linux Scheduling Priority Order

On a standard (non-RT) Linux kernel, execution priority — from highest to lowest — looks like this:

Two points beginners frequently miss:

• Everything inside “real-time threads” — your own SCHED_FIFO application threads and kernel IRQ threads — competes on the single 1-99 priority scale. A kernel thread at a given priority gets a small implicit edge over a userspace thread at the exact same numeric priority.
• A hardware interrupt always wins, no matter how high your real-time thread’s priority is set, because it runs in a completely different context that isn’t scheduled at all in the normal sense — it simply preempts the CPU.

Why This Matters for Deadlines

Imagine a real-time application with three SCHED_FIFO threads — A, B, and C — running at real-time priorities 30, 45, and 60. Suppose thread B has a worst-case deadline of 12 milliseconds to finish its work once its event occurs. If a lower-priority SCHED_OTHER thread X is running when B’s event fires, B correctly preempts X immediately, because the fundamental real-time scheduling rule is that the highest-priority runnable thread must be the one running.

But if a hardware interrupt fires during that same window — even one completely unrelated to thread B — it preempts B too, because hardirqs sit above every SCHED_FIFO priority. This is precisely why keeping hardirq handlers extremely short matters: a slow hardirq handler can silently blow through B’s 12 ms deadline even though B’s own priority looks safely high.

Inspecting and Tuning IRQ Thread Priority

Every threaded IRQ shows up as a normal kernel thread you can inspect with standard tools:

# List all IRQ threads and their scheduling policy/priority
ps -eLo pid,comm,policy,rtprio | grep irq/

# Example output
#   431 irq/47-ep-button  FF   50

You can change an individual IRQ thread’s real-time priority at runtime without touching driver code, using chrt:

# Raise the priority of IRQ thread with PID 431 to 80
chrt -f -p 80 431

# Verify the change
chrt -p 431

Tip: Only raise an IRQ thread’s priority above your application’s real-time threads if that interrupt genuinely needs to win the race — for example, a safety-critical sensor read. Raising priorities carelessly just moves the starvation problem elsewhere.

PREEMPT_RT on Kernel 6.x: A Bigger Picture Change

Since the PREEMPT_RT patch set was merged into mainline Linux starting with the 6.12 series, the priority picture above gets an important addition. On a kernel configured with CONFIG_PREEMPT_RT:

• Most hardirq handlers are force-threaded by default, so far fewer code paths actually run in true interrupt context — most “interrupt handling” you observe is really SCHED_FIFO thread execution.
• Spinlocks are converted into sleepable “rt-mutexes” internally, which changes some assumptions long-time kernel developers had about spinlock behavior — this is a deliberate design trade-off to bound worst-case latency.
• System administrators get much finer control: because more work now happens in schedulable threads, more of the system’s timing behavior can be tuned with standard scheduler tools like chrt and cgroup CPU controllers, instead of requiring kernel recompilation.

Common Mistakes and Troubleshooting

Best Practices

• Set IRQ thread priorities deliberately and document why, rather than defaulting to the highest possible value.
• Use ps -eLo pid,comm,policy,rtprio as a quick health check when debugging latency issues.
• On PREEMPT_RT systems, budget for the fact that spinlock-protected critical sections may now be preempted.

Performance and Security Considerations

Performance: Correct priority assignment across IRQ threads and application threads is often more impactful for real-time performance than any single code optimization.

Security: A userspace process with sufficient privilege can call chrt against unrelated real-time threads, including IRQ threads, and starve the system. Restrict CAP_SYS_NICE appropriately in production.

Real-World Use Cases

• Industrial control loops that must service a sensor interrupt within a strict microsecond window
• Audio subsystems relying on PREEMPT_RT to avoid buffer underruns
• Robotics and motor control firmware built on embedded Linux with RT patches

Key Takeaways

• Hardware interrupts always preempt every SCHED_FIFO thread, no matter its priority.
• Threaded IRQ handlers default to SCHED_FIFO priority 50, and can be tuned at runtime with chrt.
• PREEMPT_RT, mainline since the 6.12 series, force-threads most hardirqs by default for better worst-case latency.
• Deliberate, documented priority assignment beats blanket “set everything to maximum priority” tuning.

Frequently Asked Questions

1. Does a real-time thread ever preempt a hardware interrupt?

No. Hardware interrupts always have the highest priority on Linux and preempt any real-time or normal thread.

2. What is the default real-time priority of a threaded IRQ handler?

50, out of the usable SCHED_FIFO/SCHED_RR range of 1 to 99.

3. How do I change an IRQ thread’s priority without recompiling the driver?

Find its PID with ps -eLo pid,comm,policy,rtprio | grep irq/, then use chrt -f -p <priority> <pid>.

4. What is PREEMPT_RT and is it in mainline Linux?

PREEMPT_RT is a set of real-time preemption changes to the kernel scheduler and locking primitives. It was merged into mainline Linux starting with the 6.12 kernel series.

5. Does PREEMPT_RT thread every interrupt automatically?

It force-threads most hardirq handlers by default, though some latency-critical or non-threadable interrupts remain exceptions.

6. Should I always raise my IRQ thread’s priority to the maximum?

No. Raise priority only for interrupts with a genuine hard real-time deadline, since over-prioritizing one thread can starve others.

7. How does spinlock behavior change under PREEMPT_RT?

Spinlocks are converted internally into sleepable rt-mutexes, so code that assumed a spinlock-protected section could never be preempted needs to be reviewed.

Conclusion

Understanding where threaded IRQ handlers sit in the Linux scheduling priority order — and how PREEMPT_RT changes that picture on kernel 6.x — is essential for anyone building real-time or latency-sensitive embedded Linux systems. Combine short hardirq handlers, deliberately tuned SCHED_FIFO priorities, and, where needed, a PREEMPT_RT kernel to build systems with predictable worst-case behavior. This wraps up our two-part series on threaded interrupt handling in this free Linux kernel programming course.

« Previous Lecture | Next Lecture »