What is NMI Interrupt and Magic SysRq: Live Debugging in Linux Kernel 6.x-Linux Device Driver Training Online

NMI Interrupt and Magic SysRq: Live Debugging in Linux Kernel 6.x
Free Linux Kernel Programming Course — EmbeddedPathashala
NMI interrupt linux kernel magic sysrq sysrq-trigger kernel debugging Free Embedded Systems Course

The NMI interrupt linux kernel mechanism is one of the most powerful, and least understood, debugging tools available to embedded and kernel developers. Unlike ordinary interrupts, the non-maskable interrupt cannot be turned off by any of the enable/disable APIs we studied earlier, which makes it perfect for emergency debugging when a system appears completely frozen. In this lecture of our free Linux kernel programming course, we explore the NMI and the magic SysRq facility that builds on top of it, updated for kernel 6.x systems.

What You Will Learn
What makes NMI different Magic SysRq basics /proc/sysrq-trigger Backtrace-all-CPUs debugging Securing SysRq in production

Prerequisites

  • Completion of the previous two lectures on interrupt handling and enable/disable IRQ APIs
  • Root access to a Linux system (physical board, VM, or single-board computer)
  • Basic familiarity with /proc pseudo-filesystem

What Makes the NMI Special

Every API we’ve covered so far — local_irq_disable(), disable_irq(), and friends — has one thing in common: they all work by masking ordinary maskable interrupts. The non-maskable interrupt (NMI) is architecture-specific and deliberately designed so that none of those APIs can block it. It exists so the system always has one channel of communication left even when something has gone badly wrong, such as a hung CPU core or a hardware watchdog firing.

Maskable Interrupts vs the NMI
Maskable IRQs

Can be disabled with local_irq_disable(), disable_irq(), etc. Used for normal devices.

vs
NMI

Cannot be masked by software. Reserved for watchdogs, hardware faults, and emergency debugging.

Two important properties worth remembering: NMI lines cannot be shared between devices, and they are handled through architecture-specific code rather than the generic IRQ framework used for regular device interrupts.

Magic SysRq: A Practical Use of NMI

The kernel’s magic SysRq facility is the most common real-world use of the NMI that most developers will actually touch. On a physical keyboard it is triggered with the Alt+SysRq+<letter> key combination, but a far more scriptable and remote-friendly method is to write the letter to a special file in /proc.

Security first: Magic SysRq is disabled or restricted by default on most modern distributions. You must be root, and the kernel parameter kernel.sysrq controls which SysRq functions are permitted.
# Check current sysrq setting
cat /proc/sys/kernel/sysrq

# Enable all sysrq functions temporarily (root required)
echo 1 > /proc/sys/kernel/sysrq

# List available sysrq commands in the kernel log
echo h > /proc/sysrq-trigger
dmesg | tail -20

Dumping a Backtrace on Every CPU

One of the most useful commands for a developer debugging a system that seems to be hanging is the “show backtrace on all active CPUs” function. It works by sending an NMI to every core in the system simultaneously, so it can capture exactly what each CPU is executing at that instant — even if some cores are stuck in a tight loop that would never respond to an ordinary signal.

# Trigger a kernel-mode stack backtrace on every active CPU core
echo l > /proc/sysrq-trigger

# View the captured backtraces
dmesg | tail -100

Other Frequently Used SysRq Commands

Letter Effect
lShow backtrace on all active CPUs (uses NMI)
tShow state of all tasks
wShow all blocked (uninterruptible sleep) tasks
mShow current memory usage
sSync all mounted filesystems
bImmediately reboot the system (no sync, use with caution)

A Simple Kernel Module That Reacts to NMI-Triggered Debugging

While you cannot register your own generic NMI handler through the normal request_irq()-style APIs, you can register a kernel notifier so your driver logs helpful diagnostic state whenever a kernel panic or oops occurs — often the same moment an engineer would reach for a SysRq backtrace.

#include <linux/module.h>
#include <linux/notifier.h>
#include <linux/kdebug.h>

static int ep_diag_die_notify(struct notifier_block *nb,
                               unsigned long event, void *data)
{
    if (event == DIE_OOPS)
        pr_emerg("ep_diag: kernel oops detected, dumping driver state\n");

    return NOTIFY_DONE;
}

static struct notifier_block ep_diag_nb = {
    .notifier_call = ep_diag_die_notify,
};

static int __init ep_diag_init(void)
{
    register_die_notifier(&ep_diag_nb);
    pr_info("ep_diag: diagnostic notifier registered\n");
    return 0;
}

static void __exit ep_diag_exit(void)
{
    unregister_die_notifier(&ep_diag_nb);
}

module_init(ep_diag_init);
module_exit(ep_diag_exit);
MODULE_LICENSE("GPL");

Real-World Use Cases

  • Diagnosing a soft-locked embedded board in the field by triggering an “l” SysRq command over a serial console.
  • Hardware watchdogs that raise an NMI to force a diagnostic dump before a forced reboot.
  • CI/lab test rigs that script SysRq commands via /proc/sysrq-trigger to capture state during automated stress tests.

Common Mistakes and Troubleshooting

  • Leaving kernel.sysrq permanently set to 1 in production. This exposes powerful commands (including immediate reboot) to anyone with local console or /proc access; restrict it with a bitmask instead.
  • Expecting SysRq output on the screen. On headless or remote systems, the output goes to the kernel log (dmesg), not necessarily the console you’re typing in.
  • Confusing NMI watchdog panics with normal kernel panics. An NMI watchdog timeout usually points to a CPU stuck with interrupts disabled for too long — check dmesg for “NMI watchdog: BUG: soft lockup” style messages.

Best Practices and Security Considerations

Practice Why
Use a restrictive kernel.sysrq bitmask in productionLimits which SysRq functions are exposed, reducing attack surface.
Keep SysRq fully enabled on development/lab boards onlySpeeds up debugging without risking production systems.
Log SysRq-triggered dumps to persistent storage or a remote syslogEnsures diagnostic data survives a reboot after a crash.

Summary / Key Takeaways

  • The NMI cannot be masked by any of the enable/disable IRQ APIs, making it ideal for emergency debugging.
  • Magic SysRq exposes NMI-based and other diagnostic commands through simple keypresses or /proc/sysrq-trigger.
  • The “l” command dumps a backtrace from every active CPU using the NMI, which is invaluable for diagnosing hangs.
  • Always restrict kernel.sysrq on production and internet-facing systems.

Conclusion

The NMI interrupt linux kernel facility, and the magic SysRq tools built on top of it, give embedded and kernel developers a last line of defense when a system seems completely unresponsive. Practicing with /proc/sysrq-trigger on a lab board today will make you far more effective the day a production system locks up. This wraps up our three-part series on interrupt handling in the free Linux kernel programming and device drivers course — check the course index for what comes next.

FAQ

Q1. What is the NMI used for in the Linux kernel?
It is used for hardware watchdogs, critical fault detection, and debugging features such as dumping a stack backtrace on every CPU, precisely because it cannot be masked by software.

Q2. Can I share an NMI line between two devices?
No, NMI interrupt lines cannot be shared, unlike regular maskable IRQ lines.

Q3. How do I trigger magic SysRq without a physical keyboard?
Echo the relevant letter to /proc/sysrq-trigger as root, for example: echo l > /proc/sysrq-trigger.

Q4. Is magic SysRq enabled by default?
It depends on the distribution; many enable a restricted subset by default. Check /proc/sys/kernel/sysrq to see the current setting.

Q5. What does the sysrq “l” command actually do?
It sends an NMI to every active CPU core and asks each one to print its current kernel-mode stack backtrace to the kernel log.

Q6. Is it safe to leave sysrq fully enabled on a production server?
Generally no — it’s safer to use a restrictive bitmask so only the diagnostic commands you actually need are available, reducing the risk if the system is compromised.

Q7. Can a kernel module register a custom handler for the NMI?
Not through the standard request_irq()-family APIs; NMI handling is architecture-specific. Drivers typically hook into related mechanisms like die notifiers instead, as shown in this lecture’s example module.

Continue Learning — Free Linux Kernel Programming Course

This lecture is part of EmbeddedPathashala’s free Linux kernel programming and device drivers course.

Visit EmbeddedPathashala Next Lecture →

2 Comments

Leave a Reply

Your email address will not be published. Required fields are marked *