Prerequisites

This lecture assumes you can already build and load a basic out-of-tree kernel module (covered in our device driver chapters) and are comfortable reading a Makefile that invokes the kernel build system. No prior debugging experience is assumed — that is exactly what this lecture builds.

Oops vs. Panic: What Actually Happens

A linux kernel oops is the kernel’s way of saying “something in kernel context just did something illegal, but I can probably keep the rest of the system running.” Typical triggers are a NULL or wild pointer dereference, an unaligned access on architectures that forbid it, or an explicit BUG(). The kernel prints diagnostic state, marks itself tainted, and — if the fault happened in process context and not inside an interrupt handler or while holding a critical lock — kills the current task and carries on.

A panic is different: it is unrecoverable. It happens when the fault occurs somewhere the kernel cannot safely continue from — inside interrupt context, inside the scheduler itself, or when panic_on_oops is set (the default on most production and embedded boards, because a half-alive kernel is often more dangerous than a clean reboot). Everything you learn about reading an oops applies directly to reading the oops that precedes a panic too — the panic dump is the same structure, just followed by a hang or reboot instead of a live shell.

Anatomy of a Kernel Oops

Every oops carries the same core fields regardless of architecture, though the register list changes. The fields that matter most when you are hunting a bug are:

Building an Original Fault Demo Module

To practice safely, we will build a small original module, ep_oops_demo, that only triggers its fault when you explicitly ask it to via a debugfs file — never on load. This keeps the demo predictable and lets you reload it between attempts.

// ep_oops_demo.c — deliberately triggers a kernel oops on demand
#include <linux/module.h>
#include <linux/debugfs.h>
#include <linux/uaccess.h>

struct ep_fault_data {
    int value;
};

/* Intentionally never allocated -- this is the bug we want to see */
static struct ep_fault_data *ep_bad_ptr;

static ssize_t ep_trigger_write(struct file *f, const char __user *buf,
                                 size_t len, loff_t *off)
{
    /* Writing anything to this debugfs file dereferences a NULL
     * pointer on purpose, producing a controlled oops for study. */
    ep_bad_ptr->value = 42;
    return len;
}

static const struct file_operations ep_trigger_fops = {
    .owner = THIS_MODULE,
    .write = ep_trigger_write,
};

static struct dentry *ep_debug_dir;

static int __init ep_oops_demo_init(void)
{
    ep_debug_dir = debugfs_create_dir("ep_oops_demo", NULL);
    debugfs_create_file("trigger", 0200, ep_debug_dir, NULL,
                         &ep_trigger_fops);
    pr_info("ep_oops_demo: loaded, write to "
            "/sys/kernel/debug/ep_oops_demo/trigger to fault\n");
    return 0;
}

static void __exit ep_oops_demo_exit(void)
{
    debugfs_remove_recursive(ep_debug_dir);
}

module_init(ep_oops_demo_init);
module_exit(ep_oops_demo_exit);
MODULE_LICENSE("GPL");
MODULE_DESCRIPTION("EmbeddedPathashala oops-decoding demo");

Build it against your running kernel headers and load it:

$ make -C /lib/modules/$(uname -r)/build M=$PWD modules
$ sudo insmod ep_oops_demo.ko
$ echo 1 | sudo tee /sys/kernel/debug/ep_oops_demo/trigger

Immediately check dmesg — you will see an oops similar to this (addresses will differ on every run because of KASLR, which we cover below):

[  102.884112] BUG: kernel NULL pointer dereference, address: 0000000000000000
[  102.884560] #PF: supervisor write access in kernel mode
[  102.884561] #PF: error_code(0x0002) - not-present page
[  102.884890] Oops: 0002 [#1] PREEMPT SMP NOPTI
[  102.885012] CPU: 2 PID: 1842 Comm: tee Tainted: G           OE      6.9.7 #1
[  102.885340] RIP: 0010:ep_trigger_write+0x14/0x40 [ep_oops_demo]
[  102.885690] Call Trace:
[  102.885691]  <TASK>
[  102.885692]  vfs_write+0xcc/0x3b0
[  102.885820]  ksys_write+0x67/0xe0
[  102.885890]  do_syscall_64+0x5c/0x90
[  102.885960]  entry_SYSCALL_64_after_hwframe+0x76/0x7e
[  102.886030]  </TASK>

Pinning the Fault to a Source Line

The line RIP: 0010:ep_trigger_write+0x14/0x40 [ep_oops_demo] tells you the fault happened 0x14 bytes into a 0x40-byte function. On current kernels the fastest way to resolve this to a source line is the in-tree scripts/faddr2line helper, which needs a kernel/module build with CONFIG_DEBUG_INFO=y:

$ ./scripts/faddr2line /path/to/ep_oops_demo.ko ep_trigger_write+0x14
ep_trigger_write+0x14/0x40:
ep_trigger_write at /home/ravi/ep_oops_demo/ep_oops_demo.c:20

If you only have objdump available, the manual version of the same lookup still works exactly as it did on older kernels — disassemble with source interleaved and look at the target offset:

$ objdump -dS ep_oops_demo.ko | grep -A5 "<ep_trigger_write>:"
0000000000000000 <ep_trigger_write>:
   ep_bad_ptr->value = 42;
   0:   48 8b 05 00 00 00 00    mov    0x0(%rip),%rax
   14: c7 00 2a 00 00 00       movl   $0x2a,(%rax)

Offset 0x14 lands exactly on the movl $0x2a,(%rax) instruction — the write through the uninitialized ep_bad_ptr, confirming the bug is precisely where we expected: line 20 of ep_oops_demo.c.

Preserving an Oops When There Is No Console

The hardest oops to debug is the one you never saw — a crash during early boot, right after a suspend/resume, or on a headless board with no attached serial cable. The modern, standard answer to this on embedded Linux is pstore backed by ramoops, which persists kernel log data across a reset by writing it to a reserved region of RAM that survives a warm reboot.

To enable ramoops, reserve a memory region in the device tree and enable the driver in your kernel config:

// devicetree fragment
reserved-memory {
    #address-cells = <1>;
    #size-cells = <1>;
    ramoops@8f000000 {
        compatible = "ramoops";
        reg = <0x8f000000 0x100000>;  /* 1 MiB reserved */
        record-size   = <0x8000>;
        console-size  = <0x8000>;
    };
};

$ zcat /proc/config.gz | grep PSTORE
CONFIG_PSTORE=y
CONFIG_PSTORE_RAM=y

After a crash and reboot, mount pstore (often auto-mounted at boot) and read back the preserved console log:

$ sudo mount -t pstore pstore /sys/fs/pstore
$ ls /sys/fs/pstore
console-ramoops-0  dmesg-ramoops-0
$ cat /sys/fs/pstore/console-ramoops-0

The legacy technique of reading the raw kernel log ring buffer address (__log_buf from System.map or /proc/kallsyms) out of physical memory from the bootloader still exists as a fallback when nothing else was configured ahead of time, but on any board you control, setting up ramoops in advance is far more reliable — you get a clean, parsed log instead of a raw hex dump you must reconstruct by hand.

Common Mistakes

• Chasing a raw hex address across reboots instead of the symbol+offset pair — KASLR makes raw addresses non-reproducible.
• Ignoring the “Tainted” flags and spending hours in mainline source when the fault is in your own out-of-tree module.
• Building without CONFIG_DEBUG_INFO, then being unable to resolve offsets to source lines with faddr2line.
• Assuming every oops is fatal — many are fully recoverable and only kill the offending process.

Best Practices

• Keep a debug build of your kernel/module with matching vmlinux/.ko symbols archived per release, so you can resolve any field oops later.
• Enable ramoops (or netconsole on boards with early network) on every board that ships without a permanently attached console.
• Read the call trace bottom-to-top to reconstruct the real call order, then confirm the top frame with faddr2line before touching any code.
• Treat a tainted kernel report as a strong prior, not proof — verify with the offset lookup before assuming which module is at fault.

Security consideration: oops and panic output can leak kernel addresses and internal state; avoid exposing raw dmesg or pstore output on production devices with untrusted physical or console access, and consider CONFIG_RANDOMIZE_BASE plus restricted dmesg (kernel.dmesg_restrict) on any board handled outside a trusted environment.

Summary

Reading a linux kernel oops comes down to four repeatable steps: identify the fault type and tainted state, resolve the faulting instruction to symbol+offset, map that offset to an exact source line with faddr2line or objdump, and — if the crash happens where you cannot see it live — make sure pstore/ramoops was already configured to catch it. Every advanced kernel debugging tool you will meet later, from kgdb to ftrace, builds on this same habit of reading register and stack state carefully before touching code.

Conclusion

This lecture is part of EmbeddedPathashala’s continuing free linux kernel development course and sits alongside our device driver and embedded Linux material as one more free resource for engineers learning production-grade debugging. Load the ep_oops_demo module on a spare board or VM, trigger it deliberately, and practice the faddr2line workflow until it is automatic — that repetition is what turns an intimidating wall of hex into a five-minute diagnosis.

FAQ