PREV_LEC NEXT_LEC

What You Will Learn

• What an Oops actually is, and how it differs from a full kernel panic
• How to write and load a small module that deliberately triggers a NULL pointer dereference
• How to read every field of a modern Oops: the fault description, the register dump, RIP/PC, and the Call Trace
• How to turn raw addresses into source file and line numbers using current tooling
• How to capture a backtrace on demand using the sysrq interface
• Common mistakes engineers make when reading Oops output, and how to avoid them

Prerequisites

This lecture assumes you are comfortable with the basics already covered earlier in this free Linux kernel development course:

• Writing and building a basic loadable kernel module (LKM)
• Using insmod, rmmod, and reading dmesg
• A working kernel build environment with headers matching your running kernel
• Basic familiarity with the C language and pointers

What Exactly Is a Kernel Oops

An Oops is the kernel’s exception handler firing because something the kernel was executing hit an illegal condition — most commonly a NULL or otherwise invalid pointer dereference, a bad memory access, or a failed assertion. Unlike a userspace segfault, the kernel does not have a safety net underneath it. When something goes wrong in kernel context, the kernel prints as much diagnostic information as it can — the faulting instruction, the full register state, and the call stack that led there — and then makes a judgment call: kill the offending task and limp on, or decide the damage is unrecoverable and panic.

An Oops is therefore a report, not necessarily a full system failure. A panic is what happens when the kernel decides it cannot safely continue at all, for example when the fault occurred inside interrupt context, inside the idle task, or when panic_on_oops is set. Every panic starts life as an Oops-style report, but not every Oops turns into a panic.

Building an Original Crash Module

To see a real Oops, we need a controlled way to trigger one. The following module is written from scratch for this free Linux kernel development course and deliberately dereferences a NULL pointer inside a dedicated function, so the fault shows up cleanly in the call trace instead of being buried inside an inlined helper.

// ep_oops_demo.c
#include <linux/kernel.h>
#include <linux/module.h>
#include <linux/init.h>

static noinline void ep_trigger_fault(void)
{
	int *bad_ptr = NULL;

	pr_info("ep_oops_demo: about to dereference a NULL pointer\n");
	*bad_ptr = 0xdead;
}

static int __init ep_demo_init(void)
{
	pr_info("ep_oops_demo: module loaded, triggering fault now\n");
	ep_trigger_fault();
	return 0;
}

static void __exit ep_demo_exit(void)
{
	pr_info("ep_oops_demo: module unloaded\n");
}

module_init(ep_demo_init);
module_exit(ep_demo_exit);
MODULE_LICENSE("GPL");
MODULE_DESCRIPTION("EmbeddedPathashala demo module for Oops decoding");

A quick note on style: we mark ep_trigger_fault() as noinline so the compiler keeps it as a distinct symbol. On modern GCC/Clang with kernel build flags, ordinary functions are frequently inlined into their caller, and an inlined fault would make the call trace harder to read for a first example.

Build and Load Steps

A minimal Makefile for an out-of-tree module looks like this:

obj-m += ep_oops_demo.o

all:
	make -C /lib/modules/$(shell uname -r)/build M=$(PWD) modules

clean:
	make -C /lib/modules/$(shell uname -r)/build M=$(PWD) clean

$ make
$ sudo insmod ep_oops_demo.ko
$ dmesg | tail -n 40

Reading a Modern Oops Line by Line

On a current 6.x kernel with CONFIG_KALLSYMS enabled (the default on virtually every distribution kernel), you will see output shaped like this. The exact addresses and offsets will differ on your machine — that is expected and normal.

BUG: kernel NULL pointer dereference, address: 0000000000000000
#PF: supervisor write access in kernel mode
#PF: error_code(0x0002) - not-present page
PGD 0 P4D 0
Oops: 0002 [#1] PREEMPT SMP NOPTI
CPU: 2 PID: 4417 Comm: insmod Tainted: G           OE      6.8.0-generic #1
RIP: 0010:ep_trigger_fault+0x19/0x30 [ep_oops_demo]
Code: 55 48 89 e5 bf 61 00 00 00 e8 4a 3f 8a e0 c7 04 25 00 00 00 00 ad de
RSP: 0018:ffffb2c1c0abbdd8 EFLAGS: 00010246
RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffffffffc0a02008 RDI: ffffffffc0a02040
RBP: ffffb2c1c0abbdd8 R08: 0000000000000000 R09: 0000000000000000
Call Trace:
 <TASK>
 ep_demo_init+0x17/0x1000 [ep_oops_demo]
 do_one_initcall+0x5f/0x220
 do_init_module+0x60/0x230
 __se_sys_init_module+0x180/0x1e0
 do_syscall_64+0x5b/0x90
 entry_SYSCALL_64_after_hwframe+0x6e/0x76
 </TASK>
---[ end trace 0000000000000000 ]---

Working through this field by field:

• BUG line — a one-line human-readable summary of the fault class. Here it is a NULL pointer write, which matches our module.
• #PF lines — page-fault detail: whether the access was a read or write, and whether it happened in supervisor (kernel) or user mode.
• RIP — the instruction pointer at the moment of the fault, already resolved to a symbol name and offset thanks to CONFIG_KALLSYMS. This is the modern equivalent of manually looking up a raw PC value, and it is the single most useful line in the whole dump.
• Code — a hex dump of the machine instructions around the faulting one, primarily useful when you need to disassemble by hand.
• Register block (RAX–R15, RSP, EFLAGS) — the full CPU state at fault time, useful for reconstructing argument values or verifying your theory about what went wrong.
• Call Trace — the chain of function calls leading up to the fault, from most recent to oldest. Because ep_trigger_fault was inlined-adjacent but kept noinline, it appears explicitly at RIP, with ep_demo_init directly below it as the caller.
• Tainted flags — the letters after Tainted: tell you whether out-of-tree modules, proprietary code, or prior Oopses have already put the kernel in a degraded diagnostic state.

Turning Addresses Into Source Lines

With CONFIG_KALLSYMS, function names and offsets are already resolved for you, which removes most of the manual address arithmetic that older debugging workflows relied on. When you need to go further and map an offset to an exact source line, the modern approach is the kernel’s own scripts/decode_stacktrace.sh script, run against your build tree and the saved dmesg output:

$ dmesg > oops.log
$ ./scripts/decode_stacktrace.sh vmlinux . modules-dir < oops.log

For an individual symbol, addr2line remains useful when you only have one offset to resolve:

$ addr2line -e ep_oops_demo.ko -i 0x19

Capturing a Backtrace on Demand

You do not always need to wait for an accidental fault. The kernel’s sysrq interface lets you request a controlled crash to practice reading a trace, which is a safe exercise to run in a virtual machine as part of this free Linux kernel development course:

$ echo 1 | sudo tee /proc/sys/kernel/sysrq
$ echo c | sudo tee /proc/sysrq-trigger
$ dmesg | tail -n 30

This produces a full Call Trace without you having to write any code at all, and it is a quick way to sanity-check that your symbol resolution and decode_stacktrace.sh setup are working correctly before you rely on them for a real bug.

Real-World Use Cases

• Diagnosing a driver crash reported from a customer device where you only have a saved dmesg log, not a live system
• Verifying a fix by confirming the exact faulting line before and after a patch
• Triaging a fleet of embedded boards by grouping crash reports by RIP symbol and offset
• Bisecting a regression by comparing Call Trace shapes across kernel versions

Common Mistakes and Troubleshooting

• Ignoring the Tainted flags. If a prior Oops already tainted the kernel, later dumps can be misleading because the system may already be in a partially corrupted state.
• Reading Call Trace strictly top to bottom without checking for inlining. An aggressively inlined function can hide the true faulting line; rebuilding with -fno-inline-functions-called-once or checking the disassembly clears this up.
• Assuming every Oops means the module is broken. Sometimes the fault originates from bad data passed in from elsewhere in the kernel or from user space; the Call Trace’s deeper frames often reveal the real origin.
• Losing the log. On embedded targets without persistent storage for dmesg, always configure pstore or a serial console capture before you need it, not after.
• Mismatched symbols. Running decode_stacktrace.sh or addr2line against a vmlinux or module build that does not exactly match the running kernel produces confidently wrong line numbers.

Best Practices in a Free Linux Kernel Development Course Workflow

• Always build modules with debug info (CONFIG_DEBUG_INFO) during development so addresses resolve cleanly.
• Keep a copy of the exact vmlinux and module .ko files matched to every kernel build you ship, so old Oops logs stay decodable later.
• Enable panic_on_oops only where a controlled reboot is safer than a degraded but still-running kernel — know which behavior your product needs.
• Rate-limit or capture kernel logs to persistent storage on embedded targets, since the ring buffer is volatile and will overwrite itself.

Performance and Security Considerations

Debug aids like CONFIG_KASAN or full debug symbols make Oops output far richer but add measurable runtime and image-size overhead, so production builds typically trade some of that diagnostic detail for performance. On the security side, an Oops dump can leak kernel addresses, register contents, and internal structure layout; treat saved crash logs as sensitive, and be deliberate about whether dmesg access should be restricted on production devices via kernel.dmesg_restrict.

Summary and Key Takeaways

• An Oops is a structured diagnostic report the kernel prints when it hits a fault it may or may not recover from.
• Modern kernels resolve symbols automatically in the Call Trace, which removes most manual address lookup.
• scripts/decode_stacktrace.sh and addr2line remain the tools of choice when you need exact source lines from a saved log.
• The sysrq interface is a safe, repeatable way to practice reading crash output.
• Reading an Oops carefully, field by field, turns what looks like a wall of hex into a precise map back to the offending line of code.

Conclusion

Every kernel or driver engineer eventually meets their first Oops, usually at the worst possible time. The good news is that the dump is not random noise — it is a structured, decodable record of exactly what the CPU was doing and why it could not continue. By practicing on a controlled module like ep_oops_demo and getting comfortable with CONFIG_KALLSYMS, decode_stacktrace.sh, and the sysrq trigger, you turn a stressful production incident into a routine diagnostic task. That confidence is exactly what this free Linux kernel development course is aiming to build, one lecture at a time, and it carries directly into any free embedded Linux course work you do afterward on real hardware.

Frequently Asked Questions

PREV_LEC NEXT_LEC