PREV_LEC | NEXT_LEC

What You Will Learn

• Why post-mortem tools like kdump only show you a snapshot, not a timeline
• How Ftrace’s automatic oops-dump feature captures the events leading up to a crash
• How to enable it via tracefs and via a kernel boot parameter
• How to size the per-CPU trace ring buffer sensibly
• A complete, original demo module that triggers a fault and shows the resulting trace dump
• Common mistakes, performance trade-offs, and security considerations

Prerequisites

• Comfortable building and loading out-of-tree kernel modules
• Basic familiarity with Ftrace (function tracer, tracers, tracefs mount point)
• A test VM or board where you’re allowed to crash the kernel on purpose — never do this on
  production hardware

Why Post-Mortem Snapshots Aren’t Enough

Tools such as kdump/kexec combined with the crash utility are
excellent for inspecting the state of memory at the instant of a crash — register values, the
stack, active processes. What they cannot show you is the path the kernel took to get there.
If a driver corrupts a data structure ten function calls before the actual fault, a memory snapshot
alone won’t reveal that chain of events. You need a timeline, and that’s exactly what Ftrace’s
ring buffer already is: a rolling, timestamped log of function entries/exits, scheduling events, and
any custom trace points you’ve enabled.

How ftrace_dump_on_oops Works

Ftrace maintains a per-CPU ring buffer of trace events. Normally you’d read that buffer manually
through trace or trace_pipe. The ftrace_dump_on_oops option changes
that: the moment the kernel detects an oops or panic, it automatically dumps the entire buffer to the
console in plain ASCII, before the rest of the panic handling proceeds. If your console is wired to a
serial line (which is standard practice on embedded boards), that dump survives even if the system
never comes back up — you simply capture the serial log.

There are two independent ways to turn this on:

1. At runtime via sysctl / tracefs

# echo 1 > /proc/sys/kernel/ftrace_dump_on_oops

2. At boot, via the kernel command line

ftrace_dump_on_oops

Adding this to your bootloader’s kernel command line (for example in the U-Boot bootargs or
GRUB config) guarantees the feature is active from the very first boot, which matters if the crash you’re
chasing happens early — before you’d have a chance to enable it manually.

Sizing the Trace Ring Buffer

By default the Ftrace ring buffer is comfortably larger than 1 MB per CPU, and dumping
that much text to a slow serial console can take a very long time — sometimes long enough that a
watchdog resets the board before the dump finishes. In practice you almost always want to shrink the
buffer before you arm dump-on-oops, trading history depth for a dump that actually completes.

# mount -t tracefs nodev /sys/kernel/tracing   # if not already mounted
# echo 4 > /sys/kernel/tracing/buffer_size_kb

Note two things modern kernels handle slightly differently from older references you may have seen:
tracefs is its own filesystem type nowadays and is commonly mounted directly at
/sys/kernel/tracing, independent of debugfs. If your system still mounts debugfs at
/sys/kernel/debug, the same file is also reachable at
/sys/kernel/debug/tracing/buffer_size_kb — both paths point at the same tracefs instance.
The value you write is always per CPU, not a total, so a 4 CPU board with
buffer_size_kb set to 4 is actually holding roughly 16 KB of trace history in total.

Hands-On: Triggering and Capturing a Dump

Let’s build a small, original demo module — ep_oops_demo — that deliberately
dereferences a NULL pointer inside a function a few calls deep, so you can see the trace dump actually
capture that call chain.

// ep_oops_demo.c
#include <linux/init.h>
#include <linux/module.h>
#include <linux/kernel.h>

static noinline void ep_level_three(void)
{
    int *bad = NULL;

    pr_info("ep_oops_demo: about to fault in %s\n", __func__);
    *bad = 42;   /* deliberate NULL pointer write */
}

static noinline void ep_level_two(void)
{
    pr_info("ep_oops_demo: entering %s\n", __func__);
    ep_level_three();
}

static noinline void ep_level_one(void)
{
    pr_info("ep_oops_demo: entering %s\n", __func__);
    ep_level_two();
}

static int __init ep_oops_demo_init(void)
{
    pr_info("ep_oops_demo: loaded, triggering fault chain\n");
    ep_level_one();
    return 0; /* unreachable in practice */
}

static void __exit ep_oops_demo_exit(void)
{
    pr_info("ep_oops_demo: unloaded\n");
}

module_init(ep_oops_demo_init);
module_exit(ep_oops_demo_exit);
MODULE_LICENSE("GPL");
MODULE_DESCRIPTION("EmbeddedPathashala Ftrace dump-on-oops demo");

# Makefile
obj-m += ep_oops_demo.o

all:
	make -C /lib/modules/$(shell uname -r)/build M=$(PWD) modules

clean:
	make -C /lib/modules/$(shell uname -r)/build M=$(PWD) clean

Build it, then arm Ftrace before loading:

$ make
# mount -t tracefs nodev /sys/kernel/tracing
# echo function > /sys/kernel/tracing/current_tracer
# echo 4 > /sys/kernel/tracing/buffer_size_kb
# echo 1 > /sys/kernel/tracing/tracing_on
# echo 1 > /proc/sys/kernel/ftrace_dump_on_oops

# insmod ep_oops_demo.ko

The expected dmesg output shows the module’s own log lines, followed by the oops itself, followed
by the automatically flushed trace buffer showing the exact call sequence
ep_oops_demo_init → ep_level_one → ep_level_two → ep_level_three right before the fault:

$ dmesg | tail -n 30
ep_oops_demo: loaded, triggering fault chain
ep_oops_demo: entering ep_level_one
ep_oops_demo: entering ep_level_two
ep_oops_demo: about to fault in ep_level_three
BUG: kernel NULL pointer dereference, address: 0000000000000000
...
RIP: 0010:ep_level_three+0x1c/0x30 [ep_oops_demo]
...
---[ end trace ]---
Dumping ftrace buffer:
---------------------------------
 -0     [000] ....   123.456: ep_oops_demo_init <-do_one_initcall
 -0     [000] ....   123.457: ep_level_one <-ep_oops_demo_init
 -0     [000] ....   123.457: ep_level_two <-ep_level_one
 -0     [000] ....   123.458: ep_level_three <-ep_level_two
---------------------------------

That flushed buffer is the whole point: even though the crash happened inside
ep_level_three, you have the complete call chain that led there, with timestamps, without
needing a debugger attached at the moment of the crash.

kdump vs Ftrace Dump-on-Oops

In real projects these two techniques complement each other rather than compete: dump-on-oops tells
you the story, kdump lets you examine the final scene in detail.

Common Mistakes and Troubleshooting

• Buffer too small — if you shrink buffer_size_kb too aggressively,
  the relevant history may have already scrolled out of the ring buffer before the crash happens.
  Start conservative, then shrink once you know how far back you need to see.
• No tracer selected — dump-on-oops only dumps whatever the currently active
  tracer has been recording. If current_tracer is still nop, the buffer will
  be essentially empty. Enable function or a relevant event tracer first.
• tracing_on left at 0 — a common oversight; the master switch has to be on for
  anything to accumulate in the buffer at all.
• Console too slow — on a heavily loaded serial console, the dump itself can take
  long enough to trip a hardware watchdog. Either shrink the buffer further or raise the watchdog
  timeout during debugging sessions.

Best Practices

• Route your console to a serial line (or a persistent log target) whenever you’re chasing an
  intermittent crash — a dump that scrolls off an unlogged terminal is wasted effort.
• Pair ftrace_dump_on_oops with a narrow, purpose-built tracer (function filtering,
  specific event tracers) rather than tracing the entire kernel — it keeps the dump readable and the
  overhead low.
• Set the boot-time parameter on boards where crashes happen during early init, since you won’t get
  a chance to enable it manually in time.

Performance Considerations

Ftrace tracing is not free — the function tracer adds measurable overhead per traced call. For
long debugging sessions on production-adjacent hardware, prefer targeted tracers (specific functions
or tracepoints) over the blanket function tracer, and keep the ring buffer only as large as you need.

Security Considerations

Trace dumps can include function names, addresses, and sometimes argument values that reveal kernel
layout — useful information for an attacker trying to defeat KASLR. Treat trace dumps the same way you
treat other kernel debug output: restrict console/log access on production or externally reachable
systems, and disable dump-on-oops outside of controlled debugging environments.

Summary and Key Takeaways

• kdump/crash gives you a snapshot; Ftrace’s dump-on-oops gives you the timeline leading to it
• ftrace_dump_on_oops can be toggled at runtime or set as a boot parameter
• The ring buffer size is per CPU — size it deliberately before you need it
• Combine a targeted tracer with dump-on-oops for a readable, low-overhead crash trail

Conclusion

Automatic ftrace dumps turn an oops from a dead end into a lead. Instead of staring at a single
faulting instruction, you get the sequence of calls that walked the kernel into that state — often
enough, on its own, to spot the real bug. It’s a small amount of setup for a disproportionately useful
payoff, which is exactly why it belongs in every kernel developer’s debugging toolkit, and in this
free linux kernel development course.

FAQ

PREV_LEC | NEXT_LEC