• Comfortable building and loading kernel modules
• Root access to a Linux test machine or VM
• Basic understanding of what a system call and a call stack are
• Ideally, completed the previous lecture on spinlocks in this course

Ftrace is the kernel’s built-in function and event tracer, exposed through debugfs, normally mounted at /sys/kernel/tracing on modern distributions (the older /sys/kernel/debug/tracing path still works as a compatibility mount on most systems). Everything you need is a set of plain files you read and write as root; no separate driver has to be loaded.

sudo mount -t tracefs none /sys/kernel/tracing 2>/dev/null
cd /sys/kernel/tracing
ls

Before reaching for a tracer, always check dmesg first; many bugs, including atomic-context violations and NULL pointer dereferences, dump a full call stack automatically.

sudo dmesg -T | tail -n 60

Read a kernel call stack from the bottom upward: the bottom-most frames are the oldest calls (closest to the system call entry), and the top-most frames are the deepest, most recent calls, usually pointing at the exact line that misbehaved. Frames marked with a leading ? are unreliable leftovers from earlier stack usage and can generally be ignored.

When dmesg alone is not enough, the function_graph tracer shows you the full call tree, with entry and exit timestamps, for any function you choose. Here is a minimal, from-scratch example tracing a hypothetical driver write handler named my_write:

cd /sys/kernel/tracing
echo function_graph > current_tracer
echo my_write > set_graph_function
echo 1 > tracing_on

# now trigger the code path from user space, e.g. write to your device node

echo 0 > tracing_on
cat trace > /tmp/my_write_trace.txt

Reset the tracer when you are done so you do not leave tracing running on a production system:

echo nop > current_tracer

A function_graph trace shows nested entries and exits with a duration column. Below is the general shape you will see, expressed as a flow rather than as raw trace text:

The indentation depth tells you the call depth, and the duration column (when present) tells you how long each function took, which is invaluable for spotting an unexpectedly slow function hiding inside your driver.

On systems with eBPF support, bpftrace gives you the same kind of visibility with far less setup and much lower overhead than a full function_graph trace. A one-liner to trace entry into any kernel function matching a pattern looks like this:

sudo bpftrace -e 'kprobe:my_write { printf("%s called by PID %d\n", probe, pid); }'

For most day-to-day debugging on a modern kernel, reach for bpftrace first; fall back to raw ftrace when you need the full call graph with timing, or when eBPF is unavailable on the target.

• Confirming exactly which driver entry point a user-space call reaches, and in what order, before adding new logic.
• Measuring how long a specific kernel function takes under real workload, to catch latency regressions.
• Proving to a code reviewer that a fix actually changed the call path, with a captured trace as evidence.
• Root-causing an intermittent bug report by capturing a trace the moment it happens on a lab device.

• Always start with dmesg; it is free and often already has your answer.
• Filter traces to the smallest possible function set before capturing.
• Turn tracing off immediately after capturing what you need.
• Prefer bpftrace for quick, low-overhead checks; use function_graph when you need full nested call timing.
• Save every useful trace to a file with a clear name; you will want to compare it against future traces.

The function_graph tracer instruments every traced function call, which adds measurable overhead, especially if the filter is too broad. On a production or performance-sensitive system, always scope the trace to the narrowest function set possible, and prefer bpftrace or perf’s sampling mode when you only need a rough picture rather than a complete call graph.

Access to tracefs and eBPF tracing requires root or equivalent capabilities for good reason: a trace can expose kernel addresses, timing side-channels, and sensitive data flowing through the system. Never leave tracing enabled on a shared or production system, and be mindful of what a captured trace file reveals before sharing it outside your team.

• Always check dmesg first; many bugs self-report a full call stack.
• Ftrace lives at /sys/kernel/tracing and needs no extra kernel module.
• The function_graph tracer, filtered with set_graph_function, shows a precise nested call tree with timing.
• bpftrace is the lower-overhead, modern option for quick targeted checks.
• Always disable tracing when you are done capturing data.

Tracing turns a vague bug report into a precise, provable call sequence. Once you are comfortable moving between dmesg, ftrace’s function_graph tracer, and bpftrace, you can confidently answer “what exactly did the kernel do, and in what order” for almost any driver problem you encounter. Combine this with the spinlock discipline from the previous lecture and you have the two core debugging skills every kernel and driver developer relies on daily.

Q1. Where does ftrace live on a modern Linux system?
Under the tracefs filesystem, normally mounted at /sys/kernel/tracing, with the older debugfs path still available as a compatibility mount on most distributions.

Q2. Do I need to load a kernel module to use ftrace?
No, ftrace is built into the kernel and controlled entirely through files under tracefs.

Q3. What does the function_graph tracer show?
A nested call tree of function entries and exits, including how long each call took, for whichever function or functions you filter it to.

Q4. How do I trace only one function instead of the whole kernel?
Write the function name into set_graph_function before enabling tracing, which limits the graph to that function and everything it calls.

Q5. Is bpftrace better than ftrace?
They solve overlapping problems; bpftrace is generally lower overhead and faster to write one-off probes with, while ftrace’s function_graph tracer gives a more complete nested call picture with timing.

Q6. How should I read a call stack from dmesg?
Bottom to top: the bottom-most frames are closest to the system call entry point, and the top-most frames are the deepest, most recent calls.

Q7. Why should I turn tracing off after I am done?
Tracing adds CPU overhead and can expose sensitive kernel data; leaving it running unnecessarily wastes resources and widens your security exposure.