What You Will Learn

Prerequisites

Where Ftrace Lives Today

On current mainline kernels, the tracing filesystem (tracefs) is usually auto-mounted at /sys/kernel/tracing. Older material you may find online still refers to /sys/kernel/debug/tracing, which was the path when tracefs was only reachable through debugfs. Both paths still work on most distributions today because tracefs is bind-mounted under debugfs for compatibility, but the standalone tracefs mount is the one to prefer, since it does not require debugfs to be mounted at all.

# Confirm tracefs is mounted
mount | grep tracefs

# If it is not mounted, mount it directly
sudo mount -t tracefs nodev /sys/kernel/tracing

cd /sys/kernel/tracing
ls

An Original Demo Module To Trace

Rather than tracing an existing kernel path, we will build a tiny original module, ep_ftrace_demo, that does a small amount of deliberately traceable work every time we write to a sysfs attribute. This gives us a controlled, repeatable target for both tracers.

// ep_ftrace_demo.c
#include <linux/module.h>
#include <linux/kobject.h>
#include <linux/sysfs.h>
#include <linux/delay.h>

static struct kobject *ep_kobj;

static u32 ep_step_one(u32 seed)
{
    return (seed * 2654435761u) ^ 0x9e3779b9u;
}

static u32 ep_step_two(u32 value)
{
    udelay(50);
    return value + ep_step_one(value);
}

static void ep_run_pipeline(u32 seed)
{
    u32 result = ep_step_two(seed);
    pr_info("ep_ftrace_demo: pipeline result = %u\n", result);
}

static ssize_t trigger_store(struct kobject *kobj, struct kobj_attribute *attr,
                              const char *buf, size_t count)
{
    u32 seed;

    if (kstrtou32(buf, 10, &seed))
        return -EINVAL;

    ep_run_pipeline(seed);
    return count;
}

static struct kobj_attribute trigger_attr = __ATTR_WO(trigger);

static int __init ep_ftrace_demo_init(void)
{
    ep_kobj = kobject_create_and_add("ep_ftrace_demo", kernel_kobj);
    if (!ep_kobj)
        return -ENOMEM;

    return sysfs_create_file(ep_kobj, &trigger_attr.attr);
}

static void __exit ep_ftrace_demo_exit(void)
{
    kobject_put(ep_kobj);
}

module_init(ep_ftrace_demo_init);
module_exit(ep_ftrace_demo_exit);
MODULE_LICENSE("GPL");
MODULE_DESCRIPTION("EmbeddedPathashala Ftrace demo module");

# Build and load
make -C /lib/modules/$(uname -r)/build M=$PWD modules
sudo insmod ep_ftrace_demo.ko

# Trigger the traced path
echo 7 | sudo tee /sys/kernel/kernel/ep_ftrace_demo/trigger

Using the Function Tracer

The function tracer records every traced function entry along with the function that called it. It is the lightest-weight tracer Ftrace offers and a good starting point before reaching for heavier tools like perf.

cd /sys/kernel/tracing
echo function > current_tracer
echo "ep_step_one" > set_ftrace_filter
echo "ep_step_two" >> set_ftrace_filter
echo 1 > tracing_on

echo 7 | sudo tee /sys/kernel/kernel/ep_ftrace_demo/trigger

echo 0 > tracing_on
cat trace

Each line reads right to left: the function on the right called the function on the left. This is enough to answer “was my function called, and by what?” without any instrumentation code of your own.

Reading the function_graph Tracer

The function_graph tracer goes further: it shows entry and exit as matching braces and reports how long each call took. This makes nested call relationships and hot paths immediately visible.

echo function_graph > current_tracer
echo 1 > tracing_on
echo 9 | sudo tee /sys/kernel/kernel/ep_ftrace_demo/trigger
echo 0 > tracing_on
cat trace

A + before the duration means the call took more than 10 microseconds; an ! would mean more than 100 microseconds. Here the udelay(50) inside ep_step_two() is clearly visible as the dominant cost, which is exactly the kind of insight the flat function tracer cannot give you at a glance.

Dynamic Ftrace and Filtering at Scale

A stock kernel exposes tens of thousands of traceable functions in available_filter_functions. Tracing all of them at once would be both noisy and expensive. CONFIG_DYNAMIC_FTRACE, enabled on virtually every distribution kernel today, solves this in two ways.

You move a function name from available_filter_functions into set_ftrace_filter to trace it, and into set_ftrace_notrace to explicitly exclude it. Wildcards are supported, which is useful for tracing an entire subsystem by prefix.

# Trace every function starting with ep_
echo "ep_*" > set_ftrace_filter

# Exclude one noisy function from that set
echo "ep_step_one" > set_ftrace_notrace

# Clear all filters (trace everything again)
echo > set_ftrace_filter

Real-World Use Cases

• Driver bring-up: confirming a probe path or interrupt handler is actually being entered on real hardware.
• Latency hunting: using function_graph durations to spot an unexpectedly slow function in an interrupt or IPC path.
• Regression triage: filtering to one subsystem prefix (e.g. tcp_*) after a kernel bump to see if call patterns changed.

Common Mistakes and Troubleshooting

Best Practices

• Always set current_tracer before touching set_ftrace_filter, to avoid a moment where an unfiltered tracer is briefly active.
• Use function_graph when you need timing; use plain function when you only need call presence, since it is cheaper.
• Reset state with echo nop > current_tracer and clear filters when you finish a debugging session, especially on shared systems.

Performance and Security Considerations

Summary and Key Takeaways

Conclusion

Ftrace’s function and function_graph tracers are often the fastest path from “something feels slow” to “here is the exact function and duration responsible.” Because they ship in the mainline kernel with zero extra tooling, they belong in every kernel developer’s first line of debugging tools — well before reaching for heavier profilers. In the next lecture of this free linux kernel development course, we go one level deeper with trace events, which attach real parameters to each recorded call.

Frequently Asked Questions