What You Will Learn

• The difference between static and dynamic tracepoints
• How the events directory in tracefs organizes tracepoints by subsystem
• How to enable/disable individual event tracing without touching function tracers
• How to write a custom tracepoint in a kernel module using TRACE_EVENT
• How to scope tracing to a single process with set_ftrace_pid
• Common mistakes that make event tracing look “broken” when it isn’t

Prerequisites

• Comfortable with tracefs mounting and the nop, function, and function_graph tracers (previous lecture in this free embedded linux course)
• A Linux 6.x kernel with CONFIG_FTRACE, CONFIG_DYNAMIC_FTRACE, and tracefs enabled
• Root access on a test VM or board — never experiment with tracing on production systems

Static vs Dynamic Tracepoints

A tracepoint is a fixed instrumentation point the kernel developer places inside a function. When the code path hits it, a trace record can be written into the ring buffer. Tracepoints come in two flavors:

• Static tracepoints — compiled directly into the kernel image at a known location. When disabled, they cost only a few bytes (a no-op jump) and a small metadata structure placed in a separate ELF section. They do not call any function unless explicitly enabled, so the runtime overhead when off is effectively zero.
• Dynamic tracepoints — created and attached at runtime, most commonly through kprobes. They carry more overhead because a probe handler function has to be invoked and checked on every hit.

Almost every core subsystem — scheduler, memory management, block I/O, networking, IRQ handling — already ships static tracepoints. You don’t have to add anything to trace them; you just have to enable the ones you want.

Discovering Available Events

Every static tracepoint in the kernel shows up under /sys/kernel/debug/tracing/events/, organized as <subsystem>/<event_name>. A flat listing of every event uses the <subsystem>:<event> format:

# cat /sys/kernel/debug/tracing/available_events
sched:sched_switch
sched:sched_wakeup
irq:irq_handler_entry
irq:irq_handler_exit
kmem:kmalloc
kmem:kfree
block:block_rq_issue
[...]

That flat list gets unwieldy fast on a modern kernel — you’ll see hundreds of entries. The events directory itself is friendlier because it’s organized as one subdirectory per subsystem:

# ls /sys/kernel/debug/tracing/events
block   fib    irq    napi     rcu    sched   timer
compaction  filelock  kmem   oom      regmap  signal  udp
ext4    gpio   migrate  power   regulator  skb  workqueue
[...]

Drilling into one subsystem — say, timer — shows every individual tracepoint available for that subsystem, each as its own directory with its own enable and filter files:

# ls /sys/kernel/debug/tracing/events/timer
enable   hrtimer_expire_entry   hrtimer_start   timer_expire_entry
filter   hrtimer_expire_exit    itimer_expire   timer_expire_exit
format   hrtimer_cancel         itimer_state    timer_init
hrtimer_init  tick_stop         timer_cancel    timer_start

The format file inside each event directory describes the exact fields recorded for that event — useful when you later want to filter on a specific field value.

Enabling Event Tracing Step by Step

Unlike the function or function_graph tracers, event tracing works fine with current_tracer set to nop — you’re not tracing function entry/exit at all, only the specific events you flip on. Here’s the standard workflow:

# cd /sys/kernel/debug/tracing
# echo 0 > tracing_on            # stop tracing first
# echo > trace                   # clear the ring buffer
# echo nop > current_tracer      # no function tracer noise
# echo 1 > events/timer/enable   # enable all timer subsystem events
# echo 1 > tracing_on            # start recording
# sleep 1
# echo 0 > tracing_on            # stop recording
# echo 0 > events/timer/enable   # disable the event
# cat trace | head -20

You can enable a single event instead of a whole subsystem by pointing at its own enable file, e.g. echo 1 > events/timer/hrtimer_start/enable. There’s also a global shortcut — echo 1 > events/enable turns on every tracepoint in the kernel, which is almost never what you want on a real system since the ring buffer fills instantly.

Writing Your Own Tracepoint

You aren’t limited to tracing existing kernel events — you can define your own in a module using the TRACE_EVENT macro. This is the same mechanism the kernel itself uses internally. Below is an original, minimal example module, ep_tracepoint_demo, that defines and fires a custom tracepoint on every timer tick of a kernel timer.

// ep_trace_events.h
#undef TRACE_SYSTEM
#define TRACE_SYSTEM ep_demo

#if !defined(_EP_TRACE_EVENTS_H) || defined(TRACE_HEADER_MULTI_READ)
#define _EP_TRACE_EVENTS_H

#include <linux/tracepoint.h>

TRACE_EVENT(ep_tick,
    TP_PROTO(unsigned int count),
    TP_ARGS(count),
    TP_STRUCT__entry(
        __field(unsigned int, count)
    ),
    TP_fast_assign(
        __entry->count = count;
    ),
    TP_printk("ep_demo tick count=%u", __entry->count)
);

#endif /* _EP_TRACE_EVENTS_H */

#undef TRACE_INCLUDE_PATH
#define TRACE_INCLUDE_PATH .
#undef TRACE_INCLUDE_FILE
#define TRACE_INCLUDE_FILE ep_trace_events
#include <trace/define_trace.h>

// ep_trace_demo.c
#define CREATE_TRACE_POINTS
#include "ep_trace_events.h"

#include <linux/module.h>
#include <linux/timer.h>

static struct timer_list ep_timer;
static unsigned int ep_count;

static void ep_timer_cb(struct timer_list *t)
{
    ep_count++;
    trace_ep_tick(ep_count);
    mod_timer(&ep_timer, jiffies + HZ);
}

static int __init ep_trace_demo_init(void)
{
    timer_setup(&ep_timer, ep_timer_cb, 0);
    mod_timer(&ep_timer, jiffies + HZ);
    pr_info("ep_trace_demo: loaded, firing ep_tick every second\n");
    return 0;
}

static void __exit ep_trace_demo_exit(void)
{
    del_timer_sync(&ep_timer);
    pr_info("ep_trace_demo: unloaded\n");
}

module_init(ep_trace_demo_init);
module_exit(ep_trace_demo_exit);
MODULE_LICENSE("GPL");
MODULE_DESCRIPTION("EmbeddedPathashala custom tracepoint demo");

Build and load it, then discover and enable the new event exactly like a built-in one:

$ make
$ sudo insmod ep_trace_demo.ko
$ ls /sys/kernel/debug/tracing/events/ep_demo
enable  filter  format  ep_tick
$ sudo sh -c 'echo 1 > /sys/kernel/debug/tracing/events/ep_demo/ep_tick/enable'
$ sleep 3
$ sudo cat /sys/kernel/debug/tracing/trace | tail -5

          <...>-2211  [001] .... 481.201033: ep_tick: ep_demo tick count=1
          <...>-2211  [001] .... 482.201098: ep_tick: ep_demo tick count=2
          <...>-2211  [001] .... 483.201155: ep_tick: ep_demo tick count=3

Scoping a Trace to One Process

Both function tracing and event tracing normally capture activity system-wide, across every process. That’s often too much noise when you’re debugging one misbehaving application. The set_ftrace_pid pseudo-file restricts tracing to a single PID, which you can discover with pgrep for an already-running process:

# pgrep myapp
2451
# echo 2451 > /sys/kernel/debug/tracing/set_ftrace_pid
# echo function_graph > /sys/kernel/debug/tracing/current_tracer
# echo 1 > /sys/kernel/debug/tracing/tracing_on

If the process isn’t running yet, wrap it in a small launcher script that captures its own PID ($$) before exec-ing into the real command, so tracing is armed for that exact PID from the very first instruction:

#!/bin/sh
# ep_trace_launch.sh
echo $$ > /sys/kernel/debug/tracing/set_ftrace_pid
echo function_graph > /sys/kernel/debug/tracing/current_tracer
echo 1 > /sys/kernel/debug/tracing/tracing_on
exec "$@"

$ sudo ./ep_trace_launch.sh ./myapp --run
$ sudo cat /sys/kernel/debug/tracing/trace | head -20

This same trick works with event tracing too — set_ftrace_pid filters any tracer, not just function tracers, so you can combine it with events/<subsystem>/enable to see only the events one process triggers.

Event vs Function Tracing — When to Use Which

Common Mistakes and Troubleshooting

• Forgetting to clear the ring buffer. Old records from a previous session make new output confusing — always echo > trace before a fresh capture.
• Enabling events/enable globally on a busy system. The ring buffer wraps almost instantly, and you lose the events you actually wanted.
• Custom tracepoint doesn’t show up under events/. Almost always a missing #define CREATE_TRACE_POINTS in exactly one .c file, or an include-guard mismatch between the header and source.
• Using set_ftrace_pid without setting a tracer. The PID filter has no effect if current_tracer is still nop and no events are enabled — there’s nothing to filter.
• Reading trace instead of trace_pipe for live monitoring. trace is a static snapshot; trace_pipe streams new entries as they arrive, which is what you want while a workload is running.

Best Practices

• Always disable tracing (tracing_on to 0) and disable events you enabled once you’re done — leftover tracepoints add permanent per-event overhead.
• Prefer subsystem-scoped or single-event enables over the global events/enable switch in any real debugging session.
• When adding custom tracepoints to a driver you maintain, keep the header self-contained and match the naming convention of the subsystem you’re instrumenting — it makes the events discoverable and predictable for other engineers.
• Performance: disabled static tracepoints cost effectively nothing; the expense only shows up once you enable them and the format/write path runs on every hit — budget accordingly on latency-sensitive paths.
• Security: tracefs access requires root by default for good reason — trace output can expose kernel addresses, process arguments, and timing information; never leave broad tracing enabled on a production or multi-tenant system.

Interview Questions

FAQ