What You Will Learn

Prerequisites

Why Trace Events Exist

A raw function tracer entry only tells you a function was called. A trace event, by contrast, is hand-written by the subsystem’s developer to capture the specific parameters that matter — for example, the size and pointer returned by an allocator, or the device and register touched by a driver. Because trace events are defined once with TRACE_EVENT and compiled into the kernel, there are well over a thousand of them available today across memory management, scheduling, block I/O, networking, and more, listed at runtime in available_events.

Each event lives under a subdirectory named tracing/events/<subsystem>/<event>, for example events/kmem/kmalloc, and exposes a small, consistent set of files:

An Original Demo Event

To see this end to end, we define a small original trace event, ep_alloc, inside a demo module that mimics a simple allocation path. This is written from scratch and does not reuse any existing kernel subsystem’s event definitions.

// ep_trace_events.h
#undef TRACE_SYSTEM
#define TRACE_SYSTEM ep_demo

#if !defined(_EP_TRACE_EVENTS_H) || defined(TRACE_HEADER_MULTI_READ)
#define _EP_TRACE_EVENTS_H

#include <linux/tracepoint.h>

TRACE_EVENT(ep_alloc,
    TP_PROTO(size_t bytes, void *ptr),
    TP_ARGS(bytes, ptr),

    TP_STRUCT__entry(
        __field(size_t, bytes)
        __field(void *, ptr)
    ),

    TP_fast_assign(
        __entry->bytes = bytes;
        __entry->ptr = ptr;
    ),

    TP_printk("bytes=%zu ptr=%p", __entry->bytes, __entry->ptr)
);

#endif

#undef TRACE_INCLUDE_PATH
#define TRACE_INCLUDE_PATH .
#undef TRACE_INCLUDE_FILE
#define TRACE_INCLUDE_FILE ep_trace_events
#include <trace/define_trace.h>

// ep_trace_demo.c
#define CREATE_TRACE_POINTS
#include "ep_trace_events.h"

#include <linux/module.h>
#include <linux/kobject.h>
#include <linux/slab.h>

static struct kobject *ep_kobj;

static ssize_t alloc_store(struct kobject *kobj, struct kobj_attribute *attr,
                            const char *buf, size_t count)
{
    size_t bytes;
    void *ptr;

    if (kstrtoul(buf, 10, &bytes))
        return -EINVAL;

    ptr = kmalloc(bytes, GFP_KERNEL);
    trace_ep_alloc(bytes, ptr);
    kfree(ptr);

    return count;
}

static struct kobj_attribute alloc_attr = __ATTR_WO(alloc);

static int __init ep_trace_demo_init(void)
{
    ep_kobj = kobject_create_and_add("ep_trace_demo", kernel_kobj);
    if (!ep_kobj)
        return -ENOMEM;

    return sysfs_create_file(ep_kobj, &alloc_attr.attr);
}

static void __exit ep_trace_demo_exit(void)
{
    kobject_put(ep_kobj);
}

module_init(ep_trace_demo_init);
module_exit(ep_trace_demo_exit);
MODULE_LICENSE("GPL");
MODULE_DESCRIPTION("EmbeddedPathashala trace event demo module");

make -C /lib/modules/$(uname -r)/build M=$PWD modules
sudo insmod ep_trace_demo.ko

Enabling and Reading the Event

Event tracing is independent of the function tracers, so we start from the nop tracer and enable only the event we defined.

cd /sys/kernel/tracing
echo nop > current_tracer
echo 1 > events/ep_demo/ep_alloc/enable

echo 256 | sudo tee /sys/kernel/kernel/ep_trace_demo/alloc
echo 4096 | sudo tee /sys/kernel/kernel/ep_trace_demo/alloc

cat trace

Unlike the plain function tracer, this line carries real payload: the exact byte count and pointer for each allocation, exactly as the developer chose to record it.

You can also enable multiple events at once by writing to set_event instead of toggling each enable file individually:

echo "ep_demo:ep_alloc" > set_event

Filtering by Parameter

The real power of trace events is the filter file, which lets you match on the fields you saw in format — without writing any kernel code.

# Only record allocations larger than 1KB
echo "bytes > 1024" > events/ep_demo/ep_alloc/filter

echo 256 | sudo tee /sys/kernel/kernel/ep_trace_demo/alloc
echo 4096 | sudo tee /sys/kernel/kernel/ep_trace_demo/alloc

cat trace
# Only the 4096-byte allocation appears

Real-World Use Cases

• Memory debugging: watching kmem:kmalloc and kmem:kfree to spot allocation size regressions or leaks.
• Driver validation: confirming an interrupt handler fires with the expected register values during hardware bring-up.
• Filtered production diagnostics: enabling one narrowly filtered event on a live system with negligible overhead, instead of a full function trace.

Common Mistakes and Troubleshooting

Best Practices

• Prefer trace events over function tracing whenever you need parameter data, not just call presence.
• Use filter to narrow high-frequency events (like memory allocation) before enabling them on a busy system.
• Clear set_event and any filters when you finish, to avoid leaving hidden overhead for the next person using the system.

Performance and Security Considerations

Summary and Key Takeaways

Conclusion

Trace events close the gap that function tracing leaves open: they tell you not just that something happened, but exactly what data was involved. Combined with the function and function_graph tracers from the previous lecture, you now have the two core building blocks of Ftrace-based kernel debugging — one for call flow, one for parameter-level detail. This wraps up the Ftrace portion of our free linux kernel development course.

Frequently Asked Questions