Listing Available Kernel Probes

Before enabling anything, see what LTTng’s kernel tracer can already observe. Every probe here maps to a tracepoint compiled into your running kernel:

$ sudo lttng list --kernel
Kernel events:
-------------
    sched_switch (loglevel: TRACE_EMERG (0)) (type: tracepoint)
    sched_process_fork (loglevel: TRACE_EMERG (0)) (type: tracepoint)
    syscall_entry_openat (loglevel: TRACE_EMERG (0)) (type: tracepoint)
    syscall_exit_openat (loglevel: TRACE_EMERG (0)) (type: tracepoint)
[...]

This list is long — most kernels expose several hundred tracepoints. Piping through grep for a subsystem you care about (scheduler, block layer, a specific syscall) is the fastest way to find what you need instead of scrolling.

Creating A Tracing Session

A session is the container LTTng records into. Give it a name you’ll recognise later — avoid the generic names tutorials often use, since you’ll be running several of these while learning:

$ sudo lttng create ep_trace_demo
Session ep_trace_demo created.
Traces will be written in /home/root/lttng-traces/ep_trace_demo-20260831-101500

$ sudo lttng list
Available tracing sessions:
  1) ep_trace_demo (/home/root/lttng-traces/ep_trace_demo-20260831-101500) [inactive]

The session exists but nothing is being recorded yet — no channel has any events enabled, and even if it did, lttng start hasn’t been called.

Building An Original Demo Workload

Rather than trace an event that just happens in the background, write something small that deliberately generates the events you want to see: a handful of child processes, each opening and closing a file a few times.

/* ep_workload.c — generates fork + openat/close activity for LTTng to capture */
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/wait.h>
#include <fcntl.h>

#define EP_CHILDREN   3
#define EP_OPEN_LOOPS 5

static void ep_child_task(int id)
{
    for (int i = 0; i < EP_OPEN_LOOPS; i++) {
        int fd = open("/tmp/ep_workload.tmp", O_CREAT | O_WRONLY, 0644);
        if (fd >= 0)
            close(fd);
    }
    printf("ep_workload: child %d done\n", id);
    exit(0);
}

int main(void)
{
    printf("ep_workload: spawning %d children\n", EP_CHILDREN);

    for (int i = 0; i < EP_CHILDREN; i++) {
        pid_t pid = fork();
        if (pid == 0)
            ep_child_task(i);
    }

    for (int i = 0; i < EP_CHILDREN; i++)
        wait(NULL);

    printf("ep_workload: all children finished\n");
    return 0;
}

$ gcc -O2 -o ep_workload ep_workload.c
$ ./ep_workload
ep_workload: spawning 3 children
ep_workload: child 0 done
ep_workload: child 1 done
ep_workload: child 2 done
ep_workload: all children finished

Each run of this forks three children and performs fifteen open/close pairs total — enough activity to see clearly in a trace without drowning in noise.

Enabling Events And Capturing A Trace

Enable exactly the tracepoints the demo exercises — this is the “don’t reach for --all in anything you’ll actually keep” habit from the previous lecture:

$ sudo lttng enable-event --kernel sched_process_fork,sched_switch,syscall_entry_openat,syscall_exit_openat
Kernel event sched_process_fork created in channel channel0
Kernel event sched_switch created in channel channel0
Kernel event syscall_entry_openat created in channel channel0
Kernel event syscall_exit_openat created in channel channel0

Confirm the session picked them up, then start recording, run the workload, and stop:

$ sudo lttng start
Tracing started for session ep_trace_demo

$ ./ep_workload
ep_workload: spawning 3 children
ep_workload: child 0 done
ep_workload: child 1 done
ep_workload: child 2 done
ep_workload: all children finished

$ sudo lttng stop
Waiting for data availability
Tracing stopped for session ep_trace_demo

The window between start and stop is your trace — anything the enabled tracepoints saw system-wide during that window is now sitting in the session’s trace directory, not just events from ep_workload specifically.

Reading The Trace With babeltrace2

Point babeltrace2 at the session directory to get a human-readable, one-event-per-line dump:

$ babeltrace2 /home/root/lttng-traces/ep_trace_demo-20260831-101500
[10:15:03.221184532] (+?.?????????) ep-target sched_process_fork: { cpu_id = 0 }, { parent_pid = 4210, child_pid = 4211, ... }
[10:15:03.221198760] (+0.000014228) ep-target syscall_entry_openat: { cpu_id = 0 }, { filename = "/tmp/ep_workload.tmp", flags = 0x241 }
[10:15:03.221201990] (+0.000003230) ep-target syscall_exit_openat: { cpu_id = 0 }, { ret = 7 }
[10:15:03.221340102] (+0.000138112) ep-target sched_switch: { cpu_id = 0 }, { prev_comm = "ep_workload", next_comm = "ep_workload" }
[...]

Because it’s plain text, piping this through grep openat or grep 4211 to follow one specific child process is often faster than opening a graphical viewer for a quick check.

Reading The Trace With Trace Compass

For a zoomable, multi-track timeline — genuinely useful once you’re correlating scheduler activity against syscall latency — import the same trace into the Trace Compass plug-in for Eclipse:

1. Open the Tracing perspective in Eclipse.
2. File → New → Tracing Project, give it a name, click Finish.
3. Right-click the new project → Import.
4. Expand Tracing, select Trace Import.
5. Browse to your ep_trace_demo-* directory, tick the kernel subdirectory, click Finish.
6. Expand the project → Traces → double-click kernel to open the timeline view.

The scheduler track will show your three ep_workload children as distinct rows switching in and out, with the openat/close pairs visible as short-lived events layered on top — a much faster way to spot an unexpectedly long gap than scrolling text output.

Common Mistakes And Troubleshooting

• Forgetting lttng start — a session with events enabled but never started produces an empty trace directory; this is the single most common first-run mistake.
• Running the workload before the session starts — LTTng only records events inside the start/stop window, so a workload that finishes before lttng start leaves nothing to capture.
• Permission errors on lttng create — kernel tracing needs root or membership in the tracing group; a “Warning: Kernel tracer not available” message almost always traces back to this.
• Trace directory grows unexpectedly large — usually caused by leaving sched_switch enabled with --all on a busy multi-core system for longer than intended; scope events narrowly as shown above.

Summary And Key Takeaways

A complete LTTng capture in this free linux kernel development course comes down to six steps: list probes, create a session, enable only the events you need, start, run your workload, and stop. What you enable and how you read the result back — text with babeltrace2, or a graphical timeline with Trace Compass — is entirely up to the question you’re answering. Combined with the ftrace and perf lectures earlier in this free embedded linux course, you now have the full standard toolkit for tracing kernel behaviour in a real embedded system.

FAQ