Why LTTng Belongs In A Free Linux Kernel Development Course

ftrace is excellent for “what is my kernel doing right now” questions on a single machine. perf is excellent for “where is the CPU spending its cycles” questions. Neither was designed from the ground up to merge kernel events and application events into one timeline, stream that timeline off an embedded target with low overhead, and hand you a format that a graphical viewer can render and zoom into. LTTng was designed for precisely that job. It grew out of the original Linux Trace Toolkit, was re-architected by Mathieu Desnoyers into the “next generation” tracer, and today ships as the LTTng 2.15 release series (“Péché Mortel”), maintained by EfficiOS with an active open-source community.

For anyone building embedded systems where a live JTAG debugger isn’t practical — a deployed gateway, a robotics controller, a set-top box — LTTng’s ability to capture a low-overhead, timestamped, correlated trace and pull it off the device after the fact is the reason it keeps appearing in this free linux kernel development course.

LTTng Architecture: Session Daemon, Tracer, Consumer

LTTng is not one binary — it’s a small distributed system that runs entirely on your target. Three pieces cooperate:

• Session daemon (lttng-sessiond) — the control-plane process. Every lttng command you type talks to this daemon over a Unix socket; it tracks sessions, enabled events, and channel configuration.
• Kernel tracer — a set of kernel modules (built by the lttng-modules package) that hook into the same tracepoint infrastructure ftrace uses, but write records into LTTng’s own high-throughput ring buffers instead of the ftrace ring buffer.
• User-space tracer (liblttng-ust) — a library your applications link against to emit their own custom trace events, timestamped on the same clock as the kernel events.

A fourth piece, the consumer daemon (lttng-consumerd), is spawned automatically by the session daemon the moment you create your first event. It drains the ring buffers to disk (or streams them over the network to a relay daemon) so the tracer itself never blocks on I/O.

To actually view what got captured you need a separate reader — babeltrace2 for a text dump on the command line, or the Trace Compass Eclipse plug-in for a graphical, zoomable timeline. Neither ships as part of LTTng itself; both are separate packages you install alongside it. We’ll use both in the next lecture.

Kernel Requirements For LTTng

Because the kernel tracer rides on the standard tracepoint mechanism, your kernel .config needs the same option ftrace needs:

CONFIG_TRACEPOINTS=y

In menuconfig this is pulled in automatically as soon as you enable any tracer under Kernel hacking → Tracers — enabling the function tracer, for instance, is enough. It’s worth double-checking directly rather than assuming:

$ zcat /proc/config.gz | grep CONFIG_TRACEPOINTS
CONFIG_TRACEPOINTS=y

You’ll also need matching versions of lttng-modules and lttng-tools — the modules package builds out-of-tree kernel modules against your exact kernel headers, so a version skew between the two is one of the most common first-time failures (see Troubleshooting below).

Adding LTTng To A Yocto Image

The OpenEmbedded/Yocto layers already carry recipes for all three LTTng packages, so integration is a one-line addition to your image recipe or local.conf. Using current bitbake override syntax:

IMAGE_INSTALL:append = " lttng-tools lttng-modules lttng-ust"

If you plan to read traces directly on the target instead of copying them to a host, pull in the reader too:

IMAGE_INSTALL:append = " babeltrace2"

Two things trip people up here. First, lttng-modules is a kernel-version-specific recipe — a bitbake virtual/kernel -c cleanall followed by a rebuild is the fastest fix if it fails to build after a kernel bump. Second, remember the legacy _append underscore syntax still works on most Yocto releases for backward compatibility, but the colon form (:append) is the current, recommended syntax and is what you should reach for in new recipes.

Adding LTTng To A Buildroot Image

Buildroot exposes each LTTng component as its own menuconfig option rather than a single meta-package. Enable these under Target packages → Debugging, profiling and benchmark:

Buildroot Option	What It Enables
BR2_PACKAGE_LTTNG_MODULES	Out-of-tree kernel tracer modules
BR2_PACKAGE_LTTNG_TOOLS	The lttng CLI and session/consumer daemons
BR2_PACKAGE_LTTNG_LIBUST	User-space tracing library (under Target packages → Libraries → Other)
BR2_PACKAGE_BABELTRACE2	Command-line trace reader for the target

Buildroot also builds a host-side copy of babeltrace2 automatically and drops it in output/host/usr/bin/, so you can read traces on your build machine without installing anything extra there.

LTTng vs Ftrace vs Perf

Tool	Best For	User-Space Events	Typical Overhead
ftrace	Ad-hoc, in-kernel-only investigation	No	Low, single-CPU friendly
perf	CPU profiling, hotspot analysis	Via probes, sampling-based	Low to moderate
LTTng	Correlated kernel + app timelines, production capture	Yes, first-class	Very low, designed for continuous tracing

None of these tools replace each other — most embedded engineers who complete this free linux kernel development course keep all three in rotation and pick based on the question they’re asking, not out of habit.

Common Mistakes And Troubleshooting

• Version mismatch between lttng-tools and lttng-modules — the session daemon logs a module load failure at session creation; keep both packages pinned to the same LTTng release series.
• Forgetting the consumer daemon needs disk space — a long-running session with no lttng stop will happily fill your rootfs; always bound sessions with a channel size or a clear stop point during development.
• Assuming CONFIG_TRACEPOINTS is on by default — many minimal embedded kernel configs strip it out; verify with /proc/config.gz before debugging “no kernel events” further upstream.

Summary And Key Takeaways

LTTng adds a fourth option alongside ftrace and perf in any free linux device drivers course toolkit: a distributed, low-overhead tracer built from a session daemon, kernel modules, and a user-space library, with Yocto and Buildroot both offering first-class packaging. The architecture — control plane, tracer, consumer, and a separate reader — is what lets it scale from a five-minute debugging session to a continuously-running production trace pipeline. The next lecture in this free embedded linux course puts all of this to work with an actual capture session.

FAQ