Introducing perf

perf is short for the Linux performance event counter subsystem, perf_events, and it is also the name of the command-line tool used to drive it. The subsystem has shipped in the mainline kernel for well over a decade, and the tool that talks to it lives directly in the kernel source tree under tools/perf. Because the tool and the kernel subsystem are versioned together, you should always build the perf binary from the exact same kernel source tree you are running — a perf built against a different kernel version can misbehave or fail outright.

The original motivation for perf was to give every architecture a unified way to read the registers of the performance measurement unit (PMU), a block of hardware counters built into essentially every modern CPU core. Once that abstraction existed, it made sense to extend perf to cover software counters and tracepoints too — so today perf is really a general framework of event counters governed by rules about what to count and when.

Those rules are what make perf so flexible for embedded and system-level debugging. You can scope a capture to the whole system, to just the kernel, or to a single process and its children; you can run it across every CPU or pin it to one core. That lets you start broad — “what is this whole board doing?” — and then zero in on a specific driver, application, or library call that looks suspicious.

Configuring the Kernel for perf

Two things need to be true before perf can do anything useful: the kernel must be built with perf_events support, and a matching perf binary must exist on the target.

If you also plan to profile using tracepoints — a topic for a later lecture in this free linux kernel development course — make sure the Ftrace-related options are enabled too, and it is worth turning on CONFIG_DEBUG_INFO at the same time so the kernel itself carries useful debug information.

Building perf With the Yocto Project

A standard linux-yocto kernel already has perf_events enabled, so there is nothing to change on the kernel config side. To pull the perf tool itself onto the image — plus debug symbols and the matching vmlinux — add the following to your conf/local.conf:

EXTRA_IMAGE_FEATURES = "debug-tweaks dbg-pkgs tools-profile"
IMAGE_INSTALL:append = " kernel-vmlinux"

The tools-profile feature pulls in perf along with a few related profiling utilities, and dbg-pkgs ensures the packages on your image keep their debug symbols instead of being stripped.

Building perf With Buildroot

Buildroot’s default kernel configurations frequently leave perf_events disabled, so check your kernel config first using the path shown above. Then, from Buildroot’s own menuconfig, enable:

• BR2_LINUX_KERNEL_TOOL_PERF under Kernel → Linux Kernel Tools
• BR2_ENABLE_DEBUG under Build options → build packages with debugging symbols
• BR2_STRIP = none under Build options → strip command for binaries on target, so debug symbols aren’t stripped away on install

$ make clean
$ make

Once the build finishes, copy the generated vmlinux into your target image manually — Buildroot does not do this for you automatically.

Recording a Profile: perf record and perf report

Creating a profile with perf is a two-stage process. perf record captures samples and writes them to a file — perf.data by default. perf report then reads that file and presents the results. Both commands run on the target. The default event counter is cycles, a generic hardware counter mapped to a PMU register that counts core clock cycles, and samples are filtered to the command you launch and its children.

Here is an original demo program that deliberately spends most of its time in one function, so you can watch perf correctly identify it:

/* ep_cpu_burn.c */
#include <stdio.h>

static void ep_light_work(void)
{
    volatile int i;
    for (i = 0; i < 5000; i++)
        ;
}

static void ep_heavy_work(void)
{
    volatile long i;
    for (i = 0; i < 500000000L; i++)
        ;
}

int main(void)
{
    int round;
    for (round = 0; round < 20; round++) {
        ep_light_work();
        ep_heavy_work();
    }
    printf("ep_cpu_burn: done\n");
    return 0;
}

$ gcc -g -O0 -o ep_cpu_burn ep_cpu_burn.c

Now record a profile of it running under perf:

# perf record ./ep_cpu_burn
[ perf record: Woken up 3 times to write data ]
[ perf record: Captured and wrote 0.412 MB perf.data (~9800 samples) ]

# ls -l perf.data
-rw------- 1 root root 421280 Aug 31 2026 perf.data

Read the results back with perf report:

# perf report

Just as the code predicts, the overwhelming majority of samples land in ep_heavy_work. This is the same statistical principle you saw with the GDB-based poor man’s profiler in the previous lecture, but here perf gathered nearly ten thousand samples automatically instead of the handful you could realistically collect by hand.

Choosing a perf report Interface

perf report offers three ways to browse the results, selectable on the command line:

On a headless embedded target reached only over a serial console or SSH, --tui is almost always the right default — which is exactly why the tool picks it automatically when you run perf report with no flags.

Real-World Use Cases

• Driver call overhead: profile a user-space test harness that exercises your kernel driver via ioctl calls to see whether time is spent in the driver, in libc, or in your own application logic.
• Boot-time investigation: record system-wide (no target command, just a duration) during boot to find which service or driver init routine is taking longest.
• Library function regression: compare two perf reports taken before and after a library upgrade to see whether a specific function’s share of samples increased.

Common Mistakes

• Mismatched perf and kernel versions. Since the tool and subsystem are developed together, a perf binary built from a different kernel tree can report garbage or fail to start.
• Missing debug symbols. Without them, perf report shows raw hex addresses instead of function names — always keep vmlinux and unstripped binaries available.
• Forgetting CONFIG_PERF_EVENTS on a custom Buildroot kernel config. Unlike Yocto’s default kernel, Buildroot kernel configs frequently ship with it off.
• Profiling too short a run. A workload that finishes in well under a second may not generate enough samples for a meaningful report.

Performance Considerations

perf’s PMU-driven sampling has dramatically lower overhead than debugger-based sampling because the hardware itself triggers each sample, rather than a human or script repeatedly stopping the process. This lets you gather thousands of samples per second with minimal distortion of the program’s real timing — a meaningful advantage on latency-sensitive embedded workloads where the GDB technique from the previous lecture would itself skew the measurement.

Summary and Key Takeaways

perf gives you the same statistical-sampling idea as the poor man’s profiler, but implemented with PMU hardware counters instead of a debugger, which means far lower overhead and far more samples. Getting it running takes two steps — enabling CONFIG_PERF_EVENTS in the kernel and building the perf tool for your target, whether via Yocto’s tools-profile feature or Buildroot’s BR2_LINUX_KERNEL_TOOL_PERF — after which perf record and perf report give you a real, statistically sound picture of where a program spends its time.

Frequently Asked Questions