Why a Flat Function List Is Not Enough

A plain perf report ranks functions by the percentage of samples that landed inside them. That is useful for a first pass, but on any real workload the top of the list tends to fill up with generic low-level routines — memory allocators, string comparisons, memcpy-style functions — that are already about as optimized as they are going to get. Knowing that malloc consumed 5% of samples does not tell you which part of your code is calling malloc too often. To act on the data you need the calling context, not just the leaf function.

This is exactly the gap call graphs fill. Instead of recording a single instruction pointer per sample, perf can also unwind the stack at the moment of the sample and record the full chain of callers. Once that chain is available, perf report can group samples by call path instead of by function alone, and you can expand a hot leaf function to see who keeps calling it.

Capturing a Call Graph

Call graph capture is enabled with a single flag on perf record:

# perf record -g -a sleep 10

The -g flag tells perf to capture a backtrace with every sample, not just the single instruction pointer where the sample landed. Everything else about recording — the event selected, the sampling frequency set with -F, the target process chosen with -p or a command to launch — works exactly as covered in the previous lecture; -g simply adds stack-unwinding on top.

Stack unwinding is not free — it costs extra CPU and memory per sample, and how well it works depends on the toolchain. There are two unwinding methods available on modern kernels, selected with --call-graph:

# perf record --call-graph dwarf -a sleep 10

Reading the Call Graph in perf report

Open the recorded call graph the same way as a flat report:

# perf report -g

Each hot leaf function now carries a small marker — typically a plus sign — showing that it has an expandable call chain beneath it. Pressing the expand key on that line reveals the callers, each annotated with its own percentage of the parent’s samples. Reading down that expanded chain from a hot leaf function toward main is how you go from “malloc is expensive” to “the JSON parser in module X calls malloc once per token instead of once per document.”

The percentage at each expansion level is relative to the samples of the row above it, not to the whole recording. A caller showing 96% of its parent’s samples is telling you that almost every time this leaf function ran, it was this one caller that invoked it — a strong signal for where to focus optimization effort.

perf annotate: Stepping Inside a Function

Once a call graph has told you which function-and-caller pair matters, the next question is which instructions inside that function are actually slow. That is the job of perf annotate. It reuses the same recorded sample data as perf report, but instead of showing you function-level percentages, it disassembles the function — by invoking a copy of objdump installed locally — and attributes each sample to the exact instruction it landed on.

# perf annotate re_search_internal

Run with no argument, perf annotate opens on the hottest function from the recording; pass a function name to jump straight to it. Because it needs to map addresses back to disassembly, perf annotate requires the same symbol tables as perf report — the executable and any shared libraries involved must be available with their debug information, and kernel symbols must be resolvable, exactly as covered when we first configured perf for symbol resolution.

Each line of disassembly is prefixed with the percentage of samples that landed on that exact instruction. A single instruction carrying a disproportionate share of the samples — a memory load waiting on a cache miss, or a branch that mispredicts often — points at a concrete, fixable hotspot rather than a vague “this function is slow.”

Interleaving Source Code with Assembly

Raw assembly percentages are useful, but matching them back to your own source lines is far more actionable. perf annotate can interleave source code with the disassembly when it can locate the original source file, using the path recorded in the binary’s debug information at compile time.

To find where the compiler recorded the source directory, inspect the DWARF debug info directly:

$ objdump --dwarf lib/mylib.so | grep DW_AT_comp_dir

 <3f>   DW_AT_comp_dir    : /home/dev/build/mylib

For perf annotate to show source lines, the same path — /home/dev/build/mylib in this example — must exist on the machine running perf annotate, with the matching source file(s) inside it. If you built with Yocto and enabled the dbg-pkgs image feature, or installed the individual -dbg package for a component built with Buildroot, the source is typically already installed under /usr/src/debug on the target, and this step is handled for you. Otherwise, copy the relevant source tree to match the recorded compile directory before annotating.

A Worked Example: Annotating a Deliberately Slow Loop

To see the full workflow end to end, build a small, original demo with an obvious hotspot, profile it, and annotate the result.

/* ep_scanloop.c — deliberately inefficient linear scan, for demonstration */
#include <stdio.h>
#include <stdlib.h>

#define N 20000000

static int haystack[N];

int ep_find(int target) {
    int hits = 0;
    for (int i = 0; i < N; i++) {
        if (haystack[i] == target)
            hits++;
    }
    return hits;
}

int main(void) {
    for (int i = 0; i < N; i++)
        haystack[i] = i % 97;

    int total = 0;
    for (int pass = 0; pass < 5; pass++)
        total += ep_find(pass);

    printf("total hits: %d\n", total);
    return 0;
}

Build it with frame pointers kept and debug symbols embedded, so both unwind methods and source interleaving are available:

$ gcc -O2 -fno-omit-frame-pointer -g -o ep_scanloop ep_scanloop.c

Record a call-graph-enabled profile while it runs, then inspect the report and annotate the hot function:

$ perf record -g ./ep_scanloop
$ perf report -g
$ perf annotate ep_find

Expected shape of the output — exact percentages vary by machine:

total hits: N
[ perf record: Captured and wrote perf.data ]

# perf report -g
   98.7%  ep_scanloop  ep_scanloop  [.] ep_find
      - ep_find
          100.00%  main

# perf annotate ep_find
  61.20%   cmp    dword [rax], edi
  22.40%   je     +0x1a
  16.40%   add    rax, 0x4

The call graph confirms ep_find is only ever reached from main — unsurprising here, but on a real codebase this is exactly how you’d discover an unexpected caller. The annotation shows the comparison instruction inside the loop body dominating the samples, which is expected for a linear scan over 20 million integers — and matches where a real optimization (early exit, a better data structure, or vectorization) would actually pay off.

Other Profilers: OProfile and gprof

Before perf existed, two other statistical profilers were the standard tools on Linux, and you will still encounter them in older build systems and legacy documentation. Both are functional subsets of what perf now provides in one tool, but they remain worth recognizing.

OProfile

OProfile is a kernel-level profiler with a long history; modern versions build on the same perf_events kernel infrastructure that perf itself uses, rather than a separate sampling mechanism. It needs two kernel config options enabled — general profiling support, and OProfile’s own kernel component. On a Yocto-built image its userspace tools come from the tools-profile image feature; on Buildroot, from the corresponding package option.

# operf ./ep_scanloop
# opreport

operf collects samples while your program runs (or until you press Ctrl+C), writing them under a local oprofile_data/samples directory; opreport then turns those samples into a summary, similar in spirit to perf report.

gprof

gprof is older still, part of the standard GNU toolchain. Unlike perf and OProfile, it does not require any kernel support at all — it works by combining compile-time instrumentation with a fixed sampling rate. You opt in by adding a compiler and linker flag:

$ gcc -O2 -pg -o ep_scanloop ep_scanloop.c
$ ./ep_scanloop
$ gprof ep_scanloop gmon.out

The -pg flag injects call-counting code into every function’s preamble at compile time. Running the resulting binary produces a gmon.out file in the working directory, and the gprof command reads that file together with the original binary to print a call-count and time summary.

Common Mistakes and Troubleshooting

Best Practices

• Capture a call graph only once you already know, from a flat perf report, roughly which function deserves deeper attention — recording -g for everything by default adds avoidable overhead.
• Prefer --call-graph dwarf for optimized production builds; reserve frame-pointer unwinding for debug builds where you control the compile flags.
• Re-run perf annotate against a normal release build after drawing conclusions from a debug build, since debug flags and lower optimization change instruction-level timing.
• Keep a matching source tree available at the recorded DW_AT_comp_dir path in your build pipeline so annotation with source stays usable long after the original build machine is gone.

Summary and Key Takeaways

• A flat perf report tells you which functions are hot; call graphs (-g / --call-graph) tell you who is calling them.
• Choose fp unwinding for speed on frame-pointer-preserving builds, dwarf unwinding for optimized builds without frame pointers.
• perf annotate attributes samples down to individual instructions, and can interleave original source when the compile-time path is available locally.
• OProfile and gprof predate perf and cover subsets of its functionality — OProfile now shares perf’s kernel sampling infrastructure, while gprof needs no kernel support at all but requires compile-time instrumentation.

Conclusion

Between call graphs and perf annotate, you now have a complete path from “the system is slow” down to the specific instruction responsible — the natural depth limit of statistical, sampling-based profiling. The next lecture in this free linux kernel development course moves from sampling-based profiling to event-based tracing with Ftrace, where instead of periodic samples you get an exact, ordered record of kernel events as they happen.

FAQ