The Observer Effect: Measurement Has a Cost

In physics, checking the current in a wire means inserting a small resistor and reading the voltage drop across it — but that resistor changes the very current you are trying to measure. Software profiling has the identical problem. A profiler that samples the program counter every millisecond, a tracer that logs every function entry, or a debugger single-stepping through code all consume CPU cycles, evict cache lines, and sometimes touch disk or network I/O to store their own output. None of that activity would exist if you weren’t watching.

This is not a reason to distrust profiling — it is a reason to be deliberate about how you profile. Three consequences follow directly from the observer effect:

• Timing shifts. A function that looked fast in isolation may look slower under a sampling profiler simply because the profiler’s own interrupt handler is stealing cycles from it.
• Cache and TLB pollution. Instrumentation code and the buffers it writes into compete for the same L1/L2 cache lines as your workload, so cache-miss-sensitive code paths can look artificially worse than they are in production.
• Heisenbugs in reverse. Sometimes a bug that reproduces reliably in normal operation becomes intermittent or vanishes entirely once a tracer is attached, because the timing window that triggered it no longer exists.

Why a Free Linux Kernel Development Course Spends Time on Symbol Tables

Once you accept that every measurement has overhead, the next question is: what does a profiler actually give you back? Under the hood, a sampling profiler like perf or a tracer like Ftrace does not record “function ep_fib ran for 4ms.” It records raw instruction pointer addresses — plain 64-bit numbers. Turning those numbers into function names and line numbers you can act on is a completely separate step, and it depends entirely on two things being present: a symbol table and, ideally, debug information.

A symbol table is a section inside an ELF binary (typically .symtab, or .dynsym for the exported subset) that maps addresses to names — function names, global variable names, and so on. Debug information is a much richer, separate set of ELF sections (in the DWARF format on Linux) that additionally maps addresses to source file names, line numbers, and even local variable locations. A binary can have a symbol table with no debug info, debug info with a stripped symbol table, or both, or neither. What you compiled with decides which of these you get.

Compile Flags That Decide What a Profiler Can See

The single most common mistake in any Linux profiling workflow — and one this free Linux kernel development course wants you to avoid from day one — is profiling a binary that was never built to be profiled. The table below summarizes the flags that matter and what each one actually buys you.

Hands-On: Watching Symbols Disappear Under Optimization

The clearest way to see why compile flags matter is to build the exact same source file twice — once the way you would ship it, and once the way you would profile it — and compare what a symbol table actually contains. Here is a small, original workload written for this lecture.

/* ep_profile_demo.c
 * A tiny CPU-bound workload used to explore how compiler flags
 * affect symbol tables and debug information. */

#include <stdio.h>
#include <stdlib.h>

static long ep_fib(int n)
{
    if (n < 2)
        return n;
    return ep_fib(n - 1) + ep_fib(n - 2);
}

static long ep_sum_squares(int limit)
{
    long total = 0;
    for (int i = 0; i < limit; i++)
        total += (long)i * i;
    return total;
}

int main(int argc, char **argv)
{
    int n = (argc > 1) ? atoi(argv[1]) : 30;

    long fib_result = ep_fib(n);
    long sum_result = ep_sum_squares(20000000);

    printf("fib(%d) = %ld\n", n, fib_result);
    printf("sum_squares = %ld\n", sum_result);

    return 0;
}

Now build it two different ways:

# A "shipping" release build — no debug info, aggressive optimization
gcc -O2 -o ep_profile_release ep_profile_demo.c

# A "profiling-ready" build — debug info, frame pointers kept
gcc -O0 -g -fno-omit-frame-pointer -o ep_profile_debug ep_profile_demo.c

Compare what each binary’s symbol table actually contains:

$ nm ep_profile_release | grep ep_
$ nm ep_profile_debug | grep ep_
0000000000401136 t ep_fib
00000000004011a2 t ep_sum_squares

With the debug build, you can go one step further and resolve a raw address back to a source line:

$ objdump -d ep_profile_debug | grep -A1 "<ep_fib>:"
0000000000401136 <ep_fib>:
  401136:   f3 0f 1e fa             endbr64

$ addr2line -e ep_profile_debug -f 0x401136
ep_fib
/home/user/ep_profile_demo.c:8

Run the same addr2line command against ep_profile_release and it will either fail to resolve the address to any function you recognize, or point you at main instead — because that is genuinely where the inlined code now lives. This small experiment is the entire “symbol tables and compile flags” problem in miniature: the system you observe and the system you can meaningfully report on are not automatically the same system.

Continuing This Free Linux Kernel Development Course: Applying It to Real Tools

Every profiling and tracing tool covered later in this series inherits this same dependency:

• perf works best when the running binary — application or kernel — has an unstripped copy with matching symbols available, either installed on the target or supplied via perf‘s build-id lookup mechanism.
• Ftrace and LTTng for kernel-side tracing need the kernel itself built with the relevant tracer config options enabled, which in practice means building and deploying a kernel image specifically for this kind of work rather than assuming your production kernel already has it.
• gprof requires the -pg instrumentation flag at compile time — without it, there is no call-graph data to collect no matter how the program is run afterward.

The pattern across all three: decide what you need to measure, then choose your build flags before you start, not after you’ve already collected a confusing set of hex addresses.

Common Mistakes and Troubleshooting

Best Practices

• Treat “profiling build” as its own build configuration — release optimization plus -g plus -fno-omit-frame-pointer — rather than reusing either your debug build or your stripped release build unmodified.
• Keep unstripped binaries archived alongside every release you ship, indexed by version or build ID, so you can symbolize a crash or a profile from production weeks later.
• Change one variable at a time. If you need call graphs, add frame pointers. If you need line-level attribution, add debug info. Don’t drop to -O0 “just in case” — it changes the very performance characteristics you’re trying to measure.
• Adopt a wait-and-see approach to instrumentation: only add tracepoints or recompile with instrumentation once you have a specific hypothesis to test, since every change you make to the system is a change to what you’re measuring.

Performance Considerations

Debug information adds disk and binary size but no runtime cost, since the loader never touches DWARF sections during execution. Frame pointer retention typically costs low single-digit percent overhead on modern x86-64 and ARM64 targets because it reserves one register that would otherwise be available for the optimizer — usually an acceptable trade for reliable call graphs in production profiling.

Security Considerations

Debug information can reveal internal function names, file paths, and sometimes source line content indirectly through crash reports — treat unstripped debug binaries for shipped products as sensitive build artifacts, store them separately from what you distribute publicly, and strip symbols from anything that leaves your build infrastructure.

Summary and Key Takeaways

• Every profiling or tracing tool changes the system it observes — plan measurements with that overhead in mind, and prefer target hardware, release builds, and realistic data.
• A symbol table maps addresses to names; DWARF debug info (from -g) additionally maps addresses to source file and line — a profiler needs at least the former and ideally both.
• Optimization can delete the very functions you want to measure through inlining; frame pointer omission can break call-graph unwinding; instrumentation-based tools like gprof need their own dedicated compile flag.
• The fix in every case is deciding your build configuration before you start profiling, not reverse-engineering hex addresses afterward.

This foundation — the observer effect on one side, symbol tables and compile flags on the other — is what makes every tool later in this free Linux kernel development course actually usable. Skip it, and even the most sophisticated tracer will hand you a report full of addresses you can’t act on.

Interview Questions

Frequently Asked Questions