Why Sampling Works

A profiler’s job is to tell you where a program spends its time. There are two broad families of technique. Instrumenting profilers insert extra code at every function entry and exit and measure exact elapsed time — accurate, but they change the program’s timing and can be heavy to set up. Statistical profilers take the opposite approach: they interrupt the running program at intervals and simply record which function it happened to be in at that instant. A single sample tells you almost nothing. A few hundred samples tell you a great deal, because functions where the program spends more time are proportionally more likely to be caught by a random interrupt. This is the same statistical principle behind opinion polling — you don’t need to ask every voter to get a good estimate of the result.

The “poor man’s profiler” is simply this idea applied using a tool every embedded developer already has installed: GDB. You attach, note the stack, resume, and repeat. No extra packages, no kernel reconfiguration, no cross-compiling a profiling agent for your target — which is exactly why it is worth knowing even in 2026, when far more capable tools exist.

The Manual Sampling Loop

The procedure has four steps, repeated as many times as you have patience for:

Attaching Locally vs. Remotely

On a desktop or a target where GDB itself runs natively, you attach directly to a process ID:

$ gdb -p $(pidof ep_hotspot_demo)
(gdb) bt
(gdb) continue

On a resource-constrained embedded target, running full GDB on the board is often impractical. Instead you run the small gdbserver stub on the target and drive it from a cross-GDB on your build host:

# On the target board:
target$ gdbserver --attach :2345 $(pidof ep_hotspot_demo)

# On the development host:
host$ arm-linux-gnueabihf-gdb ep_hotspot_demo
(gdb) target remote target-ip:2345
(gdb) bt
(gdb) continue

Hands-On Demo: Catching a Hidden Hotspot

Here is a small original demo program with one obviously slow function buried among several fast ones, so you can watch the sampling loop find it.

/* ep_hotspot_demo.c */
#include <stdio.h>
#include <unistd.h>

static void ep_fast_task(void)
{
    volatile int i;
    for (i = 0; i < 1000; i++)
        ;
}

static void ep_slow_task(void)
{
    volatile long i;
    for (i = 0; i < 200000000L; i++)
        ;
}

static void ep_work_cycle(void)
{
    ep_fast_task();
    ep_slow_task();   /* the hidden hotspot */
    ep_fast_task();
}

int main(void)
{
    while (1)
        ep_work_cycle();
    return 0;
}

Build it with debug symbols so backtraces show real function names and line numbers:

$ gcc -g -O0 -o ep_hotspot_demo ep_hotspot_demo.c
$ ./ep_hotspot_demo &

Now attach and sample it a few times:

$ gdb -p $(pidof ep_hotspot_demo)
(gdb) bt
#0  ep_slow_task () at ep_hotspot_demo.c:15
#1  ep_work_cycle () at ep_hotspot_demo.c:21
#2  main () at ep_hotspot_demo.c:27
(gdb) continue
Continuing.
^C
Program received signal SIGINT, Interrupt.
ep_slow_task () at ep_hotspot_demo.c:15
15          for (i = 0; i < 200000000L; i++)
(gdb) bt
#0  ep_slow_task () at ep_hotspot_demo.c:15
#1  ep_work_cycle () at ep_hotspot_demo.c:21
#2  main () at ep_hotspot_demo.c:27
(gdb) continue

Repeat the Ctrl+C / bt / continue cycle ten or fifteen times and you will find that the overwhelming majority of your backtraces land inside ep_slow_task, even though it is a single call among three in ep_work_cycle. That is statistical profiling working exactly as designed — the function consuming the most CPU time gets caught the most often.

Common Mistakes

• Too few samples. Five backtraces prove nothing; aim for at least 20-30 before drawing conclusions.
• Sampling only during idle periods. If you attach right as the program starts, you may only be watching one-time setup code, not steady-state behavior.
• Building without debug symbols. Without -g, backtraces show raw addresses instead of function names, making the samples nearly useless.
• Forgetting the intrusiveness. On latency-sensitive code, this technique can itself trigger the very timing problems you are trying to diagnose.

Summary and Key Takeaways

The poor man’s profiler turns an ordinary debugger into a rough-and-ready statistical profiler: attach, record the stack, resume, repeat. It costs nothing to set up, works on any target with GDB, and is intrusive enough that you should treat its results as a starting hint rather than a final answer. In the next lecture of this free linux kernel development course, we build on the same statistical idea using the kernel’s own perf_events subsystem, which samples with far lower overhead and far higher sample rates.

Frequently Asked Questions