What You Will Learn

Prerequisites

Why Measuring a System Changes It

Every profiling tool borrows resources from the very system it is trying to describe. A sampling profiler steals CPU cycles to take samples. A tracer writes buffers to memory or disk. Even a lightweight tool like top allocates memory and reads /proc on every refresh. None of this is free, and that has a name in engineering: the observer effect — the act of measuring a quantity disturbs the quantity you are measuring.

This matters practically. If you profile a system that is idling, under synthetic load, with extra logging turned on, or running a debug kernel, you are measuring that variant of the system — not the one your customers will run. Two rules keep profiling data trustworthy:

• Measure on the actual target hardware, not a desktop simulator or emulator, whenever the numbers matter.
• Measure release builds with a realistic workload and as few extra background services as you can get away with.

Notice that top is not a profiler in the strict sense — it never tells you which line of code is expensive. Its job is triage: it points you at which process or subsystem deserves a real profiler’s attention, in seconds, with zero setup.

Symbol Tables and Compile Flags

Before reaching for perf, Ftrace, or LTTng later in this chapter, it helps to understand why profiling tools are picky about how your software was built:

Every one of these changes the binary you ship. The rule of thumb: the more you modify a system to make it observable, the less confident you can be that the measurement still reflects production behaviour. Add debug symbols and frame pointers where you must, but do it deliberately, and re-measure the unmodified build once you’ve found the hotspot.

Reading the top Summary Line

top ships in two common flavours on embedded targets: the compact BusyBox version, and the fuller procps build (also present in Buildroot and the Yocto Project). Both put a CPU summary line near the top of the screen — the second line in BusyBox, the third line in procps. htop is a popular third option with a friendlier interface, built on the same underlying counters.

Reading this line correctly is the whole trick. A system spending most of its time in us/usr is user-space bound — look at the top process. A system spending most of its time in sy/sys is kernel-bound — the culprit is usually a syscall-heavy application, an interrupt storm, or a driver doing more work than it should.

Hands-On: Two Deliberately Different Workloads

Rather than reusing a stock example, let’s build two tiny, original programs — ep_cpuburn and ep_syscall_hammer — that each push a system into one of the two states above, so you can see the difference in your own top output.

1. A user-space-bound workload. This program does nothing but arithmetic in a tight loop — no syscalls, no I/O — so every cycle it burns should show up as us/usr time.

/* ep_cpuburn.c — pure user-space CPU load, no syscalls in the hot loop */
#include <stdio.h>

int main(void)
{
    volatile unsigned long acc = 0;

    for (unsigned long i = 0; i < 4000000000UL; i++)
        acc += i ^ (i << 3);

    printf("done, acc=%lu\n", acc);
    return 0;
}

$ gcc -O0 -o ep_cpuburn ep_cpuburn.c
$ ./ep_cpuburn &
$ top

Mem: 61200K used, 441900K free, 40K shrd, 3100K buff, 33800K cached
CPU:  97% usr   2% sys   0% nic   0% idle   0% io   1% irq   0% sirq
Load average: 0.62 0.20 0.07 2/50 210
  PID PPID USER     STAT   VSZ %VSZ %CPU COMMAND
  210  188 root     R     1868   0%  97% ep_cpuburn

Almost the entire CPU budget is usr, and ep_cpuburn is the one line responsible. This is exactly the case where the next lecture’s tool, perf, should be pointed at ep_cpuburn — not at the kernel.

2. A kernel-bound workload. This one repeatedly asks the kernel to do something trivial but expensive to enter and exit: a syscall loop with no real work in user space.

/* ep_syscall_hammer.c — spends its time crossing into the kernel, not computing */
#include <unistd.h>

int main(void)
{
    for (long i = 0; i < 200000000L; i++)
        getppid();   /* cheap syscall, called relentlessly */

    return 0;
}

$ gcc -O0 -o ep_syscall_hammer ep_syscall_hammer.c
$ ./ep_syscall_hammer &
$ top

Mem: 13500K used, 489500K free, 40K shrd, 0K buff, 2700K cached
CPU:   3% usr  96% sys   0% nic   0% idle   0% io   1% irq   0% sirq
Load average: 0.55 0.14 0.05 2/48 214
  PID PPID USER     STAT   VSZ %VSZ %CPU COMMAND
  214  188 root     R     1912   0%  96% ep_syscall_hammer

Here almost all the time is sys, even though ep_syscall_hammer is still the top line. Profiling this binary’s own code would tell you almost nothing useful — the real work is happening on the kernel side of every getppid() call, which is a job for a whole-system profile with perf, or for Ftrace/LTTng if you need the exact sequence of kernel events.

Per-thread and per-CPU views. By default top aggregates all threads of a process into one line and sums usage across every core. Press H in either BusyBox or procps top to break a multi-threaded process out by thread — essential once you suspect one thread is starving the others. In procps top, press 1 to toggle a per-CPU breakdown, which is how you catch a single-core bottleneck hiding inside a healthy-looking system-wide average.

Common Mistakes and Troubleshooting

Best Practices

Summary and Key Takeaways

Conclusion

This lecture is the entry point to the rest of our free linux kernel development course’s profiling chapter. Once top has told you where to look — a single user-space process, several coupled processes, or the kernel itself — the next lectures take you deeper with perf for CPU profiling, then Ftrace and LTTng for detailed kernel-level tracing, and strace for watching the exact system calls a program makes. Master this first triage step and every tool after it will be aimed at the right target from the start.

FAQ