Complete the previous two lectures on Linux scheduling policies and the EEVDF scheduler. You need a Linux machine (VM is fine) with root access. A multicore system makes the visualizations far more interesting.

Multicore hardware means true parallelism: on a 4-core board, four threads genuinely execute at the same instant. That brings throughput — and also synchronization headaches with shared data, cache-line bouncing when a thread migrates cores, and mysterious latency spikes when a high-priority thread preempts yours. Reading top tells you how much CPU each process used; it cannot tell you when, on which core, or who preempted whom. For that you need a processor timeline, and perf is the modern, in-tree way to record one. Other capable tracers exist too — ftrace (built into every kernel), LTTng, and Trace Compass for GUI analysis — and we will position them at the end.

perf is unusual: although it is a userspace tool, it lives inside the kernel source tree and is tightly coupled to the kernel version it runs on. Distributions therefore package it per-kernel:

# Ubuntu / Debian
sudo apt install linux-tools-common linux-tools-$(uname -r)

# Fedora
sudo dnf install perf

# Verify
perf --version

Important for kernel developers: if you boot a custom-built kernel, the distro package for your exact version will not exist. Either boot a standard distro kernel while learning perf, or build perf yourself from your kernel source tree (it lives under tools/perf/ inside the source):

# Build perf from your own kernel source
cd ~/kernels/linux
cd tools/perf
make
sudo cp perf /usr/local/bin/perf-custom

Think of perf top as a microscope version of top: instead of processes, it shows the hottest functions (kernel and userspace) consuming CPU right now, refreshed live.

# Basic live profile of the whole system
sudo perf top

# Group hot spots by process name and shared object
sudo perf top --sort comm,dso

# Even finer: per PID, per symbol
sudo perf top --sort pid,comm,dso,symbol

Here comm means the command (process) name and dso means dynamic shared object — the binary or library the sampled instruction belongs to. Generate some load in another terminal (for example yes > /dev/null) and watch it climb the list instantly.

The most direct tool to visualize Linux CPU scheduling with perf is the perf sched subcommand. It records every scheduler event (context switches, wakeups, migrations) and offers several analysis views.

# Record all scheduler events system-wide for 5 seconds
sudo perf sched record -- sleep 5

# View a per-CPU map: which task occupied which core, over time
sudo perf sched map

# Per-task scheduling latency report (wait time before running)
sudo perf sched latency --sort max

# Timeline of individual events with timestamps
sudo perf sched timehist

How to read the outputs:

• perf sched map prints one column per CPU; each letter is a task. Watching a letter jump between columns literally shows a thread migrating between cores.
• perf sched latency shows, per task, how long it sat runnable but waiting. A large max value is a smoking gun for preemption or priority problems.
• perf sched timehist gives an event-by-event history: wait time, scheduling delay, and run duration for every switch.

perf timechart records scheduler and power events, then renders them as an SVG picture you can open in any browser — a genuine processor timeline showing every CPU as a horizontal lane and every task as colored segments.

# Record system-wide events; press Ctrl+C to stop
sudo perf timechart record

# The recording lands in a binary file: perf.data
# Convert it to an SVG timeline
sudo perf timechart -o timeline.svg

# Open timeline.svg in a browser and zoom in

# Record only one workload instead of the whole system
sudo perf timechart record -- ./my_workload

In the SVG, look for: long solid bars (CPU-bound tasks), rapid thin slivers (interactive tasks waking often), and the same task color appearing in different CPU lanes (migrations).

Try this complete experiment to connect all three lectures of this free embedded systems course track:

# Terminal 1: a normal CPU hog pinned to CPU 1
taskset -c 1 sh -c 'while :; do :; done' &

# Terminal 2: an RT hog on the same CPU (careful: 3 seconds only!)
sudo timeout 3 taskset -c 1 chrt --fifo 50 sh -c 'while :; do :; done'

# Terminal 3: record what happened
sudo perf sched record -- sleep 5
sudo perf sched latency --sort max | head

# Cleanup
kill %1

In the latency report you will see the normal shell loop accumulating huge wait times during those 3 seconds — visible proof that the rt class starves the fair class, exactly as the class hierarchy predicts.

perf is not the only lens. Three complementary tools deserve a place on your radar:

• ftrace — built into every kernel, controlled via /sys/kernel/tracing. Enable sched_switch events and you get raw scheduler traces with zero extra software; the trace-cmd and kernelshark tools make it friendlier.
• LTTng — a low-overhead tracer suited to long recordings on production and embedded systems.
• Trace Compass — a graphical analyzer that turns LTTng/ftrace/perf traces into interactive timelines, resource views, and critical-path analysis.

# 10-second taste of ftrace scheduler tracing
cd /sys/kernel/tracing
echo 1 | sudo tee events/sched/sched_switch/enable
sudo cat trace_pipe | head -20
echo 0 | sudo tee events/sched/sched_switch/enable

• perf version mismatch: “WARNING: perf not found for kernel” means the linux-tools package for your running kernel is missing — install linux-tools-$(uname -r) or build from source.
• Permission errors: either run with sudo or relax /proc/sys/kernel/perf_event_paranoid (understand the security trade-off first).
• Missing symbols (hex addresses everywhere): install debug symbol packages, and check /proc/sys/kernel/kptr_restrict for kernel symbols.
• Huge perf.data files: scheduler events are high-frequency; keep recordings short or scope them to one command.
• Tracing overhead on tiny embedded boards: recording itself consumes CPU; prefer short windows or LTTng for long sessions.

• Reproduce the problem first, then record the shortest window that captures it.
• Prefer perf sched latency as your first stop for “my thread is slow” complaints — it directly measures time spent waiting for a CPU.
• Pin workloads with taskset during experiments to make timelines easier to read.
• Keep before/after recordings when tuning priorities so improvements are provable, not anecdotal.

Q1. Why must the perf tool version match the running kernel?

perf lives in the kernel source tree and depends on kernel-version-specific event interfaces, so distributions ship it per-kernel as linux-tools-$(uname -r); on custom kernels you build it from the same source tree.

Q2. Which perf subcommand shows scheduling latency per task?

perf sched latency (after perf sched record) reports how long each task waited between becoming runnable and actually running, including the maximum delay.

Q3. How can you see task migrations between CPU cores?

perf sched map shows a per-CPU column view where a task’s letter moving across columns indicates migration; perf timechart shows the same graphically in SVG.

Q4. What is the difference between top and perf top?

top aggregates CPU usage per process; perf top samples the instruction pointer and attributes usage to individual functions/symbols in both kernel and userspace, live.

Q5. When would you choose ftrace or LTTng over perf?

ftrace when you want zero-install tracing on any kernel (including tiny embedded builds); LTTng for low-overhead, long-duration production tracing; both pair with Trace Compass for graphical analysis.

Q6. What file does perf record its data into?

A binary file named perf.data in the current directory by default, later consumed by report/analysis subcommands.

• perf wiki and tutorials: perfwiki.github.io
• ftrace documentation: docs.kernel.org/trace/ftrace
• man pages: perf-sched(1), perf-top(1), perf-timechart(1)