What is Debugging the Linux Kernel with KGDB-Embedded Linux Training for Beginners

PREV_LEC NEXT_LEC

Debugging the Linux Kernel with KGDB

Break into a live kernel and step through its source with source-level debugging — free linux kernel development course

Everything covered so far in this free linux kernel development course has debugged user-space applications. The kernel itself is a different animal: it has no parent process to attach GDB to, it manages the very memory mappings GDB depends on, and a naive Ctrl-C won’t interrupt it the way it interrupts a normal program. kgdb is the kernel’s own source-level debugging facility, and this lecture walks through configuring it, connecting to a live target over serial, and stepping through real kernel code with the same GDB commands you already know.

Topics Covered

kgdb kgdboc vmlinux symbols sysrq trigger early boot debugging

What You Will Learn

How kgdb differs from user-space debugging Kernel config options required for kgdb Connecting kgdb over a serial console with kgdboc Forcing a break into a running kernel Debugging code that runs before the console is ready

Prerequisites

You need a board you can rebuild and reflash the kernel on, a serial console connection to it, and a host-side GDB matching your target architecture. Familiarity with kernel config (menuconfig) and the boot command line from earlier lectures in this course will help.

Why Kernel Debugging Is Different

GDB debugging an application relies on the kernel: it uses ptrace to pause the process, read its memory, and resume it. There’s no layer underneath the kernel to provide that same service to the kernel itself. kgdb solves this by embedding a small debug stub directly inside the kernel that can freeze all CPUs, hand control to a remote GDB over a serial link, and single-step or set breakpoints in kernel code the same way GDB steps through a user-space program.

Because the kernel is a real-time system managing memory mappings and scheduling, single-stepping through code that touches those subsystems can produce results that look stranger than a normal application debug session — that’s expected, not a sign something is broken.

kgdb Connection Path

Host Target board —- ———— gdb vmlinux kernel running kgdboc | | |—– serial cable ——-| | (shared with console) | | | target remote /dev/ttyUSB0 | |————————->| | connection refused | | until kernel is | | trapped via sysrq | | | # echo g > /proc/sysrq-trigger | |<————————-| | kernel halts, kgdb | | stub takes over | | | interactive GDB session <——-> frozen kernel

Required Kernel Configuration

Before any of this works, the kernel has to be built with debug support enabled. In menuconfig, under the Kernel hacking submenu, turn on:

Config optionWhat it enables
CONFIG_DEBUG_INFOCompiles the kernel with debug symbols GDB can read
CONFIG_FRAME_POINTERReliable stack unwinding for backtraces (architecture dependent)
CONFIG_KGDBThe kgdb debug stub itself
CONFIG_KGDB_SERIAL_CONSOLEThe kgdboc serial transport

You’ll also need the uncompressed kernel image in ELF form — vmlinux, produced in your kernel build directory — since that’s what carries the symbol table GDB loads. The compressed zImage or uImage you actually boot from doesn’t carry usable debug symbols.

Watch out: the kernel is built assuming at least -O1 optimization; unlike user-space code you can’t debug an unoptimized kernel build. Some variable values may appear optimized-away during a step — that’s expected, not a symbol-loading bug.

Connecting kgdb Over Serial (kgdboc)

The most common transport, kgdboc (“kgdb over console”), shares the same serial line as your normal console. You can set it either from the kernel command line at boot, or at runtime through sysfs.

Option 1 — kernel command line

kgdboc=ttyS0,115200

Option 2 — set it after boot

# echo ttyS0 > /sys/module/kgdboc/parameters/kgdboc

On the host, start GDB against the matching vmlinux and connect over the same serial device the target uses:

$ arm-none-linux-gnueabihf-gdb ~/linux-build/vmlinux
(gdb) set remotebaud 115200
(gdb) target remote /dev/ttyUSB0
Remote debugging using /dev/ttyUSB0
Bogus trace status reply from target: qTStatus

That “bogus trace status” reply is expected at this point — kgdb isn’t actively listening yet. Close any terminal program that’s holding the console open (it will conflict with GDB on the same line), then force the kernel to trap into the debugger from a second connection to the board, such as SSH over the network interface if one is available:

# echo g > /proc/sysrq-trigger

The target freezes immediately. Reconnect from GDB and you’re now in an interactive kernel debug session:

(gdb) target remote /dev/ttyUSB0
Remote debugging using /dev/ttyUSB0
0xc009a59c in arch_kgdb_breakpoint ()
(gdb) break sys_sync
Breakpoint 1 at 0xc0128a88: file fs/sync.c, line 103.
(gdb) c
Continuing.

Typing sync on the target now hits that breakpoint, and standard GDB commands — step, print, bt — work against live kernel state exactly as they did against a user-space process. To release the kernel and disable kgdboc when you’re done:

# echo "" > /sys/module/kgdboc/parameters/kgdboc

Debugging Code That Runs Before Boot Finishes

The sysrq-trigger approach only works once the system is far enough along to run a shell. For bugs in early init code, append kgdbwait after kgdboc on the boot command line, and the kernel pauses itself and waits for GDB before continuing any further:

kgdboc=ttyS0,115200 kgdbwait

On boot the console prints something like this, and holds there until a debugger connects:

[    1.10] console [ttyS0] enabled
[    1.11] kgdb: Registered I/O driver kgdboc.
[    1.11] kgdb: Waiting for connection from remote gdb...

At that point, connect from the host exactly as before — target remote /dev/ttyUSB0 — and you’re debugging code that runs before user space even starts, which is otherwise nearly impossible to instrument any other way.

Common Mistakes

Trying Ctrl-C in GDB to interrupt the kernel — it doesn’t work, use sysrq Loading a vmlinux that doesn’t match the running kernel build exactly Leaving a terminal emulator attached to the console while GDB uses the same line Debugging without CONFIG_FRAME_POINTER and getting an unreliable backtrace

Best Practices

Keep a debug-enabled kernel build separate from your production image — the config options above add overhead you don’t want shipping. If your board has a spare UART, dedicate it to kgdboc so you’re not fighting for the same line as your normal console during a session. And always confirm the vmlinux you load matches the exact running kernel, since even a one-commit mismatch can produce a nonsensical backtrace.

Summary

kgdb embeds a debug stub inside the kernel itself, letting a remote GDB connect over serial, freeze all CPUs, and step through kernel source the same way it steps through an application — with the key difference that you have to explicitly trap the kernel via sysrq (or kgdbwait for early boot) rather than relying on Ctrl-C. It’s the deepest level of source debugging this free linux kernel development course covers, and the natural next step once user-space GDB workflows feel familiar.

FAQ

Why doesn’t Ctrl-C interrupt the kernel like it does an application?

The kernel has no parent debugger relationship the way a ptraced process does; you have to explicitly force a trap, typically by writing ‘g’ to /proc/sysrq-trigger.

What’s the difference between vmlinux and the zImage I actually boot?

vmlinux is the uncompressed ELF build carrying full debug symbols; zImage/uImage are the compressed, bootable images with no usable symbol table for GDB.

Can I use kgdb without a dedicated serial port?

kgdboc commonly shares your existing console UART, though this means you lose console access during the debug session.

How do I debug code that runs before init starts?

Add kgdbwait after kgdboc on the kernel command line, which pauses boot and waits for GDB to connect before continuing.

Does kgdb work with TUI?

Yes, launching gdb -tui against vmlinux works the same way as any other GDB session, giving you a live source window during the kgdb connection.

Is kgdb safe to leave enabled in production?

Generally no, the debug config options add overhead and an exposed debug entry point, so it’s best kept to development and diagnostic builds only.

Keep Building Your Kernel Debugging Skills

This lecture is part of EmbeddedPathashala’s free linux kernel development course and free linux device drivers course. Explore more chapters on kernel internals and driver development.

Browse the Course Join Free
PREV_LEC NEXT_LEC

2 Comments

Leave a Reply

Your email address will not be published. Required fields are marked *