What You Will Learn

• Why GDB cannot debug a loaded module without extra help, and where the kernel publishes each section’s runtime address
• How to attach GDB to a running module with add-symbol-file and set a breakpoint inside it
• How to enable and drive kdb, the built-in debug shell that runs entirely on the target
• How to read the fields of a kernel oops — PC, LR, and the backtrace — well enough to know which function crashed

Prerequisites For This Free Linux Kernel Development Course Lecture

• A board or QEMU target you can build and load kernel modules on, with a serial console
• A cross-compiled kernel with CONFIG_DEBUG_INFO, CONFIG_KGDB, and CONFIG_KGDB_KDB enabled
• Basic familiarity with writing and loading a simple character driver (covered earlier in this course)
• A host toolchain matching the target (for gdb and objdump)

Why A Loaded Module Confuses GDB

A kernel module is position-independent object code. When insmod loads it, the kernel picks wherever there is free space in module memory and relocates every symbol to that address. Your .ko file on disk has no idea what that address will be — it is decided fresh on every load. If you just run gdb vmlinux and try to break on a function inside your module, GDB will refuse: it has no symbol table entry at any address, because the module was never linked into vmlinux.

The kernel does publish the answer, though — through sysfs. Every loaded module gets an entry under /sys/module/<name>/sections/, one file per ELF section, and each file’s content is the runtime address the kernel relocated that section to.

Because ELF section names begin with a dot, they are hidden files — remember ls -a or they simply will not show up.

$ ls -a /sys/module/ep_dbgdemo/sections/
.  ..  .bss  .data  .text  __versions

A Small Demo Driver To Debug

Rather than reuse a textbook example, build this tiny character driver. It deliberately calls a function pointer that is never initialised, so you get a real, reproducible oops to practise on.

// ep_dbgdemo.c — minimal char driver used to practise module debugging
#include <linux/module.h>
#include <linux/fs.h>
#include <linux/cdev.h>
#include <linux/uaccess.h>
#include <linux/slab.h>

#define EP_BUF_LEN 128

struct ep_dbg_state {
    char buf[EP_BUF_LEN];
    size_t len;
};

static dev_t ep_devno;
static struct cdev ep_cdev;
static struct class *ep_class;

static ssize_t ep_write(struct file *filp, const char __user *ubuf,
                         size_t count, loff_t *off)
{
    /* filp->private_data is never set by ep_open() below — dereferencing
     * it here is the intentional bug this lecture debugs. */
    struct ep_dbg_state *st = filp->private_data;

    if (count > EP_BUF_LEN)
        count = EP_BUF_LEN;

    st->len = count;                       /* <-- NULL pointer dereference */
    if (copy_from_user(st->buf, ubuf, count))
        return -EFAULT;

    return count;
}

static int ep_open(struct inode *inode, struct file *filp)
{
    return 0;   /* private_data intentionally left NULL */
}

static const struct file_operations ep_fops = {
    .owner = THIS_MODULE,
    .open  = ep_open,
    .write = ep_write,
};

static int __init ep_dbgdemo_init(void)
{
    int ret;

    ret = alloc_chrdev_region(&ep_devno, 0, 1, "ep_dbgdemo");
    if (ret)
        return ret;

    cdev_init(&ep_cdev, &ep_fops);
    ret = cdev_add(&ep_cdev, ep_devno, 1);
    if (ret)
        goto err_region;

    ep_class = class_create("ep_dbgdemo");
    if (IS_ERR(ep_class)) {
        ret = PTR_ERR(ep_class);
        goto err_cdev;
    }

    device_create(ep_class, NULL, ep_devno, NULL, "ep_dbgdemo");
    pr_info("ep_dbgdemo: loaded, major=%d\n", MAJOR(ep_devno));
    return 0;

err_cdev:
    cdev_del(&ep_cdev);
err_region:
    unregister_chrdev_region(ep_devno, 1);
    return ret;
}

static void __exit ep_dbgdemo_exit(void)
{
    device_destroy(ep_class, ep_devno);
    class_destroy(ep_class);
    cdev_del(&ep_cdev);
    unregister_chrdev_region(ep_devno, 1);
}

module_init(ep_dbgdemo_init);
module_exit(ep_dbgdemo_exit);
MODULE_LICENSE("GPL");
MODULE_DESCRIPTION("EmbeddedPathashala GDB/kdb debugging demo driver");

Build it against your kernel tree with a one-line Makefile (obj-m += ep_dbgdemo.o), copy the .ko to the target, then load it and provoke the bug:

$ insmod ep_dbgdemo.ko
$ echo hello > /dev/ep_dbgdemo
[  41.208112] Unable to handle kernel NULL pointer dereference at virtual address 00000008
[  41.208601] Internal error: Oops: 96000006 [#1] SMP
[  41.208734] Modules linked in: ep_dbgdemo(O)
[  41.208977] CPU: 0 PID: 214 Comm: sh Tainted: G  O  6.6.0 #1
[  41.209288] pc : ep_write+0x18/0x74 [ep_dbgdemo]
[  41.209421] lr : vfs_write+0xd8/0x38c

Loading Module Symbols In GDB

Read the three addresses sysfs published while the module is still loaded on the target:

target$ cat /sys/module/ep_dbgdemo/sections/.text
0xffffffc008a40000
target$ cat /sys/module/ep_dbgdemo/sections/.data
0xffffffc008a43e18
target$ cat /sys/module/ep_dbgdemo/sections/.bss
0xffffffc008a44120

Then, on the host, hand those exact addresses to GDB alongside the unstripped .ko:

(gdb) add-symbol-file ep_dbgdemo.ko 0xffffffc008a40000 \
      -s .data 0xffffffc008a43e18 -s .bss 0xffffffc008a44120
add symbol table from file "ep_dbgdemo.ko" at
    .text_addr = 0xffffffc008a40000
    .data_addr = 0xffffffc008a43e18
    .bss_addr = 0xffffffc008a44120

(gdb) break ep_write
Breakpoint 1 at 0xffffffc008a40018: file ep_dbgdemo.c, line 22.
(gdb) continue

From this point GDB behaves exactly as it does with vmlinux — you can inspect locals, step, and watch structures inside the module, because the symbol table now lines up with where the code actually sits in memory.

Debugging With kdb When You Have No Host

kdb trades GDB’s features for zero external dependencies — it runs entirely on the target over a serial console, so it works even when you cannot attach a host debugger at all. It sits on top of the same kgdb infrastructure; enable it with:

CONFIG_KGDB=y
CONFIG_KGDB_KDB=y   # Kernel hacking -> KGDB: kernel debugger -> KGDB_KDB

Force the kernel to drop into kdb from the console with the sysrq trigger:

# echo g > /proc/sysrq-trigger
[   58.114203] SysRq : DEBUG
Entering kdb (current=0xffffffc0091a4080, pid 92) due to Keyboard Entry
kdb>

The commands you will reach for most often:

kdb> bp ep_write
Instruction(i) BP #0 at 0xffffffc008a40018 (ep_write)
  is enabled   addr at 0xffffffc008a40018, hardtype=0 installed=0
kdb> go

Because kdb is not a source-level debugger, it cannot show you the C line that triggered a fault — only the raw backtrace. That is enough to identify which function crashed and its call chain, which is often all you need on a headless target.

Reading Your First Kernel Oops

When the kernel hits an invalid memory access, it writes an oops to the kernel log rather than always crashing the whole system outright. Two fields matter most for a first pass:

[  41.209288] pc : ep_write+0x18/0x74 [ep_dbgdemo]
[  41.209421] lr : vfs_write+0xd8/0x38c
[  41.209544] Call trace:
[  41.209544]  ep_write+0x18/0x74 [ep_dbgdemo]
[  41.209611]  vfs_write+0xd8/0x38c
[  41.209678]  ksys_write+0x74/0x110
[  41.209744]  __arm64_sys_write+0x1c/0x28

pc : ep_write+0x18/0x74 [ep_dbgdemo] already tells you most of the story: the fault happened 0x18 bytes into a 0x74-byte function named ep_write, inside the ep_dbgdemo module. The next lecture in this free Linux kernel development course shows how to turn that offset into an exact source line with objdump, and how to preserve this log even when the console never got a chance to print it.

Common Mistakes And Troubleshooting

• Stale addresses: re-reading add-symbol-file values after a module reload is easy to forget — GDB will silently point at the wrong code.
• Stripped modules: build with CONFIG_DEBUG_INFO and do not strip the .ko you copy to the host, or GDB has nothing to load.
• Hidden sysfs files: forgetting ls -a on sections/ makes it look like the addresses do not exist.
• kdb not appearing: confirm both CONFIG_KGDB and CONFIG_KGDB_KDB are set, and that the console is a serial console kdb can take over.

Best Practices

• Keep an unstripped copy of every module .ko you ship to a test board, matched to the exact build.
• Script the sysfs address read plus add-symbol-file generation — typing three hex addresses by hand invites typos.
• Reach for kdb first on boards where you have serial but no reliable network path for gdbserver or full kgdb.

Summary And Key Takeaways

• A loaded module’s addresses live under /sys/module/<name>/sections/, and GDB needs them via add-symbol-file before it can debug the module.
• kdb gives you breakpoints, backtraces, and memory inspection with zero host dependency, driven from a serial console.
• An oops’s pc line already names the function and byte offset where the fault happened — the starting point for real root-causing.

Conclusion

Getting comfortable reading a raw oops and attaching GDB to a live module is one of the highest-leverage skills in this free Linux kernel development course — it is the difference between staring at a crash and knowing exactly which line to open. The next lecture builds directly on this one, turning the pc offset into an exact source line and showing how to recover an oops even when it never reached the console.

Frequently Asked Questions