Linux Kernel Debug Configuration: Free Linux Kernel Development Course | EmbeddedPathashala

Linux Kernel Debug Configuration: Building a Debug Kernel for Safe Module Development

A complete guide to setting up a Linux 6.x debug kernel so you can catch bugs early — covering memory sanitizers, lock validators, ftrace, and the right development workflow.

📚 Chapter 4, Part 1 ⏱ ~18 min read 🎓 Free Linux Kernel Programming Course

← PREV_LEC NEXT_LEC →

🎓 What You Will Learn

Why a debug kernel is essential for Linux kernel module development
How the Linux 6.x kernel debug configuration system works
Which CONFIG_* options to enable and exactly what each one does
How memory sanitizers like KASAN catch kernel bugs at runtime
How lockdep detects deadlocks and lock ordering violations
How to use ftrace to trace kernel function calls
The correct two-kernel development workflow used by professional kernel engineers

Prerequisites

You should have already built the Linux 6.x kernel from source (covered in earlier chapters of this free Linux kernel development course). You should know how to run make menuconfig and understand what a loadable kernel module (LKM) is. Basic C programming knowledge is assumed.

Why You Need a Debug Kernel for Linux Module Development

When you write a Linux kernel module, a bug does not just crash your program — it can panic the entire system, corrupt memory silently, or cause a deadlock that freezes everything. This is completely different from writing user-space applications where a segfault only kills your process.

In this free Linux kernel development course, we follow the same professional practice that Linux kernel maintainers use: always test your kernel module code on a debug kernel first. A debug kernel is a specially configured Linux kernel that has extra instrumentation turned on. These extra checks cost some performance, but they will catch bugs during development that would never show up on a production kernel — bugs that might only appear under very specific race conditions or memory access patterns.

The Linux 6.x kernel has a rich set of debug configuration options that have improved significantly compared to older versions like 5.4. This tutorial walks through what each group of options does, how to enable them in make menuconfig, and how to structure your workflow so you never ship buggy kernel code.

Key Principle

Every kernel module you write during this free Linux kernel programming course should be tested on a debug kernel before you consider it ready. Think of a debug kernel as your safety net — expensive to run, but invaluable for catching problems early.

How Linux Kernel Debug Configuration Works

The Linux kernel’s entire build system is driven by a configuration mechanism called Kconfig. Each kernel feature or option is controlled by a CONFIG_* variable that can be set to y (built-in), m (module), or n (disabled). Debug options are almost always either y or n.

You interact with these options through the make menuconfig terminal UI. The debug-related options are grouped under the Kernel Hacking sub-menu. Here is an overview of how the configuration flows from your choices to the compiled kernel:

Linux Kernel Debug Configuration Flow

Developer runs
make menuconfig → Kernel Hacking
Sub-menu → CONFIG_* options
set in .config → Debug Kernel
Compiled with
Extra Checks

When you enable a debug option, the kernel build system compiles additional code paths into the kernel image. These code paths perform extra checking at runtime — for example, checking every memory allocation boundary or validating every lock acquisition order. This is why a debug kernel is larger and slower than a production kernel.

Navigating to Debug Options in make menuconfig (Linux 6.x)

Step	Menu Path	What You Find Here
1	`make menuconfig` top level	Root kernel configuration menu
2	Kernel hacking →	All kernel debug and tracing options
3	Kernel hacking → Memory Debugging	KASAN, SLUB_DEBUG, heap corruption detection
4	Kernel hacking → Lock Debugging	lockdep, lock statistics, atomic sleep checks
5	Kernel hacking → Tracers	ftrace, function graph tracer, IRQ off tracer

Essential Debug Config Options for Linux 6.x Kernel Module Development

Let us go through each group of debug options. These are the options every serious Linux kernel developer enables during development. In Linux 6.x, some of these have improved significantly — for example, KASAN now supports both x86_64 and ARM64 architectures.

⚙ Group 1 — Basic Kernel Debug Options

CONFIG_DEBUG_INFO

Compiles the kernel with full debug symbol information (DWARF format). This is required if you want to use gdb, crash, or any debugger tool against the kernel. Without this, stack traces in kernel oops messages are much less useful. Always enable this during development.

CONFIG_DEBUG_FS

Enables the debugfs virtual filesystem, typically mounted at /sys/kernel/debug/. Many kernel subsystems and drivers expose diagnostic information through debugfs. It is also used by ftrace internally. You will often need this enabled to access real-time kernel debug data from user space.

CONFIG_MAGIC_SYSRQ

Enables the Magic SysRq key feature. If your kernel hangs or panics, you can press specific key combinations (Alt+SysRq+key) to trigger recovery actions like syncing filesystems, killing processes, or rebooting — even when the system is mostly unresponsive. Extremely useful during kernel debugging sessions.

CONFIG_DEBUG_KERNEL

A master switch that enables several other debug options. Enabling this makes many other debug CONFIG_ options visible in menuconfig that are otherwise hidden. In Linux 6.x, this is usually a prerequisite for enabling the more specific debug options below.

CONFIG_EARLY_PRINTK

Enables very early kernel console output during boot, before the normal console subsystem is initialized. If your kernel is crashing during early boot (before normal logging starts), this lets you see printk output from those early stages.

Memory Debugging Options — The Most Important Group

Memory bugs are the most dangerous and hardest-to-find class of kernel bugs. A use-after-free bug or a heap buffer overflow in a kernel module can corrupt memory that is used by a completely unrelated part of the kernel, causing a crash much later with no obvious connection to your code. The memory debugging options below catch these bugs immediately when they happen.

🔌 Group 2 — Memory Debugging Options

CONFIG_SLUB_DEBUG

Enables debug checks inside the SLUB memory allocator, which is the default slab allocator in modern Linux kernels. It adds red zones around allocations to catch buffer overflows, tracks allocation/free patterns to catch use-after-free bugs, and validates that objects are not freed twice. The SLUB allocator manages the kernel’s small-object memory pool — anything you allocate with kmalloc() comes from here.

CONFIG_DEBUG_MEMORY_INIT

Causes the kernel to fill freshly allocated memory pages with a specific poison pattern before handing them to callers. If your code reads from memory before properly initializing it, you will see the poison pattern values in your data, making the bug immediately obvious instead of silently using stale or garbage data.

CONFIG_KASAN

The Kernel Address SANitizer — the most powerful memory safety tool available in the Linux kernel. KASAN instruments every memory access at compile time and then validates each access at runtime using shadow memory. It catches out-of-bounds reads/writes, use-after-free, and stack overflows the moment they happen. In Linux 6.x, KASAN supports both x86_64 and ARM64 and has improved performance compared to earlier versions. The shadow memory overhead is roughly 1/8th of your total RAM. If you have 8 GB of RAM on your development machine, KASAN will use about 1 GB for its shadow map.

How KASAN Shadow Memory Works

Every 8 bytes of kernel memory mapped to 1 byte of shadow memory
Kernel Memory Byte 0-7	Kernel Memory Byte 8-15	Kernel Memory Byte 16-23	Redzones (invalid)	Free’d region (poisoned)
↓	↓	↓	↓	↓
Shadow: 0x00 (valid)	Shadow: 0x00 (valid)	Shadow: 0x03 (partial)	Shadow: 0xF1 (redzone)	Shadow: 0xFD (freed)
On any access to poisoned shadow regions, KASAN immediately reports a bug with full stack trace

Lock Debugging Options — Catch Deadlocks Before They Happen

Locking bugs are another major category of kernel bugs. A deadlock can freeze your entire system with no error message and no obvious cause. The lock debugging options below make the kernel actively verify that lock acquisitions always follow a consistent order, catching potential deadlocks the first time a suspicious pattern is detected — even if the actual deadlock has not occurred yet.

🔒 Group 3 — Lock Debugging Options

CONFIG_PROVE_LOCKING

Enables lockdep — the Linux kernel’s lock dependency validator. This is one of the most powerful debugging tools in the entire kernel. Lockdep tracks every lock acquisition across the entire kernel and builds a graph of lock ordering relationships. If it detects a potential circular dependency (which could cause a deadlock), it reports it immediately, even if no actual deadlock has occurred yet. Enabling this also automatically enables CONFIG_LOCK_STAT and several other related options.

CONFIG_LOCK_STAT

Collects runtime statistics about lock contention — how long each lock is held, how often it causes contention, and which code paths are the most frequent lock holders. You can read this data from /proc/lock_stat. This is useful for performance optimization once your module is functionally correct.

CONFIG_DEBUG_ATOMIC_SLEEP

Detects the serious bug of calling a sleeping function while holding a spinlock or while inside an atomic context. In the Linux kernel, spinlocks disable preemption. If you hold a spinlock and then call something that might sleep (like kmalloc() with GFP_KERNEL, or copy_from_user()), the kernel can deadlock or panic. This option makes the kernel emit a warning the moment it detects such a pattern.

How lockdep Detects Potential Deadlocks

Thread A	Thread B	lockdep Detection
Acquires Lock 1 ✓	Acquires Lock 2 ✓	Records: A holds L1, B holds L2
Tries to acquire Lock 2 → BLOCKS	Tries to acquire Lock 1 → BLOCKS	⚠ lockdep reports circular dependency BEFORE deadlock!

Tracing Options — See Exactly What Your Kernel Is Doing

📊 Group 4 — Kernel Tracing Options

CONFIG_FTRACE

Enables ftrace — the Linux kernel’s built-in function tracer. With ftrace, you can trace every function call inside the kernel in real time, measure how long each function takes, and see exactly which code path led to a particular function being called. In Linux 6.x, ftrace is often enabled by default, but its individual tracer plugins may not be. You should specifically enable at least the function tracer and the function graph tracer in the sub-menu.

CONFIG_STACKTRACE

Enables the kernel’s ability to capture and print full stack traces. This is required for lockdep to show you where a suspicious lock pattern was detected, and it makes kernel oops/panic messages much more informative — instead of just a program counter, you see the full call chain leading to the bug.

CONFIG_DEBUG_BUGVERBOSE

Makes BUG() and WARN() macros print the exact source file name and line number where they were triggered, in addition to the stack trace. Without this, you only get the function name. With it, you get the precise line of code that detected the bug.

Additional Safety Options

🛡 Group 5 — Additional Safety Checks

CONFIG_UBSAN

The Undefined Behavior SANitizer for the kernel. Catches C undefined behavior such as signed integer overflow, shift-out-of-bounds, misaligned pointer access, and array index out-of-bounds at runtime. These are bugs that the C standard says are “undefined” but in practice often produce subtle, hard-to-reproduce errors. UBSAN makes them immediately visible.

CONFIG_BUG_ON_DATA_CORRUPTION

Enables additional validation checks across various kernel data structures (linked lists, radix trees, etc.). If the kernel detects that one of these structures has been corrupted — which usually indicates a memory safety bug — it immediately panics with a descriptive message rather than continuing to run with corrupted state.

CONFIG_SCHED_STACK_END_CHECK

Checks for kernel stack overflow on every context switch. Each kernel thread has a fixed-size stack. If your code uses deep recursion or very large local variables in interrupt context, the stack can overflow and silently corrupt adjacent memory. This option detects that pattern by placing a canary value at the bottom of each stack and checking it on every context switch.

CONFIG_UNWINDER_FRAME_POINTER

Selects the frame-pointer-based stack unwinder. This produces more reliable and complete stack traces compared to the ORC unwinder in some scenarios, especially during panics. It requires the kernel and all modules to be compiled with -fno-omit-frame-pointer, which adds a small overhead but greatly improves debuggability.

CONFIG_KGDB

Optional — enables the kernel’s built-in GDB stub, allowing you to attach a remote GDB debugger to the kernel over a serial connection or network. This is the most interactive debugging approach, letting you set breakpoints and inspect kernel state live. It requires two machines or a VM setup and is optional for most development scenarios.

Quick Reference: All Debug Options at a Glance

CONFIG Option	Category	What It Catches	Performance Impact
`CONFIG_DEBUG_INFO`	General	Enables readable stack traces and GDB support	Larger kernel image
`CONFIG_SLUB_DEBUG`	Memory	Heap overflows, double-free, use-after-free	Moderate (~10-15%)
`CONFIG_KASAN`	Memory	All memory safety violations immediately	High (~2x), uses ~1/8 RAM
`CONFIG_DEBUG_MEMORY_INIT`	Memory	Use of uninitialized memory	Low
`CONFIG_PROVE_LOCKING`	Locking	Potential deadlocks, lock ordering violations	Moderate
`CONFIG_DEBUG_ATOMIC_SLEEP`	Locking	Sleeping in atomic/spinlock context	Very Low
`CONFIG_FTRACE`	Tracing	Function call tracing and profiling	Low when inactive
`CONFIG_UBSAN`	Safety	Integer overflow, invalid shifts, misaligned access	Low-Moderate
`CONFIG_SCHED_STACK_END_CHECK`	Safety	Kernel stack overflow	Very Low
`CONFIG_BUG_ON_DATA_CORRUPTION`	Safety	Corrupted kernel data structures	Very Low

Enabling Debug Options: Step-by-Step Commands for Linux 6.x

Open your Linux 6.x kernel source directory and run:

# Navigate to your kernel source directory
cd ~/linux-6.x

# Open the configuration menu
make menuconfig

Inside menuconfig, navigate to Kernel hacking and enable the options listed above. Alternatively, you can set them directly in your .config file. Here is a shell snippet that uses scripts/config to enable the most important ones automatically:

#!/bin/bash
# Enable essential debug options for Linux 6.x kernel development
# Run this inside your kernel source directory

./scripts/config --enable CONFIG_DEBUG_INFO
./scripts/config --enable CONFIG_DEBUG_FS
./scripts/config --enable CONFIG_MAGIC_SYSRQ
./scripts/config --enable CONFIG_DEBUG_KERNEL
./scripts/config --enable CONFIG_SLUB_DEBUG
./scripts/config --enable CONFIG_DEBUG_MEMORY_INIT
./scripts/config --enable CONFIG_KASAN
./scripts/config --enable CONFIG_PROVE_LOCKING
./scripts/config --enable CONFIG_LOCK_STAT
./scripts/config --enable CONFIG_DEBUG_ATOMIC_SLEEP
./scripts/config --enable CONFIG_STACKTRACE
./scripts/config --enable CONFIG_DEBUG_BUGVERBOSE
./scripts/config --enable CONFIG_FTRACE
./scripts/config --enable CONFIG_BUG_ON_DATA_CORRUPTION
./scripts/config --enable CONFIG_SCHED_STACK_END_CHECK
./scripts/config --enable CONFIG_UBSAN
./scripts/config --enable CONFIG_EARLY_PRINTK
./scripts/config --enable CONFIG_UNWINDER_FRAME_POINTER

# Regenerate .config to resolve any new dependencies
make olddefconfig

echo "Debug config applied. Now run: make -j$(nproc)"

Important Note for Linux 6.x

In Linux 6.x, CONFIG_DEBUG_INFO has been split into several sub-options (CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT, CONFIG_DEBUG_INFO_DWARF5, etc.). If scripts/config --enable CONFIG_DEBUG_INFO does not work on your exact kernel version, run make menuconfig and manually select the debug info format under Kernel hacking → Compile-time checks and compiler options → Debug information.

The Professional Two-Kernel Development Workflow

Having a debug kernel is not just about enabling options — it is about using the right kernel at the right time. Professional Linux kernel engineers and embedded Linux developers always maintain at least two kernel builds:

Two-Kernel Development Workflow

🛠

Debug Kernel

✓ KASAN enabled
✓ lockdep active
✓ SLUB_DEBUG on
✓ ftrace ready
✓ Full debug symbols
✗ Slower performance
✗ Higher RAM usage

Use during development & testing

⇄

🚀

Production Kernel

✗ KASAN disabled
✗ lockdep off
✗ SLUB_DEBUG off
✗ Minimal tracing
✗ Stripped symbols
✓ Maximum performance
✓ Minimal RAM overhead

Use for final validation & deployment

The workflow is:

Write your kernel module code.
Build and load it on your debug kernel. Run your tests. Let KASAN, lockdep, and UBSAN catch any bugs.
Fix any bugs reported. Repeat until the module runs cleanly on the debug kernel.
Build and load it on your production kernel. Run the same tests to confirm behavior under real-world conditions.
Only declare the module ready for deployment after passing both stages.

Pro Tip

If you are doing embedded Linux development targeting ARM (such as a Raspberry Pi or BeagleBone), you can run the debug kernel on an x86_64 virtual machine during early development, since most kernel bugs are architecture-independent. Switch to the actual ARM target for the final testing stage.

What Changed from Linux 5.x to Linux 6.x for Debug Configuration

Feature	Linux 5.4 (old reference)	Linux 6.x (current)
KASAN architecture support	x86_64 only officially	x86_64, ARM64, RISC-V, and more
KASAN mode	Classic shadow memory only	Classic + Hardware Tag-Based KASAN (ARM64)
Debug info format	Single CONFIG_DEBUG_INFO option	Multiple sub-options (DWARF4, DWARF5, split)
Stack unwinder	ORC default, frame pointer optional	ORC improved; frame pointer still available
lockdep scalability	Performance issues with many locks	Improved hash tables, better scalability
UBSAN checks	Basic set	Extended: local-bounds, pointer-overflow added

Common Mistakes When Setting Up a Debug Kernel

Mistake 1: Testing only on the production kernel

Many beginners skip the debug kernel step because it is slower. The result is that bugs slip through that would have been caught immediately. Always test on the debug kernel first.

Mistake 2: Ignoring WARN_ON() messages

When a debug kernel prints a WARN_ON() message, it means the kernel detected a suspicious condition. Do not ignore these — treat them as bugs to fix, even if the system keeps running.

Mistake 3: Not enabling KASAN due to RAM constraints

KASAN needs roughly 1/8th of your RAM for shadow memory. If your development VM only has 2 GB of RAM, KASAN may cause the system to run out of memory. Use at least 4 GB of RAM for a KASAN-enabled debug kernel — 8 GB is more comfortable.

Mistake 4: Using GFP_KERNEL inside spinlock-protected sections

This is the classic kernel programming mistake that CONFIG_DEBUG_ATOMIC_SLEEP catches. kmalloc(GFP_KERNEL) may sleep; if you call it while holding a spinlock, the kernel can deadlock. Use GFP_ATOMIC inside spinlock-protected sections.

Best Practices for Debug Kernel Configuration in Professional Kernel Development

Keep a dedicated .config file for your debug kernel and check it into version control so your team uses consistent debug settings.
Use scripts/config to script your configuration changes rather than manually editing .config — it is safer and reproducible.
After enabling new debug options, run make olddefconfig to resolve any new dependency options that need to be set.
If you are working on embedded hardware with limited RAM, consider enabling KASAN only on your x86_64 development VM and doing ARM-specific testing without it.
Enable CONFIG_FTRACE even if you are not actively tracing — it is lightweight when not in use, and having it available lets you quickly trace any function when something unexpected happens.
Run your module under stress conditions (concurrent access, memory pressure, rapid load/unload cycles) on the debug kernel to expose race conditions that do not show up under normal usage.

🌟 Key Takeaways

A debug kernel is a Linux kernel built with extra runtime checking enabled — it is slower but catches bugs immediately instead of silently corrupting state.
The essential debug groups are: basic info (DEBUG_INFO, DEBUG_FS), memory safety (KASAN, SLUB_DEBUG), lock validation (lockdep, DEBUG_ATOMIC_SLEEP), and tracing (ftrace).
KASAN is the most powerful tool — it catches memory safety bugs the moment they happen using shadow memory instrumentation.
lockdep validates lock ordering across the entire kernel and catches potential deadlocks before they actually freeze your system.
The correct professional workflow is: develop and test on a debug kernel first, then validate on a production kernel before deployment.
Linux 6.x has improved KASAN support (now includes ARM64), better UBSAN checks, and improved lockdep scalability compared to Linux 5.x.

Frequently Asked Questions

Q1. Does enabling all these debug options mean my kernel module code needs to change?

No. The debug options instrument the kernel itself, not your module code. Your module code remains the same. However, the debug kernel will detect bugs in your module’s behavior that would be silent on a production kernel — so you may end up fixing bugs you did not know existed.

Q2. How much slower is a KASAN-enabled debug kernel?

Typically about 1.5x to 2x slower for memory-intensive workloads. For many development tasks this is perfectly acceptable. If you need to benchmark performance, always use the production kernel without KASAN for those measurements.

Q3. Can I run a debug kernel directly on physical hardware like a Raspberry Pi?

Yes. You can cross-compile a debug kernel for ARM (including Raspberry Pi) with KASAN and other debug options enabled — provided the target has enough RAM. For a Raspberry Pi 4 with 4 GB, KASAN is feasible. For a Raspberry Pi Zero or similar low-RAM devices, KASAN may cause out-of-memory issues.

Q4. What is the difference between KASAN and Valgrind?

Valgrind is a user-space tool — it runs user-space programs in a sandboxed environment to detect memory bugs. KASAN is a kernel-space tool that is compiled into the kernel itself. Valgrind cannot detect kernel memory bugs. KASAN cannot detect user-space bugs. They are complementary tools operating in different privilege levels.

Q5. Does lockdep catch all possible deadlocks?

lockdep catches all potential deadlocks that arise from circular lock dependency patterns. It will report a suspicious pattern the very first time it is encountered, even if the actual deadlock never happened in your test. It is conservative — it may report false positives in very unusual locking patterns, but it will never miss a real circular dependency.

Q6. I enabled CONFIG_FTRACE but I do not know how to use it. Where do I start?

ftrace is accessed through debugfs, typically at /sys/kernel/debug/tracing/. To start a basic function trace, run: echo function > /sys/kernel/debug/tracing/current_tracer and then read /sys/kernel/debug/tracing/trace. The trace-cmd user-space tool makes this much easier — it is the recommended interface for everyday ftrace use.

Q7. What is CONFIG_DEBUG_SHIRQ and should I enable it?

CONFIG_DEBUG_SHIRQ tests that your interrupt handler correctly handles being called when the interrupt line is being shared with another device. When enabled, the kernel will deliberately call your IRQ handler an extra time during free_irq() to verify the handler correctly handles spurious invocations. Useful if you are writing interrupt-driven device drivers.

Q8. Is there a way to enable these debug options without rebuilding the kernel from scratch?

Not fully. Most debug options like KASAN and lockdep require code to be compiled into the kernel image itself. You cannot enable them dynamically at runtime. However, some lighter-weight options like slub_debug can be set as a kernel boot parameter without recompiling. For serious kernel module development, rebuilding with proper debug options is the right approach.

References and Further Reading

← Previous: Writing Your First LKM Next: Cross-Compiling Kernel Modules →

Continue Your Free Linux Kernel Development Journey

EmbeddedPathashala offers a completely free Linux kernel programming course covering everything from writing your first kernel module to advanced device drivers.

View Full Course Index Subscribe on YouTube

embeddedpathashala.com

Linux Kernel Debugging: Building a Debug Kernel for Module Development