Beginners often confuse these three. Each behaves differently depending on how your kernel is built.

Key point about pr_devel(): Even if CONFIG_DYNAMIC_DEBUG=y is set in your kernel, pr_devel() does not get compiled in unless you explicitly compile with -DDEBUG. It is intentionally kept out of the dynamic debug system. Use it for very low-level debug lines you never want accidentally enabled on any production system — for example, per-packet tracing in a network driver.

When CONFIG_DYNAMIC_DEBUG is enabled in your kernel (check with grep DYNAMIC_DEBUG /boot/config-$(uname -r)), every pr_debug() and dev_dbg() call is handled in a special way at compile time.

The compiler creates a small metadata record — called a _ddebug descriptor — for each callsite. This descriptor stores: the module name, the file name, the function name, the line number, and the format string. All these descriptors are packed into a special section of the kernel binary called __dyndbg.

This mechanism uses the kernel’s static keys / jump labels feature. The actual binary code at the callsite is modified in memory — no branch that must be evaluated every time. When disabled, the instruction stream flows straight through without any branch taken.

The control file is at /sys/kernel/debug/dynamic_debug/control. You need debugfs mounted (it usually is automatically) and root access.

# First, confirm dynamic debug is available
ls /sys/kernel/debug/dynamic_debug/control

# See all registered debug callsites (there are thousands in a real kernel)
cat /sys/kernel/debug/dynamic_debug/control | head -20

Each line in that file shows one debug callsite in this format:

# filename:lineno [module]function flags format
drivers/usb/core/hub.c:1234 [usbcore]hub_activate =_ "port %d: ..."

The flags field shows what is currently enabled: =_ means nothing enabled (all off). The flags you can set are:

Practical commands for your module

# Enable all pr_debug() in your module (simplest — just add 'p' flag)
echo 'module mymodule +p' > /sys/kernel/debug/dynamic_debug/control

# Enable with function name and line number shown
echo 'module mymodule +pfl' > /sys/kernel/debug/dynamic_debug/control

# Enable only in a specific file
echo 'file mymodule_core.c +p' > /sys/kernel/debug/dynamic_debug/control

# Enable only a specific function
echo 'func mymodule_probe +p' > /sys/kernel/debug/dynamic_debug/control

# Enable only a specific line
echo 'file mymodule_core.c line 87 +p' > /sys/kernel/debug/dynamic_debug/control

# Disable all debug messages in your module
echo 'module mymodule -p' > /sys/kernel/debug/dynamic_debug/control

# Disable all flags completely
echo 'module mymodule =_' > /sys/kernel/debug/dynamic_debug/control

Problem: What if you need debug messages from the module’s init function — which runs at insmod time, before you can write to the control file?

# Solution: pass dyndbg as a module parameter to modprobe
modprobe mymodule dyndbg=+p

# Or, to enable at boot for a built-in module
# Add this to your kernel command line in GRUB:
mymodule.dyndbg=+p

If your kernel does not have CONFIG_DYNAMIC_DEBUG, or if you want pr_devel() (which dynamic debug ignores), you compile with -DDEBUG. There are two modern ways to do this in a kernel module Makefile.

Method 1: ccflags-y (recommended, affects all .c files in this Makefile)

# In your module's Makefile
obj-m := mymodule.o
mymodule-objs := mymodule_core.o mymodule_spi.o

# Enable DEBUG for all source files
ccflags-y += -DDEBUG

Method 2: CFLAGS_filename.o (only one specific file)

# In your module's Makefile — only mymodule_core.c gets -DDEBUG
obj-m := mymodule.o
mymodule-objs := mymodule_core.o mymodule_spi.o

CFLAGS_mymodule_core.o := -DDEBUG

Method 3: Inside the source file itself (quickest for testing)

/*
 * Add this as the VERY FIRST LINE of your .c file,
 * before any #include. This enables pr_debug() and pr_devel()
 * for this file only.
 */
#define DEBUG

#include <linux/module.h>
/* ... rest of your file ... */

When you write a device driver, use the dev_*() family instead of pr_*(). The difference is that every dev_*() message automatically includes the driver name and device identity (bus address, device name) so you can immediately tell which hardware instance produced the message.

#include <linux/device.h>

static int mydriver_probe(struct platform_device *pdev)
{
    /* dev_info adds "mydriver mydevice0: " prefix automatically */
    dev_info(&pdev->dev, "probe called, IRQ = %d\n", pdev->resource->start);

    if (some_error) {
        dev_err(&pdev->dev, "failed to map registers: %d\n", ret);
        return ret;
    }

    dev_warn(&pdev->dev, "operating in fallback mode\n");
    dev_dbg(&pdev->dev, "register dump: ctrl=0x%08x\n", ctrl);

    return 0;
}

The output in dmesg will look like this — notice the driver and device identity is there automatically:

[   12.345678] mydriver mydevice0: probe called, IRQ = 47
[   12.345700] mydriver mydevice0: register dump: ctrl=0x00010001

There are also dev_err_probe() and dev_warn_probe() specifically for the probe function — they handle the common -EPROBE_DEFER case cleanly and are the preferred style in modern drivers.

The dmesg command reads the kernel ring buffer. Here are the most useful options you will use every day as a kernel developer.

# Read all messages (may need sudo if kernel.dmesg_restrict=1)
dmesg

# Human-readable timestamps (shows real date/time, not seconds since boot)
dmesg -T

# Best for daily use: color + pager + readable timestamps
dmesg -H

# Follow — stay open and print new messages as they arrive (like tail -f)
dmesg -w

# Show only errors and warnings
dmesg -l err,warn

# Show only your module's messages
dmesg | grep mymodule

# Clear the ring buffer (useful before testing your module)
sudo dmesg -C

# Show and then clear
sudo dmesg -c

# Show last 20 lines only
dmesg | tail -20

# Change console log level to 7 (show everything including KERN_DEBUG)
sudo dmesg -n 7

Show Everything with ignore_loglevel

Sometimes during debugging you want to see all messages on the console — including KERN_DEBUG — without having to change /proc/sys/kernel/printk. Use the ignore_loglevel toggle:

# Force all messages to the console regardless of level
echo Y | sudo tee /sys/module/printk/parameters/ignore_loglevel

# Restore normal loglevel filtering
echo N | sudo tee /sys/module/printk/parameters/ignore_loglevel

You can also enable this permanently by adding ignore_loglevel to your kernel boot parameters in GRUB.