The Kernel Ring Buffer — Where All Messages Live

Every printk() call — regardless of log level — writes to the kernel ring buffer. This is a fixed-size circular buffer kept in kernel memory. When it fills up, the oldest messages are overwritten. On Linux 6.x the ring buffer is fully lockless, meaning messages can be written even from non-maskable interrupt (NMI) context without any risk of deadlock.

The buffer size is set at build time by CONFIG_LOG_BUF_SHIFT. The default is usually 17, giving a 128 KB buffer. You can increase it at boot time with the kernel parameter log_buf_len=4M if you need more history.

Userspace reads the ring buffer through the character device /dev/kmsg. Both dmesg and journalctl -k ultimately read from this device.

The Four Values in /proc/sys/kernel/printk

Reading this file gives you four integers on one line:

$ cat /proc/sys/kernel/printk
7    4    1    7

Each number has a specific meaning:

dmesg — Reading the Ring Buffer

dmesg is the standard command to read kernel messages. On modern systems it reads directly from /dev/kmsg. Here are the most useful options for kernel developers:

Practical workflow when testing a module load:

# Terminal 1: watch the log live with timestamps
dmesg -wT

# Terminal 2: load your module
sudo insmod ep_uart.ko

# You will see your pr_info/pr_err messages appear in Terminal 1 immediately

journalctl — Kernel Logs via systemd

On any system running systemd, systemd-journald collects kernel messages alongside service and application logs. You query all of it through journalctl. The advantage over dmesg is that journal logs persist across reboots — essential for debugging crashes.

# Show all kernel messages from the current boot
journalctl -k

# Same thing, written out:
journalctl --dmesg

# Kernel messages from the previous boot — invaluable after a crash
journalctl -k -b -1

# List all stored boot sessions
journalctl --list-boots

# Follow kernel messages live (like dmesg -w)
journalctl -k -f

# Show only error level and above (levels 0-3)
journalctl -k -p err

# Show a range: warning through alert (levels 1-4)
journalctl -k -p warning..alert

# Filter by time range
journalctl -k --since "1 hour ago"
journalctl -k --since "2025-06-25 09:00:00" --until "2025-06-25 10:00:00"

# Combine filters: kernel errors in the last hour
journalctl -k -p err --since "1 hour ago"

# Follow messages from a specific service
journalctl -f -u apache2.service

Live Monitoring with journalctl -f

The -f flag on journalctl works like tail -f — it continuously prints new messages as they arrive. This is very useful during driver development to watch what your module produces in real time:

# Watch all kernel messages live
journalctl -k -f

# Watch a specific service (e.g. your custom daemon) live
journalctl -f -u my-driver-daemon.service

# Watch live AND filter by level — only warnings and above
journalctl -k -f -p warning

Rate Limiting — Preventing Log Floods

Imagine a driver that detects a hardware error 10,000 times per second and calls pr_err() each time. The kernel log fills up instantly, the console freezes trying to scroll text, and the system may become unresponsive. Rate limiting solves this by capping how many messages a call site can produce in a time window.

The Modern Way: pr_*_ratelimited()

Each of the pr_* macros has a rate-limited version. The naming pattern is pr_LEVEL_ratelimited():

pr_emerg_ratelimited(fmt, ...)
pr_alert_ratelimited(fmt, ...)
pr_crit_ratelimited(fmt, ...)
pr_err_ratelimited(fmt, ...)
pr_warn_ratelimited(fmt, ...)
pr_notice_ratelimited(fmt, ...)
pr_info_ratelimited(fmt, ...)

The default rate limit per call site is 10 messages per 5 seconds. Once exceeded, messages are silently dropped until the window resets. After the window resets and the next message goes through, the kernel automatically prints how many messages were suppressed:

/* Example usage in an IRQ error path */
static irqreturn_t ep_uart_irq(int irq, void *dev_id)
{
    if (hw_error_detected()) {
        /* Safe in hot path — will not flood the log */
        pr_err_ratelimited("RX FIFO overflow on port %d\n", port_num);
        clear_hw_error();
    }
    return IRQ_HANDLED;
}

/* What you see in dmesg when the error fires 1000 times in 5 seconds: */
/* [  45.123] ep_uart: RX FIFO overflow on port 0   (message 1) */
/* [  45.124] ep_uart: RX FIFO overflow on port 0   (message 2) */
/* ...up to message 10...                                        */
/* [  50.001] ep_uart: 990 callbacks suppressed      (auto summary) */
/* [  50.001] ep_uart: RX FIFO overflow on port 0   (window reset) */

Print Only Once: pr_*_once()

For messages that should appear at most one time for the lifetime of the running kernel, use the _once variants:

pr_err_once("legacy hardware detected, some features disabled\n");
pr_warn_once("firmware version is outdated, consider upgrading\n");

These use a static boolean to ensure the message fires exactly once per boot, no matter how many times the code path is reached.

Quick Reference: Common Developer Workflows

# ===== DURING MODULE DEVELOPMENT =====

# 1. Open a terminal and follow the kernel log live
dmesg -wT

# 2. In another terminal, load your module
sudo insmod my_module.ko

# 3. See your pr_info/pr_err output appear immediately in terminal 1

# 4. Remove module
sudo rmmod my_module

# ===== AFTER A SYSTEM CRASH OR HANG =====

# Show kernel messages from the boot before the last one
journalctl -k -b -1

# Show only errors from that boot
journalctl -k -b -1 -p err

# ===== ADJUSTING CONSOLE OUTPUT =====

# Show debug messages on console (temporary)
sudo echo 8 > /proc/sys/kernel/printk

# Quiet the console (show only errors and above)
sudo echo 4 > /proc/sys/kernel/printk

# Restore default
sudo echo 7 > /proc/sys/kernel/printk

# ===== FILTER MESSAGES FROM YOUR MODULE =====

# dmesg: grep for your module name
dmesg | grep ep_uart

# journalctl: filter by syslog identifier
journalctl -k | grep ep_uart