What You Will Learn

• How to use objdump -S to turn a bare oops offset into the exact line of C source that faulted
• Why an oops can vanish before it ever reaches the console, and how the kernel log buffer survives a crash
• How to locate __log_buf in System.map and translate it from a kernel logical address to a physical one
• How to read that recovered log straight out of your bootloader with no OS running at all

Prerequisites For This Free Linux Device Drivers Course Lecture

• Lecture 11 of this series — an oops in hand and a module you can rebuild with debug info
• A matching cross toolchain with objdump on the host
• Access to your board’s bootloader console (this lecture uses U-Boot as the example)
• The kernel’s System.map for the exact build that crashed

From Offset To Source Line With objdump

Picking up the oops from the previous lecture — pc : ep_write+0x18/0x74 [ep_dbgdemo] — the module was 0x74 bytes long and the fault sat 0x18 bytes into ep_write. Disassemble the unstripped module with source interleaved and look at that exact offset:

$ aarch64-linux-gnu-objdump -S ep_dbgdemo.ko | less

ssize_t ep_write(struct file *filp, const char __user *ubuf,
                  size_t count, loff_t *off)
{
   0:   a9be7bfd    stp x29, x30, [sp, #-32]!
   4:   910003fd    mov x29, sp
    struct ep_dbg_state *st = filp->private_data;
   8:   f9401000    ldr x0, [x0, #32]
    if (count > EP_BUF_LEN)
        count = EP_BUF_LEN;
  10:   f100...     cmp x2, #128
  14:   9a82...     csel x2, x2, x3, ls
    st->len = count;
  18:   f9000002    str x2, [x0]        <-- fault: x0 (st) is NULL here

Offset 0x18 lands squarely on st->len = count; — confirming what the register dump already hinted: x0, which holds st, is 0 at that point because ep_open() never set filp->private_data. That single line is the fix: allocate a struct ep_dbg_state in ep_open() and store it before ep_write() ever runs.

When The Oops Never Reaches The Console

Everything so far assumes the oops actually got printed somewhere you could read it. That assumption breaks in two common situations: a crash during early boot before the console driver is up, or a crash right after a suspend/resume transition. In both cases the kernel log buffer, __log_buf, still holds the message in RAM — it just never made it out over serial.

__log_buf is a plain ring buffer of text. As long as a reset does not clear or corrupt RAM — which on most boards it does not — you can reboot straight into the bootloader and read that buffer directly with a memory-dump command, no kernel required.

Locating And Translating The Log Buffer

First, find the symbol’s kernel logical address in System.map for the exact kernel build that crashed:

$ grep __log_buf System.map
ffffffc008f9e428 b __log_buf

A bootloader like U-Boot has no concept of the kernel’s virtual memory mapping, so that logical address has to be translated into a physical one before it means anything to md. Subtract the kernel’s PAGE_OFFSET and add the physical base address of RAM on your board:

physical_addr = logical_addr - PAGE_OFFSET + RAM_BASE

# example, 32-bit ARM board with RAM starting at 0x80000000:
# 0xc0f72428 - 0xc0000000 + 0x80000000 = 0x80f72428

Reading The Log From The Bootloader

With the physical address in hand, dump memory from U-Boot before the kernel is even started:

U-Boot# md 0x80f72428
80f72428: 00000000 00000000 00210034 c6000000    ........4.!.....
80f72438: 746f6f42 20676e69 756e694c 6e6f2078    Booting Linux on
80f72448: 79687020 61636973 5043206c 78302055    physical CPU 0x
80f72478: 2e34206e 30312e31 68632820 40736972    n 4.1.10 (chris@

Reading the ASCII column on the right, the boot messages are recognisable even in raw hex. Real production kernels are usually built with symmetric multiprocessing and additional metadata; from Linux 3.5 onward each log line carries a small binary header encoding a timestamp and log level ahead of the text, so expect a few non-printable bytes between messages rather than one continuous string.

Common Mistakes And Troubleshooting

• Wrong System.map: the symbol address is only valid for the exact kernel build that crashed — a mismatched System.map gives you garbage.
• Forgetting the physical translation: feeding the bootloader a kernel logical address directly reads the wrong memory entirely.
• Assuming RAM survived: always sanity-check the dump against known boot-message text before trusting it.
• objdump on a stripped module: without debug info compiled in, objdump -S falls back to pure disassembly with no source lines at all.

Best Practices

• Keep a System.map archived per kernel build you deploy, named to match the image — you cannot regenerate it after the fact.
• On boards where early-boot or suspend crashes are a real risk, look into logging to an MTD partition or persistent RAM (pstore) rather than relying on manual bootloader recovery every time.
• Automate the offset-to-source lookup for your team with a small wrapper script around objdump -S, rather than scrolling through output by hand each time.

Security Considerations

A recovered kernel log can contain pointers, stack contents, and sometimes fragments of in-flight data — treat crash dumps pulled this way with the same care as any other diagnostic data leaving the device, especially on shared or production hardware.

Summary And Key Takeaways

• objdump -S against an unstripped module maps an oops offset straight to the C source line that faulted.
• The kernel log buffer, __log_buf, survives most warm resets in RAM even if the console never printed it.
• Translating a logical address to physical — subtract PAGE_OFFSET, add the RAM base — is what lets a bootloader read that buffer with no kernel running.

Conclusion

Between GDB with add-symbol-file, kdb on the console, and this offline recovery technique, you now have a full toolbox for interactive debugging and after-the-fact crash analysis — which closes out the GDB chapter of this free Linux device drivers course. The two approaches complement each other: reach for the interactive tools when you can reproduce a bug live, and fall back to log recovery when a crash only happens once, in the field, before anyone was watching. From here, the course moves on from debugging into profiling and tracing — measuring where a correctly-running kernel is spending its time, not just chasing crashes.

Frequently Asked Questions