You should be comfortable building a C program with debug symbols (-g -O0) and have gdb installed on your development host. Root access on the target device is needed to change the core dump pattern, since it lives under /proc/sys/kernel.

When a process receives a fatal signal such as SIGSEGV, SIGABRT, or SIGBUS, and that signal isn’t caught, the kernel’s default action is to write the process’s memory pages, registers, and open file descriptor table to a file before terminating it. Whether that file gets written at all depends on two things: the process’s core-file resource limit (ulimit -c), and where the kernel is told to put it.

By default many distributions disable core dumps (limit of 0) or drop them silently into the working directory as a plain file named core. For embedded debugging you want dumps collected somewhere predictable, named so you can tell crashes apart, and — ideally — piped through a small handler that can filter or compress them before they hit flash.

The kernel reads its dump destination and naming template from /proc/sys/kernel/core_pattern at the moment of the crash. You can point it at an absolute path with format specifiers, or at a pipe to an external program.

Enable core dumps for the current shell

$ ulimit -c unlimited

Then set a pattern that collects every dump into one directory, tagged with the executable name and a timestamp:

# echo /var/crash/core.%e.%p.%t > /proc/sys/kernel/core_pattern

The commonly used specifiers are:

Because the setting lives under /proc, it resets on reboot unless you persist it through sysctl.conf or an init script — something worth baking into your board’s rootfs rather than setting by hand each time.

Instead of a path, core_pattern can start with a pipe character followed by a program and arguments. The kernel then runs that program while the crashed process’s memory is still resident, streaming the core image to the handler’s standard input rather than a file:

# echo "|/usr/local/bin/ep_crash_collector %e %p %s %t" > /proc/sys/kernel/core_pattern

A handler like this can inspect /proc/<pid> while it still exists, strip large or sensitive memory regions before writing to flash, or compress the image on the fly. Once the handler finishes reading standard input, the kernel reclaims the process and that /proc entry disappears — so any inspection has to happen while data is still arriving on the pipe, not afterward.

Here is a small original demo that deliberately dereferences a null pointer, so we have something real to load into GDB:

/* ep_crashdemo.c */
#include <stdio.h>
#include <stdlib.h>

struct ep_node {
    int value;
    struct ep_node *next;
};

static void ep_bump(struct ep_node *n)
{
    n->value += 1;   /* crashes if n is NULL */
}

int main(void)
{
    struct ep_node *head = NULL;
    printf("about to bump a null node\n");
    ep_bump(head);
    return 0;
}

Build with debug info and run it

$ gcc -g -O0 -o ep_crashdemo ep_crashdemo.c
$ ulimit -c unlimited
$ ./ep_crashdemo
about to bump a null node
Segmentation fault (core dumped)
$ ls /var/crash/
core.ep_crashdemo.4211.1735489213

Load the binary together with its core file:

$ gdb ./ep_crashdemo /var/crash/core.ep_crashdemo.4211.1735489213
...
Core was generated by `./ep_crashdemo'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0000555555555149 in ep_bump (n=0x0) at ep_crashdemo.c:11
11          n->value += 1;

GDB has already told us the culprit: n is 0x0. Two commands confirm the picture. list shows the surrounding source, and backtrace (or bt) shows the call chain that led here:

(gdb) list
6       struct ep_node {
7           int value;
8           struct ep_node *next;
9       };
10
11          n->value += 1;   /* crashes if n is NULL */
12      }
13
(gdb) bt
#0  0x0000555555555149 in ep_bump (n=0x0) at ep_crashdemo.c:11
#1  0x0000555555555172 in main () at ep_crashdemo.c:18

The backtrace pinpoints exactly where a NULL check was skipped: main() passed an uninitialized-on-purpose null head straight into ep_bump(). On real crashes the same two commands — list and bt — are usually enough to get you from “it crashed somewhere” to “it crashed here, because of this.”

Keep a debug-symbol copy of every release binary you ship, even if the field image is stripped — you can point GDB at the separate symbol file with symbol-file. Rotate or cap the crash directory so a crash loop doesn’t fill flash. And prefer piping to a small collector over writing raw dumps directly when memory images might contain sensitive data.

core_pattern controls where and how the kernel saves a crashed process’s memory image, either as a named file or a stream to a handler process. GDB reopens that image against the matching binary, and bt plus list turn “segmentation fault” into an exact line of code and call path — a core workflow in any free linux device drivers course or field-debugging toolkit.