What You Will Learn

• How to attach GDB to a process that’s already running, without restarting it
• How GDB handles a process that calls fork(), and gdbserver’s key limitation there
• How to control which threads stay stopped at a breakpoint using scheduler-locking
• What core files are, why they aren’t generated by default, and how to control their naming

Prerequisites

• Completed the earlier lectures on connecting GDB/gdbserver and library debugging
• A target program you can safely stop and resume for the attach demo

Just-in-Time Debugging With attach

Sometimes a long-running program starts misbehaving after it’s been up for a while, and you want to look inside without restarting it and losing whatever state led to the problem. GDB’s attach feature does exactly that — it works for both native and remote sessions, and is genuinely useful in production troubleshooting, not just development.

For remote debugging, find the PID of the process on the target and pass it to gdbserver with --attach:

# On the target, ep_datalogger has been running for hours, PID 341
target# gdbserver --attach :2345 341
Attached; pid = 341
Listening on port 2345

This forces the process to halt as if it had hit a breakpoint, without terminating it. From here, connect your host GDB exactly as in a normal remote session:

(gdb) target remote 192.168.1.50:2345
Remote debugging using 192.168.1.50:2345

Inspect whatever you need — variables, the call stack, thread state — then detach cleanly when you’re done, letting the program continue running exactly as before, undisturbed by the debugger:

(gdb) detach
Detaching from program: /opt/ep_datalogger/ep_datalogger, process 341
Ending remote debugging.

Debugging Programs That Fork

When a debugged program calls fork(), GDB has to decide which side of the split to keep following — the parent or the newly created child. This is controlled by the follow-fork-mode setting, which accepts parent (the default) or child:

(gdb) set follow-fork-mode child

The important caveat: current gdbserver builds don’t support this option at all — it only works when debugging natively. If you’re stuck needing to debug the child specifically while using gdbserver, the practical workaround is a small code change: have the child spin on a flag variable immediately after the fork, giving you time to attach a fresh gdbserver session to its PID and then set the flag to release it from the loop. It’s a manual workaround, but it’s the standard one for this gdbserver limitation.

Debugging Threads

When any thread in a multithreaded process hits a breakpoint, GDB’s default behavior is to halt every thread in the process — not just the one that hit the breakpoint. This is usually the right default: it lets you inspect shared state without other threads mutating it out from under you while you look. The one time this becomes a problem is single-stepping — resuming causes every stopped thread to start moving again, even though you only meant to step the one you’re focused on, which can make the behavior you’re chasing disappear.

The scheduler-locking parameter controls this. Setting it to on keeps every thread except the one currently at the breakpoint frozen, giving you a clean, uncontaminated view of that thread’s behavior alone:

(gdb) set scheduler-locking on
(gdb) step

Turn it back off once you’re done isolating that thread, or other threads will remain artificially frozen for the rest of the session:

(gdb) set scheduler-locking off

Unlike follow-fork-mode, gdbserver does support scheduler-locking, so this works identically in both native and remote sessions.

Introduction to Core Files

A core file captures the complete state of a program at the exact moment it crashed — registers, memory, the call stack, everything needed to reconstruct what was happening. That means you don’t need a live debugger session running at the moment of the crash; the evidence is preserved for later. Seeing Segmentation fault (core dumped) should be treated as an opportunity, not just a failure message — that core file is a genuine goldmine of information if you know where to find it.

Core files are not generated by default — only when the core file resource limit for the shell is non-zero. Remove any size limit for your current shell with ulimit:

$ ulimit -c unlimited

By default, a core file is simply named core and dropped into the crashing process’s current working directory. This default scheme has real problems in practice: on a device generating multiple crash dumps, several files all named core make it impossible to tell which program produced which; the working directory might be read-only; or there may not be enough free space to write the dump at all.

Two kernel-level files give you much finer control over naming and placement. /proc/sys/kernel/core_uses_pid, when set to 1, appends the crashing process’s PID to the filename — useful if you can correlate that PID back to a program name from your logs:

$ echo 1 | sudo tee /proc/sys/kernel/core_uses_pid

Far more powerful is /proc/sys/kernel/core_pattern, which lets you fully customize the dump path and filename using meta-characters, including at minimum:

A pattern like this puts every dump in one predictable location with a self-describing name, solving the ambiguity of the plain core default:

$ echo "/var/crash/core.%p.%u" | sudo tee /proc/sys/kernel/core_pattern

We’ll pick this thread back up in a dedicated core file analysis lecture — loading a dump into GDB, walking its backtrace, and inspecting memory exactly as if the crash were still happening live.

Real-World Use Case

A field-deployed gateway occasionally reboots with no log entry explaining why. Setting ulimit -c unlimited in the service’s startup script and a sensible core_pattern pointing at a persistent partition means the next crash leaves behind a self-contained core file you can pull off the device and analyze on your development host — turning an unreproducible field failure into a concrete backtrace.

Common Mistakes and Troubleshooting

• Assuming follow-fork-mode child works over gdbserver — it doesn’t; use the spin-and-attach workaround for remote fork debugging.
• Leaving scheduler-locking on after you’re done — other threads stay frozen unexpectedly for the rest of the session; always turn it back off.
• Expecting a core file after a crash with no ulimit -c set — the default limit is often zero, silently suppressing core dumps entirely.
• Multiple crashes overwriting the same core file — set core_uses_pid or a custom core_pattern to avoid losing earlier dumps.

Best Practices

• Set a descriptive core_pattern pointing at a writable, persistent location on any device you expect to debug crashes on in the field.
• Use attach/detach for production troubleshooting rather than restarting a service and losing the exact conditions that triggered a bug.
• Turn scheduler-locking off as soon as you’re done isolating a specific thread’s behavior.

Summary and Key Takeaways

• gdbserver --attach :PORT PID lets you inspect an already-running process without restarting it, and detach resumes it unharmed.
• follow-fork-mode controls parent-vs-child debugging after fork(), but only works natively — not over gdbserver.
• scheduler-locking on keeps other threads frozen while you step through one thread in isolation; gdbserver does support this one.
• Core files require a non-zero ulimit -c, and core_uses_pid/core_pattern control their naming and placement.

Conclusion

This closes out the core GDB toolkit for this free linux kernel development course chapter: build flags, remote sessions, the essential command set, library and source debugging, attaching to live processes, and enough core-file groundwork to know what to reach for after a crash. Together these cover the everyday reality of debugging embedded Linux systems — where the bug rarely announces itself conveniently and the tooling has to meet you wherever it happens.

FAQ