Monitoring the Dynamic Linker LD_DEBUG_OUTPUT · Process ID Tracing

42.6 — Monitoring the Dynamic Linker

LD_DEBUG · All Keywords · LD_DEBUG_OUTPUT · Process ID Tracing | Chapter 42 · TLPI

EmbeddedPathashala.com — Free Embedded & Linux Tutorials

What Is LD_DEBUG?

LD_DEBUG is an environment variable recognized by the glibc dynamic linker (ld.so / ld-linux.so). When set, it causes the dynamic linker to print detailed diagnostic messages about its internal operations to standard error.

Think of it as a built-in debugger for the loading and linking process. You can see exactly which directories were searched for a library, which symbol matched which library, how relocations were performed, and what version requirements are being checked.

When is it useful?

Diagnosing “library not found” errors
Understanding why the wrong version of a library is being picked up
Debugging symbol interposition issues
Learning how the dynamic linker works internally
Verifying that version scripts are working correctly

Key Concepts

LD_DEBUG libs keyword symbols keyword bindings keyword versions keyword reloc keyword files keyword all keyword help keyword LD_DEBUG_OUTPUT setuid restriction

All LD_DEBUG Keywords

LD_DEBUG accepts a comma-separated list of keywords. Each keyword enables a different category of diagnostic output:

Keyword	What It Shows	Typical Use
libs	Library search paths, which library is finally loaded for each dependency	Diagnose “cannot find shared library” errors
symbols	Symbol table lookups — which library each symbol was found in	Understand which library provides a given function
bindings	Symbol binding details — what binds to what and where	Debug interposition and symbol resolution order
versions	Version dependency checks between libraries	Debug symbol versioning issues
reloc	Relocation operations performed by the dynamic linker	Advanced: debug PLT/GOT and relocation entries
files	Progress of loading individual object files (maps, init functions)	Trace the complete file loading sequence
scopes	Symbol scope information (which scope each lookup goes through)	Debug RTLD_LOCAL/RTLD_GLOBAL scope issues
statistics	Summary statistics at the end: number of relocations, symbols looked up, etc.	Performance analysis of startup linking
unused	Lists unused direct dependencies (libraries linked but not actually used)	Identify unnecessary -l flags to clean up build
all	All of the above categories at once	Full diagnostic dump (very verbose)
help	Prints the list of valid keywords and exits	Quick reference

# Quick reference — print all valid keywords:
LD_DEBUG=help /bin/ls

Understanding LD_DEBUG Output Format

Each line of LD_DEBUG output starts with the process ID followed by a colon, making it easy to identify which process generated the line (important for multi-process programs):

#  Format: PROCESS_ID:   CATEGORY   message
#  Example:
#     12345:	   find library=libm.so.6 [0]; searching
#     12345:	    search cache=/etc/ld.so.cache
#     12345:	      trying file=/lib/x86_64-linux-gnu/libm.so.6
#     12345:	   found /lib/x86_64-linux-gnu/libm.so.6

Redirecting output with LD_DEBUG_OUTPUT:

# By default, LD_DEBUG output goes to stderr (fd 2)
# LD_DEBUG_OUTPUT redirects it to a file (adds PID as suffix):
LD_DEBUG=libs LD_DEBUG_OUTPUT=/tmp/linker_trace ./my_program

# This creates /tmp/linker_trace.12345 (where 12345 is the PID)
# Very useful when stderr is polluted by the program's own output

setuid restriction: Just like LD_PRELOAD, the LD_DEBUG variable is completely ignored for setuid and setgid programs. This prevents information disclosure (loading paths, symbol names) that could aid privilege escalation attacks.

Code Example 1: Diagnosing Library Search with LD_DEBUG=libs

This is the most commonly used keyword — shows exactly where the linker looks for each library:

/* simple_prog.c — just uses printf and math */
#include <stdio.h>
#include <math.h>

int main(void) {
    printf("sin(0) = %f\n", sin(0.0));
    return 0;
}

gcc simple_prog.c -lm -o simple_prog

# Run with LD_DEBUG=libs to see library search:
LD_DEBUG=libs ./simple_prog 2>&1 | head -30

# Sample output (format: PID: info):
     12345:	initialize program: ./simple_prog
     12345:
     12345:	find library=libm.so.6 [0]; searching
     12345:	 search cache=/etc/ld.so.cache
     12345:	  trying file=/lib/x86_64-linux-gnu/libm.so.6
     12345:
     12345:	find library=libc.so.6 [0]; searching
     12345:	 search cache=/etc/ld.so.cache
     12345:	  trying file=/lib/x86_64-linux-gnu/libc.so.6
     12345:
     12345:	calling init: /lib/x86_64-linux-gnu/libm.so.6
     12345:	calling init: /lib/x86_64-linux-gnu/libc.so.6
     12345:	calling init: ./simple_prog
     12345:
sin(0) = 0.000000
     12345:	calling fini: ./simple_prog
     12345:	calling fini: /lib/x86_64-linux-gnu/libm.so.6

How to use this for debugging: If your program fails to start with “error while loading shared libraries: libfoo.so.1: cannot open shared object file”, run with LD_DEBUG=libs to see exactly which directories were searched and which file was tried — this instantly shows you whether the library is in the wrong location, has the wrong name, or is missing from the cache.

Code Example 2: Symbol Resolution with LD_DEBUG=symbols,bindings

# See which library each function symbol is resolved from:
LD_DEBUG=symbols ./simple_prog 2>&1 | grep "printf\|sin" | head -10

# Sample output:
     12345:	symbol=printf;  lookup in file=./simple_prog [0]
     12345:	symbol=printf;  lookup in file=/lib/x86_64-linux-gnu/libm.so.6 [0]
     12345:	symbol=printf;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
     12345:	binding file ./simple_prog to /lib/.../libc.so.6: normal symbol `printf' [GLIBC_2.2.5]
     12345:	symbol=sin;  lookup in file=./simple_prog [0]
     12345:	symbol=sin;  lookup in file=/lib/x86_64-linux-gnu/libm.so.6 [0]
     12345:	binding file ./simple_prog to /lib/.../libm.so.6: normal symbol `sin'

This output shows the complete search path for each symbol — you can see that printf was searched in the main program, then libm (not found), then libc where it was found and bound. The version tag GLIBC_2.2.5 is also shown.

Code Example 3: Debugging LD_PRELOAD Interposition

Combine LD_PRELOAD with LD_DEBUG=bindings to verify your interposition is working:

/* tiny_override.c — override puts() */
#define _GNU_SOURCE
#include <dlfcn.h>
#include <stdio.h>

int puts(const char *s) {
    static int (*real_puts)(const char *) = NULL;
    if (!real_puts) {
        dlerror();
        *(void **)(&real_puts) = dlsym(RTLD_NEXT, "puts");
    }
    fprintf(stderr, "[INTERCEPT] puts(\"%s\")\n", s);
    return real_puts(s);
}

gcc -fPIC -shared -o tiny_override.so tiny_override.c -ldl

# Use LD_DEBUG=bindings to verify the binding order:
LD_PRELOAD=./tiny_override.so LD_DEBUG=bindings echo "hello" 2>&1 | grep puts

# Expected output shows tiny_override.so wins the puts binding:
#   binding /bin/echo [0] to ./tiny_override.so [0]: normal symbol `puts'
# (means echo's puts reference was bound to our override!)

Code Example 4: Checking Version Dependencies

# Check version dependencies for a program:
LD_DEBUG=versions /bin/ls 2>&1 | head -20

# Sample output:
     12345:	checking for version `GLIBC_2.2.5' in file /lib/.../libc.so.6 required by /bin/ls
     12345:	checking for version `GLIBC_2.3.4' in file /lib/.../libc.so.6 required by /bin/ls
     12345:	checking for version `GLIBC_2.5'   in file /lib/.../libc.so.6 required by /bin/ls
     12345:	checking for version `GLIBC_2.17'  in file /lib/.../libc.so.6 required by /bin/ls

This is invaluable when a binary fails with “GLIBC_2.29 not found” — it shows exactly which versions are required and which are missing. You can then check whether the installed glibc supports those versions with:

strings /lib/x86_64-linux-gnu/libc.so.6 | grep "^GLIBC_" | sort -V

Code Example 5: Linker Statistics for Performance Analysis

# Show startup linking statistics:
LD_DEBUG=statistics /bin/ls /tmp 2>&1 | grep -A 10 "runtime linker statistics"

# Sample output:
runtime linker statistics:
  total startup time in dynamic loader: 1234567 cycles
            time needed for relocation: 345678 cycles (27.9%)
                 number of relocations: 1234
      number of relocations from cache: 890
        number of relative relocations: 345
             time needed to load maps: 234567 cycles (19.0%)
  number of objects with versioned sym: 4

Practical use: If your program has a slow startup, LD_DEBUG=statistics tells you whether the bottleneck is relocation processing (meaning too many symbols to resolve) or library loading (meaning too many libraries or slow filesystem access to the library cache).

Quick Reference: Common LD_DEBUG Commands

# Print all valid LD_DEBUG keywords:
LD_DEBUG=help /bin/true

# Most useful: see which libraries are loaded and where from:
LD_DEBUG=libs ./myprogram

# Debug a "symbol not found" error:
LD_DEBUG=symbols ./myprogram 2>&1 | grep "symbol=missing_func"

# Verify LD_PRELOAD interposition is working:
LD_PRELOAD=./wrapper.so LD_DEBUG=bindings ./myprogram 2>&1 | grep "wrapped_func"

# Save output to file (avoids mixing with program's stderr):
LD_DEBUG=libs LD_DEBUG_OUTPUT=/tmp/debug ./myprogram

# Check GLIBC version requirements:
LD_DEBUG=versions ./myprogram 2>&1 | grep "checking for version"

# Check for unused library dependencies:
LD_DEBUG=unused ./myprogram 2>&1 | grep unused

# Get linker performance stats:
LD_DEBUG=statistics ./myprogram 2>&1 | grep "statistics" -A 10

# Full dump (very verbose — best with LD_DEBUG_OUTPUT):
LD_DEBUG=all LD_DEBUG_OUTPUT=/tmp/full_trace ./myprogram

Interview Questions & Answers

Q1. What is LD_DEBUG and what is its primary purpose?

LD_DEBUG is an environment variable recognized by the glibc dynamic linker. When set, it causes the dynamic linker to print detailed diagnostic information about its operations to standard error. Its primary purpose is debugging dynamic linking issues — finding why a library cannot be located, understanding which library a symbol resolves to, diagnosing version conflicts, and verifying that interposition is working as expected.

Q2. Which LD_DEBUG keyword would you use to diagnose a “cannot find shared library” error?

The libs keyword. LD_DEBUG=libs causes the dynamic linker to print every directory it searches for each library, and every file it tries to open. When a library cannot be found, this output shows exactly which directories were searched — you can then determine whether the library is in the wrong location, has the wrong filename (e.g., libfoo.so vs libfoo.so.1), or is missing from the ld.so.cache (in which case ldconfig needs to be run).

Q3. What is the difference between LD_DEBUG=symbols and LD_DEBUG=bindings?

LD_DEBUG=symbols shows each symbol lookup — it prints every library that was searched for each symbol in order. This shows the search path. LD_DEBUG=bindings shows the final binding result — which specific library a symbol reference was bound to. The bindings output is more focused and shows the definitive resolution. When debugging symbol interposition (e.g., verifying an LD_PRELOAD override took effect), LD_DEBUG=bindings is the most useful because it directly shows whether your wrapper library or the original library “won” for each symbol.

Q4. Why is LD_DEBUG ignored for setuid programs?

For setuid programs, LD_DEBUG could reveal sensitive information: the full paths where libraries are loaded from, symbol resolution paths, and the internal structure of the dynamic linking process. This information could help an attacker understand how to craft an attack against a privileged program. Additionally, on some older systems LD_DEBUG interactions with the linker could potentially be exploited. The dynamic linker therefore ignores LD_DEBUG entirely for setuid/setgid executables as a security measure.

Q5. What does LD_DEBUG_OUTPUT do and why is it useful?

By default, LD_DEBUG writes to standard error. LD_DEBUG_OUTPUT=path redirects the output to a file, appending the process ID as a suffix (e.g., path.12345). This is useful for three reasons: first, it keeps the linker diagnostics separate from the program’s own stderr output so they do not interleave; second, for multi-process programs (fork/exec), each child process gets its own file with its PID in the name; third, the file can be analyzed after the program runs without scrolling through mixed output.

Q6. A program fails at startup with “GLIBC_2.29 not found”. How do you use LD_DEBUG to investigate?

Use LD_DEBUG=versions to see all version checks the program performs: LD_DEBUG=versions ./myprogram 2>&1 | grep “checking for version”. This lists every GLIBC_X.Y version the program requires. Then check what the installed glibc actually provides: strings /lib/x86_64-linux-gnu/libc.so.6 | grep “^GLIBC_” | sort -V. If GLIBC_2.29 appears in the first list but not the second, the installed glibc is too old and the binary was compiled on a newer system. The solution is either to upgrade glibc or recompile the program on a matching system.

← Prev: LD_PRELOAD ← Back to Index

embeddedpathashala.com