Systemd Watchdog For Embedded Linux

Why “Is The Process Alive?” Isn’t Enough

A crashed process is the easy failure mode — the kernel reaps it, systemd notices the exit, and a Restart= policy brings it back. The dangerous failure mode is different: the process is technically alive, its PID still shows up in ps, but it’s spinning in a bad loop, blocked on a mutex it will never get, or wedged waiting on hardware that stopped responding. From the outside this looks identical to a healthy service. This is exactly the gap systemd’s watchdog mechanism closes — instead of asking “does the process exist,” it asks “is the process still making forward progress.”

Two Independent Watchdog Layers

systemd actually gives you two separate protections, and it’s worth being clear about which one solves which problem:

Think of it as a chain of trust: your service is watched by systemd, and systemd is watched by the hardware. If any link stops responding, the layer below it resets that link.

Building A Service That Feeds Its Own Watchdog

Let’s write ep-watchdogd, a small original daemon that does real (if trivial) work in a loop, and reports its own health to systemd on schedule. The core idea: read WATCHDOG_USEC from the environment, and call sd_notify() with a WATCHDOG=1 keepalive at roughly half that interval.

// ep_watchdogd.c — original demo daemon for this lecture
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <systemd/sd-daemon.h>

static int ep_do_work_cycle(void)
{
    /* Stand-in for real daemon work: sensor poll, queue drain, etc. */
    printf("ep-watchdogd: work cycle complete\n");
    return 0;
}

int main(void)
{
    uint64_t watchdog_usec = 0;
    int watchdog_enabled = sd_watchdog_enabled(0, &watchdog_usec);

    sd_notify(0, "READY=1");

    for (;;) {
        ep_do_work_cycle();

        if (watchdog_enabled > 0) {
            /* Feed at roughly half the configured timeout */
            sd_notify(0, "WATCHDOG=1");
        }

        sleep(watchdog_enabled > 0 ? (watchdog_usec / 2) / 1000000 : 5);
    }
}

Build it against libsystemd:

$ gcc -o ep-watchdogd ep_watchdogd.c $(pkg-config --cflags --libs libsystemd)
$ sudo cp ep-watchdogd /usr/local/bin/

The Unit File: WatchdogSec And Restart Limits

Now tell systemd how long it should wait for each heartbeat before it decides ep-watchdogd is stuck:

[Unit]
Description=EP Watchdog Demo Daemon

[Service]
ExecStart=/usr/local/bin/ep-watchdogd
WatchdogSec=20s
Restart=on-watchdog
StartLimitIntervalSec=5min
StartLimitBurst=4
StartLimitAction=reboot-force

[Install]
WantedBy=multi-user.target

Reading this unit: the daemon has 20 seconds of silence budget before systemd considers it stuck. If it gets restarted more than four times within a five-minute window, systemd concludes the restart loop itself isn’t fixing anything, and forces a full reboot rather than restarting forever.

Arming The Hardware Watchdog

The per-service watchdog above only protects you if systemd itself is alive to enforce it. If the kernel panics, or the whole system locks up, systemd can’t act. That’s what the hardware watchdog layer is for. Enable it in /etc/systemd/system.conf:

[Manager]
RuntimeWatchdogSec=20s

With this set, systemd itself pets the hardware watchdog device (typically /dev/watchdog) every fraction of that interval. If systemd stops running — for any reason, including a kernel lockup — nobody pets the hardware timer, and the board resets itself. Verify it’s active with:

$ sudo systemctl show -p RuntimeWatchdogUSec
RuntimeWatchdogUSec=20s

$ cat /sys/class/watchdog/watchdog0/timeout
20

Implications For Embedded Boards

Watchdogs are close to free reliability, but systemd itself is not free — and this matters on storage-constrained targets. A minimal build of just the core pieces most embedded products actually use — systemd, systemd-udevd, and systemd-journald, plus their shared library dependencies — lands close to the 10 MiB mark. On a board with an 8 MiB SPI-NOR rootfs, that single fact can rule systemd out regardless of how good its watchdog handling is.

The second constraint is version coupling: systemd’s release cadence tracks the kernel closely, and a given systemd release generally assumes it’s running on a kernel from roughly the same era — not one that’s a year or two older. If your BSP is frozen on an older kernel for board-support reasons, pulling in a brand-new systemd can introduce compatibility friction that has nothing to do with your application logic.

Common Mistakes And Troubleshooting

• Forgetting the halving rule: feed the watchdog at roughly half of WatchdogSec, not right at the deadline — jitter in the scheduler can push a late notification past the limit.
• Not checking sd_watchdog_enabled(): if the unit doesn’t set WatchdogSec, WATCHDOG_USEC won’t be set either — calling sd_notify(0, "WATCHDOG=1") unconditionally is harmless but wasteful; checking first avoids needless wakeups.
• Setting RuntimeWatchdogSec without testing the reset path: verify a real reset actually happens (e.g. by suspending systemd’s PID with SIGSTOP in a lab, never in production) before you trust it in the field.
• Diagnosing a mystery reboot: check journalctl -b -1 -k | grep -i watchdog after an unexpected reset — the previous boot’s kernel log usually records that the watchdog fired.

Best Practices

• Set StartLimitBurst and StartLimitIntervalSec together — a lone restart policy without a burst limit can loop forever on a truly broken service.
• Log the reason for a watchdog reset somewhere persistent (not just RAM-backed journald) so a field failure is diagnosable after the fact.
• Test your watchdog timeout under real worst-case load, not idle conditions.
• Treat RuntimeWatchdogSec as the last line of defense, not the primary reliability mechanism — fix the root cause of hangs first.

FAQ