What You Will Learn

• How to start and supervise a logging daemon with BusyBox’s syslogd and klogd
• Why writing logs to flash storage is risky on embedded devices
• The three ways device nodes get created: static mknod, devtmpfs, mdev, and udev
• How to pick the right device node strategy for your hardware constraints
• A working mdev.conf example with ownership and permission rules

Prerequisites

This lecture builds on the init/inittab and user account lectures earlier in this series — you should already have a working rcS script and a non-root service account available.

Starting a Daemon via inittab

Any long-running background service can be supervised the same way: add a respawn line to /etc/inittab. BusyBox’s syslogd applet is the classic example — it collects log messages from other programs into a central log:

::sysinit:/etc/init.d/rcS
::respawn:/sbin/syslogd -n
::respawn:/sbin/klogd -n
::respawn:/sbin/getty 115200 console

The -n flag keeps each daemon in the foreground so init can supervise it directly — daemonizing itself would hide it from init’s process tracking, breaking the automatic restart. klogd forwards kernel log messages into syslogd so they end up in the same place as everything else, typically /var/log/messages.

Writing logs to raw flash on every boot wears it out over time. For production embedded devices, consider a ring buffer in RAM, rate-limited logging, or shipping logs off-device instead of writing continuously to flash.

Managing Device Nodes: Three Approaches

Historically, device nodes were created by hand with mknod, specifying a major/minor number for every device you might ever plug in. That doesn’t scale to modern hardware with hotpluggable USB, SD cards, and dynamically assigned devices. Three better approaches exist:

Using devtmpfs Alone

Mounting devtmpfs in rcS is often enough for simple targets:

mount -t devtmpfs devtmpfs /dev

Nodes appear with sane defaults — root ownership, mode 0600, with a handful of well-known exceptions like /dev/null and /dev/random defaulting to 0666 so any process can use them.

Adding mdev for Custom Rules

When you need specific ownership or extra actions on device arrival (for example, granting a non-root group access to a serial port), layer mdev on top of devtmpfs:

mount -t devtmpfs devtmpfs /dev
echo /sbin/mdev > /proc/sys/kernel/hotplug
mdev -s

An original /etc/mdev.conf example granting a custom group access to USB serial adapters:

# device_regex  uid:gid  mode
ttyUSB[0-9]+     root:ep_serial  0660
ep_sensor0       ep_user:ep_user 0644

Choosing udev

If your target already runs systemd and has the storage/RAM budget for it, udev gives you the same rules-based flexibility desktop Linux uses, plus integration with systemd unit dependencies. For genuinely constrained boards, that overhead usually isn’t worth it compared to mdev.

Real-World Use Case

Consider a board with a USB-attached sensor and a debug UART. With devtmpfs alone, both nodes appear with root-only access — fine for a bring-up shell, but not for a non-root application process. Adding mdev rules lets that application’s dedicated user open the sensor node directly without ever needing root, closing off a whole class of privilege-escalation risk if the application is ever compromised.

Common Mistakes and Troubleshooting

• syslogd keeps restarting in a loop: usually a missing -n flag — the daemon forked into the background and init lost track of it.
• Device node never appears: devtmpfs isn’t mounted, or the driver never generates a uevent — check with dmesg.
• mdev rules ignored: /proc/sys/kernel/hotplug wasn’t set to point at mdev, so the kernel has nowhere to dispatch uevents.
• Permission denied opening a device as a non-root app: default devtmpfs permissions are 0600 — you need an mdev/udev rule to relax them for your specific device.

Best Practices

• Always pass -n to BusyBox daemons started under respawn.
• Prefer mdev over hand-rolled mknod scripts for anything beyond a handful of fixed devices.
• Grant device access to dedicated groups rather than widening permissions to 0666 wholesale.

Performance Considerations

Both mdev and udev add a small amount of boot-time and hotplug latency compared to raw devtmpfs, since each uevent triggers a userspace lookup against the rules file. For most embedded workloads this is negligible; for extremely latency-sensitive hotplug scenarios, benchmark it on your actual hardware.

Summary and Key Takeaways

• Supervise long-running daemons with respawn in inittab, and always run BusyBox daemons in foreground mode (-n).
• devtmpfs gives you automatic device nodes with sane defaults out of the box.
• mdev adds lightweight customization for embedded targets; udev is the full-featured option for systemd-based systems.

Conclusion

With logging and device node management in place, your root filesystem now behaves like a proper Linux system rather than a static image — services restart themselves, and hardware appears in /dev as it’s discovered, with exactly the permissions your applications need. That completes this stretch of our free linux kernel development course on rootfs runtime services.

FAQ