What Will You Learn?
Since daemons have no terminal, they cannot use printf() or fprintf(stderr) to report errors and events. The syslog system is the standard Linux mechanism for daemon logging. In this tutorial you will understand how syslog works end-to-end โ from a C function call in your daemon all the way to a log file on disk.
Key Concepts
Regular programs write messages using printf() to stdout or fprintf(stderr, ...) to stderr. But a daemon:
All redirected to /dev/null โ printf() output disappears
Nobody is watching a screen for output
Events happen at 3am โ must be stored persistently
The solution is syslog โ a centralized logging system that: collects messages from all daemons, adds timestamps and metadata, routes messages to different files based on rules, and can even send logs to a remote server over the network.
|
sshd
syslog(LOG_INFO, “Login”) cron
syslog(LOG_DEBUG, “Job start”) kernel
printk() โ /proc/kmsg Applications & Daemons
|
โ |
/dev/log
Unix Domain Socket (SOCK_DGRAM) |
โ |
syslogd / rsyslogd
Reads messages, applies rules from /etc/syslog.conf /var/log/syslog
/var/log/auth.log
/var/log/kern.log
Remote server (UDP 514)
|
Step by step:
- A daemon calls
syslog(priority, format, ...) - The C library formats the message and writes it to the Unix domain socket
/dev/log - The syslogd daemon (or its modern replacement rsyslogd) is listening on
/dev/logand reads the message - syslogd checks its configuration file
/etc/syslog.confto decide where to route the message (which log file, which remote server) - The message is written to the appropriate file(s) under
/var/log/
Every syslog message carries a facility code that identifies the general category of the source. The facility is used by syslogd to route the message to different files. Think of it as a “tag” identifying the subsystem that generated the log.
| Facility Constant | Value | Intended Use |
|---|---|---|
LOG_KERN |
0 | Linux kernel messages (not for user programs) |
LOG_USER |
1 | Generic user-level messages (default if none specified) |
LOG_MAIL |
2 | Mail system (sendmail, postfix) |
LOG_DAEMON |
3 | System daemons โ use this for most daemons |
LOG_AUTH |
4 | Authentication & authorization (login, sudo, sshd) |
LOG_SYSLOG |
5 | syslogd internal messages |
LOG_LPR |
6 | Line printer subsystem |
LOG_NEWS |
7 | USENET news |
LOG_CRON |
9 | Clock daemon (cron, at) |
LOG_LOCAL0..7 |
16โ23 | Local use โ for your own custom daemons |
LOG_DAEMON if it is a system-level daemon, or one of LOG_LOCAL0 through LOG_LOCAL7 if you want fine-grained control and separate log routing in syslog.conf.Every syslog message also has a priority (also called severity level) that indicates how serious the event is. There are 8 levels, from most critical to least:
| Level | Value | Meaning | Example |
|---|---|---|---|
LOG_EMERG |
0 | System is unusable | “Kernel panic โ not syncing” |
LOG_ALERT |
1 | Action must be taken immediately | “Disk nearly full” |
LOG_CRIT |
2 | Critical conditions | “Hardware failure detected” |
LOG_ERR |
3 | Error conditions | “Connection to DB failed” |
LOG_WARNING |
4 | Warning conditions | “Config value not set, using default” |
LOG_NOTICE |
5 | Normal but significant conditions | “Daemon started successfully” |
LOG_INFO |
6 | Informational messages | “User ravi logged in” |
LOG_DEBUG |
7 | Debug-level messages | “Processing item 423 of 1000” |
In syslog, the priority passed to syslog() is actually a combination of the facility and the severity level. The C library provides the LOG_MAKEPRI(facility, level) macro to combine them, but you usually just OR them together:
/* Combine facility and level with bitwise OR */
syslog(LOG_DAEMON | LOG_ERR, "Connection failed: %m");
syslog(LOG_AUTH | LOG_INFO, "User %s logged in", username);
syslog(LOG_LOCAL0 | LOG_DEBUG,"Processing packet %d", seq);
/* If you only specify a level (no facility), the facility
* set in openlog() is used automatically: */
openlog("myapp", LOG_PID, LOG_DAEMON);
syslog(LOG_ERR, "This uses LOG_DAEMON facility automatically");
syslog(LOG_INFO, "So does this");
The priority integer is encoded as: priority = (facility << 3) | level. For example, LOG_DAEMON|LOG_ERR = (3<<3)|3 = 27.
#include <stdio.h>
#include <stdlib.h>
#include <syslog.h>
#include <unistd.h>
/*
* Demonstrates syslog facilities and all priority levels.
* After running, check: sudo tail -f /var/log/syslog
*/
int main(void)
{
/* Open syslog connection with program name "syslog_demo",
* include PID in messages, use LOG_DAEMON facility */
openlog("syslog_demo", LOG_PID | LOG_PERROR, LOG_DAEMON);
/* LOG_PERROR also prints to stderr โ useful during testing */
printf("Sending messages to syslog. PID=%d\n", getpid());
printf("Check: sudo tail -f /var/log/syslog\n\n");
/* Send one message at each priority level */
syslog(LOG_EMERG, "EMERG: System unusable โ test message");
syslog(LOG_ALERT, "ALERT: Action required immediately");
syslog(LOG_CRIT, "CRIT: Critical condition");
syslog(LOG_ERR, "ERR: Error condition โ errno: %m");
syslog(LOG_WARNING, "WARNING: Warning condition");
syslog(LOG_NOTICE, "NOTICE: Normal but significant");
syslog(LOG_INFO, "INFO: Informational message");
syslog(LOG_DEBUG, "DEBUG: Debug details, process=%d", getpid());
/* Send messages with different facilities */
syslog(LOG_AUTH | LOG_INFO, "AUTH: Authentication message");
syslog(LOG_CRON | LOG_INFO, "CRON: Cron-type message");
syslog(LOG_LOCAL0 | LOG_INFO, "LOCAL0: Custom app message");
syslog(LOG_LOCAL7 | LOG_DEBUG, "LOCAL7: Custom debug message");
/* Demonstrate %m โ automatic strerror(errno) expansion */
/* Trigger an error first */
FILE *f = fopen("/nonexistent/path/file.txt", "r");
if (!f) {
/* %m is a syslog extension that inserts strerror(errno) */
syslog(LOG_ERR, "Failed to open file: %m");
}
closelog();
printf("Done. Check /var/log/syslog for messages.\n");
return 0;
}
gcc -o syslog_demo syslog_demo.c
./syslog_demo
sudo grep "syslog_demo" /var/log/syslog | tail -20
Kernel messages (from printk()) also enter syslog via a different path: /proc/kmsg. This example shows how to read kernel log messages directly.
#include <stdio.h>
#include <stdlib.h>
#include <fcntl.h>
#include <unistd.h>
#include <string.h>
/*
* Kernel log priority levels (same as syslog levels)
* In /proc/kmsg, each line starts with <N> where N is the level
*/
const char *kernel_level_name(int level)
{
switch (level) {
case 0: return "EMERG";
case 1: return "ALERT";
case 2: return "CRIT";
case 3: return "ERR";
case 4: return "WARNING";
case 5: return "NOTICE";
case 6: return "INFO";
case 7: return "DEBUG";
default: return "UNKNOWN";
}
}
/*
* Read and print kernel log messages from /proc/kmsg
* Must run as root.
*
* Alternative: use dmesg command or the klogctl() syscall.
*/
int main(void)
{
int fd;
char buf[1024];
ssize_t n;
fd = open("/proc/kmsg", O_RDONLY | O_NONBLOCK);
if (fd == -1) {
perror("open /proc/kmsg (need root?)");
return 1;
}
printf("Kernel log messages (reading /proc/kmsg):\n");
printf("%-10s %s\n", "LEVEL", "MESSAGE");
printf("%-10s %s\n", "-----", "-------");
int count = 0;
while (count < 20) { /* Read up to 20 messages */
n = read(fd, buf, sizeof(buf) - 1);
if (n <= 0) break;
buf[n] = '\0';
/* Parse: <level>message text */
char *p = buf;
while (*p) {
if (*p == '<') {
int level = *(p+1) - '0';
char *msg_start = strchr(p, '>');
if (msg_start) {
msg_start++;
char *newline = strchr(msg_start, '\n');
if (newline) *newline = '\0';
printf("%-10s %s\n", kernel_level_name(level), msg_start);
count++;
p = newline ? newline + 1 : p + 1;
} else p++;
} else p++;
}
}
close(fd);
printf("\nNote: Use 'dmesg' for easier kernel log access\n");
printf("Or: sudo journalctl -k (systemd systems)\n");
return 0;
}
gcc -o read_kmsg read_kmsg.c
sudo ./read_kmsg
# Alternative โ show kernel messages with dmesg:
dmesg | tail -20
dmesg --level=err,warn # Only errors and warnings
dmesg -T # With human-readable timestamps
Answer: Daemons redirect stdin, stdout, and stderr to /dev/null during initialization. Any output to printf() (which writes to stdout, FD 1) is silently discarded by /dev/null. Additionally, since a daemon has no terminal, there is nobody watching for output. syslog() routes log messages to files on disk where they can be reviewed later.
Answer: /dev/log is a Unix domain socket of type SOCK_DGRAM. When a process calls syslog(), the C library formats the message and writes it as a datagram to /dev/log. The syslogd daemon is permanently listening on this socket and reads each message as it arrives, then routes it according to /etc/syslog.conf rules.
Answer: The facility identifies the category/subsystem that generated the message (LOG_DAEMON, LOG_AUTH, LOG_KERN, etc.). The priority (severity) indicates how serious the message is (LOG_EMERG=0 being most critical, LOG_DEBUG=7 being least). Both are combined into a single integer when calling syslog(): e.g., LOG_DAEMON|LOG_ERR.
Answer: %m is a syslog-specific format specifier that is automatically replaced with the string returned by strerror(errno) โ the human-readable description of the current error. It is equivalent to writing “…: %s”, strerror(errno) but more concise. Example: syslog(LOG_ERR, “Cannot open file: %m”); outputs “Cannot open file: No such file or directory”.
Answer: These 8 facilities are reserved for local use โ site-specific or application-specific logging. A system administrator can configure /etc/syslog.conf to route LOG_LOCAL0 messages to one file and LOG_LOCAL3 messages to another, allowing different custom applications to have their own separate log files without interfering with system daemon logs.
Answer: The Linux kernel uses printk() internally to generate log messages. These are stored in a kernel ring buffer accessible via /proc/kmsg. The syslogd daemon reads from /proc/kmsg (in addition to /dev/log) and routes kernel messages (facility LOG_KERN) to the appropriate log file, typically /var/log/kern.log.
Chapter Summary
syslog is the standard daemon logging system. Messages flow from daemon โ /dev/log (Unix domain socket) โ syslogd โ log files. Each message carries a facility (source category) and a priority (severity level 0-7). Facilities like LOG_DAEMON and LOG_LOCAL0-7 are used by application daemons. LOG_DEBUG through LOG_EMERG define the importance of the message.
