The syslog Logging System

 

The syslog Logging System
Chapter 37 โ€“ Part 5: Architecture, Facility, Severity & How Logs Flow
๐Ÿ“‹ syslog Architecture
๐Ÿท๏ธ Facility & Priority
๐Ÿ“‚ /var/log

What Will You Learn?

Since daemons have no terminal, they cannot use printf() or fprintf(stderr) to report errors and events. The syslog system is the standard Linux mechanism for daemon logging. In this tutorial you will understand how syslog works end-to-end โ€” from a C function call in your daemon all the way to a log file on disk.

Key Concepts

syslog() syslogd / rsyslogd Facility Priority / Severity /dev/log Unix Domain Socket LOG_KERN / LOG_USER LOG_DEBUG to LOG_EMERG

1. The Daemon Logging Problem

Regular programs write messages using printf() to stdout or fprintf(stderr, ...) to stderr. But a daemon:

stdin/stdout/stderr
All redirected to /dev/null โ€” printf() output disappears
No Terminal
Nobody is watching a screen for output
Long-Running
Events happen at 3am โ€” must be stored persistently

The solution is syslog โ€” a centralized logging system that: collects messages from all daemons, adds timestamps and metadata, routes messages to different files based on rules, and can even send logs to a remote server over the network.

2. syslog Architecture โ€” How Logs Flow

sshd
syslog(LOG_INFO, “Login”)
cron
syslog(LOG_DEBUG, “Job start”)
kernel
printk() โ†’ /proc/kmsg
Applications & Daemons
โ†’
/dev/log
Unix Domain Socket
(SOCK_DGRAM)
โ†’
syslogd / rsyslogd
Reads messages, applies rules from /etc/syslog.conf
/var/log/syslog
/var/log/auth.log
/var/log/kern.log
Remote server (UDP 514)

Step by step:

  1. A daemon calls syslog(priority, format, ...)
  2. The C library formats the message and writes it to the Unix domain socket /dev/log
  3. The syslogd daemon (or its modern replacement rsyslogd) is listening on /dev/log and reads the message
  4. syslogd checks its configuration file /etc/syslog.conf to decide where to route the message (which log file, which remote server)
  5. The message is written to the appropriate file(s) under /var/log/

3. Facility โ€” Where Did the Message Come From?

Every syslog message carries a facility code that identifies the general category of the source. The facility is used by syslogd to route the message to different files. Think of it as a “tag” identifying the subsystem that generated the log.

Facility Constant Value Intended Use
LOG_KERN 0 Linux kernel messages (not for user programs)
LOG_USER 1 Generic user-level messages (default if none specified)
LOG_MAIL 2 Mail system (sendmail, postfix)
LOG_DAEMON 3 System daemons โ€” use this for most daemons
LOG_AUTH 4 Authentication & authorization (login, sudo, sshd)
LOG_SYSLOG 5 syslogd internal messages
LOG_LPR 6 Line printer subsystem
LOG_NEWS 7 USENET news
LOG_CRON 9 Clock daemon (cron, at)
LOG_LOCAL0..7 16โ€“23 Local use โ€” for your own custom daemons
โœ… Best Practice: For your own daemon, use LOG_DAEMON if it is a system-level daemon, or one of LOG_LOCAL0 through LOG_LOCAL7 if you want fine-grained control and separate log routing in syslog.conf.

4. Priority (Severity) โ€” How Important Is the Message?

Every syslog message also has a priority (also called severity level) that indicates how serious the event is. There are 8 levels, from most critical to least:

Level Value Meaning Example
LOG_EMERG 0 System is unusable “Kernel panic โ€” not syncing”
LOG_ALERT 1 Action must be taken immediately “Disk nearly full”
LOG_CRIT 2 Critical conditions “Hardware failure detected”
LOG_ERR 3 Error conditions “Connection to DB failed”
LOG_WARNING 4 Warning conditions “Config value not set, using default”
LOG_NOTICE 5 Normal but significant conditions “Daemon started successfully”
LOG_INFO 6 Informational messages “User ravi logged in”
LOG_DEBUG 7 Debug-level messages “Processing item 423 of 1000”
Important: Lower number = higher priority. LOG_EMERG (0) is the most critical. LOG_DEBUG (7) is the least important. This ordering is used by setlogmask() to filter which messages actually get logged.

5. The Priority Value: Combining Facility and Level

In syslog, the priority passed to syslog() is actually a combination of the facility and the severity level. The C library provides the LOG_MAKEPRI(facility, level) macro to combine them, but you usually just OR them together:

/* Combine facility and level with bitwise OR */
syslog(LOG_DAEMON | LOG_ERR,  "Connection failed: %m");
syslog(LOG_AUTH   | LOG_INFO, "User %s logged in", username);
syslog(LOG_LOCAL0 | LOG_DEBUG,"Processing packet %d", seq);

/* If you only specify a level (no facility), the facility
 * set in openlog() is used automatically: */
openlog("myapp", LOG_PID, LOG_DAEMON);
syslog(LOG_ERR,  "This uses LOG_DAEMON facility automatically");
syslog(LOG_INFO, "So does this");

The priority integer is encoded as: priority = (facility << 3) | level. For example, LOG_DAEMON|LOG_ERR = (3<<3)|3 = 27.

๐Ÿ’ป Code Example 1: Exploring syslog Facilities and Priorities
#include <stdio.h>
#include <stdlib.h>
#include <syslog.h>
#include <unistd.h>

/*
 * Demonstrates syslog facilities and all priority levels.
 * After running, check: sudo tail -f /var/log/syslog
 */
int main(void)
{
    /* Open syslog connection with program name "syslog_demo",
     * include PID in messages, use LOG_DAEMON facility */
    openlog("syslog_demo", LOG_PID | LOG_PERROR, LOG_DAEMON);

    /* LOG_PERROR also prints to stderr โ€” useful during testing */

    printf("Sending messages to syslog. PID=%d\n", getpid());
    printf("Check: sudo tail -f /var/log/syslog\n\n");

    /* Send one message at each priority level */
    syslog(LOG_EMERG,   "EMERG:   System unusable โ€” test message");
    syslog(LOG_ALERT,   "ALERT:   Action required immediately");
    syslog(LOG_CRIT,    "CRIT:    Critical condition");
    syslog(LOG_ERR,     "ERR:     Error condition โ€” errno: %m");
    syslog(LOG_WARNING, "WARNING: Warning condition");
    syslog(LOG_NOTICE,  "NOTICE:  Normal but significant");
    syslog(LOG_INFO,    "INFO:    Informational message");
    syslog(LOG_DEBUG,   "DEBUG:   Debug details, process=%d", getpid());

    /* Send messages with different facilities */
    syslog(LOG_AUTH   | LOG_INFO,  "AUTH:    Authentication message");
    syslog(LOG_CRON   | LOG_INFO,  "CRON:    Cron-type message");
    syslog(LOG_LOCAL0 | LOG_INFO,  "LOCAL0:  Custom app message");
    syslog(LOG_LOCAL7 | LOG_DEBUG, "LOCAL7:  Custom debug message");

    /* Demonstrate %m โ€” automatic strerror(errno) expansion */
    /* Trigger an error first */
    FILE *f = fopen("/nonexistent/path/file.txt", "r");
    if (!f) {
        /* %m is a syslog extension that inserts strerror(errno) */
        syslog(LOG_ERR, "Failed to open file: %m");
    }

    closelog();

    printf("Done. Check /var/log/syslog for messages.\n");
    return 0;
}
gcc -o syslog_demo syslog_demo.c
./syslog_demo
sudo grep "syslog_demo" /var/log/syslog | tail -20

๐Ÿ’ป Code Example 2: Reading /proc/kmsg โ€” Kernel Log Messages

Kernel messages (from printk()) also enter syslog via a different path: /proc/kmsg. This example shows how to read kernel log messages directly.

#include <stdio.h>
#include <stdlib.h>
#include <fcntl.h>
#include <unistd.h>
#include <string.h>

/*
 * Kernel log priority levels (same as syslog levels)
 * In /proc/kmsg, each line starts with <N> where N is the level
 */
const char *kernel_level_name(int level)
{
    switch (level) {
        case 0: return "EMERG";
        case 1: return "ALERT";
        case 2: return "CRIT";
        case 3: return "ERR";
        case 4: return "WARNING";
        case 5: return "NOTICE";
        case 6: return "INFO";
        case 7: return "DEBUG";
        default: return "UNKNOWN";
    }
}

/*
 * Read and print kernel log messages from /proc/kmsg
 * Must run as root.
 *
 * Alternative: use dmesg command or the klogctl() syscall.
 */
int main(void)
{
    int fd;
    char buf[1024];
    ssize_t n;

    fd = open("/proc/kmsg", O_RDONLY | O_NONBLOCK);
    if (fd == -1) {
        perror("open /proc/kmsg (need root?)");
        return 1;
    }

    printf("Kernel log messages (reading /proc/kmsg):\n");
    printf("%-10s %s\n", "LEVEL", "MESSAGE");
    printf("%-10s %s\n", "-----", "-------");

    int count = 0;
    while (count < 20) {  /* Read up to 20 messages */
        n = read(fd, buf, sizeof(buf) - 1);
        if (n <= 0) break;

        buf[n] = '\0';

        /* Parse: <level>message text */
        char *p = buf;
        while (*p) {
            if (*p == '<') {
                int level = *(p+1) - '0';
                char *msg_start = strchr(p, '>');
                if (msg_start) {
                    msg_start++;
                    char *newline = strchr(msg_start, '\n');
                    if (newline) *newline = '\0';
                    printf("%-10s %s\n", kernel_level_name(level), msg_start);
                    count++;
                    p = newline ? newline + 1 : p + 1;
                } else p++;
            } else p++;
        }
    }

    close(fd);

    printf("\nNote: Use 'dmesg' for easier kernel log access\n");
    printf("Or: sudo journalctl -k  (systemd systems)\n");
    return 0;
}
gcc -o read_kmsg read_kmsg.c
sudo ./read_kmsg

# Alternative โ€” show kernel messages with dmesg:
dmesg | tail -20
dmesg --level=err,warn   # Only errors and warnings
dmesg -T                 # With human-readable timestamps

๐ŸŽฏ Interview Questions โ€” syslog Overview
Q1. Why can’t daemons use printf() for logging?
Answer: Daemons redirect stdin, stdout, and stderr to /dev/null during initialization. Any output to printf() (which writes to stdout, FD 1) is silently discarded by /dev/null. Additionally, since a daemon has no terminal, there is nobody watching for output. syslog() routes log messages to files on disk where they can be reviewed later.
Q2. What is /dev/log and what role does it play in syslog?
Answer: /dev/log is a Unix domain socket of type SOCK_DGRAM. When a process calls syslog(), the C library formats the message and writes it as a datagram to /dev/log. The syslogd daemon is permanently listening on this socket and reads each message as it arrives, then routes it according to /etc/syslog.conf rules.
Q3. What is the difference between a syslog facility and a syslog priority?
Answer: The facility identifies the category/subsystem that generated the message (LOG_DAEMON, LOG_AUTH, LOG_KERN, etc.). The priority (severity) indicates how serious the message is (LOG_EMERG=0 being most critical, LOG_DEBUG=7 being least). Both are combined into a single integer when calling syslog(): e.g., LOG_DAEMON|LOG_ERR.
Q4. What does the %m format specifier in syslog() do?
Answer: %m is a syslog-specific format specifier that is automatically replaced with the string returned by strerror(errno) โ€” the human-readable description of the current error. It is equivalent to writing “…: %s”, strerror(errno) but more concise. Example: syslog(LOG_ERR, “Cannot open file: %m”); outputs “Cannot open file: No such file or directory”.
Q5. What are LOG_LOCAL0 through LOG_LOCAL7 used for?
Answer: These 8 facilities are reserved for local use โ€” site-specific or application-specific logging. A system administrator can configure /etc/syslog.conf to route LOG_LOCAL0 messages to one file and LOG_LOCAL3 messages to another, allowing different custom applications to have their own separate log files without interfering with system daemon logs.
Q6. How do kernel messages reach syslogd?
Answer: The Linux kernel uses printk() internally to generate log messages. These are stored in a kernel ring buffer accessible via /proc/kmsg. The syslogd daemon reads from /proc/kmsg (in addition to /dev/log) and routes kernel messages (facility LOG_KERN) to the appropriate log file, typically /var/log/kern.log.

Chapter Summary

syslog is the standard daemon logging system. Messages flow from daemon โ†’ /dev/log (Unix domain socket) โ†’ syslogd โ†’ log files. Each message carries a facility (source category) and a priority (severity level 0-7). Facilities like LOG_DAEMON and LOG_LOCAL0-7 are used by application daemons. LOG_DEBUG through LOG_EMERG define the importance of the message.

โ† SIGHUP Reinitialize Next: syslog API โ†’

Leave a Reply

Your email address will not be published. Required fields are marked *