State of the art
System logs are generated by a system logger daemon, usually syslogd. It should be started at boot, and logging management is controlled by a configuration file, usually
/etc/syslog.conf. This configuration file defines what system messages will go to which log: messages, auth, or syslog. I say “usually”, because each flavor of Linux and UNIX have evolved over time, and they now do manage their logs differently. Not to mention NAS, routers and embedded systems that usually don’t even have a configuration file and use only messages, when it’s enabled.
Because there is no such organism that federate or unify OS’ system logs, and because rules are changing over time, it’s quite hard to explain which system log should contain what message. We cannot even rely on best practices in the matter because each Linux distribution team have their own view. And there are literally hundreds of Linux distributions.
At best, we can look at some major systems manual pages and draw a generic picture of what exists.
A best practice in describing UX system files, structure and logs, is to chose a specific system and then only work on this one.
Syslog Protocol: RFC-5424
Hopefully, for system messages, the Syslog Protocol is defined by RFC-5424. Although some systems may chose not to rely on it (IBM AIX for instance), it gives basic definitions that are followed by hundreds of Linux distributions. One can identify 2 important variables:
- facility: source service of the message. Each log message is associated with an application subsystem (called “facility” in the documentation):
|syslog||messages generated internally by syslogd|
|lpr||line printer subsystem|
- level: priority level of the message. Each message is also associated with a priority level. Here is the exhaustive list in decreasing order:
|0||Emergency||system is unusable / panic|
|1||Alert||action must be taken immediately|
|5||Notice||normal but significant condition|
The historic logging daemon is called syslogd. It follows the rules from RFC-5424 (replaces RFC-3164) for facility and level declaration. It is itself an evolution from the standard Berkeley utility program. This daemon listen and writes logs locally to the system, however a new daemon is taking more and more importance in the scene.
RSYSLOG is the rocket-fast system for log processing:
Rsyslog is an enhanced multi-threaded syslogd with a focus on security and reliability. Among others, it offers support for on-demand disk buffering, reliable syslog over TCP, SSL, TLS and RELP, writing to databases (MySQL and PostgreSQL), email alerting. It is a drop-in replacement for syslogd.
It has to be noted that some log messages are configured to the /dev/console file, which actually print them on the console for every logged in users at the time the message is sent.
Log naming rules
Unfortunately, there is no RFC that defines what should go in which logs, nor is there any rules federating the log names. It’s set according to each system developer teams, as developed in the next section.
However, it would make sense to put the kernel messages in some log named kern.log, and authentication messages into a log named auth.log, wouldn’t it? But then, what do you use
/var/log/messages for? Ubuntu decided to dump it, because it was a potpourri of system messages at different levels.
Whereas no single team for the Linux/UNIX world could come to an agreement on the file names, at least we could develop on what facilities are. The following facility descriptions are referenced by the Ubuntu community. Other communities may have other opinions.
The Authorization Log tracks usage of authorization systems, the mechanisms for authorizing users which prompt for user passwords, such as
- the Pluggable Authentication Module (PAM) system
- the sudo command
- SSH remote logins to sshd
- FTP logins to ftpd
- SFTP logins to sftpd
- and so on.
The Authorization Log file may be accessed at
/var/log/auth.log. This log is useful for learning about user logins and usage of the sudo command.
A daemon is a program that runs in the background, generally without human intervention, performing some operation important to the proper running of your system. A service is a daemon.
The daemon log at
/var/log/daemon.log contains information about running system and application daemons such as
- the Gnome Display Manager daemon gdm
- the Bluetooth HCI daemon hcid
- the MySQL database daemon mysqld
- the DHCP client daemon dhclient
- any other system daemon you may install…
This can help you trouble-shoot problems with a particular daemon.
The debug log at
/var/log/debug provides detailed debug messages from the system and applications which log to syslogd at the DEBUG level.
Please note how the Ubuntu team chose not to name it debug.log…
The kernel log at
/var/log/kern.log provides a detailed log of messages from the Linux kernel. These messages may prove useful for trouble-shooting a new or custom-built kernel, for example.
Kernel Ring Buffer (binary log)
The kernel ring buffer is not really a log file per se, but rather an area in the running kernel you can query for kernel bootup messages via the dmesg utility. To see the messages, use this:
Or to search for lines that mention the Plug & Play system, for example, use grep like this:
dmesg | grep pnp | more
[ 0.667083] pnp: PnP ACPI init [ 0.690161] pnp 00:02: Plug and Play ACPI device, IDs PNP0b00 (active) [ 0.690199] pnp 00:03: Plug and Play ACPI device, IDs PNP0f13 (active) [ 0.690228] pnp 00:04: Plug and Play ACPI device, IDs PNP0303 PNP030b (active)
By default, the system initialization script
/etc/init.d/bootmisc.sh sends all bootup messages to the binary file
/var/log/dmesg as well. You can view and search this file the usual way.
The system log on Ubuntu typically contains the greatest deal of information by default about your system. It is located at
/var/log/syslog, and may contain information other logs do not. Consult the System Log when you can’t locate the desired log information in another log. It also contains everything that used to be in
As you can see, the potpourri log just changed its name in Ubuntu (since natty 11.04, 2011).
A utility exists on many systems, that allow scripts to log messages in system logs. Depending on parameters used, the message will go to either log.
The default priority is user.notice
- Send notice messages to
logger "Hello World"
logger -t scriptname "Hello World"
Mar 27 08:50:51 localhost ubuntu: Hello World Mar 27 08:51:02 localhost scriptname: Hello World
- Send error messages to /var/log/auth.log:
logger -t scriptname-p auth.error"Hello World"
Mar 27 11:43:09 localhost scriptname: Hello World
- Append a kernel critical message intokern.log:
logger -p kern.crit " kernel critical message example!"