Having logs but not reading them is maybe the most common computation power waste since invention on population counting (census). It’s always better to solve the problem than to silence it! That applies to both: computers and people! 😉
Why do we have logs? Aaah… of course, if something doesn’t work so we can look it up and maybe find the error. What about security? What’s that? Yes believe it or not logging is an important part of protecting your system. In case something happen, malfunction, attack, whatever, logs should help you to find out what happen and fix the whole in the system. One of my favorite saying is: “Making mistakes is human, repeating them is stupid”.
It is also a good idea to have a separate logging machine. Especially if your system is attacked by a hacker, he will probably at try to cover his steps. Logs are the first thing to go after. If your logs are sent immediately to a remote server the hacker will have a much more difficult job. (I am assuming that you are not leaving passphrase unprotected SSH keys to your syslog machine on every system in your network.)
System Loggers
Why did I choose SYSLOG-NG and RSYSLOG system loggers? Easily: SYSLOG-NG is the default choice on Gentoo (my desktop) and RSYSLOG is the default choice for CentOS (my server). Yeah I know there is nothing like a default choice for Gentoo. Gentoo gives you all the choices and all the options. But since its mentioned in the installation manual as an example I am taking that as a “default” choice 🙂 Also if you Google a bit you will find out that SYSLOG-NG is combining the functionality of SYSLOG and METALOG plus adding a bit more. So it’s the best choice.
If you have Debian, RedHat, CentOS, … somewhere you will find out that the default system logger there is RSYSLOG. Again after a bit Googling you will find out that the reason why those distribution use RSYSLOG instead of SYSLOG-NG is because it as much awesome from the functionality point-of-view but does not have licensing issues. Those were anyway in the meantime solved, see: http://bazsi.blogs.balabit.com/2010/07/syslog-ng-contributions-redefined.
But lets make it more interesting and keep the “default” choices in the system. There is a 3rd system in the network. A NAS server with build in proprietary system logger and system logger server. Both are configurable over a web interface.
Server Part
The default port for system logging is 514 and it can be done over both, TCP and UDP. The difference is that TCP is reliable and UDP is unreliable. Basically real-time applications, application where you can loose a packet here and a packet there -> UDP, if you want reliability over performance -> TCP. I chose performance: UDP.
Since server part is via proprietary web interface it easy. For the mentioned loggers you can look up:
Client Part
SYSLOG-NG
in /etc/syslog-ng/syslog-ng.conf (Gentoo) we have the source:
source src { unix-stream("/dev/log"); internal(); };
we define the destination (the remote server):
destination syslog_server {
       #tcp("hostname.local" port(514));
       #udp("hostname.local" port(514));
       tcp("xxx.xxx.xxx.xxx" port(514));
       #udp("xxx.xxx.xxx.xxx" port(514));
};
Should be self explanatory. You can use both, IP or hostnames. Just remember that if in case you use hostname and your DNS server stops working you are not logging to the remote server!
Finally we connect the source to the destination:
log { source(src); destination(syslog_server); };
RSYSLOG
Add one of the following depending on what protocol you want to use (I am using UDP):
# Provides UDP forwarding. The IP is the server's IP address
*.* @xxx.xxx.xxx.xxx:514
# Provides TCP forwarding. But the current server runs on UDP
#*.* @@xxx.xxx.xxx.xxx:514
RSYSLOG was on my CentOS machine not logging remotely when I used hostnames. Only IPs. Didn’t figured out why. The source and filter are in RSYSLOG represented by the “*.*”. Meaning everything without using a filter.
Additionally I set up an email notification if the log severity is critical so that should force me to read logs 🙂
Hostnames
It is a good idea to setup static hostnames on each host. The default once are usually something like: localhost, localhost.localdomain, etc. If all your computers will send logs using such hostname it will not significantly contribute to readability of such logs. In case you are using own DNS server and it stops working you have the same problem. Hostname can be change in:
- Gentoo: /etc/conf.d/hostname
- CentOS: /etc/sysconfig/network
- Others: use Google 🙂