syslog-ng: Setting up a Remote Logging Server and Client

For anyone who wants a secure server, there are thousands of guides and opinions on how to protect it best. One thing the system must certainly do is to record all
the operations that are carried out; this is so that, on the one hand, you have useful information for solving any malfunctions and, on the other, you have a control tool against attacks or intrusion attempts.

Syslog-ng will monitor our servers locally and, with the right configuration, will duplicate the logs onto a central remote server.

So, if some reckless good-for-nothing decided to launch an attack against one of our servers, they could certainly hide their tracks well, deleting the signs of their activity from the log files, but they would have to do the same on a second server! It’s hard to escape a well-devised attack, but at least let’s make their life difficult!

We have already revealed our choice regarding the log server we’ll use for our purpose, namely syslog-ng, both for its excellent granularity in defining filtering rules and for the availability of more documentation.

Let’s install our package on each machine:

apt-get install syslog-ng

this will remove rsyslog. So be careful!!!

Syslog-ng works according to a simple scheme: the log from a given source is filtered and sent to a destination. If this doesn’t seem complicated to you, reading the configuration file won’t be difficult at all.

So here’s what I did: on what will be the central server collecting the logs from the other servers, I added a source coming from the network, I used the same filters as the local system, and I directed everything to files placed in separate folders for each client server.

Let’s call the remote log-storage server “central.example.com” and give the various servers the names “client1.example.com”, “client2.example.com” … “client4.example.com” and so on.

Let’s open the configuration file located in the /etc/syslog-ng/ folder on the “central” server and add the highlighted lines. There shouldn’t be any major trouble understanding them:

...

# First, set some global options.
options { chain_hostnames(off); flush_lines(0); use_dns(no); use_fqdn(no);
	  owner("root"); group("adm"); perm(0640); stats_freq(0);
	  bad_hostname("^gconfd$");
	  keep_hostname(yes);
	  chain_hostnames(on);
	  create_dirs(yes);
};

...

source s_net { tcp( ip(192.168.1.50) port(1000) keep-alive (yes)); };

...

destination net_auth { file("/var/log/$HOST/$YEAR/$MONTH/$DAY/auth.log"); };
destination net_cron { file("/var/log/$HOST/$YEAR/$MONTH/$DAY/cron.log"); };
destination net_daemon { file("/var/log/$HOST/$YEAR/$MONTH/$DAY/daemon.log"); };
destination net_kern { file("/var/log/$HOST/$YEAR/$MONTH/$DAY/kern.log"); };
destination net_lpr { file("/var/log/$HOST/$YEAR/$MONTH/$DAY/lpr.log"); };
destination net_mail { file("/var/log/$HOST/$YEAR/$MONTH/$DAY/mail.log"); };
destination net_syslog { file("/var/log/$HOST/$YEAR/$MONTH/$DAY/syslog"); };
destination net_user { file("/var/log/$HOST/$YEAR/$MONTH/$DAY/user.log"); };
destination net_uucp { file("/var/log/$HOST/$YEAR/$MONTH/$DAY/uucp.log"); };
destination net_mailinfo { file("/var/log/$HOST/$YEAR/$MONTH/$DAY/mail.info"); };
destination net_mailwarn { file("/var/log/$HOST/$YEAR/$MONTH/$DAY/mail.warn"); };
destination net_mailerr { file("/var/log/$HOST/$YEAR/$MONTH/$DAY/mail.err"); };
destination net_newscrit { file("/var/log/$HOST/$YEAR/$MONTH/$DAY/news/news.crit"); };
destination net_newserr { file("/var/log/$HOST/$YEAR/$MONTH/$DAY/news/news.err"); };
destination net_newsnotice { file("/var/log/$HOST/$YEAR/$MONTH/$DAY/news/news.notice"); };
destination net_debug { file("/var/log/$HOST/$YEAR/$MONTH/$DAY/debug"); };
destination net_error { file("/var/log/$HOST/$YEAR/$MONTH/$DAY/error"); };
destination net_messages { file("/var/log/$HOST/$YEAR/$MONTH/$DAY/messages"); };
destination net_ppp { file("/var/log$HOST/$YEAR/$MONTH/$DAY/ppp.log"); };

...

log { source(s_net); filter(f_auth); destination(net_auth); };
log { source(s_net); filter(f_cron); destination(net_cron); };
log { source(s_net); filter(f_daemon); destination(net_daemon); };
log { source(s_net); filter(f_kern); destination(net_kern); };
log { source(s_net); filter(f_lpr); destination(net_lpr); };
log { source(s_net); filter(f_syslog3); destination(net_syslog); };
log { source(s_net); filter(f_user); destination(net_user); };
log { source(s_net); filter(f_uucp); destination(net_uucp); };
log { source(s_net); filter(f_mail); destination(net_mail); };
log { source(s_net); filter(f_news); filter(f_crit); destination(net_newscrit); };
log { source(s_net); filter(f_news); filter(f_err); destination(net_newserr); };
log { source(s_net); filter(f_news); filter(f_notice); destination(net_newsnotice); };
log { source(s_net); filter(f_debug); destination(net_debug); };
log { source(s_net); filter(f_error); destination(net_error); };
log { source(s_net); filter(f_messages); destination(net_messages); };
log { source(s_net); filter(f_console); destination(d_console_all); destination(d_xconsole); };
log { source(s_net); filter(f_crit); destination(d_console); };

...

In the first lines of general options, create_dirs is important!

I didn’t add any particular filters; I referred to those already present. Furthermore, I used syslog-ng’s own macros to categorize the log files into subfolders ($HOST $YEAR $MONTH $DAY). Take a look also at a couple of particular destinations, namely: destination(d_console_all); destination(d_xconsole) — using these parameters, the clients’ console messages are directed to the server’s console in real time!

Once done, restart the daemon so it can start listening on port 1000, waiting for the clients’ logs.

/etc/init.d/syslog-ng restart

Let’s move on to the “clients”. Ingeniously simple, these will just have to route the logs to the remote server. So we’ll add to the clients’ configuration file a “log” directive that routes the same messages from the local sources to a remote destination.

Add two lines like these to the configuration file of each client server:

...

destination d_net { tcp( "192.168.1.50" port(1000) log_fifo_size(1000) ); };
# All messages send to a remote site
#
log { source(s_src); destination(d_net); };

...

Restart all the client daemons so they open the connection to our centralized log server:

/etc/init.d/syslog-ng restart

If all went well, you’ll notice that in the /var/log folder on the “central.example.com” server a directory tree like this one has been created

# tree /var/log/client1
/var/log/client1
└── 2013
    └── 12
        ├── 03
        │   ├── auth.log
        │   ├── cron.log
        │   ├── daemon.log
        │   ├── debug
        │   ├── error
        │   ├── kern.log
        │   ├── mail.log
        │   ├── messages
        │   ├── syslog
        │   └── user.log
        ├── 04
        │   ├── auth.log
        │   ├── cron.log
        │   ├── daemon.log
        │   ├── debug
        │   ├── error
        │   ├── kern.log
        │   ├── mail.log
        │   ├── messages
        │   ├── syslog
        │   └── user.log
        └── 05
            ├── auth.log
            ├── cron.log
            ├── daemon.log
            ├── debug
            ├── error
            ├── kern.log
            ├── mail.log
            ├── messages
            ├── syslog
            └── user.log

5 directories, 30 files

NOTE: all the machines are placed under a VPN, which is why I didn’t bother to encrypt the log exchange again. Otherwise, we would have had to think not only about using a TLS protocol to encrypt the message but also about a mutual authentication system between client and server based on x509 certificates.

WARNING: the “central.example.com” machine must be particularly well protected against a possible attack. Let me explain: isn’t it often the case that the root password of the client servers is the same as that of the central server? All this work would be completely undone by such crude mistakes!!!