Jack Moore

Email: jack(at)jmoore53.com
Project Updates

Syslog-NG with Elasticsearch

14 Nov 2020 » system configuration, sysadmin, homelab, server build, monitoring

First I needed a firewall, then I needed Monitoring!

Setting up the syslog-ng server:

sudo apt install syslog-ng

Configuration

I left in all the other boilerplate configuration options for s_src, but made some tweaks and added the following:

# Kept the same
source s_src {
       system();
       internal();
};

# Added this source for remote servers/applicances
source s_net { tcp(ip(0.0.0.0) port(514) max-connections (5000)); udp(); };

# Added this destination for the server
destination d_syslog { file("/var/log/remotelogs/$HOST/syslog" owner("logstash") group("logstash") perm(0600) create_dirs(yes) dir_perm(0770)); };

# Added this destination for Elasticsearch
destination d_elasticsearch_http {
    elasticsearch-http(
        index("syslog-ng")
        type("")
        url("http://localhost:9200/_bulk")
        template("$(format-json --scope rfc5424 --scope dot-nv-pairs
        --rekey .* --shift 1 --scope nv-pairs
        --exclude DATE --key ISODATE @timestamp=${ISODATE})")
    );
};

# Added these to capture the logs
log { source(s_net); destination(d_syslog); };
log { source(s_net); destination(d_elasticsearch_http); };

This allowed me to configure syslog-ng with elastic search.

Logrotate (Incorrect and Correct Versions!)

Straight from the manpage, logrotate is designed to ease administration of systems that generate large numbers of log files. It allows automatic rotation, compression, removal, and mailing of log files. Each log file may be handled daily, weekly, monthly, or when it grows too large.

Insecure (and incorrect) log rotating via:

vim /etc/logrotate.d/remote
/var/log/remotelogs/*/*
{
  create 0755 root adm
  rotate 90
  daily
  missingok
  compress
  su root adm
}

This will rotate logs daily and keep them for 90 days. To test the config use:

logrotate -d --force /etc/logrotate.d/remote

… so as a bit of an update, the above configuration was incorrect. This configuration was filling up all the inodes on my logging/monitoring server causing massive issues with the disk. Inodes would fill up the disk, all applications would come to a hault requiring me to log on and remove all excess files.

The fix was simple, the new logrotate config, located at /etc/logrotate.d/remote, looked like the following:

/var/log/remotelogs/*.log
{
    create 0755 root adm
    rotate 3
    daily
    dateformat -%d%m%Y
    size 500M
    missingok
    compress
    su root adm
}

I also updated /etc/syslog-ng/syslog-ng.conf to look like:

# Remote Destination
#
#destination d_syslog { file("/var/log/remotelogs/${HOST}/${YEAR}_${MONTH}_${DAY}.log" owner("logstash") group("logstash") perm(0600) create_dirs(yes) dir_perm(0770)); };
destination d_syslog { file("/var/log/remotelogs/${HOST}.log" owner("logstash") group("logstash") perm(0770) create_dirs(yes) dir_perm(0770)); };

Some of the common scripts I used to check the size of folders and checking for failures look like the following:

# Get Size of files
# Modify --block size with 1/1M/1G for bytes/megabytes/gigabytes
du -csh --block-size=1 ./*

# Get failures except for...
sudo egrep "Failed|Failure" ./* | grep -v "10.0.1.45|hlvmg1"

Permissions

The one thing I had to allow was read and write to the /var/log/remotelogs/remote.ip.addr.here.

chmod 644 -R /var/log/remote/logs/*/*

This needs to be set to allow logstash to read the file on the host.

I also modified the docker-compose.yml file for logstash to point to a bindmount at:

- type: bind
  source: /var/log/remotelogs
  target: /var/log/remotelogs
  read_only: true

Setting up Clients

This is done for most clients in /etc/rsyslog.conf at the end to send logs over to the remote server via port 514:


*.* @10.0.0.X:514

Setting up Docker to go to syslog

Create the daemon.json file.

sudo touch /etc/docker/daemon.json

And in the file put:

{
  "log-driver": "syslog"
}

© Jack Moore