Setting up remote logging with rsyslogd and haproxy on RHEL8

The highly available haproxy/keepalived that I am setting up needs to log all connections. I've decided to set this up using a rsyslogd on a remote machine.

The system is currently running in OpenStack on a private network with a bastion for external access. There's also an externally accessible vip for keepalived for access to the haproxy instances.

Internet -|
          |
          |
          |-- Bastion------------------------|
          |                                  |
          |-- keepalived vip                 |
              |                              |
              |-- HAProxy Main --------------|
              |                              |
              |-- HaProxy Spare -------------|
                                             |
                                             |
                  Backend 01 ----------------|
                                             |
                  Backend 02 ----------------|
                                             |
                  Backend 03 ----------------|

This is a proof of concept design for a setup that may eventually be used to manage OpenShift/kubernetes ingress.

ryslogd actually gets installed on all of these machines as part of the RHEL8 base, but it's not configured for remote logging.

UDP not TCP

My first mistake here was to assume haproxy would use TCP for logging. It actually uses UDP. I spent a lot of time wondering what was going on as a result.

Step 1 is to enable udp input to rsyslogd on the bastion by uncommenting some lines in /etc/rsyslogd.conf:

These two lines:

#module(load="imudp") # needs to be done just once
#input(type="imudp" port="514")

becomes

module(load="imudp") # needs to be done just once
input(type="imudp" port="514")

In ansible something like this should do the trick:

    - name: 360 - Enable rsyslogd imudp module
      replace:
        path: /etc/rsylog.conf
        regexp: '^#(module\(load="imudp"\) # needs to be done just once)'
        replace: '\1'
    - name: 370 - Enable imudp input
      replace:
        path: /etc/rsylog.conf
        regexp: '^#(input\(type="imudp" port="514"\))'
        replace: '\1'

Restart syslogd

service rsyslog restart

Next to test that ryslogd is accepting messages over udp. You can just use nc for this, but I did have to install it first.

dnf install nmap-ncat

echo "Test message" | nc -u localhost 514

Check to see the message has actually been logged:

# grep "Test message" /var/log/messages
Mar  9 20:01:38 Test message 

Now test from one of the haproxy machines

[root@ha-proxy-main ~]# echo "Test message from haproxy main" | nc -u xx.xx.xx.xx 514

Note: substitute the actual IP address of your rsyslog server for xx.xx.xx.xx.

In OpenStack you'll need a security group rule allowing access to port 514. Best to restrict that rule to the security group that the haproxy machines are in. If you're using a host based firewall (iptables, firewall-daemon, ufw etc) you'll need to allow incoming traffic on port 514.

Check that the message arrived:

[root@ha-bastion ~]# grep "Test message" /var/log/messages
Mar  9 20:01:38 Test message 
Mar  9 20:30:39 Test message from haproxy main

Here we see the first test message and the a new message from the haproxy main server. Everything seems to be working.

Now to configure haproxy to log connections.

This is basically as simple as adding to the global options

log xx.xx.xx.xx   local0

This is going to generate a lot of log entries. I decided to at least log them to a separate file. I did this by adding a config file to the /etc/rsyslog.d directory and restarting ryslog.

/etc/rsyslog.d/50-haproxy.conf

if $programname == 'haproxy' then action(type="omfile" file="/var/log/haproxy.log")

& stop