Setting up remote logging with rsyslogd and haproxy on RHEL8
The highly available haproxy/keepalived that I am setting up needs to log all
connections. I've decided to set this up using a rsyslogd on a remote machine.
The system is currently running in OpenStack on a private network with a bastion
for external access. There's also an externally accessible vip for keepalived
for access to the haproxy instances.
Internet -|
|
|
|-- Bastion------------------------|
| |
|-- keepalived vip |
| |
|-- HAProxy Main --------------|
| |
|-- HaProxy Spare -------------|
|
|
Backend 01 ----------------|
|
Backend 02 ----------------|
|
Backend 03 ----------------|
This is a proof of concept design for a setup that may eventually be used to
manage OpenShift/kubernetes ingress.
ryslogd actually gets installed on all of these machines as part of the RHEL8
base, but it's not configured for remote logging.
UDP not TCP
My first mistake here was to assume haproxy would use TCP for logging. It
actually uses UDP. I spent a lot of time wondering what was going on as a
result.
Step 1 is to enable udp input to rsyslogd on the bastion by uncommenting some
lines in /etc/rsyslogd.conf:
These two lines:
#module(load="imudp") # needs to be done just once
#input(type="imudp" port="514")
becomes
module(load="imudp") # needs to be done just once
input(type="imudp" port="514")
In ansible something like this should do the trick:
- name: 360 - Enable rsyslogd imudp module
replace:
path: /etc/rsylog.conf
regexp: '^#(module\(load="imudp"\) # needs to be done just once)'
replace: '\1'
- name: 370 - Enable imudp input
replace:
path: /etc/rsylog.conf
regexp: '^#(input\(type="imudp" port="514"\))'
replace: '\1'
Restart syslogd
Next to test that ryslogd is accepting messages over udp. You can just use nc
for this, but I did have to install it first.
echo "Test message" | nc -u localhost 514
Check to see the message has actually been logged:
# grep "Test message" /var/log/messages
Mar 9 20:01:38 Test message
Now test from one of the haproxy machines
[root@ha-proxy-main ~]# echo "Test message from haproxy main" | nc -u xx.xx.xx.xx 514
Note: substitute the actual IP address of your rsyslog server for xx.xx.xx.xx.
In OpenStack you'll need a security group rule allowing access to port 514. Best
to restrict that rule to the security group that the haproxy machines are in. If
you're using a host based firewall (iptables, firewall-daemon, ufw etc) you'll
need to allow incoming traffic on port 514.
Check that the message arrived:
[root@ha-bastion ~]# grep "Test message" /var/log/messages
Mar 9 20:01:38 Test message
Mar 9 20:30:39 Test message from haproxy main
Here we see the first test message and the a new message from the haproxy main
server. Everything seems to be working.
Now to configure haproxy to log connections.
This is basically as simple as adding to the global options
This is going to generate a lot of log entries. I decided to at least log them
to a separate file. I did this by adding a config file to the /etc/rsyslog.d
directory and restarting ryslog.
/etc/rsyslog.d/50-haproxy.conf
if $programname == 'haproxy' then action(type="omfile" file="/var/log/haproxy.log")
& stop