03- Patroni Cluster on Rocky Linux9.5 - Setup HAPRoxy and KeepAlived

In this part, we are going to install and configure HAProxy & Keepalived. Our nodes:

haproxy01: 192.168.1.141

haproxy02: 192.168.1.142

VIP for PostgreSQL access: 192.168.1.140

HAProxy:

Install HAProxy on Both Nodes (haproxy01 & haproxy02)

sudo su
dnf install -y haproxy

Configure HAProxy. HAProxy listens to the request on port 5432 and checks the primary backend node and then forwards the request to its primary backend node. It checks the backend nodes' health by using port 8008.

 mv /etc/haproxy/haproxy.cfg /etc/haproxy/oldhaproxy.cfg
nano /etc/haproxy/haproxy.cfg

#Add these into haproxy.cfg. Only essential configuration for PostgreSQL + Patroni HAProxy setup
#--------------------------#
#     Global settings      #
#--------------------------#
global
    log         127.0.0.1 local2
    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     4000
    user        haproxy
    group       haproxy
    daemon
    stats socket /var/lib/haproxy/stats
    ssl-default-bind-ciphers PROFILE=SYSTEM
    ssl-default-server-ciphers PROFILE=SYSTEM

#--------------------------#
#     Default settings     #
#--------------------------#
defaults
    mode                    tcp
    log                     global
    option                  tcplog
    option                  dontlognull
    option http-server-close
    option                  redispatch
    retries                 3
    timeout http-request    10s
    timeout queue           1m
    timeout connect         10s
    timeout client          1m
    timeout server          1m
    timeout http-keep-alive 10s
    timeout check           10s
    maxconn                 3000

#--------------------------#
#      Patroni Cluster     #
#--------------------------#
frontend postgres_frontend
    bind *:5432
    timeout client 30s
    mode tcp
    option tcplog
    default_backend postgres_backend

backend postgres_backend
    mode tcp
    option tcp-check
    option httpchk GET /primary
    http-check expect status 200
    server postgres01 192.168.1.144:5432 port 8008 check
    server postgres02 192.168.1.145:5432 port 8008 check
    server postgres03 192.168.1.146:5432 port 8008 check

Allows HAProxy to initiate outbound network connections to any address or port

#Enable the proper SELinux boolean
setsebool -P haproxy_connect_any=1

#Verify
getsebool haproxy_connect_any

Check if config file is valid & enable HAProxy service 

#Validate the config first
haproxy -c -f /etc/haproxy/haproxy.cfg

#Enable HAProxy
systemctl enable --now haproxy
systemctl status haproxy

postgresql-01 is the Leader, and Patroni returns 200 OK for /primary on that node.

postgresql-02 and postgresql-03 are Replicas, and they correctly return 503 Service Unavailable, which is expected from non-leader nodes.

Keepalived:

Install Keepalived on Both Nodes (haproxy01 & haproxy02)

dnf install -y keepalived iproute

Create the HAProxy Health Check Script

nano /etc/keepalived/check_haproxy.sh

#Add the following into "check_haproxy.sh"
#!/bin/bash
PORT=5432

if ! pidof haproxy > /dev/null; then
    echo "HAProxy is not running"
    exit 1
fi

if ! ss -ltn | grep -q ":${PORT}"; then
    echo "HAProxy is not listening on port ${PORT}"
    exit 2
fi

exit 0

Make it executable and assign proper ownership:

useradd -r -s /bin/false keepalived_script
chmod +x /etc/keepalived/check_haproxy.sh
chown keepalived_script:keepalived_script /etc/keepalived/check_haproxy.sh
chmod 700 /etc/keepalived/check_haproxy.sh

Configure Keepalived on HAProxy01(MASTER)

Make sure you change your interface id accordingly

mv  /etc/keepalived/keepalived.conf /etc/keepalived/oldkeepalived.conf
nano /etc/keepalived/keepalived.conf

#Add the following into keepalived.conf
global_defs {
    enable_script_security
    script_user keepalived_script
}

vrrp_script check_haproxy {
    script "/etc/keepalived/check_haproxy.sh"
    interval 2
    fall 3
    rise 2
}

vrrp_instance VI_1 {
    state MASTER
    interface ens160  #Change this to your actual NIC (e.g., ens18, enp1s0)
    virtual_router_id 51
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 34Istanb
    }
    virtual_ipaddress {
        192.168.1.140
    }
    track_script {
        check_haproxy
    }
}

Configure Keepalived on HAProxy02(BACKUP)

Same config as above, but change (state BACKUP, priority 90 and interface whateever you use)

global_defs {
    enable_script_security
    script_user keepalived_script
}

vrrp_script check_haproxy {
    script "/etc/keepalived/check_haproxy.sh"
    interval 2
    fall 3
    rise 2
}

vrrp_instance VI_1 {
    state BACKUP
    interface ens160  #Change this to your actual NIC (e.g., ens18, enp1s0)
    virtual_router_id 51
    priority 90
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 34Istanb
    }
    virtual_ipaddress {
        192.168.1.140
    }
    track_script {
        check_haproxy
    }
}

Allow keepalived on SELinux

setsebool -P keepalived_connect_any=1

Enable and Start Keepalived

systemctl enable --now keepalived
journalctl -u keepalived -f

haproxy01 is currently MASTER and broadcasting gratuitous ARPs for the VIP 192.168.1.140

haproxy02 is in BACKUP state and ready to take over if haproxy01 fails.

You can go ahead and follow the test on this article section name "Data Integrity and Replication Test with PGAdmin".