Episode - 019 HA Pi and Glusterfs
Background
So I have been running 4 Raspberry Pi 3B as docker hosts for my infrastructure for a few years, they run my internal DNS ( PiHole ) , And other services that I use like VPN.
I wanted to have all the docker volumes on an NFS and then get the hosts to use that as a mounted file system. This way I could move containers to different hosts and they could just pick up from where they left off as the data is in all one place.
This caused issues on the Older Raspberry Pi 3b, when all 4 pi where trying to write to the 1 Pi that I designated the NFS server, and every day it would crash or do some weird things. This pi was also the ingress pi for anything external using LinuxServer.io SWAG which was nginx reverse proxy with certbot and fail2ban thrown in for good measure. Pi 1 was also my wireguard entry. But when this pied died, I couldn't get to anything.
Easy fix, move the SWAG container and the Wireguard Container to another Pi. Problem solved for that ingress issue. But didnt fix my storage issue of wanting the Pi to have a central place for all docker volumes.
Setting up
To make sure I had a way in I decided to make my SWAG and Wireguard set up in HA ( High Availability ) using PCS , Pacemaker and corosync. I had 2 extra Pi laying about, so bought some SD cards for them, flashed raspberrypiOS on and started setting up my Pi HA cluster.After the usual set up pieces with my docker-pi ansible playbook I had to work out how to share the storage between these two Pi , so when i need to fail over or one Pi dies they have the same data on each.
At first I considered just using rsync and syncing primary to secondary, but I thought this would get messy and out of sync if I dont failover properly and uncomment the rsync command in crontab.
I then thought about work and what we use for shared storage, now we have NFS, and we also use Ceph for Openstack, but Gluster was also something that we used, and it reminded me of an article I read about gluster and SBC as a low cost NAS.
Gluster it is, It would be a perfect fit for this, as im using the set up in an active/passive set up, the active would always be written to and gluster would manage the replication to the other node, and when i failed over the same thing would happen then.
First Steps
After setting up the Pi with a static IP and installing docker and pulling my docker github which has all my compose files, the next thing was to set up PCS.
Set up hosts file /etc/hosts
Install packages sudo apt install corosync pacemaker pcs
Enable and start the pcsd services
sudo systemctl enable pcsd && sudo systemctl start pcsd
Creating the cluster :
root@rpi-05:~# pcs cluster auth
root@rpi-05:~# pcs host auth rpi-05 rpi-06
root@rpi-05:~# pcs cluster setup clustername rpi-05 rpi-06 --force
root@rpi-05:~# pcs cluster enable --all
root@rpi-05:~# pcs cluster start --all
Disabling STONITH pcs property set stonith-enabled=false
Setting up the resources
pcs resource create floating_ip ocf:heartbeat:IPaddr2 ip=192.168.1.7 cidr_netmask=24 nic=eth0 op monitor interval=60s
The 192.168.1.7 IP address i have used to act as the VIP or floating IP and assigned it to eth0. This way when I do my port forward on my router I point it to this instead of the Pi IP of 192.168.1.5 or 192.168.1.6
Thats it for PCS currently, after creating the resource it was running on one of the Pi, I can check using pcs status
root@rpi-05:~# pcs status
Cluster name: pi-cluster
Status of pacemakerd: 'Pacemaker is running' (last updated 2023-12-20 17:01:31Z)
Cluster Summary:
* Stack: corosync
* Current DC: rpi-06 (version 2.1.5-a3f44794f94) - partition with quorum
* Last updated: Wed Dec 20 17:01:32 2023
* Last change: Wed Dec 20 08:35:24 2023 by root via cibadmin on rpi-06
* 2 nodes configured
* 1 resource instance configured
Node List:
* Online: [ rpi-05 rpi-06 ]
Full List of Resources:
* floating_ip (ocf:heartbeat:IPaddr2): Started rpi-06
Daemon Status:
corosync: active/enabled
pacemaker: active/enabled
pcsd: active/enabled
Its currently running on rpi-06. Great success
Gluster setup
Next job was to set up Gluster. I ran sudo apt install glusterfs-server on both Pi's and thats step 1.
Then set up storage, This should be done using an external drive like a USB, so lsblk showed that the USB is labled as /dev/sdb on both so I ran the below on both: parted /dev/sdb mklabel gpt parted /dev/sdb mkpart primary 0% 100% mkfs.ext4 /dev/sdb1
This set the drive, now to make a mount point for the drive.
sudo mkdir /data now I can mount the drive to /data First I needed to get the UUID of the disk to put into /etc/fstab.
blkid showed UUID of the drive, in my case this is the output of that.
dev/mmcblk0p1: LABEL_FATBOOT="bootfs" LABEL="bootfs" UUID="0B22-2966" BLOCK_SIZE="512" TYPE="vfat" PARTUUID="645b9b92-01"
/dev/mmcblk0p2: LABEL="rootfs" UUID="3ad7386b-e1ae-4032-ae33-0c40f5ecc4ac" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="645b9b92-02"
/dev/sda: UUID="65557b2d-5448-4afd-a58c-cab7c77410d2" BLOCK_SIZE="4096" TYPE="ext4"
So in my fstab sudo nano /etc/fstab I added the following line: UUID="65557b2d-5448-4afd-a58c-cab7c77410d2" /data ext4 defaults 0 0
After this I ran sudo mount -a and it mounted the drive to /data.
Now to prepare for the gluster brick mount. sudo mkdir -p /glusterfs/data
Time to get the Gluster cluster set up. To start, on one of the Pi ( rpi-05 ) I ran gluster peer probe rpi-06.
Now I can create the gluster volume:
volume create data replica 2 transport tcp 192.168.1.5:/data/gluster/bricks/brick1/data 192.168.1.6:/data/gluster/bricks/brick1/data
Now to mount the gluster volume. I added the below to the end of the /etc/fstab localhost:/data /glusterfs/data glusterfs defaults,_netdev 0 0
Then ran the sudo mount -a command again, and now I can run df -h and see that /data is mounted to the /dev/sdb USB device and /gluster is mounted localhost:/data 113G 9.3G 99G 9% /glusterfs/data
Now all I had to do was stop the SWAG container on rpi-04 and rsync the data over to rpi-05 and then let gluster replicate the data over.
After that was done, it was as easy as editing the docker-compose.yml file to point the volume to the new /gluster/data mount point and start the container.
Finally ,I updated my router to port forward to the new floating IP/VIP and test.
It worked, now to test the failover. sudo pcs node standby rpi-06 this should stop the resource on rpi-06 and startup on rpi-05
Lets test:
root@rpi-05:~# pcs status
Cluster name: pi-cluster
Status of pacemakerd: 'Pacemaker is running' (last updated 2023-12-20 17:29:26Z)
Cluster Summary:
* Stack: corosync
* Current DC: rpi-06 (version 2.1.5-a3f44794f94) - partition with quorum
* Last updated: Wed Dec 20 17:29:27 2023
* Last change: Wed Dec 20 08:35:24 2023 by root via cibadmin on rpi-06
* 2 nodes configured
* 1 resource instance configured
Node List:
*Online: [ rpi-05 rpi-06 ]
Full List of Resources:
* floating_ip (ocf:heartbeat:IPaddr2): Started rpi-06
Daemon Status:
corosync: active/enabled
pacemaker: active/enabled
pcsd: active/enabled
root@rpi-05:~# pcs node standby rpi-06
root@rpi-05:~# pcs status
Cluster name: pi-cluster
Status of pacemakerd: 'Pacemaker is running' (last updated 2023-12-20 17:29:43Z)
Cluster Summary:
* Stack: corosync
* Current DC: rpi-06 (version 2.1.5-a3f44794f94) - partition with quorum
* Last updated: Wed Dec 20 17:29:44 2023
* Last change: Wed Dec 20 17:29:39 2023 by root via cibadmin on rpi-05
* 2 nodes configured
* 1 resource instance configured
Node List:
* Node rpi-06: standby
* Online: [ rpi-05 ]
Full List of Resources:
* floating_ip (ocf:heartbeat:IPaddr2): Started rpi-05
Daemon Status:
corosync: active/enabled
pacemaker: active/enabled
pcsd: active/enabled
It worked and I didnt even get a notification from Uptime Kuma to say that it blipped.
My next step is just to add the Wireguard container and volumes to this set up and then my HA ingress is complete.
Now I have to set up rpi-01,2,3,4 all with glusterfs and rsync all that data around and change docker-compose files around to mount the new glusterfs mount point and fingers crossed I have a decent set up for some old Raspberry Pi. Maybe I document it and write about it next time.
Now you are probablly wondering why not just get better Raspberry Pi or buy a new NAS to fix your issue. And I say, why not try something different with what you have and learn something in the process?
Let me know your thoughts over on twitter or email feedback@0lzi.com