Homelab<\/h2>\n
We have two CentOS 7 servers installed which we want to configure as follows:<\/p>\n
storage1.hl.local (10.11.1.15) \u2013 Pacemaker cluster node
\nstorage2.hl.local (10.11.1.16) \u2013 Pacemaker cluster node<\/p>\n
SELinux set to enforcing mode.<\/p>\n
See the image below to identify the homelab part this article applies to.<\/p>\n
![\"\"](\"https:\/\/www.lisenet.com\/wp-content\/uploads\/2018\/04\/lisenet-homelab-diagram_nfs.png\")
<\/p>\n
Cluster Requirements<\/h2>\n
To configure the cluster, we are going to need the following:<\/p>\n
- \n
- A virtual IP address, required for the NFS server.<\/li>\n
- Shared storage for the NFS nodes in the cluster.<\/li>\n
- A power fencing device for each node of the cluster.<\/li>\n<\/ol>\n
The virtual IP is 10.11.1.31 (with the DNS name of nfsvip.hl.local).<\/p>\n
With regards to shared storage, while I agree that iSCSI would be ideal, the truth is that \u201cwe don\u2019t have that kind of money<\/a>\u201c. We will have to make it with a shared disk among different VMs on same Proxmox host.<\/p>\n
In terms of fencing, as mentioned earlier, Proxmox does not use libvirt, therefore Pacemaker clusters cannot be fenced by using fence-agents-virsh. There is fence_pve available, but we won\u2019t find it in CentOS\/RHEL. We\u2019ll need to compile it from source.<\/p>\n
Proxmox and Disk Sharing<\/h2>\n
I was unable to find a WebUI way to add an existing disk to another VM. Proxmox forum<\/a> was somewhat helpful, and I ended up manually editing the VM\u2019s config file since the WebUI would not let me assign the same disk to two VMs.<\/p>\n
Take a look at the following image, showing two disks attached to the storage1.hl.local node:<\/p>\n
![\"\"](\"https:\/\/www.lisenet.com\/wp-content\/uploads\/2018\/04\/lisenet-homelab-shared-disk.png\")
<\/p>\nWe want to use the smaller (2GB) disk for NFS.<\/p>\n
The VM ID of the storage2.hl.local node is 208 (see here<\/a>), therefore we can add the disk by editing the node\u2019s configuration file.<\/p>\n
# cat \/etc\/pve\/qemu-server\/208.conf
\nboot: cn
\nbootdisk: scsi0
\ncores: 1
\nhotplug: disk,cpu
\nmemory: 768
\nname: storage2.hl.local
\nnet0: virtio=00:22:FF:00:00:16,bridge=vmbr0
\nonboot: 1
\nostype: l26
\nscsi0: data_ssd:208\/vm-208-disk-1.qcow2,size=32G
\nscsi1: data_ssd:207\/vm-207-disk-3.qcow2,size=2G
\nscsihw: virtio-scsi-pci
\nsmbios1: uuid=030e28da-72e6-412d-be77-a79f06862351
\nsockets: 1
\nstartup: order=208<\/p>\nThe disk that we\u2019ve added is scsi1. Note how it references the VM ID 207.<\/p>\n
The disk will be visible on both nodes as \/dev\/disk\/by-id\/scsi-0QEMU_QEMU_HARDDISK_drive-scsi1.<\/p>\n
Configuration with Puppet<\/h2>\n
Puppet master runs on the Katello<\/a> server.<\/p>\n
Puppet Modules<\/h3>\n
We use puppet-corosync<\/a> Puppet module to configure the server. We also use puppetlabs-accounts<\/a> for Linux account creation.<\/p>\n
Please see the module documentation for features supported and configuration options available.<\/p>\n
Configure Firewall<\/h3>\n
It is essential to ensure that Pacemaker servers can talk to each other. The following needs applying to both cluster nodes:<\/p>\n
firewall { ‘007 accept HA cluster requests’:
\ndport => [‘2224’, ‘3121’, ‘5403’, ‘21064’],
\nproto => ‘tcp’,
\nsource => ‘10.11.1.0\/24’,
\naction => ‘accept’,
\n}->
\nfirewall { ‘008 accept HA cluster requests’:
\ndport => [‘5404’, ‘5405’],
\nproto => ‘udp’,
\nsource => ‘10.11.1.0\/24’,
\naction => ‘accept’,
\n}->
\nfirewall { ‘009 accept NFS requests’:
\ndport => [‘2049’],
\nproto => ‘tcp’,
\nsource => ‘10.11.1.0\/24’,
\naction => ‘accept’,
\n}->
\nfirewall { ‘010 accept TCP mountd requests’:
\ndport => [‘20048’],
\nproto => ‘tcp’,
\nsource => ‘10.11.1.0\/24’,
\naction => ‘accept’,
\n}->
\nfirewall { ‘011 accept UDP mountd requests’:
\ndport => [‘20048’],
\nproto => ‘udp’,
\nsource => ‘10.11.1.0\/24’,
\naction => ‘accept’,
\n}->
\nfirewall { ‘012 accept TCP rpc-bind requests’:
\ndport => [‘111’],
\nproto => ‘tcp’,
\nsource => ‘10.11.1.0\/24’,
\naction => ‘accept’,
\n}->
\nfirewall { ‘013 accept UDP rpc-bind requests’:
\ndport => [‘111’],
\nproto => ‘udp’,
\nsource => ‘10.11.1.0\/24’,
\naction => ‘accept’,
\n}<\/p>\nCreate Apache User and NFS Mountpoint<\/h3>\n
Before we configure the cluster, we need to make sure that we have the nfs-utils package installed and that the nfs-lock service is disabled \u2013 it will be managed by pacemaker.<\/p>\n
The Apache user is created in order to match ownership and allow web servers to write to the NFS share.<\/p>\n
The following needs applying to both cluster nodes:<\/p>\n
package { ‘nfs-utils’: ensure => ‘installed’ }->
\nservice { ‘nfs-lock’: enable => false }->
\naccounts::user { ‘apache’:
\ncomment => ‘Apache’,
\nuid => ’48’,
\ngid => ’48’,
\nshell => ‘\/sbin\/nologin’,
\npassword => ‘!!’,
\nhome => ‘\/usr\/share\/httpd’,
\nhome_mode => ‘0755’,
\nlocked => false,
\n}->
\nfile {‘\/nfsshare’:
\nensure => ‘directory’,
\nowner => ‘root’,
\ngroup => ‘root’,
\nmode => ‘0755’,
\n}<\/p>\nConfigure Pacemaker\/Corosync on storage1.hl.local<\/h3>\n
We disable STONITH initially because the fencing agent fence_pve is simply not available yet. We will compile it later, however, it\u2019s not required in order to get the cluster into an operational state.<\/p>\n
We use colocations to keep primitives together. While colocation defines that a set of primitives must live together on the same node, order definitions will define the order of which each primitive is started. This is importat, as we want to make sure that we start cluster resources in the correct order.<\/p>\n
Note how we configure NFS exports to be available to two specific clients only: web1.hl.local and web2.hl.local. In reality there is no need for any other homelab server to have access to the NFS share.<\/p>\n
We make the apache user the owner of the NFS share, and export it with no_all_squash.<\/p>\n
class { ‘corosync’:
\nauthkey => ‘\/etc\/puppetlabs\/puppet\/ssl\/certs\/ca.pem’,
\nbind_address => $::ipaddress,
\ncluster_name => ‘nfs_cluster’,
\nenable_secauth => true,
\nenable_corosync_service => true,
\nenable_pacemaker_service => true,
\nset_votequorum => true,
\nquorum_members => [ ‘storage1.hl.local’, ‘storage2.hl.local’ ],
\n}
\ncorosync::service { ‘pacemaker’:
\n## See: https:\/\/wiki.clusterlabs.org\/wiki\/Pacemaker<\/em>
\nversion => ‘1.1’,
\n}->
\ncs_property { ‘stonith-enabled’:
\nvalue => ‘false’,
\n}->
\ncs_property { ‘no-quorum-policy’:
\nvalue => ‘ignore’,
\n}->
\ncs_primitive { ‘nfsshare’:
\nprimitive_class => ‘ocf’,
\nprimitive_type => ‘Filesystem’,
\nprovided_by => ‘heartbeat’,
\nparameters => { ‘device’ => ‘\/dev\/disk\/by-id\/scsi-0QEMU_QEMU_HARDDISK_drive-scsi1’, ‘directory’ => ‘\/nfsshare’, ‘fstype’ => ‘ext4’ },
\n}->
\ncs_primitive { ‘nfsd’:
\nprimitive_class => ‘ocf’,
\nprimitive_type => ‘nfsserver’,
\nprovided_by => ‘heartbeat’,
\nparameters => { ‘nfs_shared_infodir’ => ‘\/nfsshare\/nfsinfo’ },
\nrequire => Cs_primitive[‘nfsshare’],
\n}->
\ncs_primitive { ‘nfsroot1’:
\nprimitive_class => ‘ocf’,
\nprimitive_type => ‘exportfs’,
\nprovided_by => ‘heartbeat’,
\nparameters => { ‘clientspec’ => ‘web1.hl.local’, ‘options’ => ‘rw,async,no_root_squash,no_all_squash’, ‘directory’ => ‘\/nfsshare’, ‘fsid’ => ‘0’ },
\nrequire => Cs_primitive[‘nfsd’],
\n}->
\ncs_primitive { ‘nfsroot2’:
\nprimitive_class => ‘ocf’,
\nprimitive_type => ‘exportfs’,
\nprovided_by => ‘heartbeat’,
\nparameters => { ‘clientspec’ => ‘web2.hl.local’, ‘options’ => ‘rw,async,no_root_squash,no_all_squash’, ‘directory’ => ‘\/nfsshare’, ‘fsid’ => ‘0’ },
\nrequire => Cs_primitive[‘nfsd’],
\n}->
\ncs_primitive { ‘nfsvip’:
\nprimitive_class => ‘ocf’,
\nprimitive_type => ‘IPaddr2’,
\nprovided_by => ‘heartbeat’,
\nparameters => { ‘ip’ => ‘10.11.1.31’, ‘cidr_netmask’ => ’24’ },
\nrequire => Cs_primitive[‘nfsroot1′,’nfsroot2’],
\n}->
\ncs_colocation { ‘nfsshare_nfsd_nfsroot_nfsvip’:
\nprimitives => [
\n[ ‘nfsshare’, ‘nfsd’, ‘nfsroot1’, ‘nfsroot2’, ‘nfsvip’ ],
\n}->
\ncs_order { ‘nfsshare_before_nfsd’:
\nfirst => ‘nfsshare’,
\nsecond => ‘nfsd’,
\nrequire => Cs_colocation[‘nfsshare_nfsd_nfsroot_nfsvip’],
\n}->
\ncs_order { ‘nfsd_before_nfsroot1’:
\nfirst => ‘nfsd’,
\nsecond => ‘nfsroot1’,
\nrequire => Cs_colocation[‘nfsshare_nfsd_nfsroot_nfsvip’],
\n}->
\ncs_order { ‘nfsroot1_before_nfsroot2’:
\nfirst => ‘nfsroot1’,
\nsecond => ‘nfsroot2’,
\nrequire => Cs_colocation[‘nfsshare_nfsd_nfsroot_nfsvip’],
\n}->
\ncs_order { ‘nfsroot2_before_nfsvip’:
\nfirst => ‘nfsroot2’,
\nsecond => ‘nfsvip’,
\nrequire => Cs_colocation[‘nfsshare_nfsd_nfsroot_nfsvip’],
\n}->
\nfile {‘\/nfsshare\/uploads’:
\nensure => ‘directory’,
\nowner => ‘apache’,
\ngroup => ‘root’,
\nmode => ‘0755’,
\n}<\/p>\nConfigure Pacemaker\/Corosync on storage2.hl.local<\/h3>\n
class { ‘corosync’:
\nauthkey => ‘\/etc\/puppetlabs\/puppet\/ssl\/certs\/ca.pem’,
\nbind_address => $::ipaddress,
\ncluster_name => ‘nfs_cluster’,
\nenable_secauth => true,
\nenable_corosync_service => true,
\nenable_pacemaker_service => true,
\nset_votequorum => true,
\nquorum_members => [ ‘storage1.hl.local’, ‘storage2.hl.local’ ],
\n}
\ncorosync::service { ‘pacemaker’:
\nversion => ‘1.1’,
\n}->
\ncs_property { ‘stonith-enabled’:
\nvalue => ‘false’,
\n}<\/p>\nCluster Status<\/h3>\n
If all went well, we should have our cluster up and running at this point.<\/p>\n
[[email protected]<\/a> ~]# pcs status
\nCluster name: nfs_cluster
\nStack: corosync
\nCurrent DC: storage2.hl.local (version 1.1.16-12.el7_4.8-94ff4df) – partition with quorum
\nLast updated: Sun Apr 29 17:04:50 2018
\nLast change: Sun Apr 29 16:56:25 2018 by root via cibadmin on storage1.hl.local<\/p>\n2 nodes configured
\n5 resources configured<\/p>\nOnline: [ storage1.hl.local storage2.hl.local ]<\/p>\n
Full list of resources:<\/p>\n
nfsshare (ocf::heartbeat:Filesystem): Started storage1.hl.local
\nnfsd (ocf::heartbeat:nfsserver): Started storage1.hl.local
\nnfsroot1 (ocf::heartbeat:exportfs): Started storage1.hl.local
\nnfsroot2 (ocf::heartbeat:exportfs): Started storage1.hl.local
\nnfsvip (ocf::heartbeat:IPaddr2): Started storage1.hl.local<\/p>\nDaemon Status:
\ncorosync: active\/enabled
\npacemaker: active\/enabled
\npcsd: inactive\/disabled
\n[[email protected]<\/a> ~]# pcs status
\nCluster name: nfs_cluster
\nStack: corosync
\nCurrent DC: storage2.hl.local (version 1.1.16-12.el7_4.8-94ff4df) – partition with quorum
\nLast updated: Sun Apr 29 17:05:04 2018
\nLast change: Sun Apr 29 16:56:25 2018 by root via cibadmin on storage1.hl.local<\/p>\n2 nodes configured
\n5 resources configured<\/p>\nOnline: [ storage1.hl.local storage2.hl.local ]<\/p>\n
Full list of resources:<\/p>\n
nfsshare (ocf::heartbeat:Filesystem): Started storage1.hl.local
\nnfsd (ocf::heartbeat:nfsserver): Started storage1.hl.local
\nnfsroot1 (ocf::heartbeat:exportfs): Started storage1.hl.local
\nnfsroot2 (ocf::heartbeat:exportfs): Started storage1.hl.local
\nnfsvip (ocf::heartbeat:IPaddr2): Started storage1.hl.local<\/p>\nDaemon Status:
\ncorosync: active\/enabled
\npacemaker: active\/enabled
\npcsd: inactive\/disabled<\/p>\nTest cluster failover by putting the active node into standby:<\/p>\n
[[email protected]<\/a> ~]# pcs node standby<\/p>\n
Services should become available on the other cluster node:<\/p>\n
[[email protected]<\/a> ~]# pcs status
\nCluster name: nfs_cluster
\nStack: corosync
\nCurrent DC: storage2.hl.local (version 1.1.16-12.el7_4.8-94ff4df) – partition with quorum
\nLast updated: Sun Apr 29 17:06:36 2018
\nLast change: Sun Apr 29 16:56:25 2018 by root via cibadmin on storage1.hl.local<\/p>\n2 nodes configured
\n5 resources configured<\/p>\nNode storage1.hl.local: standby
\nOnline: [ storage2.hl.local ]<\/p>\nFull list of resources:<\/p>\n
nfsshare (ocf::heartbeat:Filesystem): Started storage2.hl.local
\nnfsd (ocf::heartbeat:nfsserver): Started storage2.hl.local
\nnfsroot1 (ocf::heartbeat:exportfs): Started storage2.hl.local
\nnfsroot2 (ocf::heartbeat:exportfs): Started storage2.hl.local
\nnfsvip (ocf::heartbeat:IPaddr2): Started storage2.hl.local<\/p>\nDaemon Status:
\ncorosync: active\/enabled
\npacemaker: active\/enabled
\npcsd: inactive\/disabled<\/p>\nDo showmount on the virtual IP address:<\/p>\n
[[email protected]<\/a> ~]# showmount -e 10.11.1.31
\nExport list for 10.11.1.31:
\n\/nfsshare web1.hl.local,web2.hl.local<\/p>\nCompile fence_pve on CentOS 7<\/h2>\n
This is where the automated part ends I\u2019m afraid, however, there is nothing that stops you from putting the manual steps below into a Puppet manifest.<\/p>\n
Install Packages<\/h3>\n
# yum install git gcc make automake autoconf libtool
\npexpect python-requests<\/p>\nDownload Source and Compile<\/h3>\n
# git clone https:\/\/github.com\/ClusterLabs\/fence-agents.git<\/p>\n
Note the configuration part, we are interested in compiling one fencing agent only, fence_pve.<\/p>\n
# cd fence-agents\/
\n# .\/autogen.sh
\n# .\/configure –with-agents=pve
\n# make && make install<\/p>\nVerify:<\/p>\n
# fence_pve –version
\n4.1.1.51-6e6d<\/p>\nConfigure Pacemaker to Use fence_pve<\/h3>\n
Big thanks to Igor Cicimov\u2019s blog post<\/a> which helped me to get it working with minimal effort.<\/p>\n
To test the fencing agent, do the following:<\/p>\n
[[email protected]<\/a> ~]# fence_pve –ip=10.11.1.5 –nodename=pve
\n[email protected]<\/a> –password=passwd
\n–plug=208 –action=off<\/p>\nWhere 10.11.1.5 is the IP of the Proxmox hypervisor, pve is the name of the Proxmox node, and the plug is the VM ID. In this case we fenced the storage2.hl.local node.<\/p>\n
To configure Pacemaker, we can create two STONITH configurations, one for each node that we want to be able to fence.<\/p>\n
[[email protected]<\/a> ~]# pcs stonith create my_proxmox_fence207 fence_pve
\nipaddr=”10.11.1.5″ inet4_only=”true” vmtype=”qemu”
\nlogin=”[email protected]<\/a>” passwd=”passwd”
\nnode_name=”pve” delay=”15″
\nport=”207″
\npcmk_host_check=static-list
\npcmk_host_list=”storage1.hl.local”
\n[[email protected]<\/a> ~]# pcs stonith create my_proxmox_fence208 fence_pve
\nipaddr=”10.11.1.5″ inet4_only=”true” vmtype=”qemu”
\nlogin=”[email protected]<\/a>” passwd=”passwd”
\nnode_name=”pve” delay=”15″
\nport=”208″
\npcmk_host_check=static-list
\npcmk_host_list=”storage2.hl.local”<\/p>\nVerify:<\/p>\n
[[email protected]<\/a> ~]# stonith_admin -L
\nmy_proxmox_fence207
\nmy_proxmox_fence208
\n2 devices found
\n[[email protected]<\/a> ~]# pcs status
\nCluster name: nfs_cluster
\nStack: corosync
\nCurrent DC: storage1.hl.local (version 1.1.16-12.el7_4.8-94ff4df) – partition with quorum
\nLast updated: Sun Apr 29 17:50:59 2018
\nLast change: Sun Apr 29 17:50:55 2018 by root via cibadmin on storage1.hl.local<\/p>\n2 nodes configured
\n7 resources configured<\/p>\nOnline: [ storage1.hl.local ]
\nOFFLINE: [ storage2.hl.local ]<\/p>\nFull list of resources:<\/p>\n
nfsshare (ocf::heartbeat:Filesystem): Started storage1.hl.local
\nnfsd (ocf::heartbeat:nfsserver): Started storage1.hl.local
\nnfsroot1 (ocf::heartbeat:exportfs): Started storage1.hl.local
\nnfsroot2 (ocf::heartbeat:exportfs): Started storage1.hl.local
\nnfsvip (ocf::heartbeat:IPaddr2): Started storage1.hl.local
\nmy_proxmox_fence207 (stonith:fence_pve): Started storage1.hl.local
\nmy_proxmox_fence208 (stonith:fence_pve): Started storage1.hl.local<\/p>\nDaemon Status:
\ncorosync: active\/enabled
\npacemaker: active\/enabled
\npcsd: inactive\/disabled<\/p>\nNote how the storage2.hl.local node is down, because we\u2019ve fenced it.<\/p>\n
If you decide to use test configuration, do not forget to stop the Puppet agent on the cluster nodes as it will disable STONITH (we set stonith-enabled to false in the manifest).<\/p>\n
For more info, do the following:<\/p>\n
# pcs stonith describe fence_pve<\/p>\n
This will give you a list of other STONITH options available.<\/p>\n