[ClusterLabs] fence-scsi question
I have a 2-node CentOS7 cluster running ZFS. The two nodes (vsphere appliances on different hosts) access 2 SAS SSD in a Supermicro JBOD with 2 mini-SAS connectors. It all works fine - failover and all. My quandary was how to implement fencing. I was able to get both of the vmware SOAP and REST fencing agents to work - it just isn't reliable enough. If the vcenter server appliance is busy, fencing requests timeout. I know I can increase the timeouts, but in at least one test run, even a minute wasn't enough, and my concern is that too long switching over, and vmware will put the datastore in APD, hosing guests. I confirmed that both SSD work properly with the fence-scsi agent. Fencing the host who actively owns the ZFS pool also works perfectly (ZFS flushes data to the datastore every 5 seconds or so, so withdrawing the SCSI-3 persistent reservations causes a fatal write error to the pool, and setting the pool in failmode=panic will cause the fenced cluster node to reboot automatically.) The problem (maybe it isn't really one?) is that fencing the node that does *not* own the pool has no effect, since it holds no reservations on the devices in the pool.) I'd love to be sure this isn't an issue at all. ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] fence-scsi question
On 2020-02-10 00:06, Strahil Nikolov wrote: On February 10, 2020 2:07:01 AM GMT+02:00, Dan Swartzendruber wrote: I have a 2-node CentOS7 cluster running ZFS. The two nodes (vsphere appliances on different hosts) access 2 SAS SSD in a Supermicro JBOD with 2 mini-SAS connectors. It all works fine - failover and all. My quandary was how to implement fencing. I was able to get both of the vmware SOAP and REST fencing agents to work - it just isn't reliable enough. If the vcenter server appliance is busy, fencing requests timeout. I know I can increase the timeouts, but in at least one test run, even a minute wasn't enough, and my concern is that too long switching over, and vmware will put the datastore in APD, hosing guests. I confirmed that both SSD work properly with the fence-scsi agent. Fencing the host who actively owns the ZFS pool also works perfectly (ZFS flushes data to the datastore every 5 seconds or so, so withdrawing the SCSI-3 persistent reservations causes a fatal write error to the pool, and setting the pool in failmode=panic will cause the fenced cluster node to reboot automatically.) The problem (maybe it isn't really one?) is that fencing the node that does *not* own the pool has no effect, since it holds no reservations on the devices in the pool.) I'd love to be sure this isn't an issue at all. ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/ Hi Dan, You can configure multiple fencing mechanisms in your cluster. For example, you can set the first fencing mechanism to be via VmWare and if it fails (being busy or currrently unavailable), then the scsi fencing can kick in to ensure a failover can be done. What you observe is normal - no scsi reservations -> no fencing. That's why major vendors require , when using fence_multipath/fence_scsi, the shared storage to be a dependency (a File system in use by the application) and not just an add-on. I personally don't like scsi reservations, as there is no guarantee that other resources (services, IPs, etc) are actually down , but the risk is low. In your case fence_scsi stonith can be a second layer of protection. Best Regards, Strahil Nikolov Okay, thanks. I'll look into multi-level then. ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] Stonith configuration
On 2020-02-14 13:06, Strahil Nikolov wrote: On February 14, 2020 4:44:53 PM GMT+02:00, "BASDEN, ALASTAIR G." wrote: Hi Strahil, Note2: Consider adding a third node /for example a VM/ or a qdevice on a separate node (allows to be on a separate network, so a simple routing is the only requirement ) and reconfigure the cluster , so you have 'Expected votes: 3' . This will protect you from split brain and is highly recommended. Highly recommend qdevice. I spun one up on a small (paperback size) 'router' running CentOS7. ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] How to unfence without reboot (fence_mpath)
Many people don't have red hat access, so linking those urls is not useful. On February 17, 2020, at 1:40 AM, Strahil Nikolov wrote: Hello Ondrej, thanks for your reply. I really appreciate that. I have picked fence_multipath as I'm preparing for my EX436 and I can't know what agent will be useful on the exam. Also ,according to https://access.redhat.com/solutions/3201072 , there could be a race condition with fence_scsi. So, I've checked the cluster when fencing and the node immediately goes offline. Last messages from pacemaker are: Feb 17 08:21:57 node1.localdomain stonith-ng[23808]: notice: Client stonith_admin.controld.23888.b57ceee7 wants to fence (reboot) 'node1.localdomain' with device '(any)' Feb 17 08:21:57 node1.localdomain stonith-ng[23808]: notice: Requesting peer fencing (reboot) of node1.localdomain Feb 17 08:21:57 node1.localdomain stonith-ng[23808]: notice: FENCING can fence (reboot) node1.localdomain (aka. '1'): static-list Feb 17 08:21:58 node1.localdomain stonith-ng[23808]: notice: Operation reboot of node1.localdomain by node2.localdomain for stonith_admin.controld.23888@node1.localdomain.ede38ffb: OK Feb 17 08:21:58 node1.localdomain crmd[23812]: crit: We were allegedly just fenced by node2.localdomain for node1.localdomai Which for me means - node1 just got fenced again. Actually fencing works ,as I/O is immediately blocked and the reservation is removed. I've used https://access.redhat.com/solutions/2766611 to setup the fence_mpath , but I could have messed up something. Cluster config is: [root@node3 ~]# pcs config show Cluster Name: HACLUSTER2 Corosync Nodes: node1.localdomain node2.localdomain node3.localdomain Pacemaker Nodes: node1.localdomain node2.localdomain node3.localdomain Resources: Clone: dlm-clone Meta Attrs: interleave=true ordered=true Resource: dlm (class=ocf provider=pacemaker type=controld) Operations: monitor interval=30s on-fail=fence (dlm-monitor-interval-30s) start interval=0s timeout=90 (dlm-start-interval-0s) stop interval=0s timeout=100 (dlm-stop-interval-0s) Clone: clvmd-clone Meta Attrs: interleave=true ordered=true Resource: clvmd (class=ocf provider=heartbeat type=clvm) Operations: monitor interval=30s on-fail=fence (clvmd-monitor-interval-30s) start interval=0s timeout=90s (clvmd-start-interval-0s) stop interval=0s timeout=90s (clvmd-stop-interval-0s) Clone: TESTGFS2-clone Meta Attrs: interleave=true Resource: TESTGFS2 (class=ocf provider=heartbeat type=Filesystem) Attributes: device=/dev/TEST/gfs2 directory=/GFS2 fstype=gfs2 options=noatime run_fsck=no Operations: monitor interval=15s on-fail=fence OCF_CHECK_LEVEL=20 (TESTGFS2-monitor-interval-15s) notify interval=0s timeout=60s (TESTGFS2-notify-interval-0s) start interval=0s timeout=60s (TESTGFS2-start-interval-0s) stop interval=0s timeout=60s (TESTGFS2-stop-interval-0s) Stonith Devices: Resource: FENCING (class=stonith type=fence_mpath) Attributes: devices=/dev/mapper/36001405cb123d000 pcmk_host_argument=key pcmk_host_map=node1.localdomain:1;node2.localdomain:2;node3.localdomain:3 pcmk_monitor_action=metadata pcmk_reboot_action=off Meta Attrs: provides=unfencing Operations: monitor interval=60s (FENCING-monitor-interval-60s) Fencing Levels: Location Constraints: Ordering Constraints: start dlm-clone then start clvmd-clone (kind:Mandatory) (id:order-dlm-clone-clvmd-clone-mandatory) start clvmd-clone then start TESTGFS2-clone (kind:Mandatory) (id:order-clvmd-clone-TESTGFS2-clone-mandatory) Colocation Constraints: clvmd-clone with dlm-clone (score:INFINITY) (id:colocation-clvmd-clone-dlm-clone-INFINITY) TESTGFS2-clone with clvmd-clone (score:INFINITY) (id:colocation-TESTGFS2-clone-clvmd-clone-INFINITY) Ticket Constraints: Alerts: No alerts defined Resources Defaults: No defaults set [root@node3 ~]# crm_mon -r1 Stack: corosync Current DC: node3.localdomain (version 1.1.20-5.el7_7.2-3c4c782f70) - partition with quorum Last updated: Mon Feb 17 08:39:30 2020 Last change: Sun Feb 16 18:44:06 2020 by root via cibadmin on node1.localdomain 3 nodes configured 10 resources configured Online: [ node2.localdomain node3.localdomain ] OFFLINE: [ node1.localdomain ] Full list of resources: FENCING (stonith:fence_mpath): Started node2.localdomain Clone Set: dlm-clone [dlm] Started: [ node2.localdomain node3.localdomain ] Stopped: [ node1.localdomain ] Clone Set: clvmd-clone [clvmd] Started: [ node2.localdomain node3.localdomain ] Stopped: [ node1.localdomain ] Clone Set: TESTGFS2-clone [TESTGFS2] Started: [ node2.localdomain node3.localdomain ] Stopped: [ node1.localdomain ] In the logs , I've noticed that the node is first unfenced and later it is fenced again... For the unfence , I believe "meta provides=unfencing" is 'guilty', yet I'm not sure about
Re: [ClusterLabs] 2 node clusters - ipmi fencing
I believe you in fact want each fence agent to run on the other node, yes. On February 20, 2020, at 6:23 PM, Ricardo Esteves wrote: Hi, I have a question regarding fencing, i have 2 physical servers: node01, node02, each one has an ipmi card so i create 2 fence devices: fence_ipmi_node01 (with ip of ipmi card of server node01) - with constraint to prefer to run on node01 fence_ipmi_node02 (with ip of ipmi card of server node02) - with constraint to prefer to run on node02 - configured also a 20s delay on this one Is this the best practice? Like this node01 can only fence itself right? and node02 also can only fence itself right? Shouldn't i configure fence_ipmi_node01 location constraint to be placed on node02 and fence_ipmi_node02 location constraint to be placed on node01 ? So that node01 can fence node02 and vice versa? ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/ ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] 2 node clusters - ipmi fencing
On 2020-02-21 08:51, Ricardo Esteves wrote: Hi, I'm trying to understand what is the objective of the constraints to have the fencing devices running on opposite node or on its own node or running all on the same node. Can you explain the difference? IPMI fencing involves the instance interacting with the IPMI device and telling it to power-cycle/reset/whatever the target node. It waits for a confirmation, then tells the cluster SW the fencing operation was successful. If the instance is running on the same node, it will be blown away when the node is reset. ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] connection timed out fence_virsh monitor stonith
On 2020-02-24 12:17, Strahil Nikolov wrote: On February 24, 2020 4:56:07 PM GMT+02:00, Luke Camilleri wrote: Hello users, I would like to ask for assistance on the below setup please, mainly on the monitor fence timeout: I notice that the issue happens at 00:00 on both days . Have you checked for a backup or other cron job that is 'overloading' the virtualization host ? This is a very good point. I had a similar problem with a vsphere cluster. Two hyper-converged storage appliances. I used the fence-vmware-rest (or soap) stonith agent to fence the storage apps. Worked just fine. Until the vcenter server appliance got busy doing something or other. Next thing I know, I'm getting stonith agent timeouts. I ended up switching to fence_scsi. Not sure there is a good answer. I saw on a vmware forum a recommendation to increase the stonith timeout, but the recommended timeout was close to a minute, which is enough to be a problem for the VMs in that cluster... ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] Preferred node for a service (not constrained)
On 2020-11-30 23:21, Petr Bena wrote: Hello, Is there a way to setup a preferred node for a service? I know how to create constrain that will make it possible to run a service ONLY on certain node, or constrain that will make it impossible to run 2 services on same node, but I don't want any of that, as in catastrophical scenarios when services would have to be located together on same node, this would instead disable it. Essentially what I want is for service to be always started on preferred node when it is possible, but if it's not possible (eg. node is down) it would freely run on any other node, with no restrictions and when node is back up, it would migrate back. How can I do that? I do precisely this for an active/passive NFS/ZFS storage appliance pair. One of the VSA has more memory and is less used, so I have it set to prefer that host. https://clusterlabs.org/pacemaker/doc/en-US/Pacemaker/1.1/html/Clusters_from_Scratch/_prefer_one_node_over_another.html I believe I used the value infinity, so it will prefer the 2nd host over the 1st if at all possible. My 'pcs constraint': [root@centos-vsa2 ~]# pcs constraint Location Constraints: Resource: group-zfs Enabled on: centos-vsa2 (score:INFINITY) Ordering Constraints: Colocation Constraints: Ticket Constraints: ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
[ClusterLabs] Fencing with a 3-node (1 for quorum only) cluster
I'm setting up an HA NFS server to serve up storage to a couple of vsphere hosts. I have a virtual IP, and it depends on a ZFS resource agent which imports or exports a pool. So far, with stonith disabled, it all works perfectly. I was dubious about a 2-node solution, so I created a 3rd node which runs as a virtual machine on one of the hosts. All it is for is quorum. So, looking at fencing next. The primary server is a poweredge R905, which has DRAC for fencing. The backup storage node is a Supermicro X9-SCL-F (with IPMI). So I would be using the DRAC agent for the former and the ipmilan for the latter? I was reading about location constraints, where you tell each instance of the fencing agent not to run on the node that would be getting fenced. So, my first thought was to configure the drac agent and tell it not to fence node 1, and configure the ipmilan agent and tell it not to fence node 2. The thing is, there is no agent available for the quorum node. Would it make more sense instead to tell the drac agent to only run on node 2, and the ipmilan agent to only run on node 1? Thanks! ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] Fencing with a 3-node (1 for quorum only) cluster
On 2016-08-04 19:03, Digimer wrote: On 04/08/16 06:56 PM, Dan Swartzendruber wrote: I'm setting up an HA NFS server to serve up storage to a couple of vsphere hosts. I have a virtual IP, and it depends on a ZFS resource agent which imports or exports a pool. So far, with stonith disabled, it all works perfectly. I was dubious about a 2-node solution, so I created a 3rd node which runs as a virtual machine on one of the hosts. All it is for is quorum. So, looking at fencing next. The primary server is a poweredge R905, which has DRAC for fencing. The backup storage node is a Supermicro X9-SCL-F (with IPMI). So I would be using the DRAC agent for the former and the ipmilan for the latter? I was reading about location constraints, where you tell each instance of the fencing agent not to run on the node that would be getting fenced. So, my first thought was to configure the drac agent and tell it not to fence node 1, and configure the ipmilan agent and tell it not to fence node 2. The thing is, there is no agent available for the quorum node. Would it make more sense instead to tell the drac agent to only run on node 2, and the ipmilan agent to only run on node 1? Thanks! This is a common mistake. Fencing and quorum solve different problems and are not interchangeable. In short; Fencing is a tool when things go wrong. Quorum is a tool when things are working. The only impact that having quorum has with regard to fencing is that it avoids a scenario when both nodes try to fence each other and the faster one wins (which is itself OK). Even then, you can add 'delay=15' the node you want to win and it will win is such a case. In the old days, it would also prevent a fence loop if you started the cluster on boot and comms were down. Now though, you set 'wait_for_all' and you won't get a fence loop, so that solves that. Said another way; Quorum is optional, fencing is not (people often get that backwards). As for DRAC vs IPMI, no, they are not two things. In fact, I am pretty certain that fence_drac is a symlink to fence_ipmilan. All DRAC is (same with iRMC, iLO, RSA, etc) is "IPMI + features". Fundamentally, the fence action; rebooting the node, works via the basic IPMI standard using the DRAC's BMC. To do proper redundant fencing, which is a great idea, you want something like switched PDUs. This is how we do it (with two node clusters). IPMI first, and if that fails, a pair of PDUs (one for each PSU, each PDU going to independent UPSes) as backup. Thanks for the quick response. I didn't mean to give the impression that I didn't know the different between quorum and fencing. The only reason I (currently) have the quorum node was to prevent a deathmatch (which I had read about elsewhere.) If it is as simple as adding a delay as you describe, I'm inclined to go that route. At least on CentOS7, fence_ipmilan and fence_drac are not the same. e.g. they are both python scripts that are totally different. ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] Fencing with a 3-node (1 for quorum only) cluster
On 2016-08-04 19:33, Digimer wrote: On 04/08/16 07:21 PM, Dan Swartzendruber wrote: On 2016-08-04 19:03, Digimer wrote: On 04/08/16 06:56 PM, Dan Swartzendruber wrote: I'm setting up an HA NFS server to serve up storage to a couple of vsphere hosts. I have a virtual IP, and it depends on a ZFS resource agent which imports or exports a pool. So far, with stonith disabled, it all works perfectly. I was dubious about a 2-node solution, so I created a 3rd node which runs as a virtual machine on one of the hosts. All it is for is quorum. So, looking at fencing next. The primary server is a poweredge R905, which has DRAC for fencing. The backup storage node is a Supermicro X9-SCL-F (with IPMI). So I would be using the DRAC agent for the former and the ipmilan for the latter? I was reading about location constraints, where you tell each instance of the fencing agent not to run on the node that would be getting fenced. So, my first thought was to configure the drac agent and tell it not to fence node 1, and configure the ipmilan agent and tell it not to fence node 2. The thing is, there is no agent available for the quorum node. Would it make more sense instead to tell the drac agent to only run on node 2, and the ipmilan agent to only run on node 1? Thanks! This is a common mistake. Fencing and quorum solve different problems and are not interchangeable. In short; Fencing is a tool when things go wrong. Quorum is a tool when things are working. The only impact that having quorum has with regard to fencing is that it avoids a scenario when both nodes try to fence each other and the faster one wins (which is itself OK). Even then, you can add 'delay=15' the node you want to win and it will win is such a case. In the old days, it would also prevent a fence loop if you started the cluster on boot and comms were down. Now though, you set 'wait_for_all' and you won't get a fence loop, so that solves that. Said another way; Quorum is optional, fencing is not (people often get that backwards). As for DRAC vs IPMI, no, they are not two things. In fact, I am pretty certain that fence_drac is a symlink to fence_ipmilan. All DRAC is (same with iRMC, iLO, RSA, etc) is "IPMI + features". Fundamentally, the fence action; rebooting the node, works via the basic IPMI standard using the DRAC's BMC. To do proper redundant fencing, which is a great idea, you want something like switched PDUs. This is how we do it (with two node clusters). IPMI first, and if that fails, a pair of PDUs (one for each PSU, each PDU going to independent UPSes) as backup. Thanks for the quick response. I didn't mean to give the impression that I didn't know the different between quorum and fencing. The only reason I (currently) have the quorum node was to prevent a deathmatch (which I had read about elsewhere.) If it is as simple as adding a delay as you describe, I'm inclined to go that route. At least on CentOS7, fence_ipmilan and fence_drac are not the same. e.g. they are both python scripts that are totally different. The delay is perfectly fine. We've shipped dozens of two-node systems over the last five or so years and all were 2-node and none have had trouble. Where node failures have occurred, fencing operated properly and services were recovered. So in my opinion, in the interest of minimizing complexity, I recommend the two-node approach. As for the two agents not being symlinked, OK. It still doesn't change the core point through that both fence_ipmilan and fence_drac would be acting on the same target. Note; If you lose power to the mainboard (which we've seen, failed mainboard voltage regulator did this once), you lose the IPMI (DRAC) BMC. This scenario will leave your cluster blocked without an external secondary fence method, like switched PDUs. cheers Thanks! ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] Fencing with a 3-node (1 for quorum only) cluster
A lot of good suggestions here. Unfortunately, my budget is tapped out for the near future at least (this is a home lab/soho setup). I'm inclined to go with Digimer's two-node approach, with IPMI fencing. I understand mobos can die and such. In such a long-shot, manual intervention is fine. So, when I get a chance, I need to remove the quorum node from the cluster and switch it to two_node mode. Thanks for the info! ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] Fencing with a 3-node (1 for quorum only) cluster
Okay, I almost have this all working. fence_ipmilan for the supermicro host. Had to specify lanplus for it to work. fence_drac5 for the R905. That was failing to complete due to timeout. Found a couple of helpful posts that recommended increase the retry count to 3 and the timeout to 60. That worked also. The only problem now, is that it takes well over a minute to complete the fencing operation. In that interim, the fenced host shows as UNCLEAN (offline), and because the fencing operation hasn't completed, the other node has to wait to import the pool and share out the filesystem. This causes the vsphere hosts to declare the NFS datastore down. I hadn't gotten exact timing, but I think the fencing operation took a little over a minute. I'm wondering if I could change the timeout to a smaller value, but increase the retries? Like back to the default 20 second timeout, but change retries from 1 to 5? ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] Fencing with a 3-node (1 for quorum only) cluster
On 2016-08-06 19:46, Digimer wrote: On 06/08/16 07:33 PM, Dan Swartzendruber wrote: Okay, I almost have this all working. fence_ipmilan for the supermicro host. Had to specify lanplus for it to work. fence_drac5 for the R905. That was failing to complete due to timeout. Found a couple of helpful posts that recommended increase the retry count to 3 and the timeout to 60. That worked also. The only problem now, is that it takes well over a minute to complete the fencing operation. In that interim, the fenced host shows as UNCLEAN (offline), and because the fencing operation hasn't completed, the other node has to wait to import the pool and share out the filesystem. This causes the vsphere hosts to declare the NFS datastore down. I hadn't gotten exact timing, but I think the fencing operation took a little over a minute. I'm wondering if I could change the timeout to a smaller value, but increase the retries? Like back to the default 20 second timeout, but change retries from 1 to 5? Did you try the fence_ipmilan against the DRAC? It *should* work. Would be interesting to see if it had the same issue. Can you check the DRAC's host's power state using ipmitool directly without delay? Yes, I did try fence_ipmilan, but it got the timeout waiting for power off (or whatever). I have to admit, I switched to fence_drac and had the same issue, but after increasing the timeout and retries, got it to work, so it is possible, that fence_ipmilan is okay. They both seemed to take more than 60 seconds to complete the operation. I have to say that when I do a power cycle through the drac web interface, it takes awhile, so that might be normal. I think I will try again with 20 seconds and 5 retries and see how that goes... ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] Fencing with a 3-node (1 for quorum only) cluster
On 2016-08-06 21:59, Digimer wrote: On 06/08/16 08:22 PM, Dan Swartzendruber wrote: On 2016-08-06 19:46, Digimer wrote: On 06/08/16 07:33 PM, Dan Swartzendruber wrote: (snip) What about using ipmitool directly? I can't imagine that such a long time is normal. Maybe there is a firmware update for the DRAC and/or BIOS? (I know with Fujitsu, they recommend updating the IPMI BMC and BIOS together). Unfortunately, the R905 is EoL, so any updates are not likely. Over a minute to fence is, strictly speaking, OK. However, that's a significant delay in time to recover. The thing that concerns me, though, is the delay in I/O for vsphere clients. I know 2 or more retries of 60 seconds caused issues. I'm going to try again with 5 20-second retries, and see how that works. If this doesn't cooperate, I may need to look into an PDU or something... ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] Fencing with a 3-node (1 for quorum only) cluster
On 2016-08-06 21:59, Digimer wrote: On 06/08/16 08:22 PM, Dan Swartzendruber wrote: On 2016-08-06 19:46, Digimer wrote: On 06/08/16 07:33 PM, Dan Swartzendruber wrote: (snip) What about using ipmitool directly? I can't imagine that such a long time is normal. Maybe there is a firmware update for the DRAC and/or BIOS? (I know with Fujitsu, they recommend updating the IPMI BMC and BIOS together). Over a minute to fence is, strictly speaking, OK. However, that's a significant delay in time to recover. Okay, I tested with 20 second timeout and 5 retries, using fence_drac5 at the command line. Ran 'date' on both sides to see how long it took. Just under a minute. It's too late now to mess around any more for tonight. I do need to verify that that works okay for vsphere. I will post back my results. Thanks! ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] ClusterLabs] Failing over NFSv4/TCP exports
Thanks for the info. I only use esxi, which likely explains why I never had issues... Patrick Zwahlen wrote: >Hi, > >> -Original Message- >> From: Andreas Kurz [mailto:andreas.k...@gmail.com] >> Sent: mercredi, 17 août 2016 23:16 >> To: Cluster Labs - All topics related to open-source clustering welcomed >> >> Subject: Re: [ClusterLabs] Failing over NFSv4/TCP exports >> >> This is a known problem ... have a look into the portblock RA - it has >> the feature to send out TCP tickle ACKs to reset such hanging sessions. >> So you can configure a portblock resource that blocks the tcp port >> before starting the VIP and another portblock resource that unblocks the >> port afterwards and sends out that tickle ACKs. > >Thanks Andreas for pointing me to the portblock RA. I wasn't aware of it and >will read/test. > >I also made some further testing using ESXi and I found out that the ESXi NFS >client behaves in a completely different way when compared to the Linux client >and at first sight it actually seems to work (where the Linux client fails). > >It's mainly due to 2 things: > >1) Their NFS client is much more aggressive in terms of monitoring the server >and restarting sessions. > >2) Every new TCP session comes from a different source port compared to the >Linux client which seems to stick to a single source port. This actually >solves the issue of failing back to a node with FIN_WAIT1 sessions. > >Regards, Patrick > >** >This email and any files transmitted with it are confidential and >intended solely for the use of the individual or entity to whom they >are addressed. If you have received this email in error please notify >the system manager. "postmas...@navixia.com" Navixia SA >** >___ >Users mailing list: Users@clusterlabs.org >http://clusterlabs.org/mailman/listinfo/users > >Project Home: http://www.clusterlabs.org >Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf >Bugs: http://bugs.clusterlabs.org ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] Solved: converting configuration
On 2016-08-25 10:24, Gabriele Bulfon wrote: YESSS!!! That was it! :))) Upgraded to 1.1.15, rebuilt and the rng files contain a lot more stuff. Packaged, published, installed on the test machine: got all my instructions as is!!! :))) ...now last stepsmaking our custom agents/shells work on this new setup ;) For example, what does the yellow "stopped" state mean? Here is the last rows of crm output after my config instructions : Full list of resources: xstorage1-stonith (stonith:external/ssh-sonicle): Stopped xstorage2-stonith (stonith:external/ssh-sonicle): Stopped I believe you will see this if the cluster is in maintenance mode or stonith is disabled. Possibly other reasons, but these I have seen... ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
[ClusterLabs] fence_apc delay?
So, I was testing my ZFS dual-head JBOD 2-node cluster. Manual failovers worked just fine. I then went to try an acid-test by logging in to node A and doing 'systemctl stop network'. Sure enough, pacemaker told the APC fencing agent to power-cycle node A. The ZFS pool moved to node B as expected. As soon as node A was back up, I migrated the pool/IP back to node A. I *thought* all was okay, until a bit later, I did 'zpool status', and saw checksum errors on both sides of several of the vdevs. After much digging and poking, the only theory I could come up with was that maybe the fencing operation was considered complete too quickly? I googled for examples using this, and the best tutorial I found showed using a power-wait=5, whereas the default seems to be power-wait=0? (this is CentOS 7, btw...) I changed it to use 5 instead of 0, and did a several fencing operations while a guest VM (vsphere via NFS) was writing to the pool. So far, no evidence of corruption. BTW, the way I was creating and managing the cluster was with the lcmc java gui. Possibly the power-wait default of 0 comes from there, I can't really tell. Any thoughts or ideas appreciated :) ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] fence_apc delay?
It occurred to me folks reading this might not have any knowledge about ZFS. Think of my setup as an mdraid pool with a filesystem mounted on it, shared out via NFS. Same basic idea... ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] fence_apc delay?
On 2016-09-02 10:09, Ken Gaillot wrote: On 09/02/2016 08:14 AM, Dan Swartzendruber wrote: So, I was testing my ZFS dual-head JBOD 2-node cluster. Manual failovers worked just fine. I then went to try an acid-test by logging in to node A and doing 'systemctl stop network'. Sure enough, pacemaker told the APC fencing agent to power-cycle node A. The ZFS pool moved to node B as expected. As soon as node A was back up, I migrated the pool/IP back to node A. I *thought* all was okay, until a bit later, I did 'zpool status', and saw checksum errors on both sides of several of the vdevs. After much digging and poking, the only theory I could come up with was that maybe the fencing operation was considered complete too quickly? I googled for examples using this, and the best tutorial I found showed using a power-wait=5, whereas the default seems to be power-wait=0? (this is CentOS 7, btw...) I changed it to use 5 instead That's a reasonable theory -- that's why power_wait is available. It would be nice if there were a page collecting users' experience with the ideal power_wait for various devices. Even better if fence-agents used those values as the defaults. Ken, thanks. FWIW, this is a Dell Poweredge R905. I have no idea how long the power supplies in that thing can keep things going when A/C goes away. Always wary of small sample sizes, but I got filesystem corruption after 1 fencing event with power_wait=0, and none after 3 fencing events with power_wait=5. ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] fence_apc delay?
On 2016-09-03 08:41, Marek Grac wrote: Hi, There are two problems mentioned in the email. 1) power-wait Power-wait is a quite advanced option and there are only few fence devices/agent where it makes sense. And only because the HW/firmware on the device is somewhat broken. Basically, when we execute power ON/OFF operation, we wait for power-wait seconds before we send next command. I don't remember any issue with APC and this kind of problems. 2) the only theory I could come up with was that maybe the fencing operation was considered complete too quickly? That is virtually not possible. Even when power ON/OFF is asynchronous, we test status of device and fence agent wait until status of the plug/VM/... matches what user wants. I think you misunderstood my point (possibly I wasn't clear.) Not saying anything is wrong with either the fencing agent or the PDU, rather, my theory is that if the agent flips the power off, then back on, if the interval it is off is 'too short', possibly a host like the R905 can continue to operate for a couple of seconds, continuing to write data to the disks past the point where the other node begins to do likewise. If power_wait is not the right way to wait, say, 10 seconds to make 100% sure node A is dead as a doornail, what *is* the right way? ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] Antw: Re: fence_apc delay?
On 2016-09-05 03:04, Ulrich Windl wrote: Marek Grac schrieb am 03.09.2016 um 14:41 in Nachricht : Hi, There are two problems mentioned in the email. 1) power-wait Power-wait is a quite advanced option and there are only few fence devices/agent where it makes sense. And only because the HW/firmware on the device is somewhat broken. Basically, when we execute power ON/OFF operation, we wait for power-wait seconds before we send next command. I don't remember any issue with APC and this kind of problems. 2) the only theory I could come up with was that maybe the fencing operation was considered complete too quickly? That is virtually not possible. Even when power ON/OFF is asynchronous, we test status of device and fence agent wait until status of the plug/VM/... matches what user wants. I can imagine that a powerful power supply can deliver up to one second of power even if the mains is disconnected. If the cluster is very quick after fencing, it might be a problem. I'd suggest a 5 to 10 second delay between fencing action and cluster reaction. Ulrich, please see the response I just posted to Marek. Thanks! ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] fence_apc delay?
On 2016-09-06 10:59, Ken Gaillot wrote: [snip] I thought power-wait was intended for this situation, where the node's power supply can survive a brief outage, so a delay is needed to ensure it drains. In any case, I know people are using it for that. Are there any drawbacks to using power-wait for this purpose, even if that wasn't its original intent? Is it just that the "on" will get the delay as well? I can't speak to the first part of your question, but for me the second part is a definite YES. The issue is that I want a long enough delay to be sure the host is D E A D and not writing to the pool anymore; but that delay is now multiplied by 2, and if it gets "too long", vsphere guests can start getting disk I/O errors... *) Configure fence device to not use reboot but OFF, ON Very same to the situation when there are multiple power circuits; you have to switch them all OFF and afterwards turn them ON. FYI, no special configuration is needed for this with recent pacemaker versions. If multiple devices are listed in a topology level, pacemaker will automatically convert reboot requests into all-off-then-all-on. My understanding was that applied to 1.1.14? My CentOS 7 host has pacemaker 1.1.13 :( [snip] ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] fence_apc delay?
On 2016-09-06 10:59, Ken Gaillot wrote: On 09/05/2016 09:38 AM, Marek Grac wrote: Hi, [snip] FYI, no special configuration is needed for this with recent pacemaker versions. If multiple devices are listed in a topology level, pacemaker will automatically convert reboot requests into all-off-then-all-on. Hmmm, thinking about this some more, this just puts me back in the current situation (e.g. having an 'extra' delay.) The issue for me would be having two fencing devices, each of which needs a brief delay to let its target's PS drain. If a single PDU fencing agent does this (with proposed change): power-off wait N seconds power-on that is cool. Unfortunately, with the all-off-then-all-on pacemaker would do, I would get this: power-off node A wait N seconds power-off node B wait N seconds power-on node A power-on node B or am I missing something? If not, seems like it would be nice to have some sort of delay at the pacemaker level. e.g. tell pacemaker to convert a reboot of node A into a 'turn off node A, wait N seconds, turn on node A'? ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
[ClusterLabs] Location constraints for fencing resources?
Posting this as a separate thread from my fence_apc one. As I said in that thread, I created two fence_apc agents, one to fence node A and one to fence node B. Each was configured using a static pcmk node mapping, and constrained to only run on the other node. In the process of testing this, I discovered a bug (feature?) in the LCMC GUI I am using to manage the cluster. To wit: when I click on a fence object, it never seems to fetch the resource constraints (e.g. they show in the GUI as "nothing selected"), so if I changes something (say the power_wait timeout) and then click "Apply", the location constraints are deleted from that fencing resource. I also noticed that if I connect to port 2224 (pcsd), regular resources show me the location constraints, whereas fence resources don't, which is making me wonder if this is not supported? I'm thinking I can set up a pcmk_host_map to tell it which APC outlet manages node A and which manages node B, in which case I can just use one fence_apc resource with a dynamic pcmk host list? Thanks! ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] Location constraints for fencing resources?
On 2016-09-12 10:48, Dan Swartzendruber wrote: Posting this as a separate thread from my fence_apc one. As I said in that thread, I created two fence_apc agents, one to fence node A and one to fence node B. Each was configured using a static pcmk node mapping, and constrained to only run on the other node. In the process of testing this, I discovered a bug (feature?) in the LCMC GUI I am using to manage the cluster. To wit: when I click on a fence object, it never seems to fetch the resource constraints (e.g. they show in the GUI as "nothing selected"), so if I changes something (say the power_wait timeout) and then click "Apply", the location constraints are deleted from that fencing resource. I also noticed that if I connect to port 2224 (pcsd), regular resources show me the location constraints, whereas fence resources don't, which is making me wonder if this is not supported? I'm thinking I can set up a pcmk_host_map to tell it which APC outlet manages node A and which manages node B, in which case I can just use one fence_apc resource with a dynamic pcmk host list? Thanks! Okay, this *seems* to work. e.g. pcmk_host_list has the two hosts. pmck_host_map says nas1:8;nas2:2. The fencing agent was running on nas2. I logged in to nas2 and did 'systemctl stop network'. pacemaker moved the fencing resource to nas1, then power-cycled nas2. Looking good... ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] Location constraints for fencing resources?
On 2016-09-13 00:20, Klaus Wenninger wrote: Location-constraints for fencing-resources are definitely supported and don't just work by accident - if this was the question. On 09/13/2016 02:43 AM, Dan Swartzendruber wrote: On 2016-09-12 10:48, Dan Swartzendruber wrote: Posting this as a separate thread from my fence_apc one. As I said in that thread, I created two fence_apc agents, one to fence node A and one to fence node B. Each was configured using a static pcmk node mapping, and constrained to only run on the other node. In the process of testing this, I discovered a bug (feature?) in the LCMC GUI I am using to manage the cluster. To wit: when I click on a fence object, it never seems to fetch the resource constraints (e.g. they show in the GUI as "nothing selected"), so if I changes something (say the power_wait timeout) and then click "Apply", the location constraints are deleted from that fencing resource. I also noticed that if I connect to port 2224 (pcsd), regular resources show me the location constraints, whereas fence resources don't, which is making me wonder if this is not supported? I'm thinking I can set up a pcmk_host_map to tell it which APC outlet manages node A and which manages node B, in which case I can just use one fence_apc resource with a dynamic pcmk host list? Thanks! Okay, this *seems* to work. e.g. pcmk_host_list has the two hosts. pmck_host_map says nas1:8;nas2:2. The fencing agent was running on nas2. I logged in to nas2 and did 'systemctl stop network'. pacemaker moved the fencing resource to nas1, then power-cycled nas2. Looking good... Basically, yes. I was puzzled, since the web GUI pcsd serves up gives no apparent way to edit location constraints for stonith resources, and LCMC apparently doesn't read them from the config, so if you edit a stonith resource and change anything, then click on "Apply", it nukes any change you *had* made. For the latter, I will open a ticket with the lcmc developer(s). Thanks! ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
[ClusterLabs] where do I find the null fencing device?
I wanted to do some experiments, and the null fencing agent seemed to be just what I wanted. I don't find it anywhere, even after installing fence-agents-all and cluster-glue (this is on CentOS 7, btw...) Thanks... ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
