Re: [ClusterLabs] Two-node cluster fencing
On Sun, 2018-05-13 at 22:58 +0800, Confidential Company wrote: > Hi Casey, > > 1. I tried shutting down my VM while testing, the "ClusterIP" > resource switched automatically onto the standy node(Node2) > 2. I do "systemctl enable corosnyc/pacemaker" - so that after reboot, > corosync and pacemaker will automatically start. > 3. As I turn-on Node1, I experienced downtime (maybe syncing of nodes > will result of downtime), but my cluster still works as expected --> > The active node is still Node2. There shouldn't be any downtime directly as a result of a node joining. However if you didn't set any resource stickiness, it's possible the cluster moved the IP back to the newly joined node. > 4. If I choose ESXI as my fence device, if the physical server goes > down, would it still be reasonable because its on one host? I'm not too familiar with ESXI so hopefully someone else can comment on the specifics there, but in general, with VM fencing, you do still have an issue if the hypervisor that the fence device talks to doesn't respond for whatever reason. If the cluster can't get a confirmation that the VM was fenced, it won't recover any resources from it. One way around this is a fencing topology i.e. a fallback device. You would use VM fencing as the first topology level, so that if it succeeds, that's sufficient, but if it fails, a second topology level (e.g. a power strip connected to the physical machine) would be tried. > > Thanks Casey, I want to understand more about fencing. > > > Regards, > > imnotarobot > > > > > > > > > > > >>Without fencing, if the primary is powered off abruptly (e.g. if > one of your ESX servers crashes), the standby will not become > primary, and you will need to promote it manually. We had exactly > this scenario happen last week with a 2-node cluster. Without > fencing, you don't have high availability. If you don't need high > availability, you probably don't need pacemaker. > > There are instructions for setting up fencing with vmware here: http > s://www.hastexo.com/resources/hints-and-kinks/fencing-vmware- > virtualized-pacemaker-nodes/ > > One note - rather than the SDK, I believe you actually need the CLI > package, which can be found here: https://my.vmware.com/web/vmware/d > etails?downloadGroup=VCLI600&productId=491 > > Good luck - I haven't managed to get it to build yet - vmware gives > you a black box installer script that compiles a bunch of dependent > perl modules, and it ends up getting hung with 100% CPU usage for > days - digging into this further with lsof and friends, it seems to > be prompting for where your apache source code is to compile > mod_perl. Why does it need mod_perl for the CLI?? Anyways, I > haven't managed to get past that roadblock yet. I'm using Ubuntu 16 > so it may happen to just work better on your RHEL instances. If you > have a different ESX version than 6.0, you may have better luck as > well. > > Best wishes, > -- > Casey > > > On May 11, 2018, at 10:31 PM, Confidential Company .com> wrote: > > > > Hi, > > > > This is my setup: > > > > 1. I have Two vMware-ESXI hosts with one virtual machine (RHEL 7.4) > on each. > > 2. On my physical machine, I have four vmnic --> vmnic 0,1 for > uplink going to switchA and switchB --> vmnic 2,3 for heartbeat > corosync traffic (direct connect to other ESXI host) > > 3. I plan on clustering my two virtual machines via corosync and > create a virtual-IP via pacemaker. > > 4. I plan on using the uplink interface for data and totem > interface for corosync packets(heartbeat messages). > > 5. These two virtual machines doesnt need for a shared storage, or > a shared LUN because the application is, by nature, a standalone > application that doesnt need to have a centralized location as it > does not store any data that needs to be synchronized between two > servers. > > 6. I have a PC that only needs to contact the Virtual IP of the > rhel virtual servers. > > 7. Seamless failover from primary to secondary is not required. > > 8. Active/Passive setup > > > > > > Given the setup above, > > 1. Is there any drawbacks? > > 2. Do I need fencing? Can you explain me by giving a scenario on > the above setup? What instances will occur if I didnt put a fence > device? > > 3. If I need a fence device? what fence device you recommend? SAN, > vmWare, or PDU? > > > > > > Thanks, > > > > imnotarobot -- Ken Gaillot ___ Users mailing list: Users@clusterlabs.org https://lists.clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] Two-node cluster fencing
On Sat, 2018-05-12 at 12:51 -0600, Casey & Gina Shobe wrote: > Without fencing, if the primary is powered off abruptly (e.g. if one > of your ESX servers crashes), the standby will not become primary, > and you will need to promote it manually. We had exactly this > scenario happen last week with a 2-node cluster. Without fencing, > you don't have high availability. If you don't need high > availability, you probably don't need pacemaker. To go into this a bit more, the reason it's not safe to operate without fencing is the situation where the active node still has the IP, but has stopped responding for whatever reason (crippling load, failed disk, flaky network card, etc.). If the passive node brought up the IP in this case without first fencing the active node, both VMs would advertise the IP, and packets would (practically speaking) randomly go to one or the other, making any sort of communication impossible. > There are instructions for setting up fencing with vmware here: http > s://www.hastexo.com/resources/hints-and-kinks/fencing-vmware- > virtualized-pacemaker-nodes/ > > One note - rather than the SDK, I believe you actually need the CLI > package, which can be found here: https://my.vmware.com/web/vmware/d > etails?downloadGroup=VCLI600&productId=491 > > Good luck - I haven't managed to get it to build yet - vmware gives > you a black box installer script that compiles a bunch of dependent > perl modules, and it ends up getting hung with 100% CPU usage for > days - digging into this further with lsof and friends, it seems to > be prompting for where your apache source code is to compile > mod_perl. Why does it need mod_perl for the CLI?? Anyways, I > haven't managed to get past that roadblock yet. I'm using Ubuntu 16 > so it may happen to just work better on your RHEL instances. If you > have a different ESX version than 6.0, you may have better luck as > well. > > Best wishes, -- Ken Gaillot ___ Users mailing list: Users@clusterlabs.org https://lists.clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
[ClusterLabs] Two-node cluster fencing
Message: 2 Date: Sun, 13 May 2018 10:20:36 +0300 From: Andrei Borzenkov To: users@clusterlabs.org Subject: Re: [ClusterLabs] Two-node cluster fencing Message-ID: Content-Type: text/plain; charset=utf-8 12.05.2018 07:31, Confidential Company ?: > Hi, > > This is my setup: > > 1. I have Two vMware-ESXI hosts with one virtual machine (RHEL 7.4) on each. > 2. On my physical machine, I have four vmnic --> vmnic 0,1 for uplink going > to switchA and switchB --> vmnic 2,3 for heartbeat corosync traffic (direct > connect to other ESXI host) > 3. I plan on clustering my two virtual machines via corosync and create a > virtual-IP via pacemaker. > 4. I plan on using the uplink interface for data and totem interface for > corosync packets(heartbeat messages). > 5. These two virtual machines doesnt need for a shared storage, or a shared > LUN because the application is, by nature, a standalone application that > doesnt need to have a centralized location as it does not store any data > that needs to be synchronized between two servers. >Then why you need fialover cluster in the first place? Why cannot you have both nodes running at the same time and use virtual server to let clients connect to either of them? *Reply to Andrei: *I want to it to be simple as possible. If I setup them as active/active, i need a loadbalancer (nginx I assume). Or maybe you have suggestions. Thanks bro > 6. I have a PC that only needs to contact the Virtual IP of the rhel > virtual servers. > 7. Seamless failover from primary to secondary is not required.> 8. Active/Passive setup > > > Given the setup above, > 1. Is there any drawbacks? > 2. Do I need fencing? Can you explain me by giving a scenario on the above > setup? What instances will occur if I didnt put a fence device? > 3. If I need a fence device? what fence device you recommend? SAN, vmWare, > or PDU? > > > Thanks, > > imnotarobot > > > > ___ ___ Users mailing list: Users@clusterlabs.org https://lists.clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
[ClusterLabs] Two-node cluster fencing
Hi Casey, 1. I tried shutting down my VM while testing, the "ClusterIP" resource switched automatically onto the standy node(Node2) 2. I do "systemctl enable corosnyc/pacemaker" - so that after reboot, corosync and pacemaker will automatically start. 3. As I turn-on Node1, I experienced downtime (maybe syncing of nodes will result of downtime), but my cluster still works as expected --> The active node is still Node2. 4. If I choose ESXI as my fence device, if the physical server goes down, would it still be reasonable because its on one host? Thanks Casey, I want to understand more about fencing. Regards, imnotarobot >>Without fencing, if the primary is powered off abruptly (e.g. if one of your ESX servers crashes), the standby will not become primary, and you will need to promote it manually. We had exactly this scenario happen last week with a 2-node cluster. Without fencing, you don't have high availability. If you don't need high availability, you probably don't need pacemaker. There are instructions for setting up fencing with vmware here: https://www.hastexo.com/resources/hints-and-kinks/fencing-vmware-virtualized-pacemaker-nodes/ One note - rather than the SDK, I believe you actually need the CLI package, which can be found here: https://my.vmware.com/web/vmware/details?downloadGroup=VCLI600&productId=491 Good luck - I haven't managed to get it to build yet - vmware gives you a black box installer script that compiles a bunch of dependent perl modules, and it ends up getting hung with 100% CPU usage for days - digging into this further with lsof and friends, it seems to be prompting for where your apache source code is to compile mod_perl. Why does it need mod_perl for the CLI?? Anyways, I haven't managed to get past that roadblock yet. I'm using Ubuntu 16 so it may happen to just work better on your RHEL instances. If you have a different ESX version than 6.0, you may have better luck as well. Best wishes, -- Casey > On May 11, 2018, at 10:31 PM, Confidential Company wrote: > > Hi, > > This is my setup: > > 1. I have Two vMware-ESXI hosts with one virtual machine (RHEL 7.4) on each. > 2. On my physical machine, I have four vmnic --> vmnic 0,1 for uplink going to switchA and switchB --> vmnic 2,3 for heartbeat corosync traffic (direct connect to other ESXI host) > 3. I plan on clustering my two virtual machines via corosync and create a virtual-IP via pacemaker. > 4. I plan on using the uplink interface for data and totem interface for corosync packets(heartbeat messages). > 5. These two virtual machines doesnt need for a shared storage, or a shared LUN because the application is, by nature, a standalone application that doesnt need to have a centralized location as it does not store any data that needs to be synchronized between two servers. > 6. I have a PC that only needs to contact the Virtual IP of the rhel virtual servers. > 7. Seamless failover from primary to secondary is not required. > 8. Active/Passive setup > > > Given the setup above, > 1. Is there any drawbacks? > 2. Do I need fencing? Can you explain me by giving a scenario on the above setup? What instances will occur if I didnt put a fence device? > 3. If I need a fence device? what fence device you recommend? SAN, vmWare, or PDU? > > > Thanks, > > imnotarobot ___ Users mailing list: Users@clusterlabs.org https://lists.clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] Two-node cluster fencing
12.05.2018 07:31, Confidential Company пишет: > Hi, > > This is my setup: > > 1. I have Two vMware-ESXI hosts with one virtual machine (RHEL 7.4) on each. > 2. On my physical machine, I have four vmnic --> vmnic 0,1 for uplink going > to switchA and switchB --> vmnic 2,3 for heartbeat corosync traffic (direct > connect to other ESXI host) > 3. I plan on clustering my two virtual machines via corosync and create a > virtual-IP via pacemaker. > 4. I plan on using the uplink interface for data and totem interface for > corosync packets(heartbeat messages). > 5. These two virtual machines doesnt need for a shared storage, or a shared > LUN because the application is, by nature, a standalone application that > doesnt need to have a centralized location as it does not store any data > that needs to be synchronized between two servers. Then why you need fialover cluster in the first place? Why cannot you have both nodes running at the same time and use virtual server to let clients connect to either of them? > 6. I have a PC that only needs to contact the Virtual IP of the rhel > virtual servers. > 7. Seamless failover from primary to secondary is not required.> 8. > Active/Passive setup > > > Given the setup above, > 1. Is there any drawbacks? > 2. Do I need fencing? Can you explain me by giving a scenario on the above > setup? What instances will occur if I didnt put a fence device? > 3. If I need a fence device? what fence device you recommend? SAN, vmWare, > or PDU? > > > Thanks, > > imnotarobot > > > > ___ > Users mailing list: Users@clusterlabs.org > https://lists.clusterlabs.org/mailman/listinfo/users > > Project Home: http://www.clusterlabs.org > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf > Bugs: http://bugs.clusterlabs.org > ___ Users mailing list: Users@clusterlabs.org https://lists.clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] Two-node cluster fencing
Without fencing, if the primary is powered off abruptly (e.g. if one of your ESX servers crashes), the standby will not become primary, and you will need to promote it manually. We had exactly this scenario happen last week with a 2-node cluster. Without fencing, you don't have high availability. If you don't need high availability, you probably don't need pacemaker. There are instructions for setting up fencing with vmware here: https://www.hastexo.com/resources/hints-and-kinks/fencing-vmware-virtualized-pacemaker-nodes/ One note - rather than the SDK, I believe you actually need the CLI package, which can be found here: https://my.vmware.com/web/vmware/details?downloadGroup=VCLI600&productId=491 Good luck - I haven't managed to get it to build yet - vmware gives you a black box installer script that compiles a bunch of dependent perl modules, and it ends up getting hung with 100% CPU usage for days - digging into this further with lsof and friends, it seems to be prompting for where your apache source code is to compile mod_perl. Why does it need mod_perl for the CLI?? Anyways, I haven't managed to get past that roadblock yet. I'm using Ubuntu 16 so it may happen to just work better on your RHEL instances. If you have a different ESX version than 6.0, you may have better luck as well. Best wishes, -- Casey > On May 11, 2018, at 10:31 PM, Confidential Company > wrote: > > Hi, > > This is my setup: > > 1. I have Two vMware-ESXI hosts with one virtual machine (RHEL 7.4) on each. > 2. On my physical machine, I have four vmnic --> vmnic 0,1 for uplink going > to switchA and switchB --> vmnic 2,3 for heartbeat corosync traffic (direct > connect to other ESXI host) > 3. I plan on clustering my two virtual machines via corosync and create a > virtual-IP via pacemaker. > 4. I plan on using the uplink interface for data and totem interface for > corosync packets(heartbeat messages). > 5. These two virtual machines doesnt need for a shared storage, or a shared > LUN because the application is, by nature, a standalone application that > doesnt need to have a centralized location as it does not store any data that > needs to be synchronized between two servers. > 6. I have a PC that only needs to contact the Virtual IP of the rhel virtual > servers. > 7. Seamless failover from primary to secondary is not required. > 8. Active/Passive setup > > > Given the setup above, > 1. Is there any drawbacks? > 2. Do I need fencing? Can you explain me by giving a scenario on the above > setup? What instances will occur if I didnt put a fence device? > 3. If I need a fence device? what fence device you recommend? SAN, vmWare, or > PDU? > > > Thanks, > > imnotarobot > > ___ > Users mailing list: Users@clusterlabs.org > https://lists.clusterlabs.org/mailman/listinfo/users > > Project Home: http://www.clusterlabs.org > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf > Bugs: http://bugs.clusterlabs.org ___ Users mailing list: Users@clusterlabs.org https://lists.clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
[ClusterLabs] Two-node cluster fencing
Hi, This is my setup: 1. I have Two vMware-ESXI hosts with one virtual machine (RHEL 7.4) on each. 2. On my physical machine, I have four vmnic --> vmnic 0,1 for uplink going to switchA and switchB --> vmnic 2,3 for heartbeat corosync traffic (direct connect to other ESXI host) 3. I plan on clustering my two virtual machines via corosync and create a virtual-IP via pacemaker. 4. I plan on using the uplink interface for data and totem interface for corosync packets(heartbeat messages). 5. These two virtual machines doesnt need for a shared storage, or a shared LUN because the application is, by nature, a standalone application that doesnt need to have a centralized location as it does not store any data that needs to be synchronized between two servers. 6. I have a PC that only needs to contact the Virtual IP of the rhel virtual servers. 7. Seamless failover from primary to secondary is not required. 8. Active/Passive setup Given the setup above, 1. Is there any drawbacks? 2. Do I need fencing? Can you explain me by giving a scenario on the above setup? What instances will occur if I didnt put a fence device? 3. If I need a fence device? what fence device you recommend? SAN, vmWare, or PDU? Thanks, imnotarobot ___ Users mailing list: Users@clusterlabs.org https://lists.clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
