Re: [ClusterLabs] Still Beginner STONITH Problem
>With kvm please use the qemu-watchdog and try to >prevent using softdogwith SBD. >Especially if you are aiming for a production-cluster ... You can tell it to the previous company I worked for :D . All clusters were using softdog on SLES 11/12 despite the hardware had it's own. We had no issues with fencing, but we got plenty of san issues to test the fencing :) Best Regards, Strahil Nikolov ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] Still Beginner STONITH Problem
I can't find fence_virtd for Ubuntu18, but it is available for Ubuntu20. Your other option is to get an iSCSI from your quorum system and use that for SBD. For watchdog, you can use 'softdog' kernel module or you can use KVM to present one to the VMs. You can also check the '-P' flag for SBD. Best Regards, Strahil Nikolov На 7 юли 2020 г. 10:11:38 GMT+03:00, "stefan.schm...@farmpartner-tec.com" написа: > >What does 'virsh list' > >give you onthe 2 hosts? Hopefully different names for > >the VMs ... > >Yes, each host shows its own > ># virsh list > IdName Status > > 2 kvm101 running > ># virsh list > IdName State > > 1 kvm102 running > > > > >Did you try 'fence_xvm -a {mcast-ip} -o list' on the > >guests as well? > >fence_xvm sadly does not work on the Ubuntu guests. The howto said to >install "yum install fence-virt fence-virtd" which do not exist as >such >in Ubuntu 18.04. After we tried to find the appropiate packages we >installed "libvirt-clients" and "multipath-tools". Is there maybe >something misisng or completely wrong? >Though we can connect to both hosts using "nc -z -v -u 192.168.1.21 >1229", that just works fine. > > > >Usually, the biggest problem is the multicast traffic - as in many > >environments it can be dropped by firewalls. > >To make sure I have requested our Datacenter techs to verify that >multicast Traffic can move unhindered in our local Network. But in the >past on multiple occasions they have confirmed, that local traffic is >not filtered in any way. But Since now I have never specifically asked >for multicast traffic, which I now did. I am waiting for an answer to >that question. > > >kind regards >Stefan Schmitz > >Am 06.07.2020 um 11:24 schrieb Klaus Wenninger: >> On 7/6/20 10:10 AM, stefan.schm...@farmpartner-tec.com wrote: >>> Hello, >>> > # fence_xvm -o list > kvm102 >bab3749c-15fc-40b7-8b6c-d4267b9f0eb9 > on >>> This should show both VMs, so getting to that point will likely >solve your problem. fence_xvm relies on multicast, there could be some obscure network configuration to get that working on the VMs. >> You said you tried on both hosts. What does 'virsh list' >> give you onthe 2 hosts? Hopefully different names for >> the VMs ... >> Did you try 'fence_xvm -a {mcast-ip} -o list' on the >> guests as well? >> Did you try pinging via the physical network that is >> connected tothe bridge configured to be used for >> fencing? >> If I got it right fence_xvm should supportcollecting >> answersfrom multiple hosts but I found a suggestion >> to do a setup with 2 multicast-addresses & keys for >> each host. >> Which route did you go? >> >> Klaus >>> >>> Thank you for pointing me in that direction. We have tried to solve >>> that but with no success. We were using an howto provided here >>> https://wiki.clusterlabs.org/wiki/Guest_Fencing >>> >>> Problem is, it specifically states that the tutorial does not yet >>> support the case where guests are running on multiple hosts. There >are >>> some short hints what might be necessary to do, but working through >>> those sadly just did not work nor where there any clues which would >>> help us finding a solution ourselves. So now we are completely stuck >>> here. >>> >>> Has someone the same configuration with Guest VMs on multiple hosts? >>> And how did you manage to get that to work? What do we need to do to >>> resolve this? Is there maybe even someone who would be willing to >take >>> a closer look at our server? Any help would be greatly appreciated! >>> >>> Kind regards >>> Stefan Schmitz >>> >>> >>> >>> Am 03.07.2020 um 02:39 schrieb Ken Gaillot: On Thu, 2020-07-02 at 17:18 +0200, >stefan.schm...@farmpartner-tec.com wrote: > Hello, > > I hope someone can help with this problem. We are (still) trying >to > get > Stonith to achieve a running active/active HA Cluster, but sadly >to > no > avail. > > There are 2 Centos Hosts. On each one there is a virtual Ubuntu >VM. > The > Ubuntu VMs are the ones which should form the HA Cluster. > > The current status is this: > > # pcs status > Cluster name: pacemaker_cluster > WARNING: corosync and pacemaker node names do not match (IPs used >in > setup?) > Stack: corosync > Current DC: server2ubuntu1 (version 1.1.18-2b07d5c5a9) - partition > with > quorum > Last updated: Thu Jul 2 17:03:53 2020 > Last change: Thu Jul 2 14:33:14 2020 by root via cibadmin on > server4ubuntu1 > > 2 nodes configured > 13 resources configured > > Online: [ server2ubuntu1 server4ubuntu1 ] > > Full list of resources: > > stonith_id_1 (stonith:external/libvirt):
Re: [ClusterLabs] Still Beginner STONITH Problem
On 7/7/20 11:12 AM, Strahil Nikolov wrote: >> With kvm please use the qemu-watchdog and try to >> prevent using softdogwith SBD. >> Especially if you are aiming for a production-cluster ... > You can tell it to the previous company I worked for :D . > All clusters were using softdog on SLES 11/12 despite the hardware had it's > own. Yes I know opinions regarding softdog do diverge a bit. Going through some possible kernel-paths at least leaves a bad taste. Doesn't mean you will have issues though. Just something where testing won't give you an easy answer. May as well depend heavily on the hardware you are running on. As long as there are better possibilities one should at least consider them. Remember to have defaulted to softdog on a pre-configured product-installer with the documentation stating that softdog has it's shortcomings and an advise to configure something else if available, you know what you are doing and you have tested it (testing if a hardware watchdog actually fires is easy while it is merely impossible to test-verify if softdog is really reliable enough). Klaus > > We had no issues with fencing, but we got plenty of san issues to test the > fencing :) > > Best Regards, > Strahil Nikolov > ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] Still Beginner STONITH Problem
On 7/7/20 10:33 AM, Strahil Nikolov wrote: > I can't find fence_virtd for Ubuntu18, but it is available for Ubuntu20. > > Your other option is to get an iSCSI from your quorum system and use that for > SBD. > For watchdog, you can use 'softdog' kernel module or you can use KVM to > present one to the VMs. > You can also check the '-P' flag for SBD. With kvm please use the qemu-watchdog and try to prevent using softdogwith SBD. Especially if you are aiming for a production-cluster ... Adding something like that to libvirt-xml should do the trick: > > Best Regards, > Strahil Nikolov > > На 7 юли 2020 г. 10:11:38 GMT+03:00, "stefan.schm...@farmpartner-tec.com" > написа: >>> What does 'virsh list' >>> give you onthe 2 hosts? Hopefully different names for >>> the VMs ... >> Yes, each host shows its own >> >> # virsh list >> IdName Status >> >> 2 kvm101 running >> >> # virsh list >> IdName State >> >> 1 kvm102 running >> >> >> >>> Did you try 'fence_xvm -a {mcast-ip} -o list' on the >>> guests as well? >> fence_xvm sadly does not work on the Ubuntu guests. The howto said to >> install "yum install fence-virt fence-virtd" which do not exist as >> such >> in Ubuntu 18.04. After we tried to find the appropiate packages we >> installed "libvirt-clients" and "multipath-tools". Is there maybe >> something misisng or completely wrong? >> Though we can connect to both hosts using "nc -z -v -u 192.168.1.21 >> 1229", that just works fine. >> without fence-virt you can't expect the whole thing to work. maybe you can build it for your ubuntu-version from sources of a package for another ubuntu-version if it doesn't exist yet. btw. which pacemaker-version are you using? There was a convenience-fix on the master-branch for at least a couple of days (sometimes during 2.0.4 release-cycle) that wasn't compatible with fence_xvm. >>> Usually, the biggest problem is the multicast traffic - as in many >>> environments it can be dropped by firewalls. >> To make sure I have requested our Datacenter techs to verify that >> multicast Traffic can move unhindered in our local Network. But in the >> past on multiple occasions they have confirmed, that local traffic is >> not filtered in any way. But Since now I have never specifically asked >> for multicast traffic, which I now did. I am waiting for an answer to >> that question. >> >> >> kind regards >> Stefan Schmitz >> >> Am 06.07.2020 um 11:24 schrieb Klaus Wenninger: >>> On 7/6/20 10:10 AM, stefan.schm...@farmpartner-tec.com wrote: Hello, >> # fence_xvm -o list >> kvm102 >> bab3749c-15fc-40b7-8b6c-d4267b9f0eb9 >> on > This should show both VMs, so getting to that point will likely >> solve > your problem. fence_xvm relies on multicast, there could be some > obscure network configuration to get that working on the VMs. >>> You said you tried on both hosts. What does 'virsh list' >>> give you onthe 2 hosts? Hopefully different names for >>> the VMs ... >>> Did you try 'fence_xvm -a {mcast-ip} -o list' on the >>> guests as well? >>> Did you try pinging via the physical network that is >>> connected tothe bridge configured to be used for >>> fencing? >>> If I got it right fence_xvm should supportcollecting >>> answersfrom multiple hosts but I found a suggestion >>> to do a setup with 2 multicast-addresses & keys for >>> each host. >>> Which route did you go? >>> >>> Klaus Thank you for pointing me in that direction. We have tried to solve that but with no success. We were using an howto provided here https://wiki.clusterlabs.org/wiki/Guest_Fencing Problem is, it specifically states that the tutorial does not yet support the case where guests are running on multiple hosts. There >> are some short hints what might be necessary to do, but working through those sadly just did not work nor where there any clues which would help us finding a solution ourselves. So now we are completely stuck here. Has someone the same configuration with Guest VMs on multiple hosts? And how did you manage to get that to work? What do we need to do to resolve this? Is there maybe even someone who would be willing to >> take a closer look at our server? Any help would be greatly appreciated! Kind regards Stefan Schmitz Am 03.07.2020 um 02:39 schrieb Ken Gaillot: > On Thu, 2020-07-02 at 17:18 +0200, >> stefan.schm...@farmpartner-tec.com > wrote: >> Hello, >> >> I hope someone can help with this problem. We are (still) trying >> to >> get >> Stonith to achieve a running active/active HA Cluster, but sadly >> to >> no >> avail. >> >> There are 2 Centos
Re: [ClusterLabs] Still Beginner STONITH Problem
>What does 'virsh list' >give you onthe 2 hosts? Hopefully different names for >the VMs ... Yes, each host shows its own # virsh list IdName Status 2 kvm101 running # virsh list IdName State 1 kvm102 running >Did you try 'fence_xvm -a {mcast-ip} -o list' on the >guests as well? fence_xvm sadly does not work on the Ubuntu guests. The howto said to install "yum install fence-virt fence-virtd" which do not exist as such in Ubuntu 18.04. After we tried to find the appropiate packages we installed "libvirt-clients" and "multipath-tools". Is there maybe something misisng or completely wrong? Though we can connect to both hosts using "nc -z -v -u 192.168.1.21 1229", that just works fine. >Usually, the biggest problem is the multicast traffic - as in many >environments it can be dropped by firewalls. To make sure I have requested our Datacenter techs to verify that multicast Traffic can move unhindered in our local Network. But in the past on multiple occasions they have confirmed, that local traffic is not filtered in any way. But Since now I have never specifically asked for multicast traffic, which I now did. I am waiting for an answer to that question. kind regards Stefan Schmitz Am 06.07.2020 um 11:24 schrieb Klaus Wenninger: On 7/6/20 10:10 AM, stefan.schm...@farmpartner-tec.com wrote: Hello, # fence_xvm -o list kvm102 bab3749c-15fc-40b7-8b6c-d4267b9f0eb9 on This should show both VMs, so getting to that point will likely solve your problem. fence_xvm relies on multicast, there could be some obscure network configuration to get that working on the VMs. You said you tried on both hosts. What does 'virsh list' give you onthe 2 hosts? Hopefully different names for the VMs ... Did you try 'fence_xvm -a {mcast-ip} -o list' on the guests as well? Did you try pinging via the physical network that is connected tothe bridge configured to be used for fencing? If I got it right fence_xvm should supportcollecting answersfrom multiple hosts but I found a suggestion to do a setup with 2 multicast-addresses & keys for each host. Which route did you go? Klaus Thank you for pointing me in that direction. We have tried to solve that but with no success. We were using an howto provided here https://wiki.clusterlabs.org/wiki/Guest_Fencing Problem is, it specifically states that the tutorial does not yet support the case where guests are running on multiple hosts. There are some short hints what might be necessary to do, but working through those sadly just did not work nor where there any clues which would help us finding a solution ourselves. So now we are completely stuck here. Has someone the same configuration with Guest VMs on multiple hosts? And how did you manage to get that to work? What do we need to do to resolve this? Is there maybe even someone who would be willing to take a closer look at our server? Any help would be greatly appreciated! Kind regards Stefan Schmitz Am 03.07.2020 um 02:39 schrieb Ken Gaillot: On Thu, 2020-07-02 at 17:18 +0200, stefan.schm...@farmpartner-tec.com wrote: Hello, I hope someone can help with this problem. We are (still) trying to get Stonith to achieve a running active/active HA Cluster, but sadly to no avail. There are 2 Centos Hosts. On each one there is a virtual Ubuntu VM. The Ubuntu VMs are the ones which should form the HA Cluster. The current status is this: # pcs status Cluster name: pacemaker_cluster WARNING: corosync and pacemaker node names do not match (IPs used in setup?) Stack: corosync Current DC: server2ubuntu1 (version 1.1.18-2b07d5c5a9) - partition with quorum Last updated: Thu Jul 2 17:03:53 2020 Last change: Thu Jul 2 14:33:14 2020 by root via cibadmin on server4ubuntu1 2 nodes configured 13 resources configured Online: [ server2ubuntu1 server4ubuntu1 ] Full list of resources: stonith_id_1 (stonith:external/libvirt): Stopped Master/Slave Set: r0_pacemaker_Clone [r0_pacemaker] Masters: [ server4ubuntu1 ] Slaves: [ server2ubuntu1 ] Master/Slave Set: WebDataClone [WebData] Masters: [ server2ubuntu1 server4ubuntu1 ] Clone Set: dlm-clone [dlm] Started: [ server2ubuntu1 server4ubuntu1 ] Clone Set: ClusterIP-clone [ClusterIP] (unique) ClusterIP:0 (ocf::heartbeat:IPaddr2): Started server2ubuntu1 ClusterIP:1 (ocf::heartbeat:IPaddr2): Started server4ubuntu1 Clone Set: WebFS-clone [WebFS] Started: [ server4ubuntu1 ] Stopped: [ server2ubuntu1 ] Clone Set: WebSite-clone [WebSite] Started: [ server4ubuntu1 ] Stopped: [ server2ubuntu1 ] Failed Actions: * stonith_id_1_start_0 on server2ubuntu1 'unknown error' (1): call=201, status=Error,