Re: [ClusterLabs] Pacemaker/corosync behavior in case of partial split brain
On 2021-08-05 2:25 p.m., Andrei Borzenkov wrote: Three nodes A, B, C. Communication between A and B is blocked (completely - no packet can come in both direction). A and B can communicate with C. I expected that result will be two partitions - (A, C) and (B, C). To my surprise, A went offline leaving (B, C) running. It was always the same node (with node id 1 if it matters, out of 1, 2, 3). How surviving partition is determined in this case? Can I be sure the same will also work in case of multiple nodes? I.e. if I have two sites with equal number of nodes and the third site as witness and connectivity between multi-node sites is lost but each site can communicate with witness. Will one site go offline? Which one? In your case, your nodes were otherwise healthy so quorum worked. To properly avoid a split brain (when a node is not behaving properly, ie: lockups, bad RAM/CPU, etc) you realy need actual fencing. In such a case, whichever nodes maintain quorum, will fence the lost node (be it because it became inquorate or stopped behaving properly). As for the mechanics of how quorum is determined in your case above, I'll let one of the corosync people decide. -- Digimer Papers and Projects: https://alteeve.com/w/ "I am, somehow, less interested in the weight and convolutions of Einstein’s brain than in the near certainty that people of equal talent have lived and died in cotton fields and sweatshops." - Stephen Jay Gould ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
[ClusterLabs] Pacemaker/corosync behavior in case of partial split brain
Three nodes A, B, C. Communication between A and B is blocked (completely - no packet can come in both direction). A and B can communicate with C. I expected that result will be two partitions - (A, C) and (B, C). To my surprise, A went offline leaving (B, C) running. It was always the same node (with node id 1 if it matters, out of 1, 2, 3). How surviving partition is determined in this case? Can I be sure the same will also work in case of multiple nodes? I.e. if I have two sites with equal number of nodes and the third site as witness and connectivity between multi-node sites is lost but each site can communicate with witness. Will one site go offline? Which one? ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] Sub‑clusters / super‑clusters - working :)
On Thursday 05 August 2021 at 15:44:18, Ulrich Windl wrote: > Hi! > > Nice to hear. What could be "interesting" is how stable the WAN-type of > corosync communication works. Well, between cityA and cityB it should be pretty good, because these are two data centres on opposite sides of England run by the same hosting provider (with private dark fibre between them, not dependent on the Internet). > If it's not that stable, the cluster could try to fence nodes rather > frequently. OK, you disabled fencing; maybe it works without. I'm going to find out :) > Did you tune the parameters? No: a) I only just got it working today :) b) I got it working on a bunch of VMs in my own personal hosting environment; I haven't tried it in the real data centres yet. At the moment I regard it as a Proof of Concept to show that the design works. Antony. -- Heisenberg, Gödel, and Chomsky walk in to a bar. Heisenberg says, "Clearly this is a joke, but how can we work out if it's funny or not?" Gödel replies, "We can't know that because we're inside the joke." Chomsky says, "Of course it's funny. You're just saying it wrong." Please reply to the list; please *don't* CC me. ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
[ClusterLabs] Antw: [EXT] Re: Sub‑clusters / super‑clusters - working :)
Hi! Nice to hear. What could be "interesting" is how stable the WAN-type of corosync communication works. If it's not that stable, the cluster could try to fence nodes rather frequently. OK, you disabled fencing; maybe it works without. Did you tune the parameters? Regards, Ulrich >>> Antony Stone schrieb am 05.08.2021 um 14:44 in Nachricht <202108051444.39919.antony.st...@ha.open.source.it>: > On Thursday 05 August 2021 at 10:51:37, Antony Stone wrote: > >> On Thursday 05 August 2021 at 07:48:37, Ulrich Windl wrote: >> > >> > Have you ever tried to find out why this happens? (Talking about logs) >> >> Not in detail, no, but just in case there's a chance of getting this >> working as suggested simply using location constraints, I shall look >> further. > > I now have a working solution ‑ thank you to everyone who has helped. > > The answer to the problem above was simple ‑ with a 6‑node cluster, 3 votes is > > not quorum. > > I added a 7th node (in "city C") and adjusted the location constraints to > ensure that cluster A resources run in city A, cluster B resources run in > city > B, and the "anywhere" resource runs in either city A or city B. > > I've even added a colocation constraint to ensure that the "anywhere" > resource > runs on the same machine in either city A or city B as is running the local > resources there (which wasn't a strict requirement, but is very useful). > > For anyone interested in the detail of how to do this (without needing > booth), > here is my cluster.conf file, as in "crm configure load replace > cluster.conf": > > > node tom attribute site=cityA > node dick attribute site=cityA > node harry attribute site=cityA > > node fred attribute site=cityB > node george attribute site=cityB > node ron attribute site=cityB > > primitive A‑float IPaddr2 params ip=192.168.32.250 cidr_netmask=24 meta > migration‑threshold=3 failure‑timeout=60 op monitor interval=5 timeout=20 on‑ > fail=restart > primitive B‑float IPaddr2 params ip=192.168.42.250 cidr_netmask=24 meta > migration‑threshold=3 failure‑timeout=60 op monitor interval=5 timeout=20 on‑ > fail=restart > primitive Asterisk asterisk meta migration‑threshold=3 failure‑timeout=60 op > monitor interval=5 timeout=20 on‑fail=restart > > group GroupA A‑float4 resource‑stickiness=100 > group GroupB B‑float4 resource‑stickiness=100 > group Anywhere Asterisk resource‑stickiness=100 > > location pref_A GroupA rule ‑inf: site ne cityA > location pref_B GroupB rule ‑inf: site ne cityB > location no_pref Anywhere rule ‑inf: site ne cityA and site ne cityB > > colocation Ast 100: Anywhere [ cityA cityB ] > > property cib‑bootstrap‑options: stonith‑enabled=no no‑quorum‑policy=stop > start‑failure‑is‑fatal=false cluster‑recheck‑interval=60s > > > Of course, the group definitions are not needed for single resources, but I > shall in practice be using multiple resources which do need groups, so I > wanted to ensure I was creating something which would work with that. > > I have tested it by: > > ‑ bringing up one node at a time: as soon as any 4 nodes are running, all > possible resources are running > > ‑ bringing up 5 or more nodes: all resources run > > ‑ taking down one node at a time to a maximum of three nodes offline: if at > least one node in a given city is running, the resources at that city are > running > > ‑ turning off (using "halt", so that corosync dies nicely) all three nodes > in > a city simultaneously: that city's resources stop running, the other city > continues working, as well as the "anywhere" resource > > ‑ causing a network failure at one city (so it simply disappears without > stopping corosync neatly): the other city continues its resources (plus the > "anywhere" resource), the isolated city stops > > For me, this is the solution I wanted, and in fact it's even slightly better > > than the previous two isolated 3‑node clusters I had, because I can now have > resources running on a single active node in cityA (provided it can see at > least 3 other nodes in cityB or cityC), which wasn't possible before. > > > Once again, thanks to everyone who has helped me to achieve this result :) > > > Antony. > > ‑‑ > "The future is already here. It's just not evenly distributed yet." > > ‑ William Gibson > >Please reply to the list; > please *don't* CC > me. > ___ > Manage your subscription: > https://lists.clusterlabs.org/mailman/listinfo/users > > ClusterLabs home: https://www.clusterlabs.org/ ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] Sub‑clusters / super‑clusters - working :)
On Thursday 05 August 2021 at 10:51:37, Antony Stone wrote: > On Thursday 05 August 2021 at 07:48:37, Ulrich Windl wrote: > > > > Have you ever tried to find out why this happens? (Talking about logs) > > Not in detail, no, but just in case there's a chance of getting this > working as suggested simply using location constraints, I shall look > further. I now have a working solution - thank you to everyone who has helped. The answer to the problem above was simple - with a 6-node cluster, 3 votes is not quorum. I added a 7th node (in "city C") and adjusted the location constraints to ensure that cluster A resources run in city A, cluster B resources run in city B, and the "anywhere" resource runs in either city A or city B. I've even added a colocation constraint to ensure that the "anywhere" resource runs on the same machine in either city A or city B as is running the local resources there (which wasn't a strict requirement, but is very useful). For anyone interested in the detail of how to do this (without needing booth), here is my cluster.conf file, as in "crm configure load replace cluster.conf": node tom attribute site=cityA node dick attribute site=cityA node harry attribute site=cityA node fred attribute site=cityB node george attribute site=cityB node ron attribute site=cityB primitive A-float IPaddr2 params ip=192.168.32.250 cidr_netmask=24 meta migration-threshold=3 failure-timeout=60 op monitor interval=5 timeout=20 on- fail=restart primitive B-float IPaddr2 params ip=192.168.42.250 cidr_netmask=24 meta migration-threshold=3 failure-timeout=60 op monitor interval=5 timeout=20 on- fail=restart primitive Asterisk asterisk meta migration-threshold=3 failure-timeout=60 op monitor interval=5 timeout=20 on-fail=restart group GroupA A-float4 resource-stickiness=100 group GroupB B-float4 resource-stickiness=100 group Anywhere Asterisk resource-stickiness=100 location pref_A GroupA rule -inf: site ne cityA location pref_B GroupB rule -inf: site ne cityB location no_pref Anywhere rule -inf: site ne cityA and site ne cityB colocation Ast 100: Anywhere [ cityA cityB ] property cib-bootstrap-options: stonith-enabled=no no-quorum-policy=stop start-failure-is-fatal=false cluster-recheck-interval=60s Of course, the group definitions are not needed for single resources, but I shall in practice be using multiple resources which do need groups, so I wanted to ensure I was creating something which would work with that. I have tested it by: - bringing up one node at a time: as soon as any 4 nodes are running, all possible resources are running - bringing up 5 or more nodes: all resources run - taking down one node at a time to a maximum of three nodes offline: if at least one node in a given city is running, the resources at that city are running - turning off (using "halt", so that corosync dies nicely) all three nodes in a city simultaneously: that city's resources stop running, the other city continues working, as well as the "anywhere" resource - causing a network failure at one city (so it simply disappears without stopping corosync neatly): the other city continues its resources (plus the "anywhere" resource), the isolated city stops For me, this is the solution I wanted, and in fact it's even slightly better than the previous two isolated 3-node clusters I had, because I can now have resources running on a single active node in cityA (provided it can see at least 3 other nodes in cityB or cityC), which wasn't possible before. Once again, thanks to everyone who has helped me to achieve this result :) Antony. -- "The future is already here. It's just not evenly distributed yet." - William Gibson Please reply to the list; please *don't* CC me. ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] Pacemaker problems with pingd
On Wed, Aug 4, 2021 at 5:30 PM Janusz Jaskiewicz < janusz.jaskiew...@gmail.com> wrote: > Hello. > > Please forgive the length of this email but I wanted to provide as much > details as possible. > > I'm trying to set up a cluster of two nodes for my service. > I have a problem with a scenario where the network between two nodes gets > broken and they can no longer see each other. > This causes split-brain. > I know that proper way of implementing this would be to employ STONITH, > but it is not feasible for me now (I don't have necessary hardware support > and I don't want to introduce another point of failure by introducing > shared storage based STONITH). > > In order to work-around the split-brain scenario I introduced pingd to my > cluster, which in theory should do what I expect. > pingd pings a network device, so when the NIC is broken on one of my > nodes, this node should not run the resources because pingd would fail for > it. > As we've discussed on this list in multiple previous threads already there are lots of failure scenarios where cluster-nodes don't see each other but both can ping something else on the network. Important cases where your approach wouldn't work are as well those where nodes are just partially alive - leads to corosync membership being lost & node not able to stop resources properly anymore. Thus it is highly recommended to have all these setups that rely on some kind of self-fencing or bringing down of resources within some timeout being guarded by a (hardware)-watchdog. Previously you probably were referring to SBD which implements such a watchdog-guarded approach. As you've probably figured out you can't directly use SBD in a 2-node-setup without a shared-disk. Pure watchdog-fencing needs quorum decision made by at least 3 instances. If you don't want a full blown 3rd node you can consider qdevice - can be used by multiple 2-node-clusters for quorum evaluation. Otherwise you can use SBD with a shared disk. You are right that both, a shared disk and any kind of 3rd node are an additional point of failure. Important is that in both cases we are talking about a point of failure but not of a single point of failure - meaning it failing it would not necessarily impose services to be shutdown. Klaus > > pingd resource is configured to update the value of variable 'pingd' > (interval: 5s, dampen: 3s, multiplier:1000). > Based on the value of pingd I have a location constraint which sets score > to -INFINITY for resource DimProdClusterIP when 'pingd' is not 1000. > All other resources are colocated with DimProdClusterIP, and > DimProdClusterIP should start before all other resources. > > Based on that setup I would expect that when the resources run on > dimprod01 and I disconnect dimprod02 from the network, the resources will > not start on dimprod02. > Unfortunately I see that after a token interval + consensus interval my > resources are brought up for a moment and then go down again. > This is undesirable, as it causes DRBD split-brain inconsistency and > cluster IP may also be taken over by the node which is down. > > I tried to debug it, but I can't figure out why it doesn't work. > I would appreciate any help/pointers. > > > Following are some details of my setup and snippet of pacemaker logs with > comments: > > Setup details: > > pcs status: > Cluster name: dimprodcluster > Cluster Summary: > * Stack: corosync > * Current DC: dimprod02 (version 2.0.5-9.el8_4.1-ba59be7122) - partition > with quorum > * Last updated: Tue Aug 3 08:20:32 2021 > * Last change: Mon Aug 2 18:24:39 2021 by root via cibadmin on > dimprod01 > * 2 nodes configured > * 8 resource instances configured > > Node List: > * Online: [ dimprod01 dimprod02 ] > > Full List of Resources: > * DimProdClusterIP (ocf::heartbeat:IPaddr2): Started dimprod01 > * WyrDimProdServer (systemd:wyr-dim): Started dimprod01 > * Clone Set: WyrDimProdServerData-clone [WyrDimProdServerData] > (promotable): > * Masters: [ dimprod01 ] > * Slaves: [ dimprod02 ] > * WyrDimProdFS (ocf::heartbeat:Filesystem): Started dimprod01 > * DimTestClusterIP (ocf::heartbeat:IPaddr2): Started dimprod01 > * Clone Set: ping-clone [ping]: > * Started: [ dimprod01 dimprod02 ] > > Daemon Status: > corosync: active/enabled > pacemaker: active/enabled > pcsd: active/enabled > > > pcs constraint > Location Constraints: > Resource: DimProdClusterIP > Constraint: location-DimProdClusterIP > Rule: score=-INFINITY > Expression: pingd ne 1000 > Ordering Constraints: > start DimProdClusterIP then promote WyrDimProdServerData-clone > (kind:Mandatory) > promote WyrDimProdServerData-clone then start WyrDimProdFS > (kind:Mandatory) > start WyrDimProdFS then start WyrDimProdServer (kind:Mandatory) > start WyrDimProdServer then start DimTestClusterIP (kind:Mandatory) > Colocation Constraints: > WyrDimProdServer with DimProdClusterIP (score:INFINITY) > DimTestClusterIP with
Re: [ClusterLabs] Antw: [EXT] Re: Sub‑clusters / super‑clusters?
On 05/08/2021 00:11, Frank D. Engel, Jr. wrote: In theory if you could have an independent voting infrastructure among the three clusters which serves to effectively create a second cluster infrastructure interconnecting them to support resource D, you could Yes. It's called booth. have D running on one of the clusters so long as at least two of them can communicate with each other. In other words, give each cluster one vote, then as long as two of them can communicate there are two votes which makes quorum, thus resource D can run on one of those two clusters. If all three clusters lose contact with each other, then D still cannot safely run. To keep the remaining resources working when contact is lost between the clusters, the vote for this would need to be independent of the vote within each individual cluster, effectively meaning that each node would belong to two clusters at once: its own local cluster (A/B/C) plus a "global" cluster spread across the three locations. I don't know offhand if that is readily possible to support with the current software. On 8/4/21 5:01 PM, Antony Stone wrote: On Wednesday 04 August 2021 at 22:06:39, Frank D. Engel, Jr. wrote: There is no safe way to do what you are trying to do. If the resource is on cluster A and contact is lost between clusters A and B due to a network failure, how does cluster B know if the resource is still running on cluster A or not? It has no way of knowing if cluster A is even up and running. In that situation it cannot safely start the resource. I am perfectly happy to have an additional machine at a third location in order to avoid this split-brain between two clusters. However, what I cannot have is for the resources which should be running on cluster A to get started on cluster B. If cluster A is down, then its resources should simply not run - as happens right now with two independent clusters. Suppose for a moment I had three clusters at three locations: A, B and C. Is there a method by which I can have: 1. Cluster A resources running on cluster A if cluster A is functional and not running anywhere if cluster A is non-functional. 2. Cluster B resources running on cluster B if cluster B is functional and not running anywhere if cluster B is non-functional. 3. Cluster C resources running on cluster C if cluster C is functional and not running anywhere if cluster C is non-functional. 4. Resource D running _somewhere_ on clusters A, B or C, but only a single instance of D at a single location at any time. Requirements 1, 2 and 3 are easy to achieve - don't connect the clusters. Requirement 4 is the one I'm stuck with how to implement. If the three nodes comprising cluster A can manage resources such that they run on only one of the three nodes at any time, surely there must be a way of doing the same thing with a resource running on one of three clusters? Antony. ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/ ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] Antw: [EXT] Re: Sub‑clusters / super‑clusters?
On Thursday 05 August 2021 at 07:43:30, Andrei Borzenkov wrote: > On 05.08.2021 00:01, Antony Stone wrote: > > > > Requirements 1, 2 and 3 are easy to achieve - don't connect the clusters. > > > > Requirement 4 is the one I'm stuck with how to implement. > > You either have single cluster and define appropriate location > constraints or you have multiple clusters and configure geo-cluster on > top of them. But you already have been told it multiple times. > > > If the three nodes comprising cluster A can manage resources such that > > they run on only one of the three nodes at any time, surely there must > > be a way of doing the same thing with a resource running on one of three > > clusters? > > You need something that coordinates resources between three clusters and > that is booth. Indeed: On Wednesday 04 August 2021 at 12:48:37, Antony Stone wrote: > I'm going to look into booth as suggested by others. Thanks, Antony. -- +++ Divide By Cucumber Error. Please Reinstall Universe And Reboot +++ Please reply to the list; please *don't* CC me. ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] Antw: Re: Antw: [EXT] Re: Sub‑clusters / super‑clusters?
On Thursday 05 August 2021 at 07:48:37, Ulrich Windl wrote: > Antony Stone schrieb am 04.08.2021 um 21:27: > > > > As soon as I connect the clusters at city A and city B, and apply the > > location contraints and weighting rules you have suggested: > > > > 1. everything works, including the single resource at either city A or > > city B, so long as both clusters are operational. > > > > 2. as soon as one cluster fails (all three of its nodes nodes become > > unavailable), then the other cluster stops running all its resources as > > well. This is even with quorum=2. > > Have you ever tried to find out why this happens? (Talking about logs) Not in detail, no, but just in case there's a chance of getting this working as suggested simply using location constraints, I shall look further. Thanks, Antony. -- This sentence contains exacly three erors. Please reply to the list; please *don't* CC me. ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
[ClusterLabs] Antw: Antw: Re: Antw: [EXT] Re: Sub‑clusters / super‑clusters?
>>> "Ulrich Windl" schrieb am 05.08.2021 um 08:03 in Nachricht <610b7f1d02a100042...@gwsmtp.uni-regensburg.de>: "Frank D. Engel, Jr." schrieb am 05.08.2021 um 00:11 in > Nachricht : >> In theory if you could have an independent voting infrastructure among >> the three clusters which serves to effectively create a second cluster >> infrastructure interconnecting them to support resource D, you could >> have D running on one of the clusters so long as at least two of them >> can communicate with each other. > > Hi! > Sorry, pre-coffeine time, so many mis-spellings: > That's what I thought too, BUT: > If yoiu have some common (NAS) storage where each cluster sends a "proof of "yoiu" is "you" > life" periodically, what will happen if the network is down? > Each cluster will thing the other is dead, so no help. "thing" = "think" > Maybe it's time for an independent communication channel. > Maybe quorum‑voting via SMS or packet radio? ;‑) > > Regards, > Ulrich >> >> >> In other words, give each cluster one vote, then as long as two of them >> can communicate there are two votes which makes quorum, thus resource D >> can run on one of those two clusters. >> >> If all three clusters lose contact with each other, then D still cannot >> safely run. >> >> >> To keep the remaining resources working when contact is lost between the >> clusters, the vote for this would need to be independent of the vote >> within each individual cluster, effectively meaning that each node would >> belong to two clusters at once: its own local cluster (A/B/C) plus a >> "global" cluster spread across the three locations. I don't know >> offhand if that is readily possible to support with the current software. >> >> >> On 8/4/21 5:01 PM, Antony Stone wrote: >>> On Wednesday 04 August 2021 at 22:06:39, Frank D. Engel, Jr. wrote: >>> There is no safe way to do what you are trying to do. If the resource is on cluster A and contact is lost between clusters A and B due to a network failure, how does cluster B know if the resource is still running on cluster A or not? It has no way of knowing if cluster A is even up and running. In that situation it cannot safely start the resource. >>> I am perfectly happy to have an additional machine at a third location in >>> order to avoid this split‑brain between two clusters. >>> >>> However, what I cannot have is for the resources which should be running on >>> cluster A to get started on cluster B. >>> >>> If cluster A is down, then its resources should simply not run ‑ as happens >>> right now with two independent clusters. >>> >>> Suppose for a moment I had three clusters at three locations: A, B and C. >>> >>> Is there a method by which I can have: >>> >>> 1. Cluster A resources running on cluster A if cluster A is functional and >> not >>> running anywhere if cluster A is non‑functional. >>> >>> 2. Cluster B resources running on cluster B if cluster B is functional and >> not >>> running anywhere if cluster B is non‑functional. >>> >>> 3. Cluster C resources running on cluster C if cluster C is functional and >> not >>> running anywhere if cluster C is non‑functional. >>> >>> 4. Resource D running _somewhere_ on clusters A, B or C, but only a single >>> instance of D at a single location at any time. >>> >>> Requirements 1, 2 and 3 are easy to achieve ‑ don't connect the clusters. >>> >>> Requirement 4 is the one I'm stuck with how to implement. >>> >>> If the three nodes comprising cluster A can manage resources such that they >>> run on only one of the three nodes at any time, surely there must be a way >> of >>> doing the same thing with a resource running on one of three clusters? >>> >>> >>> Antony. >>> >> >> ___ >> Manage your subscription: >> https://lists.clusterlabs.org/mailman/listinfo/users >> >> ClusterLabs home: https://www.clusterlabs.org/ > > > > > ___ > Manage your subscription: > https://lists.clusterlabs.org/mailman/listinfo/users > > ClusterLabs home: https://www.clusterlabs.org/ ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
[ClusterLabs] Antw: Re: Antw: [EXT] Re: Sub‑clusters / super‑clusters?
>>> "Frank D. Engel, Jr." schrieb am 05.08.2021 um 00:11 in Nachricht : > In theory if you could have an independent voting infrastructure among > the three clusters which serves to effectively create a second cluster > infrastructure interconnecting them to support resource D, you could > have D running on one of the clusters so long as at least two of them > can communicate with each other. Hi! That's what I thought too, BUT: If yoiu have some common (NAS) storage where each cluster sends a "proof of life" periodically, what will happen if the network is down? Each cluster will thing the other is dead, so no help. Maybe it's time for an independent communication channel. Maybe quorum-voting via SMS or packet radio? ;-) Regards, Ulrich > > > In other words, give each cluster one vote, then as long as two of them > can communicate there are two votes which makes quorum, thus resource D > can run on one of those two clusters. > > If all three clusters lose contact with each other, then D still cannot > safely run. > > > To keep the remaining resources working when contact is lost between the > clusters, the vote for this would need to be independent of the vote > within each individual cluster, effectively meaning that each node would > belong to two clusters at once: its own local cluster (A/B/C) plus a > "global" cluster spread across the three locations. I don't know > offhand if that is readily possible to support with the current software. > > > On 8/4/21 5:01 PM, Antony Stone wrote: >> On Wednesday 04 August 2021 at 22:06:39, Frank D. Engel, Jr. wrote: >> >>> There is no safe way to do what you are trying to do. >>> >>> If the resource is on cluster A and contact is lost between clusters A >>> and B due to a network failure, how does cluster B know if the resource >>> is still running on cluster A or not? >>> >>> It has no way of knowing if cluster A is even up and running. >>> >>> In that situation it cannot safely start the resource. >> I am perfectly happy to have an additional machine at a third location in >> order to avoid this split-brain between two clusters. >> >> However, what I cannot have is for the resources which should be running on >> cluster A to get started on cluster B. >> >> If cluster A is down, then its resources should simply not run - as happens >> right now with two independent clusters. >> >> Suppose for a moment I had three clusters at three locations: A, B and C. >> >> Is there a method by which I can have: >> >> 1. Cluster A resources running on cluster A if cluster A is functional and > not >> running anywhere if cluster A is non-functional. >> >> 2. Cluster B resources running on cluster B if cluster B is functional and > not >> running anywhere if cluster B is non-functional. >> >> 3. Cluster C resources running on cluster C if cluster C is functional and > not >> running anywhere if cluster C is non-functional. >> >> 4. Resource D running _somewhere_ on clusters A, B or C, but only a single >> instance of D at a single location at any time. >> >> Requirements 1, 2 and 3 are easy to achieve - don't connect the clusters. >> >> Requirement 4 is the one I'm stuck with how to implement. >> >> If the three nodes comprising cluster A can manage resources such that they >> run on only one of the three nodes at any time, surely there must be a way > of >> doing the same thing with a resource running on one of three clusters? >> >> >> Antony. >> > > ___ > Manage your subscription: > https://lists.clusterlabs.org/mailman/listinfo/users > > ClusterLabs home: https://www.clusterlabs.org/ ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
