Re: [ClusterLabs] Gracefully stop nodes one by one with disk-less sbd
On 8/12/19 9:24 PM, Klaus Wenninger wrote: [...] > If you shutdown solely pacemaker one-by-one on all nodes > and these shutdowns are considered graceful then you are > not gonna experience any reboots (e.g. 3 node cluster). While revisit what you said, then run `systemctl stop pacemaker` one by one. At this point corosync is still running on all nodes. ( NOTE: make sure no "StopWhenUnneded=yes" in corosync.service to get hijacked when stopping pacemaker. ) > Afterwards you can shutdown corosync one-by-one as well > without experiencing reboots as without the cib-connection > sbd isn't gonna check for quorum anymore (all resources > down so no need to reboot in case of quorum-loss - extra > care has to be taken care of with unmanaged resources but > that isn't particular with sbd). > Then, `systemctl stop corosync` one by one. Nice! disk-less sbd does play the trick as above. All nodes stay! Thanks, Roger ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] Gracefully stop nodes one by one with disk-less sbd
On 8/12/19 3:24 PM, Klaus Wenninger wrote: > On 8/12/19 2:30 PM, Yan Gao wrote: >> Hi Klaus, >> >> On 8/12/19 1:39 PM, Klaus Wenninger wrote: >>> On 8/9/19 9:06 PM, Yan Gao wrote: On 8/9/19 6:40 PM, Andrei Borzenkov wrote: > 09.08.2019 16:34, Yan Gao пишет: >> Hi, >> >> With disk-less sbd, it's fine to stop cluster service from the cluster >> nodes all at the same time. >> >> But if to stop the nodes one by one, for example with a 3-node cluster, >> after stopping the 2nd node, the only remaining node resets itself with: >> > That is sort of documented in SBD manual page: > > --><-- > However, while the cluster is in such a degraded state, it can > neither successfully fence nor be shutdown cleanly (as taking the > cluster below the quorum threshold will immediately cause all remaining > nodes to self-fence). > --><-- > > SBD in shared-nothing mode is basically always in such degraded state > and cannot tolerate loss of quorum. Well, the context here is it loses quorum *expectedly* since the other nodes gracefully shut down. >> Aug 09 14:30:20 opensuse150-1 sbd[1079]: pcmk:debug: >> notify_parent: Not notifying parent: state transient (2) >> Aug 09 14:30:20 opensuse150-1 sbd[1080]:cluster:debug: >> notify_parent: Notifying parent: healthy >> Aug 09 14:30:20 opensuse150-1 sbd[1078]: warning: inquisitor_child: >> Latency: No liveness for 4 s exceeds threshold of 3 s (healthy servants: >> 0) >> >> I can think of the way to manipulate quorum with last_man_standing and >> potentially also auto_tie_breaker, not to mention >> last_man_standing_window would also be a factor... But is there a better >> solution? >> > Lack of cluster wide shutdown mode was mentioned more than once on this > list. I guess the only workaround is to use higher level tools which > basically simply try to stop cluster on all nodes at once. It is still > susceptible to race condition. Gracefully stopping nodes one by one on purpose is still a reasonable need though ... >>> If you do the teardown as e.g. pcs is doing it - first tear down >>> pacemaker-instances and then corosync/sbd - it is at >>> least possible to tear down the pacemaker-instances one-by one >>> without risking a reboot due to quorum-loss. >>> With kind of current sbd having in >>> - >>> https://github.com/ClusterLabs/sbd/commit/824fe834c67fb7bae7feb87607381f9fa8fa2945 >>> - >>> https://github.com/ClusterLabs/sbd/commit/79b778debfee5b4ab2d099b2bfc7385f45597f70 >>> - >>> https://github.com/ClusterLabs/sbd/commit/a716a8ddd3df615009bcff3bd96dd9ae64cb5f68 >>> this should be pretty robust although we are still thinking >>> (probably together with some heartbeat to pacemakerd >>> that assures pacemakerd is checking liveness of sub-daemons >>> properly) of having a cleaner way to detect graceful >>> pacemaker-shutdown. >> These are all good improvements, thanks! >> >> But in this case the remaining node is not shutting down yet, or it's >> intentionally not being shut down :-) Loss of quorum is as expected, so >> is following no-quorum-policy, but self-reset is probably too much? > Hmm ... not sure if I can follow ... > If you shutdown solely pacemaker one-by-one on all nodes > and these shutdowns are considered graceful then you are > not gonna experience any reboots (e.g. 3 node cluster). > Afterwards you can shutdown corosync one-by-one as well > without experiencing reboots as without the cib-connection > sbd isn't gonna check for quorum anymore (all resources > down so no need to reboot in case of quorum-loss - extra > care has to be taken care of with unmanaged resources but > that isn't particular with sbd). I meant if users would like shut down only 2 out of 3 nodes in the cluster and keep the last one online and alive, it's simply not possible for now, although the loss of quorum is expected. Regards, Yan ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] Gracefully stop nodes one by one with disk-less sbd
On 8/12/19 2:30 PM, Yan Gao wrote: > Hi Klaus, > > On 8/12/19 1:39 PM, Klaus Wenninger wrote: >> On 8/9/19 9:06 PM, Yan Gao wrote: >>> On 8/9/19 6:40 PM, Andrei Borzenkov wrote: 09.08.2019 16:34, Yan Gao пишет: > Hi, > > With disk-less sbd, it's fine to stop cluster service from the cluster > nodes all at the same time. > > But if to stop the nodes one by one, for example with a 3-node cluster, > after stopping the 2nd node, the only remaining node resets itself with: > That is sort of documented in SBD manual page: --><-- However, while the cluster is in such a degraded state, it can neither successfully fence nor be shutdown cleanly (as taking the cluster below the quorum threshold will immediately cause all remaining nodes to self-fence). --><-- SBD in shared-nothing mode is basically always in such degraded state and cannot tolerate loss of quorum. >>> Well, the context here is it loses quorum *expectedly* since the other >>> nodes gracefully shut down. >>> > Aug 09 14:30:20 opensuse150-1 sbd[1079]: pcmk:debug: > notify_parent: Not notifying parent: state transient (2) > Aug 09 14:30:20 opensuse150-1 sbd[1080]:cluster:debug: > notify_parent: Notifying parent: healthy > Aug 09 14:30:20 opensuse150-1 sbd[1078]: warning: inquisitor_child: > Latency: No liveness for 4 s exceeds threshold of 3 s (healthy servants: > 0) > > I can think of the way to manipulate quorum with last_man_standing and > potentially also auto_tie_breaker, not to mention > last_man_standing_window would also be a factor... But is there a better > solution? > Lack of cluster wide shutdown mode was mentioned more than once on this list. I guess the only workaround is to use higher level tools which basically simply try to stop cluster on all nodes at once. It is still susceptible to race condition. >>> Gracefully stopping nodes one by one on purpose is still a reasonable >>> need though ... >> If you do the teardown as e.g. pcs is doing it - first tear down >> pacemaker-instances and then corosync/sbd - it is at >> least possible to tear down the pacemaker-instances one-by one >> without risking a reboot due to quorum-loss. >> With kind of current sbd having in >> - >> https://github.com/ClusterLabs/sbd/commit/824fe834c67fb7bae7feb87607381f9fa8fa2945 >> - >> https://github.com/ClusterLabs/sbd/commit/79b778debfee5b4ab2d099b2bfc7385f45597f70 >> - >> https://github.com/ClusterLabs/sbd/commit/a716a8ddd3df615009bcff3bd96dd9ae64cb5f68 >> this should be pretty robust although we are still thinking >> (probably together with some heartbeat to pacemakerd >> that assures pacemakerd is checking liveness of sub-daemons >> properly) of having a cleaner way to detect graceful >> pacemaker-shutdown. > These are all good improvements, thanks! > > But in this case the remaining node is not shutting down yet, or it's > intentionally not being shut down :-) Loss of quorum is as expected, so > is following no-quorum-policy, but self-reset is probably too much? Hmm ... not sure if I can follow ... If you shutdown solely pacemaker one-by-one on all nodes and these shutdowns are considered graceful then you are not gonna experience any reboots (e.g. 3 node cluster). Afterwards you can shutdown corosync one-by-one as well without experiencing reboots as without the cib-connection sbd isn't gonna check for quorum anymore (all resources down so no need to reboot in case of quorum-loss - extra care has to be taken care of with unmanaged resources but that isn't particular with sbd). Klaus > > Regards, >Yan ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] Gracefully stop nodes one by one with disk-less sbd
Hi Klaus, On 8/12/19 1:39 PM, Klaus Wenninger wrote: > On 8/9/19 9:06 PM, Yan Gao wrote: >> On 8/9/19 6:40 PM, Andrei Borzenkov wrote: >>> 09.08.2019 16:34, Yan Gao пишет: Hi, With disk-less sbd, it's fine to stop cluster service from the cluster nodes all at the same time. But if to stop the nodes one by one, for example with a 3-node cluster, after stopping the 2nd node, the only remaining node resets itself with: >>> That is sort of documented in SBD manual page: >>> >>> --><-- >>> However, while the cluster is in such a degraded state, it can >>> neither successfully fence nor be shutdown cleanly (as taking the >>> cluster below the quorum threshold will immediately cause all remaining >>> nodes to self-fence). >>> --><-- >>> >>> SBD in shared-nothing mode is basically always in such degraded state >>> and cannot tolerate loss of quorum. >> Well, the context here is it loses quorum *expectedly* since the other >> nodes gracefully shut down. >> >>> Aug 09 14:30:20 opensuse150-1 sbd[1079]: pcmk:debug: notify_parent: Not notifying parent: state transient (2) Aug 09 14:30:20 opensuse150-1 sbd[1080]:cluster:debug: notify_parent: Notifying parent: healthy Aug 09 14:30:20 opensuse150-1 sbd[1078]: warning: inquisitor_child: Latency: No liveness for 4 s exceeds threshold of 3 s (healthy servants: 0) I can think of the way to manipulate quorum with last_man_standing and potentially also auto_tie_breaker, not to mention last_man_standing_window would also be a factor... But is there a better solution? >>> Lack of cluster wide shutdown mode was mentioned more than once on this >>> list. I guess the only workaround is to use higher level tools which >>> basically simply try to stop cluster on all nodes at once. It is still >>> susceptible to race condition. >> Gracefully stopping nodes one by one on purpose is still a reasonable >> need though ... > If you do the teardown as e.g. pcs is doing it - first tear down > pacemaker-instances and then corosync/sbd - it is at > least possible to tear down the pacemaker-instances one-by one > without risking a reboot due to quorum-loss. > With kind of current sbd having in > - > https://github.com/ClusterLabs/sbd/commit/824fe834c67fb7bae7feb87607381f9fa8fa2945 > - > https://github.com/ClusterLabs/sbd/commit/79b778debfee5b4ab2d099b2bfc7385f45597f70 > - > https://github.com/ClusterLabs/sbd/commit/a716a8ddd3df615009bcff3bd96dd9ae64cb5f68 > this should be pretty robust although we are still thinking > (probably together with some heartbeat to pacemakerd > that assures pacemakerd is checking liveness of sub-daemons > properly) of having a cleaner way to detect graceful > pacemaker-shutdown. These are all good improvements, thanks! But in this case the remaining node is not shutting down yet, or it's intentionally not being shut down :-) Loss of quorum is as expected, so is following no-quorum-policy, but self-reset is probably too much? Regards, Yan ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] Gracefully stop nodes one by one with disk-less sbd
On 8/9/19 9:06 PM, Yan Gao wrote: > On 8/9/19 6:40 PM, Andrei Borzenkov wrote: >> 09.08.2019 16:34, Yan Gao пишет: >>> Hi, >>> >>> With disk-less sbd, it's fine to stop cluster service from the cluster >>> nodes all at the same time. >>> >>> But if to stop the nodes one by one, for example with a 3-node cluster, >>> after stopping the 2nd node, the only remaining node resets itself with: >>> >> That is sort of documented in SBD manual page: >> >> --><-- >> However, while the cluster is in such a degraded state, it can >> neither successfully fence nor be shutdown cleanly (as taking the >> cluster below the quorum threshold will immediately cause all remaining >> nodes to self-fence). >> --><-- >> >> SBD in shared-nothing mode is basically always in such degraded state >> and cannot tolerate loss of quorum. > Well, the context here is it loses quorum *expectedly* since the other > nodes gracefully shut down. > >> >> >>> Aug 09 14:30:20 opensuse150-1 sbd[1079]: pcmk:debug: >>> notify_parent: Not notifying parent: state transient (2) >>> Aug 09 14:30:20 opensuse150-1 sbd[1080]:cluster:debug: >>> notify_parent: Notifying parent: healthy >>> Aug 09 14:30:20 opensuse150-1 sbd[1078]: warning: inquisitor_child: >>> Latency: No liveness for 4 s exceeds threshold of 3 s (healthy servants: 0) >>> >>> I can think of the way to manipulate quorum with last_man_standing and >>> potentially also auto_tie_breaker, not to mention >>> last_man_standing_window would also be a factor... But is there a better >>> solution? >>> >> Lack of cluster wide shutdown mode was mentioned more than once on this >> list. I guess the only workaround is to use higher level tools which >> basically simply try to stop cluster on all nodes at once. It is still >> susceptible to race condition. > Gracefully stopping nodes one by one on purpose is still a reasonable > need though ... If you do the teardown as e.g. pcs is doing it - first tear down pacemaker-instances and then corosync/sbd - it is at least possible to tear down the pacemaker-instances one-by one without risking a reboot due to quorum-loss. With kind of current sbd having in - https://github.com/ClusterLabs/sbd/commit/824fe834c67fb7bae7feb87607381f9fa8fa2945 - https://github.com/ClusterLabs/sbd/commit/79b778debfee5b4ab2d099b2bfc7385f45597f70 - https://github.com/ClusterLabs/sbd/commit/a716a8ddd3df615009bcff3bd96dd9ae64cb5f68 this should be pretty robust although we are still thinking (probably together with some heartbeat to pacemakerd that assures pacemakerd is checking liveness of sub-daemons properly) of having a cleaner way to detect graceful pacemaker-shutdown. Klaus > > Regards, >Yan > ___ > Manage your subscription: > https://lists.clusterlabs.org/mailman/listinfo/users > > ClusterLabs home: https://www.clusterlabs.org/ ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] Gracefully stop nodes one by one with disk-less sbd
On 8/9/19 6:40 PM, Andrei Borzenkov wrote: > 09.08.2019 16:34, Yan Gao пишет: >> Hi, >> >> With disk-less sbd, it's fine to stop cluster service from the cluster >> nodes all at the same time. >> >> But if to stop the nodes one by one, for example with a 3-node cluster, >> after stopping the 2nd node, the only remaining node resets itself with: >> > > That is sort of documented in SBD manual page: > > --><-- > However, while the cluster is in such a degraded state, it can > neither successfully fence nor be shutdown cleanly (as taking the > cluster below the quorum threshold will immediately cause all remaining > nodes to self-fence). > --><-- > > SBD in shared-nothing mode is basically always in such degraded state > and cannot tolerate loss of quorum. Well, the context here is it loses quorum *expectedly* since the other nodes gracefully shut down. > > > >> Aug 09 14:30:20 opensuse150-1 sbd[1079]: pcmk:debug: >> notify_parent: Not notifying parent: state transient (2) >> Aug 09 14:30:20 opensuse150-1 sbd[1080]:cluster:debug: >> notify_parent: Notifying parent: healthy >> Aug 09 14:30:20 opensuse150-1 sbd[1078]: warning: inquisitor_child: >> Latency: No liveness for 4 s exceeds threshold of 3 s (healthy servants: 0) >> >> I can think of the way to manipulate quorum with last_man_standing and >> potentially also auto_tie_breaker, not to mention >> last_man_standing_window would also be a factor... But is there a better >> solution? >> > > Lack of cluster wide shutdown mode was mentioned more than once on this > list. I guess the only workaround is to use higher level tools which > basically simply try to stop cluster on all nodes at once. It is still > susceptible to race condition. Gracefully stopping nodes one by one on purpose is still a reasonable need though ... Regards, Yan ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] Gracefully stop nodes one by one with disk-less sbd
09.08.2019 16:34, Yan Gao пишет: > Hi, > > With disk-less sbd, it's fine to stop cluster service from the cluster > nodes all at the same time. > > But if to stop the nodes one by one, for example with a 3-node cluster, > after stopping the 2nd node, the only remaining node resets itself with: > That is sort of documented in SBD manual page: --><-- However, while the cluster is in such a degraded state, it can neither successfully fence nor be shutdown cleanly (as taking the cluster below the quorum threshold will immediately cause all remaining nodes to self-fence). --><-- SBD in shared-nothing mode is basically always in such degraded state and cannot tolerate loss of quorum. > Aug 09 14:30:20 opensuse150-1 sbd[1079]: pcmk:debug: > notify_parent: Not notifying parent: state transient (2) > Aug 09 14:30:20 opensuse150-1 sbd[1080]:cluster:debug: > notify_parent: Notifying parent: healthy > Aug 09 14:30:20 opensuse150-1 sbd[1078]: warning: inquisitor_child: > Latency: No liveness for 4 s exceeds threshold of 3 s (healthy servants: 0) > > I can think of the way to manipulate quorum with last_man_standing and > potentially also auto_tie_breaker, not to mention > last_man_standing_window would also be a factor... But is there a better > solution? > Lack of cluster wide shutdown mode was mentioned more than once on this list. I guess the only workaround is to use higher level tools which basically simply try to stop cluster on all nodes at once. It is still susceptible to race condition. ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
