Re: [ClusterLabs] what is the "best" way to completely shutdown a two-node cluster ?
- On Feb 10, 2022, at 4:40 PM, Jehan-Guillaume de Rorthais j...@dalibo.com wrote: > > I wonder if after the cluster shutdown complete, the target-role=Stopped could > be removed/edited offline with eg. crmadmin? That would make VirtualDomain > startable on boot. > > I suppose this would not be that simple as it would require to update it on > all > nodes, taking care of the CIB version, hash, etc... But maybe some tooling > could take care of this? > > Last, if Bernd need to stop gracefully the VirtualDomain paying attention to > the I/O load, maybe he doesn't want them start automatically on boot for the > exact same reason anyway? I start the cluster manually (systemctl start pacemaker) and don't have a problem to start the VirtualDomains by hand one after the other. I prefer that towards an "automatic" solution. Bernd smime.p7s Description: S/MIME Cryptographic Signature ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] what is the "best" way to completely shutdown a two-node cluster ?
On Thu, 10 Feb 2022 22:15:07 +0800 Roger Zhou via Users wrote: > > On 2/9/22 17:46, Lentes, Bernd wrote: > > > > > > - On Feb 7, 2022, at 4:13 PM, Jehan-Guillaume de Rorthais > > j...@dalibo.com wrote: > > > >> On Mon, 7 Feb 2022 14:24:44 +0100 (CET) > >> "Lentes, Bernd" wrote: > >> > >>> Hi, > >>> > >>> i'm currently changing a bit in my cluster because i realized that my > >>> configuration for a power outtage didn't work as i expected. My idea is > >>> currently: > >>> - first stop about 20 VirtualDomains, which are my services. This will > >>> surely takes some minutes. I'm thinking of stopping each with a time > >>> difference of about 20 seconds for not getting to much IO load. and then > >>> ... > > This part is tricky. At one hand, it is good thinking to throttle IO load. > > On the other hand, as Jehan and Ulrich mentioned, `crm resource stop ` > introduces "target‑role=Stopped" for each VirtualDomain, and have to do `crm > resource start ` to changed it back to "target‑role=Started" to start > them after the power outage. I wonder if after the cluster shutdown complete, the target-role=Stopped could be removed/edited offline with eg. crmadmin? That would make VirtualDomain startable on boot. I suppose this would not be that simple as it would require to update it on all nodes, taking care of the CIB version, hash, etc... But maybe some tooling could take care of this? Last, if Bernd need to stop gracefully the VirtualDomain paying attention to the I/O load, maybe he doesn't want them start automatically on boot for the exact same reason anyway? ++ ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] what is the "best" way to completely shutdown a two-node cluster ?
On 2/9/22 17:46, Lentes, Bernd wrote: - On Feb 7, 2022, at 4:13 PM, Jehan-Guillaume de Rorthais j...@dalibo.com wrote: On Mon, 7 Feb 2022 14:24:44 +0100 (CET) "Lentes, Bernd" wrote: Hi, i'm currently changing a bit in my cluster because i realized that my configuration for a power outtage didn't work as i expected. My idea is currently: - first stop about 20 VirtualDomains, which are my services. This will surely takes some minutes. I'm thinking of stopping each with a time difference of about 20 seconds for not getting to much IO load. and then ... This part is tricky. At one hand, it is good thinking to throttle IO load. On the other hand, as Jehan and Ulrich mentioned, `crm resource stop ` introduces "target‑role=Stopped" for each VirtualDomain, and have to do `crm resource start ` to changed it back to "target‑role=Started" to start them after the power outage. - how to stop the other resources ? I would set cluster option "stop-all-resources" so all remaining resources are stopped gracefully by the cluster. Then you can stop both nodes using eg. "crm cluster stop". Here, for SLES12SP5, `crm cluster run "crm cluster stop"` could help a little. From crmsh-4.4.0 onward, `crm cluster stop --all` is recommended to simplify the whole procedure to do the cluster wide shutdown. BR, Roger On restart, after both nodes are up and joined to the cluster, you can set "stop-all-resources=false", then start your VirtualDomains. Aren't the VirtualDomains already started by "stop-all-resources=false" ? I wrote a script for the whole procedure which is triggered by the UPS. As i am not a big schellscript-writer please have a look and tell me your opinion. You find it here: https://nc-mcd.helmholtz-muenchen.de/nextcloud/s/rEA9bFxs5Ay6fYG Thanks. Bernd ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/ ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] what is the "best" way to completely shutdown a two-node cluster ?
One drawback of that approach is that adding a the resource stop command will also prevent the resources from starting once the UPS gets enough power and start the servers. Of course a script in cron (@reboot) or in systemd can overcome it. Best Regards,Strahil Nikolov On Thu, Feb 10, 2022 at 11:15, Jehan-Guillaume de Rorthais wrote: On Wed, 9 Feb 2022 17:42:35 + (UTC) Strahil Nikolov via Users wrote: > If you gracefully shutdown a node - pacemaker will migrate all resources away > so you need to shut them down simultaneously and all resources should be > stopped by the cluster. > > Shutting down the nodes would be my choice. If you want to gracefully shutdown your cluster, then you can add one manual step to first gracefully stop your resources instead of betting the cluster will do the good things. As far as I remember, there's no way the DC/CRM can orchestrate the whole cluster shutdown gracefully in the same transition. So I prefer standing on the safe side and add one step to my procedure. I even add it with commands like `pcs cluster stop --all` which tries to shutdown all the nodes "kind of" in the same time everywhere. At least, I know where my resources were stopped and how they will start. It might be important when you deal with eg. permanent promotion scores. ++ ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] what is the "best" way to completely shutdown a two-node cluster ?
On Wed, 9 Feb 2022 17:42:35 + (UTC) Strahil Nikolov via Users wrote: > If you gracefully shutdown a node - pacemaker will migrate all resources away > so you need to shut them down simultaneously and all resources should be > stopped by the cluster. > > Shutting down the nodes would be my choice. If you want to gracefully shutdown your cluster, then you can add one manual step to first gracefully stop your resources instead of betting the cluster will do the good things. As far as I remember, there's no way the DC/CRM can orchestrate the whole cluster shutdown gracefully in the same transition. So I prefer standing on the safe side and add one step to my procedure. I even add it with commands like `pcs cluster stop --all` which tries to shutdown all the nodes "kind of" in the same time everywhere. At least, I know where my resources were stopped and how they will start. It might be important when you deal with eg. permanent promotion scores. ++ ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] what is the "best" way to completely shutdown a two-node cluster ?
If you gracefully shutdown a node - pacemaker will migrate all resources away so you need to shut them down simultaneously and all resources should be stopped by the cluster. Shutting down the nodes would be my choice. Best Regards,Strahil Nikolov On Wed, Feb 9, 2022 at 12:52, Lentes, Bernd wrote: - On Feb 9, 2022, at 11:26 AM, Jehan-Guillaume de Rorthais j...@dalibo.com wrote: > > I'm not sure how "crm resource stop " actually stop a resource. I thought > it would set "target-role=Stopped", but I might be wrong. > > If "crm resource stop" actually use "target-role=Stopped", I believe the > resources would not start automatically after setting > "stop-all-resources=false". > ha-idg-2:~ # crm resource help stop Stop resources Stop one or more resources using the target-role attribute. If there are multiple meta attributes sets, the attribute is set in all of them. If the resource is a clone, all target-role attributes are removed from the children resources. For details on group management see options manage-children. Usage: stop [ ...] Bernd ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] what is the "best" way to completely shutdown a two-node cluster ?
- On Feb 9, 2022, at 11:26 AM, Jehan-Guillaume de Rorthais j...@dalibo.com wrote: > > I'm not sure how "crm resource stop " actually stop a resource. I thought > it would set "target-role=Stopped", but I might be wrong. > > If "crm resource stop" actually use "target-role=Stopped", I believe the > resources would not start automatically after setting > "stop-all-resources=false". > ha-idg-2:~ # crm resource help stop Stop resources Stop one or more resources using the target-role attribute. If there are multiple meta attributes sets, the attribute is set in all of them. If the resource is a clone, all target-role attributes are removed from the children resources. For details on group management see options manage-children. Usage: stop [ ...] Bernd smime.p7s Description: S/MIME Cryptographic Signature ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] what is the "best" way to completely shutdown a two-node cluster ?
On Wed, 9 Feb 2022 10:46:30 +0100 (CET) "Lentes, Bernd" wrote: > - On Feb 7, 2022, at 4:13 PM, Jehan-Guillaume de Rorthais j...@dalibo.com > wrote: > > > On Mon, 7 Feb 2022 14:24:44 +0100 (CET) > > "Lentes, Bernd" wrote: > > > >> Hi, > >> > >> i'm currently changing a bit in my cluster because i realized that my > >> configuration for a power outtage didn't work as i expected. My idea is > >> currently: > >> - first stop about 20 VirtualDomains, which are my services. This will > >> surely takes some minutes. I'm thinking of stopping each with a time > >> difference of about 20 seconds for not getting to much IO load. and then > >> ... > >> - how to stop the other resources ? > > > > I would set cluster option "stop-all-resources" so all remaining resources > > are stopped gracefully by the cluster. > > > > Then you can stop both nodes using eg. "crm cluster stop". > > > > On restart, after both nodes are up and joined to the cluster, you can set > > "stop-all-resources=false", then start your VirtualDomains. > > Aren't the VirtualDomains already started by "stop-all-resources=false" ? I'm not sure how "crm resource stop " actually stop a resource. I thought it would set "target-role=Stopped", but I might be wrong. If "crm resource stop" actually use "target-role=Stopped", I believe the resources would not start automatically after setting "stop-all-resources=false". ++ ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] what is the "best" way to completely shutdown a two-node cluster ?
- On Feb 7, 2022, at 4:13 PM, Jehan-Guillaume de Rorthais j...@dalibo.com wrote: > On Mon, 7 Feb 2022 14:24:44 +0100 (CET) > "Lentes, Bernd" wrote: > >> Hi, >> >> i'm currently changing a bit in my cluster because i realized that my >> configuration for a power outtage didn't work as i expected. My idea is >> currently: >> - first stop about 20 VirtualDomains, which are my services. This will surely >> takes some minutes. I'm thinking of stopping each with a time difference of >> about 20 seconds for not getting to much IO load. and then ... >> - how to stop the other resources ? > > I would set cluster option "stop-all-resources" so all remaining resources are > stopped gracefully by the cluster. > > Then you can stop both nodes using eg. "crm cluster stop". > > On restart, after both nodes are up and joined to the cluster, you can set > "stop-all-resources=false", then start your VirtualDomains. Aren't the VirtualDomains already started by "stop-all-resources=false" ? I wrote a script for the whole procedure which is triggered by the UPS. As i am not a big schellscript-writer please have a look and tell me your opinion. You find it here: https://nc-mcd.helmholtz-muenchen.de/nextcloud/s/rEA9bFxs5Ay6fYG Thanks. Bernd smime.p7s Description: S/MIME Cryptographic Signature ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] what is the "best" way to completely shutdown a two-node cluster ?
On Mon, 7 Feb 2022 14:24:44 +0100 (CET) "Lentes, Bernd" wrote: > Hi, > > i'm currently changing a bit in my cluster because i realized the my > configuration for a power outtage didn't work as i expected. My idea is > currently: > - first stop about 20 VirtualDomains, which are my services. This will surely > takes some minutes. I'm thinking of stopping each with a time difference of > about 20 seconds for not getting to much IO load. and then ... > - how to stop the other resources ? I would set cluster option "stop-all-resources" so all remaining resources are stopped gracefully by the cluster. Then you can stop both nodes using eg. "crm cluster stop". On restart, after both nodes are up and joined to the cluster, you can set "stop-all-resources=false", then start your VirtualDomains. ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
[ClusterLabs] what is the "best" way to completely shutdown a two-node cluster ?
Hi, i'm currently changing a bit in my cluster because i realized the my configuration for a power outtage didn't work as i expected. My idea is currently: - first stop about 20 VirtualDomains, which are my services. This will surely takes some minutes. I'm thinking of stopping each with a time difference of about 20 seconds for not getting to much IO load. and then ... - how to stop the other resources ? - put the nodes into standby or offline ? - do a systemctl stop pacemaker ? - or do a crm cluster stop ? And what is if both nodes are running ? Can i do that simultaneously on both nodes ? My OS is SLES 12 SP5, pacemaker is 1.1.23, corosync is 2.3.6-9.13.1 Thanks for your help. Bernd -- Bernd Lentes System Administrator Institute for Metabolism and Cell Death (MCD) Building 25 - office 122 HelmholtzZentrum München bernd.len...@helmholtz-muenchen.de phone: +49 89 3187 1241 fax: +49 89 3187 2294 http://www.helmholtz-muenchen.de/mcd Public key: 30 82 01 0a 02 82 01 01 00 b3 72 3e ce 2c 0a 6f 58 49 2c 92 23 c7 b9 c1 ff 6c 3a 53 be f7 9e e9 24 b7 49 fa 3c e8 de 28 85 2c d3 ed f7 70 03 3f 4d 82 fc cc 96 4f 18 27 1f df 25 b3 13 00 db 4b 1d ec 7f 1b cf f9 cd e8 5b 1f 11 b3 a7 48 f8 c8 37 ed 41 ff 18 9f d7 83 51 a9 bd 86 c2 32 b3 d6 2d 77 ff 32 83 92 67 9e ae ae 9c 99 ce 42 27 6f bf d8 c2 a1 54 fd 2b 6b 12 65 0e 8a 79 56 be 53 89 70 51 02 6a eb 76 b8 92 25 2d 88 aa 57 08 42 ef 57 fb fe 00 71 8e 90 ef b2 e3 22 f3 34 4f 7b f1 c4 b1 7c 2f 1d 6f bd c8 a6 a1 1f 25 f3 e4 4b 6a 23 d3 d2 fa 27 ae 97 80 a3 f0 5a c4 50 4a 45 e3 45 4d 82 9f 8b 87 90 d0 f9 92 2d a7 d2 67 53 e6 ae 1e 72 3e e9 e0 c9 d3 1c 23 e0 75 78 4a 45 60 94 f8 e3 03 0b 09 85 08 d0 6c f3 ff ce fa 50 25 d9 da 81 7b 2a dc 9e 28 8b 83 04 b4 0a 9f 37 b8 ac 58 f1 38 43 0e 72 af 02 03 01 00 01 smime.p7s Description: S/MIME Cryptographic Signature ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
