Re: Self-fencing when storage not available
I saw your suggestion in the ticket ... Thanx, Paul On 07.02.2016 20:51, Paul Angus wrote: Well, to start off with I'll add the suggestion to the ticket. https://issues.apache.org/jira/browse/CLOUDSTACK-3367 Paul Angus VP Technology , ShapeBlue t: @cloudyangus<tel:@cloudyangus> e: paul.an...@shapeblue.com<mailto:paul.an...@shapeblue.com>| w: www.shapeblue.com<http://www.shapeblue.com> -Original Message- From: Vladislav Nazarenko [mailto:vladislav.nazare...@gmail.com] Sent: Friday, February 5, 2016 5:29 PM To: users@cloudstack.apache.org Subject: Re: Self-fencing when storage not available right, Paul ... this might be a good suggestion ... How could we bring it to the developers? On 04.02.2016 20:59, Paul Angus wrote: Just to give some background (as I asked the same questions years ago). *IF* you only have one shared storage pool per cluster (and no local storage) and it goes wonky, then most likely fix is to restart the host(s). The VMs have already crashed because the primary storage is disconnected and the host is unable to reconnect it. That’s why it reboots. As everyone who has a cloud larger than their laptop knows, it also nukes the good VMs on other storage pools and local storage. A global setting which allows operators to choose the behaviour they want might be a solution? Paul Angus VP Technology , ShapeBlue t: @cloudyangus<tel:@cloudyangus> e: paul.an...@shapeblue.com<mailto:paul.an...@shapeblue.com> | w: www.shapeblue.com<http://www.shapeblue.com> -Original Message- From: Vladislav Nazarenko [mailto:vladislav.nazare...@gmail.com] Sent: Thursday, February 4, 2016 5:35 PM To: users@cloudstack.apache.org Subject: Re: Self-fencing when storage not available So ... found some information about this ... looks like this is an annoying problem (bug), which isn't going to be fixed ... But at least there is a possibility to change the behavior modifying the script - https://admintweets.wordpress.com/tag/cloudstack-2/ - https://www.rootusers.com/edit-the-xenserver-storage-heartbeat/ - https://issues.apache.org/jira/browse/CLOUDSTACK-3367 Thanks a lot, guys .. On 03.02.2016 20:05, Glenn Wagner wrote: Hi As Lucian said , you can edit the script and remove the reboot , but yes this is the normal behaviour of the script Glenn -Original Message- From: Nux! [mailto:n...@li.nux.ro] Sent: Wednesday, 03 February 2016 8:32 PM To: users@cloudstack.apache.org Subject: Re: Self-fencing when storage not available You can modify the script to not reboot, but until we find a better way to deal with it this is correct behaviour. It sucks it reboots VMs on healthy storage though. -- Sent from the Delta quadrant using Borg technology! Nux! www.nux.ro - Original Message - From: "Vladislav Nazarenko" <vladislav.nazare...@gmail.com> To: users@cloudstack.apache.org Sent: Wednesday, 3 February, 2016 16:24:34 Subject: Re: Self-fencing when storage not available Hi Glenn, we use KVM ... I also found the script: /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/kvmheartbeat.s h Just wondering if we can avoid rebooting ... Thanks Vlad On 03.02.2016 16:57, Glenn Wagner wrote: Hi Vlad, Can I ask what hypervisor are you using? The default action is to reboot the hosts This is done in the heatbeat file on xenserver /opt/xensource/bin/xenheartbeat.sh Thanks Glenn ShapeBlue <http://www.shapeblue.com> Glenn Wagner Senior Consultant , ShapeBlue d: * | s: +27 21 527 0091* <tel:%7C%20s:%20+27%2021%20527%200091> | m: *+27 73 917 4111* <tel:+27%2073%20917%204111> e: *glenn.wag...@shapeblue.com | t: * <mailto:glenn.wag...@shapeblue.com%20%7C%20t:> | w: *www.shapeblue.com* <http://www.shapeblue.com> a: 2nd Floor, Oudehuis Centre, 122 Main Rd, Somerset West Cape Town 7130 South Africa Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue Services India LLP is a company incorporated in India and is operated under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil and is operated under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic of South Africa and is traded under license from Shape Blue Ltd. ShapeBlue is a registered trademark. This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Shape Blue Ltd or related companies. If you are not the intended recipient of this email, you must neither take any action based upon its contents, nor copy or show it to anyone. Please contact the sender if you believe you have received this email in error. -Original Message- From: Vladislav Nazarenko [mailto:vladislav.nazare..
RE: Self-fencing when storage not available
Well, to start off with I'll add the suggestion to the ticket. https://issues.apache.org/jira/browse/CLOUDSTACK-3367 Paul Angus VP Technology , ShapeBlue t: @cloudyangus<tel:@cloudyangus> e: paul.an...@shapeblue.com<mailto:paul.an...@shapeblue.com>| w: www.shapeblue.com<http://www.shapeblue.com> -Original Message- From: Vladislav Nazarenko [mailto:vladislav.nazare...@gmail.com] Sent: Friday, February 5, 2016 5:29 PM To: users@cloudstack.apache.org Subject: Re: Self-fencing when storage not available right, Paul ... this might be a good suggestion ... How could we bring it to the developers? On 04.02.2016 20:59, Paul Angus wrote: > Just to give some background (as I asked the same questions years ago). *IF* > you only have one shared storage pool per cluster (and no local storage) and > it goes wonky, then most likely fix is to restart the host(s). The VMs have > already crashed because the primary storage is disconnected and the host is > unable to reconnect it. > > That’s why it reboots. > > As everyone who has a cloud larger than their laptop knows, it also nukes the > good VMs on other storage pools and local storage. > > A global setting which allows operators to choose the behaviour they want > might be a solution? > > > > > Paul Angus > VP Technology , ShapeBlue > > > t: @cloudyangus<tel:@cloudyangus> > > e: paul.an...@shapeblue.com<mailto:paul.an...@shapeblue.com> | w: > www.shapeblue.com<http://www.shapeblue.com> > > > > > > -Original Message- > From: Vladislav Nazarenko [mailto:vladislav.nazare...@gmail.com] > Sent: Thursday, February 4, 2016 5:35 PM > To: users@cloudstack.apache.org > Subject: Re: Self-fencing when storage not available > > So ... found some information about this ... looks like this is an annoying > problem (bug), which isn't going to be fixed ... But at least there is a > possibility to change the behavior modifying the script > > - https://admintweets.wordpress.com/tag/cloudstack-2/ > - https://www.rootusers.com/edit-the-xenserver-storage-heartbeat/ > - https://issues.apache.org/jira/browse/CLOUDSTACK-3367 > > Thanks a lot, guys .. > > On 03.02.2016 20:05, Glenn Wagner wrote: >> Hi >> >> As Lucian said , you can edit the script and remove the reboot , but >> yes this is the normal behaviour of the script >> >> Glenn >> >> >> -Original Message- >> From: Nux! [mailto:n...@li.nux.ro] >> Sent: Wednesday, 03 February 2016 8:32 PM >> To: users@cloudstack.apache.org >> Subject: Re: Self-fencing when storage not available >> >> You can modify the script to not reboot, but until we find a better way to >> deal with it this is correct behaviour. It sucks it reboots VMs on healthy >> storage though. >> >> -- >> Sent from the Delta quadrant using Borg technology! >> >> Nux! >> www.nux.ro >> >> - Original Message - >>> From: "Vladislav Nazarenko" <vladislav.nazare...@gmail.com> >>> To: users@cloudstack.apache.org >>> Sent: Wednesday, 3 February, 2016 16:24:34 >>> Subject: Re: Self-fencing when storage not available Hi Glenn, >>> >>> we use KVM ... I also found the script: >>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/kvmheartbeat.s >>> h >>> >>> Just wondering if we can avoid rebooting ... >>> >>> Thanks >>> Vlad >>> >>> On 03.02.2016 16:57, Glenn Wagner wrote: >>>> Hi Vlad, >>>> >>>> Can I ask what hypervisor are you using? >>>> The default action is to reboot the hosts >>>> >>>> This is done in the heatbeat file on xenserver >>>> /opt/xensource/bin/xenheartbeat.sh >>>> >>>> Thanks >>>> Glenn >>>> >>>> >>>> >>>> ShapeBlue <http://www.shapeblue.com> Glenn Wagner >>>> Senior Consultant , ShapeBlue >>>> >>>> d: * | s: +27 21 527 0091* <tel:%7C%20s:%20+27%2021%20527%200091> >>>> | m: *+27 73 917 4111* <tel:+27%2073%20917%204111> >>>> >>>> e: *glenn.wag...@shapeblue.com | t: * >>>> <mailto:glenn.wag...@shapeblue.com%20%7C%20t:> | w: >>>> *www.shapeblue.com* <http://www.shapeblue.com> >>>> >>>> a: >>>> 2nd Floor, Oudehuis Centre, 122 Main Rd, Somerset West Cape Town >>>> 7130 South Africa >>>> >>>> >>>> Shape Blue Ltd is a company incorporated in
Re: Self-fencing when storage not available
right, Paul ... this might be a good suggestion ... How could we bring it to the developers? On 04.02.2016 20:59, Paul Angus wrote: Just to give some background (as I asked the same questions years ago). *IF* you only have one shared storage pool per cluster (and no local storage) and it goes wonky, then most likely fix is to restart the host(s). The VMs have already crashed because the primary storage is disconnected and the host is unable to reconnect it. That’s why it reboots. As everyone who has a cloud larger than their laptop knows, it also nukes the good VMs on other storage pools and local storage. A global setting which allows operators to choose the behaviour they want might be a solution? Paul Angus VP Technology , ShapeBlue t: @cloudyangus<tel:@cloudyangus> e: paul.an...@shapeblue.com<mailto:paul.an...@shapeblue.com>| w: www.shapeblue.com<http://www.shapeblue.com> -Original Message- From: Vladislav Nazarenko [mailto:vladislav.nazare...@gmail.com] Sent: Thursday, February 4, 2016 5:35 PM To: users@cloudstack.apache.org Subject: Re: Self-fencing when storage not available So ... found some information about this ... looks like this is an annoying problem (bug), which isn't going to be fixed ... But at least there is a possibility to change the behavior modifying the script - https://admintweets.wordpress.com/tag/cloudstack-2/ - https://www.rootusers.com/edit-the-xenserver-storage-heartbeat/ - https://issues.apache.org/jira/browse/CLOUDSTACK-3367 Thanks a lot, guys .. On 03.02.2016 20:05, Glenn Wagner wrote: Hi As Lucian said , you can edit the script and remove the reboot , but yes this is the normal behaviour of the script Glenn -Original Message- From: Nux! [mailto:n...@li.nux.ro] Sent: Wednesday, 03 February 2016 8:32 PM To: users@cloudstack.apache.org Subject: Re: Self-fencing when storage not available You can modify the script to not reboot, but until we find a better way to deal with it this is correct behaviour. It sucks it reboots VMs on healthy storage though. -- Sent from the Delta quadrant using Borg technology! Nux! www.nux.ro - Original Message - From: "Vladislav Nazarenko" <vladislav.nazare...@gmail.com> To: users@cloudstack.apache.org Sent: Wednesday, 3 February, 2016 16:24:34 Subject: Re: Self-fencing when storage not available Hi Glenn, we use KVM ... I also found the script: /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/kvmheartbeat.s h Just wondering if we can avoid rebooting ... Thanks Vlad On 03.02.2016 16:57, Glenn Wagner wrote: Hi Vlad, Can I ask what hypervisor are you using? The default action is to reboot the hosts This is done in the heatbeat file on xenserver /opt/xensource/bin/xenheartbeat.sh Thanks Glenn ShapeBlue <http://www.shapeblue.com> Glenn Wagner Senior Consultant , ShapeBlue d: * | s: +27 21 527 0091* <tel:%7C%20s:%20+27%2021%20527%200091> | m: *+27 73 917 4111* <tel:+27%2073%20917%204111> e: *glenn.wag...@shapeblue.com | t: * <mailto:glenn.wag...@shapeblue.com%20%7C%20t:> | w: *www.shapeblue.com* <http://www.shapeblue.com> a: 2nd Floor, Oudehuis Centre, 122 Main Rd, Somerset West Cape Town 7130 South Africa Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue Services India LLP is a company incorporated in India and is operated under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil and is operated under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic of South Africa and is traded under license from Shape Blue Ltd. ShapeBlue is a registered trademark. This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Shape Blue Ltd or related companies. If you are not the intended recipient of this email, you must neither take any action based upon its contents, nor copy or show it to anyone. Please contact the sender if you believe you have received this email in error. -Original Message- From: Vladislav Nazarenko [mailto:vladislav.nazare...@gmail.com] Sent: Wednesday, 03 February 2016 5:08 PM To: users@cloudstack.apache.org Subject: Self-fencing when storage not available Hi All, I'm testing Cloudstack 4.6 now When doing some tests with storage (nfs), I was wondering, that the hosts self-fence itself by reboot, when a storage not writable ... Even more, my cluster had a working storage and I added one more, which was not writable due to missing user permissions... So effectively all the VMs hard drive were located on the working storage, but the problem with permissions on the new one causes the entire cluster to reboot :( Is this a bug or a correct behavior? Is it
Re: Self-fencing when storage not available
So ... found some information about this ... looks like this is an annoying problem (bug), which isn't going to be fixed ... But at least there is a possibility to change the behavior modifying the script - https://admintweets.wordpress.com/tag/cloudstack-2/ - https://www.rootusers.com/edit-the-xenserver-storage-heartbeat/ - https://issues.apache.org/jira/browse/CLOUDSTACK-3367 Thanks a lot, guys .. On 03.02.2016 20:05, Glenn Wagner wrote: Hi As Lucian said , you can edit the script and remove the reboot , but yes this is the normal behaviour of the script Glenn -Original Message- From: Nux! [mailto:n...@li.nux.ro] Sent: Wednesday, 03 February 2016 8:32 PM To: users@cloudstack.apache.org Subject: Re: Self-fencing when storage not available You can modify the script to not reboot, but until we find a better way to deal with it this is correct behaviour. It sucks it reboots VMs on healthy storage though. -- Sent from the Delta quadrant using Borg technology! Nux! www.nux.ro - Original Message - From: "Vladislav Nazarenko" <vladislav.nazare...@gmail.com> To: users@cloudstack.apache.org Sent: Wednesday, 3 February, 2016 16:24:34 Subject: Re: Self-fencing when storage not available Hi Glenn, we use KVM ... I also found the script: /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/kvmheartbeat.sh Just wondering if we can avoid rebooting ... Thanks Vlad On 03.02.2016 16:57, Glenn Wagner wrote: Hi Vlad, Can I ask what hypervisor are you using? The default action is to reboot the hosts This is done in the heatbeat file on xenserver /opt/xensource/bin/xenheartbeat.sh Thanks Glenn ShapeBlue <http://www.shapeblue.com> Glenn Wagner Senior Consultant, ShapeBlue d: * | s: +27 21 527 0091* <tel:%7C%20s:%20+27%2021%20527%200091> | m: *+27 73 917 4111* <tel:+27%2073%20917%204111> e: *glenn.wag...@shapeblue.com | t: * <mailto:glenn.wag...@shapeblue.com%20%7C%20t:>| w: *www.shapeblue.com* <http://www.shapeblue.com> a: 2nd Floor, Oudehuis Centre, 122 Main Rd, Somerset West Cape Town 7130 South Africa Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue Services India LLP is a company incorporated in India and is operated under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil and is operated under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic of South Africa and is traded under license from Shape Blue Ltd. ShapeBlue is a registered trademark. This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Shape Blue Ltd or related companies. If you are not the intended recipient of this email, you must neither take any action based upon its contents, nor copy or show it to anyone. Please contact the sender if you believe you have received this email in error. -Original Message- From: Vladislav Nazarenko [mailto:vladislav.nazare...@gmail.com] Sent: Wednesday, 03 February 2016 5:08 PM To: users@cloudstack.apache.org Subject: Self-fencing when storage not available Hi All, I'm testing Cloudstack 4.6 now When doing some tests with storage (nfs), I was wondering, that the hosts self-fence itself by reboot, when a storage not writable ... Even more, my cluster had a working storage and I added one more, which was not writable due to missing user permissions... So effectively all the VMs hard drive were located on the working storage, but the problem with permissions on the new one causes the entire cluster to reboot :( Is this a bug or a correct behavior? Is it able to avoid rebooting at least? Thank you in advance Vlad Find out more about ShapeBlue and our range of CloudStack related services: IaaS Cloud Design & Build <http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge – rapid IaaS deployment framework <http://shapeblue.com/csforge/> CloudStack Consulting <http://shapeblue.com/cloudstack-consultancy/> | CloudStack Software Engineering <http://shapeblue.com/cloudstack-software-engineering/> CloudStack Infrastructure Support <http://shapeblue.com/cloudstack-infrastructure-support/> | CloudStack Bootcamp Training Courses <http://shapeblue.com/cloudstack-training/> Find out more about ShapeBlue and our range of CloudStack related services: IaaS Cloud Design & Build<http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge – rapid IaaS deployment framework<http://shapeblue.com/csforge/> CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/> | CloudStack Software Engineering<http://shapeblue.com/cloudstack-software-engineering/> CloudStack Infrastructure Support<http://shapeblue.com/cloudstack-infrastructure-support/> | CloudStack Bootcamp Training Courses<http://shapeblue.com/cloudstack-training/>
RE: Self-fencing when storage not available
Just to give some background (as I asked the same questions years ago). *IF* you only have one shared storage pool per cluster (and no local storage) and it goes wonky, then most likely fix is to restart the host(s). The VMs have already crashed because the primary storage is disconnected and the host is unable to reconnect it. That’s why it reboots. As everyone who has a cloud larger than their laptop knows, it also nukes the good VMs on other storage pools and local storage. A global setting which allows operators to choose the behaviour they want might be a solution? Paul Angus VP Technology , ShapeBlue t: @cloudyangus<tel:@cloudyangus> e: paul.an...@shapeblue.com<mailto:paul.an...@shapeblue.com>| w: www.shapeblue.com<http://www.shapeblue.com> -Original Message- From: Vladislav Nazarenko [mailto:vladislav.nazare...@gmail.com] Sent: Thursday, February 4, 2016 5:35 PM To: users@cloudstack.apache.org Subject: Re: Self-fencing when storage not available So ... found some information about this ... looks like this is an annoying problem (bug), which isn't going to be fixed ... But at least there is a possibility to change the behavior modifying the script - https://admintweets.wordpress.com/tag/cloudstack-2/ - https://www.rootusers.com/edit-the-xenserver-storage-heartbeat/ - https://issues.apache.org/jira/browse/CLOUDSTACK-3367 Thanks a lot, guys .. On 03.02.2016 20:05, Glenn Wagner wrote: > Hi > > As Lucian said , you can edit the script and remove the reboot , but > yes this is the normal behaviour of the script > > Glenn > > > -Original Message- > From: Nux! [mailto:n...@li.nux.ro] > Sent: Wednesday, 03 February 2016 8:32 PM > To: users@cloudstack.apache.org > Subject: Re: Self-fencing when storage not available > > You can modify the script to not reboot, but until we find a better way to > deal with it this is correct behaviour. It sucks it reboots VMs on healthy > storage though. > > -- > Sent from the Delta quadrant using Borg technology! > > Nux! > www.nux.ro > > - Original Message - >> From: "Vladislav Nazarenko" <vladislav.nazare...@gmail.com> >> To: users@cloudstack.apache.org >> Sent: Wednesday, 3 February, 2016 16:24:34 >> Subject: Re: Self-fencing when storage not available Hi Glenn, >> >> we use KVM ... I also found the script: >> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/kvmheartbeat.s >> h >> >> Just wondering if we can avoid rebooting ... >> >> Thanks >> Vlad >> >> On 03.02.2016 16:57, Glenn Wagner wrote: >>> Hi Vlad, >>> >>> Can I ask what hypervisor are you using? >>> The default action is to reboot the hosts >>> >>> This is done in the heatbeat file on xenserver >>> /opt/xensource/bin/xenheartbeat.sh >>> >>> Thanks >>> Glenn >>> >>> >>> >>> ShapeBlue <http://www.shapeblue.com> Glenn Wagner >>> Senior Consultant , ShapeBlue >>> >>> d: * | s: +27 21 527 0091* <tel:%7C%20s:%20+27%2021%20527%200091> >>> | m: *+27 73 917 4111* <tel:+27%2073%20917%204111> >>> >>> e: *glenn.wag...@shapeblue.com | t: * >>> <mailto:glenn.wag...@shapeblue.com%20%7C%20t:> | w: >>> *www.shapeblue.com* <http://www.shapeblue.com> >>> >>> a: >>> 2nd Floor, Oudehuis Centre, 122 Main Rd, Somerset West Cape Town >>> 7130 South Africa >>> >>> >>> Shape Blue Ltd is a company incorporated in England & Wales. >>> ShapeBlue Services India LLP is a company incorporated in India and >>> is operated under license from Shape Blue Ltd. Shape Blue Brasil >>> Consultoria Ltda is a company incorporated in Brasil and is operated >>> under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company >>> registered by The Republic of South Africa and is traded under >>> license from Shape Blue Ltd. ShapeBlue is a registered trademark. >>> This email and any attachments to it may be confidential and are >>> intended solely for the use of the individual to whom it is addressed. >>> Any views or opinions expressed are solely those of the author and >>> do not necessarily represent those of Shape Blue Ltd or related >>> companies. If you are not the intended recipient of this email, you >>> must neither take any action based upon its contents, nor copy or >>> show it to anyone. Please contact the sender if you believe you have >>> received this email in error. >>> >>> >>> -Original Messag
Self-fencing when storage not available
Hi All, I'm testing Cloudstack 4.6 now When doing some tests with storage (nfs), I was wondering, that the hosts self-fence itself by reboot, when a storage not writable ... Even more, my cluster had a working storage and I added one more, which was not writable due to missing user permissions... So effectively all the VMs hard drive were located on the working storage, but the problem with permissions on the new one causes the entire cluster to reboot :( Is this a bug or a correct behavior? Is it able to avoid rebooting at least? Thank you in advance Vlad
Re: Self-fencing when storage not available
Hi Glenn, we use KVM ... I also found the script: /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/kvmheartbeat.sh Just wondering if we can avoid rebooting ... Thanks Vlad On 03.02.2016 16:57, Glenn Wagner wrote: Hi Vlad, Can I ask what hypervisor are you using? The default action is to reboot the hosts This is done in the heatbeat file on xenserver /opt/xensource/bin/xenheartbeat.sh Thanks Glenn ShapeBlue <http://www.shapeblue.com> Glenn Wagner Senior Consultant , ShapeBlue d: * | s: +27 21 527 0091* <tel:%7C%20s:%20+27%2021%20527%200091> | m: *+27 73 917 4111* <tel:+27%2073%20917%204111> e: *glenn.wag...@shapeblue.com | t: * <mailto:glenn.wag...@shapeblue.com%20%7C%20t:> | w: *www.shapeblue.com* <http://www.shapeblue.com> a: 2nd Floor, Oudehuis Centre, 122 Main Rd, Somerset West Cape Town 7130 South Africa Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue Services India LLP is a company incorporated in India and is operated under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil and is operated under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic of South Africa and is traded under license from Shape Blue Ltd. ShapeBlue is a registered trademark. This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Shape Blue Ltd or related companies. If you are not the intended recipient of this email, you must neither take any action based upon its contents, nor copy or show it to anyone. Please contact the sender if you believe you have received this email in error. -Original Message- From: Vladislav Nazarenko [mailto:vladislav.nazare...@gmail.com] Sent: Wednesday, 03 February 2016 5:08 PM To: users@cloudstack.apache.org Subject: Self-fencing when storage not available Hi All, I'm testing Cloudstack 4.6 now When doing some tests with storage (nfs), I was wondering, that the hosts self-fence itself by reboot, when a storage not writable ... Even more, my cluster had a working storage and I added one more, which was not writable due to missing user permissions... So effectively all the VMs hard drive were located on the working storage, but the problem with permissions on the new one causes the entire cluster to reboot :( Is this a bug or a correct behavior? Is it able to avoid rebooting at least? Thank you in advance Vlad Find out more about ShapeBlue and our range of CloudStack related services: IaaS Cloud Design & Build <http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge – rapid IaaS deployment framework <http://shapeblue.com/csforge/> CloudStack Consulting <http://shapeblue.com/cloudstack-consultancy/> | CloudStack Software Engineering <http://shapeblue.com/cloudstack-software-engineering/> CloudStack Infrastructure Support <http://shapeblue.com/cloudstack-infrastructure-support/> | CloudStack Bootcamp Training Courses <http://shapeblue.com/cloudstack-training/>
RE: Self-fencing when storage not available
Hi Vlad, Can I ask what hypervisor are you using? The default action is to reboot the hosts This is done in the heatbeat file on xenserver /opt/xensource/bin/xenheartbeat.sh Thanks Glenn [ShapeBlue]<http://www.shapeblue.com> Glenn Wagner Senior Consultant , ShapeBlue d: | s: +27 21 527 0091<tel:|%20s:%20+27%2021%20527%200091>| m: +27 73 917 4111<tel:+27%2073%20917%204111> e: glenn.wag...@shapeblue.com | t: <mailto:glenn.wag...@shapeblue.com%20|%20t:> | w: www.shapeblue.com<http://www.shapeblue.com> a: 2nd Floor, Oudehuis Centre, 122 Main Rd, Somerset West Cape Town 7130 South Africa [cid:image24bf30.png@0df405d7.41a47b58] Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue Services India LLP is a company incorporated in India and is operated under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil and is operated under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic of South Africa and is traded under license from Shape Blue Ltd. ShapeBlue is a registered trademark. This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Shape Blue Ltd or related companies. If you are not the intended recipient of this email, you must neither take any action based upon its contents, nor copy or show it to anyone. Please contact the sender if you believe you have received this email in error. -Original Message- From: Vladislav Nazarenko [mailto:vladislav.nazare...@gmail.com] Sent: Wednesday, 03 February 2016 5:08 PM To: users@cloudstack.apache.org Subject: Self-fencing when storage not available Hi All, I'm testing Cloudstack 4.6 now When doing some tests with storage (nfs), I was wondering, that the hosts self-fence itself by reboot, when a storage not writable ... Even more, my cluster had a working storage and I added one more, which was not writable due to missing user permissions... So effectively all the VMs hard drive were located on the working storage, but the problem with permissions on the new one causes the entire cluster to reboot :( Is this a bug or a correct behavior? Is it able to avoid rebooting at least? Thank you in advance Vlad Find out more about ShapeBlue and our range of CloudStack related services: IaaS Cloud Design & Build<http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge – rapid IaaS deployment framework<http://shapeblue.com/csforge/> CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/> | CloudStack Software Engineering<http://shapeblue.com/cloudstack-software-engineering/> CloudStack Infrastructure Support<http://shapeblue.com/cloudstack-infrastructure-support/> | CloudStack Bootcamp Training Courses<http://shapeblue.com/cloudstack-training/>
RE: Self-fencing when storage not available
Hi As Lucian said , you can edit the script and remove the reboot , but yes this is the normal behaviour of the script Glenn -Original Message- From: Nux! [mailto:n...@li.nux.ro] Sent: Wednesday, 03 February 2016 8:32 PM To: users@cloudstack.apache.org Subject: Re: Self-fencing when storage not available You can modify the script to not reboot, but until we find a better way to deal with it this is correct behaviour. It sucks it reboots VMs on healthy storage though. -- Sent from the Delta quadrant using Borg technology! Nux! www.nux.ro - Original Message - > From: "Vladislav Nazarenko" <vladislav.nazare...@gmail.com> > To: users@cloudstack.apache.org > Sent: Wednesday, 3 February, 2016 16:24:34 > Subject: Re: Self-fencing when storage not available > Hi Glenn, > > we use KVM ... I also found the script: > /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/kvmheartbeat.sh > > Just wondering if we can avoid rebooting ... > > Thanks > Vlad > > On 03.02.2016 16:57, Glenn Wagner wrote: >> Hi Vlad, >> >> Can I ask what hypervisor are you using? >> The default action is to reboot the hosts >> >> This is done in the heatbeat file on xenserver >> /opt/xensource/bin/xenheartbeat.sh >> >> Thanks >> Glenn >> >> >> >> ShapeBlue <http://www.shapeblue.com> >> Glenn Wagner >> Senior Consultant, ShapeBlue >> >> d: * | s: +27 21 527 0091* <tel:%7C%20s:%20+27%2021%20527%200091> >> | m: *+27 73 917 4111* <tel:+27%2073%20917%204111> >> >> e: *glenn.wag...@shapeblue.com | t: * >> <mailto:glenn.wag...@shapeblue.com%20%7C%20t:>| w: >> *www.shapeblue.com* <http://www.shapeblue.com> >> >> a: >> 2nd Floor, Oudehuis Centre, 122 Main Rd, Somerset West Cape Town 7130 >> South Africa >> >> >> Shape Blue Ltd is a company incorporated in England & Wales. >> ShapeBlue Services India LLP is a company incorporated in India and >> is operated under license from Shape Blue Ltd. Shape Blue Brasil >> Consultoria Ltda is a company incorporated in Brasil and is operated >> under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company >> registered by The Republic of South Africa and is traded under >> license from Shape Blue Ltd. ShapeBlue is a registered trademark. >> This email and any attachments to it may be confidential and are >> intended solely for the use of the individual to whom it is addressed. >> Any views or opinions expressed are solely those of the author and do >> not necessarily represent those of Shape Blue Ltd or related >> companies. If you are not the intended recipient of this email, you >> must neither take any action based upon its contents, nor copy or >> show it to anyone. Please contact the sender if you believe you have >> received this email in error. >> >> >> -Original Message- >> From: Vladislav Nazarenko [mailto:vladislav.nazare...@gmail.com] >> Sent: Wednesday, 03 February 2016 5:08 PM >> To: users@cloudstack.apache.org >> Subject: Self-fencing when storage not available >> >> Hi All, >> >> I'm testing Cloudstack 4.6 now >> >> When doing some tests with storage (nfs), I was wondering, that the >> hosts self-fence itself by reboot, when a storage not writable ... >> >> Even more, my cluster had a working storage and I added one more, >> which was not writable due to missing user permissions... So >> effectively all the VMs hard drive were located on the working >> storage, but the problem with permissions on the new one causes the >> entire cluster to reboot :( >> >> Is this a bug or a correct behavior? >> >> Is it able to avoid rebooting at least? >> >> Thank you in advance >> Vlad >> >> >> Find out more about ShapeBlue and our range of CloudStack related >> services: >> IaaS Cloud Design & Build >> <http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge – >> rapid IaaS deployment framework <http://shapeblue.com/csforge/> >> CloudStack Consulting <http://shapeblue.com/cloudstack-consultancy/> >> | CloudStack Software Engineering >> <http://shapeblue.com/cloudstack-software-engineering/> >> CloudStack Infrastructure Support >> <http://shapeblue.com/cloudstack-infrastructure-support/> | >> CloudStack > > Bootcamp Training Courses > > <http://shapeblue.com/cloudstack-training/> Find out more about ShapeBlue and our range of CloudStack related services: IaaS Cloud Design & Build<http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge – rapid IaaS deployment framework<http://shapeblue.com/csforge/> CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/> | CloudStack Software Engineering<http://shapeblue.com/cloudstack-software-engineering/> CloudStack Infrastructure Support<http://shapeblue.com/cloudstack-infrastructure-support/> | CloudStack Bootcamp Training Courses<http://shapeblue.com/cloudstack-training/>
Re: Self-fencing when storage not available
There were probably many discussions around this implementation? Do you have any links to read about that? I'm really interesting in the background, would like to understand why implemented this way ... On 03.02.2016 19:34, Frank Louwers wrote: On 03 Feb 2016, at 19:32, Nux! <n...@li.nux.ro> wrote: You can modify the script to not reboot, but until we find a better way to deal with it this is correct behaviour. It sucks it reboots VMs on healthy storage though. Not only does it reboot on healty nodes, it only works on NFS. Not on another type of primary storage, not on local storage etc... -- Sent from the Delta quadrant using Borg technology! Nux! www.nux.ro - Original Message - From: "Vladislav Nazarenko" <vladislav.nazare...@gmail.com> To: users@cloudstack.apache.org Sent: Wednesday, 3 February, 2016 16:24:34 Subject: Re: Self-fencing when storage not available Hi Glenn, we use KVM ... I also found the script: /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/kvmheartbeat.sh Just wondering if we can avoid rebooting ... Thanks Vlad On 03.02.2016 16:57, Glenn Wagner wrote: Hi Vlad, Can I ask what hypervisor are you using? The default action is to reboot the hosts This is done in the heatbeat file on xenserver /opt/xensource/bin/xenheartbeat.sh Thanks Glenn ShapeBlue <http://www.shapeblue.com> Glenn Wagner Senior Consultant , ShapeBlue d: * | s: +27 21 527 0091* <tel:%7C%20s:%20+27%2021%20527%200091> | m: *+27 73 917 4111* <tel:+27%2073%20917%204111> e: *glenn.wag...@shapeblue.com | t: * <mailto:glenn.wag...@shapeblue.com%20%7C%20t:> | w: *www.shapeblue.com* <http://www.shapeblue.com> a: 2nd Floor, Oudehuis Centre, 122 Main Rd, Somerset West Cape Town 7130 South Africa Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue Services India LLP is a company incorporated in India and is operated under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil and is operated under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic of South Africa and is traded under license from Shape Blue Ltd. ShapeBlue is a registered trademark. This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Shape Blue Ltd or related companies. If you are not the intended recipient of this email, you must neither take any action based upon its contents, nor copy or show it to anyone. Please contact the sender if you believe you have received this email in error. -Original Message- From: Vladislav Nazarenko [mailto:vladislav.nazare...@gmail.com] Sent: Wednesday, 03 February 2016 5:08 PM To: users@cloudstack.apache.org Subject: Self-fencing when storage not available Hi All, I'm testing Cloudstack 4.6 now When doing some tests with storage (nfs), I was wondering, that the hosts self-fence itself by reboot, when a storage not writable ... Even more, my cluster had a working storage and I added one more, which was not writable due to missing user permissions... So effectively all the VMs hard drive were located on the working storage, but the problem with permissions on the new one causes the entire cluster to reboot :( Is this a bug or a correct behavior? Is it able to avoid rebooting at least? Thank you in advance Vlad Find out more about ShapeBlue and our range of CloudStack related services: IaaS Cloud Design & Build <http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge – rapid IaaS deployment framework <http://shapeblue.com/csforge/> CloudStack Consulting <http://shapeblue.com/cloudstack-consultancy/> | CloudStack Software Engineering <http://shapeblue.com/cloudstack-software-engineering/> CloudStack Infrastructure Support <http://shapeblue.com/cloudstack-infrastructure-support/> | CloudStack Bootcamp Training Courses <http://shapeblue.com/cloudstack-training/>
Re: Self-fencing when storage not available
You can modify the script to not reboot, but until we find a better way to deal with it this is correct behaviour. It sucks it reboots VMs on healthy storage though. -- Sent from the Delta quadrant using Borg technology! Nux! www.nux.ro - Original Message - > From: "Vladislav Nazarenko" <vladislav.nazare...@gmail.com> > To: users@cloudstack.apache.org > Sent: Wednesday, 3 February, 2016 16:24:34 > Subject: Re: Self-fencing when storage not available > Hi Glenn, > > we use KVM ... I also found the script: > /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/kvmheartbeat.sh > > Just wondering if we can avoid rebooting ... > > Thanks > Vlad > > On 03.02.2016 16:57, Glenn Wagner wrote: >> Hi Vlad, >> >> Can I ask what hypervisor are you using? >> The default action is to reboot the hosts >> >> This is done in the heatbeat file on xenserver >> /opt/xensource/bin/xenheartbeat.sh >> >> Thanks >> Glenn >> >> >> >> ShapeBlue <http://www.shapeblue.com> >> Glenn Wagner >> Senior Consultant, ShapeBlue >> >> d: * | s: +27 21 527 0091* <tel:%7C%20s:%20+27%2021%20527%200091> >> | m: *+27 73 917 4111* <tel:+27%2073%20917%204111> >> >> e: *glenn.wag...@shapeblue.com | t: * >> <mailto:glenn.wag...@shapeblue.com%20%7C%20t:>| w: >> *www.shapeblue.com* <http://www.shapeblue.com> >> >> a: >> 2nd Floor, Oudehuis Centre, 122 Main Rd, Somerset West Cape Town 7130 South >> Africa >> >> >> Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue >> Services India LLP is a company incorporated in India and is operated >> under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda >> is a company incorporated in Brasil and is operated under license from >> Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The >> Republic of South Africa and is traded under license from Shape Blue >> Ltd. ShapeBlue is a registered trademark. >> This email and any attachments to it may be confidential and are >> intended solely for the use of the individual to whom it is addressed. >> Any views or opinions expressed are solely those of the author and do >> not necessarily represent those of Shape Blue Ltd or related >> companies. If you are not the intended recipient of this email, you >> must neither take any action based upon its contents, nor copy or show >> it to anyone. Please contact the sender if you believe you have >> received this email in error. >> >> >> -Original Message- >> From: Vladislav Nazarenko [mailto:vladislav.nazare...@gmail.com] >> Sent: Wednesday, 03 February 2016 5:08 PM >> To: users@cloudstack.apache.org >> Subject: Self-fencing when storage not available >> >> Hi All, >> >> I'm testing Cloudstack 4.6 now >> >> When doing some tests with storage (nfs), I was wondering, that the >> hosts self-fence itself by reboot, when a storage not writable ... >> >> Even more, my cluster had a working storage and I added one more, >> which was not writable due to missing user permissions... So >> effectively all the VMs hard drive were located on the working >> storage, but the problem with permissions on the new one causes the >> entire cluster to reboot :( >> >> Is this a bug or a correct behavior? >> >> Is it able to avoid rebooting at least? >> >> Thank you in advance >> Vlad >> >> >> Find out more about ShapeBlue and our range of CloudStack related >> services: >> IaaS Cloud Design & Build >> <http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge – rapid >> IaaS deployment framework <http://shapeblue.com/csforge/> >> CloudStack Consulting <http://shapeblue.com/cloudstack-consultancy/> | >> CloudStack Software Engineering >> <http://shapeblue.com/cloudstack-software-engineering/> >> CloudStack Infrastructure Support >> <http://shapeblue.com/cloudstack-infrastructure-support/> | CloudStack > > Bootcamp Training Courses <http://shapeblue.com/cloudstack-training/>
Re: Self-fencing when storage not available
> On 03 Feb 2016, at 19:32, Nux! <n...@li.nux.ro> wrote: > > You can modify the script to not reboot, but until we find a better way to > deal with it this is correct behaviour. It sucks it reboots VMs on healthy > storage though. Not only does it reboot on healty nodes, it only works on NFS. Not on another type of primary storage, not on local storage etc... > > -- > Sent from the Delta quadrant using Borg technology! > > Nux! > www.nux.ro > > - Original Message - >> From: "Vladislav Nazarenko" <vladislav.nazare...@gmail.com> >> To: users@cloudstack.apache.org >> Sent: Wednesday, 3 February, 2016 16:24:34 >> Subject: Re: Self-fencing when storage not available > >> Hi Glenn, >> >> we use KVM ... I also found the script: >> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/kvmheartbeat.sh >> >> Just wondering if we can avoid rebooting ... >> >> Thanks >> Vlad >> >> On 03.02.2016 16:57, Glenn Wagner wrote: >>> Hi Vlad, >>> >>> Can I ask what hypervisor are you using? >>> The default action is to reboot the hosts >>> >>> This is done in the heatbeat file on xenserver >>> /opt/xensource/bin/xenheartbeat.sh >>> >>> Thanks >>> Glenn >>> >>> >>> >>> ShapeBlue <http://www.shapeblue.com> >>> Glenn Wagner >>> Senior Consultant , ShapeBlue >>> >>> d: * | s: +27 21 527 0091* <tel:%7C%20s:%20+27%2021%20527%200091> >>> | m: *+27 73 917 4111* <tel:+27%2073%20917%204111> >>> >>> e: *glenn.wag...@shapeblue.com | t: * >>> <mailto:glenn.wag...@shapeblue.com%20%7C%20t:> | w: >>> *www.shapeblue.com* <http://www.shapeblue.com> >>> >>> a: >>> 2nd Floor, Oudehuis Centre, 122 Main Rd, Somerset West Cape Town 7130 South >>> Africa >>> >>> >>> Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue >>> Services India LLP is a company incorporated in India and is operated >>> under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda >>> is a company incorporated in Brasil and is operated under license from >>> Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The >>> Republic of South Africa and is traded under license from Shape Blue >>> Ltd. ShapeBlue is a registered trademark. >>> This email and any attachments to it may be confidential and are >>> intended solely for the use of the individual to whom it is addressed. >>> Any views or opinions expressed are solely those of the author and do >>> not necessarily represent those of Shape Blue Ltd or related >>> companies. If you are not the intended recipient of this email, you >>> must neither take any action based upon its contents, nor copy or show >>> it to anyone. Please contact the sender if you believe you have >>> received this email in error. >>> >>> >>> -Original Message- >>> From: Vladislav Nazarenko [mailto:vladislav.nazare...@gmail.com] >>> Sent: Wednesday, 03 February 2016 5:08 PM >>> To: users@cloudstack.apache.org >>> Subject: Self-fencing when storage not available >>> >>> Hi All, >>> >>> I'm testing Cloudstack 4.6 now >>> >>> When doing some tests with storage (nfs), I was wondering, that the >>> hosts self-fence itself by reboot, when a storage not writable ... >>> >>> Even more, my cluster had a working storage and I added one more, >>> which was not writable due to missing user permissions... So >>> effectively all the VMs hard drive were located on the working >>> storage, but the problem with permissions on the new one causes the >>> entire cluster to reboot :( >>> >>> Is this a bug or a correct behavior? >>> >>> Is it able to avoid rebooting at least? >>> >>> Thank you in advance >>> Vlad >>> >>> >>> Find out more about ShapeBlue and our range of CloudStack related >>> services: >>> IaaS Cloud Design & Build >>> <http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge – rapid >>> IaaS deployment framework <http://shapeblue.com/csforge/> >>> CloudStack Consulting <http://shapeblue.com/cloudstack-consultancy/> | >>> CloudStack Software Engineering >>> <http://shapeblue.com/cloudstack-software-engineering/> >>> CloudStack Infrastructure Support >>> <http://shapeblue.com/cloudstack-infrastructure-support/> | CloudStack >>> Bootcamp Training Courses <http://shapeblue.com/cloudstack-training/>
