[jira] [Commented] (CLOUDSTACK-3367) When one primary storage fails, all XenServer hosts get rebooted, killing all VMs, even those not on this primary storage.
[ https://issues.apache.org/jira/browse/CLOUDSTACK-3367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15138908#comment-15138908 ] France commented on CLOUDSTACK-3367: :-/ 3 years after, the same issue persists. And it is not just this one. This is why, we have given up on CS and are slowly migrating to Proxmox VE. > When one primary storage fails, all XenServer hosts get rebooted, killing all > VMs, even those not on this primary storage. > -- > > Key: CLOUDSTACK-3367 > URL: https://issues.apache.org/jira/browse/CLOUDSTACK-3367 > Project: CloudStack > Issue Type: Bug > Security Level: Public(Anyone can view this level - this is the > default.) > Components: Management Server, XenServer >Affects Versions: 4.1.0, 4.2.0, 4.5.0, 4.3.1 > Environment: CentOS 6.3, XenServer 6.0.2 + all hotfixes, CloudStack > 4.1.0 >Reporter: France >Assignee: Abhinandan Prateek > Fix For: Future > > > As the title says: if only one of the primary storages fails, all XenServer > hosts get rebooted one by one. Because i have many primary storages, which > are/were running fine with other VMs, rebooting XenServer Hipervisor is an > overkill. Please disable this or implement just stopping/killing the VMs > running on that storage and try to re-attach that storage only. > Problem was reported on the mailing list, as well as a workaround for > XenServer. So i'm not the only one hit by this "bug/feature". Workaround for > now is as follows: > 1. Modify /opt/xensource/bin/xenheartbeat.sh on all your Hosts, commenting > out the two entries which have "reboot -f" > 2. Identify the PID of the script - pidof -x xenheartbeat.sh > 3. Restart the Script - kill > 4. Force reconnect Host from the UI, the script will then re-launch on > reconnect -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CLOUDSTACK-3367) When one primary storage fails, all XenServer hosts get rebooted, killing all VMs, even those not on this primary storage.
[ https://issues.apache.org/jira/browse/CLOUDSTACK-3367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15138445#comment-15138445 ] Paul Angus commented on CLOUDSTACK-3367: OK. But the current solution is actually worse than the original problem. > When one primary storage fails, all XenServer hosts get rebooted, killing all > VMs, even those not on this primary storage. > -- > > Key: CLOUDSTACK-3367 > URL: https://issues.apache.org/jira/browse/CLOUDSTACK-3367 > Project: CloudStack > Issue Type: Bug > Security Level: Public(Anyone can view this level - this is the > default.) > Components: Management Server, XenServer >Affects Versions: 4.1.0, 4.2.0, 4.5.0, 4.3.1 > Environment: CentOS 6.3, XenServer 6.0.2 + all hotfixes, CloudStack > 4.1.0 >Reporter: France >Assignee: Abhinandan Prateek > Fix For: Future > > > As the title says: if only one of the primary storages fails, all XenServer > hosts get rebooted one by one. Because i have many primary storages, which > are/were running fine with other VMs, rebooting XenServer Hipervisor is an > overkill. Please disable this or implement just stopping/killing the VMs > running on that storage and try to re-attach that storage only. > Problem was reported on the mailing list, as well as a workaround for > XenServer. So i'm not the only one hit by this "bug/feature". Workaround for > now is as follows: > 1. Modify /opt/xensource/bin/xenheartbeat.sh on all your Hosts, commenting > out the two entries which have "reboot -f" > 2. Identify the PID of the script - pidof -x xenheartbeat.sh > 3. Restart the Script - kill > 4. Force reconnect Host from the UI, the script will then re-launch on > reconnect -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CLOUDSTACK-3367) When one primary storage fails, all XenServer hosts get rebooted, killing all VMs, even those not on this primary storage.
[ https://issues.apache.org/jira/browse/CLOUDSTACK-3367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15137723#comment-15137723 ] Paul Angus commented on CLOUDSTACK-3367: [~abhi_shapeblue] The behaviour of heartbeat.sh reboots the host if it detects ANY of the storage pools have gone offline. it's too late to protect/shutdown the VMs on that storage as it's already unavailable. As the current behaviour causes the host to restart, any VMs running on 'still good' storage are shut down / reset. - This is the problem. > When one primary storage fails, all XenServer hosts get rebooted, killing all > VMs, even those not on this primary storage. > -- > > Key: CLOUDSTACK-3367 > URL: https://issues.apache.org/jira/browse/CLOUDSTACK-3367 > Project: CloudStack > Issue Type: Bug > Security Level: Public(Anyone can view this level - this is the > default.) > Components: Management Server, XenServer >Affects Versions: 4.1.0, 4.2.0, 4.5.0, 4.3.1 > Environment: CentOS 6.3, XenServer 6.0.2 + all hotfixes, CloudStack > 4.1.0 >Reporter: France >Assignee: Abhinandan Prateek > Fix For: Future > > > As the title says: if only one of the primary storages fails, all XenServer > hosts get rebooted one by one. Because i have many primary storages, which > are/were running fine with other VMs, rebooting XenServer Hipervisor is an > overkill. Please disable this or implement just stopping/killing the VMs > running on that storage and try to re-attach that storage only. > Problem was reported on the mailing list, as well as a workaround for > XenServer. So i'm not the only one hit by this "bug/feature". Workaround for > now is as follows: > 1. Modify /opt/xensource/bin/xenheartbeat.sh on all your Hosts, commenting > out the two entries which have "reboot -f" > 2. Identify the PID of the script - pidof -x xenheartbeat.sh > 3. Restart the Script - kill > 4. Force reconnect Host from the UI, the script will then re-launch on > reconnect -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CLOUDSTACK-3367) When one primary storage fails, all XenServer hosts get rebooted, killing all VMs, even those not on this primary storage.
[ https://issues.apache.org/jira/browse/CLOUDSTACK-3367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15136794#comment-15136794 ] Frank Louwers commented on CLOUDSTACK-3367: --- Please note this is *not* Xenserver specific: KVM has the same issue! > When one primary storage fails, all XenServer hosts get rebooted, killing all > VMs, even those not on this primary storage. > -- > > Key: CLOUDSTACK-3367 > URL: https://issues.apache.org/jira/browse/CLOUDSTACK-3367 > Project: CloudStack > Issue Type: Bug > Security Level: Public(Anyone can view this level - this is the > default.) > Components: Management Server, XenServer >Affects Versions: 4.1.0, 4.2.0, 4.5.0, 4.3.1 > Environment: CentOS 6.3, XenServer 6.0.2 + all hotfixes, CloudStack > 4.1.0 >Reporter: France >Assignee: Abhinandan Prateek > Fix For: Future > > > As the title says: if only one of the primary storages fails, all XenServer > hosts get rebooted one by one. Because i have many primary storages, which > are/were running fine with other VMs, rebooting XenServer Hipervisor is an > overkill. Please disable this or implement just stopping/killing the VMs > running on that storage and try to re-attach that storage only. > Problem was reported on the mailing list, as well as a workaround for > XenServer. So i'm not the only one hit by this "bug/feature". Workaround for > now is as follows: > 1. Modify /opt/xensource/bin/xenheartbeat.sh on all your Hosts, commenting > out the two entries which have "reboot -f" > 2. Identify the PID of the script - pidof -x xenheartbeat.sh > 3. Restart the Script - kill > 4. Force reconnect Host from the UI, the script will then re-launch on > reconnect -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CLOUDSTACK-3367) When one primary storage fails, all XenServer hosts get rebooted, killing all VMs, even those not on this primary storage.
[ https://issues.apache.org/jira/browse/CLOUDSTACK-3367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15136564#comment-15136564 ] Abhinandan Prateek commented on CLOUDSTACK-3367: [~paulangus] the reboot behaviour is to save VMs have disk on the down primary storage. With later releases of Xen it is possible to shutdown those only the VMs that have disk on bad PS. It makes sense to just implement this feature now. This will require the setting to be manipulatable on each host and will not be as simple to control from MS. > When one primary storage fails, all XenServer hosts get rebooted, killing all > VMs, even those not on this primary storage. > -- > > Key: CLOUDSTACK-3367 > URL: https://issues.apache.org/jira/browse/CLOUDSTACK-3367 > Project: CloudStack > Issue Type: Bug > Security Level: Public(Anyone can view this level - this is the > default.) > Components: Management Server, XenServer >Affects Versions: 4.1.0, 4.2.0, 4.5.0, 4.3.1 > Environment: CentOS 6.3, XenServer 6.0.2 + all hotfixes, CloudStack > 4.1.0 >Reporter: France >Assignee: Abhinandan Prateek > Fix For: Future > > > As the title says: if only one of the primary storages fails, all XenServer > hosts get rebooted one by one. Because i have many primary storages, which > are/were running fine with other VMs, rebooting XenServer Hipervisor is an > overkill. Please disable this or implement just stopping/killing the VMs > running on that storage and try to re-attach that storage only. > Problem was reported on the mailing list, as well as a workaround for > XenServer. So i'm not the only one hit by this "bug/feature". Workaround for > now is as follows: > 1. Modify /opt/xensource/bin/xenheartbeat.sh on all your Hosts, commenting > out the two entries which have "reboot -f" > 2. Identify the PID of the script - pidof -x xenheartbeat.sh > 3. Restart the Script - kill > 4. Force reconnect Host from the UI, the script will then re-launch on > reconnect -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CLOUDSTACK-3367) When one primary storage fails, all XenServer hosts get rebooted, killing all VMs, even those not on this primary storage.
[ https://issues.apache.org/jira/browse/CLOUDSTACK-3367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15136411#comment-15136411 ] Paul Angus commented on CLOUDSTACK-3367: [~aprateek] is it feasible to add a global setting which controls this behaviour? While the current behaviour can be useful in some circumstances there are quite a few where it is highly undesirable It would be good if the cloud operators could have control over it. > When one primary storage fails, all XenServer hosts get rebooted, killing all > VMs, even those not on this primary storage. > -- > > Key: CLOUDSTACK-3367 > URL: https://issues.apache.org/jira/browse/CLOUDSTACK-3367 > Project: CloudStack > Issue Type: Bug > Security Level: Public(Anyone can view this level - this is the > default.) > Components: Management Server, XenServer >Affects Versions: 4.1.0, 4.2.0, 4.5.0, 4.3.1 > Environment: CentOS 6.3, XenServer 6.0.2 + all hotfixes, CloudStack > 4.1.0 >Reporter: France >Assignee: Abhinandan Prateek > Fix For: Future > > > As the title says: if only one of the primary storages fails, all XenServer > hosts get rebooted one by one. Because i have many primary storages, which > are/were running fine with other VMs, rebooting XenServer Hipervisor is an > overkill. Please disable this or implement just stopping/killing the VMs > running on that storage and try to re-attach that storage only. > Problem was reported on the mailing list, as well as a workaround for > XenServer. So i'm not the only one hit by this "bug/feature". Workaround for > now is as follows: > 1. Modify /opt/xensource/bin/xenheartbeat.sh on all your Hosts, commenting > out the two entries which have "reboot -f" > 2. Identify the PID of the script - pidof -x xenheartbeat.sh > 3. Restart the Script - kill > 4. Force reconnect Host from the UI, the script will then re-launch on > reconnect -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CLOUDSTACK-3367) When one primary storage fails, all XenServer hosts get rebooted, killing all VMs, even those not on this primary storage.
[ https://issues.apache.org/jira/browse/CLOUDSTACK-3367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14626340#comment-14626340 ] Keerthiraja commented on CLOUDSTACK-3367: - Will this be taken care on current 4.5.2 release > When one primary storage fails, all XenServer hosts get rebooted, killing all > VMs, even those not on this primary storage. > -- > > Key: CLOUDSTACK-3367 > URL: https://issues.apache.org/jira/browse/CLOUDSTACK-3367 > Project: CloudStack > Issue Type: Bug > Security Level: Public(Anyone can view this level - this is the > default.) > Components: Management Server, XenServer >Affects Versions: 4.1.0, 4.2.0, 4.5.0, 4.3.1 > Environment: CentOS 6.3, XenServer 6.0.2 + all hotfixes, CloudStack > 4.1.0 >Reporter: France >Assignee: Abhinandan Prateek > Fix For: Future > > > As the title says: if only one of the primary storages fails, all XenServer > hosts get rebooted one by one. Because i have many primary storages, which > are/were running fine with other VMs, rebooting XenServer Hipervisor is an > overkill. Please disable this or implement just stopping/killing the VMs > running on that storage and try to re-attach that storage only. > Problem was reported on the mailing list, as well as a workaround for > XenServer. So i'm not the only one hit by this "bug/feature". Workaround for > now is as follows: > 1. Modify /opt/xensource/bin/xenheartbeat.sh on all your Hosts, commenting > out the two entries which have "reboot -f" > 2. Identify the PID of the script - pidof -x xenheartbeat.sh > 3. Restart the Script - kill > 4. Force reconnect Host from the UI, the script will then re-launch on > reconnect -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CLOUDSTACK-3367) When one primary storage fails, all XenServer hosts get rebooted, killing all VMs, even those not on this primary storage.
[ https://issues.apache.org/jira/browse/CLOUDSTACK-3367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14233185#comment-14233185 ] John Kinsella commented on CLOUDSTACK-3367: --- Folks - you'll have better success by finding one or more developers who are familiar with the XenServer integration code and asking for their help than just griping in a jira ticket that isn't assigned to anybody. Search the dev list or commits for folks who have worked on the appropriate code in the past, or just start a thread on dev@ with a subject to attract the appropriate folks. Griping, while it might be justified, doesn't tend to gain favor in open source projects where people are volunteering their time. > When one primary storage fails, all XenServer hosts get rebooted, killing all > VMs, even those not on this primary storage. > -- > > Key: CLOUDSTACK-3367 > URL: https://issues.apache.org/jira/browse/CLOUDSTACK-3367 > Project: CloudStack > Issue Type: Bug > Security Level: Public(Anyone can view this level - this is the > default.) > Components: Management Server, XenServer >Affects Versions: 4.1.0, 4.2.0, 4.5.0, 4.3.1 > Environment: CentOS 6.3, XenServer 6.0.2 + all hotfixes, CloudStack > 4.1.0 >Reporter: France > Fix For: Future > > > As the title says: if only one of the primary storages fails, all XenServer > hosts get rebooted one by one. Because i have many primary storages, which > are/were running fine with other VMs, rebooting XenServer Hipervisor is an > overkill. Please disable this or implement just stopping/killing the VMs > running on that storage and try to re-attach that storage only. > Problem was reported on the mailing list, as well as a workaround for > XenServer. So i'm not the only one hit by this "bug/feature". Workaround for > now is as follows: > 1. Modify /opt/xensource/bin/xenheartbeat.sh on all your Hosts, commenting > out the two entries which have "reboot -f" > 2. Identify the PID of the script - pidof -x xenheartbeat.sh > 3. Restart the Script - kill > 4. Force reconnect Host from the UI, the script will then re-launch on > reconnect -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CLOUDSTACK-3367) When one primary storage fails, all XenServer hosts get rebooted, killing all VMs, even those not on this primary storage.
[ https://issues.apache.org/jira/browse/CLOUDSTACK-3367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14232986#comment-14232986 ] France commented on CLOUDSTACK-3367: Soon will be in third year of this critical bug reported... ...still no one cares, if failure on one of primary storages, which happens to be non redundant, hard reboots WHOLE cloud. Or is this not the case anymore with new releases ad it has been fixed? > When one primary storage fails, all XenServer hosts get rebooted, killing all > VMs, even those not on this primary storage. > -- > > Key: CLOUDSTACK-3367 > URL: https://issues.apache.org/jira/browse/CLOUDSTACK-3367 > Project: CloudStack > Issue Type: Bug > Security Level: Public(Anyone can view this level - this is the > default.) > Components: Management Server, XenServer >Affects Versions: 4.1.0, 4.2.0, 4.5.0, 4.3.1 > Environment: CentOS 6.3, XenServer 6.0.2 + all hotfixes, CloudStack > 4.1.0 >Reporter: France > Fix For: Future > > > As the title says: if only one of the primary storages fails, all XenServer > hosts get rebooted one by one. Because i have many primary storages, which > are/were running fine with other VMs, rebooting XenServer Hipervisor is an > overkill. Please disable this or implement just stopping/killing the VMs > running on that storage and try to re-attach that storage only. > Problem was reported on the mailing list, as well as a workaround for > XenServer. So i'm not the only one hit by this "bug/feature". Workaround for > now is as follows: > 1. Modify /opt/xensource/bin/xenheartbeat.sh on all your Hosts, commenting > out the two entries which have "reboot -f" > 2. Identify the PID of the script - pidof -x xenheartbeat.sh > 3. Restart the Script - kill > 4. Force reconnect Host from the UI, the script will then re-launch on > reconnect -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CLOUDSTACK-3367) When one primary storage fails, all XenServer hosts get rebooted, killing all VMs, even those not on this primary storage.
[ https://issues.apache.org/jira/browse/CLOUDSTACK-3367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14146276#comment-14146276 ] France commented on CLOUDSTACK-3367: Anyone willing to pick this up? It has been well over a year by now. :-( > When one primary storage fails, all XenServer hosts get rebooted, killing all > VMs, even those not on this primary storage. > -- > > Key: CLOUDSTACK-3367 > URL: https://issues.apache.org/jira/browse/CLOUDSTACK-3367 > Project: CloudStack > Issue Type: Bug > Security Level: Public(Anyone can view this level - this is the > default.) > Components: Management Server, XenServer >Affects Versions: 4.1.0, 4.2.0, 4.5.0, 4.3.1 > Environment: CentOS 6.3, XenServer 6.0.2 + all hotfixes, CloudStack > 4.1.0 >Reporter: France > Fix For: Future > > > As the title says: if only one of the primary storages fails, all XenServer > hosts get rebooted one by one. Because i have many primary storages, which > are/were running fine with other VMs, rebooting XenServer Hipervisor is an > overkill. Please disable this or implement just stopping/killing the VMs > running on that storage and try to re-attach that storage only. > Problem was reported on the mailing list, as well as a workaround for > XenServer. So i'm not the only one hit by this "bug/feature". Workaround for > now is as follows: > 1. Modify /opt/xensource/bin/xenheartbeat.sh on all your Hosts, commenting > out the two entries which have "reboot -f" > 2. Identify the PID of the script - pidof -x xenheartbeat.sh > 3. Restart the Script - kill > 4. Force reconnect Host from the UI, the script will then re-launch on > reconnect -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CLOUDSTACK-3367) When one primary storage fails, all XenServer hosts get rebooted, killing all VMs, even those not on this primary storage.
[ https://issues.apache.org/jira/browse/CLOUDSTACK-3367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13920762#comment-13920762 ] France commented on CLOUDSTACK-3367: Just an idea for whomever picks this issue up (if anyone at all :( ). Before killing the whole hypervisor host, maybe live migrate instances who's private storage is still functioning to another hypervisor. > When one primary storage fails, all XenServer hosts get rebooted, killing all > VMs, even those not on this primary storage. > -- > > Key: CLOUDSTACK-3367 > URL: https://issues.apache.org/jira/browse/CLOUDSTACK-3367 > Project: CloudStack > Issue Type: Bug > Security Level: Public(Anyone can view this level - this is the > default.) > Components: Management Server, XenServer >Affects Versions: 4.1.0, 4.2.0 > Environment: CentOS 6.3, XenServer 6.0.2 + all hotfixes, CloudStack > 4.1.0 >Reporter: France > Fix For: Future > > > As the title says: if only one of the primary storages fails, all XenServer > hosts get rebooted one by one. Because i have many primary storages, which > are/were running fine with other VMs, rebooting XenServer Hipervisor is an > overkill. Please disable this or implement just stopping/killing the VMs > running on that storage and try to re-attach that storage only. > Problem was reported on the mailing list, as well as a workaround for > XenServer. So i'm not the only one hit by this "bug/feature". Workaround for > now is as follows: > 1. Modify /opt/xensource/bin/xenheartbeat.sh on all your Hosts, commenting > out the two entries which have "reboot -f" > 2. Identify the PID of the script - pidof -x xenheartbeat.sh > 3. Restart the Script - kill > 4. Force reconnect Host from the UI, the script will then re-launch on > reconnect -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CLOUDSTACK-3367) When one primary storage fails, all XenServer hosts get rebooted, killing all VMs, even those not on this primary storage.
[ https://issues.apache.org/jira/browse/CLOUDSTACK-3367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13917862#comment-13917862 ] Koushik Das commented on CLOUDSTACK-3367: - Busy with some other stuff, unassigned so that it can be picked up by anyone else willing to work on it. Initially had planned to fix it as part of CLOUDSTACK-5203 ( https://cwiki.apache.org/confluence/display/CLOUDSTACK/User+VM+HA+using+native+XS+HA+capabilities) but that proposal was dropped due to regressions (http://mail-archives.apache.org/mod_mbox/cloudstack-dev/201311.mbox/%3c20cf38cb4385ce4d9d1558d52a0fc05818f...@sjcpex01cl03.citrite.net%3E). > When one primary storage fails, all XenServer hosts get rebooted, killing all > VMs, even those not on this primary storage. > -- > > Key: CLOUDSTACK-3367 > URL: https://issues.apache.org/jira/browse/CLOUDSTACK-3367 > Project: CloudStack > Issue Type: Bug > Security Level: Public(Anyone can view this level - this is the > default.) > Components: Management Server, XenServer >Affects Versions: 4.1.0, 4.2.0 > Environment: CentOS 6.3, XenServer 6.0.2 + all hotfixes, CloudStack > 4.1.0 >Reporter: France > Fix For: Future > > > As the title says: if only one of the primary storages fails, all XenServer > hosts get rebooted one by one. Because i have many primary storages, which > are/were running fine with other VMs, rebooting XenServer Hipervisor is an > overkill. Please disable this or implement just stopping/killing the VMs > running on that storage and try to re-attach that storage only. > Problem was reported on the mailing list, as well as a workaround for > XenServer. So i'm not the only one hit by this "bug/feature". Workaround for > now is as follows: > 1. Modify /opt/xensource/bin/xenheartbeat.sh on all your Hosts, commenting > out the two entries which have "reboot -f" > 2. Identify the PID of the script - pidof -x xenheartbeat.sh > 3. Restart the Script - kill > 4. Force reconnect Host from the UI, the script will then re-launch on > reconnect -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CLOUDSTACK-3367) When one primary storage fails, all XenServer hosts get rebooted, killing all VMs, even those not on this primary storage.
[ https://issues.apache.org/jira/browse/CLOUDSTACK-3367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13917844#comment-13917844 ] France commented on CLOUDSTACK-3367: LOL, we're rolling backwards on the issue. We just lost assignee. :-) > When one primary storage fails, all XenServer hosts get rebooted, killing all > VMs, even those not on this primary storage. > -- > > Key: CLOUDSTACK-3367 > URL: https://issues.apache.org/jira/browse/CLOUDSTACK-3367 > Project: CloudStack > Issue Type: Bug > Security Level: Public(Anyone can view this level - this is the > default.) > Components: Management Server, XenServer >Affects Versions: 4.1.0, 4.2.0 > Environment: CentOS 6.3, XenServer 6.0.2 + all hotfixes, CloudStack > 4.1.0 >Reporter: France > Fix For: Future > > > As the title says: if only one of the primary storages fails, all XenServer > hosts get rebooted one by one. Because i have many primary storages, which > are/were running fine with other VMs, rebooting XenServer Hipervisor is an > overkill. Please disable this or implement just stopping/killing the VMs > running on that storage and try to re-attach that storage only. > Problem was reported on the mailing list, as well as a workaround for > XenServer. So i'm not the only one hit by this "bug/feature". Workaround for > now is as follows: > 1. Modify /opt/xensource/bin/xenheartbeat.sh on all your Hosts, commenting > out the two entries which have "reboot -f" > 2. Identify the PID of the script - pidof -x xenheartbeat.sh > 3. Restart the Script - kill > 4. Force reconnect Host from the UI, the script will then re-launch on > reconnect -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CLOUDSTACK-3367) When one primary storage fails, all XenServer hosts get rebooted, killing all VMs, even those not on this primary storage.
[ https://issues.apache.org/jira/browse/CLOUDSTACK-3367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13766393#comment-13766393 ] Koushik Das commented on CLOUDSTACK-3367: - In XS 6.2 I simulated SR failure by disconnecting it. After that tried shutting down the VM, the operation succeeded and the VMs power state was made 'halted'. It even allowed destroying the 'halted' VM. So looks like in XS 6.2 selective cleanup of VMs can be done instead of rebooting host. But in older versions of XS there is no way to do selective cleanup. > When one primary storage fails, all XenServer hosts get rebooted, killing all > VMs, even those not on this primary storage. > -- > > Key: CLOUDSTACK-3367 > URL: https://issues.apache.org/jira/browse/CLOUDSTACK-3367 > Project: CloudStack > Issue Type: Bug > Security Level: Public(Anyone can view this level - this is the > default.) > Components: Management Server, XenServer >Affects Versions: 4.1.0, 4.2.0 > Environment: CentOS 6.3, XenServer 6.0.2 + all hotfixes, CloudStack > 4.1.0 >Reporter: France > Fix For: Future > > > As the title says: if only one of the primary storages fails, all XenServer > hosts get rebooted one by one. Because i have many primary storages, which > are/were running fine with other VMs, rebooting XenServer Hipervisor is an > overkill. Please disable this or implement just stopping/killing the VMs > running on that storage and try to re-attach that storage only. > Problem was reported on the mailing list, as well as a workaround for > XenServer. So i'm not the only one hit by this "bug/feature". Workaround for > now is as follows: > 1. Modify /opt/xensource/bin/xenheartbeat.sh on all your Hosts, commenting > out the two entries which have "reboot -f" > 2. Identify the PID of the script - pidof -x xenheartbeat.sh > 3. Restart the Script - kill > 4. Force reconnect Host from the UI, the script will then re-launch on > reconnect -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CLOUDSTACK-3367) When one primary storage fails, all XenServer hosts get rebooted, killing all VMs, even those not on this primary storage.
[ https://issues.apache.org/jira/browse/CLOUDSTACK-3367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13729501#comment-13729501 ] Jiri Kanicky commented on CLOUDSTACK-3367: -- I agree with Alex in regards to XenServers. Disabling the reboot might cause problems on the XenServer, because any failure of storage will leave staled mounts on the host. > When one primary storage fails, all XenServer hosts get rebooted, killing all > VMs, even those not on this primary storage. > -- > > Key: CLOUDSTACK-3367 > URL: https://issues.apache.org/jira/browse/CLOUDSTACK-3367 > Project: CloudStack > Issue Type: Bug > Security Level: Public(Anyone can view this level - this is the > default.) > Components: Management Server, XenServer >Affects Versions: 4.1.0, 4.2.0 > Environment: CentOS 6.3, XenServer 6.0.2 + all hotfixes, CloudStack > 4.1.0 >Reporter: France > Fix For: Future > > > As the title says: if only one of the primary storages fails, all XenServer > hosts get rebooted one by one. Because i have many primary storages, which > are/were running fine with other VMs, rebooting XenServer Hipervisor is an > overkill. Please disable this or implement just stopping/killing the VMs > running on that storage and try to re-attach that storage only. > Problem was reported on the mailing list, as well as a workaround for > XenServer. So i'm not the only one hit by this "bug/feature". Workaround for > now is as follows: > 1. Modify /opt/xensource/bin/xenheartbeat.sh on all your Hosts, commenting > out the two entries which have "reboot -f" > 2. Identify the PID of the script - pidof -x xenheartbeat.sh > 3. Restart the Script - kill > 4. Force reconnect Host from the UI, the script will then re-launch on > reconnect -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CLOUDSTACK-3367) When one primary storage fails, all XenServer hosts get rebooted, killing all VMs, even those not on this primary storage.
[ https://issues.apache.org/jira/browse/CLOUDSTACK-3367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13721491#comment-13721491 ] Alex Huang commented on CLOUDSTACK-3367: Our experience in testing this with the 5.6 version of XenServer is that if we attempt to stop the VMs with XenServer while the storage is out, that XenServer may not shut them down cleanly due to storage problems, leading to further problems down the road. It's the reason why we chose to reboot instead of stop VMs. You also have to consider how often this happens. If a storage server needs to be taken out, the storage server should be put in maintenance mode which shutdown the vms. In that case, then it won't cause host to reboot. Therefore, this can only happen with an unscheduled outage of the storage server. We can add a few things to make this happen less often. - Don't put a heartbeat on the storage until a VM using that storage is on a host. - Remove the heartbeat on the storage when all VMs using that storage is done. - Try to stop the VMs within a short interval and if by that interval we can't stop the VMs, then reboot. > When one primary storage fails, all XenServer hosts get rebooted, killing all > VMs, even those not on this primary storage. > -- > > Key: CLOUDSTACK-3367 > URL: https://issues.apache.org/jira/browse/CLOUDSTACK-3367 > Project: CloudStack > Issue Type: Bug > Security Level: Public(Anyone can view this level - this is the > default.) > Components: Management Server, XenServer >Affects Versions: 4.1.0, 4.2.0 > Environment: CentOS 6.3, XenServer 6.0.2 + all hotfixes, CloudStack > 4.1.0 >Reporter: France >Priority: Critical > Fix For: Future > > > As the title says: if only one of the primary storages fails, all XenServer > hosts get rebooted one by one. Because i have many primary storages, which > are/were running fine with other VMs, rebooting XenServer Hipervisor is an > overkill. Please disable this or implement just stopping/killing the VMs > running on that storage and try to re-attach that storage only. > Problem was reported on the mailing list, as well as a workaround for > XenServer. So i'm not the only one hit by this "bug/feature". Workaround for > now is as follows: > 1. Modify /opt/xensource/bin/xenheartbeat.sh on all your Hosts, commenting > out the two entries which have "reboot -f" > 2. Identify the PID of the script - pidof -x xenheartbeat.sh > 3. Restart the Script - kill > 4. Force reconnect Host from the UI, the script will then re-launch on > reconnect -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CLOUDSTACK-3367) When one primary storage fails, all XenServer hosts get rebooted, killing all VMs, even those not on this primary storage.
[ https://issues.apache.org/jira/browse/CLOUDSTACK-3367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13720748#comment-13720748 ] France commented on CLOUDSTACK-3367: I agree with your path to fix, but i disagree that not killing VMs, which have no issues, is an improvement or a new feature. If you kill/destroy/stop something that's working normally and you should not have, it's definitely a bug. A mayor bug. :-) > When one primary storage fails, all XenServer hosts get rebooted, killing all > VMs, even those not on this primary storage. > -- > > Key: CLOUDSTACK-3367 > URL: https://issues.apache.org/jira/browse/CLOUDSTACK-3367 > Project: CloudStack > Issue Type: Improvement > Security Level: Public(Anyone can view this level - this is the > default.) > Components: Management Server, XenServer >Affects Versions: 4.1.0, 4.2.0 > Environment: CentOS 6.3, XenServer 6.0.2 + all hotfixes, CloudStack > 4.1.0 >Reporter: France > Fix For: Future > > > As the title says: if only one of the primary storages fails, all XenServer > hosts get rebooted one by one. Because i have many primary storages, which > are/were running fine with other VMs, rebooting XenServer Hipervisor is an > overkill. Please disable this or implement just stopping/killing the VMs > running on that storage and try to re-attach that storage only. > Problem was reported on the mailing list, as well as a workaround for > XenServer. So i'm not the only one hit by this "bug/feature". Workaround for > now is as follows: > 1. Modify /opt/xensource/bin/xenheartbeat.sh on all your Hosts, commenting > out the two entries which have "reboot -f" > 2. Identify the PID of the script - pidof -x xenheartbeat.sh > 3. Restart the Script - kill > 4. Force reconnect Host from the UI, the script will then re-launch on > reconnect -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CLOUDSTACK-3367) When one primary storage fails, all XenServer hosts get rebooted, killing all VMs, even those not on this primary storage.
[ https://issues.apache.org/jira/browse/CLOUDSTACK-3367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13720725#comment-13720725 ] Abhinandan Prateek commented on CLOUDSTACK-3367: The heartbeat script should only be modified after we fix the issue of stopping that VMs that are running on that storage. Without a reboot these VMs will become inconsistent. I think it should be classified as an improvement. > When one primary storage fails, all XenServer hosts get rebooted, killing all > VMs, even those not on this primary storage. > -- > > Key: CLOUDSTACK-3367 > URL: https://issues.apache.org/jira/browse/CLOUDSTACK-3367 > Project: CloudStack > Issue Type: Bug > Security Level: Public(Anyone can view this level - this is the > default.) > Components: Management Server, XenServer >Affects Versions: 4.1.0, 4.2.0 > Environment: CentOS 6.3, XenServer 6.0.2 + all hotfixes, CloudStack > 4.1.0 >Reporter: France > > As the title says: if only one of the primary storages fails, all XenServer > hosts get rebooted one by one. Because i have many primary storages, which > are/were running fine with other VMs, rebooting XenServer Hipervisor is an > overkill. Please disable this or implement just stopping/killing the VMs > running on that storage and try to re-attach that storage only. > Problem was reported on the mailing list, as well as a workaround for > XenServer. So i'm not the only one hit by this "bug/feature". Workaround for > now is as follows: > 1. Modify /opt/xensource/bin/xenheartbeat.sh on all your Hosts, commenting > out the two entries which have "reboot -f" > 2. Identify the PID of the script - pidof -x xenheartbeat.sh > 3. Restart the Script - kill > 4. Force reconnect Host from the UI, the script will then re-launch on > reconnect -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
