CLOUDSTACK VM HA
Hi Community, I am testing High Availability (HA) for Cloudstack VMs using KVM and NFS. To simulate a host failure, I am triggering a kernel panic. When the command is executed, the host status remains "up" for 3-4 minutes before transitioning to an "Alert '' state and not marked as "Down ". As a result, the VMs are not being migrated and still appear to be running on the downed host server. My questions are: 1. Can we reduce the detection time for a downed host to about 30 seconds? 2. What could be the reason for hosts entering an "Alert" state after a kernel panic, which is preventing VM HA from functioning properly? Alex.
Re: Cloudstack VM HA with 2 primary storage
Yu Huang, I guess nobody has done this (as you are not getting any responses) Did you do further experiments ? how did you fare? On Thu, Nov 23, 2023 at 7:50 AM Yu Huang Chan wrote: > > Hi All, > > According to the CloudStock documentation, HA for hosts and HA-enabled KVM > VMs only work with primary storage and must have iSCSI or NFS storage. > In our current setup, the primary storage is Linstor raw block storage. > To enable HA for hosts and HA-enabled KVM VM features, is it possible that > we create another primary storage NFS with a capacity of only 200 MB for > the heartbeat file? > > Hence, our setup will have two primary storages: the first is raw block > storage, and the second is a small NFS with 200MB. Is it workable in this > scenario? > > Any recommendations and best practices for production setup? > > Regards, > Yu Huang -- Daan
Cloudstack VM HA with 2 primary storage
Hi All, According to the CloudStock documentation, HA for hosts and HA-enabled KVM VMs only work with primary storage and must have iSCSI or NFS storage. In our current setup, the primary storage is Linstor raw block storage. To enable HA for hosts and HA-enabled KVM VM features, is it possible that we create another primary storage NFS with a capacity of only 200 MB for the heartbeat file? Hence, our setup will have two primary storages: the first is raw block storage, and the second is a small NFS with 200MB. Is it workable in this scenario? Any recommendations and best practices for production setup? Regards, Yu Huang
Re: AW: AW: Cloudstack VM HA
Hello Bryan, your understanding is correct to my knowledge and experience. Host HA is restarting all affected VMs on a still running Host in case of a Host failure. VM HA is restarting the VM in case the VM fails / is not running even though it should. VM HA will even (re)-start the VM in case you shutdown the VM on OS level, as according to CS the VM should be in running state. For the test case you described, shutting down a host and expecting all VMs to be restarted on a another host, you would need Host HA. I cannot stress the remark of Nux enough - you need a really stable NFS for Host HA on KVM. We learned it the hard way using Ceph with HA NFS Gateways, which, in our case, was not stable enough for Host HA resulting in all CloudStack hosts rebooting unexpectedly for example during Ceph Host reboots/updates. I am very curious about to hear about altenatives to NFS based Host HA on CCC. Best regards, Stephan > m...@swen.io hat am 11.10.2023 16:23 CEST geschrieben: > > > At the moment you need a nfs storage as nux wrote. Without it you are unable > to use host ha. > > As far as I understand you can use both ha options and host ha will start vms > on another host if a host will be unavailable. But I am not 100% sure about > this. > > Regards, > Swen > > -Ursprüngliche Nachricht- > Von: Bryan Tiang > Gesendet: Mittwoch, 11. Oktober 2023 15:48 > An: users@cloudstack.apache.org > Betreff: Re: AW: Cloudstack VM HA > > Hi Nix and Swen, > > Thanks for the input! Just curious, can VM HA and Host HA be enabled at the > same time? > > In our case, using Cloudstack + Linstor. > > And to clarify my understanding. Host HA migrates VMs to another Host if > Cloudstack detects the physical host to be unhealthy, right? That’s all? > > Regards, > Bryan > On 11 Oct 2023 at 7:48 PM +0800, m...@swen.io, wrote: > > Hi Bryan, > > > > we are testing the exact same scenario at the moment! :-) > > > > As far as I understand CS has 2 different HA. VM HA and Host HA. When > > talking about VM HA the VM needs to use an offering with ha is enabled. CS > > is now checking if the VM is running and if it is not running it will > > restart or recreate it. You can test this when destroying a vm via virsh > > destroy on KVM directly. CS will restart this VM. > > > > Host HA only works, as NUX wrote, with NFS-storage at the moment. As far as > > I know StorPool is developing a new framework so other storages can be used > > for host ha in the future. I read something on the ccc agenda. > > > > Regards, > > Swen > > > > -Ursprüngliche Nachricht- > > Von: Bryan Tiang > > Gesendet: Dienstag, 10. Oktober 2023 12:36 > > An: users@cloudstack.apache.org > > Betreff: Cloudstack VM HA > > > > Hi All, > > > > We are setting up Cloudstack + Linbit SDS (via plugin). Hypervisor is > > Ubuntu. > > > > We are trying to test the VM HA by powering down a physical node at random. > > However, the VMs doesn’t seem to be failing over to the other nodes. > > > > VM HA is enabled already, is there something we are missing? > > > > Regards, > > Bryan > > > >
Re: AW: Cloudstack VM HA
Sharing this wiki page here, the current implementation might have changed from this but still can be used as a reference: https://cwiki.apache.org/confluence/display/CLOUDSTACK/High+Availability+Developer%27s+Guide -Jithin From: Nux Date: Wednesday, 11 October 2023 at 8:38 PM To: users@cloudstack.apache.org Cc: Bryan Tiang Subject: Re: AW: Cloudstack VM HA What I learned in practice is that enabling Host HA affects VM HA.. in that VM HA no longer works. :) So what does Host HA do? It'll reboot the hypervisor via IPMI if it is deemed unreachable. While the hypervisor is down or rebooting the VMs CANNOT be moved/started on another hypervisor. What does VM HA do? It'll make sure VMs on a HA offering will be restarted (possibly on another hypervisor) if it is deemed down. Possible scenarios where VM HA would kick in: - hypervisor crashed and Cloudstack marked the VMs on it as down - user has powered off the VM from within (poweroff via ssh for example), Cloudstack will notice it is down and restart it As part of VM HA and for data integrity, a hypervisor will keep a heartbeat file (sort of lock file) on the NFS primary storage - if the the NFS share has gone away it will assume it is in a network split or has lost access to the storage and will forcefully reboot itself. This is where that happens: https://github.com/apache/cloudstack/blob/d2ad9363a264290e9e5ee58db4a745cbb0e1c62a/scripts/vm/hypervisor/kvm/kvmheartbeat.sh#L162 HTH On 2023-10-11 14:47, Bryan Tiang wrote: > Hi Nix and Swen, > > Thanks for the input! Just curious, can VM HA and Host HA be enabled at > the same time? > > In our case, using Cloudstack + Linstor. > > And to clarify my understanding. Host HA migrates VMs to another Host > if Cloudstack detects the physical host to be unhealthy, right? That’s > all? > > Regards, > Bryan > On 11 Oct 2023 at 7:48 PM +0800, m...@swen.io, wrote: >> Hi Bryan, >> >> we are testing the exact same scenario at the moment! :-) >> >> As far as I understand CS has 2 different HA. VM HA and Host HA. When >> talking about VM HA the VM needs to use an offering with ha is >> enabled. CS is now checking if the VM is running and if it is not >> running it will restart or recreate it. You can test this when >> destroying a vm via virsh destroy on KVM directly. CS will restart >> this VM. >> >> Host HA only works, as NUX wrote, with NFS-storage at the moment. As >> far as I know StorPool is developing a new framework so other storages >> can be used for host ha in the future. I read something on the ccc >> agenda. >> >> Regards, >> Swen >> >> -----Ursprüngliche Nachricht- >> Von: Bryan Tiang >> Gesendet: Dienstag, 10. Oktober 2023 12:36 >> An: users@cloudstack.apache.org >> Betreff: Cloudstack VM HA >> >> Hi All, >> >> We are setting up Cloudstack + Linbit SDS (via plugin). Hypervisor is >> Ubuntu. >> >> We are trying to test the VM HA by powering down a physical node at >> random. However, the VMs doesn’t seem to be failing over to the other >> nodes. >> >> VM HA is enabled already, is there something we are missing? >> >> Regards, >> Bryan >> >>
Re: AW: Cloudstack VM HA
What I learned in practice is that enabling Host HA affects VM HA.. in that VM HA no longer works. :) So what does Host HA do? It'll reboot the hypervisor via IPMI if it is deemed unreachable. While the hypervisor is down or rebooting the VMs CANNOT be moved/started on another hypervisor. What does VM HA do? It'll make sure VMs on a HA offering will be restarted (possibly on another hypervisor) if it is deemed down. Possible scenarios where VM HA would kick in: - hypervisor crashed and Cloudstack marked the VMs on it as down - user has powered off the VM from within (poweroff via ssh for example), Cloudstack will notice it is down and restart it As part of VM HA and for data integrity, a hypervisor will keep a heartbeat file (sort of lock file) on the NFS primary storage - if the the NFS share has gone away it will assume it is in a network split or has lost access to the storage and will forcefully reboot itself. This is where that happens: https://github.com/apache/cloudstack/blob/d2ad9363a264290e9e5ee58db4a745cbb0e1c62a/scripts/vm/hypervisor/kvm/kvmheartbeat.sh#L162 HTH On 2023-10-11 14:47, Bryan Tiang wrote: Hi Nix and Swen, Thanks for the input! Just curious, can VM HA and Host HA be enabled at the same time? In our case, using Cloudstack + Linstor. And to clarify my understanding. Host HA migrates VMs to another Host if Cloudstack detects the physical host to be unhealthy, right? That’s all? Regards, Bryan On 11 Oct 2023 at 7:48 PM +0800, m...@swen.io, wrote: Hi Bryan, we are testing the exact same scenario at the moment! :-) As far as I understand CS has 2 different HA. VM HA and Host HA. When talking about VM HA the VM needs to use an offering with ha is enabled. CS is now checking if the VM is running and if it is not running it will restart or recreate it. You can test this when destroying a vm via virsh destroy on KVM directly. CS will restart this VM. Host HA only works, as NUX wrote, with NFS-storage at the moment. As far as I know StorPool is developing a new framework so other storages can be used for host ha in the future. I read something on the ccc agenda. Regards, Swen -Ursprüngliche Nachricht- Von: Bryan Tiang Gesendet: Dienstag, 10. Oktober 2023 12:36 An: users@cloudstack.apache.org Betreff: Cloudstack VM HA Hi All, We are setting up Cloudstack + Linbit SDS (via plugin). Hypervisor is Ubuntu. We are trying to test the VM HA by powering down a physical node at random. However, the VMs doesn’t seem to be failing over to the other nodes. VM HA is enabled already, is there something we are missing? Regards, Bryan
Re: Cloudstack VM HA
Hi Nux, My deployment is using KVM on centos 7 and NFS as primary storage. Even after enabling HA HA State is showing as "Ineligible" on all 3 KVM hosts. Did I miss something? Reg, Jon On Tue, 10 Oct, 2023, 19:06 Nux, wrote: > Hello, > > You need a stable NFS primary storage for the hearbeat file. > You can keep it in disabled state after the testing - so VMs do not get > created there - but it needs to be present. > Watch out, if the NFS storage becomes unstable or unreachable via > network (switch fault etc) the hypervisors will force reboot themselves. > > > On 2023-10-10 11:35, Bryan Tiang wrote: > > Hi All, > > > > We are setting up Cloudstack + Linbit SDS (via plugin). Hypervisor is > > Ubuntu. > > > > We are trying to test the VM HA by powering down a physical node at > > random. However, the VMs doesn’t seem to be failing over to the other > > nodes. > > > > VM HA is enabled already, is there something we are missing? > > > > Regards, > > Bryan >
AW: AW: Cloudstack VM HA
At the moment you need a nfs storage as nux wrote. Without it you are unable to use host ha. As far as I understand you can use both ha options and host ha will start vms on another host if a host will be unavailable. But I am not 100% sure about this. Regards, Swen -Ursprüngliche Nachricht- Von: Bryan Tiang Gesendet: Mittwoch, 11. Oktober 2023 15:48 An: users@cloudstack.apache.org Betreff: Re: AW: Cloudstack VM HA Hi Nix and Swen, Thanks for the input! Just curious, can VM HA and Host HA be enabled at the same time? In our case, using Cloudstack + Linstor. And to clarify my understanding. Host HA migrates VMs to another Host if Cloudstack detects the physical host to be unhealthy, right? That’s all? Regards, Bryan On 11 Oct 2023 at 7:48 PM +0800, m...@swen.io, wrote: > Hi Bryan, > > we are testing the exact same scenario at the moment! :-) > > As far as I understand CS has 2 different HA. VM HA and Host HA. When talking > about VM HA the VM needs to use an offering with ha is enabled. CS is now > checking if the VM is running and if it is not running it will restart or > recreate it. You can test this when destroying a vm via virsh destroy on KVM > directly. CS will restart this VM. > > Host HA only works, as NUX wrote, with NFS-storage at the moment. As far as I > know StorPool is developing a new framework so other storages can be used for > host ha in the future. I read something on the ccc agenda. > > Regards, > Swen > > -Ursprüngliche Nachricht- > Von: Bryan Tiang > Gesendet: Dienstag, 10. Oktober 2023 12:36 > An: users@cloudstack.apache.org > Betreff: Cloudstack VM HA > > Hi All, > > We are setting up Cloudstack + Linbit SDS (via plugin). Hypervisor is Ubuntu. > > We are trying to test the VM HA by powering down a physical node at random. > However, the VMs doesn’t seem to be failing over to the other nodes. > > VM HA is enabled already, is there something we are missing? > > Regards, > Bryan > >
Re: AW: Cloudstack VM HA
Hi Nix and Swen, Thanks for the input! Just curious, can VM HA and Host HA be enabled at the same time? In our case, using Cloudstack + Linstor. And to clarify my understanding. Host HA migrates VMs to another Host if Cloudstack detects the physical host to be unhealthy, right? That’s all? Regards, Bryan On 11 Oct 2023 at 7:48 PM +0800, m...@swen.io, wrote: > Hi Bryan, > > we are testing the exact same scenario at the moment! :-) > > As far as I understand CS has 2 different HA. VM HA and Host HA. When talking > about VM HA the VM needs to use an offering with ha is enabled. CS is now > checking if the VM is running and if it is not running it will restart or > recreate it. You can test this when destroying a vm via virsh destroy on KVM > directly. CS will restart this VM. > > Host HA only works, as NUX wrote, with NFS-storage at the moment. As far as I > know StorPool is developing a new framework so other storages can be used for > host ha in the future. I read something on the ccc agenda. > > Regards, > Swen > > -Ursprüngliche Nachricht- > Von: Bryan Tiang > Gesendet: Dienstag, 10. Oktober 2023 12:36 > An: users@cloudstack.apache.org > Betreff: Cloudstack VM HA > > Hi All, > > We are setting up Cloudstack + Linbit SDS (via plugin). Hypervisor is Ubuntu. > > We are trying to test the VM HA by powering down a physical node at random. > However, the VMs doesn’t seem to be failing over to the other nodes. > > VM HA is enabled already, is there something we are missing? > > Regards, > Bryan > >
AW: Cloudstack VM HA
Hi Bryan, we are testing the exact same scenario at the moment! :-) As far as I understand CS has 2 different HA. VM HA and Host HA. When talking about VM HA the VM needs to use an offering with ha is enabled. CS is now checking if the VM is running and if it is not running it will restart or recreate it. You can test this when destroying a vm via virsh destroy on KVM directly. CS will restart this VM. Host HA only works, as NUX wrote, with NFS-storage at the moment. As far as I know StorPool is developing a new framework so other storages can be used for host ha in the future. I read something on the ccc agenda. Regards, Swen -Ursprüngliche Nachricht- Von: Bryan Tiang Gesendet: Dienstag, 10. Oktober 2023 12:36 An: users@cloudstack.apache.org Betreff: Cloudstack VM HA Hi All, We are setting up Cloudstack + Linbit SDS (via plugin). Hypervisor is Ubuntu. We are trying to test the VM HA by powering down a physical node at random. However, the VMs doesn’t seem to be failing over to the other nodes. VM HA is enabled already, is there something we are missing? Regards, Bryan
Re: Cloudstack VM HA
Hello, You need a stable NFS primary storage for the hearbeat file. You can keep it in disabled state after the testing - so VMs do not get created there - but it needs to be present. Watch out, if the NFS storage becomes unstable or unreachable via network (switch fault etc) the hypervisors will force reboot themselves. On 2023-10-10 11:35, Bryan Tiang wrote: Hi All, We are setting up Cloudstack + Linbit SDS (via plugin). Hypervisor is Ubuntu. We are trying to test the VM HA by powering down a physical node at random. However, the VMs doesn’t seem to be failing over to the other nodes. VM HA is enabled already, is there something we are missing? Regards, Bryan
Cloudstack VM HA
Hi All, We are setting up Cloudstack + Linbit SDS (via plugin). Hypervisor is Ubuntu. We are trying to test the VM HA by powering down a physical node at random. However, the VMs doesn’t seem to be failing over to the other nodes. VM HA is enabled already, is there something we are missing? Regards, Bryan
RE: Urgent Issue : Cloudstack VM HA not working
On my log cloudstack is showing that the vm's are restarted after he ditects the host in down state,but physically the vm's are not started . Need to investigate this, can you post the logs somewhere so we can take a look? In addition, can you verify if the host (host_id in table vm_instance) where CS thinks the VM is running is the same as the host where the VMs is in halted state? Regards, Somesh -Original Message- From: Fedi Ben Ali [mailto:ben.ali.fe...@gmail.com] Sent: Saturday, August 15, 2015 5:41 AM To: users@cloudstack.apache.org Cc: prapul cool Subject: Re: Urgent Issue : Cloudstack VM HA not working Hi , I'm working on FC as primary storage ,can that be related to this issue ? On my log cloudstack is showing that the vm's are restarted after he ditects the host in down state,but physically the vm's are not started . on my xenserver i have all the xenserver patches installed. 2015-08-15 9:21 GMT+03:00 Remi Bergsma rberg...@schubergphilis.com: Hi, Next to CloudStack reporting Up, also double check the management log en verify it can actually connect to the hypervisor and talk to it. 4.4.2 had issues with these kind of things. What patches have you installed on xenserver? Can you put the mgt logs somewhere so we can take a look? Please consider upgrading as unfortunately 4.4.2 wasn't our best release. The latest 4.4.4 is great and has 100+ bugs fixed. I am not saying it will fix this, but it would be better anyway. Regards, Remi On 14 Aug 2015, at 23:36, Somesh Naidu somesh.na...@citrix.com wrote: in the cloudDB its running That would explain why HA did not trigger on the VM. Is the host marked as Up, as in connected to CS? Regards, Somesh -Original Message- From: Fedi Ben Ali [mailto:ben.ali.fe...@gmail.com] Sent: Friday, August 14, 2015 5:23 PM To: users@cloudstack.apache.org Cc: prapul cool Subject: Re: Urgent Issue : Cloudstack VM HA not working hi, On the xencenter its halted and in the cloudDB its running. The vmsync process is not getting any information about the VM. 2015-08-14 18:46 GMT+03:00 Somesh Naidu somesh.na...@citrix.com: Fedi, I am assuming VM has HA flag set in its Compute Offering. When you stop the VM from XenCenter, what state is it in XenSever (halted or does it disappear) and in Cloud DB (Running or Stopped)? What does vmsync for that host do with this VM (look at management server logs for this)? Regards, Somesh -Original Message- From: Fedi Ben Ali [mailto:ben.ali.fe...@gmail.com] Sent: Friday, August 14, 2015 7:59 AM To: users@cloudstack.apache.org Cc: prapul cool Subject: Re: Urgent Issue : Cloudstack VM HA not working Hi, thanks for your replys, but all those links are basically discribing how to do the pool-master failover (i already made those the configuration of xenserver ha ). but in my case i'm not recovering from a slave faillure ,even when i stop a vm from xencenter ,cloudstack do not start it as described in the documentation. Best regards. 2015-08-14 12:26 GMT+03:00 Remi Bergsma rberg...@schubergphilis.com: Hi, This may also be of help: http://blog.remibergsma.com/2015/05/23/making-xenserver-and-cloudstack-sing-and-dance-together-again/ I’d also consider upgrading to 4.4.4. I just did a 4.4.2 - 4.4.4 upgrade and it went without issues. It has many fixes over 4.4.2. Regards, Remi On 14 Aug 2015, at 08:05, prapul cool prapul_cool2...@yahoo.com.INVALID mailto:prapul_cool2...@yahoo.com.INVALID wrote: Hi ,From Xenserver6.2 onwards, you need to enabled native HA at pool level for cloudstack HA to work.please refer the link for more details. http://www.shapeblue.com/xenserver-native-ha-with-cloudstack/ Regards, Prapul sriram, Cloud Architect, PSI Cloud Solutions, Psiclouds.comhttp://Psiclouds.com. On Friday, 14 August 2015 3:31 AM, Fedi Ben Ali ben.ali.fe...@gmail.commailto:ben.ali.fe...@gmail.com wrote: hello , I'm working on cloudstack 4.4.2 and xenserver 6.2 cluster with 5 hosts, i deployed many virtual machines with compute offerings having ha feature enabled. But when i stops one of the pools hosts , happens all VM steel set to running on cloudstack and no ha is done. Even when i stop a vm from the inside (setting command init 0 ) ,cloudstack suppose to restart the VM ,but he is not doing so. This is very weird but can any one help ,this sooo urgent . Thx
Re: Urgent Issue : Cloudstack VM HA not working
Hi , I'm working on FC as primary storage ,can that be related to this issue ? On my log cloudstack is showing that the vm's are restarted after he ditects the host in down state,but physically the vm's are not started . on my xenserver i have all the xenserver patches installed. 2015-08-15 9:21 GMT+03:00 Remi Bergsma rberg...@schubergphilis.com: Hi, Next to CloudStack reporting Up, also double check the management log en verify it can actually connect to the hypervisor and talk to it. 4.4.2 had issues with these kind of things. What patches have you installed on xenserver? Can you put the mgt logs somewhere so we can take a look? Please consider upgrading as unfortunately 4.4.2 wasn't our best release. The latest 4.4.4 is great and has 100+ bugs fixed. I am not saying it will fix this, but it would be better anyway. Regards, Remi On 14 Aug 2015, at 23:36, Somesh Naidu somesh.na...@citrix.com wrote: in the cloudDB its running That would explain why HA did not trigger on the VM. Is the host marked as Up, as in connected to CS? Regards, Somesh -Original Message- From: Fedi Ben Ali [mailto:ben.ali.fe...@gmail.com] Sent: Friday, August 14, 2015 5:23 PM To: users@cloudstack.apache.org Cc: prapul cool Subject: Re: Urgent Issue : Cloudstack VM HA not working hi, On the xencenter its halted and in the cloudDB its running. The vmsync process is not getting any information about the VM. 2015-08-14 18:46 GMT+03:00 Somesh Naidu somesh.na...@citrix.com: Fedi, I am assuming VM has HA flag set in its Compute Offering. When you stop the VM from XenCenter, what state is it in XenSever (halted or does it disappear) and in Cloud DB (Running or Stopped)? What does vmsync for that host do with this VM (look at management server logs for this)? Regards, Somesh -Original Message- From: Fedi Ben Ali [mailto:ben.ali.fe...@gmail.com] Sent: Friday, August 14, 2015 7:59 AM To: users@cloudstack.apache.org Cc: prapul cool Subject: Re: Urgent Issue : Cloudstack VM HA not working Hi, thanks for your replys, but all those links are basically discribing how to do the pool-master failover (i already made those the configuration of xenserver ha ). but in my case i'm not recovering from a slave faillure ,even when i stop a vm from xencenter ,cloudstack do not start it as described in the documentation. Best regards. 2015-08-14 12:26 GMT+03:00 Remi Bergsma rberg...@schubergphilis.com: Hi, This may also be of help: http://blog.remibergsma.com/2015/05/23/making-xenserver-and-cloudstack-sing-and-dance-together-again/ I’d also consider upgrading to 4.4.4. I just did a 4.4.2 - 4.4.4 upgrade and it went without issues. It has many fixes over 4.4.2. Regards, Remi On 14 Aug 2015, at 08:05, prapul cool prapul_cool2...@yahoo.com.INVALID mailto:prapul_cool2...@yahoo.com.INVALID wrote: Hi ,From Xenserver6.2 onwards, you need to enabled native HA at pool level for cloudstack HA to work.please refer the link for more details. http://www.shapeblue.com/xenserver-native-ha-with-cloudstack/ Regards, Prapul sriram, Cloud Architect, PSI Cloud Solutions, Psiclouds.comhttp://Psiclouds.com. On Friday, 14 August 2015 3:31 AM, Fedi Ben Ali ben.ali.fe...@gmail.commailto:ben.ali.fe...@gmail.com wrote: hello , I'm working on cloudstack 4.4.2 and xenserver 6.2 cluster with 5 hosts, i deployed many virtual machines with compute offerings having ha feature enabled. But when i stops one of the pools hosts , happens all VM steel set to running on cloudstack and no ha is done. Even when i stop a vm from the inside (setting command init 0 ) ,cloudstack suppose to restart the VM ,but he is not doing so. This is very weird but can any one help ,this sooo urgent . Thx
Re: Urgent Issue : Cloudstack VM HA not working
Hi, Next to CloudStack reporting Up, also double check the management log en verify it can actually connect to the hypervisor and talk to it. 4.4.2 had issues with these kind of things. What patches have you installed on xenserver? Can you put the mgt logs somewhere so we can take a look? Please consider upgrading as unfortunately 4.4.2 wasn't our best release. The latest 4.4.4 is great and has 100+ bugs fixed. I am not saying it will fix this, but it would be better anyway. Regards, Remi On 14 Aug 2015, at 23:36, Somesh Naidu somesh.na...@citrix.com wrote: in the cloudDB its running That would explain why HA did not trigger on the VM. Is the host marked as Up, as in connected to CS? Regards, Somesh -Original Message- From: Fedi Ben Ali [mailto:ben.ali.fe...@gmail.com] Sent: Friday, August 14, 2015 5:23 PM To: users@cloudstack.apache.org Cc: prapul cool Subject: Re: Urgent Issue : Cloudstack VM HA not working hi, On the xencenter its halted and in the cloudDB its running. The vmsync process is not getting any information about the VM. 2015-08-14 18:46 GMT+03:00 Somesh Naidu somesh.na...@citrix.com: Fedi, I am assuming VM has HA flag set in its Compute Offering. When you stop the VM from XenCenter, what state is it in XenSever (halted or does it disappear) and in Cloud DB (Running or Stopped)? What does vmsync for that host do with this VM (look at management server logs for this)? Regards, Somesh -Original Message- From: Fedi Ben Ali [mailto:ben.ali.fe...@gmail.com] Sent: Friday, August 14, 2015 7:59 AM To: users@cloudstack.apache.org Cc: prapul cool Subject: Re: Urgent Issue : Cloudstack VM HA not working Hi, thanks for your replys, but all those links are basically discribing how to do the pool-master failover (i already made those the configuration of xenserver ha ). but in my case i'm not recovering from a slave faillure ,even when i stop a vm from xencenter ,cloudstack do not start it as described in the documentation. Best regards. 2015-08-14 12:26 GMT+03:00 Remi Bergsma rberg...@schubergphilis.com: Hi, This may also be of help: http://blog.remibergsma.com/2015/05/23/making-xenserver-and-cloudstack-sing-and-dance-together-again/ I’d also consider upgrading to 4.4.4. I just did a 4.4.2 - 4.4.4 upgrade and it went without issues. It has many fixes over 4.4.2. Regards, Remi On 14 Aug 2015, at 08:05, prapul cool prapul_cool2...@yahoo.com.INVALID mailto:prapul_cool2...@yahoo.com.INVALID wrote: Hi ,From Xenserver6.2 onwards, you need to enabled native HA at pool level for cloudstack HA to work.please refer the link for more details. http://www.shapeblue.com/xenserver-native-ha-with-cloudstack/ Regards, Prapul sriram, Cloud Architect, PSI Cloud Solutions, Psiclouds.comhttp://Psiclouds.com. On Friday, 14 August 2015 3:31 AM, Fedi Ben Ali ben.ali.fe...@gmail.commailto:ben.ali.fe...@gmail.com wrote: hello , I'm working on cloudstack 4.4.2 and xenserver 6.2 cluster with 5 hosts, i deployed many virtual machines with compute offerings having ha feature enabled. But when i stops one of the pools hosts , happens all VM steel set to running on cloudstack and no ha is done. Even when i stop a vm from the inside (setting command init 0 ) ,cloudstack suppose to restart the VM ,but he is not doing so. This is very weird but can any one help ,this sooo urgent . Thx
Re: Urgent Issue : Cloudstack VM HA not working
Hi ,From Xenserver6.2 onwards, you need to enabled native HA at pool level for cloudstack HA to work.please refer the link for more details.http://www.shapeblue.com/xenserver-native-ha-with-cloudstack/ Regards, Prapul sriram, Cloud Architect, PSI Cloud Solutions, Psiclouds.com. On Friday, 14 August 2015 3:31 AM, Fedi Ben Ali ben.ali.fe...@gmail.com wrote: hello , I'm working on cloudstack 4.4.2 and xenserver 6.2 cluster with 5 hosts, i deployed many virtual machines with compute offerings having ha feature enabled. But when i stops one of the pools hosts , happens all VM steel set to running on cloudstack and no ha is done. Even when i stop a vm from the inside (setting command init 0 ) ,cloudstack suppose to restart the VM ,but he is not doing so. This is very weird but can any one help ,this sooo urgent . Thx
Re: Urgent Issue : Cloudstack VM HA not working
Hi, This may also be of help: http://blog.remibergsma.com/2015/05/23/making-xenserver-and-cloudstack-sing-and-dance-together-again/ I’d also consider upgrading to 4.4.4. I just did a 4.4.2 - 4.4.4 upgrade and it went without issues. It has many fixes over 4.4.2. Regards, Remi On 14 Aug 2015, at 08:05, prapul cool prapul_cool2...@yahoo.com.INVALIDmailto:prapul_cool2...@yahoo.com.INVALID wrote: Hi ,From Xenserver6.2 onwards, you need to enabled native HA at pool level for cloudstack HA to work.please refer the link for more details.http://www.shapeblue.com/xenserver-native-ha-with-cloudstack/ Regards, Prapul sriram, Cloud Architect, PSI Cloud Solutions, Psiclouds.comhttp://Psiclouds.com. On Friday, 14 August 2015 3:31 AM, Fedi Ben Ali ben.ali.fe...@gmail.commailto:ben.ali.fe...@gmail.com wrote: hello , I'm working on cloudstack 4.4.2 and xenserver 6.2 cluster with 5 hosts, i deployed many virtual machines with compute offerings having ha feature enabled. But when i stops one of the pools hosts , happens all VM steel set to running on cloudstack and no ha is done. Even when i stop a vm from the inside (setting command init 0 ) ,cloudstack suppose to restart the VM ,but he is not doing so. This is very weird but can any one help ,this sooo urgent . Thx
RE: Urgent Issue : Cloudstack VM HA not working
Fedi, I am assuming VM has HA flag set in its Compute Offering. When you stop the VM from XenCenter, what state is it in XenSever (halted or does it disappear) and in Cloud DB (Running or Stopped)? What does vmsync for that host do with this VM (look at management server logs for this)? Regards, Somesh -Original Message- From: Fedi Ben Ali [mailto:ben.ali.fe...@gmail.com] Sent: Friday, August 14, 2015 7:59 AM To: users@cloudstack.apache.org Cc: prapul cool Subject: Re: Urgent Issue : Cloudstack VM HA not working Hi, thanks for your replys, but all those links are basically discribing how to do the pool-master failover (i already made those the configuration of xenserver ha ). but in my case i'm not recovering from a slave faillure ,even when i stop a vm from xencenter ,cloudstack do not start it as described in the documentation. Best regards. 2015-08-14 12:26 GMT+03:00 Remi Bergsma rberg...@schubergphilis.com: Hi, This may also be of help: http://blog.remibergsma.com/2015/05/23/making-xenserver-and-cloudstack-sing-and-dance-together-again/ I’d also consider upgrading to 4.4.4. I just did a 4.4.2 - 4.4.4 upgrade and it went without issues. It has many fixes over 4.4.2. Regards, Remi On 14 Aug 2015, at 08:05, prapul cool prapul_cool2...@yahoo.com.INVALID mailto:prapul_cool2...@yahoo.com.INVALID wrote: Hi ,From Xenserver6.2 onwards, you need to enabled native HA at pool level for cloudstack HA to work.please refer the link for more details. http://www.shapeblue.com/xenserver-native-ha-with-cloudstack/ Regards, Prapul sriram, Cloud Architect, PSI Cloud Solutions, Psiclouds.comhttp://Psiclouds.com. On Friday, 14 August 2015 3:31 AM, Fedi Ben Ali ben.ali.fe...@gmail.commailto:ben.ali.fe...@gmail.com wrote: hello , I'm working on cloudstack 4.4.2 and xenserver 6.2 cluster with 5 hosts, i deployed many virtual machines with compute offerings having ha feature enabled. But when i stops one of the pools hosts , happens all VM steel set to running on cloudstack and no ha is done. Even when i stop a vm from the inside (setting command init 0 ) ,cloudstack suppose to restart the VM ,but he is not doing so. This is very weird but can any one help ,this sooo urgent . Thx
Re: Urgent Issue : Cloudstack VM HA not working
hi, On the xencenter its halted and in the cloudDB its running. The vmsync process is not getting any information about the VM. 2015-08-14 18:46 GMT+03:00 Somesh Naidu somesh.na...@citrix.com: Fedi, I am assuming VM has HA flag set in its Compute Offering. When you stop the VM from XenCenter, what state is it in XenSever (halted or does it disappear) and in Cloud DB (Running or Stopped)? What does vmsync for that host do with this VM (look at management server logs for this)? Regards, Somesh -Original Message- From: Fedi Ben Ali [mailto:ben.ali.fe...@gmail.com] Sent: Friday, August 14, 2015 7:59 AM To: users@cloudstack.apache.org Cc: prapul cool Subject: Re: Urgent Issue : Cloudstack VM HA not working Hi, thanks for your replys, but all those links are basically discribing how to do the pool-master failover (i already made those the configuration of xenserver ha ). but in my case i'm not recovering from a slave faillure ,even when i stop a vm from xencenter ,cloudstack do not start it as described in the documentation. Best regards. 2015-08-14 12:26 GMT+03:00 Remi Bergsma rberg...@schubergphilis.com: Hi, This may also be of help: http://blog.remibergsma.com/2015/05/23/making-xenserver-and-cloudstack-sing-and-dance-together-again/ I’d also consider upgrading to 4.4.4. I just did a 4.4.2 - 4.4.4 upgrade and it went without issues. It has many fixes over 4.4.2. Regards, Remi On 14 Aug 2015, at 08:05, prapul cool prapul_cool2...@yahoo.com.INVALID mailto:prapul_cool2...@yahoo.com.INVALID wrote: Hi ,From Xenserver6.2 onwards, you need to enabled native HA at pool level for cloudstack HA to work.please refer the link for more details. http://www.shapeblue.com/xenserver-native-ha-with-cloudstack/ Regards, Prapul sriram, Cloud Architect, PSI Cloud Solutions, Psiclouds.comhttp://Psiclouds.com. On Friday, 14 August 2015 3:31 AM, Fedi Ben Ali ben.ali.fe...@gmail.commailto:ben.ali.fe...@gmail.com wrote: hello , I'm working on cloudstack 4.4.2 and xenserver 6.2 cluster with 5 hosts, i deployed many virtual machines with compute offerings having ha feature enabled. But when i stops one of the pools hosts , happens all VM steel set to running on cloudstack and no ha is done. Even when i stop a vm from the inside (setting command init 0 ) ,cloudstack suppose to restart the VM ,but he is not doing so. This is very weird but can any one help ,this sooo urgent . Thx
RE: Urgent Issue : Cloudstack VM HA not working
in the cloudDB its running That would explain why HA did not trigger on the VM. Is the host marked as Up, as in connected to CS? Regards, Somesh -Original Message- From: Fedi Ben Ali [mailto:ben.ali.fe...@gmail.com] Sent: Friday, August 14, 2015 5:23 PM To: users@cloudstack.apache.org Cc: prapul cool Subject: Re: Urgent Issue : Cloudstack VM HA not working hi, On the xencenter its halted and in the cloudDB its running. The vmsync process is not getting any information about the VM. 2015-08-14 18:46 GMT+03:00 Somesh Naidu somesh.na...@citrix.com: Fedi, I am assuming VM has HA flag set in its Compute Offering. When you stop the VM from XenCenter, what state is it in XenSever (halted or does it disappear) and in Cloud DB (Running or Stopped)? What does vmsync for that host do with this VM (look at management server logs for this)? Regards, Somesh -Original Message- From: Fedi Ben Ali [mailto:ben.ali.fe...@gmail.com] Sent: Friday, August 14, 2015 7:59 AM To: users@cloudstack.apache.org Cc: prapul cool Subject: Re: Urgent Issue : Cloudstack VM HA not working Hi, thanks for your replys, but all those links are basically discribing how to do the pool-master failover (i already made those the configuration of xenserver ha ). but in my case i'm not recovering from a slave faillure ,even when i stop a vm from xencenter ,cloudstack do not start it as described in the documentation. Best regards. 2015-08-14 12:26 GMT+03:00 Remi Bergsma rberg...@schubergphilis.com: Hi, This may also be of help: http://blog.remibergsma.com/2015/05/23/making-xenserver-and-cloudstack-sing-and-dance-together-again/ I’d also consider upgrading to 4.4.4. I just did a 4.4.2 - 4.4.4 upgrade and it went without issues. It has many fixes over 4.4.2. Regards, Remi On 14 Aug 2015, at 08:05, prapul cool prapul_cool2...@yahoo.com.INVALID mailto:prapul_cool2...@yahoo.com.INVALID wrote: Hi ,From Xenserver6.2 onwards, you need to enabled native HA at pool level for cloudstack HA to work.please refer the link for more details. http://www.shapeblue.com/xenserver-native-ha-with-cloudstack/ Regards, Prapul sriram, Cloud Architect, PSI Cloud Solutions, Psiclouds.comhttp://Psiclouds.com. On Friday, 14 August 2015 3:31 AM, Fedi Ben Ali ben.ali.fe...@gmail.commailto:ben.ali.fe...@gmail.com wrote: hello , I'm working on cloudstack 4.4.2 and xenserver 6.2 cluster with 5 hosts, i deployed many virtual machines with compute offerings having ha feature enabled. But when i stops one of the pools hosts , happens all VM steel set to running on cloudstack and no ha is done. Even when i stop a vm from the inside (setting command init 0 ) ,cloudstack suppose to restart the VM ,but he is not doing so. This is very weird but can any one help ,this sooo urgent . Thx
Re: Urgent Issue : Cloudstack VM HA not working
Hi, thanks for your replys, but all those links are basically discribing how to do the pool-master failover (i already made those the configuration of xenserver ha ). but in my case i'm not recovering from a slave faillure ,even when i stop a vm from xencenter ,cloudstack do not start it as described in the documentation. Best regards. 2015-08-14 12:26 GMT+03:00 Remi Bergsma rberg...@schubergphilis.com: Hi, This may also be of help: http://blog.remibergsma.com/2015/05/23/making-xenserver-and-cloudstack-sing-and-dance-together-again/ I’d also consider upgrading to 4.4.4. I just did a 4.4.2 - 4.4.4 upgrade and it went without issues. It has many fixes over 4.4.2. Regards, Remi On 14 Aug 2015, at 08:05, prapul cool prapul_cool2...@yahoo.com.INVALID mailto:prapul_cool2...@yahoo.com.INVALID wrote: Hi ,From Xenserver6.2 onwards, you need to enabled native HA at pool level for cloudstack HA to work.please refer the link for more details. http://www.shapeblue.com/xenserver-native-ha-with-cloudstack/ Regards, Prapul sriram, Cloud Architect, PSI Cloud Solutions, Psiclouds.comhttp://Psiclouds.com. On Friday, 14 August 2015 3:31 AM, Fedi Ben Ali ben.ali.fe...@gmail.commailto:ben.ali.fe...@gmail.com wrote: hello , I'm working on cloudstack 4.4.2 and xenserver 6.2 cluster with 5 hosts, i deployed many virtual machines with compute offerings having ha feature enabled. But when i stops one of the pools hosts , happens all VM steel set to running on cloudstack and no ha is done. Even when i stop a vm from the inside (setting command init 0 ) ,cloudstack suppose to restart the VM ,but he is not doing so. This is very weird but can any one help ,this sooo urgent . Thx
Urgent Issue : Cloudstack VM HA not working
hello , I'm working on cloudstack 4.4.2 and xenserver 6.2 cluster with 5 hosts, i deployed many virtual machines with compute offerings having ha feature enabled. But when i stops one of the pools hosts , happens all VM steel set to running on cloudstack and no ha is done. Even when i stop a vm from the inside (setting command init 0 ) ,cloudstack suppose to restart the VM ,but he is not doing so. This is very weird but can any one help ,this sooo urgent . Thx
