Re: 4.11 without Host-HA framework
Thanks Jon. I've been following that thread too, have now much clear understanding of how VM HA behaves in cloudstack. Thanks for finding a bug and saving time of others :) Regards, Parth Patel On Mon 11 Jun, 2018, 16:30 Jon Marshall, wrote: > Hi Parth > > > Just in case you have not seen my other thread, it turns out that all this > time it has been a bug. > > > Using multiple NICs with basic networking and using zone wide NFS VM HA > just does not work. If you change to cluster wide NFS then it works fine > (and quite quickly as well :)) > > > I am now going to setup Host HA and see make sure that all works as well > using cluster NFS. > > > Got there in the end :) > > > Jon > > > > > > > From: Parth Patel > Sent: 24 May 2018 06:52 > To: users@cloudstack.apache.org > Subject: Re: 4.11 without Host-HA framework > > Hi Jon and Angus, > > I did not shutdown the VMs as Yiping Zhang said, but I have confirmed this > and discussed earlier in the users list that my HA-enabled VMs got started > on another suitable available host in the cluster even when I didn't have > IPMI-enabled hardware and did no configuration for OOBM and Host-HA. I > simply pulled the ethernet cable connecting the host to entire network (I > did use just one NIC) and according to the value set in ping timeout event, > the HA-enabled VMs were restarted on another available host. I tested the > scenario using both the scenarios: the echo command as well as good old > plugging out the NIC from the host. My VMs were successfully started on > another available host after CS manager confirmed they were not reachable. > > I too want to understand how the failover mechanism in CloudStack actually > works. I used ACS 4.11 packages available here: > http://cloudstack.apt-get.eu/centos/7/4.11/ > > Regards, > Parth Patel > > > On Thu, 24 May 2018 at 10:53 Paul Angus wrote: > > > I'm afraid that is not a host crash. When shutting down the guest OS, > the > > CloudStack agent on the host is still able to report to the management > > server that the VM has stopped. > > > > This is my point. VM-HA relies on the management sever communication with > > the host agent. > > > > Kind regards, > > > > Paul Angus > > > > paul.an...@shapeblue.com > > www.shapeblue.com<http://www.shapeblue.com> > > 53 Chandos Place, Covent Garden, London WC2N 4HSUK > > @shapeblue > > > > > > > > > > -Original Message- > > From: Yiping Zhang > > Sent: 24 May 2018 00:44 > > To: users@cloudstack.apache.org > > Subject: Re: 4.11 without Host-HA framework > > > > I can say for fact that VM's using a HA enabled service offering will be > > restarted by CS on another host, assuming there are enough > > capacity/resources in the cluster, when their original host crashes, > > regardless that host comes back or not. > > > > The simplest way to test VM HA feature with a VM instance using HA > enabled > > service offering is to issue shutdown command in guest OS, and watching > it > > gets restarted by CS manager. > > > > On 5/23/18, 1:23 PM, "Paul Angus" wrote: > > > > Hi Jon, > > > > Don't worry, TBH I'm dubious about those claiming to have VM-HA > > working when a host crashes (but doesn't restart). > > I'll check in with the guys that set values for host-ha when testing, > > to see which ones they change and what they set them to. > > > > paul.an...@shapeblue.com > > www.shapeblue.com<http://www.shapeblue.com> > > 53 Chandos Place, Covent Garden, London WC2N 4HSUK > > @shapeblue > > > > > > > > > > -Original Message- > > From: Jon Marshall > > Sent: 23 May 2018 21:10 > > To: users@cloudstack.apache.org > > Subject: Re: 4.11 without Host-HA framework > > > > Rohit / Paul > > > > > > Thanks again for answering. > > > > > > I am a Cisco guy with an ex Unix background but no virtualisation > > experience and I can honestly say I have never felt this stupid before > > > > > > I have Cloudstack working but failover is killing me. > > > > > > When you say VM HA relies on the host telling CS the VM is down how > > does that work because if you crash the host how does it tell CS > anything ? > > And when you say tell CS do you mean the CS manager ? > > > > > > I guess I am just not understanding all the moving parts
Re: 4.11 without Host-HA framework
Hi Parth Just in case you have not seen my other thread, it turns out that all this time it has been a bug. Using multiple NICs with basic networking and using zone wide NFS VM HA just does not work. If you change to cluster wide NFS then it works fine (and quite quickly as well :)) I am now going to setup Host HA and see make sure that all works as well using cluster NFS. Got there in the end :) Jon From: Parth Patel Sent: 24 May 2018 06:52 To: users@cloudstack.apache.org Subject: Re: 4.11 without Host-HA framework Hi Jon and Angus, I did not shutdown the VMs as Yiping Zhang said, but I have confirmed this and discussed earlier in the users list that my HA-enabled VMs got started on another suitable available host in the cluster even when I didn't have IPMI-enabled hardware and did no configuration for OOBM and Host-HA. I simply pulled the ethernet cable connecting the host to entire network (I did use just one NIC) and according to the value set in ping timeout event, the HA-enabled VMs were restarted on another available host. I tested the scenario using both the scenarios: the echo command as well as good old plugging out the NIC from the host. My VMs were successfully started on another available host after CS manager confirmed they were not reachable. I too want to understand how the failover mechanism in CloudStack actually works. I used ACS 4.11 packages available here: http://cloudstack.apt-get.eu/centos/7/4.11/ Regards, Parth Patel On Thu, 24 May 2018 at 10:53 Paul Angus wrote: > I'm afraid that is not a host crash. When shutting down the guest OS, the > CloudStack agent on the host is still able to report to the management > server that the VM has stopped. > > This is my point. VM-HA relies on the management sever communication with > the host agent. > > Kind regards, > > Paul Angus > > paul.an...@shapeblue.com > www.shapeblue.com<http://www.shapeblue.com> > 53 Chandos Place, Covent Garden, London WC2N 4HSUK > @shapeblue > > > > > -Original Message- > From: Yiping Zhang > Sent: 24 May 2018 00:44 > To: users@cloudstack.apache.org > Subject: Re: 4.11 without Host-HA framework > > I can say for fact that VM's using a HA enabled service offering will be > restarted by CS on another host, assuming there are enough > capacity/resources in the cluster, when their original host crashes, > regardless that host comes back or not. > > The simplest way to test VM HA feature with a VM instance using HA enabled > service offering is to issue shutdown command in guest OS, and watching it > gets restarted by CS manager. > > On 5/23/18, 1:23 PM, "Paul Angus" wrote: > > Hi Jon, > > Don't worry, TBH I'm dubious about those claiming to have VM-HA > working when a host crashes (but doesn't restart). > I'll check in with the guys that set values for host-ha when testing, > to see which ones they change and what they set them to. > > paul.an...@shapeblue.com > www.shapeblue.com<http://www.shapeblue.com> > 53 Chandos Place, Covent Garden, London WC2N 4HSUK > @shapeblue > > > > > -Original Message- > From: Jon Marshall > Sent: 23 May 2018 21:10 > To: users@cloudstack.apache.org > Subject: Re: 4.11 without Host-HA framework > > Rohit / Paul > > > Thanks again for answering. > > > I am a Cisco guy with an ex Unix background but no virtualisation > experience and I can honestly say I have never felt this stupid before > > > I have Cloudstack working but failover is killing me. > > > When you say VM HA relies on the host telling CS the VM is down how > does that work because if you crash the host how does it tell CS anything ? > And when you say tell CS do you mean the CS manager ? > > > I guess I am just not understanding all the moving parts. I have had > HOST HA working (to an extent) although it takes a long time to failover > even after tweaking the timers but the fact that I keep finding references > to people saying even without HOST HA it should failover (and mine doesn't) > makes me think I have configured it incorrectly somewhere along the line. > > > I have configured a compute offering with HA and I am crashing the > host with the echo command as suggested but still nothing. > > > I understand what you are saying Paul about it not being a good idea > to rely on VM HA so I will go back to Host HA and try to speed up failover > times. > > > Can I ask, from your experiences, what is a realistic fail over time > for CS ie. if a host fails for example ? > > > Jon > > > > > > Fro
Re: 4.11 without Host-HA framework
As mentioned if I use just the one NIC for all traffic then VM HA works. I have been using this document to understand CS network concepts - https://www.shapeblue.com/understanding-cloudstacks-physical-networking-architecture/ I have been assuming that the manager node only needs an interface in the management network and it is the compute nodes that I have split the traffic across 3 NICs as per the above doc. Does the manager need NICs in the other networks as well ? Jon From: Paul Angus Sent: 25 May 2018 07:37 To: users@cloudstack.apache.org Subject: RE: 4.11 without Host-HA framework I'm on leave next week, but I'll pick this up again when I'm back ... paul.an...@shapeblue.com www.shapeblue.com<http://www.shapeblue.com> 53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue -Original Message- From: Jon Marshall Sent: 24 May 2018 11:20 To: users@cloudstack.apache.org Subject: Re: 4.11 without Host-HA framework Hi Parth I remember you saying this worked for you in a previous thread. I am beginning to wonder if it is the fact I have used 3 separate NICs, one for management, one for the VM traffic and the third for storage that I am not seeing the behaviour you saw. That is why, I too would like to understand exactly what is talking to what and doing checks for both non Host-HA and Host-HA. I did get failover working in some scenarios with Host-HA and OOBM using IPMI but it was slow even after tweaking the timers eg. for a crashed host the best time i got was around 8 minutes which seems a long time but perhaps that is an acceptable time for CS, I just don't know. Not expecting it to be instantaneous as it needs to do checks etc. Jon From: Parth Patel Sent: 24 May 2018 06:52 To: users@cloudstack.apache.org Subject: Re: 4.11 without Host-HA framework Hi Jon and Angus, I did not shutdown the VMs as Yiping Zhang said, but I have confirmed this and discussed earlier in the users list that my HA-enabled VMs got started on another suitable available host in the cluster even when I didn't have IPMI-enabled hardware and did no configuration for OOBM and Host-HA. I simply pulled the ethernet cable connecting the host to entire network (I did use just one NIC) and according to the value set in ping timeout event, the HA-enabled VMs were restarted on another available host. I tested the scenario using both the scenarios: the echo command as well as good old plugging out the NIC from the host. My VMs were successfully started on another available host after CS manager confirmed they were not reachable. I too want to understand how the failover mechanism in CloudStack actually works. I used ACS 4.11 packages available here: http://cloudstack.apt-get.eu/centos/7/4.11/ Regards, Parth Patel On Thu, 24 May 2018 at 10:53 Paul Angus wrote: > I'm afraid that is not a host crash. When shutting down the guest OS, > the CloudStack agent on the host is still able to report to the > management server that the VM has stopped. > > This is my point. VM-HA relies on the management sever communication > with the host agent. > > Kind regards, > > Paul Angus > > paul.an...@shapeblue.com > www.shapeblue.com<http://www.shapeblue.com> > 53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue > > > > > -Original Message- > From: Yiping Zhang > Sent: 24 May 2018 00:44 > To: users@cloudstack.apache.org > Subject: Re: 4.11 without Host-HA framework > > I can say for fact that VM's using a HA enabled service offering will be > restarted by CS on another host, assuming there are enough > capacity/resources in the cluster, when their original host crashes, > regardless that host comes back or not. > > The simplest way to test VM HA feature with a VM instance using HA enabled > service offering is to issue shutdown command in guest OS, and watching it > gets restarted by CS manager. > > On 5/23/18, 1:23 PM, "Paul Angus" wrote: > > Hi Jon, > > Don't worry, TBH I'm dubious about those claiming to have VM-HA > working when a host crashes (but doesn't restart). > I'll check in with the guys that set values for host-ha when testing, > to see which ones they change and what they set them to. > > paul.an...@shapeblue.com > www.shapeblue.com<http://www.shapeblue.com> > 53 Chandos Place, Covent Garden, London WC2N 4HSUK > @shapeblue > > > > > -Original Message- > From: Jon Marshall > Sent: 23 May 2018 21:10 > To: users@cloudstack.apache.org > Subject: Re: 4.11 without Host-HA framework > > Rohit / Paul > > > Thanks again for answering. > > > I am a Cisco guy with an ex Unix background but no virtualisation > e
Re: 4.11 without Host-HA framework
Update on this. I put everything (management, storage and guest VMs) on single NIC so all in same subnet and VM HA failover worked. It took about 6 1/2 minutes with default timers before the VM was responding to a ping after being migrated. So it looks like it is something with the network setup I am doing. The manager node hast just a single NIC in the management subnet - 172.30.3.0/27 and the IP is assigned directly to the NIC. Each compute node has - 1) a NIC in the management subnet - 172.30.3.0/27 2) a NIC in the guest VM subnet - 172.30.4.0/25 3) a NIC in the storage subnet - 172.30.5.0/28 (NFS server is also in this subnet) None of the NICs are vlan aware but the ports they connect to on the switch are in different vlans. 3 bridges are used on each node - cloudbr0 for management, cloudbr1 for guest VMs and cloudbr2 for storage. Only the ifcfg-cloudbr1 configuration references a default gateway because I read somewhere that is what should be used and I seemed to remember I had trouble with SSVM until I did this. When setting up the cloud I exclude the already used management IPs on the nodes from the range you enter as I had issues with the system VMs picking up IPs already in use. Same reasoning behind storage ie. I exclude IPs already used for NFS server and compute nodes. Can anyone see where any of the above could be causing an issue ? Many thanks for any help given From: Rohit Yadav Sent: 23 May 2018 10:45 To: users@cloudstack.apache.org Subject: Re: 4.11 without Host-HA framework Jon, In the VM's compute offering, make sure that HA is ticked/enabled. Then use that HA-enabled VM offering while deploying a VM. Around testing - it depends how you're crashing. In case of KVM, you can try to cause host crash (example: echo c > /proc/sysrq-trigger) and see if HA-enabled VMs gets started on a different host. - Rohit <https://cloudstack.apache.org> [https://cloudstack.apache.org/images/monkey-144.png]<https://cloudstack.apache.org/> Apache CloudStack: Open Source Cloud Computing<https://cloudstack.apache.org/> cloudstack.apache.org CloudStack is open source cloud computing software for creating, managing, and deploying infrastructure cloud services From: Jon Marshall Sent: Tuesday, May 22, 2018 8:28:06 PM To: users@cloudstack.apache.org Subject: Re: 4.11 without Host-HA framework Hi Rohit Thanks for responding. I have not had much luck with HA at all. I crash a server and nothing happens in terms of VMs migrating to another host. Monitoring the management log file it seems the management server recognises the host has stopped responding to pings but doesn't think it has to do anything. I am currently running v4.11 with basic network but 3 separate NICs, one for management, one for storage and one for VMs themselves. Should it make it any difference ie. would it be worth trying to run management and storage over the same NIC ? I am just lost as to why I see no failover at all whereas others are reporting it works fine. Jon From: Rohit Yadav Sent: 22 May 2018 12:12 To: users@cloudstack.apache.org Subject: Re: 4.11 without Host-HA framework Hi Jon, Yes, Host-HA is different from VM-HA and without Host HA enabled a HA enabled VM should be recovered/run on a different host when it crashes. Historically the term 'HA' in CloudStack is used around high availability of a VM. Host HA as the name tries to imply is around HA of a physical hypervisor host by means of out-of-band management technologies such as ipmi and currently supporting ipmi as OOBM and KVM hosts with NFS storage. - Rohit <https://cloudstack.apache.org> [https://cloudstack.apache.org/images/monkey-144.png]<https://cloudstack.apache.org/> Apache CloudStack: Open Source Cloud Computing<https://cloudstack.apache.org/> cloudstack.apache.org CloudStack is open source cloud computing software for creating, managing, and deploying infrastructure cloud services From: Jon Marshall Sent: Monday, May 21, 2018 8:36:04 PM To: users@cloudstack.apache.org Subject: 4.11 without Host-HA framework I keep seeing conflicting information about this in the mailing lists and in blogs etc. If I run 4.11 without enabling Host HA framework should HA still work if I crash a compute node because my understanding was the new framework was added for certain cases only. It doesn't work for me but I can find a number of people saying you don't need to enable the new framework for it to work. Thanks Jon rohit.ya...@shapeblue.com www.shapeblue.com<http://www.shapeblue.com> 53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue rohit.ya...@shapeblue.com www.shapeblue.com<http://www.shapeblue.com> 53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue
Re: 4.11 without Host-HA framework
Hi Rohit In an attempt to make things simpler I am now running the management and storage (NFS) across the same NIC with a separate NIC for the guest VMs. So basic networking, one subnet for management/storage and a different one for guest VMs which means two bridges. I am also just testing VM HA (not Host HA at present) 1 manager and 3 compute nodes. I crash a compute node or pull the power on the node and monitor the management server log. It reports the ping timeouts and once then after the ping interval * ping timoeut time it marks the host as state Alert in the UI. So far so good. But it never tries to migrate the VM running on the crashed node. Not a single message about attempting to restart, nothing. The VM has been setup with a compute offering with HA enabled. Any thoughts as to why it is not trying to restart the VM on another of the nodes (there is capacity as one of the nodes has no VMs on it) . The only other thing I can try is to use just one NIC for everything and see if I get anywhere with that. Jon From: Rohit Yadav Sent: 23 May 2018 10:45 To: users@cloudstack.apache.org Subject: Re: 4.11 without Host-HA framework Jon, In the VM's compute offering, make sure that HA is ticked/enabled. Then use that HA-enabled VM offering while deploying a VM. Around testing - it depends how you're crashing. In case of KVM, you can try to cause host crash (example: echo c > /proc/sysrq-trigger) and see if HA-enabled VMs gets started on a different host. - Rohit <https://cloudstack.apache.org> [https://cloudstack.apache.org/images/monkey-144.png]<https://cloudstack.apache.org/> Apache CloudStack: Open Source Cloud Computing<https://cloudstack.apache.org/> cloudstack.apache.org CloudStack is open source cloud computing software for creating, managing, and deploying infrastructure cloud services From: Jon Marshall Sent: Tuesday, May 22, 2018 8:28:06 PM To: users@cloudstack.apache.org Subject: Re: 4.11 without Host-HA framework Hi Rohit Thanks for responding. I have not had much luck with HA at all. I crash a server and nothing happens in terms of VMs migrating to another host. Monitoring the management log file it seems the management server recognises the host has stopped responding to pings but doesn't think it has to do anything. I am currently running v4.11 with basic network but 3 separate NICs, one for management, one for storage and one for VMs themselves. Should it make it any difference ie. would it be worth trying to run management and storage over the same NIC ? I am just lost as to why I see no failover at all whereas others are reporting it works fine. Jon From: Rohit Yadav Sent: 22 May 2018 12:12 To: users@cloudstack.apache.org Subject: Re: 4.11 without Host-HA framework Hi Jon, Yes, Host-HA is different from VM-HA and without Host HA enabled a HA enabled VM should be recovered/run on a different host when it crashes. Historically the term 'HA' in CloudStack is used around high availability of a VM. Host HA as the name tries to imply is around HA of a physical hypervisor host by means of out-of-band management technologies such as ipmi and currently supporting ipmi as OOBM and KVM hosts with NFS storage. - Rohit <https://cloudstack.apache.org> [https://cloudstack.apache.org/images/monkey-144.png]<https://cloudstack.apache.org/> Apache CloudStack: Open Source Cloud Computing<https://cloudstack.apache.org/> cloudstack.apache.org CloudStack is open source cloud computing software for creating, managing, and deploying infrastructure cloud services From: Jon Marshall Sent: Monday, May 21, 2018 8:36:04 PM To: users@cloudstack.apache.org Subject: 4.11 without Host-HA framework I keep seeing conflicting information about this in the mailing lists and in blogs etc. If I run 4.11 without enabling Host HA framework should HA still work if I crash a compute node because my understanding was the new framework was added for certain cases only. It doesn't work for me but I can find a number of people saying you don't need to enable the new framework for it to work. Thanks Jon rohit.ya...@shapeblue.com www.shapeblue.com<http://www.shapeblue.com> 53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue rohit.ya...@shapeblue.com www.shapeblue.com<http://www.shapeblue.com> 53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue
RE: 4.11 without Host-HA framework
I'm on leave next week, but I'll pick this up again when I'm back ... paul.an...@shapeblue.com www.shapeblue.com 53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue -Original Message- From: Jon Marshall <jms@hotmail.co.uk> Sent: 24 May 2018 11:20 To: users@cloudstack.apache.org Subject: Re: 4.11 without Host-HA framework Hi Parth I remember you saying this worked for you in a previous thread. I am beginning to wonder if it is the fact I have used 3 separate NICs, one for management, one for the VM traffic and the third for storage that I am not seeing the behaviour you saw. That is why, I too would like to understand exactly what is talking to what and doing checks for both non Host-HA and Host-HA. I did get failover working in some scenarios with Host-HA and OOBM using IPMI but it was slow even after tweaking the timers eg. for a crashed host the best time i got was around 8 minutes which seems a long time but perhaps that is an acceptable time for CS, I just don't know. Not expecting it to be instantaneous as it needs to do checks etc. Jon From: Parth Patel <parthpatel2...@gmail.com> Sent: 24 May 2018 06:52 To: users@cloudstack.apache.org Subject: Re: 4.11 without Host-HA framework Hi Jon and Angus, I did not shutdown the VMs as Yiping Zhang said, but I have confirmed this and discussed earlier in the users list that my HA-enabled VMs got started on another suitable available host in the cluster even when I didn't have IPMI-enabled hardware and did no configuration for OOBM and Host-HA. I simply pulled the ethernet cable connecting the host to entire network (I did use just one NIC) and according to the value set in ping timeout event, the HA-enabled VMs were restarted on another available host. I tested the scenario using both the scenarios: the echo command as well as good old plugging out the NIC from the host. My VMs were successfully started on another available host after CS manager confirmed they were not reachable. I too want to understand how the failover mechanism in CloudStack actually works. I used ACS 4.11 packages available here: http://cloudstack.apt-get.eu/centos/7/4.11/ Regards, Parth Patel On Thu, 24 May 2018 at 10:53 Paul Angus <paul.an...@shapeblue.com> wrote: > I'm afraid that is not a host crash. When shutting down the guest OS, > the CloudStack agent on the host is still able to report to the > management server that the VM has stopped. > > This is my point. VM-HA relies on the management sever communication > with the host agent. > > Kind regards, > > Paul Angus > > paul.an...@shapeblue.com > www.shapeblue.com<http://www.shapeblue.com> > 53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue > > > > > -Original Message- > From: Yiping Zhang <yzh...@marketo.com> > Sent: 24 May 2018 00:44 > To: users@cloudstack.apache.org > Subject: Re: 4.11 without Host-HA framework > > I can say for fact that VM's using a HA enabled service offering will be > restarted by CS on another host, assuming there are enough > capacity/resources in the cluster, when their original host crashes, > regardless that host comes back or not. > > The simplest way to test VM HA feature with a VM instance using HA enabled > service offering is to issue shutdown command in guest OS, and watching it > gets restarted by CS manager. > > On 5/23/18, 1:23 PM, "Paul Angus" <paul.an...@shapeblue.com> wrote: > > Hi Jon, > > Don't worry, TBH I'm dubious about those claiming to have VM-HA > working when a host crashes (but doesn't restart). > I'll check in with the guys that set values for host-ha when testing, > to see which ones they change and what they set them to. > > paul.an...@shapeblue.com > www.shapeblue.com<http://www.shapeblue.com> > 53 Chandos Place, Covent Garden, London WC2N 4HSUK > @shapeblue > > > > > -Original Message- > From: Jon Marshall <jms@hotmail.co.uk> > Sent: 23 May 2018 21:10 > To: users@cloudstack.apache.org > Subject: Re: 4.11 without Host-HA framework > > Rohit / Paul > > > Thanks again for answering. > > > I am a Cisco guy with an ex Unix background but no virtualisation > experience and I can honestly say I have never felt this stupid before > > > I have Cloudstack working but failover is killing me. > > > When you say VM HA relies on the host telling CS the VM is down how > does that work because if you crash the host how does it tell CS anything ? > And when you say tell CS do you mean the CS manager ? > > > I guess I am just not understanding all the moving parts. I have had > HOST HA working
Re: 4.11 without Host-HA framework
Hi Parth I remember you saying this worked for you in a previous thread. I am beginning to wonder if it is the fact I have used 3 separate NICs, one for management, one for the VM traffic and the third for storage that I am not seeing the behaviour you saw. That is why, I too would like to understand exactly what is talking to what and doing checks for both non Host-HA and Host-HA. I did get failover working in some scenarios with Host-HA and OOBM using IPMI but it was slow even after tweaking the timers eg. for a crashed host the best time i got was around 8 minutes which seems a long time but perhaps that is an acceptable time for CS, I just don't know. Not expecting it to be instantaneous as it needs to do checks etc. Jon From: Parth Patel <parthpatel2...@gmail.com> Sent: 24 May 2018 06:52 To: users@cloudstack.apache.org Subject: Re: 4.11 without Host-HA framework Hi Jon and Angus, I did not shutdown the VMs as Yiping Zhang said, but I have confirmed this and discussed earlier in the users list that my HA-enabled VMs got started on another suitable available host in the cluster even when I didn't have IPMI-enabled hardware and did no configuration for OOBM and Host-HA. I simply pulled the ethernet cable connecting the host to entire network (I did use just one NIC) and according to the value set in ping timeout event, the HA-enabled VMs were restarted on another available host. I tested the scenario using both the scenarios: the echo command as well as good old plugging out the NIC from the host. My VMs were successfully started on another available host after CS manager confirmed they were not reachable. I too want to understand how the failover mechanism in CloudStack actually works. I used ACS 4.11 packages available here: http://cloudstack.apt-get.eu/centos/7/4.11/ Regards, Parth Patel On Thu, 24 May 2018 at 10:53 Paul Angus <paul.an...@shapeblue.com> wrote: > I'm afraid that is not a host crash. When shutting down the guest OS, the > CloudStack agent on the host is still able to report to the management > server that the VM has stopped. > > This is my point. VM-HA relies on the management sever communication with > the host agent. > > Kind regards, > > Paul Angus > > paul.an...@shapeblue.com > www.shapeblue.com<http://www.shapeblue.com> > 53 Chandos Place, Covent Garden, London WC2N 4HSUK > @shapeblue > > > > > -Original Message- > From: Yiping Zhang <yzh...@marketo.com> > Sent: 24 May 2018 00:44 > To: users@cloudstack.apache.org > Subject: Re: 4.11 without Host-HA framework > > I can say for fact that VM's using a HA enabled service offering will be > restarted by CS on another host, assuming there are enough > capacity/resources in the cluster, when their original host crashes, > regardless that host comes back or not. > > The simplest way to test VM HA feature with a VM instance using HA enabled > service offering is to issue shutdown command in guest OS, and watching it > gets restarted by CS manager. > > On 5/23/18, 1:23 PM, "Paul Angus" <paul.an...@shapeblue.com> wrote: > > Hi Jon, > > Don't worry, TBH I'm dubious about those claiming to have VM-HA > working when a host crashes (but doesn't restart). > I'll check in with the guys that set values for host-ha when testing, > to see which ones they change and what they set them to. > > paul.an...@shapeblue.com > www.shapeblue.com<http://www.shapeblue.com> > 53 Chandos Place, Covent Garden, London WC2N 4HSUK > @shapeblue > > > > > -Original Message- > From: Jon Marshall <jms@hotmail.co.uk> > Sent: 23 May 2018 21:10 > To: users@cloudstack.apache.org > Subject: Re: 4.11 without Host-HA framework > > Rohit / Paul > > > Thanks again for answering. > > > I am a Cisco guy with an ex Unix background but no virtualisation > experience and I can honestly say I have never felt this stupid before > > > I have Cloudstack working but failover is killing me. > > > When you say VM HA relies on the host telling CS the VM is down how > does that work because if you crash the host how does it tell CS anything ? > And when you say tell CS do you mean the CS manager ? > > > I guess I am just not understanding all the moving parts. I have had > HOST HA working (to an extent) although it takes a long time to failover > even after tweaking the timers but the fact that I keep finding references > to people saying even without HOST HA it should failover (and mine doesn't) > makes me think I have configured it incorrectly somewhere along the line. > > > I have configured a compute offering with HA and I am crashi
Re: 4.11 without Host-HA framework
Hi Jon and Angus, I did not shutdown the VMs as Yiping Zhang said, but I have confirmed this and discussed earlier in the users list that my HA-enabled VMs got started on another suitable available host in the cluster even when I didn't have IPMI-enabled hardware and did no configuration for OOBM and Host-HA. I simply pulled the ethernet cable connecting the host to entire network (I did use just one NIC) and according to the value set in ping timeout event, the HA-enabled VMs were restarted on another available host. I tested the scenario using both the scenarios: the echo command as well as good old plugging out the NIC from the host. My VMs were successfully started on another available host after CS manager confirmed they were not reachable. I too want to understand how the failover mechanism in CloudStack actually works. I used ACS 4.11 packages available here: http://cloudstack.apt-get.eu/centos/7/4.11/ Regards, Parth Patel On Thu, 24 May 2018 at 10:53 Paul Angus <paul.an...@shapeblue.com> wrote: > I'm afraid that is not a host crash. When shutting down the guest OS, the > CloudStack agent on the host is still able to report to the management > server that the VM has stopped. > > This is my point. VM-HA relies on the management sever communication with > the host agent. > > Kind regards, > > Paul Angus > > paul.an...@shapeblue.com > www.shapeblue.com > 53 Chandos Place, Covent Garden, London WC2N 4HSUK > @shapeblue > > > > > -Original Message- > From: Yiping Zhang <yzh...@marketo.com> > Sent: 24 May 2018 00:44 > To: users@cloudstack.apache.org > Subject: Re: 4.11 without Host-HA framework > > I can say for fact that VM's using a HA enabled service offering will be > restarted by CS on another host, assuming there are enough > capacity/resources in the cluster, when their original host crashes, > regardless that host comes back or not. > > The simplest way to test VM HA feature with a VM instance using HA enabled > service offering is to issue shutdown command in guest OS, and watching it > gets restarted by CS manager. > > On 5/23/18, 1:23 PM, "Paul Angus" <paul.an...@shapeblue.com> wrote: > > Hi Jon, > > Don't worry, TBH I'm dubious about those claiming to have VM-HA > working when a host crashes (but doesn't restart). > I'll check in with the guys that set values for host-ha when testing, > to see which ones they change and what they set them to. > > paul.an...@shapeblue.com > www.shapeblue.com > 53 Chandos Place, Covent Garden, London WC2N 4HSUK > @shapeblue > > > > > -Original Message- > From: Jon Marshall <jms@hotmail.co.uk> > Sent: 23 May 2018 21:10 > To: users@cloudstack.apache.org > Subject: Re: 4.11 without Host-HA framework > > Rohit / Paul > > > Thanks again for answering. > > > I am a Cisco guy with an ex Unix background but no virtualisation > experience and I can honestly say I have never felt this stupid before > > > I have Cloudstack working but failover is killing me. > > > When you say VM HA relies on the host telling CS the VM is down how > does that work because if you crash the host how does it tell CS anything ? > And when you say tell CS do you mean the CS manager ? > > > I guess I am just not understanding all the moving parts. I have had > HOST HA working (to an extent) although it takes a long time to failover > even after tweaking the timers but the fact that I keep finding references > to people saying even without HOST HA it should failover (and mine doesn't) > makes me think I have configured it incorrectly somewhere along the line. > > > I have configured a compute offering with HA and I am crashing the > host with the echo command as suggested but still nothing. > > > I understand what you are saying Paul about it not being a good idea > to rely on VM HA so I will go back to Host HA and try to speed up failover > times. > > > Can I ask, from your experiences, what is a realistic fail over time > for CS ie. if a host fails for example ? > > > Jon > > > > > > From: Paul Angus <paul.an...@shapeblue.com> > Sent: 23 May 2018 19:55 > To: users@cloudstack.apache.org > Subject: RE: 4.11 without Host-HA framework > > Jon, > > As Rohit says, it is very important to understand the difference > between VM HA and host HA. > VM HA relies on the HOST telling CloudStack that the VM is down on > order for CloudStack start it again (wherever that ends up being). > Any sequence of events that ends up with VM HA re
RE: 4.11 without Host-HA framework
I'm afraid that is not a host crash. When shutting down the guest OS, the
CloudStack agent on the host is still able to report to the management server
that the VM has stopped.
This is my point. VM-HA relies on the management sever communication with the
host agent.
Kind regards,
Paul Angus
paul.an...@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London WC2N 4HSUK
@shapeblue
-Original Message-
From: Yiping Zhang <yzh...@marketo.com>
Sent: 24 May 2018 00:44
To: users@cloudstack.apache.org
Subject: Re: 4.11 without Host-HA framework
I can say for fact that VM's using a HA enabled service offering will be
restarted by CS on another host, assuming there are enough capacity/resources
in the cluster, when their original host crashes, regardless that host comes
back or not.
The simplest way to test VM HA feature with a VM instance using HA enabled
service offering is to issue shutdown command in guest OS, and watching it gets
restarted by CS manager.
On 5/23/18, 1:23 PM, "Paul Angus" <paul.an...@shapeblue.com> wrote:
Hi Jon,
Don't worry, TBH I'm dubious about those claiming to have VM-HA working
when a host crashes (but doesn't restart).
I'll check in with the guys that set values for host-ha when testing, to
see which ones they change and what they set them to.
paul.an...@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London WC2N 4HSUK
@shapeblue
-Original Message-
From: Jon Marshall <jms@hotmail.co.uk>
Sent: 23 May 2018 21:10
To: users@cloudstack.apache.org
Subject: Re: 4.11 without Host-HA framework
Rohit / Paul
Thanks again for answering.
I am a Cisco guy with an ex Unix background but no virtualisation
experience and I can honestly say I have never felt this stupid before
I have Cloudstack working but failover is killing me.
When you say VM HA relies on the host telling CS the VM is down how does
that work because if you crash the host how does it tell CS anything ? And when
you say tell CS do you mean the CS manager ?
I guess I am just not understanding all the moving parts. I have had HOST
HA working (to an extent) although it takes a long time to failover even after
tweaking the timers but the fact that I keep finding references to people
saying even without HOST HA it should failover (and mine doesn't) makes me
think I have configured it incorrectly somewhere along the line.
I have configured a compute offering with HA and I am crashing the host
with the echo command as suggested but still nothing.
I understand what you are saying Paul about it not being a good idea to
rely on VM HA so I will go back to Host HA and try to speed up failover times.
Can I ask, from your experiences, what is a realistic fail over time for CS
ie. if a host fails for example ?
Jon
From: Paul Angus <paul.an...@shapeblue.com>
Sent: 23 May 2018 19:55
To: users@cloudstack.apache.org
Subject: RE: 4.11 without Host-HA framework
Jon,
As Rohit says, it is very important to understand the difference between VM
HA and host HA.
VM HA relies on the HOST telling CloudStack that the VM is down on order
for CloudStack start it again (wherever that ends up being).
Any sequence of events that ends up with VM HA restarting the VM when
CloudStack can't contact the host is luck/fluke/unreliable/bad(tm)
The purpose of Host HA was to create a reliable mechanism to determine that
a host has 'crashed' and that the VMs within it are inoperative. Then take
appropriate action, including ultimately telling VM HA to restart the VM
elsewhere.
paul.an...@shapeblue.com
www.shapeblue.com<http://www.shapeblue.com>
Shapeblue - The CloudStack Company<http://www.shapeblue.com/>
www.shapeblue.com
ShapeBlue are the largest independent integrator of CloudStack technologies
globally and are specialists in the design and implementation of IaaS cloud
infrastructures for both private and public cloud implementations.
53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue
-Original Message-
From: Rohit Yadav <rohit.ya...@shapeblue.com>
Sent: 23 May 2018 10:45
To: users@cloudstack.apache.org
Subject: Re: 4.11 without Host-HA framework
Jon,
In the VM's compute offering, make sure that HA is ticked/enabled. Then use
that HA-enabled VM offering while deploying a VM. Around testing - it depends
how you're crashing. In case of KVM, you can try to cause host crash (example:
echo c > /proc/sysrq-trigger) and see if HA-enabled VMs gets started on a
Re: 4.11 without Host-HA framework
I can say for fact that VM's using a HA enabled service offering will be
restarted by CS on another host, assuming there are enough capacity/resources
in the cluster, when their original host crashes, regardless that host comes
back or not.
The simplest way to test VM HA feature with a VM instance using HA enabled
service offering is to issue shutdown command in guest OS, and watching it gets
restarted by CS manager.
On 5/23/18, 1:23 PM, "Paul Angus" <paul.an...@shapeblue.com> wrote:
Hi Jon,
Don't worry, TBH I'm dubious about those claiming to have VM-HA working
when a host crashes (but doesn't restart).
I'll check in with the guys that set values for host-ha when testing, to
see which ones they change and what they set them to.
paul.an...@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London WC2N 4HSUK
@shapeblue
-Original Message-
From: Jon Marshall <jms@hotmail.co.uk>
Sent: 23 May 2018 21:10
To: users@cloudstack.apache.org
Subject: Re: 4.11 without Host-HA framework
Rohit / Paul
Thanks again for answering.
I am a Cisco guy with an ex Unix background but no virtualisation
experience and I can honestly say I have never felt this stupid before
I have Cloudstack working but failover is killing me.
When you say VM HA relies on the host telling CS the VM is down how does
that work because if you crash the host how does it tell CS anything ? And when
you say tell CS do you mean the CS manager ?
I guess I am just not understanding all the moving parts. I have had HOST
HA working (to an extent) although it takes a long time to failover even after
tweaking the timers but the fact that I keep finding references to people
saying even without HOST HA it should failover (and mine doesn't) makes me
think I have configured it incorrectly somewhere along the line.
I have configured a compute offering with HA and I am crashing the host
with the echo command as suggested but still nothing.
I understand what you are saying Paul about it not being a good idea to
rely on VM HA so I will go back to Host HA and try to speed up failover times.
Can I ask, from your experiences, what is a realistic fail over time for CS
ie. if a host fails for example ?
Jon
From: Paul Angus <paul.an...@shapeblue.com>
Sent: 23 May 2018 19:55
To: users@cloudstack.apache.org
Subject: RE: 4.11 without Host-HA framework
Jon,
As Rohit says, it is very important to understand the difference between VM
HA and host HA.
VM HA relies on the HOST telling CloudStack that the VM is down on order
for CloudStack start it again (wherever that ends up being).
Any sequence of events that ends up with VM HA restarting the VM when
CloudStack can't contact the host is luck/fluke/unreliable/bad(tm)
The purpose of Host HA was to create a reliable mechanism to determine that
a host has 'crashed' and that the VMs within it are inoperative. Then take
appropriate action, including ultimately telling VM HA to restart the VM
elsewhere.
paul.an...@shapeblue.com
www.shapeblue.com<http://www.shapeblue.com>
Shapeblue - The CloudStack Company<http://www.shapeblue.com/>
www.shapeblue.com
ShapeBlue are the largest independent integrator of CloudStack technologies
globally and are specialists in the design and implementation of IaaS cloud
infrastructures for both private and public cloud implementations.
53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue
-Original Message-
From: Rohit Yadav <rohit.ya...@shapeblue.com>
Sent: 23 May 2018 10:45
To: users@cloudstack.apache.org
Subject: Re: 4.11 without Host-HA framework
Jon,
In the VM's compute offering, make sure that HA is ticked/enabled. Then use
that HA-enabled VM offering while deploying a VM. Around testing - it depends
how you're crashing. In case of KVM, you can try to cause host crash (example:
echo c > /proc/sysrq-trigger) and see if HA-enabled VMs gets started on a
different host.
- Rohit
<https://cloudstack.apache.org>
From: Jon Marshall <jms@hotmail.co.uk>
Sent: Tuesday, May 22, 2018 8:28:06 PM
To: users@cloudstack.apache.org
Subject: Re: 4.11 without Host-HA framework
Hi Rohit
Thanks for responding.
I have not had much luck with HA at all. I crash a server and nothing
happens in terms of VMs migrating to another host. Monitoring the management
log file it seems the management server recognises the h
RE: 4.11 without Host-HA framework
Hi Jon, Don't worry, TBH I'm dubious about those claiming to have VM-HA working when a host crashes (but doesn't restart). I'll check in with the guys that set values for host-ha when testing, to see which ones they change and what they set them to. paul.an...@shapeblue.com www.shapeblue.com 53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue -Original Message- From: Jon Marshall <jms@hotmail.co.uk> Sent: 23 May 2018 21:10 To: users@cloudstack.apache.org Subject: Re: 4.11 without Host-HA framework Rohit / Paul Thanks again for answering. I am a Cisco guy with an ex Unix background but no virtualisation experience and I can honestly say I have never felt this stupid before I have Cloudstack working but failover is killing me. When you say VM HA relies on the host telling CS the VM is down how does that work because if you crash the host how does it tell CS anything ? And when you say tell CS do you mean the CS manager ? I guess I am just not understanding all the moving parts. I have had HOST HA working (to an extent) although it takes a long time to failover even after tweaking the timers but the fact that I keep finding references to people saying even without HOST HA it should failover (and mine doesn't) makes me think I have configured it incorrectly somewhere along the line. I have configured a compute offering with HA and I am crashing the host with the echo command as suggested but still nothing. I understand what you are saying Paul about it not being a good idea to rely on VM HA so I will go back to Host HA and try to speed up failover times. Can I ask, from your experiences, what is a realistic fail over time for CS ie. if a host fails for example ? Jon From: Paul Angus <paul.an...@shapeblue.com> Sent: 23 May 2018 19:55 To: users@cloudstack.apache.org Subject: RE: 4.11 without Host-HA framework Jon, As Rohit says, it is very important to understand the difference between VM HA and host HA. VM HA relies on the HOST telling CloudStack that the VM is down on order for CloudStack start it again (wherever that ends up being). Any sequence of events that ends up with VM HA restarting the VM when CloudStack can't contact the host is luck/fluke/unreliable/bad(tm) The purpose of Host HA was to create a reliable mechanism to determine that a host has 'crashed' and that the VMs within it are inoperative. Then take appropriate action, including ultimately telling VM HA to restart the VM elsewhere. paul.an...@shapeblue.com www.shapeblue.com<http://www.shapeblue.com> Shapeblue - The CloudStack Company<http://www.shapeblue.com/> www.shapeblue.com ShapeBlue are the largest independent integrator of CloudStack technologies globally and are specialists in the design and implementation of IaaS cloud infrastructures for both private and public cloud implementations. 53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue -Original Message- From: Rohit Yadav <rohit.ya...@shapeblue.com> Sent: 23 May 2018 10:45 To: users@cloudstack.apache.org Subject: Re: 4.11 without Host-HA framework Jon, In the VM's compute offering, make sure that HA is ticked/enabled. Then use that HA-enabled VM offering while deploying a VM. Around testing - it depends how you're crashing. In case of KVM, you can try to cause host crash (example: echo c > /proc/sysrq-trigger) and see if HA-enabled VMs gets started on a different host. - Rohit <https://cloudstack.apache.org> From: Jon Marshall <jms@hotmail.co.uk> Sent: Tuesday, May 22, 2018 8:28:06 PM To: users@cloudstack.apache.org Subject: Re: 4.11 without Host-HA framework Hi Rohit Thanks for responding. I have not had much luck with HA at all. I crash a server and nothing happens in terms of VMs migrating to another host. Monitoring the management log file it seems the management server recognises the host has stopped responding to pings but doesn't think it has to do anything. I am currently running v4.11 with basic network but 3 separate NICs, one for management, one for storage and one for VMs themselves. Should it make it any difference ie. would it be worth trying to run management and storage over the same NIC ? I am just lost as to why I see no failover at all whereas others are reporting it works fine. Jon From: Rohit Yadav <rohit.ya...@shapeblue.com> Sent: 22 May 2018 12:12 To: users@cloudstack.apache.org Subject: Re: 4.11 without Host-HA framework Hi Jon, Yes, Host-HA is different from VM-HA and without Host HA enabled a HA enabled VM should be recovered/run on a different host when it crashes. Historically the term 'HA' in CloudStack is used around high availability of a VM. Host HA as the name tries to imply is around HA of a physical hypervisor host by means of out-of-band
Re: 4.11 without Host-HA framework
Rohit / Paul Thanks again for answering. I am a Cisco guy with an ex Unix background but no virtualisation experience and I can honestly say I have never felt this stupid before I have Cloudstack working but failover is killing me. When you say VM HA relies on the host telling CS the VM is down how does that work because if you crash the host how does it tell CS anything ? And when you say tell CS do you mean the CS manager ? I guess I am just not understanding all the moving parts. I have had HOST HA working (to an extent) although it takes a long time to failover even after tweaking the timers but the fact that I keep finding references to people saying even without HOST HA it should failover (and mine doesn't) makes me think I have configured it incorrectly somewhere along the line. I have configured a compute offering with HA and I am crashing the host with the echo command as suggested but still nothing. I understand what you are saying Paul about it not being a good idea to rely on VM HA so I will go back to Host HA and try to speed up failover times. Can I ask, from your experiences, what is a realistic fail over time for CS ie. if a host fails for example ? Jon From: Paul Angus <paul.an...@shapeblue.com> Sent: 23 May 2018 19:55 To: users@cloudstack.apache.org Subject: RE: 4.11 without Host-HA framework Jon, As Rohit says, it is very important to understand the difference between VM HA and host HA. VM HA relies on the HOST telling CloudStack that the VM is down on order for CloudStack start it again (wherever that ends up being). Any sequence of events that ends up with VM HA restarting the VM when CloudStack can't contact the host is luck/fluke/unreliable/bad(tm) The purpose of Host HA was to create a reliable mechanism to determine that a host has 'crashed' and that the VMs within it are inoperative. Then take appropriate action, including ultimately telling VM HA to restart the VM elsewhere. paul.an...@shapeblue.com www.shapeblue.com<http://www.shapeblue.com> Shapeblue - The CloudStack Company<http://www.shapeblue.com/> www.shapeblue.com ShapeBlue are the largest independent integrator of CloudStack technologies globally and are specialists in the design and implementation of IaaS cloud infrastructures for both private and public cloud implementations. 53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue -Original Message- From: Rohit Yadav <rohit.ya...@shapeblue.com> Sent: 23 May 2018 10:45 To: users@cloudstack.apache.org Subject: Re: 4.11 without Host-HA framework Jon, In the VM's compute offering, make sure that HA is ticked/enabled. Then use that HA-enabled VM offering while deploying a VM. Around testing - it depends how you're crashing. In case of KVM, you can try to cause host crash (example: echo c > /proc/sysrq-trigger) and see if HA-enabled VMs gets started on a different host. - Rohit <https://cloudstack.apache.org> From: Jon Marshall <jms@hotmail.co.uk> Sent: Tuesday, May 22, 2018 8:28:06 PM To: users@cloudstack.apache.org Subject: Re: 4.11 without Host-HA framework Hi Rohit Thanks for responding. I have not had much luck with HA at all. I crash a server and nothing happens in terms of VMs migrating to another host. Monitoring the management log file it seems the management server recognises the host has stopped responding to pings but doesn't think it has to do anything. I am currently running v4.11 with basic network but 3 separate NICs, one for management, one for storage and one for VMs themselves. Should it make it any difference ie. would it be worth trying to run management and storage over the same NIC ? I am just lost as to why I see no failover at all whereas others are reporting it works fine. Jon From: Rohit Yadav <rohit.ya...@shapeblue.com> Sent: 22 May 2018 12:12 To: users@cloudstack.apache.org Subject: Re: 4.11 without Host-HA framework Hi Jon, Yes, Host-HA is different from VM-HA and without Host HA enabled a HA enabled VM should be recovered/run on a different host when it crashes. Historically the term 'HA' in CloudStack is used around high availability of a VM. Host HA as the name tries to imply is around HA of a physical hypervisor host by means of out-of-band management technologies such as ipmi and currently supporting ipmi as OOBM and KVM hosts with NFS storage. - Rohit <https://cloudstack.apache.org> [https://cloudstack.apache.org/images/monkey-144.png]<https://cloudstack.apache.org/> Apache CloudStack: Open Source Cloud Computing<https://cloudstack.apache.org/> cloudstack.apache.org CloudStack is open source cloud computing software for creating, managing, and deploying infrastructure cloud services From: Jon Marshall <jms.
RE: 4.11 without Host-HA framework
Jon, As Rohit says, it is very important to understand the difference between VM HA and host HA. VM HA relies on the HOST telling CloudStack that the VM is down on order for CloudStack start it again (wherever that ends up being). Any sequence of events that ends up with VM HA restarting the VM when CloudStack can't contact the host is luck/fluke/unreliable/bad(tm) The purpose of Host HA was to create a reliable mechanism to determine that a host has 'crashed' and that the VMs within it are inoperative. Then take appropriate action, including ultimately telling VM HA to restart the VM elsewhere. paul.an...@shapeblue.com www.shapeblue.com 53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue -Original Message- From: Rohit Yadav <rohit.ya...@shapeblue.com> Sent: 23 May 2018 10:45 To: users@cloudstack.apache.org Subject: Re: 4.11 without Host-HA framework Jon, In the VM's compute offering, make sure that HA is ticked/enabled. Then use that HA-enabled VM offering while deploying a VM. Around testing - it depends how you're crashing. In case of KVM, you can try to cause host crash (example: echo c > /proc/sysrq-trigger) and see if HA-enabled VMs gets started on a different host. - Rohit <https://cloudstack.apache.org> From: Jon Marshall <jms@hotmail.co.uk> Sent: Tuesday, May 22, 2018 8:28:06 PM To: users@cloudstack.apache.org Subject: Re: 4.11 without Host-HA framework Hi Rohit Thanks for responding. I have not had much luck with HA at all. I crash a server and nothing happens in terms of VMs migrating to another host. Monitoring the management log file it seems the management server recognises the host has stopped responding to pings but doesn't think it has to do anything. I am currently running v4.11 with basic network but 3 separate NICs, one for management, one for storage and one for VMs themselves. Should it make it any difference ie. would it be worth trying to run management and storage over the same NIC ? I am just lost as to why I see no failover at all whereas others are reporting it works fine. Jon From: Rohit Yadav <rohit.ya...@shapeblue.com> Sent: 22 May 2018 12:12 To: users@cloudstack.apache.org Subject: Re: 4.11 without Host-HA framework Hi Jon, Yes, Host-HA is different from VM-HA and without Host HA enabled a HA enabled VM should be recovered/run on a different host when it crashes. Historically the term 'HA' in CloudStack is used around high availability of a VM. Host HA as the name tries to imply is around HA of a physical hypervisor host by means of out-of-band management technologies such as ipmi and currently supporting ipmi as OOBM and KVM hosts with NFS storage. - Rohit <https://cloudstack.apache.org> [https://cloudstack.apache.org/images/monkey-144.png]<https://cloudstack.apache.org/> Apache CloudStack: Open Source Cloud Computing<https://cloudstack.apache.org/> cloudstack.apache.org CloudStack is open source cloud computing software for creating, managing, and deploying infrastructure cloud services From: Jon Marshall <jms@hotmail.co.uk> Sent: Monday, May 21, 2018 8:36:04 PM To: users@cloudstack.apache.org Subject: 4.11 without Host-HA framework I keep seeing conflicting information about this in the mailing lists and in blogs etc. If I run 4.11 without enabling Host HA framework should HA still work if I crash a compute node because my understanding was the new framework was added for certain cases only. It doesn't work for me but I can find a number of people saying you don't need to enable the new framework for it to work. Thanks Jon rohit.ya...@shapeblue.com www.shapeblue.com<http://www.shapeblue.com> 53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue rohit.ya...@shapeblue.com www.shapeblue.com 53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue
Re: 4.11 without Host-HA framework
Jon, In the VM's compute offering, make sure that HA is ticked/enabled. Then use that HA-enabled VM offering while deploying a VM. Around testing - it depends how you're crashing. In case of KVM, you can try to cause host crash (example: echo c > /proc/sysrq-trigger) and see if HA-enabled VMs gets started on a different host. - Rohit <https://cloudstack.apache.org> From: Jon Marshall <jms@hotmail.co.uk> Sent: Tuesday, May 22, 2018 8:28:06 PM To: users@cloudstack.apache.org Subject: Re: 4.11 without Host-HA framework Hi Rohit Thanks for responding. I have not had much luck with HA at all. I crash a server and nothing happens in terms of VMs migrating to another host. Monitoring the management log file it seems the management server recognises the host has stopped responding to pings but doesn't think it has to do anything. I am currently running v4.11 with basic network but 3 separate NICs, one for management, one for storage and one for VMs themselves. Should it make it any difference ie. would it be worth trying to run management and storage over the same NIC ? I am just lost as to why I see no failover at all whereas others are reporting it works fine. Jon From: Rohit Yadav <rohit.ya...@shapeblue.com> Sent: 22 May 2018 12:12 To: users@cloudstack.apache.org Subject: Re: 4.11 without Host-HA framework Hi Jon, Yes, Host-HA is different from VM-HA and without Host HA enabled a HA enabled VM should be recovered/run on a different host when it crashes. Historically the term 'HA' in CloudStack is used around high availability of a VM. Host HA as the name tries to imply is around HA of a physical hypervisor host by means of out-of-band management technologies such as ipmi and currently supporting ipmi as OOBM and KVM hosts with NFS storage. - Rohit <https://cloudstack.apache.org> [https://cloudstack.apache.org/images/monkey-144.png]<https://cloudstack.apache.org/> Apache CloudStack: Open Source Cloud Computing<https://cloudstack.apache.org/> cloudstack.apache.org CloudStack is open source cloud computing software for creating, managing, and deploying infrastructure cloud services From: Jon Marshall <jms@hotmail.co.uk> Sent: Monday, May 21, 2018 8:36:04 PM To: users@cloudstack.apache.org Subject: 4.11 without Host-HA framework I keep seeing conflicting information about this in the mailing lists and in blogs etc. If I run 4.11 without enabling Host HA framework should HA still work if I crash a compute node because my understanding was the new framework was added for certain cases only. It doesn't work for me but I can find a number of people saying you don't need to enable the new framework for it to work. Thanks Jon rohit.ya...@shapeblue.com www.shapeblue.com<http://www.shapeblue.com> 53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue rohit.ya...@shapeblue.com www.shapeblue.com 53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue
Re: 4.11 without Host-HA framework
Hi Rohit Thanks for responding. I have not had much luck with HA at all. I crash a server and nothing happens in terms of VMs migrating to another host. Monitoring the management log file it seems the management server recognises the host has stopped responding to pings but doesn't think it has to do anything. I am currently running v4.11 with basic network but 3 separate NICs, one for management, one for storage and one for VMs themselves. Should it make it any difference ie. would it be worth trying to run management and storage over the same NIC ? I am just lost as to why I see no failover at all whereas others are reporting it works fine. Jon From: Rohit Yadav <rohit.ya...@shapeblue.com> Sent: 22 May 2018 12:12 To: users@cloudstack.apache.org Subject: Re: 4.11 without Host-HA framework Hi Jon, Yes, Host-HA is different from VM-HA and without Host HA enabled a HA enabled VM should be recovered/run on a different host when it crashes. Historically the term 'HA' in CloudStack is used around high availability of a VM. Host HA as the name tries to imply is around HA of a physical hypervisor host by means of out-of-band management technologies such as ipmi and currently supporting ipmi as OOBM and KVM hosts with NFS storage. - Rohit <https://cloudstack.apache.org> [https://cloudstack.apache.org/images/monkey-144.png]<https://cloudstack.apache.org/> Apache CloudStack: Open Source Cloud Computing<https://cloudstack.apache.org/> cloudstack.apache.org CloudStack is open source cloud computing software for creating, managing, and deploying infrastructure cloud services From: Jon Marshall <jms@hotmail.co.uk> Sent: Monday, May 21, 2018 8:36:04 PM To: users@cloudstack.apache.org Subject: 4.11 without Host-HA framework I keep seeing conflicting information about this in the mailing lists and in blogs etc. If I run 4.11 without enabling Host HA framework should HA still work if I crash a compute node because my understanding was the new framework was added for certain cases only. It doesn't work for me but I can find a number of people saying you don't need to enable the new framework for it to work. Thanks Jon rohit.ya...@shapeblue.com www.shapeblue.com<http://www.shapeblue.com> 53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue
Re: 4.11 without Host-HA framework
Hi Jon, Yes, Host-HA is different from VM-HA and without Host HA enabled a HA enabled VM should be recovered/run on a different host when it crashes. Historically the term 'HA' in CloudStack is used around high availability of a VM. Host HA as the name tries to imply is around HA of a physical hypervisor host by means of out-of-band management technologies such as ipmi and currently supporting ipmi as OOBM and KVM hosts with NFS storage. - Rohit <https://cloudstack.apache.org> From: Jon Marshall <jms@hotmail.co.uk> Sent: Monday, May 21, 2018 8:36:04 PM To: users@cloudstack.apache.org Subject: 4.11 without Host-HA framework I keep seeing conflicting information about this in the mailing lists and in blogs etc. If I run 4.11 without enabling Host HA framework should HA still work if I crash a compute node because my understanding was the new framework was added for certain cases only. It doesn't work for me but I can find a number of people saying you don't need to enable the new framework for it to work. Thanks Jon rohit.ya...@shapeblue.com www.shapeblue.com 53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue
4.11 without Host-HA framework
I keep seeing conflicting information about this in the mailing lists and in blogs etc. If I run 4.11 without enabling Host HA framework should HA still work if I crash a compute node because my understanding was the new framework was added for certain cases only. It doesn't work for me but I can find a number of people saying you don't need to enable the new framework for it to work. Thanks Jon
