Thanks Jon. I've been following that thread too, have now much clear understanding of how VM HA behaves in cloudstack. Thanks for finding a bug and saving time of others :)
Regards, Parth Patel On Mon 11 Jun, 2018, 16:30 Jon Marshall, <[email protected]> wrote: > Hi Parth > > > Just in case you have not seen my other thread, it turns out that all this > time it has been a bug. > > > Using multiple NICs with basic networking and using zone wide NFS VM HA > just does not work. If you change to cluster wide NFS then it works fine > (and quite quickly as well :)) > > > I am now going to setup Host HA and see make sure that all works as well > using cluster NFS. > > > Got there in the end :) > > > Jon > > > > > > ________________________________ > From: Parth Patel <[email protected]> > Sent: 24 May 2018 06:52 > To: [email protected] > Subject: Re: 4.11 without Host-HA framework > > Hi Jon and Angus, > > I did not shutdown the VMs as Yiping Zhang said, but I have confirmed this > and discussed earlier in the users list that my HA-enabled VMs got started > on another suitable available host in the cluster even when I didn't have > IPMI-enabled hardware and did no configuration for OOBM and Host-HA. I > simply pulled the ethernet cable connecting the host to entire network (I > did use just one NIC) and according to the value set in ping timeout event, > the HA-enabled VMs were restarted on another available host. I tested the > scenario using both the scenarios: the echo command as well as good old > plugging out the NIC from the host. My VMs were successfully started on > another available host after CS manager confirmed they were not reachable. > > I too want to understand how the failover mechanism in CloudStack actually > works. I used ACS 4.11 packages available here: > http://cloudstack.apt-get.eu/centos/7/4.11/ > > Regards, > Parth Patel > > > On Thu, 24 May 2018 at 10:53 Paul Angus <[email protected]> wrote: > > > I'm afraid that is not a host crash. When shutting down the guest OS, > the > > CloudStack agent on the host is still able to report to the management > > server that the VM has stopped. > > > > This is my point. VM-HA relies on the management sever communication with > > the host agent. > > > > Kind regards, > > > > Paul Angus > > > > [email protected] > > www.shapeblue.com<http://www.shapeblue.com> > > 53 Chandos Place, Covent Garden, London WC2N 4HSUK > > @shapeblue > > > > > > > > > > -----Original Message----- > > From: Yiping Zhang <[email protected]> > > Sent: 24 May 2018 00:44 > > To: [email protected] > > Subject: Re: 4.11 without Host-HA framework > > > > I can say for fact that VM's using a HA enabled service offering will be > > restarted by CS on another host, assuming there are enough > > capacity/resources in the cluster, when their original host crashes, > > regardless that host comes back or not. > > > > The simplest way to test VM HA feature with a VM instance using HA > enabled > > service offering is to issue shutdown command in guest OS, and watching > it > > gets restarted by CS manager. > > > > On 5/23/18, 1:23 PM, "Paul Angus" <[email protected]> wrote: > > > > Hi Jon, > > > > Don't worry, TBH I'm dubious about those claiming to have VM-HA > > working when a host crashes (but doesn't restart). > > I'll check in with the guys that set values for host-ha when testing, > > to see which ones they change and what they set them to. > > > > [email protected] > > www.shapeblue.com<http://www.shapeblue.com> > > 53 Chandos Place, Covent Garden, London WC2N 4HSUK > > @shapeblue > > > > > > > > > > -----Original Message----- > > From: Jon Marshall <[email protected]> > > Sent: 23 May 2018 21:10 > > To: [email protected] > > Subject: Re: 4.11 without Host-HA framework > > > > Rohit / Paul > > > > > > Thanks again for answering. > > > > > > I am a Cisco guy with an ex Unix background but no virtualisation > > experience and I can honestly say I have never felt this stupid before 😊 > > > > > > I have Cloudstack working but failover is killing me. > > > > > > When you say VM HA relies on the host telling CS the VM is down how > > does that work because if you crash the host how does it tell CS > anything ? > > And when you say tell CS do you mean the CS manager ? > > > > > > I guess I am just not understanding all the moving parts. I have had > > HOST HA working (to an extent) although it takes a long time to failover > > even after tweaking the timers but the fact that I keep finding > references > > to people saying even without HOST HA it should failover (and mine > doesn't) > > makes me think I have configured it incorrectly somewhere along the line. > > > > > > I have configured a compute offering with HA and I am crashing the > > host with the echo command as suggested but still nothing. > > > > > > I understand what you are saying Paul about it not being a good idea > > to rely on VM HA so I will go back to Host HA and try to speed up > failover > > times. > > > > > > Can I ask, from your experiences, what is a realistic fail over time > > for CS ie. if a host fails for example ? > > > > > > Jon > > > > > > > > > > ________________________________ > > From: Paul Angus <[email protected]> > > Sent: 23 May 2018 19:55 > > To: [email protected] > > Subject: RE: 4.11 without Host-HA framework > > > > Jon, > > > > As Rohit says, it is very important to understand the difference > > between VM HA and host HA. > > VM HA relies on the HOST telling CloudStack that the VM is down on > > order for CloudStack start it again (wherever that ends up being). > > Any sequence of events that ends up with VM HA restarting the VM when > > CloudStack can't contact the host is luck/fluke/unreliable/bad(tm) > > > > The purpose of Host HA was to create a reliable mechanism to > determine > > that a host has 'crashed' and that the VMs within it are inoperative. > Then > > take appropriate action, including ultimately telling VM HA to restart > the > > VM elsewhere. > > > > > > > > > > > > [email protected] > > www.shapeblue.com<http://www.shapeblue.com> > > Shapeblue - The CloudStack Company<http://www.shapeblue.com/> > > www.shapeblue.com<http://www.shapeblue.com> > > ShapeBlue are the largest independent integrator of CloudStack > > technologies globally and are specialists in the design and > implementation > > of IaaS cloud infrastructures for both private and public cloud > > implementations. > > > > > > > > 53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue > > > > > > > > > > -----Original Message----- > > From: Rohit Yadav <[email protected]> > > Sent: 23 May 2018 10:45 > > To: [email protected] > > Subject: Re: 4.11 without Host-HA framework > > > > Jon, > > > > > > In the VM's compute offering, make sure that HA is ticked/enabled. > > Then use that HA-enabled VM offering while deploying a VM. Around > testing - > > it depends how you're crashing. In case of KVM, you can try to cause host > > crash (example: echo c > /proc/sysrq-trigger) and see if HA-enabled VMs > > gets started on a different host. > > > > > > - Rohit > > > > <https://cloudstack.apache.org> > > > > > > > > ________________________________ > > From: Jon Marshall <[email protected]> > > Sent: Tuesday, May 22, 2018 8:28:06 PM > > To: [email protected] > > Subject: Re: 4.11 without Host-HA framework > > > > Hi Rohit > > > > > > Thanks for responding. > > > > > > I have not had much luck with HA at all. I crash a server and > nothing > > happens in terms of VMs migrating to another host. Monitoring the > > management log file it seems the management server recognises the host > has > > stopped responding to pings but doesn't think it has to do anything. > > > > > > I am currently running v4.11 with basic network but 3 separate NICs, > > one for management, one for storage and one for VMs themselves. > > > > > > Should it make it any difference ie. would it be worth trying to run > > management and storage over the same NIC ? > > > > > > I am just lost as to why I see no failover at all whereas others are > > reporting it works fine. > > > > > > Jon > > > > > > ________________________________ > > From: Rohit Yadav <[email protected]> > > Sent: 22 May 2018 12:12 > > To: [email protected] > > Subject: Re: 4.11 without Host-HA framework > > > > Hi Jon, > > > > > > Yes, Host-HA is different from VM-HA and without Host HA enabled a HA > > enabled VM should be recovered/run on a different host when it crashes. > > Historically the term 'HA' in CloudStack is used around high availability > > of a VM. > > > > > > Host HA as the name tries to imply is around HA of a physical > > hypervisor host by means of out-of-band management technologies such as > > ipmi and currently supporting ipmi as OOBM and KVM hosts with NFS > storage. > > > > > > - Rohit > > > > <https://cloudstack.apache.org> > > [https://cloudstack.apache.org/images/monkey-144.png]< > > https://cloudstack.apache.org/> > > > > Apache CloudStack: Open Source Cloud Computing< > > https://cloudstack.apache.org/> > > cloudstack.apache.org > > CloudStack is open source cloud computing software for creating, > > managing, and deploying infrastructure cloud services > > > > > > > > > > > > > > ________________________________ > > From: Jon Marshall <[email protected]> > > Sent: Monday, May 21, 2018 8:36:04 PM > > To: [email protected] > > Subject: 4.11 without Host-HA framework > > > > I keep seeing conflicting information about this in the mailing lists > > and in blogs etc. > > > > If I run 4.11 without enabling Host HA framework should HA still work > > if I crash a compute node because my understanding was the new framework > > was added for certain cases only. > > > > It doesn't work for me but I can find a number of people saying you > > don't need to enable the new framework for it to work. > > > > Thanks > > > > Jon > > > > [email protected] > > www.shapeblue.com<http://www.shapeblue.com> > > 53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue > > > > > > > > > > [email protected] > > www.shapeblue.com<http://www.shapeblue.com> > > 53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue > > > > > > > > > > > > >
