Hi Paul
I did make some progress with this and seem to remember that after it said Recovered it then went back to Suspect and finally Fenced. I am going to rerun a lot of the tests after changing some of the kvm_ha_ timers to try and speed things up a bit. Will update here after I have run tests to check if that is what I should be seeing. Many thanks Jon ________________________________ From: Paul Angus <paul.an...@shapeblue.com> Sent: 28 March 2018 20:01 To: users@cloudstack.apache.org Subject: Re: Failover for VMs Ah. Did you wait after the node said recovered? That message is spurious. I've seen it also. It should say recovering. at that time. ________________________________ From: Jon Marshall <jms....@hotmail.co.uk> Sent: Tuesday, 27 March 2018 10:42 am To: users@cloudstack.apache.org Subject: Re: Failover for VMs Just as an update to this before I forget what I did :) - I used "echo c /proc/sysrq-trigger" on one of the compute nodes and there was no VM failover. Instead HA reported suspect and then IPMI rebooted the machine, it came back online and the VM started responding to pings again. IPMI is out of band so that seems to be reasonable behaviour but no use in testing HA. Next I just pulled all 3 NIC cables from the same compute node and again HA reported suspect. Again IPMI rebooted but then HA state changed to "Recovered" which I don't understand as the NIC cables were still disconnected so VM was not reachable and no failover. I don't understand how it can think the node is recovered as apart from the IPMI out of band connection there are no network connections to this server. Finally pulled power lead and this time HA went from suspect to Fencing and then stayed that way. Again no VM failover. This makes sense as no power means IPMI cannot reboot server so it never moves to Fenced I assume. Again no failover. I am wondering if it is to do with out of band IPMI or the way I have the NICs setup. The management node only has one NIC in the management network but I assume this is okay. I may try reloading with CS v4.9 and just try failover without the new HA KVM to see if I see anything different. Jon ________________________________ From: Jon Marshall <jms....@hotmail.co.uk> Sent: 27 March 2018 10:10 To: users@cloudstack.apache.org Subject: Re: Failover for VMs Thanks Paul, will pick up after Easter break. Doing some more testing with HA KVM at the moment so any progress will update this thread i ________________________________ From: Paul Angus <paul.an...@shapeblue.com Sent: 27 March 2018 10:07 To: users@cloudstack.apache.org Subject: RE: Failover for VMr Jon, I've been updating the Ansible to move our physical hosts from Centos6 to Centos7, now that's done I'll run through an HA setup and post answers (probably after easter break). paul.an...@shapeblue.com www.shapeblue.com<http://www.shapeblue.com> [http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png]<http://www.shapeblue.com/> Shapeblue - The CloudStack Company<http://www.shapeblue.com/> www.shapeblue.com Rapid deployment framework for Apache CloudStack IaaS Clouds. CSForge is a framework developed by ShapeBlue to deliver the rapid deployment of a standardised ... [http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png]<http://www.shapeblue.com/> Shapeblue - The CloudStack Company<http://www.shapeblue.com/> www.shapeblue.com<http://www.shapeblue.com> Rapid deployment framework for Apache CloudStack IaaS Clouds. CSForge is a framework developed by ShapeBlue to deliver the rapid deployment of a standardised ... [http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png]<http://www.shapeblue.com/> Shapeblue - The CloudStack Company<http://www.shapeblue.com/> www.shapeblue.com<http://www.shapeblue.com> Rapid deployment framework for Apache CloudStack IaaS Clouds. CSForge is a framework developed by ShapeBlue to deliver the rapid deployment of a standardised ... 53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue paul.an...@shapeblue.com www.shapeblue.com<http://www.shapeblue.com> 53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue -----Original Message----- From: Jon Marshall <jms....@hotmail.co.uk> Sent: 27 March 2018 09:19 To: users@cloudstack.apache.org Subject: Failover for VMs After 3 weeks of trying multiple different setups I still have not managed to get a VM to failover between compute nodes and am just running out of ideas. I have 3 compute nodes each with 3 NICS (management, VMs traffic, storage), one management node with just a single NIC connection in the management network and a separate NFS server. I have tried with and without the new Host HA KVM in CS v4.11 as from what I have read even without enabling the new Host HA KVM when you power off or reboot a compute node your VMs should still migrate. I have tried powering off a compute node, pulling the power lead, removing the management and NFS network cables and the management server just seems to carry on as if nothing has happened. Could someone explain exactly how HA is meant to work so I can look at where it is going wrong.
