James,
Try just configuring a single NFS server and see if your setup works. If you have 3 NFS shares, across all 3 hosts, i'm wondering whether ACS is picking the one you rebooted as the storage for your VMs and when that storage goes away (when you bounce the host), all storage for your VMs vanishes and ACS tries to reboot your other hosts. Normally in a simple ACS setup, you would have a separate storage server that can serve up NFS to all hosts. If a host dies, then a VM would be brought up on a spare hosts since all hosts have access to the same storage. Your other option is to use local storage, but that won't provide HA. - Si ________________________________ From: McClune, James <mcclu...@norwalktruckers.net> Sent: Monday, October 30, 2017 2:26 PM To: users@cloudstack.apache.org Subject: Re: Problems with KVM HA & STONITH Hi Dag, Thank you for responding back. I am currently running ACS 4.9 on an Ubuntu 14.04 VM. I have the three nodes, each having about 1TB of primary storage (NFS) and 1TB of secondary storage (NFS). I added each NFS share into ACS. All nodes are in a cluster. Maybe I'm not understanding the setup or misconfigured something. I'm trying to setup an HA environment where if one node goes down, running an HA marked VM, the VM will start on another host. When I simulate a network disconnect or reboot of a host, all of the nodes go down (STONITH?). I am unsure on how to setup an HA environment, if all the nodes in the cluster go down. Any help is much appreciated! Thanks, James On Mon, Oct 30, 2017 at 3:49 AM, Dag Sonstebo <dag.sonst...@shapeblue.com> wrote: > Hi James, > > I think you possibly have over-configured your KVM hosts. If you use NFS > (and no clustered file system like CLVM) then there should be no need to > configure STONITH. CloudStack takes care of your HA, so this is not > something you offload to the KVM host. > > (As mentioned the only time I have played with STONITH and CloudStack was > for CLVM – and I eventually found it not fit for purpose, too unstable and > causing too many issues like you describe. Note this was for block storage > though – not NFS). > > Regards, > Dag Sonstebo > Cloud Architect > ShapeBlue > > On 28/10/2017, 03:40, "Ivan Kudryavtsev" <kudryavtsev...@bw-sw.com> wrote: > > Hi. If the node losts nfs host it reboots (acs agent behaviour). If you > really have 3 storages, you'll go clusterwide reboot everytime your > host is > down. > > 28 окт. 2017 г. 3:02 пользователь "Simon Weller" > <swel...@ena.com.invalid> > написал: > > > Hi James, > > > > > > Can you elaborate a bit further on the storage? You say you're > running NFS > > on all 3 nodes, can you explain how it is setup? > > > > Also, what version of ACS are you running? > > > > > > - Si > > > > > > > > > > ________________________________ > > From: McClune, James <mcclu...@norwalktruckers.net> > > Sent: Friday, October 27, 2017 2:21 PM > > To: users@cloudstack.apache.org > > Subject: Problems with KVM HA & STONITH > > > > Hello Apache CloudStack Community, > > > > My setup consists of the following: > > > > - Three nodes (NODE1, NODE2, and NODE3) > > NODE1 is running Ubuntu 16.04.3, NODE2 is running Ubuntu 16.04.3, > and NODE3 > > is running Ubuntu 14.04.5. > > - Management Server (running on separate VM, not in cluster) > > > > The three nodes use KVM as the hypervisor. I also configured primary > and > > secondary storage on all three of the nodes. I'm using NFS for the > primary > > & secondary storage. VM operations work great. Live migration works > great. > > > > However, when a host goes down, the HA functionality does not work > at all. > > Instead of spinning up the VM on another available host, the down > host > > seems to trigger STONITH. When STONITH happens, all hosts in the > cluster go > > down. This not only causes no HA, but also downs perfectly good > VM's. I > > have read countless articles and documentation related to this > issue. I > > still cannot find a viable solution for this issue. I really want to > use > > Apache CloudStack, but cannot implement this in production when > STONITH > > happens. > > > > I think I have something misconfigured. I thought I would reach out > to the > > CloudStack community and ask for some friendly assistance. > > > > If there is anything (system-wise) you request in order to further > > troubleshoot this issue, please let me know and I'll send. I > appreciate any > > help in this issue! > > > > -- > > > > Thanks, > > > > James > > > > > > dag.sonst...@shapeblue.com > www.shapeblue.com<http://www.shapeblue.com> [http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png]<http://www.shapeblue.com/> Shapeblue - The CloudStack Company<http://www.shapeblue.com/> www.shapeblue.com Rapid deployment framework for Apache CloudStack IaaS Clouds. CSForge is a framework developed by ShapeBlue to deliver the rapid deployment of a standardised ... > 53 Chandos Place, Covent Garden, London WC2N 4HSUK > @shapeblue > > > > -- James McClune Technical Support Specialist Norwalk City Schools Phone: 419-660-6590 mcclu...@norwalktruckers.net
