Each (KVM) cloudstacl agent executes a heartbeat script (every 1min or so), trying to write to a primary NFS server, ensuring it has the connectivity working.
If this heartbeat fails - agent will say - "ok I'm stupid and less reliable, NFS storage must be HA, so let me reboot myself/host in order to regain access to NFS" (which is on your mgmt server) /usr/lib64/cloud/agent/scripts/vm/hypervisor/kvm/kvmheartbeat.sh Comment the line " echo b > /proc/sysrq-trigger " and KVM agent will NOT reboot itself when NFS is unavailable. Something that is also wise to do in production as well (trust me.... ... ) Cheers, On Mon, 28 Jun 2021 at 18:57, Brian Fitzpatrick <[email protected]> wrote: > Hi Andrija, > > Thanks for the reply > > There is NFS primary storage on the management server but it doesn't have > any vm volumes on it, they are on other NFS primary servers. > > So I thought I should have been safe to reboot > > All vms on other hosts, vm volumes on other nfs shares living on other > servers, system vms and routers also on other servers > > So all (I am fairly certain) that was on the server I rebooted was the one > and only Management server and mysql > > Thanks > > Brian > > -----Original Message----- > From: Andrija Panic <[email protected]<mailto: > andrija%20panic%20%[email protected]%3e>> > Reply-To: [email protected]<mailto:[email protected]> > To: users <[email protected]<mailto: > users%20%[email protected]%3e>> > Subject: Re: Management server reboot appears to cause vms on other hosts > to shutdown? > Date: Mon, 28 Jun 2021 17:44:57 +0200 > > > CAUTION ! > > > This email was NOT sent using a University of Chester account, so we are > unable to verify the identity of the sender. Do not click links or open > attachments unless you recognise the sender and know the content is safe. > > > ===== > > > > NFS primary storage also on your management server? > > > On Sat, 26 Jun 2021 at 10:36, Brian Fitzpatrick < > > <mailto:[email protected]> > > [email protected] > > > > > wrote: > > > Hi Jordan, > > > Thanks for your reply. Apologies I might not have been clear. > > > The management server is aware of the vm and when I set the host that is > > also the same server that is running the management server (and mysql) into > > maintenance mode, I can see it not longer has any running vms on it. They > > have migrated to other hosts. Cloud stack can see them. Bu when I then to > > an apt update and reboot the management server, the vms on the other hosts > > seem to have shutdown. > > > The reboot did take a while (15-20 mins), but I am surprised that it has > > affected other kvm hosts, which I thought should just carry on running. > > Unless I have missed something that was still on the management(and mysql) > > server. > > > Thanks > > > Brian > > > -----Original Message----- > > From: Yordan Kostov < > > <mailto:[email protected]> > > [email protected] > > <mailto: > > <mailto:yordan%20kostov%20%[email protected]> > > yordan%20kostov%20%[email protected] > > %3e>> > > Reply-To: > > <mailto:[email protected]> > > [email protected] > > <mailto: > > <mailto:[email protected]> > > [email protected] > > > > > To: > > <mailto:[email protected]> > > [email protected] > > < > > <mailto:[email protected]> > > [email protected] > > <mailto:% > > <mailto:[email protected]> > > [email protected] > > <mailto:%22%20%[email protected]> > > %22%20%[email protected] > > %3e>> > > Subject: RE: Management server reboot appears to cause vms on other hosts > > to shutdown? > > Date: Fri, 25 Jun 2021 09:10:44 +0000 > > > > CAUTION ! > > > > This email was NOT sent using a University of Chester account, so we are > > unable to verify the identity of the sender. Do not click links or open > > attachments unless you recognise the sender and know the content is safe. > > > > ===== > > > > > Hello Brian, > > > > May be I did not understand very well but from what you say I get > > that the management server + SQL and NFS are on the same physical hosts > > that are being managed by cloudstack? > > > If those VMs are not visible in Cloudstack, the system is not > > aware that they exist so it wont try to roll them to another host if you > > perform hypervisor host reboot. > > > > Best regards, > > > Jordan > > > > -----Original Message----- > > > From: Brian Fitzpatrick < > > > <mailto: > > <mailto:[email protected]> > > [email protected] > > > > > > <mailto:[email protected]> > > [email protected] > > > > > > Sent: Friday, June 25, 2021 12:06 PM > > > To: > > > <mailto: > > <mailto:[email protected]> > > [email protected] > > > > > > <mailto:[email protected]> > > [email protected] > > > > > Subject: Management server reboot appears to cause vms on other hosts to > > shutdown? > > > > > [X] This message came from outside your organization > > > > > Hi all, > > > > Still relatively new to CloudStack and learning, testing etc. > > > > I have created 1 management server with mysql on it and created 2 clusters > > with a nfs primary storage server in each and a number of hosts in each. > > > > I have been working through the servers, putting them in maintenance mode > > (noting the vm migrations), updating and rebooting them. All working fine > > > > I then wanted to update and reboot the server running the management and > > mysql. It is also a host, so I set it in maintenance mode so no vms running > > on it. > > > > I thought if I update it and reboot, all I would lose for a period of time > > was access to the management server, the vms should keep running on their > > various hosts > > > > The reboot, took longer than usual, it seemed to hang for 15-20mins before > > shutting down and rebooting. To my surprise though I lost contact to all > > the vms on the other hosts. > > > > They all shut down. > > > > Apologies, if I have missed something here, I thought I understood. All > > virtual routers and system vms appeared to be running on the other hosts. > > > > Is it because the management server took a while to reboot, the other > > hosts have lost contact and shutdown their vms? seems odd? > > > > Any suggestions, help welcome. As I say, still learning! > > > > Thanks > > > > Brian > > > > > -- > > > Andrija Panić > -- Andrija Panić
