> -----Original Message----- > From: Marcus Sorensen [mailto:shadow...@gmail.com] > Sent: Wednesday, August 07, 2013 1:46 PM > To: dev@cloudstack.apache.org > Subject: Re: [DISCUSS] KVM HA > > I'm not sure we can rely on IPMI to tell us much about the host status itself. > It's easy to use it for checking on basic poweron/poweroff, temperature, etc, > but not so easy to tell if something is wrong with the OS, config, or at the > software level.
Check the host status is enough, the investigator only needs to know, the host is power on or off. If host is power off, then cloudstack can start VM on another host, without worried about the split brain issue. > > However, I did mention support in that thread early on for sending an IPMI > poweroff for hosts that cloudstack has determined are down and starts > migrating vms for, as a safety precaution. > > On Wed, Aug 7, 2013 at 2:41 PM, Marcus Sorensen <shadow...@gmail.com> > wrote: > > Does KVMInvestigator work on all shared primary storage, or just NFS? > > I'm only familiar with the NFS KVMHA directories. > > > > From this it seems like a clean stop of the KVM agent still shouldn't > > trigger any issues/HA, correct? > > > > On Wed, Aug 7, 2013 at 2:28 PM, Edison Su <edison...@citrix.com> wrote: > >> There is long time issue related to KVM HA, see bug: CLOUDSTACK-3535. > Basically, HA won't be triggered, if KVM agent is stopped either normally nor > abnormally, HA only be triggered if the network between mgt server and > kvm host is disconnected and the network between KVM hosts in the same > cluster is disconnected. > >> Here is how the KVM HA works after the fix for CLOUDSTACK-3535: > >> 1. If agent is stopped, agent will send a shutdown request to mgt server, > mgt server will mark the host as disconnected, while still maintain the host > in > pingmap. Code is in AgentManagerImpl->AgentHandler- >ProcessRequest-> > disconnectWithoutInvestigation > >> 2. After ping.interval, mgt server will find the host is ping > >> timeout, then start HA investigation for the host. Code is in > AgentMonitor->run-> disconnectWithInvestigation 3. Mgt server will call all > the available Investigators to investigate the status of host. > >> The current investigators will be called for KVM host: > >> UserVmDomRInvestigator->isAgentAlive, will send PingTestCommand > to the host's neighbor. PingTestCommand will ping host's private ip address, > if ping is reachable, means host is up, otherwise, host's state is unknown. So > this investigator can only detect host is in up state. > >> KVMInvestigator, which is newly added, will send a > CheckOnHostCommand to host's neighbor. CheckOnHostCommand will > check the heartbeat of host(heartbeat is stored on shared primary storage). > Ideally, it will detect host is down or up. > >> > >> Combined with UserVmDomRInvestigator and KVMInvestigator, mgt > server should find out the status of host. But there is case, these two > investigators can report wrong status of host: > >> Host is in a network partition, while the KVM agent is > >> down(thus heartbeat is stopped) 4. After investigator reports status of > host, if host is down, then start HA for VMs created on the host. > >> > >> > >> Improvement: > >> Per suggestion from Lennert den Teuling, we'd better use IPMI to > detect host status, which is more reliable than ping and heartbeat, as IPMI > has its own network, less likely has network partition. > >> > >>
