Re: Recover VM after KVM host down (and HA not working) ?

[Jean-Francois Nadeau]

Hmm could this be the culprit ?

WARN  [c.c.h.KVMInvestigator] (AgentTaskPool-10:ctx-694feb6c)
(logid:160220c5) Agent investigation was requested on host
Host[-4-Routing], but host does not support investigation because it has no
NFS storage. Skipping investigation.

The primary storage is NFS.

On Sat, Dec 23, 2017 at 10:14 AM, Jean-Francois Nadeau <
the.jfnad...@gmail.com> wrote:

> Clearly the management server doesn't realize the instance on the failed
> host is not running...  but the host is in Alert state and powered down,
> and missing NFS heartbeats.
>
> 2017-12-23 14:57:52,427 DEBUG [c.c.h.Status] (AgentTaskPool-10:ctx-694feb6c)
> (logid:160220c5) Transition:[Resource state = Enabled, Agent event =
> AgentDisconnected, Host id = 4, name = r62-i122-36-01.domain.com]
> 2017-12-23 14:58:24,487 DEBUG [c.c.c.CapacityManagerImpl]
> (CapacityChecker:ctx-66fbe484) (logid:1f53cd63) Found 1 VMs on host 4
> 2017-12-23 14:58:24,495 DEBUG [c.c.c.CapacityManagerImpl]
> (CapacityChecker:ctx-66fbe484) (logid:1f53cd63) Found 0 VM, not running on
> host 4
>
> Next step ?
>
> On Sat, Dec 23, 2017 at 9:49 AM, Jean-Francois Nadeau <
> the.jfnad...@gmail.com> wrote:
>
>> I'd really like to get at the bottom of this.It does sound like the
>> behavior mentioned in https://issues.apache.org/j
>> ira/browse/CLOUDSTACK-5582 but should be long fixed.
>>
>> One suspect log entry (be unrelated) I noticed is this recurring
>> exception in the manger logs :
>>
>> ERROR [c.c.v.UserVmManagerImpl] (UserVm-ipfetch-3:ctx-d4c44c2b)
>> (logid:16dd70ad) Caught the Exception in VmIpFetchTask
>>
>> Which I guess is caused by the use of an external DHCP so manager fails
>> to determine a running VM IP.Which brings me to my next question
>> how is a VM marked for HA actually monitored ?
>>
>>
>> On Sat, Dec 23, 2017 at 3:38 AM, Eric Green 
>> wrote:
>>
>>> If all else fails, change its state to the correct  state in the MySQL
>>> database and restart the management  service. Sadly that is the only way
>>> I
>>> could do it when my Cloudstack got confused and stuck an instance in an
>>> intermediate state where I couldn't do anything with it.
>>>
>>> On Dec 22, 2017 at 9:09 AM, >> the.jfnad...@gmail.com>>
>>> wrote:
>>>
>>> Good morning,
>>>
>>> New to ACS and doing a POC with 4.10 on Centos 7 and KVM.
>>>
>>> Im trying to recover VMs after an host failure (powered off from OOB).
>>>
>>> Primary storage is NFS and IPMI is configured for the KVM hosts.  Zone is
>>> advanced mode with vlan separation and created a shared network with no
>>> services since I wish to use an external DHCP.
>>>
>>> First,  say I don't have a compute offering with HA enabled and a KVM
>>> host
>>> goes down...  I can't put it in maintenance mode while down and disabling
>>> it have no effect on the state of the lost VMs.  VM stays in running
>>> state
>>> according to manager.   What should I do to force restart on remaining
>>> healthy hosts ?
>>>
>>> Then I enabled  IPMI on all KVM hosts and attempted the same experience
>>> with a compute offering with HA enabled.   Same result.  Manager do see
>>> the
>>> host as disconnected and powered off but take no action.   I certainly
>>> miss
>>> something here.  Please help !
>>>
>>> Regards,
>>>
>>> Jean-Francois
>>>
>>
>>
>

Re: Upgrading to XenServer 7.x

[Pierre-Luc Dion]

Hi  Rafael,

you would see something like this in management-server.log:

Catch Exception: class com.cloud.utils.exception.CloudRuntimeException due
to com.cloud.utils.exception.CloudRuntimeException: There are 2 SRs with
name XenServer Tools
com.cloud.utils.exception.CloudRuntimeException: There are 2 SRs with name
XenServer Tools
at
com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.createPatchVbd(CitrixResourceBase.java:1065)
at
com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixStartCommandWrapper.execute(CitrixStartCommandWrapper.java:98)
at
com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixStartCommandWrapper.execute(CitrixStartCommandWrapper.java:53)
at
com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)

It does not prevent to create VMs but fail to create new VR.



On Tue, Dec 26, 2017 at 2:24 PM, Rafael Weingärtner <
rafaelweingart...@gmail.com> wrote:

> yes, those are the log entries I was talking about.
>
> On Tue, Dec 26, 2017 at 5:18 PM, Pierre-Luc Dion 
> wrote:
>
> > VR creation was failing with error logs in cloudstack-management.log.  I
> > don't have logs details in hand, is this what you are looking for ?
> >
> > I have not created a bug in Jira because it seams to be a  post upgrade
> > XenServer issue more than CloudStack at the moment, and is easy to
> resolve.
> >
> >
> > *Pierre-Luc DION*
> > Architecte de Solution Cloud | Cloud Solutions Architect
> > t 855.652.5683
> >
> > *CloudOps* Votre partenaire infonuagique* | *Cloud Solutions Experts
> > 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
> > w cloudops.com *|* tw @CloudOps_
> >
> > On Tue, Dec 26, 2017 at 11:48 AM, Rafael Weingärtner <
> > rafaelweingart...@gmail.com> wrote:
> >
> > > Did you see any exception regarding this issue to share with us?
> > >
> > > On Tue, Dec 26, 2017 at 2:12 PM, Pierre-Luc Dion 
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > Just as FYI, we recently upgrade from 6.5 to xs 7.1, so far everthing
> > is
> > > > good. Then we add some hosts to Pool or reinstall some XenServer to
> > have
> > > > proper new filesystem on dom0 which have more disk space for
> /var/log!
> > > >  then we ran into the situation where CloudStack fail to create
> Virtual
> > > > Router in a XenServer cluster. Turns out that for some unknown
> reason,
> > > > adding a fresh installed xenserver to a cluster can create new SR and
> > VDI
> > > > for xs-tools iso, this break cloudstack VR creation for some reason.
> So
> > > the
> > > > easy fix is to forget non-shared SR containing  xs-tools VDI.
> > > >
> > > > Basically, if "xe vdi-list is-tools-iso=true" return more than one
> iso,
> > > > CloudStack should fail to create Virtual-Router.
> > > >
> > >
> > >
> > >
> > > --
> > > Rafael Weingärtner
> > >
> >
>
>
>
> --
> Rafael Weingärtner
>