Re: ALL Hosts Stuck in Maintenance

[Daan Hoogland]

Steven,

I had a look at your stacktrace. It seems a vm is marked to be migrated
back during maintenance mode cancel and it doesn't exist. Did you destroy
any of the vms on the host while it was in maintenance?

There is a number of jobs for migration each of which pertains to a vm. For
one of them the vm does not exist. Doesn't exist in this case means that it
is not in the database. This is a bug, obviously. I can't be sure what
happened during the maintenance mode and thus not what the nature of the
bug exactly is. It would be nice if you could to investigate this.

Daan


On Fri, Nov 21, 2014 at 6:35 PM, Steve Searles  wrote:

>  Below is the full NPE from the catalina.out if that would help.  This is
> the same NPE for all hypervisor types.
>
>  INFO  [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-10:ctx-208629c7
> job-12679) Add job-12679 into job monitoring
> ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-10:ctx-208629c7
> job-12679) Unexpected exception while executing
> org.apache.cloudstack.api.command.admin.host.CancelMaintenanceCmd
> java.lang.NullPointerException
> at
> com.cloud.resource.ResourceManagerImpl.doCancelMaintenance(ResourceManagerImpl.java:2083)
> at
> com.cloud.resource.ResourceManagerImpl.cancelMaintenance(ResourceManagerImpl.java:2140)
> at
> com.cloud.resource.ResourceManagerImpl.cancelMaintenance(ResourceManagerImpl.java:1127)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
> at
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
> at
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
> at com.sun.proxy.$Proxy148.cancelMaintenance(Unknown Source)
> at
> org.apache.cloudstack.api.command.admin.host.CancelMaintenanceCmd.execute(CancelMaintenanceCmd.java:102)
> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:141)
> at
> com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> INFO  [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-10:ctx-208629c7
> job-12679) Remove job-12679 from job monitoring
>
>
>Steven Searles, CTO | ssear...@zimcom.net
>  Zimcom Internet Solutions  | www.zimcom.net
>  O: 513.231.9500  |  D: 513.233.4130
>
>
>  On Nov 21, 2014, at 12:31 PM, Steve Searles  wrote:
>
>  In my case the agent is connected fine.  I have tried even restarting
> the hosts.  I have the agent log in debug for the KVM server and am not
> seeing the mgmt server even trying to execute anything when coming out of
> maintenance.  I just get the NPE immediately in the management server
> logs.
>
>   Steven Searles, CTO | ssear...@zimcom.net
>  Zimcom Internet Solutions  | www.zimcom.net
>  O: 513.231.9500  |  D: 513.233.4130
>
> 
>
>  On Nov 21, 2014, at 10:54 AM, Andrija Panic 
> wrote:
>
> Experienced similar behaviour, for kvm - seems like restarting libvirtd and
> give it some time to settle, and than agent connects ... on its own...
>
> Sent from Google Nexus 4
> On Nov 21, 2014 4:43 PM, "Steve Searles"  wrote:
>
> For some reason it is affecting every host.  VMware, KVM, and XenServer.
> No hosts will come out of maintenance same NPE for all.  Storage will

Re: VMs not Expunging

[Fernandez, H.J.]

Hi Brent,

It happened to me and just simply forced the expunge operation. You could also 
handle it via API adding a parameter “expunge=True” which will force the 
expunge operation to these machines.

Hector

Linkedin: 
linkedin.com/in/hector2fernandez
Twitter:  twitter.com/hectorj2f





On Nov 22, 2014, at 6:40 AM, Alireza Eskandari 
mailto:astro.alir...@yahoo.com.INVALID>> wrote:

Hi Brent
I suggest you to change state of the VM to Stopped from database.
This action enables you to try destroy VM again fron UI.

Sent from Samsung Mobile.

 Original message From: Brent Clark 
mailto:bcl...@tendrilinc.com>> 
Date:22/11/2014  00:30  (GMT+03:30) To: 
users@cloudstack.apache.org 
Subject: VMs not Expunging 
Hello, I have an issue were a number of vms are in "Expunging" state and
have been for days. The KVM host they lived on crashed and isnt coming
back. I had to force remove the host from the cluster. This left the vms in
the CloudStack Manager in a destroyed state. 24hrs later they moved to
"Expunging" state and have been there since. (~3days now)

>From within the CloudStack manger, there is no option to do anything with
them because they are in "Expunging" state.

How do I get rid of these entries?

Thanks!

--
Brent S. Clark
Senior Cloud & Infrastructure Systems Engineer

2580 55th St.  |  Boulder, Colorado 80301
www.tendrilinc.com  |  blog 




This email and any files transmitted with it are confidential and intended 
solely for the use of the individual or entity to whom they are addressed.
If you have received this email in error please notify the sender.
Please note that any views or opinions presented in this email are solely those 
of the author and do not necessarily represent those of the company.
Finally, the recipient should check this email and any attachments for the 
presence of viruses.
The company accepts no liability for any damage caused by any virus transmitted 
by this email.