[ https://issues.apache.org/jira/browse/CLOUDSTACK-5730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Harikrishna Patnala updated CLOUDSTACK-5730: -------------------------------------------- Status: Ready To Review (was: In Progress) > [Automation]DirectAgentAttache's send command fails due to > ArrayIndexOutOfBoundsException; Unable to cleanup hypervisor's datastructures > as part of Delete Account Job > ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- > > Key: CLOUDSTACK-5730 > URL: https://issues.apache.org/jira/browse/CLOUDSTACK-5730 > Project: CloudStack > Issue Type: Bug > Security Level: Public(Anyone can view this level - this is the > default.) > Components: Management Server > Affects Versions: 4.3.0 > Reporter: Chandan Purushothama > Assignee: Harikrishna Patnala > Priority: Critical > Fix For: 4.3.0 > > > ======================================================= > ArrayIndexOutOfBoundsException during DeleteAccount Job Execution: > ======================================================= > 2014-01-03 07:39:16,752 DEBUG [c.c.v.VirtualMachineManagerImpl] > (Job-Executor-72:ctx-d82a5609 ctx-1472cd68) Destroying vm > VM[User|QA-88cfa2c8-e8ef-4133-8cf1-002ea3cb0c7b] > 2014-01-03 07:39:16,752 DEBUG [c.c.v.VirtualMachineManagerImpl] > (Job-Executor-72:ctx-d82a5609 ctx-1472cd68) Cleaning up NICS > 2014-01-03 07:39:16,752 DEBUG [o.a.c.e.o.NetworkOrchestrator] > (Job-Executor-72:ctx-d82a5609 ctx-1472cd68) Cleaning network for vm: 44 > 2014-01-03 07:39:16,763 DEBUG [c.c.n.NetworkModelImpl] > (Job-Executor-72:ctx-d82a5609 ctx-1472cd68) Service SecurityGroup is not > supported in the network id=229 > 2014-01-03 07:39:16,780 DEBUG [o.a.c.e.o.NetworkOrchestrator] > (Job-Executor-72:ctx-d82a5609 ctx-1472cd68) Removed nic id=81 > 2014-01-03 07:39:16,782 DEBUG [o.a.c.e.o.NetworkOrchestrator] > (Job-Executor-72:ctx-d82a5609 ctx-1472cd68) Revoving nic secondary ip entry > ... > 2014-01-03 07:39:16,783 DEBUG [c.c.v.VirtualMachineManagerImpl] > (Job-Executor-72:ctx-d82a5609 ctx-1472cd68) Cleaning up hypervisor data > structures (ex. SRs in XenServer) for managed storage > 2014-01-03 07:39:16,796 DEBUG [c.c.a.ApiServlet] > (catalina-exec-5:ctx-f044cb4e) ===START=== 10.147.38.149 -- GET > signature=shTK1s%2FnQv84C1gwQYEUo%2F5WFW4%3D&apiKey=SyITk_W_BpbLs73aYJqNRBu9Sir1d7k6hqoKVUmFCMSVG-vshSHHKOdj_nVVTOQHqyvfR3ZSXhDzL2QUquSdJA&command=queryAsyncJobResult&response=json&jobid=fb908e1c-cdf8-4ae9-8c0d-09116061c3f7 > 2014-01-03 07:39:16,800 DEBUG [c.c.a.t.Request] (Job-Executor-72:ctx-d82a5609 > ctx-1472cd68) Seq 1-678298558: Sending { Cmd , MgmtId: 6631563722783, via: > 1(Rack1Pod1Host15), Ver: v1, Flags: 100011, [] } > 2014-01-03 07:39:16,801 DEBUG [c.c.a.t.Request] (Job-Executor-72:ctx-d82a5609 > ctx-1472cd68) Seq 1-678298558: Executing: { Cmd , MgmtId: 6631563722783, > via: 1(Rack1Pod1Host15), Ver: v1, Flags: 100011, [] } > 2014-01-03 07:39:16,801 WARN [c.c.a.m.AgentAttache] > (Job-Executor-72:ctx-d82a5609 ctx-1472cd68) Seq 1-678298558: Unable to send > due to > java.lang.ArrayIndexOutOfBoundsException: 0 > at > com.cloud.agent.manager.DirectAgentAttache.send(DirectAgentAttache.java:100) > at com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:374) > at com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:398) > at > com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:394) > at > com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:920) > at > com.cloud.vm.VirtualMachineManagerImpl.advanceExpunge(VirtualMachineManagerImpl.java:497) > at > com.cloud.vm.VirtualMachineManagerImpl.advanceExpunge(VirtualMachineManagerImpl.java:448) > at com.cloud.vm.UserVmManagerImpl.expunge(UserVmManagerImpl.java:1704) > at sun.reflect.GeneratedMethodAccessor429.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:616) > at > org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) > at > org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) > at > org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) > at $Proxy169.expunge(Unknown Source) > at > com.cloud.user.AccountManagerImpl.cleanupAccount(AccountManagerImpl.java:629) > at > com.cloud.user.AccountManagerImpl.deleteAccount(AccountManagerImpl.java:561) > at > com.cloud.user.AccountManagerImpl.deleteUserAccount(AccountManagerImpl.java:1308) > at sun.reflect.GeneratedMethodAccessor478.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:616) > at > org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) > at > com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:50) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161) > at > org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) > at > org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) > at $Proxy82.deleteUserAccount(Unknown Source) > at > org.apache.cloudstack.region.RegionManagerImpl.deleteUserAccount(RegionManagerImpl.java:193) > at > org.apache.cloudstack.region.RegionServiceImpl.deleteUserAccount(RegionServiceImpl.java:118) > at > org.apache.cloudstack.api.command.admin.account.DeleteAccountCmd.execute(DeleteAccountCmd.java:101) > at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:161) > at > com.cloud.api.ApiAsyncJobDispatcher.runJobInContext(ApiAsyncJobDispatcher.java:109) > at > com.cloud.api.ApiAsyncJobDispatcher$1.run(ApiAsyncJobDispatcher.java:66) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > at > com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:63) > at > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:522) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:679) > 2014-01-03 07:39:16,836 DEBUG [c.c.a.m.AgentAttache] > (Job-Executor-72:ctx-d82a5609 ctx-1472cd68) Seq 1-678298558: Cancelling. > 2014-01-03 07:39:16,837 WARN [c.c.v.UserVmManagerImpl] > (Job-Executor-72:ctx-d82a5609 ctx-1472cd68) Unable to expunge > VM[User|QA-88cfa2c8-e8ef-4133-8cf1-002ea3cb0c7b] > com.cloud.exception.AgentUnavailableException: Resource [Host:1] is > unreachable: Host 1: Problem due to other exception 0 > at com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:389) > at com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:398) > at > com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:394) > at > com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:920) > at > com.cloud.vm.VirtualMachineManagerImpl.advanceExpunge(VirtualMachineManagerImpl.java:497) > at > com.cloud.vm.VirtualMachineManagerImpl.advanceExpunge(VirtualMachineManagerImpl.java:448) > at com.cloud.vm.UserVmManagerImpl.expunge(UserVmManagerImpl.java:1704) > at sun.reflect.GeneratedMethodAccessor429.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:616) > at > org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) > at > org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) > at > org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) > at $Proxy169.expunge(Unknown Source) > at > com.cloud.user.AccountManagerImpl.cleanupAccount(AccountManagerImpl.java:629) > at > com.cloud.user.AccountManagerImpl.deleteAccount(AccountManagerImpl.java:561) > at > com.cloud.user.AccountManagerImpl.deleteUserAccount(AccountManagerImpl.java:1308) > at sun.reflect.GeneratedMethodAccessor478.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:616) > at > org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) > at > com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:50) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161) > at > org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) > at > org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) > at $Proxy82.deleteUserAccount(Unknown Source) > at > org.apache.cloudstack.region.RegionManagerImpl.deleteUserAccount(RegionManagerImpl.java:193) > at > org.apache.cloudstack.region.RegionServiceImpl.deleteUserAccount(RegionServiceImpl.java:118) > at > org.apache.cloudstack.api.command.admin.account.DeleteAccountCmd.execute(DeleteAccountCmd.java:101) > at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:161) > at > com.cloud.api.ApiAsyncJobDispatcher.runJobInContext(ApiAsyncJobDispatcher.java:109) > at > com.cloud.api.ApiAsyncJobDispatcher$1.run(ApiAsyncJobDispatcher.java:66) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > at > com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:63) > at > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:522) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:679) > 2014-01-03 07:39:16,839 ERROR [c.c.u.AccountManagerImpl] > (Job-Executor-72:ctx-d82a5609 ctx-1472cd68) Unable to expunge vm: 44 > 2014-01-03 07:39:16,869 INFO [c.c.u.AccountManagerImpl] > (Job-Executor-72:ctx-d82a5609 ctx-1472cd68) deleteAccount: Deleted 1 network > groups for account 26 > 2014-01-03 07:39:16,870 INFO [c.c.u.AccountManagerImpl] > (Job-Executor-72:ctx-d82a5609 ctx-1472cd68) deleteAccount: Deleted 0 affinity > groups for account 26 > 2014-01-03 07:39:16,871 DEBUG [c.c.u.AccountManagerImpl] > (Job-Executor-72:ctx-d82a5609 ctx-1472cd68) Deleting networks for account 26 > 2014-01-03 07:39:16,884 DEBUG [c.c.a.ApiServlet] > (catalina-exec-5:ctx-f044cb4e ctx-e5d3f936 ctx-efc2b479) ===END=== > 10.147.38.149 -- GET > signature=shTK1s%2FnQv84C1gwQYEUo%2F5WFW4%3D&apiKey=SyITk_W_BpbLs73aYJqNRBu9Sir1d7k6hqoKVUmFCMSVG-vshSHHKOdj_nVVTOQHqyvfR3ZSXhDzL2QUquSdJA&command=queryAsyncJobResult&response=json&jobid=fb908e1c-cdf8-4ae9-8c0d-09116061c3f7 > 2014-01-03 07:39:16,894 WARN [o.a.c.e.o.NetworkOrchestrator] > (Job-Executor-72:ctx-d82a5609 ctx-1472cd68) Can't delete the network, not all > user vms are expunged. Vm VM[User|QA-03d9e9fa-081a-4175-9bf2-2508034b6e17] is > in Expunging state > 2014-01-03 07:39:16,894 WARN [c.c.u.AccountManagerImpl] > (Job-Executor-72:ctx-d82a5609 ctx-1472cd68) Unable to destroy network > Ntwk[229|Guest|8] as a part of account id=26 cleanup. > 2014-01-03 07:39:16,895 DEBUG [c.c.u.AccountManagerImpl] > (Job-Executor-72:ctx-d82a5609 ctx-1472cd68) Deleting vpcs for account 26 > 2014-01-03 07:39:16,897 DEBUG [c.c.u.AccountManagerImpl] > (Job-Executor-72:ctx-d82a5609 ctx-1472cd68) Deleting site-to-site VPN > customer gateways for account 26 > 2014-01-03 07:39:16,908 INFO [c.c.u.AccountManagerImpl] > (Job-Executor-72:ctx-d82a5609 ctx-1472cd68) deleteAccount: Released 0 > dedicated guest vlan ranges from account 26 > 2014-01-03 07:39:17,034 INFO [c.c.u.AccountManagerImpl] > (Job-Executor-72:ctx-d82a5609 ctx-1472cd68) Cleanup for account 26 is needed. > 2014-01-03 07:39:17,048 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > (Job-Executor-72:ctx-d82a5609 ctx-1472cd68) Complete async job-359, > jobStatus: SUCCEEDED, resultCode: 0, result: > org.apache.cloudstack.api.response.SuccessResponse/null/{"success":true} > 2014-01-03 07:39:17,056 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > (Job-Executor-72:ctx-d82a5609) Done executing > org.apache.cloudstack.api.command.admin.account.DeleteAccountCmd for job-359 -- This message was sent by Atlassian JIRA (v6.1.5#6160)