[
https://issues.apache.org/jira/browse/CLOUDSTACK-1339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13586870#comment-13586870
]
Rohit Yadav commented on CLOUDSTACK-1339:
-----------------------------------------
Kelven I just fixed CLOUDSTACK-1244, some stats and test reports to share:
Memory usage was 417 MB, after deploying a basic zone it was 377MB.
It think the agents which run in systemvms are responding slowly, I see:
INFO [agent.manager.AgentMonitor] (Thread-13:) Found the following agents
behind on ping: [1, 3]
INFO [agent.manager.AgentManagerImpl] (AgentTaskPool-1:) Investigating why
host 1 has disconnected with event PingTimeout
INFO [agent.manager.AgentManagerImpl] (AgentTaskPool-2:) Investigating why
host 3 has disconnected with event PingTimeout
INFO [agent.manager.AgentManagerImpl] (AgentTaskPool-2:) The state determined
is Up
INFO [agent.manager.AgentManagerImpl] (AgentTaskPool-2:) Agent is determined
to be up and running
Smoke test results:
$ nosetests --with-marvin --marvin-config ../../tools/devcloud/devcloud.cfg
--load -a tags="devcloud" -w smoke
[12:02:06]
test_01_nic (integration.smoke.test_nic.TestDeployVM) ... ok
Test Deploy Virtual Machine ... ok
Test Stop Virtual Machine ... ERROR
Test Start Virtual Machine ... ERROR
Test Reboot Virtual Machine ... ok
Test destroy Virtual Machine ... ok
Test recover Virtual Machine ... ERROR
Test destroy(expunge) Virtual Machine ...
Error log from mgmt server:
WARN [xen.resource.CitrixResourceBase] (DirectAgent-29:) Host 192.168.56.10
cannot do bridge firewalling
WARN [xen.resource.CitrixResourceBase] (DirectAgent-43:) Host 192.168.56.10
cannot do bridge firewalling
INFO [agent.manager.AgentMonitor] (Thread-13:) Found the following agents
behind on ping: [1, 3]
INFO [agent.manager.AgentManagerImpl] (AgentTaskPool-4:) Investigating why
host 1 has disconnected with event PingTimeout
INFO [agent.manager.AgentManagerImpl] (AgentTaskPool-3:) Investigating why
host 3 has disconnected with event PingTimeout
INFO [agent.manager.AgentManagerImpl] (AgentTaskPool-3:) The state determined
is Up
INFO [agent.manager.AgentManagerImpl] (AgentTaskPool-3:) Agent is determined
to be up and running
INFO [agent.manager.AgentManagerImpl] (AgentTaskPool-4:) The state determined
is Up
INFO [agent.manager.AgentManagerImpl] (AgentTaskPool-4:) Agent is determined
to be up and running
WARN [network.element.VpcVirtualRouterElement] (Job-Executor-14:job-14)
Network Ntwk[204|Guest|6] is not associated with any VPC
WARN [xen.resource.CitrixResourceBase] (DirectAgent-58:) Host 192.168.56.10
cannot do bridge firewalling
INFO [network.security.SecurityGroupManagerImpl] (Job-Executor-15:job-15)
Disassociated 1 network groups from uservm 20
INFO [cloud.user.AccountManagerImpl] (Job-Executor-15:job-15) deleteAccount:
Deleted 1 network groups for account 5
INFO [cloud.user.AccountManagerImpl] (Job-Executor-15:job-15) Cleanup for
account 5 is not needed.
INFO [agent.manager.AgentMonitor] (Thread-13:) Found the following agents
behind on ping: [1, 3]
INFO [agent.manager.AgentManagerImpl] (AgentTaskPool-5:) Investigating why
host 1 has disconnected with event PingTimeout
INFO [agent.manager.AgentManagerImpl] (AgentTaskPool-6:) Investigating why
host 3 has disconnected with event PingTimeout
INFO [agent.manager.AgentManagerImpl] (AgentTaskPool-6:) The state determined
is Up
INFO [agent.manager.AgentManagerImpl] (AgentTaskPool-6:) Agent is determined
to be up and running
INFO [agent.manager.AgentManagerImpl] (AgentTaskPool-5:) The state determined
is Up
INFO [agent.manager.AgentManagerImpl] (AgentTaskPool-5:) Agent is determined
to be up and running
WARN [network.element.VpcVirtualRouterElement] (Job-Executor-16:job-16)
Network Ntwk[204|Guest|6] is not associated with any VPC
WARN [xen.resource.CitrixResourceBase] (DirectAgent-65:) Host 192.168.56.10
cannot do bridge firewalling
WARN [network.element.VpcVirtualRouterElement] (Job-Executor-17:job-17)
Network Ntwk[204|Guest|6] is not associated with any VPC
WARN [xen.resource.CitrixResourceBase] (DirectAgent-8:) Ignoring VM i-6-28-VM
in transition state starting.
WARN [xen.resource.CitrixResourceBase] (DirectAgent-24:) Host 192.168.56.10
cannot do bridge firewalling
WARN [network.element.VpcVirtualRouterElement] (Job-Executor-18:job-18)
Network Ntwk[204|Guest|6] is not associated with any VPC
WARN [xen.resource.CitrixResourceBase] (DirectAgent-75:) Host 192.168.56.10
cannot do bridge firewalling
INFO [cloud.vm.UserVmManagerImpl] (UserVm-Scavenger-1:) Found 2 vms to expunge.
INFO [agent.manager.AgentMonitor] (Thread-13:) Found the following agents
behind on ping: [1, 3]
INFO [agent.manager.AgentManagerImpl] (AgentTaskPool-7:) Investigating why
host 1 has disconnected with event PingTimeout
INFO [agent.manager.AgentManagerImpl] (AgentTaskPool-8:) Investigating why
host 3 has disconnected with event PingTimeout
INFO [agent.manager.AgentManagerImpl] (AgentTaskPool-8:) The state determined
is Up
INFO [agent.manager.AgentManagerImpl] (AgentTaskPool-8:) Agent is determined
to be up and running
INFO [agent.manager.AgentManagerImpl] (AgentTaskPool-7:) The state determined
is Up
INFO [agent.manager.AgentManagerImpl] (AgentTaskPool-7:) Agent is determined
to be up and running
INFO [network.security.SecurityGroupManagerImpl] (UserVm-Scavenger-1:)
Disassociated 1 network groups from uservm 13
INFO [network.security.SecurityGroupManagerImpl] (UserVm-Scavenger-1:)
Disassociated 1 network groups from uservm 14
ERROR [cloud.api.ApiServer] (1242635335@qtp-1715433050-13:) unhandled exception
executing api command: stopVirtualMachine
java.lang.NullPointerException
at com.cloud.dao.EntityManagerImpl.findById(EntityManagerImpl.java:47)
at sun.reflect.GeneratedMethodAccessor184.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:319)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at
org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80)
at
com.cloud.utils.db.TransactionContextBuilder.AroundAnyMethod(TransactionContextBuilder.java:37)
at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621)
at
org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610)
at
org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:90)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
at $Proxy144.findById(Unknown Source)
at com.cloud.api.ApiDispatcher.processParameters(ApiDispatcher.java:283)
at com.cloud.api.ApiServer.queueCommand(ApiServer.java:450)
at com.cloud.api.ApiServer.handleRequest(ApiServer.java:347)
at sun.reflect.GeneratedMethodAccessor98.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:319)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at
org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80)
at
com.cloud.utils.db.TransactionContextBuilder.AroundAnyMethod(TransactionContextBuilder.java:37)
at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621)
at
org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610)
at
org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:90)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
at $Proxy386.handleRequest(Unknown Source)
at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:302)
at com.cloud.api.ApiServlet.doGet(ApiServlet.java:66)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)
at
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
at
org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at
org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
at
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at
org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at
org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at
org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
at
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
ERROR [cloud.async.AsyncJobManagerImpl] (Job-Executor-19:job-19) Unexpected
exception while executing org.apache.cloudstack.api.command.user.vm.StartVMCmd
com.cloud.exception.InsufficientServerCapacityException: Unable to create a
deployment for VM[User|e05bba48-296f-443e-a11c-8e3c0a8b3cd2]Scope=interface
com.cloud.dc.DataCenter; id=1
at
org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.reserveVirtualMachine(VMEntityManagerImpl.java:186)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:319)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at
org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80)
at
com.cloud.utils.db.TransactionContextBuilder.AroundAnyMethod(TransactionContextBuilder.java:37)
at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621)
at
org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610)
at
org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:90)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
at $Proxy299.reserveVirtualMachine(Unknown Source)
at
org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.reserve(VirtualMachineEntityImpl.java:197)
at
com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3859)
at
com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2573)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:319)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at
org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80)
at
com.cloud.event.ActionEventInterceptor.AroundAnyMethod(ActionEventInterceptor.java:53)
at sun.reflect.GeneratedMethodAccessor147.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621)
at
org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610)
at
org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
at
org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80)
at
com.cloud.utils.db.TransactionContextBuilder.AroundAnyMethod(TransactionContextBuilder.java:37)
at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621)
at
org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610)
at
org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:90)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
at $Proxy265.startVirtualMachine(Unknown Source)
at
org.apache.cloudstack.api.command.user.vm.StartVMCmd.execute(StartVMCmd.java:120)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:161)
at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at
org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint(Cglib2AopProxy.java:689)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at
org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80)
at
com.cloud.utils.db.TransactionContextBuilder.AroundAnyMethod(TransactionContextBuilder.java:37)
at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621)
at
org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610)
at
org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:90)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at
org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:622)
at
com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:437)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680)
ERROR [cloud.api.ApiServer] (1242635335@qtp-1715433050-13:) unhandled exception
executing api command: recoverVirtualMachine
java.lang.NullPointerException
at com.cloud.dao.EntityManagerImpl.findById(EntityManagerImpl.java:47)
at sun.reflect.GeneratedMethodAccessor184.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:319)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at
org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80)
at
com.cloud.utils.db.TransactionContextBuilder.AroundAnyMethod(TransactionContextBuilder.java:37)
at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621)
at
org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610)
at
org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:90)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
at $Proxy144.findById(Unknown Source)
at
org.apache.cloudstack.api.command.admin.vm.RecoverVMCmd.getEntityOwnerId(RecoverVMCmd.java:65)
at com.cloud.api.ApiDispatcher.doAccessChecks(ApiDispatcher.java:111)
at com.cloud.api.ApiDispatcher.processParameters(ApiDispatcher.java:318)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:133)
at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at
org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint(Cglib2AopProxy.java:689)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at
org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80)
at
com.cloud.utils.db.TransactionContextBuilder.AroundAnyMethod(TransactionContextBuilder.java:37)
at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621)
at
org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610)
at
org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:90)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at
org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:622)
at com.cloud.api.ApiServer.queueCommand(ApiServer.java:498)
at com.cloud.api.ApiServer.handleRequest(ApiServer.java:347)
at sun.reflect.GeneratedMethodAccessor98.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:319)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at
org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80)
at
com.cloud.utils.db.TransactionContextBuilder.AroundAnyMethod(TransactionContextBuilder.java:37)
at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621)
at
org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610)
at
org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:90)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
at $Proxy386.handleRequest(Unknown Source)
at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:302)
at com.cloud.api.ApiServlet.doGet(ApiServlet.java:66)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)
at
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
at
org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at
org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
at
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at
org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at
org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at
org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
at
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
INFO [agent.manager.AgentMonitor] (Thread-13:) Found the following agents
behind on ping: [1, 3]
INFO [agent.manager.AgentManagerImpl] (AgentTaskPool-9:) Investigating why
host 1 has disconnected with event PingTimeout
INFO [agent.manager.AgentManagerImpl] (AgentTaskPool-10:) Investigating why
host 3 has disconnected with event PingTimeout
INFO [agent.manager.AgentManagerImpl] (AgentTaskPool-10:) The state determined
is Up
INFO [agent.manager.AgentManagerImpl] (AgentTaskPool-10:) Agent is determined
to be up and running
INFO [agent.manager.AgentManagerImpl] (AgentTaskPool-9:) The state determined
is Up
INFO [agent.manager.AgentManagerImpl] (AgentTaskPool-9:) Agent is determined
to be up and running
INFO [agent.manager.AgentMonitor] (Thread-13:) Found the following agents
behind on ping: [1, 3]
> ASF 4.1: Management server becomes unresponsive
> -----------------------------------------------
>
> Key: CLOUDSTACK-1339
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-1339
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Components: Management Server
> Affects Versions: 4.1.0
> Reporter: Srikanteswararao Talluri
> Assignee: Kelven Yang
> Priority: Blocker
> Fix For: 4.1.0
>
> Attachments: MS_log.txt
>
>
> management server becomes unresponsive after running for a while and after
> some time the process shuts down.
> This is blocking the testing
> logs can be found @ https://citrix.sharefile.com/d/sa684d7a73fb45ba8
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
