[ https://issues.apache.org/jira/browse/CLOUDSTACK-7424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jayapal Reddy updated CLOUDSTACK-7424: -------------------------------------- Assignee: Chandan Purushothama (was: Jayapal Reddy) > [Automation] Failed to restart network > -------------------------------------- > > Key: CLOUDSTACK-7424 > URL: https://issues.apache.org/jira/browse/CLOUDSTACK-7424 > Project: CloudStack > Issue Type: Bug > Security Level: Public(Anyone can view this level - this is the > default.) > Components: Automation, Test > Affects Versions: 4.5.0 > Reporter: Chandan Purushothama > Assignee: Chandan Purushothama > Priority: Critical > Fix For: 4.5.0 > > > Error Executing the test case > *integration.component.test_multiple_ips_per_nic.TestVmNetworkOperations.test_network_restart_cleanup_false_3_VPC* > *Error Message* > Job failed: {jobprocstatus : 0, created : u'2014-08-24T07:31:03+0000', cmd : > u'org.apache.cloudstack.api.command.user.network.RestartNetworkCmd', userid : > u'77df13d2-2b4d-11e4-89bd-1e5d0e053e75', jobstatus : 2, jobid : > u'01faacdc-294b-4209-a855-2dc8ea0aa2b4', jobresultcode : 530, jobresulttype : > u'object', jobresult : {errorcode : 530, errortext : u'Failed to restart > network'}, accountid : u'77df055e-2b4d-11e4-89bd-1e5d0e053e75'} Logs > available at > http://xenrt.hq.xensource.com/control/queue.cgi?action=testlogs&id=804076&phase=Parallel&test=multiple_ips_per_n > *Stacktrace* > File "/usr/lib/python2.7/unittest/case.py", line 332, in run > testMethod() > File "/usr/local/lib/python2.7/dist-packages/ddt.py", line 114, in wrapper > return func(self, *args, **kwargs) > File > "/root/cloudstack/test/integration/component/test_multiple_ips_per_nic.py", > line 1186, in test_network_restart_cleanup_false > network.restart(self.apiclient, cleanup=False) > File "/usr/local/lib/python2.7/dist-packages/marvin/lib/base.py", line > 2626, in restart > return(apiclient.restartNetwork(cmd)) > File > "/usr/local/lib/python2.7/dist-packages/marvin/cloudstackAPI/cloudstackAPIClient.py", > line 1899, in restartNetwork > response = self.connection.marvinRequest(command, response_type=response, > method=method) > File > "/usr/local/lib/python2.7/dist-packages/marvin/cloudstackConnection.py", line > 379, in marvinRequest > raise e > 'Job failed: {jobprocstatus : 0, created : u\'2014-08-24T07:31:03+0000\', cmd > : u\'org.apache.cloudstack.api.command.user.network.RestartNetworkCmd\', > userid : u\'77df13d2-2b4d-11e4-89bd-1e5d0e053e75\', jobstatus : 2, jobid : > u\'01faacdc-294b-4209-a855-2dc8ea0aa2b4\', jobresultcode : 530, jobresulttype > : u\'object\', jobresult : {errorcode : 530, errortext : u\'Failed to restart > network\'}, accountid : u\'77df055e-2b4d-11e4-89bd-1e5d0e053e75\'}\n > Logs available at > http://xenrt.hq.xensource.com/control/queue.cgi?action=testlogs&id=804076&phase=Parallel&test=multiple_ips_per_n > > ============================== > *Error in the Management Server Log*: > ============================== > {code} > 2014-08-24 07:31:03,602 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] > (API-Job-Executor-82:ctx-6738c61c job-1529 ctx-3eaba8e6) Router r-229-VM is > in Stopping, so not sending apply static nat rules commands to the backend > 2014-08-24 07:31:03,621 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] > (API-Job-Executor-82:ctx-6738c61c job-1529 ctx-3eaba8e6) Router r-229-VM is > in Stopping, so not sending apply vpc ip association commands to the backend > 2014-08-24 07:31:03,638 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] > (API-Job-Executor-82:ctx-6738c61c job-1529 ctx-3eaba8e6) Router r-229-VM is > in Stopping, so not sending apply vpc ip association commands to the backend > 2014-08-24 07:31:03,640 DEBUG [o.a.c.e.o.NetworkOrchestrator] > (API-Job-Executor-82:ctx-6738c61c job-1529 ctx-3eaba8e6) Sending network > shutdown to VpcVirtualRouter > 2014-08-24 07:31:03,653 DEBUG [c.c.n.NetworkModelImpl] > (API-Job-Executor-82:ctx-6738c61c job-1529 ctx-3eaba8e6) Service > SecurityGroup is not supported in the network id=314 > 2014-08-24 07:31:03,654 DEBUG [c.c.n.r.VpcVirtualNetworkApplianceManagerImpl] > (API-Job-Executor-82:ctx-6738c61c job-1529 ctx-3eaba8e6) Router r-229-VM is > in Stopping, so not sending setup guest network command to the backend > 2014-08-24 07:31:03,663 DEBUG [c.c.v.VirtualMachineManagerImpl] > (API-Job-Executor-82:ctx-6738c61c job-1529 ctx-3eaba8e6) Lock is acquired for > nic id 476 as a part of remove vm VM[DomainRouter|r-229-VM] from network > Ntwk[314|Guest|37] > 2014-08-24 07:31:03,667 DEBUG [c.c.n.NetworkModelImpl] > (API-Job-Executor-82:ctx-6738c61c job-1529 ctx-3eaba8e6) Service > SecurityGroup is not supported in the network id=314 > 2014-08-24 07:31:03,668 WARN [c.c.v.VirtualMachineManagerImpl] > (API-Job-Executor-82:ctx-6738c61c job-1529 ctx-3eaba8e6) Unable to remove vm > VM[DomainRouter|r-229-VM] from network Ntwk[314|Guest|37] > 2014-08-24 07:31:03,669 DEBUG [c.c.v.VirtualMachineManagerImpl] > (API-Job-Executor-82:ctx-6738c61c job-1529 ctx-3eaba8e6) Lock is released for > nic id 476 as a part of remove vm VM[DomainRouter|r-229-VM] from network > Ntwk[314|Guest|37] > 2014-08-24 07:31:03,671 WARN [o.a.c.e.o.NetworkOrchestrator] > (API-Job-Executor-82:ctx-6738c61c job-1529 ctx-3eaba8e6) Unable to complete > shutdown of the network elements due to element: VpcVirtualRouter > com.cloud.exception.ResourceUnavailableException: Resource [DataCenter:1] is > unreachable: Unable to remove vm VM[DomainRouter|r-229-VM] from network, is > not in the right state > at > com.cloud.vm.VirtualMachineManagerImpl.orchestrateRemoveVmFromNetwork(VirtualMachineManagerImpl.java:3079) > at > com.cloud.vm.VirtualMachineManagerImpl.removeVmFromNetwork(VirtualMachineManagerImpl.java:3009) > at > com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.removeVpcRouterFromGuestNetwork(VpcVirtualNetworkApplianceManagerImpl.java:321) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at > org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) > at > org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) > at > org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) > at $Proxy190.removeVpcRouterFromGuestNetwork(Unknown Source) > at > com.cloud.network.element.VpcVirtualRouterElement.shutdown(VpcVirtualRouterElement.java:261) > at > org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.shutdownNetworkElementsAndResources(NetworkOrchestrator.java:2161) > at > org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.restartNetwork(NetworkOrchestrator.java:2468) > at > com.cloud.network.NetworkServiceImpl.restartNetwork(NetworkServiceImpl.java:1874) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at > org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) > at > org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:106) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161) > at > com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161) > at > org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) > at > org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) > at $Proxy157.restartNetwork(Unknown Source) > at > org.apache.cloudstack.api.command.user.network.RestartNetworkCmd.execute(RestartNetworkCmd.java:95) > at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:141) > at > com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108) > at > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) > at > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:722) > 2014-08-24 07:31:03,673 DEBUG [o.a.c.e.o.NetworkOrchestrator] > (API-Job-Executor-82:ctx-6738c61c job-1529 ctx-3eaba8e6) Failed to shutdown > the network elements and resources as a part of network restart: Implemented > 2014-08-24 07:31:03,673 DEBUG [o.a.c.e.o.NetworkOrchestrator] > (API-Job-Executor-82:ctx-6738c61c job-1529 ctx-3eaba8e6) Marking network > Ntwk[314|Guest|37] with restartRequired=true > 2014-08-24 07:31:03,675 WARN [c.c.n.NetworkServiceImpl] > (API-Job-Executor-82:ctx-6738c61c job-1529 ctx-3eaba8e6) Network id=314 > failed to restart. > 2014-08-24 07:31:03,680 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > (API-Job-Executor-82:ctx-6738c61c job-1529) Complete async job-1529, > jobStatus: FAILED, resultCode: 530, result: > org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Failed > to restart network"} > {code} -- This message was sent by Atlassian JIRA (v6.2#6252)