Hi All, Raised the following issue on Jira to track this bug: https://issues.apache.org/jira/browse/CLOUDSTACK-9547
With regards, Prashanth On Mon, Oct 17, 2016 at 12:38 PM, Prashanth Manthena < prashanth.manth...@nuagenetworks.net> wrote: > Hi Paul, > > First of all, thank you for your time and investigation. > > I have tried with timeouts up to 180 seconds, and that didn't help either > :) > > Yes, it doesn't explain the difference between 4.7 and 4.9, because I am > using the same vcenter/vmware setup (i.e. no vmware tools installed). > > Let me know, if you want me to raise an issue on Jira for tracking this > bug. > > On Sat, Oct 15, 2016 at 8:55 PM, Paul Angus <paul.an...@shapeblue.com> > wrote: > >> I tried extending the timeout to 30 seconds, but that didn't help. >> However, installing vmware tools (not easy in our built-in templates) >> made the problem go away. With the tools installed the removal was pretty >> much instant. >> Doesn't explain the difference 4.7 and 4.9 that you are seeing though. >> >> >> Kind regards, >> >> Paul Angus >> >> paul.an...@shapeblue.com >> www.shapeblue.com >> 53 Chandos Place, Covent Garden, London WC2N 4HSUK >> @shapeblue >> >> >> >> >> -----Original Message----- >> From: Prashanth Manthena [mailto:prashanth.manth...@nuagenetworks.net] >> Sent: 14 October 2016 13:24 >> To: dev@cloudstack.apache.org >> Subject: Re: ACS 4.9 + VMware: Unable to remove one of the NICs of a >> multi-nic VM >> >> Hi Paul, >> >> Thank you for trying it out. >> >> I am only hitting this issue for guest VMs (i.e. not with VPC VRs) >> created in ACS 4.9 (i.e. not in ACS 4.7) with VMware setups. >> >> Moreover, I get the same error when I am trying to remove the NIC (i.e. >> network adapter) directly from VMware's Vcenter. >> >> There is a possible workaround for this issue from VMware on Internet, >> which doesn't work in this scenario both from CloudStack and VMware: >> https://kb.vmware.com/selfservice/microsites/search.do? >> language=en_US&cmd=displayKC&externalId=2081503 >> >> Most likely, this issue has something to do with how we deploy >> (multi-nic) guest VMs in ACS 4.9 with VMware setups. >> >> On Fri, Oct 14, 2016 at 1:00 PM, Paul Angus <paul.an...@shapeblue.com> >> wrote: >> >> > Hi Prashanth, >> > >> > I've just tried that. I get the same error - The guest operating >> > system did not respond to a hot-remove request for device ethernet1 in >> > a timely manner. >> > >> > Kind regards, >> > >> > Paul Angus >> > >> > paul.an...@shapeblue.com >> > www.shapeblue.com >> > 53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue >> > >> > >> > >> > >> > -----Original Message----- >> > From: Prashanth Manthena [mailto:prashanth.manth...@nuagenetworks.net] >> > Sent: 14 October 2016 09:21 >> > To: dev@cloudstack.apache.org >> > Subject: Re: ACS 4.9 + VMware: Unable to remove one of the NICs of a >> > multi-nic VM >> > >> > HI All, >> > >> > Does this issue ring a bell, and is anyone else hitting this issue ? >> > >> > Let me know, if it is a known issue. >> > >> > Thanking you in advance !! >> > >> > On Thu, Oct 13, 2016 at 6:25 PM, Prashanth Manthena < >> prashanth.manthena@ >> > nuagenetworks.net> wrote: >> > >> > > Hi, >> > > >> > > I am hitting the following issue on an ACS 4.9 + VMware setup (steps >> > > to >> > > reproduce): >> > > >> > > 1) Deploy a multi-nic VM (or) add a nic to a single-nic VM >> > > >> > > 2) Remove the non-default nic from the multi-nic VM, which fails with >> > > the following error/exception in the management server log: >> > > >> > > 2016-10-05 06:13:28,251 DEBUG [c.c.a.ApiServlet] >> > > (catalina-exec-14:ctx-f8dc6bd0 ctx-ee610e01) (logid:58e9cf98) >> > > ===END=== 10.31.52.95 -- GET >> > > command=queryAsyncJobResult&jobId=9ad66ce9-6e1b-4c25-bd2e-76 >> 3f4586dd86 >> > > &response=json&_=1475673245452 >> > > 2016-10-05 06:13:29,787 ERROR [c.c.h.v.r.VmwareResource] >> > (DirectAgent-302:ctx-78a58d67 10.31.56.178, job-171/job-172, cmd: >> > UnPlugNicCommand) (logid:9ad66ce9) Unexpected exception: >> > > java.lang.RuntimeException: The guest operating system did not respond >> > to a hot-remove request for device ethernet1 in a timely manner. >> > > at com.cloud.hypervisor.vmware.util.VmwareClient.waitForTask( >> > VmwareClient.java:354) >> > > at com.cloud.hypervisor.vmware.mo.VirtualMachineMO. >> > configureVm(VirtualMachineMO.java:949) >> > > at com.cloud.hypervisor.vmware.resource.VmwareResource. >> > execute(VmwareResource.java:1103) >> > > at com.cloud.hypervisor.vmware.resource.VmwareResource. >> > executeRequest(VmwareResource.java:469) >> > > at com.cloud.agent.manager.Direct >> AgentAttache$Task.runInContext( >> > DirectAgentAttache.java:315) >> > > at org.apache.cloudstack.managed.context. >> > ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) >> > > at org.apache.cloudstack.managed.context.impl. >> > DefaultManagedContext$1.call(DefaultManagedContext.java:56) >> > > at org.apache.cloudstack.managed.context.impl. >> > DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) >> > > at org.apache.cloudstack.managed.context.impl. >> > DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) >> > > at org.apache.cloudstack.managed.context. >> > ManagedContextRunnable.run(ManagedContextRunnable.java:46) >> > > at java.util.concurrent.Executors$RunnableAdapter. >> > call(Executors.java:471) >> > > at java.util.concurrent.FutureTask.run(FutureTask.java:262) >> > > at java.util.concurrent.ScheduledThreadPoolExecutor$ >> > ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) >> > > at java.util.concurrent.ScheduledThreadPoolExecutor$ >> > ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) >> > > at java.util.concurrent.ThreadPoolExecutor.runWorker( >> > ThreadPoolExecutor.java:1145) >> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run( >> > ThreadPoolExecutor.java:615) >> > > at java.lang.Thread.run(Thread.java:745) >> > > 2016-10-05 06:13:29,788 DEBUG [c.c.a.m.DirectAgentAttache] >> > (DirectAgent-302:ctx-78a58d67) (logid:9ad66ce9) Seq >> 4-1440588930805137508: >> > Response Received: >> > > 2016-10-05 06:13:29,788 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] >> > > (DirectAgent-302:ctx-78a58d67) (logid:9ad66ce9) Seq >> > > 4-1440588930805137508: MgmtId 275619427423488: Resp: Routing to peer >> > > 2016-10-05 06:13:29,789 DEBUG [c.c.a.m.AgentAttache] >> > > (DirectAgent-302:ctx-78a58d67) (logid:9ad66ce9) Seq >> > > 4-1440588930805137508: No more commands found >> > > 2016-10-05 06:13:31,120 DEBUG [o.s.b.f.s.DefaultListableBeanFactory] >> > (API-Job-Executor-8:ctx-a6e36538 job-171 ctx-446c510f) (logid:9ad66ce9) >> > Returning cached instance of singleton bean 'messageBus' >> > > 2016-10-05 06:13:31,127 ERROR [c.c.a.ApiAsyncJobDispatcher] >> > > (API-Job-Executor-8:ctx-a6e36538 job-171) (logid:9ad66ce9) Unexpected >> > > exception while executing >> > > org.apache.cloudstack.api.command.admin.vm.RemoveNicFromVMCmdByAdmin >> > > com.cloud.utils.exception.CloudRuntimeException: Unable to remove >> > Ntwk[205|Guest|16] from VM[User|i-2-3-VM] >> > > at com.cloud.vm.UserVmManagerImpl >> .removeNicFromVirtualMachine( >> > UserVmManagerImpl.java:1291) >> > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native >> Method) >> > > at sun.reflect.NativeMethodAccessorImpl.invoke( >> > NativeMethodAccessorImpl.java:57) >> > > at sun.reflect.DelegatingMethodAccessorImpl.invoke( >> > DelegatingMethodAccessorImpl.java:43) >> > > at java.lang.reflect.Method.invoke(Method.java:606) >> > > at org.springframework.aop.support.AopUtils. >> > invokeJoinpointUsingReflection(AopUtils.java:317) >> > > at org.springframework.aop.framew >> ork.ReflectiveMethodInvocation. >> > invokeJoinpoint(ReflectiveMethodInvocation.java:183) >> > > at org.springframework.aop.framew >> ork.ReflectiveMethodInvocation. >> > proceed(ReflectiveMethodInvocation.java:150) >> > > at org.apache.cloudstack.network.contrail.management. >> > EventUtils$EventInterceptor.invoke(EventUtils.java:106) >> > > at org.springframework.aop.framew >> ork.ReflectiveMethodInvocation. >> > proceed(ReflectiveMethodInvocation.java:161) >> > > at com.cloud.event.ActionEventInterceptor.invoke( >> > ActionEventInterceptor.java:51) >> > > at org.springframework.aop.framew >> ork.ReflectiveMethodInvocation. >> > proceed(ReflectiveMethodInvocation.java:161) >> > > at org.springframework.aop.interceptor. >> > ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91) >> > > at org.springframework.aop.framew >> ork.ReflectiveMethodInvocation. >> > proceed(ReflectiveMethodInvocation.java:172) >> > > at org.springframework.aop.framework.JdkDynamicAopProxy. >> > invoke(JdkDynamicAopProxy.java:204) >> > > at com.sun.proxy.$Proxy198.remove >> NicFromVirtualMachine(Unknown >> > Source) >> > > at org.apache.cloudstack.api.command.admin.vm. >> > RemoveNicFromVMCmdByAdmin.execute(RemoveNicFromVMCmdByAdmin.java:44) >> > > at com.cloud.api.ApiDispatcher.di >> spatch(ApiDispatcher.java:150) >> > > at com.cloud.api.ApiAsyncJobDispatcher.runJob( >> > ApiAsyncJobDispatcher.java:108) >> > > at org.apache.cloudstack.framework.jobs.impl. >> > AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:554) >> > > at org.apache.cloudstack.managed.context. >> > ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) >> > > at org.apache.cloudstack.managed.context.impl. >> > DefaultManagedContext$1.call(DefaultManagedContext.java:56) >> > > at org.apache.cloudstack.managed.context.impl. >> > DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) >> > > at org.apache.cloudstack.managed.context.impl. >> > DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) >> > > at org.apache.cloudstack.managed.context. >> > ManagedContextRunnable.run(ManagedContextRunnable.java:46) >> > > at org.apache.cloudstack.framework.jobs.impl. >> > AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:502) >> > > at java.util.concurrent.Executors$RunnableAdapter. >> > call(Executors.java:471) >> > > at java.util.concurrent.FutureTask.run(FutureTask.java:262) >> > > at java.util.concurrent.ThreadPoolExecutor.runWorker( >> > ThreadPoolExecutor.java:1145) >> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run( >> > ThreadPoolExecutor.java:615) >> > > at java.lang.Thread.run(Thread.java:745) >> > > 2016-10-05 06:13:31,129 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] >> > > (API-Job-Executor-8:ctx-a6e36538 job-171) (logid:9ad66ce9) Complete >> > > async job-171, jobStatus: FAILED, resultCode: 530, result: >> > > org.apache.cloudstack.api.response.ExceptionResponse/null/{" >> uuidList": >> > > [],"errorcode":530,"errortext":"Unable to remove Ntwk[205|Guest|16] >> > > from VM[User|i-2-3-VM]"} >> > > >> > > >> > > Here is the error in the VM's vmware.log: >> > > >> > > [root@csc-1 i-2-9-VM]# tail -f vmware.log 2016-10-06T04:34:33.214Z| >> > > vcpu-0| I120: CDROM: Emulate GET CONFIGURATION RT 2 starting feature 0 >> > > 2016-10-06T04:34:33.214Z| vcpu-0| I120: CDROM: Emulate GET >> > > CONFIGURATION RT 2 starting feature 0 2016-10-06T04:36:09.127Z| vmx| >> > I120: GuestRpcSendTimedOut: message to toolbox timed out. >> > > 2016-10-06T04:36:09.128Z| vmx| I120: Vix: [80405 >> > > guestCommands.c:1924]: Error VIX_E_TOOLS_NOT_RUNNING in >> > VMAutomationTranslateGuestRpcError(): VMware Tools are not running in >> the >> > guest 2016-10-06T04:43:08.823Z| vmx| I120: Tools: No activity for 10 >> > minutes, resetting Tools version. >> > > 2016-10-06T04:43:08.824Z| vmx| I120: ToolsSetVersionWork did nothing; >> > > new tools version (0) matches old Tools version >> > > 2016-10-06T04:43:23.152Z| vmx| W110: Requesting hot-remove of >> > > ethernet1 2016-10-06T04:50:13.889Z| mks| I120: SOCKET 3 (121) recv >> > detected client closed connection 2016-10-06T04:50:13.890Z| mks| I120: >> > SOCKET 3 (121) VNC Remote Disconnect: socket closed. >> > > 2016-10-06T04:53:25.699Z| vmx| W110: VigorHotRemovePoll: device >> > ethernet1 failed to disappear. >> > > >> > > >> > > Note: I am not hitting this issue on ACS 4.7 with same VMware setup. >> > > >> > > >> > > With regards, >> > > >> > > Prashanth >> > > >> > >> > >
