[ https://issues.apache.org/jira/browse/CLOUDSTACK-9821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15897264#comment-15897264 ]
Jayapal Reddy commented on CLOUDSTACK-9821: ------------------------------------------- {noformat} Mar 6 18:49:44 xenserver-ownpphln SM: [8771] ['/bin/bash', '-c', 'ipset', '-N', 'ipsetqzvxtmp', 'nethash'] Mar 6 18:49:44 xenserver-ownpphln SM: [8771] FAILED in util.pread: (rc 2) stdout: '', stderr: 'ipset v6.11: No command specified. Mar 6 18:49:44 xenserver-ownpphln SM: [8771] Try `ipset help' for more information. Mar 6 18:49:44 xenserver-ownpphln SM: [8771] ' Mar 6 18:49:44 xenserver-ownpphln SM: [8771] ['/bin/bash', '-c', 'ipset', '-F', 'ipsetqzvxtmp'] Mar 6 18:49:44 xenserver-ownpphln SM: [8771] FAILED in util.pread: (rc 2) stdout: '', stderr: 'ipset v6.11: No command specified. Mar 6 18:49:44 xenserver-ownpphln SM: [8771] Try `ipset help' for more information. {noformat} > Unable to deploy user VM in Basic Zone > -------------------------------------- > > Key: CLOUDSTACK-9821 > URL: https://issues.apache.org/jira/browse/CLOUDSTACK-9821 > Project: CloudStack > Issue Type: Bug > Security Level: Public(Anyone can view this level - this is the > default.) > Components: Network Controller > Affects Versions: 4.10.0.0 > Environment: Ubuntu 12.04 for CS MS > Reporter: Mike Tutkowski > Priority: Blocker > Fix For: 4.10.0.0 > > > From my e-mails on dev@ concerning this issue (messages listed oldest to > newest): > I see the following exception when trying to deploy a user VM in a Basic Zone > with two XenServer 6.5 hosts in one cluster. My system VMs have all deployed > properly. The user template gets downloaded fine. I can see the user VM begin > to start on a XenServer host, then it goes away. We then automatically try on > the other host. I can see the VM begin to start there for a moment, then it > goes away. > I am just deploying the user VM’s template and root disk to NFS (same place > where the template and root disks of my system VMs are). > I am using the built-in XenServer CentOS 5.6 (64 bit) template with 1 vCPU, > 500 MHz, and 256 MB memory. > WARN [c.c.a.r.v.VirtualRoutingResource] (DirectAgent-7:ctx-35aded78) > (logid:aab9c320) Expected 1 answers while executing VmDataCommand but > received 2 > WARN [c.c.v.VirtualMachinePowerStateSyncImpl] > (DirectAgentCronJob-14:ctx-27fb1ac3) (logid:2c342f23) VM state was updated > but update time is null?! vm id: 6 > INFO [o.a.c.f.j.i.AsyncJobManagerImpl] > (AsyncJobMgr-Heartbeat-1:ctx-2c7d2dce) (logid:a56a9a8c) Begin cleanup expired > async-jobs > INFO [o.a.c.f.j.i.AsyncJobManagerImpl] > (AsyncJobMgr-Heartbeat-1:ctx-2c7d2dce) (logid:a56a9a8c) End cleanup expired > async-jobs > INFO [c.c.u.AccountManagerImpl] (AccountChecker-1:ctx-383a632c) > (logid:541e9ba5) Found 0 removed accounts to cleanup > INFO [c.c.u.AccountManagerImpl] (AccountChecker-1:ctx-383a632c) > (logid:541e9ba5) Found 0 disabled accounts to cleanup > INFO [c.c.u.AccountManagerImpl] (AccountChecker-1:ctx-383a632c) > (logid:541e9ba5) Found 0 inactive domains to cleanup > INFO [c.c.u.AccountManagerImpl] (AccountChecker-1:ctx-383a632c) > (logid:541e9ba5) Found 0 disabled projects to cleanup > WARN [c.c.h.x.r.CitrixResourceBase] (DirectAgent-16:ctx-7c901443) > (logid:aab9c320) callHostPlugin failed for cmd: default_network_rules with > args secIps: 0:, vmName: i-2-6-VM, vmID: 6, vmIP: 10.117.40.53, vmMAC: > 06:b2:f4:00:00:22, due to There was a failure communicating with the plugin. > WARN [c.c.h.x.r.w.x.CitrixStartCommandWrapper] (DirectAgent-16:ctx-7c901443) > (logid:aab9c320) Catch Exception: class > com.cloud.utils.exception.CloudRuntimeException due to > com.cloud.utils.exception.CloudRuntimeException: callHostPlugin failed for > cmd: default_network_rules with args secIps: 0:, vmName: i-2-6-VM, vmID: 6, > vmIP: 10.117.40.53, vmMAC: 06:b2:f4:00:00:22, due to There was a failure > communicating with the plugin. > com.cloud.utils.exception.CloudRuntimeException: callHostPlugin failed for > cmd: default_network_rules with args secIps: 0:, vmName: i-2-6-VM, vmID: 6, > vmIP: 10.117.40.53, vmMAC: 06:b2:f4:00:00:22, due to There was a failure > communicating with the plugin. > at > com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.callHostPlugin(CitrixResourceBase.java:338) > at > com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixStartCommandWrapper.execute(CitrixStartCommandWrapper.java:188) > at > com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixStartCommandWrapper.execute(CitrixStartCommandWrapper.java:53) > at > com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122) > at > com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1691) > at > com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > WARN [c.c.h.x.r.CitrixResourceBase] (DirectAgent-16:ctx-7c901443) > (logid:aab9c320) Unable to start i-2-6-VM due to > com.cloud.utils.exception.CloudRuntimeException: callHostPlugin failed for > cmd: default_network_rules with args secIps: 0:, vmName: i-2-6-VM, vmID: 6, > vmIP: 10.117.40.53, vmMAC: 06:b2:f4:00:00:22, due to There was a failure > communicating with the plugin. > at > com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.callHostPlugin(CitrixResourceBase.java:338) > at > com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixStartCommandWrapper.execute(CitrixStartCommandWrapper.java:188) > at > com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixStartCommandWrapper.execute(CitrixStartCommandWrapper.java:53) > at > com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122) > at > com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1691) > at > com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > INFO [o.a.c.e.o.NetworkOrchestrator] (Network-Scavenger-1:ctx-2058d5ac) > (logid:bf8885a0) NetworkGarbageCollector uses '20' seconds for GC interval. > WARN [c.c.h.x.r.CitrixResourceBase] (DirectAgent-16:ctx-7c901443) > (logid:aab9c320) Unable to clean up VBD due to > You gave an invalid object reference. The object may have recently been > deleted. The class parameter gives the type of reference given, and the > handle parameter echoes the bad value given. > at com.xensource.xenapi.Types.checkResponse(Types.java:693) > at com.xensource.xenapi.Connection.dispatch(Connection.java:395) > at > com.cloud.hypervisor.xenserver.resource.XenServerConnectionPool$XenServerConnection.dispatch(XenServerConnectionPool.java:457) > at com.xensource.xenapi.VBD.unplug(VBD.java:1109) > at > com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.handleVmStartFailure(CitrixResourceBase.java:3576) > at > com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixStartCommandWrapper.execute(CitrixStartCommandWrapper.java:210) > at > com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixStartCommandWrapper.execute(CitrixStartCommandWrapper.java:53) > at > com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122) > at > com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1691) > at > com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > WARN [c.c.h.x.r.CitrixResourceBase] (DirectAgent-16:ctx-7c901443) > (logid:aab9c320) Unable to clean up VBD due to > You gave an invalid object reference. The object may have recently been > deleted. The class parameter gives the type of reference given, and the > handle parameter echoes the bad value given. > at com.xensource.xenapi.Types.checkResponse(Types.java:693) > at com.xensource.xenapi.Connection.dispatch(Connection.java:395) > at > com.cloud.hypervisor.xenserver.resource.XenServerConnectionPool$XenServerConnection.dispatch(XenServerConnectionPool.java:457) > at com.xensource.xenapi.VBD.unplug(VBD.java:1109) > at > com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.handleVmStartFailure(CitrixResourceBase.java:3576) > at > com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixStartCommandWrapper.execute(CitrixStartCommandWrapper.java:210) > at > com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixStartCommandWrapper.execute(CitrixStartCommandWrapper.java:53) > at > com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122) > at > com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1691) > at > com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > WARN [c.c.h.x.r.CitrixResourceBase] (DirectAgent-16:ctx-7c901443) > (logid:aab9c320) Unable to cleanup VIF > You gave an invalid object reference. The object may have recently been > deleted. The class parameter gives the type of reference given, and the > handle parameter echoes the bad value given. > at com.xensource.xenapi.Types.checkResponse(Types.java:693) > at com.xensource.xenapi.Connection.dispatch(Connection.java:395) > at > com.cloud.hypervisor.xenserver.resource.XenServerConnectionPool$XenServerConnection.dispatch(XenServerConnectionPool.java:457) > at com.xensource.xenapi.VIF.unplug(VIF.java:921) > at > com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.handleVmStartFailure(CitrixResourceBase.java:3584) > at > com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixStartCommandWrapper.execute(CitrixStartCommandWrapper.java:210) > at > com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixStartCommandWrapper.execute(CitrixStartCommandWrapper.java:53) > at > com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122) > at > com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1691) > at > com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > INFO [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-2:ctx-bc104380 > job-25/job-27 ctx-8c946a5b) (logid:aab9c320) Unable to start VM on > Host[-2-Routing] due to Unable to start i-2-6-VM due to > ERROR [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-2:ctx-bc104380 > job-25/job-27 ctx-8c946a5b) (logid:aab9c320) Invocation exception, caused by: > com.cloud.exception.InsufficientServerCapacityException: Unable to create a > deployment for VM[User|i-2-6-VM]Scope=interface com.cloud.dc.DataCenter; id=1 > INFO [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-2:ctx-bc104380 > job-25/job-27 ctx-8c946a5b) (logid:aab9c320) Rethrow exception > com.cloud.exception.InsufficientServerCapacityException: Unable to create a > deployment for VM[User|i-2-6-VM]Scope=interface com.cloud.dc.DataCenter; id=1 > ERROR [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-2:ctx-bc104380 > job-25/job-27) (logid:aab9c320) Unable to complete AsyncJobVO {id:27, userId: > 2, accountId: 2, instanceType: null, instanceId: null, cmd: > com.cloud.vm.VmWorkStart, cmdInfo: > rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oABGRjSWRMAAZhdm9pZHN0ADBMY29tL2Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAljbHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVycQB-AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01hcDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAESgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAAAAAAAAAACAAAAAAAAAAIAAAAAAAAABnQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAHBwcHBwcHBwc3IAEWphdmEudXRpbC5IYXNoTWFwBQfawcMWYNEDAAJGAApsb2FkRmFjdG9ySQAJdGhyZXNob2xkeHA_QAAAAAAADHcIAAAAEAAAAAF0AApWbVBhc3N3b3JkdAAcck8wQUJYUUFEbk5oZG1Wa1gzQmhjM04zYjNKa3hw, > cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: > null, initMsid: 52237617797, completeMsid: null, lastUpdated: null, > lastPolled: null, created: Wed Mar 01 12:51:32 MST 2017}, job origin:25 > com.cloud.exception.InsufficientServerCapacityException: Unable to create a > deployment for VM[User|i-2-6-VM]Scope=interface com.cloud.dc.DataCenter; id=1 > at > com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:961) > at > com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:4661) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107) > at > com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:4822) > at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102) > at > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:554) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) > at > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:502) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-2:ctx-bc104380 > job-25/job-27) (logid:aab9c320) Remove job-27 from job monitoring > WARN [o.a.c.alerts] (API-Job-Executor-1:ctx-f787201d job-25 ctx-56356c1a) > (logid:aab9c320) alertType:: 8 // dataCenterId:: 1 // podId:: 1 // > clusterId:: null // message:: Failed to deploy Vm with Id: 6, on Host with > Id: null > ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-1:ctx-f787201d job-25) > (logid:aab9c320) Unexpected exception while executing > org.apache.cloudstack.api.command.admin.vm.DeployVMCmdByAdmin > com.cloud.utils.exception.CloudRuntimeException: Unable to start a VM due to > insufficient capacity > at > com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:623) > at > org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.deployVirtualMachine(VMEntityManagerImpl.java:242) > at > org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.deploy(VirtualMachineEntityImpl.java:212) > at > com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:4084) > at > com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3682) > at > com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3670) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) > at > org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:168) > at > com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:168) > at > org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) > at > org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213) > at com.sun.proxy.$Proxy186.startVirtualMachine(Unknown Source) > at > org.apache.cloudstack.api.command.admin.vm.DeployVMCmdByAdmin.execute(DeployVMCmdByAdmin.java:50) > at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150) > at > com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108) > at > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:554) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) > at > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:502) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Caused by: com.cloud.exception.InsufficientServerCapacityException: Unable to > create a deployment for VM[User|i-2-6-VM]Scope=interface > com.cloud.dc.DataCenter; id=1 > at > com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:961) > at > com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:4661) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > ... 18 more > ********** > According to where the code fails, though, it appears to be a networking > problem. If I set a breakpoint before the failure and change a variable to > say that security groups are not being used, then the VM starts. > I think this is a recently introduced problem because I have another branch > based off of a slightly older version of master and it works fine here. > ********** > I rolled back to my master branch at da66b06e7d562393da2e4b52206943f8bad49d10 > and it works. > It appears something that went into after that commit has broken this. It > looks like this SHA is about two weeks old and that 43 commits have gone into > master since it. -- This message was sent by Atlassian JIRA (v6.3.15#6346)