补充kvm节点上的错误日志: #agent.log 2014-08-06 01:16:37,453 WARN [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-3:null) Timed out: /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.pl -n r-12-VM -p %template=domP%name=r-12-VM%eth2ip=192.168.10.223%eth2mask=255.255.255.0%gateway=192.168.10.254%eth0ip=192.168.1.254%eth0mask=255.255.255.0%domain=cs2cloud.internal%dhcprange=192.168.1.1%eth1ip=169.254.0.55%eth1mask=255.255.0.0%type=router%disable_rp_filter=true%dns1=8.8.8.8 . Output is: 2014-08-06 01:16:58,466 WARN [kvm.resource.LibvirtComputingResource] (Script-3:null) Interrupting script.
wanghaicheng2...@gmail.com 发件人: wanghaicheng2...@gmail.com 发送时间: 2014-08-06 11:06 收件人: users-cn 主题: VXLAN模式创建vm报错 CS4.3+KVM Hypervisor: CentOS6.5 +Kernel 3.12 网络隔离:VXLAN eth0(trunk)--cloudbr0 192.168.50.107 管理、公共、存储 eth1(trunk)--cloudbr1 192.0.2.1 来宾 问题:创建第一台vm时报错,/var/log/cloudstack/management/catalina.out 报错日志: INFO [o.a.c.f.j.i.AsyncJobMonitor] (Job-Executor-51:ctx-30453c11) Add job-69 into job monitoring WARN [o.a.c.f.j.i.AsyncJobMonitor] (Timer-1:ctx-3ca98c34) Task (job-69) has been pending for 142 seconds WARN [o.a.c.f.j.i.AsyncJobMonitor] (Timer-1:ctx-27402e73) Task (job-69) has been pending for 202 seconds WARN [o.a.c.f.j.i.AsyncJobMonitor] (Timer-1:ctx-33453472) Task (job-69) has been pending for 262 seconds WARN [o.a.c.f.j.i.AsyncJobMonitor] (Timer-1:ctx-61c8019e) Task (job-69) has been pending for 322 seconds WARN [o.a.c.f.j.i.AsyncJobMonitor] (Timer-1:ctx-b2f7e70f) Task (job-69) has been pending for 382 seconds INFO [c.c.h.HighAvailabilityManagerImpl] (HA-1:ctx-709f354f) checking health of usage server WARN [o.a.c.alerts] (HA-1:ctx-709f354f) alertType:: 13 // dataCenterId:: 0 // podId:: 0 // clusterId:: null // message:: No usage server process running WARN [o.a.c.f.j.i.AsyncJobMonitor] (Timer-1:ctx-432cc75c) Task (job-69) has been pending for 442 seconds WARN [o.a.c.f.j.i.AsyncJobMonitor] (Timer-1:ctx-54f26ce2) Task (job-69) has been pending for 502 seconds WARN [o.a.c.f.j.i.AsyncJobMonitor] (Timer-1:ctx-5384af05) Task (job-69) has been pending for 562 seconds WARN [o.a.c.f.j.i.AsyncJobMonitor] (Timer-1:ctx-062f1af3) Task (job-69) has been pending for 622 seconds WARN [o.a.c.f.j.i.AsyncJobMonitor] (Timer-1:ctx-ad5358b6) Task (job-69) has been pending for 682 seconds WARN [o.a.c.f.j.i.AsyncJobMonitor] (Timer-1:ctx-c41b27b0) Task (job-69) has been pending for 742 seconds WARN [o.a.c.f.j.i.AsyncJobMonitor] (Timer-1:ctx-a46e086d) Task (job-69) has been pending for 802 seconds WARN [o.a.c.f.j.i.AsyncJobMonitor] (Timer-1:ctx-d2bf8947) Task (job-69) has been pending for 862 seconds WARN [o.a.c.f.j.i.AsyncJobMonitor] (Timer-1:ctx-96cbabe5) Task (job-69) has been pending for 922 seconds WARN [o.a.c.f.j.i.AsyncJobMonitor] (Timer-1:ctx-b2ac4671) Task (job-69) has been pending for 982 seconds INFO [c.c.n.v.VpcManagerImpl] (VpcChecker-1:ctx-ea3dafc3) Found 0 removed VPCs to cleanup INFO [c.c.u.AccountManagerImpl] (AccountChecker-1:ctx-cb1ceb0b) Found 0 removed accounts to cleanup INFO [c.c.u.AccountManagerImpl] (AccountChecker-1:ctx-cb1ceb0b) Found 0 disabled accounts to cleanup INFO [c.c.u.AccountManagerImpl] (AccountChecker-1:ctx-cb1ceb0b) Found 0 inactive domains to cleanup INFO [c.c.u.AccountManagerImpl] (AccountChecker-1:ctx-cb1ceb0b) Found 0 disabled projects to cleanup INFO [c.c.h.HighAvailabilityManagerImpl] (HA-4:ctx-33cd71d6) HA Cleanup Thread Running INFO [c.c.h.HighAvailabilityManagerImpl] (HA-2:ctx-d1a78940) checking health of usage server WARN [o.a.c.alerts] (HA-2:ctx-d1a78940) alertType:: 13 // dataCenterId:: 0 // podId:: 0 // clusterId:: null // message:: No usage server process running WARN [o.a.c.f.j.i.AsyncJobMonitor] (Timer-1:ctx-46dabe8d) Task (job-69) has been pending for 1042 seconds WARN [c.c.n.r.VirtualNetworkApplianceManagerImpl] (Job-Executor-51:ctx-30453c11 ctx-ff79e24b) Unable to ssh to the VM: Can not ping System vm r-10-VMdue to:Unable to connect INFO [c.c.v.VirtualMachineManagerImpl] (Job-Executor-51:ctx-30453c11 ctx-ff79e24b) The guru did not like the answers so stopping VM[DomainRouter|r-10-VM] WARN [o.a.c.f.j.i.AsyncJobMonitor] (Timer-1:ctx-5fbbe89b) Task (job-69) has been pending for 1102 seconds WARN [o.a.c.f.j.i.AsyncJobMonitor] (Timer-1:ctx-69b764be) Task (job-69) has been pending for 1162 seconds ERROR [c.c.v.VirtualMachineManagerImpl] (Job-Executor-51:ctx-30453c11 ctx-ff79e24b) Failed to start instance VM[DomainRouter|r-10-VM] com.cloud.utils.exception.ExecutionException: Unable to start VM[DomainRouter|r-10-VM] due to error in finalizeStart, not retrying at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1039) at com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:761) at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.start(VirtualNetworkApplianceManagerImpl.java:2845) at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startVirtualRouter(VirtualNetworkApplianceManagerImpl.java:1924) at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startRouters(VirtualNetworkApplianceManagerImpl.java:2024)at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.deployVirtualRouterInGuestNetwork(VirtualNetworkApplianceManagerImpl.java:2002) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) at com.sun.proxy.$Proxy240.deployVirtualRouterInGuestNetwork(Unknown Source) at com.cloud.network.element.VirtualRouterElement.implement(VirtualRouterElement.java:188) at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.implementNetworkElementsAndResources(NetworkOrchestrator.java:1070) at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.implementNetwork(NetworkOrchestrator.java:977) at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.prepare(NetworkOrchestrator.java:1238) at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:960) at com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:761) at com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:601) at org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.deployVirtualMachine(VMEntityManagerImpl.java:228) at org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.deploy(VirtualMachineEntityImpl.java:207) at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3581) at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3161) at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3147) ...skipping... at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3581) at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3161) at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3147) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:50) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) at com.sun.proxy.$Proxy169.startVirtualMachine(Unknown Source) at org.apache.cloudstack.api.command.user.vm.DeployVMCmd.execute(DeployVMCmd.java:443) at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:161) at com.cloud.api.ApiAsyncJobDispatcher.runJobInContext(ApiAsyncJobDispatcher.java:109) at com.cloud.api.ApiAsyncJobDispatcher$1.run(ApiAsyncJobDispatcher.java:66) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:63) at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:509) at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) Caused by: com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM[User|vm-1]Scope=interface com.cloud.dc.DataCenter; id=2 at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:921) at com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:761) at com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:601) ... 37 more INFO [o.a.c.f.j.i.AsyncJobMonitor] (Job-Executor-51:ctx-30453c11) Remove job-69 from job monitoring INFO [c.c.v.UserVmManagerImpl] (UserVm-Scavenger-1:ctx-76fbcbd0) Found 1 vms to expunge. WARN [o.a.c.f.j.AsyncJobExecutionContext] (UserVm-Scavenger-1:ctx-76fbcbd0) Job is executed without a context, setup psudo job for the executing thread INFO [c.c.h.HighAvailabilityManagerImpl] (HA-2:ctx-93ebf822) checking health of usage server WARN [o.a.c.alerts] (HA-2:ctx-93ebf822) alertType:: 13 // dataCenterId:: 0 // podId:: 0 // clusterId:: null // message:: No usage server process running INFO [c.c.h.HighAvailabilityManagerImpl] (HA-2:ctx-facbf7ea) checking health of usage server WARN [o.a.c.alerts] (HA-2:ctx-facbf7ea) alertType:: 13 // dataCenterId:: 0 // podId:: 0 // clusterId:: null // message:: No usage server process running ...skipping... at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3581) at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3161) at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3147) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:50) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) at com.sun.proxy.$Proxy169.startVirtualMachine(Unknown Source) at org.apache.cloudstack.api.command.user.vm.DeployVMCmd.execute(DeployVMCmd.java:443) at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:161) at com.cloud.api.ApiAsyncJobDispatcher.runJobInContext(ApiAsyncJobDispatcher.java:109) at com.cloud.api.ApiAsyncJobDispatcher$1.run(ApiAsyncJobDispatcher.java:66) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:63) at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:509) at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) Caused by: com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM[User|vm-1]Scope=interface com.cloud.dc.DataCenter; id=2 at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:921) at com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:761) at com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:601) ... 37 more INFO [o.a.c.f.j.i.AsyncJobMonitor] (Job-Executor-51:ctx-30453c11) Remove job-69 from job monitoring INFO [c.c.v.UserVmManagerImpl] (UserVm-Scavenger-1:ctx-76fbcbd0) Found 1 vms to expunge. WARN [o.a.c.f.j.AsyncJobExecutionContext] (UserVm-Scavenger-1:ctx-76fbcbd0) Job is executed without a context, setup psudo job for the executing thread INFO [c.c.h.HighAvailabilityManagerImpl] (HA-2:ctx-93ebf822) checking health of usage server WARN [o.a.c.alerts] (HA-2:ctx-93ebf822) alertType:: 13 // dataCenterId:: 0 // podId:: 0 // clusterId:: null // message:: No usage server process running INFO [c.c.h.HighAvailabilityManagerImpl] (HA-2:ctx-facbf7ea) checking health of usage server WARN [o.a.c.alerts] (HA-2:ctx-facbf7ea) alertType:: 13 // dataCenterId:: 0 // podId:: 0 // clusterId:: null // message:: No usage server process running wanghaicheng2...@gmail.com