hi all, And when i check agent's log i get this.
2015-01-04 01:23:20,320 ERROR [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) Internal error occurred when attempting to moun$ 2015-01-04 01:23:20,320 ERROR [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) Failed to create netfs mount: 10.8.100.201: /mnt$ org.libvirt.LibvirtException: internal error: Child process (/bin/mount 10.8.100.201:/mnt/secondary/template/tmpl/2/201 /mnt/0b82d60b-611d-320c$ at org.libvirt.ErrorHandler.processError(Unknown Source) at org.libvirt.Connect.processError(Unknown Source) at org.libvirt.Connect.processError(Unknown Source) at org.libvirt.Connect.storagePoolCreateXML(Unknown Source) at com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.createNetfsStoragePool(LibvirtStorageAdaptor.java:136) at com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.createStoragePool(LibvirtStorageAdaptor.java:500) at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:277) at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.getStoragePoolByURI(KVMStoragePoolManager.java:233) at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.createVbd(LibvirtComputingResource.java:3917) at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.execute(LibvirtComputingResource.java:3789) at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1333) at com.cloud.agent.Agent.processRequest(Agent.java:501) at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:808) at com.cloud.utils.nio.Task.run(Task.java:84) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 2015-01-04 01:23:20,321 ERROR [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) [Ljava.lang.StackTraceElement;@3854e123 2015-01-04 01:23:20,321 DEBUG [kvm.storage.KVMStoragePoolManager] (agentRequest-Handler-4:null) Disconnecting disk 26e85233-20dc-4cb1-99a0-751e$ 2015-01-04 01:23:20,352 WARN [cloud.agent.Agent] (agentRequest-Handler-4:null) Caught: com.cloud.utils.exception.CloudRuntimeException: org.libvirt.LibvirtException: internal error: Child process (/bin/mount 10.8.100.201:/mnt/seco$ Thank you On 3 January 2015 at 19:14, Mohamed Infaz <infaz...@cse.mrt.ac.lk> wrote: > Hi All, > > I am running cloudstack 4.4.1 in ubuntu 14.04 with three kvm Hosts. I was > successfully able to start the system virtual machines. And i was > successful in running the health script and no errors occurred. Now when i > am trying to create and instance i am getting two exceptions. > > 2015-01-04 02:45:06,605 ERROR [c.c.v.VirtualMachineManagerImpl] > (Work-Job-Executor-23:ctx-afbc7bf5 job-97/job-98 ctx-8772bf32) Failed to > start instance VM[User|i-2-16-VM] > com.cloud.utils.exception.CloudRuntimeException: Unable to get answer that > is of class com.cloud.agent.api.StartAnswer > at com.cloud.agent.manager.Commands.getAnswer(Commands.java:80) > at > com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1020) > at > com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5195) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107) > at > com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5340) > at > com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102) > at > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) > at > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > 2015-01-04 02:45:06,688 DEBUG [c.c.v.VirtualMachineManagerImpl] > (Work-Job-Executor-23:ctx-afbc7bf5 job-97/job-98 ctx-8772bf32) Cleaning up > resources for the vm VM[User|i-2-16-VM] in Starting state > 2015-01-04 02:45:06,690 DEBUG [c.c.a.t.Request] > (Work-Job-Executor-23:ctx-afbc7bf5 job-97/job-98 ctx-8772bf32) Seq > 6-3627649499846934559: Sending { Cmd , MgmtId: 248795600505608, via: > 6(virtualops-h4), Ver: v1, Flags: 100011, > [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"checkBeforeCleanup":false,"vmName":"i-2-16-VM","wait":0}}] > } > 2015-01-04 02:45:06,828 DEBUG [c.c.a.t.Request] > (AgentManager-Handler-13:null) Seq 6-3627649499846934559: Processing: { > Ans: , MgmtId: 248795600505608, via: 6, Ver: v1, Flags: 10, > [{"com.cloud.agent.api.StopAnswer":{"result":true,"wait":0}}] } > 2015-01-04 02:45:06,828 DEBUG [c.c.a.t.Request] > (Work-Job-Executor-23:ctx-afbc7bf5 job-97/job-98 ctx-8772bf32) Seq > 6-3627649499846934559: Received: { Ans: , MgmtId: 248795600505608, via: 6, > Ver: v1, Flags: 10, { StopAnswer } } > 2015-01-04 02:45:06,836 DEBUG [o.a.c.e.o.NetworkOrchestrator] > (Work-Job-Executor-23:ctx-afbc7bf5 job-97/job-98 ctx-8772bf32) Changing > active number of nics for network id=204 on -1 > 2015-01-04 02:45:07,043 DEBUG [o.a.c.e.o.NetworkOrchestrator] > (Work-Job-Executor-23:ctx-afbc7bf5 job-97/job-98 ctx-8772bf32) Asking > VirtualRouter to release > NicProfile[30-16-a82c2881-1c14-4494-89fa-cde39b406313-10.8.100.27-vlan://untagged > 2015-01-04 02:45:07,045 DEBUG [o.a.c.e.o.NetworkOrchestrator] > (Work-Job-Executor-23:ctx-afbc7bf5 job-97/job-98 ctx-8772bf32) Asking > SecurityGroupProvider to release > NicProfile[30-16-a82c2881-1c14-4494-89fa-cde39b406313-10.8.100.27-vlan://untagged > 2015-01-04 02:45:07,045 DEBUG [c.c.v.VirtualMachineManagerImpl] > (Work-Job-Executor-23:ctx-afbc7bf5 job-97/job-98 ctx-8772bf32) Successfully > released network resources for the vm VM[User|i-2-16-VM] > 2015-01-04 02:45:07,045 DEBUG [c.c.v.VirtualMachineManagerImpl] > (Work-Job-Executor-23:ctx-afbc7bf5 job-97/job-98 ctx-8772bf32) Successfully > cleanued up resources for the vm VM[User|i-2-16-VM] in Starting state > 2015-01-04 02:45:07,298 DEBUG [c.c.c.CapacityManagerImpl] > (Work-Job-Executor-23:ctx-afbc7bf5 job-97/job-98 ctx-8772bf32) VM state > transitted from :Starting to Stopped with event: OperationFailedvm's > original host id: null new host id: null host id before state transition: 6 > 2015-01-04 02:45:07,304 DEBUG [c.c.c.CapacityManagerImpl] > (Work-Job-Executor-23:ctx-afbc7bf5 job-97/job-98 ctx-8772bf32) Hosts's > actual total CPU: 13200 and CPU after applying overprovisioning: 13200 > 2015-01-04 02:45:07,304 DEBUG [c.c.c.CapacityManagerImpl] > (Work-Job-Executor-23:ctx-afbc7bf5 job-97/job-98 ctx-8772bf32) Hosts's > actual total RAM: 4104921088 and RAM after applying overprovisioning: > 4104921088 > 2015-01-04 02:45:07,304 DEBUG [c.c.c.CapacityManagerImpl] > (Work-Job-Executor-23:ctx-afbc7bf5 job-97/job-98 ctx-8772bf32) release cpu > from host: 6, old used: 500,reserved: 0, actual total: 13200, total with > overprovisioning: 13200; new used: 0,reserved:0; movedfromreserved: > false,moveToReserveredfalse > 2015-01-04 02:45:07,304 DEBUG [c.c.c.CapacityManagerImpl] > (Work-Job-Executor-23:ctx-afbc7bf5 job-97/job-98 ctx-8772bf32) release mem > from host: 6, old used: 536870912,reserved: 0, total: 4104921088; new used: > 0,reserved:0; movedfromreserved: false,moveToReserveredfalse > 2015-01-04 02:45:07,481 ERROR [c.c.v.VmWorkJobHandlerProxy] > (Work-Job-Executor-23:ctx-afbc7bf5 job-97/job-98 ctx-8772bf32) Invocation > exception, caused by: com.cloud.exception.AgentUnavailableException: > Resource [Host:6] is unreachable: Host 6: Unable to start instance due to > Unable to get answer that is of class com.cloud.agent.api.StartAnswer > 2015-01-04 02:45:07,482 INFO [c.c.v.VmWorkJobHandlerProxy] > (Work-Job-Executor-23:ctx-afbc7bf5 job-97/job-98 ctx-8772bf32) Rethrow > exception com.cloud.exception.AgentUnavailableException: Resource [Host:6] > is unreachable: Host 6: Unable to start instance due to Unable to get > answer that is of class com.cloud.agent.api.StartAnswer > 2015-01-04 02:45:07,482 DEBUG [c.c.v.VmWorkJobDispatcher] > (Work-Job-Executor-23:ctx-afbc7bf5 job-97/job-98) Done with run of VM work > job: com.cloud.vm.VmWorkStart for VM 16, job origin: 97 > 2015-01-04 02:45:07,482 ERROR [c.c.v.VmWorkJobDispatcher] > (Work-Job-Executor-23:ctx-afbc7bf5 job-97/job-98) Unable to complete > AsyncJobVO {id:98, userId: 2, accountId: 2, instanceType: null, instanceId: > null, cmd: com.cloud.vm.VmWorkStart, cmdInfo: > rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oABGRjSWRMAAZhdm9pZHN0ADBMY29tL2Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAljbHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVycQB-AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01hcDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAESgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAAAAAAAAAACAAAAAAAAAAIAAAAAAAAAEHQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAXBzcgAOamF2YS5sYW5nLkxvbmc7i-SQzI8j3wIAAUoABXZhbHVleHIAEGphdmEubGFuZy5OdW1iZXKGrJUdC5TgiwIAAHhwAAAAAAAAAAFzcQB-AAgAAAAAAAAABnBwcHEAfgAKcHNyABFqYXZhLnV0aWwuSGFzaE1hcAUH2sHDFmDRAwACRgAKbG9hZEZhY3RvckkACXRocmVzaG9sZHhwP0AAAAAAAAx3CAAAABAAAAABdAAKVm1QYXNzd29yZHQAHHJPMEFCWFFBRG5OaGRtVmtYM0JoYzNOM2IzSmt4cA, > cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, > result: null, initMsid: 248795600505608, completeMsid: null, lastUpdated: > null, lastPolled: null, created: Sun Jan 04 02:44:31 IST 2015}, job > origin:97 > com.cloud.exception.AgentUnavailableException: Resource [Host:6] is > unreachable: Host 6: Unable to start instance due to Unable to get answer > that is of class com.cloud.agent.api.StartAnswer > at > com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1106) > at > com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5195) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107) > at > com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5340) > at > com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102) > at > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) > at > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Caused by: com.cloud.utils.exception.CloudRuntimeException: Unable to get > answer that is of class com.cloud.agent.api.StartAnswer > at com.cloud.agent.manager.Commands.getAnswer(Commands.java:80) > at > com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1020) > ... 20 more > > The two exceptions are com.cloud.exception.AgentUnavailableException and > com.cloud.utils.exception.CloudRuntimeException. And all my agents are > running fine and they are connected to the MS. And when i create the vm > instance the virtual router is getting started and it is running fine. But > still unable to create a vm instance. Any help will be greatly appreciated. > > Thank you. >