Hello,
I've a issue with a CloudStack 4.1 installation on CentOS 6.4 with the
hypervisor KVM.
The primary storage is CLVM on iSCSI array (Dell Equallogic) on 3 nodes
(hosts) with 1 Cloudmgr server.
I've installed and configure a CLVM with locking_type = 3 and
wait_for_locks = 30. Cman like the slides:
http://fr.slideshare.net/MarcusLSorensen/cloud-stack-clvm.
I have a volume group share over iscsi on each node.
The issue come when I've 2 ou 3 nodes up. When I try to create a new
instance (from ISO or Template), somes times (> 70%) the creation
failed. (That works some times.
(==> no problem when I only have one active node, I can create a lot of
VM simultaneous without issue)
After some log inspection, I've found that the volume is "disappear"
from the node (which create the VM).
In libvirtd log, we have when it's failed:
2013-04-13 20:41:56.812+0000: 28373: error :
virStorageBackendVolOpenCheckMode:1023 : cannot stat file
'/dev/vg_baie/23761696-cca7-4d09-a65f-fa9d88fbd987': No such file or
directory
2013-04-13 20:41:56.841+0000: 28373: error : virCommandWait:2345 :
internal error Child process (/sbin/vgchange -aln vg_baie) unexpected
exit status 5: Can't deactivate volume group "vg_baie" with 16 open
logical volume(s)
(a virsh pool-list show that the vg has disappeared, I must restart
cloudstack-agent for enabled it. Note: the pool isn't with the autostart
option)
On management.log, we have:
2013-04-13 22:58:02,022 DEBUG [cloud.storage.StorageManagerImpl]
(Job-Executor-38:job-228) Creating volume: Vol[68|vm=68|ROOT]
2013-04-13 22:58:02,022 DEBUG [cloud.storage.StorageManagerImpl]
(Job-Executor-38:job-228) Trying to create in Pool[200|CLVM]
2013-04-13 22:58:02,027 DEBUG [agent.transport.Request]
(Job-Executor-38:job-228) Seq 1-1321467941: Sending { Cmd , MgmtId:
130177108533, via: 1, Ver: v1, Flags: 100111,
[{"storage.CreateCommand":{"volId":68,"pool":{"id":200,"uuid":"3425857b-3c3c-4e2a-8eba-115cefbe982b","host":"localhost","path":"/vg_baie","port":0,"type":"CLVM"},"diskCharacteristics":{"size":107374182400,"tags":[],"type":"ROOT","name":"ROOT-68","useLocalStorage":false,"recreatable":false,"diskOfferingId":5,"volumeId":68,"hyperType":"KVM"},"wait":0}}]
}
2013-04-13 22:58:02,077 DEBUG [agent.transport.Request]
(AgentManager-Handler-2:null) Seq 1-1321467941: Processing: { Ans: ,
MgmtId: 130177108533, via: 1, Ver: v1, Flags: 110,
[{"storage.CreateAnswer":{"requestTemplateReload":false,"result":false,"details":"Exception:
com.cloud.utils.exception.CloudRuntimeException\nMessage:
org.libvirt.LibvirtException: Storage pool not found: no pool with
matching uuid\nStack: com.cloud.utils.exception.CloudRuntimeException:
org.libvirt.LibvirtException: Storage pool not found: no pool with
matching uuid\n\tat
com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.getStoragePool(LibvirtStorageAdaptor.java:427)\n\tat
com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.getStoragePool(KVMStoragePoolManager.java:71)\n\tat
com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.execute(LibvirtComputingResource.java:1272)\n\tat
com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1073)\n\tat
com.cloud.agent.Agent.processRequest(Agent.java:525)\n\tat
com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:852)\n\tat
com.cloud.utils.nio.Task.run(Task.java:83)\n\tat
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)\n\tat
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\n\tat
java.lang.Thread.run(Thread.java:679)\n","wait":0}}] }
2013-04-13 22:58:02,077 DEBUG [agent.manager.AgentAttache]
(AgentManager-Handler-2:null) Seq 1-1321467941: No more commands found
2013-04-13 22:58:02,077 DEBUG [agent.transport.Request]
(Job-Executor-38:job-228) Seq 1-1321467941: Received: { Ans: , MgmtId:
130177108533, via: 1, Ver: v1, Flags: 110, { CreateAnswer } }
2013-04-13 22:58:02,077 DEBUG [cloud.storage.StorageManagerImpl]
(Job-Executor-38:job-228) Unable to create volume Vol[68|vm=68|ROOT]
2013-04-13 22:58:02,079 INFO [cloud.vm.VirtualMachineManagerImpl]
(Job-Executor-38:job-228) Unable to contact resource.
com.cloud.exception.StorageUnavailableException: Resource
[StoragePool:200] is unreachable: Unable to create Vol[68|vm=68|ROOT]
at
com.cloud.storage.StorageManagerImpl.prepare(StorageManagerImpl.java:3483)
at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:748)
at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:471)
at
org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.deployVirtualMachine(VMEntityManagerImpl.java:212)
at
org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.deploy(VirtualMachineEntityImpl.java:209)
at
com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3860)
at
com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3458)
at
com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3444)
at
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at
org.apache.cloudstack.api.command.user.vm.DeployVMCmd.execute(DeployVMCmd.java:379)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:162)
at
com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:437)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:679)
2013-04-13 22:58:02,081 DEBUG [cloud.vm.VirtualMachineManagerImpl]
(Job-Executor-38:job-228) Cleaning up resources for the vm
VM[User|Centos100go] in Starting state
2013-04-13 22:58:02,083 DEBUG [agent.transport.Request]
(Job-Executor-38:job-228) Seq 1-1321467942: Sending { Cmd , MgmtId:
130177108533, via: 1, Ver: v1, Flags: 100111,
[{"StopCommand":{"isProxy":false,"vmName":"i-2-68-VM","wait":0}}] }
(and after a message:
com.cloud.exception.InsufficientServerCapacityException: Unable to
create a deployment for VM[User...)
Perhaps that is a configuration issue on CLVM ? or iSCSI attach?
I'm not sure if CloudStack support the iSCSI/CLVM with KVM hypervisor?
yes/no.
I must change for a NFS primary storage (over iSCSI) ? or a
SharedMountPoint (GFS?).
That is very annoying for me, please have some ideas?
Thanks.
Milamber
