Sanjeev N created CLOUDSTACK-4073:
-------------------------------------
Summary: [Object_Store_Refactor] VM Deployment is failing with S3
storage in VMWare
Key: CLOUDSTACK-4073
URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4073
Project: CloudStack
Issue Type: Bug
Security Level: Public (Anyone can view this level - this is the default.)
Components: Storage Controller, VMware
Affects Versions: 4.2.0
Environment: Latest build from ACS 4.2 branch
Storage: S3 for secondary and ISCSI for primary storage
Cluster: VMWare
Reporter: Sanjeev N
Priority: Blocker
Fix For: 4.2.0
VM Deployment is failing with S3 storage in VMWare
Steps to Reproduce:
================
1.Bring up CS in advanced zone with VMWare cluster
2.Add S3 as the secondary storage, NFS as staging secondary storage and ISCSI
as the primary storage.
3.Deploy guest vm with default cent os template
Result:
======
VM Deployment failed since copycommand failed to copy cent os template from S3
storage to NFS staging storage.
Following is the log snippet after copyCommand:
2013-08-05 06:03:23,269 DEBUG [agent.transport.Request] (Job-Executor-47:job-46
= [ c355b5d9-b455-4741-aa21-68aa61c38c9c ]) Seq 9-1885405205: Sending { Cmd ,
MgmtId: 6615759585382, via: 9, Ver: v1, Flags: 100011,
[{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"template/tmpl/1/7/centos53-x64/CentOS5.3-x86_64.ova","origUrl":"http://10.147.28.7/releases/2.2.0/CentOS5.3-x86_64.ova","uuid":"edfe363e-fccc-11e2-9d3a-06045a000066","id":7,"format":"OVA","accountId":1,"checksum":"f6f881b7f2292948d8494db837fe0f47","hvm":false,"displayText":"CentOS
5.3(64-bit) no GUI
(vSphere)","imageDataStore":{"com.cloud.agent.api.to.S3TO":{"id":7,"uuid":"02049e18-46ae-4ca7-9ed6-0d435ae74e31","endPoint":"10.147.29.57:8080","bucketName":"imagestore","httpsFlag":false,"created":"Aug
5, 2013 3:26:24
AM","enableRRS":false}},"name":"centos53-x64","hypervisorType":"VMware"}},"destTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"template/tmpl/1/7","origUrl":"http://10.147.28.7/releases/2.2.0/CentOS5.3-x86_64.ova","uuid":"edfe363e-fccc-11e2-9d3a-06045a000066","id":7,"format":"OVA","accountId":1,"checksum":"f6f881b7f2292948d8494db837fe0f47","hvm":false,"displayText":"CentOS
5.3(64-bit) no GUI
(vSphere)","imageDataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.147.28.7/export/home/sanjeev/sec_esx_os","_role":"ImageCache"}},"name":"centos53-x64","hypervisorType":"VMware"}},"executeInSequence":false,"wait":10800}}]
}
2013-08-05 06:03:23,315 DEBUG [agent.transport.Request]
(AgentManager-Handler-8:null) Seq 9-1885405205: Processing: { Ans: , MgmtId:
6615759585382, via: 9, Ver: v1, Flags: 10,
[{"com.cloud.agent.api.Answer":{"result":false,"details":"not implemented
yet","wait":0}}] }
2013-08-05 06:03:23,317 DEBUG [agent.transport.Request] (Job-Executor-47:job-46
= [ c355b5d9-b455-4741-aa21-68aa61c38c9c ]) Seq 9-1885405205: Received: { Ans:
, MgmtId: 6615759585382, via: 9, Ver: v1, Flags: 10, { Answer } }
2013-08-05 06:03:23,356 DEBUG [storage.motion.AncientDataMotionStrategy]
(Job-Executor-47:job-46 = [ c355b5d9-b455-4741-aa21-68aa61c38c9c ]) copy object
failed: java.lang.NullPointerException
2013-08-05 06:03:23,356 DEBUG [storage.motion.AncientDataMotionStrategy]
(Job-Executor-47:job-46 = [ c355b5d9-b455-4741-aa21-68aa61c38c9c ]) copy failed
com.cloud.utils.exception.CloudRuntimeException: java.lang.NullPointerException
at
org.apache.cloudstack.storage.motion.AncientDataMotionStrategy.copyObject(AncientDataMotionStrategy.java:210)
at
org.apache.cloudstack.storage.motion.AncientDataMotionStrategy.copyAsync(AncientDataMotionStrategy.java:386)
at
org.apache.cloudstack.storage.motion.DataMotionServiceImpl.copyAsync(DataMotionServiceImpl.java:55)
at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.createBaseImageAsync(VolumeServiceImpl.java:434)
at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.createVolumeFromTemplateAsync(VolumeServiceImpl.java:538)
at
com.cloud.storage.VolumeManagerImpl.recreateVolume(VolumeManagerImpl.java:2488)
at
com.cloud.storage.VolumeManagerImpl.prepare(VolumeManagerImpl.java:2545)
at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:934)
at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:624)
at
org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.deployVirtualMachine(VMEntityManagerImpl.java:227)
at
org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.deploy(VirtualMachineEntityImpl.java:209)
at
com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3408)
at
com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2968)
at
com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2954)
at
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at
org.apache.cloudstack.api.command.user.vm.DeployVMCmd.execute(DeployVMCmd.java:420)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
at
com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
2013-08-05 06:03:23,388 INFO [storage.volume.VolumeServiceImpl]
(Job-Executor-47:job-46 = [ c355b5d9-b455-4741-aa21-68aa61c38c9c ]) releasing
lock for VMTemplateStoragePool 7
2013-08-05 06:03:23,389 WARN [utils.db.Merovingian2] (Job-Executor-47:job-46 =
[ c355b5d9-b455-4741-aa21-68aa61c38c9c ]) Was unable to find lock for the key
template_spool_ref7 and thread id 1287993753
2013-08-05 06:03:23,391 DEBUG [cloud.storage.VolumeManagerImpl]
(Job-Executor-47:job-46 = [ c355b5d9-b455-4741-aa21-68aa61c38c9c ]) Unable to
create Vol[15|vm=15|ROOT]:com.cloud.utils.exception.CloudRuntimeException:
java.lang.NullPointerException
2013-08-05 06:03:23,391 INFO [cloud.vm.VirtualMachineManagerImpl]
(Job-Executor-47:job-46 = [ c355b5d9-b455-4741-aa21-68aa61c38c9c ]) Unable to
contact resource.
com.cloud.exception.StorageUnavailableException: Resource [StoragePool:4] is
unreachable: Unable to create
Vol[15|vm=15|ROOT]:com.cloud.utils.exception.CloudRuntimeException:
java.lang.NullPointerException
at
com.cloud.storage.VolumeManagerImpl.recreateVolume(VolumeManagerImpl.java:2496)
at
com.cloud.storage.VolumeManagerImpl.prepare(VolumeManagerImpl.java:2545)
at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:934)
at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:624)
at
org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.deployVirtualMachine(VMEntityManagerImpl.java:227)
at
org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.deploy(VirtualMachineEntityImpl.java:209)
at
com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3408)
at
com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2968)
at
com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2954)
at
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at
org.apache.cloudstack.api.command.user.vm.DeployVMCmd.execute(DeployVMCmd.java:420)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
at
com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
2013-08-05 06:03:23,405 DEBUG [cloud.vm.VirtualMachineManagerImpl]
(Job-Executor-47:job-46 = [ c355b5d9-b455-4741-aa21-68aa61c38c9c ]) Cleaning up
resources for the vm VM[User|vm-def] in Starting state
2013-08-05 06:03:23,408 DEBUG [agent.transport.Request] (Job-Executor-47:job-46
= [ c355b5d9-b455-4741-aa21-68aa61c38c9c ]) Seq 8-1578827953: Sending { Cmd ,
MgmtId: 6615759585382, via: 8, Ver: v1, Flags: 100011,
[{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"vmName":"i-2-15-VM","wait":0}}]
}
VM deployement went fine in case of XenServer with S3 as the secondary storage,
NFS as staging and primary storage.
Attached management server log file and cloud DB.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
