sadhu suresh created CLOUDSTACK-4684: ----------------------------------------
Summary: ceph:migration of volume from one storage to another is not working Key: CLOUDSTACK-4684 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4684 Project: CloudStack Issue Type: Bug Security Level: Public (Anyone can view this level - this is the default.) Components: KVM Affects Versions: 4.2.0 Reporter: sadhu suresh 1.create a disk offering based on RBD 2.create data disk and perform storage migration from one storage to another storage (RBD based) 3.add one more kvm cluster with RBD base dprimary storage 4.perform storage migratation from one storage pool to another storage pool exists in another cluser actual results: in both cases volume migration is failing , 2013-09-17 04:42:17,606 WARN [storage.datastore.ObjectInDataStoreManagerImpl] (Job-Executor-78:job-86 = [ 3a080033-dafe-43a8-a93d-d3e53036b43c ]) Unsupported data object (VOLUME, org.apache.cloudstack.storage.datastore.PrimaryDataStoreImpl@2a3d0fc1), no need to delete from object in store ref table 2013-09-17 04:42:17,726 DEBUG [agent.transport.Request] (Job-Executor-78:job-86 = [ 3a080033-dafe-43a8-a93d-d3e53036b43c ]) Seq 1-308895640: Sending { Cmd , MgmtId: 7175246184473, via: 1, Ver: v1, Flags: 100011, [{"org.apache.cloudstack.storage.command.DeleteCommand":{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"ea77aa58-9368-411f-8df6-8d940cd9de54","volumeType":"DATADISK","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"59b50b13-75b2-3385-87c7-a452d56e2ee0","id":3,"poolType":"RBD","host":"10.147.41.3","path":"sadhu22","port":6789}},"name":"ggg","size":5368709120,"path":"c4cafbe0-8e09-486c-8d35-f94c189d0fa7","volumeId":30,"accountId":2,"format":"QCOW2","id":30,"hypervisorType":"KVM"}},"wait":0}}] } 2013-09-17 04:42:17,880 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null) SeqA 2-49564: Processing Seq 2-49564: { Cmd , MgmtId: -1, via: 2, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":2,"_loadInfo":"{\n \"connections\": []\n}","wait":0}}] } 2013-09-17 04:42:17,890 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null) SeqA 2-49564: Sending Seq 2-49564: { Ans: , MgmtId: 7175246184473, via: 2, Ver: v1, Flags: 100010, [{"com.cloud.agent.api.AgentControlAnswer":{"result":true,"wait":0}}] } 2013-09-17 04:42:17,904 DEBUG [agent.transport.Request] (AgentManager-Handler-11:null) Seq 1-308895640: Processing: { Ans: , MgmtId: 7175246184473, via: 1, Ver: v1, Flags: 10, [{"com.cloud.agent.api.Answer":{"result":true,"wait":0}}] } 2013-09-17 04:42:17,905 DEBUG [agent.transport.Request] (Job-Executor-78:job-86 = [ 3a080033-dafe-43a8-a93d-d3e53036b43c ]) Seq 1-308895640: Received: { Ans: , MgmtId: 7175246184473, via: 1, Ver: v1, Flags: 10, { Answer } } 2013-09-17 04:42:17,917 INFO [storage.volume.VolumeServiceImpl] (Job-Executor-78:job-86 = [ 3a080033-dafe-43a8-a93d-d3e53036b43c ]) Volume 30 is not referred anywhere, remove it from volumes table 2013-09-17 04:42:17,925 ERROR [cloud.storage.VolumeManagerImpl] (Job-Executor-78:job-86 = [ 3a080033-dafe-43a8-a93d-d3e53036b43c ]) migrate volume failed:com.cloud.utils.exception.CloudRuntimeException: Failed to copy sadhu1/c4cafbe0-8e09-486c-8d35-f94c189d0fa7 to 70e45ca3-fc28-4978-adb2-188dad981da3.qcow2 2013-09-17 04:42:17,925 DEBUG [cloud.storage.VolumeManagerImpl] (Job-Executor-78:job-86 = [ 3a080033-dafe-43a8-a93d-d3e53036b43c ]) Failed to migrate volume: com.cloud.exception.StorageUnavailableException: Resource [StoragePool:3] is unreachable: migrate volume failed: com.cloud.utils.exception.CloudRuntimeException: Failed to copy sadhu1/c4cafbe0-8e09-486c-8d35-f94c189d0fa7 to 70e45ca3-fc28-4978-adb2-188dad981da3.qcow2 at com.cloud.storage.VolumeManagerImpl.migrateVolume(VolumeManagerImpl.java:2254) at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125) at com.cloud.storage.VolumeManagerImpl.migrateVolume(VolumeManagerImpl.java:2238) at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125) at org.apache.cloudstack.api.command.user.volume.MigrateVolumeCmd.execute(MigrateVolumeCmd.java:103) at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158) at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:679) 2013-09-17 04:42:17,928 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Executor-78:job-86 = [ 3a080033-dafe-43a8-a93d-d3e53036b43c ]) Complete async job-86 = [ 3a080033-dafe-43a8-a93d-d3e53036b43c ], jobStatus: 2, resultCode: 530, result: Error Code: 530 Error text: Failed to migrate volume 2013-09-17 04:42:20,191 DEBUG [cloud.api.ApiServlet] (catalina-exec-4:null) ===START=== 10.150.19.243 -- GET command=queryAsyncJobResult&jobId=3a080033-dafe-43a8-a93d-d3e53036b43c&response=json&sessionkey=yitXAp912kcozh13bX9UudRDWzg%3D&_=1379353795576 2013-09-17 04:42:20,210 DEBUG [cloud.async.AsyncJobManagerImpl] (catalina-exec-4:null) Async job-86 = [ 3a080033-dafe-43a8-a93d-d3e53036b43c ] completed 2013-09-17 04:42:20,217 DEBUG [cloud.api.ApiServlet] (catalina-exec-4:null) ===END=== 10.150.19.243 -- GET command=queryAsyncJobResult&jobId=3a080033-dafe-43a8-a93d-d3e53036b43c&response=json&sessionkey=yitXAp912kcozh13bX9UudRDWzg%3D&_=1379353795576 On agent side: 2013-09-16 19:08:02,601 ERROR [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) Failed to convert rbd:sadhu44/dd66fad9-46ba-48a6-8092-096f3d7241bf:mon_host=10.147.41.3:auth_supported=cephx:id=admin:key=AQBV7AFSsLmZEBAAmGivAk/BV1iMvcaRHQfiXg== to /mnt/8d601e1f-acd8-38f4-ae1e-3a1339151745/6f75766b-cd2f-4e75-bec3-19543bb226a6.qcow2 the error was: qemu-img: Could not open 'rbd:sadhu44/dd66fad9-46ba-48a6-8092-096f3d7241bf:mon_host=10.147.41.3:auth_supported=cephx:id=admin:key=AQBV7AFSsLmZEBAAmGivAk/BV1iMvcaRHQfiXg==': Wrong medium typeqemu-img: Could not open 'rbd:sadhu44/dd66fad9-46ba-48a6-8092-096f3d7241bf:mon_host=10.147.41.3:auth_supported=cephx:id=admin:key=AQBV7AFSsLmZEBAAmGivAk/BV1iMvcaRHQfiXg==' 2013-09-16 23:03:57,680 ERROR [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-2:null) Failed to convert rbd:sadhu44/dd66fad9-46ba-48a6-8092-096f3d7241bf:mon_host=10.147.41.3:auth_supported=cephx:id=admin:key=AQBV7AFSsLmZEBAAmGivAk/BV1iMvcaRHQfiXg== to /mnt/8d601e1f-acd8-38f4-ae1e-3a1339151745/a2a7af01-c18d-41f7-8a83-68855ec634e9.qcow2 the error was: qemu-img: Could not open 'rbd:sadhu44/dd66fad9-46ba-48a6-8092-096f3d7241bf:mon_host=10.147.41.3:auth_supported=cephx:id=admin:key=AQBV7AFSsLmZEBAAmGivAk/BV1iMvcaRHQfiXg==': Wrong medium typeqemu-img: Could not open 'rbd:sadhu44/dd66fad9-46ba-48a6-8092-096f3d7241bf:mon_host=10.147.41.3:auth_supported=cephx:id=admin:key=AQBV7AFSsLmZEBAAmGivAk/BV1iMvcaRHQfiXg==' ~ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira