[
https://issues.apache.org/jira/browse/CLOUDSTACK-4423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13748409#comment-13748409
]
Wido den Hollander commented on CLOUDSTACK-4423:
------------------------------------------------
Working on this one. I'm close, but something is going wrong when calling
qemu-img in the background to do the RAW -> QCOW2 conversion. Not sure yet what
is happening here exactly.
> ceph:migrate instance form one primary to another primary failing
> ------------------------------------------------------------------
>
> Key: CLOUDSTACK-4423
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4423
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Components: Management Server
> Affects Versions: 4.2.0
> Reporter: sadhu suresh
> Assignee: Wido den Hollander
> Priority: Critical
> Attachments: management-server.rar
>
>
> 1.deploy advanced zone with kvm cluster(single cluster with ubuntu host+nfs
> as primary storage)
> 2.add RBD based primary storage
> 3.create compute offering and create a instance using this offering
> 4.stop the running VM which was created in step 3
> 5.perform migrate instance from one primary to another primary
> Actual result:
> Migrate instance is failing with below error
> "failed to migration: com.cloud.exception.StorageUnavailableException:
> Resource [StoragePool:2] is unreachable: migrate volume failed:
> com.cloud.utils.exception.CloudRuntimeException: Failed to copy
> cloudkvm/1c0f04aa-7d19-42ac-8998-5726b7b2cae5 to
> 59211ea7-d393-489d-aa56-7ed3d59e1d4e.qcow2
> content of management log:
> 2013-08-21 04:25:07,080 DEBUG [cloud.async.AsyncJobManagerImpl]
> (catalina-exec-25:null) submit async job-21 = [
> 92188256-af48-4878-8f5d-8f1d1a23ab65 ], details: AsyncJobVO {id:21, userId:
> 2, accountId: 2, sessionKey: null, instanceType: None, instanceId: null, cmd:
> org.apache.cloudstack.api.command.admin.vm.MigrateVMCmd, cmdOriginator: null,
> cmdInfo:
> {"response":"json","sessionkey":"Dle2oW4Rjxm7blpoMLnWiHE/ZVM\u003d","virtualmachineid":"694963ca-4ed6-4815-af80-5a77135b97c9","cmdEventType":"VM.MIGRATE","ctxUserId":"2","storageid":"2cd21530-1fb9-3aff-95bc-ba3b3b3f2b12","httpmethod":"GET","_":"1377019844223","ctxAccountId":"2","ctxStartEventId":"86"},
> cmdVersion: 0, callbackType: 0, callbackAddress: null, status: 0,
> processStatus: 0, resultCode: 0, result: null, initMsid: 7175246184473,
> completeMsid: null, lastUpdated: null, lastPolled: null, created: null}
> 2013-08-21 04:25:07,084 DEBUG [cloud.async.AsyncJobManagerImpl]
> (Job-Executor-9:job-21 = [ 92188256-af48-4878-8f5d-8f1d1a23ab65 ]) Executing
> org.apache.cloudstack.api.command.admin.vm.MigrateVMCmd for job-21 = [
> 92188256-af48-4878-8f5d-8f1d1a23ab65 ]
> 2013-08-21 04:25:07,110 DEBUG [cloud.capacity.CapacityManagerImpl]
> (Job-Executor-9:job-21 = [ 92188256-af48-4878-8f5d-8f1d1a23ab65 ]) VM state
> transitted from :Stopped to Migrating with event:
> StorageMigrationRequestedvm's original host id: 1 new host id: null host id
> before state transition: null
> 2013-08-21 04:25:07,193 DEBUG [storage.motion.AncientDataMotionStrategy]
> (Job-Executor-9:job-21 = [ 92188256-af48-4878-8f5d-8f1d1a23ab65 ]) copyAsync
> inspecting src type VOLUME copyAsync inspecting dest type VOLUME
> 2013-08-21 04:25:07,197 DEBUG [cache.allocator.StorageCacheRandomAllocator]
> (Job-Executor-9:job-21 = [ 92188256-af48-4878-8f5d-8f1d1a23ab65 ]) Can't find
> staging storage in zone: 1
> 2013-08-21 04:25:07,233 DEBUG [agent.transport.Request]
> (Job-Executor-9:job-21 = [ 92188256-af48-4878-8f5d-8f1d1a23ab65 ]) Seq
> 1-462424131: Sending { Cmd , MgmtId: 7175246184473, via: 1, Ver: v1, Flags:
> 100011,
> [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"306b108d-0fef-4b9e-85ff-f212d9acb22a","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"08d53ff3-8884-30a4-a1d1-ac621abcc688","id":3,"poolType":"RBD","host":"10.147.41.3","path":"cloudkvm","port":6789}},"name":"ROOT-4","size":8589934592,"path":"1c0f04aa-7d19-42ac-8998-5726b7b2cae5","volumeId":4,"vmName":"i-2-4-VM","accountId":2,"format":"RAW","id":4,"hypervisorType":"KVM"}},"destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"306b108d-0fef-4b9e-85ff-f212d9acb22a","volumeType":"ROOT","dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.147.28.7/export/home/sadhu/asf/kvmsec","_role":"Image"}},"name":"ROOT-4","size":8589934592,"path":"volumes/2/4","volumeId":4,"vmName":"i-2-4-VM","accountId":2,"format":"RAW","id":4,"hypervisorType":"KVM"}},"executeInSequence":false,"wait":10800}}]
> }
> 2013-08-21 04:25:07,571 DEBUG [agent.transport.Request]
> (Job-Executor-9:job-21 = [ 92188256-af48-4878-8f5d-8f1d1a23ab65 ]) Seq
> 1-462424131: Received: { Ans: , MgmtId: 7175246184473, via: 1, Ver: v1,
> Flags: 10, { CopyCmdAnswer } }
> 2013-08-21 04:25:07,571 DEBUG [storage.motion.AncientDataMotionStrategy]
> (Job-Executor-9:job-21 = [ 92188256-af48-4878-8f5d-8f1d1a23ab65 ]) copy to
> image store failed: com.cloud.utils.exception.CloudRuntimeException: Failed
> to copy cloudkvm/1c0f04aa-7d19-42ac-8998-5726b7b2cae5 to
> 59211ea7-d393-489d-aa56-7ed3d59e1d4e.qcow2
> 2013-08-21 04:25:07,590 DEBUG [agent.transport.Request]
> (Job-Executor-9:job-21 = [ 92188256-af48-4878-8f5d-8f1d1a23ab65 ]) Seq
> 3-891420680: Sending { Cmd , MgmtId: 7175246184473, via: 3, Ver: v1, Flags:
> 100011,
> [{"org.apache.cloudstack.storage.command.DeleteCommand":{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"306b108d-0fef-4b9e-85ff-f212d9acb22a","volumeType":"ROOT","dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.147.28.7/export/home/sadhu/asf/kvmsec","_role":"Image"}},"name":"ROOT-4","size":8589934592,"volumeId":4,"vmName":"i-2-4-VM","accountId":2,"format":"RAW","id":4,"hypervisorType":"KVM"}},"wait":0}}]
> }
> 2013-08-21 04:25:07,652 DEBUG [agent.transport.Request]
> (Job-Executor-9:job-21 = [ 92188256-af48-4878-8f5d-8f1d1a23ab65 ]) Seq
> 3-891420680: Received: { Ans: , MgmtId: 7175246184473, via: 3, Ver: v1,
> Flags: 10, { Answer } }
> 2013-08-21 04:25:07,655 WARN
> [storage.datastore.ObjectInDataStoreManagerImpl] (Job-Executor-9:job-21 = [
> 92188256-af48-4878-8f5d-8f1d1a23ab65 ]) Volume 4 is not found on image store
> 1, so no need to delete
> 2013-08-21 04:25:07,668 WARN
> [storage.datastore.ObjectInDataStoreManagerImpl] (Job-Executor-9:job-21 = [
> 92188256-af48-4878-8f5d-8f1d1a23ab65 ]) Unsupported data object (VOLUME,
> org.apache.cloudstack.storage.datastore.PrimaryDataStoreImpl@581a95c4), no
> need to delete from object in store ref table
> 2013-08-21 04:25:07,684 WARN
> [storage.datastore.ObjectInDataStoreManagerImpl] (Job-Executor-9:job-21 = [
> 92188256-af48-4878-8f5d-8f1d1a23ab65 ]) Unsupported data object (VOLUME,
> org.apache.cloudstack.storage.datastore.PrimaryDataStoreImpl@53310048), no
> need to delete from object in store ref table
> 2013-08-21 04:25:07,747 DEBUG [agent.transport.Request]
> (Job-Executor-9:job-21 = [ 92188256-af48-4878-8f5d-8f1d1a23ab65 ]) Seq
> 1-462424132: Sending { Cmd , MgmtId: 7175246184473, via: 1, Ver: v1, Flags:
> 100011,
> [{"org.apache.cloudstack.storage.command.DeleteCommand":{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"7aca5603-0b2c-4de6-8dee-45f192bf113f","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"2cd21530-1fb9-3aff-95bc-ba3b3b3f2b12","id":2,"poolType":"RBD","host":"10.147.41.3","path":"data","port":6789}},"name":"ROOT-4","size":8589934592,"path":"1c0f04aa-7d19-42ac-8998-5726b7b2cae5","volumeId":9,"vmName":"i-2-4-VM","accountId":2,"id":9,"hypervisorType":"KVM"}},"wait":0}}]
> }
> 2013-08-21 04:25:07,927 DEBUG [agent.transport.Request]
> (Job-Executor-9:job-21 = [ 92188256-af48-4878-8f5d-8f1d1a23ab65 ]) Seq
> 1-462424132: Received: { Ans: , MgmtId: 7175246184473, via: 1, Ver: v1,
> Flags: 10, { Answer } }
> 2013-08-21 04:25:07,936 INFO [storage.volume.VolumeServiceImpl]
> (Job-Executor-9:job-21 = [ 92188256-af48-4878-8f5d-8f1d1a23ab65 ]) Volume 9
> is not referred anywhere, remove it from volumes table
> 2013-08-21 04:25:07,947 ERROR [cloud.storage.VolumeManagerImpl]
> (Job-Executor-9:job-21 = [ 92188256-af48-4878-8f5d-8f1d1a23ab65 ]) migrate
> volume failed:com.cloud.utils.exception.CloudRuntimeException: Failed to copy
> cloudkvm/1c0f04aa-7d19-42ac-8998-5726b7b2cae5 to
> 59211ea7-d393-489d-aa56-7ed3d59e1d4e.qcow2
> 2013-08-21 04:25:07,948 DEBUG [cloud.vm.VirtualMachineManagerImpl]
> (Job-Executor-9:job-21 = [ 92188256-af48-4878-8f5d-8f1d1a23ab65 ]) Failed to
> migration: com.cloud.exception.StorageUnavailableException: Resource
> [StoragePool:2] is unreachable: migrate volume failed:
> com.cloud.utils.exception.CloudRuntimeException: Failed to copy
> cloudkvm/1c0f04aa-7d19-42ac-8998-5726b7b2cae5 to
> 59211ea7-d393-489d-aa56-7ed3d59e1d4e.qcow2
> 2013-08-21 04:25:07,953 DEBUG [cloud.capacity.CapacityManagerImpl]
> (Job-Executor-9:job-21 = [ 92188256-af48-4878-8f5d-8f1d1a23ab65 ]) VM state
> transitted from :Migrating to Stopped with event: AgentReportStoppedvm's
> original host id: 1 new host id: null host id before state transition: null
> 2013-08-21 04:25:07,960 DEBUG [cloud.capacity.CapacityManagerImpl]
> (Job-Executor-9:job-21 = [ 92188256-af48-4878-8f5d-8f1d1a23ab65 ]) Hosts's
> actual total CPU: 9572 and CPU after applying overprovisioning: 9572
> 2013-08-21 04:25:07,960 DEBUG [cloud.capacity.CapacityManagerImpl]
> (Job-Executor-9:job-21 = [ 92188256-af48-4878-8f5d-8f1d1a23ab65 ]) Hosts's
> actual total RAM: 16816889856 and RAM after applying overprovisioning:
> 16816889856
> 2013-08-21 04:25:07,960 DEBUG [cloud.capacity.CapacityManagerImpl]
> (Job-Executor-9:job-21 = [ 92188256-af48-4878-8f5d-8f1d1a23ab65 ]) release
> cpu from host: 1, old used: 2000,reserved: 100, actual total: 9572, total
> with overprovisioning: 9572; new used: 1900,reserved:100; movedfromreserved:
> false,moveToReserveredfalse
> 2013-08-21 04:25:07,960 DEBUG [cloud.capacity.CapacityManagerImpl]
> (Job-Executor-9:job-21 = [ 92188256-af48-4878-8f5d-8f1d1a23ab65 ]) release
> mem from host: 1, old used: 2013265920,reserved: 134217728, total:
> 16816889856; new used: 1879048192,reserved:134217728; movedfromreserved:
> false,moveToReserveredfalse
> 2013-08-21 04:25:07,974 ERROR [cloud.async.AsyncJobManagerImpl]
> (Job-Executor-9:job-21 = [ 92188256-af48-4878-8f5d-8f1d1a23ab65 ]) Unexpected
> exception while executing
> org.apache.cloudstack.api.command.admin.vm.MigrateVMCmd
> 2013-08-21 04:25:07,984 DEBUG [cloud.async.AsyncJobManagerImpl]
> (Job-Executor-9:job-21 = [ 92188256-af48-4878-8f5d-8f1d1a23ab65 ]) Complete
> async job-21 = [ 92188256-af48-4878-8f5d-8f1d1a23ab65 ], jobStatus: 2,
> resultCode: 530, result: Error Code: 530 Error text: Failed to migration:
> com.cloud.exception.StorageUnavailableException: Resource [StoragePool:2] is
> unreachable: migrate volume failed:
> com.cloud.utils.exception.CloudRuntimeException: Failed to copy
> cloudkvm/1c0f04aa-7d19-42ac-8998-5726b7b2cae5 to
> 59211ea7-d393-489d-aa56-7ed3d59e1d4e.qcow2
> 2013-08-21 04:25:10,287 DEBUG [cloud.async.AsyncJobManagerImpl]
> (catalina-exec-24:null) Async job-21 = [ 92188256-af48-4878-8f5d-8f1d1a23ab65
> ] completed
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
