[ https://issues.apache.org/jira/browse/CLOUDSTACK-3737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Abhinandan Prateek updated CLOUDSTACK-3737: ------------------------------------------- Assignee: Nitin Mehta > Uploaded volume is not getting deleted from secondary storage after attaching > it to guest vm > -------------------------------------------------------------------------------------------- > > Key: CLOUDSTACK-3737 > URL: https://issues.apache.org/jira/browse/CLOUDSTACK-3737 > Project: CloudStack > Issue Type: Bug > Security Level: Public(Anyone can view this level - this is the > default.) > Components: Storage Controller, Volumes > Affects Versions: 4.2.0 > Environment: Latest build from ACS 4.2 branch > Storage: NFS for both primary and secondary > Reporter: Sanjeev N > Assignee: Nitin Mehta > Fix For: 4.2.0 > > Attachments: management-server.rar > > > Uploaded volume is not getting deleted from secondary storage after attaching > it to guest vm > Steps to Reproduce: > ================ > 1.Bring up CS with xen hypervisor and use NFS as the storage for both primary > and secondary storage > 2.Deploy guest vm using default cent of template just with root disk > 3.Upload volume to CS using following API: > http://10.147.59.126:8096/client/api?command=uploadVolume&format=VHD&name=cent62&url=http://10.147.28.7/templates/CentOS62-64bit/280c2a70-e37f-4863-bff8-d318122fd61b.vhd&zoneid=5c5c0b8a-9d5a-4b95-8f13-b31058ffdb37&account=admin&domainid=1 > 4.After the volume download is complete to secondary storage , attache it to > vm using API: > http://10.147.59.126:8096/client/api?command=attachVolume&id=0d627eec-3824-4d35-8997-853472502454&virtualmachineid=6ef83f81-2577-4be2-9720-5fdcf3912e5f > Observations: > =========== > After step4 volume should be moved from secondary to primary. But the > observation is it is just getting copied to primary and still present in the > secondary. It should be deleted from secondary storage after successful > copying to primary storage. > Volume state in DB after attaching to VM: > mysql> select * from volumes where > uuid='0d627eec-3824-4d35-8997-853472502454'\G; > *************************** 1. row *************************** > id: 10 > account_id: 2 > domain_id: 1 > pool_id: 1 > last_pool_id: NULL > instance_id: 3 > device_id: 1 > name: cent62 > uuid: 0d627eec-3824-4d35-8997-853472502454 > size: 10737418240 > folder: NULL > path: b2b96197-665a-40cd-a52b-de0506c45a8e > pod_id: NULL > data_center_id: 1 > iscsi_name: NULL > host_ip: NULL > volume_type: DATADISK > pool_type: NULL > disk_offering_id: 6 > template_id: NULL > first_snapshot_backup_uuid: NULL > recreatable: 0 > created: 2013-07-23 12:36:22 > attached: 2013-07-23 12:52:40 > updated: 2013-07-23 13:09:08 > removed: NULL > state: Ready > chain_info: NULL > update_count: 6 > disk_type: NULL > display_volume: 0 > format: VHD > min_iops: NULL > max_iops: NULL > 1 row in set (0.00 sec) > ERROR: > No query specified > In volume_store_ref the volume state is remained in "Creating" state. Still > attaching volume was succeeded. > Here is the volume state in volume_store_ref from cloud db: > mysql> select * from volume_store_ref where id=4\G; > *************************** 1. row *************************** > id: 4 > store_id: 1 > volume_id: 10 > zone_id: 0 > created: 2013-07-23 13:09:07 > last_updated: NULL > job_id: NULL > download_pct: 0 > size: 0 > physical_size: 0 > download_state: NULL > checksum: NULL > error_str: NULL > local_path: NULL > install_path: volumes/2/10 > url: NULL > download_url: NULL > state: Creating > destroyed: 0 > update_count: 1 > ref_cnt: 0 > updated: 2013-07-23 13:09:07 > 1 row in set (0.00 sec) > ERROR: > No query specified > Log snippet for upload volume: > 2013-07-23 08:36:22,827 DEBUG [cloud.async.AsyncJobManagerImpl] > (ApiServer-1:null) submit async job-22 = [ > 63bf80a6-5142-489e-ab5d-7a5ef061140d ], details: AsyncJobVO {id:22, userId: > 1, accountId: 1, sessionKey: null, instanceType: None, instanceId: null, cmd: > org.apache.cloudstack.api.command.user.volume.UploadVolumeCmd, cmdOriginator: > null, cmdInfo: > {"cmdEventType":"VOLUME.UPLOAD","ctxUserId":"1","zoneid":"5c5c0b8a-9d5a-4b95-8f13-b31058ffdb37","name":"cent62","account":"admin","httpmethod":"GET","domainid":"1","ctxAccountId":"1","format":"VHD","ctxStartEventId":"91","url":"http://10.147.28.7/templates/CentOS62-64bit/280c2a70-e37f-4863-bff8-d318122fd61b.vhd"}, > cmdVersion: 0, callbackType: 0, callbackAddress: null, status: 0, > processStatus: 0, resultCode: 0, result: null, initMsid: 6615759585382, > completeMsid: null, lastUpdated: null, lastPolled: null, created: null} > 2013-07-23 08:36:22,837 DEBUG [cloud.async.AsyncJobManagerImpl] > (Job-Executor-22:job-22 = [ 63bf80a6-5142-489e-ab5d-7a5ef061140d ]) Executing > org.apache.cloudstack.api.command.user.volume.UploadVolumeCmd for job-22 = [ > 63bf80a6-5142-489e-ab5d-7a5ef061140d ] > 2013-07-23 08:36:23,066 DEBUG [storage.image.BaseImageStoreDriverImpl] > (Job-Executor-22:job-22 = [ 63bf80a6-5142-489e-ab5d-7a5ef061140d ]) > Downloading volume to data store 1 > 2013-07-23 08:36:23,121 DEBUG [cloudstack.storage.RemoteHostEndPoint] > (Job-Executor-22:job-22 = [ 63bf80a6-5142-489e-ab5d-7a5ef061140d ]) Sending > command org.apache.cloudstack.storage.command.DownloadCommand to host: 3 > 2013-07-23 08:36:23,125 DEBUG [agent.transport.Request] > (Job-Executor-22:job-22 = [ 63bf80a6-5142-489e-ab5d-7a5ef061140d ]) Seq > 3-1340014758: Sending { Cmd , MgmtId: 6615759585382, via: 3, Ver: v1, Flags: > 100011, > [{"org.apache.cloudstack.storage.command.DownloadCommand":{"hvm":false,"maxDownloadSizeInBytes":536870912000,"id":10,"resourceType":"VOLUME","installPath":"volumes/2/10","_store":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.147.28.7/export/home/sanjeev/sec_xen_os","_role":"Image"}},"url":"http://10.147.28.7/templates/CentOS62-64bit/280c2a70-e37f-4863-bff8-d318122fd61b.vhd","format":"VHD","accountId":2,"name":"cent62","wait":0}}] > } > 2013-07-23 08:36:23,149 DEBUG [cloud.async.AsyncJobManagerImpl] > (Job-Executor-22:job-22 = [ 63bf80a6-5142-489e-ab5d-7a5ef061140d ]) Complete > async job-22 = [ 63bf80a6-5142-489e-ab5d-7a5ef061140d ], jobStatus: 1, > resultCode: 0, result: > org.apache.cloudstack.api.response.VolumeResponse@2a5b5372 > 2013-07-23 08:36:23,163 DEBUG [cloud.async.AsyncJobManagerImpl] > (Job-Executor-22:job-22 = [ 63bf80a6-5142-489e-ab5d-7a5ef061140d ]) Done > executing org.apache.cloudstack.api.command.user.volume.UploadVolumeCmd for > job-22 = [ 63bf80a6-5142-489e-ab5d-7a5ef061140d ] > 2013-07-23 08:36:26,660 DEBUG [agent.transport.Request] > (AgentManager-Handler-14:null) Seq 3-1340014758: Processing: { Ans: , > MgmtId: 6615759585382, via: 3, Ver: v1, Flags: 10, > [{"com.cloud.agent.api.storage.DownloadAnswer":{"jobId":"9d1225f8-4dd3-44fb-b39d-391583e95281","downloadPct":1,"errorString":" > > ","downloadStatus":"DOWNLOAD_IN_PROGRESS","downloadPath":"/mnt/SecStorage/1f24310b-8f0b-3563-aecc-779179564245/volumes/2/10/dnld6777686660366560545tmp_","installPath":"volumes/2/10","templateSize":0,"templatePhySicalSize":0,"result":true,"details":" > ","wait":0}}] } > After some time volume was successfully downloaded to secondary storage: > 2013-07-23 08:45:31,785 DEBUG [agent.transport.Request] > (AgentManager-Handler-4:null) Seq 3-1340014793: Processing: { Ans: , MgmtId: > 6615759585382, via: 3, Ver: v1, Flags: 10, > [{"com.cloud.agent.api.storage.DownloadAnswer":{"jobId":"9d1225f8-4dd3-44fb-b39d-391583e95281","downloadPct":100,"errorString":"Install > completed successfully at 7/23/13 12:45 > PM","downloadStatus":"DOWNLOADED","downloadPath":"/mnt/SecStorage/1f24310b-8f0b-3563-aecc-779179564245/volumes/2/10/dnld6777686660366560545tmp_","installPath":"volumes/2/10/a58b5e0e-8548-34e3-a22f-8a4b52701ac4.vhd","templateSize":10737418240,"templatePhySicalSize":2132787712,"checkSum":"bebf14997dd834de2053ce2dd6a903b6","result":true,"details":"Install > completed successfully at 7/23/13 12:45 PM","wait":0}}] } > Executed Attachevolume cmd to attach it to guest vm: > 2013-07-23 08:50:36,484 DEBUG [cloud.async.AsyncJobManagerImpl] > (Job-Executor-23:job-23 = [ 45a8259a-4c9f-4725-bb73-43fb4d0e786b ]) Executing > org.apache.cloudstack.api.command.user.volume.AttachVolumeCmd for job-23 = [ > 45a8259a-4c9f-4725-bb73-43fb4d0e786b ] > 2013-07-23 08:50:36,549 DEBUG [storage.allocator.LocalStoragePoolAllocator] > (Job-Executor-23:job-23 = [ 45a8259a-4c9f-4725-bb73-43fb4d0e786b ]) > LocalStoragePoolAllocator trying to find storage pool to fit the vm > 2013-07-23 08:50:36,551 DEBUG > [storage.allocator.ClusterScopeStoragePoolAllocator] (Job-Executor-23:job-23 > = [ 45a8259a-4c9f-4725-bb73-43fb4d0e786b ]) ClusterScopeStoragePoolAllocator > looking for storage pool > 2013-07-23 08:50:36,551 DEBUG > [storage.allocator.ClusterScopeStoragePoolAllocator] (Job-Executor-23:job-23 > = [ 45a8259a-4c9f-4725-bb73-43fb4d0e786b ]) Looking for pools in dc: 1 pod:1 > cluster:1 > 2013-07-23 08:50:36,559 DEBUG > [storage.allocator.AbstractStoragePoolAllocator] (Job-Executor-23:job-23 = [ > 45a8259a-4c9f-4725-bb73-43fb4d0e786b ]) Checking if storage pool is suitable, > name: null ,poolId: 1 > 2013-07-23 08:50:36,567 DEBUG [cloud.storage.StorageManagerImpl] > (Job-Executor-23:job-23 = [ 45a8259a-4c9f-4725-bb73-43fb4d0e786b ]) Checking > pool 1 for storage, totalSize: 5902284816384, usedBytes: 3927302963200, > usedPct: 0.6653868942919022, disable threshold: 0.85 > 2013-07-23 08:50:36,598 DEBUG [cloud.storage.StorageManagerImpl] > (Job-Executor-23:job-23 = [ 45a8259a-4c9f-4725-bb73-43fb4d0e786b ]) Checking > pool: 1 for volume allocation [Vol[10|vm=null|DATADISK]], maxSize : > 11804569632768, totalAllocatedSize : 54609838080, askingSize : 10737418240, > allocated disable threshold: 0.85 > 2013-07-23 08:50:36,598 DEBUG > [storage.allocator.ClusterScopeStoragePoolAllocator] (Job-Executor-23:job-23 > = [ 45a8259a-4c9f-4725-bb73-43fb4d0e786b ]) FirstFitStoragePoolAllocator > returning 1 suitable storage pools > 2013-07-23 08:50:36,638 DEBUG [storage.motion.AncientDataMotionStrategy] > (Job-Executor-23:job-23 = [ 45a8259a-4c9f-4725-bb73-43fb4d0e786b ]) copyAsync > inspecting src type VOLUME copyAsync inspecting dest type VOLUME > 2013-07-23 08:50:36,653 DEBUG [agent.transport.Request] > (Job-Executor-23:job-23 = [ 45a8259a-4c9f-4725-bb73-43fb4d0e786b ]) Seq > 1-1929052680: Sending { Cmd , MgmtId: 6615759585382, via: 1, Ver: v1, Flags: > 100011, > [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"0d627eec-3824-4d35-8997-853472502454","volumeType":"DATADISK","dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.147.28.7/export/home/sanjeev/sec_xen_os","_role":"Image"}},"name":"cent62","size":10737418240,"path":"volumes/2/10/a58b5e0e-8548-34e3-a22f-8a4b52701ac4.vhd","volumeId":10,"accountId":2,"format":"VHD","id":10}},"destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"0d627eec-3824-4d35-8997-853472502454","volumeType":"DATADISK","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"c65a038a-750c-3b4f-bf26-7ce3b74e1c85","id":1,"poolType":"NetworkFilesystem","host":"10.147.28.7","path":"/export/home/sanjeev/pri_xen_os","port":2049}},"name":"cent62","size":10737418240,"volumeId":10,"accountId":2,"format":"VHD","id":10}},"executeInSequence":false,"wait":10800}}] > } > 2013-07-23 08:50:36,654 DEBUG [agent.transport.Request] > (Job-Executor-23:job-23 = [ 45a8259a-4c9f-4725-bb73-43fb4d0e786b ]) Seq > 1-1929052680: Executing: { Cmd , MgmtId: 6615759585382, via: 1, Ver: v1, > Flags: 100011, > [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"0d627eec-3824-4d35-8997-853472502454","volumeType":"DATADISK","dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.147.28.7/export/home/sanjeev/sec_xen_os","_role":"Image"}},"name":"cent62","size":10737418240,"path":"volumes/2/10/a58b5e0e-8548-34e3-a22f-8a4b52701ac4.vhd","volumeId":10,"accountId":2,"format":"VHD","id":10}},"destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"0d627eec-3824-4d35-8997-853472502454","volumeType":"DATADISK","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"c65a038a-750c-3b4f-bf26-7ce3b74e1c85","id":1,"poolType":"NetworkFilesystem","host":"10.147.28.7","path":"/export/home/sanjeev/pri_xen_os","port":2049}},"name":"cent62","size":10737418240,"volumeId":10,"accountId":2,"format":"VHD","id":10}},"executeInSequence":false,"wait":10800}}] > } > 2013-07-23 08:50:36,655 DEBUG [agent.manager.DirectAgentAttache] > (DirectAgent-17:null) Seq 1-1929052680: Executing request > 2013-07-23 08:52:39,712 DEBUG [agent.transport.Request] > (Job-Executor-23:job-23 = [ 45a8259a-4c9f-4725-bb73-43fb4d0e786b ]) Seq > 1-1929052687: Sending { Cmd , MgmtId: 6615759585382, via: 1, Ver: v1, Flags: > 100011, > [{"org.apache.cloudstack.storage.command.AttachCommand":{"disk":{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"0d627eec-3824-4d35-8997-853472502454","volumeType":"DATADISK","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"c65a038a-750c-3b4f-bf26-7ce3b74e1c85","id":1,"poolType":"NetworkFilesystem","host":"10.147.28.7","path":"/export/home/sanjeev/pri_xen_os","port":2049}},"name":"cent62","size":10737418240,"path":"b2b96197-665a-40cd-a52b-de0506c45a8e","volumeId":10,"accountId":2,"format":"VHD","id":10}},"diskSeq":1,"type":"DATADISK"},"vmName":"i-2-3-VM","_storageHost":"10.147.28.7","_storagePort":2049,"_managed":false,"wait":0}}] > } > 2013-07-23 08:52:39,713 DEBUG [agent.transport.Request] > (Job-Executor-23:job-23 = [ 45a8259a-4c9f-4725-bb73-43fb4d0e786b ]) Seq > 1-1929052687: Executing: { Cmd , MgmtId: 6615759585382, via: 1, Ver: v1, > Flags: 100011, > [{"org.apache.cloudstack.storage.command.AttachCommand":{"disk":{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"0d627eec-3824-4d35-8997-853472502454","volumeType":"DATADISK","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"c65a038a-750c-3b4f-bf26-7ce3b74e1c85","id":1,"poolType":"NetworkFilesystem","host":"10.147.28.7","path":"/export/home/sanjeev/pri_xen_os","port":2049}},"name":"cent62","size":10737418240,"path":"b2b96197-665a-40cd-a52b-de0506c45a8e","volumeId":10,"accountId":2,"format":"VHD","id":10}},"diskSeq":1,"type":"DATADISK"},"vmName":"i-2-3-VM","_storageHost":"10.147.28.7","_storagePort":2049,"_managed":false,"wait":0}}] > } > 2013-07-23 08:52:39,714 DEBUG [agent.manager.DirectAgentAttache] > (DirectAgent-399:null) Seq 1-1929052687: Executing request > 2013-07-23 08:52:40,868 DEBUG [agent.manager.DirectAgentAttache] > (DirectAgent-399:null) Seq 1-1929052687: Response Received: > 2013-07-23 08:52:40,870 DEBUG [agent.transport.Request] > (DirectAgent-399:null) Seq 1-1929052687: Processing: { Ans: , MgmtId: > 6615759585382, via: 1, Ver: v1, Flags: 10, > [{"org.apache.cloudstack.storage.command.AttachAnswer":{"disk":{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"0d627eec-3824-4d35-8997-853472502454","volumeType":"DATADISK","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"c65a038a-750c-3b4f-bf26-7ce3b74e1c85","id":1,"poolType":"NetworkFilesystem","host":"10.147.28.7","path":"/export/home/sanjeev/pri_xen_os","port":2049}},"name":"cent62","size":10737418240,"path":"b2b96197-665a-40cd-a52b-de0506c45a8e","volumeId":10,"accountId":2,"format":"VHD","id":10}},"diskSeq":1,"vdiUuid":"b2b96197-665a-40cd-a52b-de0506c45a8e","type":"DATADISK"},"result":true,"wait":0}}] > } > 2013-07-23 08:52:40,870 DEBUG [agent.transport.Request] > (Job-Executor-23:job-23 = [ 45a8259a-4c9f-4725-bb73-43fb4d0e786b ]) Seq > 1-1929052687: Received: { Ans: , MgmtId: 6615759585382, via: 1, Ver: v1, > Flags: 10, { AttachAnswer } } > 2013-07-23 08:52:40,915 DEBUG [cloud.async.AsyncJobManagerImpl] > (Job-Executor-23:job-23 = [ 45a8259a-4c9f-4725-bb73-43fb4d0e786b ]) Complete > async job-23 = [ 45a8259a-4c9f-4725-bb73-43fb4d0e786b ], jobStatus: 1, > resultCode: 0, result: > org.apache.cloudstack.api.response.VolumeResponse@50be7f47 > 2013-07-23 08:52:40,922 DEBUG [cloud.async.AsyncJobManagerImpl] > (Job-Executor-23:job-23 = [ 45a8259a-4c9f-4725-bb73-43fb4d0e786b ]) Done > executing org.apache.cloudstack.api.command.user.volume.AttachVolumeCmd for > job-23 = [ 45a8259a-4c9f-4725-bb73-43fb4d0e786b ] -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira