Srikanteswararao Talluri created CLOUDSTACK-4475:
----------------------------------------------------
Summary: [ZWPS] attaching an uploaded volume to a VM is always
going to first primary storage added
Key: CLOUDSTACK-4475
URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4475
Project: CloudStack
Issue Type: Bug
Security Level: Public (Anyone can view this level - this is the default.)
Components: Storage Controller
Affects Versions: 4.2.1
Environment: vmware esxi 5.1
Reporter: Srikanteswararao Talluri
Priority: Critical
Fix For: 4.2.1
Steps to reproduce:
==============
1. Have an advanced zone deployment with two cluster one host and one cluster
scoped primary storage each
2. add two more zone wide primary storages
3. create a deployment on zone scoped primary storage
4. upload a volume.
5. attach uploaded volume to VM created in step 3.
Observation:
=========
While attaching volume, volume is always copied to first available primary
storage in the storage_pool table and as a result attaching a volume created on
cluster scoped primary storage to a VM with its root volume on zone wide
primary storage fails.
mysql> select * from storage_pool;
+----+--------------+--------------------------------------+-------------------+------+----------------+--------+------------+---------------+----------------+--------------+-----------+------------------------------------------------+---------------------+---------+-------------+-------------+-----------------------+---------+------------+---------+---------------+
| id | name | uuid | pool_type
| port | data_center_id | pod_id | cluster_id | used_bytes | capacity_bytes
| host_address | user_info | path |
created | removed | update_time | status |
storage_provider_name | scope | hypervisor | managed | capacity_iops |
+----+--------------+--------------------------------------+-------------------+------+----------------+--------+------------+---------------+----------------+--------------+-----------+------------------------------------------------+---------------------+---------+-------------+-------------+-----------------------+---------+------------+---------+---------------+
| 1 | primaryclus1 | 722e6181-8497-3d31-9933-a0a267ae376c | NetworkFilesystem
| 2049 | 1 | 1 | 1 | 1678552014848 | 5902284800000
| 10.147.28.7 | NULL | /export/home/talluri/vmware.campo/primary |
2013-08-23 12:11:12 | NULL | NULL | Maintenance | DefaultPrimary
| CLUSTER | NULL | 0 | NULL |
| 2 | pimaryclu2 | 9fd9b0fc-c9fd-39b8-8d66-06372c5ff6d2 | NetworkFilesystem
| 2049 | 1 | 1 | 2 | 1676566495232 | 5902284800000
| 10.147.28.7 | NULL | /export/home/talluri/vmware.campo/clus1primary |
2013-08-23 12:18:14 | NULL | NULL | Up | DefaultPrimary
| CLUSTER | NULL | 0 | NULL |
| 3 | clus1p2 | 22e0c3fe-a390-38fa-8ff7-e1d965a36309 | NetworkFilesystem
| 2049 | 1 | 1 | 1 | 1660903886848 | 5902284800000
| 10.147.28.7 | NULL | /export/home/talluri/vmware.campo/clus1p2 |
2013-08-23 14:30:32 | NULL | NULL | Up | DefaultPrimary
| CLUSTER | NULL | 0 | NULL |
| 4 | clus1p3 | f2d9fb6b-c433-3c03-acf8-8f73eac48fae | NetworkFilesystem
| 2049 | 1 | 1 | 1 | 1660901400576 | 5902284800000
| 10.147.28.7 | NULL | /export/home/talluri/vmware.campo/clus1p3 |
2013-08-23 14:31:05 | NULL | NULL | Up | DefaultPrimary
| CLUSTER | NULL | 0 | NULL |
| 5 | clus2p2 | 13bf579c-51f3-317b-893a-98ff6ca8f486 | NetworkFilesystem
| 2049 | 1 | 1 | 2 | 1660900147200 | 5902284800000
| 10.147.28.7 | NULL | /export/home/talluri/vmware.campo/clus2p2 |
2013-08-23 14:31:38 | NULL | NULL | Up | DefaultPrimary
| CLUSTER | NULL | 0 | NULL |
| 7 | clus2p3 | 294ae9ff-cb02-33a0-8f31-21fdd8ff34db | NetworkFilesystem
| 2049 | 1 | 1 | 2 | 1660894195712 | 5902284800000
| 10.147.28.7 | NULL | /export/home/talluri/vmware.campo/clus2p3 |
2013-08-23 14:33:03 | NULL | NULL | Up | DefaultPrimary
| CLUSTER | NULL | 0 | NULL |
| 8 | z1 | 035188ce-7269-3440-8090-8c13da2158ba | NetworkFilesystem
| 2049 | 1 | NULL | NULL | 1626954428416 | 5902284800000
| 10.147.28.7 | NULL | /export/home/talluri/vmware.campo/z1 |
2013-08-23 16:57:56 | NULL | NULL | Up | DefaultPrimary
| ZONE | VMware | 0 | NULL |
| 9 | z2 | f0840b39-e42e-3e48-8c36-e9d121512efd | NetworkFilesystem
| 2049 | 1 | NULL | NULL | 1627066220544 | 5902284800000
| 10.147.28.7 | NULL | /export/home/talluri/vmware.campo/z2 |
2013-08-23 16:58:25 | NULL | NULL | Up | DefaultPrimary
| ZONE | VMware | 0 | NULL |
+----+--------------+--------------------------------------+-------------------+------+----------------+--------+------------+---------------+----------------+--------------+-----------+------------------------------------------------+---------------------+---------+-------------+-------------+-----------------------+---------+------------+---------+---------------+
8 rows in set (0.00 sec)
attach volume command:
===START=== 10.101.255.7 -- GET
command=attachVolume&id=bcba9a9e-de07-433f-bbad-4a9a03d5283f&virtualMachineId=67dbcb09-dbe6-4b8d-892d-e0bb02eecca4&response=json&sessionkey=Kzq5%2F2XXjxGbr%2FF4ahPYYcgKZDU%3D&_=1377264362757
2013-08-24 00:20:09,974 DEBUG [cloud.async.AsyncJobManagerImpl]
(catalina-exec-19:null) submit async job-62 = [
556e947b-98bf-4e8f-a3cd-80ba46cd9e9d ], details: AsyncJobVO {id:62, userId: 2,
accountId: 2, sessionKey: null, instanceType: Volume, instanceId: 21, cmd:
org.apache.cloudstack.api.command.user.volume.AttachVolumeCmd, cmdOriginator:
null, cmdInfo:
{"response":"json","id":"bcba9a9e-de07-433f-bbad-4a9a03d5283f","sessionkey":"Kzq5/2XXjxGbr/F4ahPYYcgKZDU\u003d","cmdEventType":"VOLUME.ATTACH","ctxUserId":"2","virtualMachineId":"67dbcb09-dbe6-4b8d-892d-e0bb02eecca4","httpmethod":"GET","_":"1377264362757","ctxAccountId":"2","ctxStartEventId":"181"},
cmdVersion: 0, callbackType: 0, callbackAddress: null, status: 0,
processStatus: 0, resultCode: 0, result: null, initMsid: 7363452993625,
completeMsid: null, lastUpdated: null, lastPolled: null, created: null}
2013-08-24 00:20:09,976 DEBUG [cloud.api.ApiServlet] (catalina-exec-19:null)
===END=== 10.101.255.7 -- GET
command=attachVolume&id=bcba9a9e-de07-433f-bbad-4a9a03d5283f&virtualMachineId=67dbcb09-dbe6-4b8d-892d-e0bb02eecca4&response=json&sessionkey=Kzq5%2F2XXjxGbr%2FF4ahPYYcgKZDU%3D&_=1377264362757
2013-08-24 00:20:09,979 DEBUG [cloud.async.AsyncJobManagerImpl]
(Job-Executor-62:job-62 = [ 556e947b-98bf-4e8f-a3cd-80ba46cd9e9d ]) Executing
org.apache.cloudstack.api.command.user.volume.AttachVolumeCmd for job-62 = [
556e947b-98bf-4e8f-a3cd-80ba46cd9e9d ]
2013-08-24 00:20:10,022 DEBUG [storage.allocator.LocalStoragePoolAllocator]
(Job-Executor-62:job-62 = [ 556e947b-98bf-4e8f-a3cd-80ba46cd9e9d ])
LocalStoragePoolAllocator trying to find storage pool to fit the vm
2013-08-24 00:20:10,023 DEBUG
[storage.allocator.ClusterScopeStoragePoolAllocator] (Job-Executor-62:job-62 =
[ 556e947b-98bf-4e8f-a3cd-80ba46cd9e9d ]) ClusterScopeStoragePoolAllocator
looking for storage pool
2013-08-24 00:20:10,024 DEBUG
[storage.allocator.ClusterScopeStoragePoolAllocator] (Job-Executor-62:job-62 =
[ 556e947b-98bf-4e8f-a3cd-80ba46cd9e9d ]) Looking for pools in dc: 1 pod:1
cluster:null
2013-08-24 00:20:10,034 DEBUG [storage.allocator.AbstractStoragePoolAllocator]
(Job-Executor-62:job-62 = [ 556e947b-98bf-4e8f-a3cd-80ba46cd9e9d ]) Checking if
storage pool is suitable, name: null ,poolId: 2
2013-08-24 00:20:10,040 DEBUG [cloud.storage.StorageManagerImpl]
(Job-Executor-62:job-62 = [ 556e947b-98bf-4e8f-a3cd-80ba46cd9e9d ]) Checking
pool 2 for storage, totalSize: 5902284800000, usedBytes: 4282525622272,
usedPct: 0.725570820010583, disable threshold: 0.85
2013-08-24 00:20:10,048 DEBUG [cloud.storage.StorageManagerImpl]
(Job-Executor-62:job-62 = [ 556e947b-98bf-4e8f-a3cd-80ba46cd9e9d ]) Checking
pool: 2 for volume allocation [Vol[21|vm=null|DATADISK]], maxSize :
11804569600000, totalAllocatedSize : 2097152000, askingSize : 2097152000,
allocated disable threshold: 0.85
2013-08-24 00:20:10,048 DEBUG
[storage.allocator.ClusterScopeStoragePoolAllocator] (Job-Executor-62:job-62 =
[ 556e947b-98bf-4e8f-a3cd-80ba46cd9e9d ]) FirstFitStoragePoolAllocator
returning 1 suitable storage pools
2013-08-24 00:20:10,078 DEBUG [storage.motion.AncientDataMotionStrategy]
(Job-Executor-62:job-62 = [ 556e947b-98bf-4e8f-a3cd-80ba46cd9e9d ]) copyAsync
inspecting src type VOLUME copyAsync inspecting dest type VOLUME
2013-08-24 00:20:10,104 DEBUG [agent.transport.Request] (Job-Executor-62:job-62
= [ 556e947b-98bf-4e8f-a3cd-80ba46cd9e9d ]) Seq 3-1450508329: Sending { Cmd ,
MgmtId: 7363452993625, via: 3, Ver: v1, Flags: 100011,
[{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"bcba9a9e-de07-433f-bbad-4a9a03d5283f","volumeType":"DATADISK","dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.147.28.7/export/home/talluri/vmware.campo/secondary","_role":"Image"}},"name":"uv","size":2097152000,"path":"volumes/2/21/da6a4f3f-fd51-3a23-ac2e-53d584936e92.ova","volumeId":21,"accountId":2,"format":"OVA","id":21,"hypervisorType":"VMware"}},"destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"bcba9a9e-de07-433f-bbad-4a9a03d5283f","volumeType":"DATADISK","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"9fd9b0fc-c9fd-39b8-8d66-06372c5ff6d2","id":2,"poolType":"NetworkFilesystem","host":"10.147.28.7","path":"/export/home/talluri/vmware.campo/clus1primary","port":2049}},"name":"uv","size":2097152000,"volumeId":21,"accountId":2,"format":"OVA","id":21,"hypervisorType":"VMware"}},"executeInSequence":false,"wait":10800}}]
}
Error:
2013-08-24 00:21:00,632 DEBUG [agent.transport.Request] (Job-Executor-62:job-62
= [ 556e947b-98bf-4e8f-a3cd-80ba46cd9e9d ]) Seq 3-1450508329: Received: { Ans:
, MgmtId: 7363452993625, via: 3, Ver: v1, Flags: 10, { CopyCmdAnswer } }
2013-08-24 00:21:00,690 ERROR [cloud.async.AsyncJobManagerImpl]
(Job-Executor-62:job-62 = [ 556e947b-98bf-4e8f-a3cd-80ba46cd9e9d ]) Unexpected
exception while executing
org.apache.cloudstack.api.command.user.volume.AttachVolumeCmd
com.cloud.utils.exception.CloudRuntimeException: Can't move volume between
scope: CLUSTER and ZONE
at
com.cloud.storage.VolumeManagerImpl.needMoveVolume(VolumeManagerImpl.java:1596)
at
com.cloud.storage.VolumeManagerImpl.attachVolumeToVM(VolumeManagerImpl.java:1871)
at
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at
org.apache.cloudstack.api.command.user.volume.AttachVolumeCmd.execute(AttachVolumeCmd.java:122)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
at
com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
2013-08-24 00:21:00,692 DEBUG [cloud.async.AsyncJobManagerImpl]
(Job-Executor-62:job-62 = [ 556e947b-98bf-4e8f-a3cd-80ba46cd9e9d ]) Complete
async job-62 = [ 556e947b-98bf-4e8f-a3cd-80ba46cd9e9d ], jobStatus: 2,
resultCode: 530, result: Error Code: 530 Error text: Can't move volume between
scope: CLUSTER and ZONE
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
