[ https://issues.apache.org/jira/browse/CLOUDSTACK-3379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13708662#comment-13708662 ]
Min Chen commented on CLOUDSTACK-3379: -------------------------------------- This is not a bu. By design, NFS cache storage is of zone scope, that is, per zone. So for muti-zone case using S3, you need to configure cache storage for each zone. > [Object_Store_Refactor] Spinning system vms failed with > CloudRuntimeException: No cache DataStore in scope id 1 type ZONE > -------------------------------------------------------------------------------------------------------------------------- > > Key: CLOUDSTACK-3379 > URL: https://issues.apache.org/jira/browse/CLOUDSTACK-3379 > Project: CloudStack > Issue Type: Bug > Security Level: Public(Anyone can view this level - this is the > default.) > Components: Management Server > Affects Versions: 4.2.0 > Environment: Latest build from master branch > Reporter: Sanjeev N > Assignee: Min Chen > Priority: Blocker > Fix For: 4.2.0 > > Attachments: management-server.rar > > > [Object_Store_Refactor] Spinning system vms failed with > CloudRuntimeException: No cache DataStore in scope id 1 type ZONE > System vms failed to come up when there are two zones in the system and only > one Cache storage. > Steps to Reproduce: > ================ > 1.Bring up CS with latest master build > 2.Add two zones say zone1 with xen and zone2 with vmware clusters > 3.Add S3 storage and Cache Storage from zone2. Does not specify any cache > storage in zone1 > 4.Enable both the zones > Observations: > =========== > 1.After step3 vmware system template was successfully download to S3 storage. > 2,After step4 CS tried to bring up system vms using the esx routing template > but failed because there is no cache storage in zone1. > mysql> select * from image_store\G; > *************************** 1. row *************************** > id: 6 > name: objectstore > image_provider_name: S3 > protocol: http > url: NULL > data_center_id: NULL > scope: REGION > role: Image > uuid: a62707cd-8cfc-490d-81c6-dfc8d706648f > parent: NULL > created: 2013-07-05 15:15:54 > removed: NULL > total_size: NULL > used_bytes: NULL > *************************** 2. row *************************** > id: 7 > name: nfs://10.147.28.7/export/home/sanjeev/sec_esx_os > image_provider_name: NFS > protocol: nfs > url: nfs://10.147.28.7/export/home/sanjeev/sec_esx_os > data_center_id: 2 > scope: ZONE > role: ImageCache > uuid: 30d3d201-d285-4ce5-a909-598c0abb8cf3 > parent: NULL > created: 2013-07-05 15:15:54 > removed: NULL > total_size: NULL > used_bytes: NULL > 2 rows in set (0.01 sec) > ERROR: > No query specified > mysql> select * from data_center; > +----+--------------+--------------------------------------+-------------+---------------+------+---------------+---------------+---------+---------+--------------------+-------------+--------------------+--------+-----------+-------------+---------------+------------------+-------------------+---------------+---------------+---------------+-------------------+------------------+--------------------------------------+---------------------------+--------------------------+---------+-------+---------+-------------+--------------+----------+----------+ > | id | name | uuid | description | > dns1 | dns2 | internal_dns1 | internal_dns2 | gateway | netmask | > router_mac_address | mac_address | guest_network_cidr | domain | domain_id | > networktype | dns_provider | gateway_provider | firewall_provider | > dhcp_provider | lb_provider | vpn_provider | userdata_provider | > allocation_state | zone_token | > is_security_group_enabled | is_local_storage_enabled | removed | owner | > created | lastUpdated | engine_state | ip6_dns1 | ip6_dns2 | > +----+--------------+--------------------------------------+-------------+---------------+------+---------------+---------------+---------+---------+--------------------+-------------+--------------------+--------+-----------+-------------+---------------+------------------+-------------------+---------------+---------------+---------------+-------------------+------------------+--------------------------------------+---------------------------+--------------------------+---------+-------+---------+-------------+--------------+----------+----------+ > | 1 | zone1 | 2b76f2bc-a859-4253-a7de-cc7d105eb82a | NULL | > 10.103.128.16 | NULL | 10.103.128.16 | NULL | NULL | NULL | > 02:00:00:00:00:01 | 21 | 10.1.1.0/24 | NULL | NULL | > Advanced | VirtualRouter | VirtualRouter | VirtualRouter | > VirtualRouter | VirtualRouter | VirtualRouter | VirtualRouter | Enabled > | 894a4b6d-9721-3bdd-821c-f4bb50c9bb70 | 0 | > 0 | NULL | NULL | NULL | NULL | Disabled > | NULL | NULL | > | 2 | zone2-vmware | 1c0f8682-73fa-42b1-a49a-a160fb688b60 | NULL | > 10.103.128.16 | NULL | 10.103.128.16 | NULL | NULL | NULL | > 02:00:00:00:00:01 | 19 | 10.1.1.0/24 | NULL | NULL | > Advanced | VirtualRouter | VirtualRouter | VirtualRouter | > VirtualRouter | VirtualRouter | VirtualRouter | VirtualRouter | Enabled > | 21e6f09b-2b7e-3ad4-8546-0c40394d9b39 | 0 | > 0 | NULL | NULL | NULL | NULL | Disabled > | NULL | NULL | > +----+--------------+--------------------------------------+-------------+---------------+------+---------------+---------------+---------+---------+--------------------+-------------+--------------------+--------+-----------+-------------+---------------+------------------+-------------------+---------------+---------------+---------------+-------------------+------------------+--------------------------------------+---------------------------+--------------------------+---------+-------+---------+-------------+--------------+----------+----------+ > 2 rows in set (0.04 sec) > 2013-07-05 11:28:21,720 DEBUG [cloud.storage.VolumeManagerImpl] > (secstorage-1:null) Checking if we need to prepare 1 volumes for > VM[SecondaryStorageVm|s-1214-VM] > 2013-07-05 11:28:22,001 DEBUG [storage.motion.AncientDataMotionStrategy] > (secstorage-1:null) copyAsync inspecting src type TEMPLATE copyAsync > inspecting dest type TEMPLATE > 2013-07-05 11:28:22,015 DEBUG [storage.motion.AncientDataMotionStrategy] > (secstorage-1:null) needCacheStorage true, dest at null dest role > Primarytemplate/tmpl/1/1/routing-1/systemvmtemplate-2013-06-23-master-xen.vhd.bz2 > src role Image > 2013-07-05 11:28:22,021 DEBUG [cache.allocator.StorageCacheRandomAllocator] > (secstorage-1:null) Can't find cache storage in zone: 1 > 2013-07-05 11:28:22,022 DEBUG [storage.motion.AncientDataMotionStrategy] > (secstorage-1:null) copy object failed: > com.cloud.utils.exception.CloudRuntimeException: No cache DataStore in scope > id 1 type ZONE > 2013-07-05 11:28:22,025 DEBUG [storage.motion.AncientDataMotionStrategy] > (secstorage-1:null) copy failed > com.cloud.utils.exception.CloudRuntimeException: > com.cloud.utils.exception.CloudRuntimeException: No cache DataStore in scope > id 1 type ZONE > at > org.apache.cloudstack.storage.motion.AncientDataMotionStrategy.copyObject(AncientDataMotionStrategy.java:204) > at > org.apache.cloudstack.storage.motion.AncientDataMotionStrategy.copyAsync(AncientDataMotionStrategy.java:351) > at > org.apache.cloudstack.storage.motion.DataMotionServiceImpl.copyAsync(DataMotionServiceImpl.java:55) > at > org.apache.cloudstack.storage.volume.VolumeServiceImpl.createBaseImageAsync(VolumeServiceImpl.java:390) > at > org.apache.cloudstack.storage.volume.VolumeServiceImpl.createVolumeFromTemplateAsync(VolumeServiceImpl.java:490) > at > com.cloud.storage.VolumeManagerImpl.recreateVolume(VolumeManagerImpl.java:2485) > at > com.cloud.storage.VolumeManagerImpl.prepare(VolumeManagerImpl.java:2542) > at > com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:852) > at > com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:557) > at > com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:550) > at > com.cloud.storage.secondary.SecondaryStorageManagerImpl.startSecStorageVm(SecondaryStorageManagerImpl.java:260) > at > com.cloud.storage.secondary.SecondaryStorageManagerImpl.allocCapacity(SecondaryStorageManagerImpl.java:683) > at > com.cloud.storage.secondary.SecondaryStorageManagerImpl.expandPool(SecondaryStorageManagerImpl.java:1290) > at > com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:123) > at > com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:50) > at > com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:104) > at > com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33) > at > com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81) > at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at > java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:679) > 2013-07-05 11:28:22,076 DEBUG [cloud.storage.VolumeManagerImpl] > (secstorage-1:null) Unable to create > Vol[1214|vm=1214|ROOT]:com.cloud.utils.exception.CloudRuntimeException: > com.cloud.utils.exception.CloudRuntimeException: No cache DataStore in scope > id 1 type ZONE > 2013-07-05 11:28:22,076 INFO [cloud.vm.VirtualMachineManagerImpl] > (secstorage-1:null) Unable to contact resource. > com.cloud.exception.StorageUnavailableException: Resource [StoragePool:2] is > unreachable: Unable to create > Vol[1214|vm=1214|ROOT]:com.cloud.utils.exception.CloudRuntimeException: > com.cloud.utils.exception.CloudRuntimeException: No cache DataStore in scope > id 1 type ZONE > at > com.cloud.storage.VolumeManagerImpl.recreateVolume(VolumeManagerImpl.java:2493) > at > com.cloud.storage.VolumeManagerImpl.prepare(VolumeManagerImpl.java:2542) > at > com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:852) > at > com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:557) > at > com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:550) > at > com.cloud.storage.secondary.SecondaryStorageManagerImpl.startSecStorageVm(SecondaryStorageManagerImpl.java:260) > at > com.cloud.storage.secondary.SecondaryStorageManagerImpl.allocCapacity(SecondaryStorageManagerImpl.java:683) > at > com.cloud.storage.secondary.SecondaryStorageManagerImpl.expandPool(SecondaryStorageManagerImpl.java:1290) > at > com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:123) > at > com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:50) > at > com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:104) > at > com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33) > at > com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81) > at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at > java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:679) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira