[ https://issues.apache.org/jira/browse/CLOUDSTACK-3498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13712834#comment-13712834 ]
Min Chen commented on CLOUDSTACK-3498: -------------------------------------- We should not even allow deleting a zone when there are zone-wide secondary storage and cache storage existing for the zone, this is a regression from our new schema change by removing secondary storage from hosts table. I will provide a fix to address this. > [Object_Store_Refactor] Deleting and adding back the zone does not create NFS > Cache store and Failure in Sytem VMs Start Up > --------------------------------------------------------------------------------------------------------------------------- > > Key: CLOUDSTACK-3498 > URL: https://issues.apache.org/jira/browse/CLOUDSTACK-3498 > Project: CloudStack > Issue Type: Bug > Security Level: Public(Anyone can view this level - this is the > default.) > Components: Management Server > Affects Versions: 4.2.0 > Environment: Latest build from ACS 4.2 branch. > Reporter: Sanjeev N > Assignee: Min Chen > Priority: Critical > Fix For: 4.2.0 > > Attachments: management-server.rar > > > Deleting a zone does not delete the NFS Cache store in that zone and adding > another zone with same NFS cache store details those were given in the > deleted zone does not create another NFS Cache store in the new zone. Hence > the Failure in System vms bring up. > Steps to Reproduce: > ================ > 1.Bring up CS with advanced zone say zone1. > 2.Add secondary storage with S3 provider and with NFS Cache Store > 3.Delete secondary storage > 4.Delete zone1 > 5.Add another zone say zone2 > 6.Add secondary storage with s3 provider and NFS Cache storage in Zone2 with > NFS Server and Path same as zone1 > Observations: > =========== > 1.Deleting zone did not delete NFS Cache store from image_store table. > 2.Adding NFS Cache store with same details in new zone did not create another > entry in image_store table for the new zone. > Impact: > ====== > CS downloads Routing template from S3 to NFS Cache store to bring up the > System VMs. Since CS did not find NFS Cache store in the new zone to copy the > routing template spinning system vms failed with following exception: > 2013-07-12 06:28:14,731 DEBUG [storage.motion.AncientDataMotionStrategy] > (consoleproxy-1:null) needCacheStorage true, dest at null dest role > Primarytemplate/tmpl/1/3/routing-3/systemvmtemplate-2013-06-25-master-kvm.qcow2.bz2 > src role Image > 2013-07-12 06:28:14,737 DEBUG [network.guru.PublicNetworkGuru] > (secstorage-1:null) public network deallocate network: networkId: 210, ip: > 10.147.48.3 > 2013-07-12 06:28:14,739 DEBUG [cache.allocator.StorageCacheRandomAllocator] > (consoleproxy-1:null) Can't find cache storage in zone: 3 > 2013-07-12 06:28:14,739 DEBUG [storage.motion.AncientDataMotionStrategy] > (consoleproxy-1:null) copy object failed: > com.cloud.utils.exception.CloudRuntimeException: No cache DataStore in scope > id 3 type ZONE > 2013-07-12 06:28:14,739 DEBUG [storage.motion.AncientDataMotionStrategy] > (consoleproxy-1:null) copy failed > com.cloud.utils.exception.CloudRuntimeException: > com.cloud.utils.exception.CloudRuntimeException: No cache DataStore in scope > id 3 type ZONE > at > org.apache.cloudstack.storage.motion.AncientDataMotionStrategy.copyObject(AncientDataMotionStrategy.java:204) > at > org.apache.cloudstack.storage.motion.AncientDataMotionStrategy.copyAsync(AncientDataMotionStrategy.java:351) > at > org.apache.cloudstack.storage.motion.DataMotionServiceImpl.copyAsync(DataMotionServiceImpl.java:55) > at > org.apache.cloudstack.storage.volume.VolumeServiceImpl.createBaseImageAsync(VolumeServiceImpl.java:390) > at > org.apache.cloudstack.storage.volume.VolumeServiceImpl.createVolumeFromTemplateAsync(VolumeServiceImpl.java:490) > at > com.cloud.storage.VolumeManagerImpl.recreateVolume(VolumeManagerImpl.java:2485) > at > com.cloud.storage.VolumeManagerImpl.prepare(VolumeManagerImpl.java:2542) > at > com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:851) > at > com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:557) > at > com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:550) > at > com.cloud.consoleproxy.ConsoleProxyManagerImpl.startProxy(ConsoleProxyManagerImpl.java:556) > at > com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:927) > at > com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1664) > at > com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:157) > at > com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:111) > at > com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33) > at > com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81) > at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at > java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:679) > 2013-07-12 06:28:14,752 DEBUG [cloud.storage.VolumeManagerImpl] > (consoleproxy-1:null) Unable to create > Vol[91|vm=79|ROOT]:com.cloud.utils.exception.CloudRuntimeException: > com.cloud.utils.exception.CloudRuntimeException: No cache DataStore in scope > id 3 type ZONE > 2013-07-12 06:28:14,752 INFO [cloud.vm.VirtualMachineManagerImpl] > (consoleproxy-1:null) Unable to contact resource. > com.cloud.exception.StorageUnavailableException: Resource [StoragePool:3] is > unreachable: Unable to create > Vol[91|vm=79|ROOT]:com.cloud.utils.exception.CloudRuntimeException: > com.cloud.utils.exception.CloudRuntimeException: No cache DataStore in scope > id 3 type ZONE > at > com.cloud.storage.VolumeManagerImpl.recreateVolume(VolumeManagerImpl.java:2493) > at > com.cloud.storage.VolumeManagerImpl.prepare(VolumeManagerImpl.java:2542) > at > com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:851) > at > com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:557) > at > com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:550) > at > com.cloud.consoleproxy.ConsoleProxyManagerImpl.startProxy(ConsoleProxyManagerImpl.java:556) > at > com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:927) > at > com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1664) > at > com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:157) > at > com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:111) > at > com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33) > at > com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81) > at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at > java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:679) > 2013-07-12 06:28:14,758 DEBUG [cloud.vm.VirtualMachineManagerImpl] > (consoleproxy-1:null) Cleaning up resources for the vm > VM[ConsoleProxy|v-79-VM] in Starting state > DB entries from Cloud table: > ====================== > mysql> select * from image_store where role='Image' and removed is null\G; > *************************** 1. row *************************** > id: 3 > name: objstore > image_provider_name: S3 > protocol: http > url: NULL > data_center_id: NULL > scope: REGION > role: Image > uuid: 0c444304-7e5f-4137-8da1-a812c3b81cf7 > parent: NULL > created: 2013-07-11 15:07:56 > removed: NULL > total_size: NULL > used_bytes: NULL > *************************** 2. row *************************** > id: 5 > name: obj_store > image_provider_name: S3 > protocol: http > url: NULL > data_center_id: NULL > scope: REGION > role: Image > uuid: 789b6388-ea95-41e6-98e6-e2ded85f119f > parent: NULL > created: 2013-07-12 10:23:09 > removed: NULL > total_size: NULL > used_bytes: NULL > 2 rows in set (0.00 sec) > mysql> select * from image_store where role='ImageCache' and removed is > null\G; > *************************** 1. row *************************** > id: 1 > name: nfs://10.147.28.7/export/home/sanjeev/sec_xen_os > image_provider_name: NFS > protocol: nfs > url: nfs://10.147.28.7/export/home/sanjeev/sec_xen_os > data_center_id: 1 > scope: ZONE > role: ImageCache > uuid: 8e43fa4b-cf0c-46c2-9e68-ead6bd481742 > parent: NULL > created: 2013-07-11 14:51:18 > removed: NULL > total_size: NULL > used_bytes: NULL > *************************** 2. row *************************** > id: 4 > name: nfs://10.147.28.7/export/home/sanjeev/sec_kvm_os > image_provider_name: NFS > protocol: nfs > url: nfs://10.147.28.7/export/home/sanjeev/sec_kvm_os > data_center_id: 2 > scope: ZONE > role: ImageCache > uuid: b42417d4-3e97-4f75-9439-25f97badcaa1 > parent: NULL > created: 2013-07-11 16:41:51 > removed: NULL > total_size: NULL > used_bytes: NULL > 2 rows in set (0.00 sec) > mysql> select * from data_center; > +----+---------+--------------------------------------+-------------+---------------+------+---------------+---------------+---------+---------+--------------------+-------------+--------------------+--------+-----------+-------------+---------------+------------------+-------------------+---------------+-----------------------+---------------+-------------------+------------------+--------------------------------------+---------------------------+--------------------------+---------------------+-------+---------+-------------+--------------+----------+----------+ > | id | name | uuid | description | dns1 > | dns2 | internal_dns1 | internal_dns2 | gateway | netmask | > router_mac_address | mac_address | guest_network_cidr | domain | domain_id | > networktype | dns_provider | gateway_provider | firewall_provider | > dhcp_provider | lb_provider | vpn_provider | userdata_provider | > allocation_state | zone_token | > is_security_group_enabled | is_local_storage_enabled | removed | > owner | created | lastUpdated | engine_state | ip6_dns1 | ip6_dns2 | > +----+---------+--------------------------------------+-------------+---------------+------+---------------+---------------+---------+---------+--------------------+-------------+--------------------+--------+-----------+-------------+---------------+------------------+-------------------+---------------+-----------------------+---------------+-------------------+------------------+--------------------------------------+---------------------------+--------------------------+---------------------+-------+---------+-------------+--------------+----------+----------+ > | 1 | Basic | d234e7a5-5cd0-4029-9b80-07a47f470e79 | NULL | > 10.103.128.16 | NULL | 10.103.128.16 | NULL | NULL | NULL | > 02:00:00:00:00:01 | 21 | NULL | NULL | NULL | > Basic | VirtualRouter | NULL | NULL | > VirtualRouter | ElasticLoadBalancerVm | NULL | VirtualRouter | > Enabled | 64c45795-5345-36e5-b52d-5530cd602a55 | > 1 | 0 | NULL | NULL | NULL | > NULL | Disabled | NULL | NULL | > | 2 | NULL | 45072eac-10d8-499d-9325-d4ee79e45026 | NULL | > 10.103.128.16 | NULL | 10.103.128.16 | NULL | NULL | NULL | > 02:00:00:00:00:01 | 19 | 10.1.1.0/24 | NULL | NULL | > Advanced | VirtualRouter | VirtualRouter | VirtualRouter | > VirtualRouter | VirtualRouter | VirtualRouter | VirtualRouter | > Disabled | b2359ba6-39c7-3dc6-a2ca-ac9091a6a06b | > 0 | 1 | 2013-07-12 10:09:14 | NULL | NULL | > NULL | Disabled | NULL | NULL | > | 3 | Adv-kvm | 37b35302-af3c-43b9-bc6c-acb17642b0d3 | NULL | > 10.103.128.16 | NULL | 10.103.128.16 | NULL | NULL | NULL | > 02:00:00:00:00:01 | 21 | 10.1.1.0/24 | NULL | NULL | > Advanced | VirtualRouter | VirtualRouter | VirtualRouter | > VirtualRouter | VirtualRouter | VirtualRouter | VirtualRouter | > Disabled | 8f2a7a84-dbfc-35e0-9201-f3066c65b414 | > 0 | 1 | NULL | NULL | NULL | > NULL | Disabled | NULL | NULL | > +----+---------+--------------------------------------+-------------+---------------+------+---------------+---------------+---------+---------+--------------------+-------------+--------------------+--------+-----------+-------------+---------------+------------------+-------------------+---------------+-----------------------+---------------+-------------------+------------------+--------------------------------------+---------------------------+--------------------------+---------------------+-------+---------+-------------+--------------+----------+----------+ > 3 rows in set (0.00 sec) > 1.Data_Center 2 was removed from the setup, however we can see the image > store with role image_cache for data_center 2 in the image_store table. > 2. Data_Center 3 was created after deleting DC 2 , and provided the same > image_store details as DC2, hence there is no image store with role > image_cache for DC3. > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira