Sanjeev N created CLOUDSTACK-3498:
-------------------------------------
Summary: [Object_Store_Refactor] Deleting and adding back the zone
does not create NFS Cache store and Failure in Sytem VMs Start Up
Key: CLOUDSTACK-3498
URL: https://issues.apache.org/jira/browse/CLOUDSTACK-3498
Project: CloudStack
Issue Type: Bug
Security Level: Public (Anyone can view this level - this is the default.)
Components: Management Server
Affects Versions: 4.2.0
Environment: Latest build from ACS 4.2 branch.
Reporter: Sanjeev N
Priority: Critical
Fix For: 4.2.0
Deleting a zone does not delete the NFS Cache store in that zone and adding
another zone with same NFS cache store details those were given in the deleted
zone does not create another NFS Cache store in the new zone. Hence the Failure
in System vms bring up.
Steps to Reproduce:
================
1.Bring up CS with advanced zone say zone1.
2.Add secondary storage with S3 provider and with NFS Cache Store
3.Delete secondary storage
4.Delete zone1
5.Add another zone say zone2
6.Add secondary storage with s3 provider and NFS Cache storage in Zone2 with
NFS Server and Path same as zone1
Observations:
===========
1.Deleting zone did not delete NFS Cache store from image_store table.
2.Adding NFS Cache store with same details in new zone did not create another
entry in image_store table for the new zone.
Impact:
======
CS downloads Routing template from S3 to NFS Cache store to bring up the System
VMs. Since CS did not find NFS Cache store in the new zone to copy the routing
template spinning system vms failed with following exception:
2013-07-12 06:28:14,731 DEBUG [storage.motion.AncientDataMotionStrategy]
(consoleproxy-1:null) needCacheStorage true, dest at null dest role
Primarytemplate/tmpl/1/3/routing-3/systemvmtemplate-2013-06-25-master-kvm.qcow2.bz2
src role Image
2013-07-12 06:28:14,737 DEBUG [network.guru.PublicNetworkGuru]
(secstorage-1:null) public network deallocate network: networkId: 210, ip:
10.147.48.3
2013-07-12 06:28:14,739 DEBUG [cache.allocator.StorageCacheRandomAllocator]
(consoleproxy-1:null) Can't find cache storage in zone: 3
2013-07-12 06:28:14,739 DEBUG [storage.motion.AncientDataMotionStrategy]
(consoleproxy-1:null) copy object failed:
com.cloud.utils.exception.CloudRuntimeException: No cache DataStore in scope id
3 type ZONE
2013-07-12 06:28:14,739 DEBUG [storage.motion.AncientDataMotionStrategy]
(consoleproxy-1:null) copy failed
com.cloud.utils.exception.CloudRuntimeException:
com.cloud.utils.exception.CloudRuntimeException: No cache DataStore in scope id
3 type ZONE
at
org.apache.cloudstack.storage.motion.AncientDataMotionStrategy.copyObject(AncientDataMotionStrategy.java:204)
at
org.apache.cloudstack.storage.motion.AncientDataMotionStrategy.copyAsync(AncientDataMotionStrategy.java:351)
at
org.apache.cloudstack.storage.motion.DataMotionServiceImpl.copyAsync(DataMotionServiceImpl.java:55)
at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.createBaseImageAsync(VolumeServiceImpl.java:390)
at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.createVolumeFromTemplateAsync(VolumeServiceImpl.java:490)
at
com.cloud.storage.VolumeManagerImpl.recreateVolume(VolumeManagerImpl.java:2485)
at
com.cloud.storage.VolumeManagerImpl.prepare(VolumeManagerImpl.java:2542)
at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:851)
at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:557)
at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:550)
at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.startProxy(ConsoleProxyManagerImpl.java:556)
at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:927)
at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1664)
at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:157)
at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:111)
at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
2013-07-12 06:28:14,752 DEBUG [cloud.storage.VolumeManagerImpl]
(consoleproxy-1:null) Unable to create
Vol[91|vm=79|ROOT]:com.cloud.utils.exception.CloudRuntimeException:
com.cloud.utils.exception.CloudRuntimeException: No cache DataStore in scope id
3 type ZONE
2013-07-12 06:28:14,752 INFO [cloud.vm.VirtualMachineManagerImpl]
(consoleproxy-1:null) Unable to contact resource.
com.cloud.exception.StorageUnavailableException: Resource [StoragePool:3] is
unreachable: Unable to create
Vol[91|vm=79|ROOT]:com.cloud.utils.exception.CloudRuntimeException:
com.cloud.utils.exception.CloudRuntimeException: No cache DataStore in scope id
3 type ZONE
at
com.cloud.storage.VolumeManagerImpl.recreateVolume(VolumeManagerImpl.java:2493)
at
com.cloud.storage.VolumeManagerImpl.prepare(VolumeManagerImpl.java:2542)
at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:851)
at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:557)
at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:550)
at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.startProxy(ConsoleProxyManagerImpl.java:556)
at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:927)
at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1664)
at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:157)
at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:111)
at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
2013-07-12 06:28:14,758 DEBUG [cloud.vm.VirtualMachineManagerImpl]
(consoleproxy-1:null) Cleaning up resources for the vm VM[ConsoleProxy|v-79-VM]
in Starting state
DB entries from Cloud table:
======================
mysql> select * from image_store where role='Image' and removed is null\G;
*************************** 1. row ***************************
id: 3
name: objstore
image_provider_name: S3
protocol: http
url: NULL
data_center_id: NULL
scope: REGION
role: Image
uuid: 0c444304-7e5f-4137-8da1-a812c3b81cf7
parent: NULL
created: 2013-07-11 15:07:56
removed: NULL
total_size: NULL
used_bytes: NULL
*************************** 2. row ***************************
id: 5
name: obj_store
image_provider_name: S3
protocol: http
url: NULL
data_center_id: NULL
scope: REGION
role: Image
uuid: 789b6388-ea95-41e6-98e6-e2ded85f119f
parent: NULL
created: 2013-07-12 10:23:09
removed: NULL
total_size: NULL
used_bytes: NULL
2 rows in set (0.00 sec)
mysql> select * from image_store where role='ImageCache' and removed is null\G;
*************************** 1. row ***************************
id: 1
name: nfs://10.147.28.7/export/home/sanjeev/sec_xen_os
image_provider_name: NFS
protocol: nfs
url: nfs://10.147.28.7/export/home/sanjeev/sec_xen_os
data_center_id: 1
scope: ZONE
role: ImageCache
uuid: 8e43fa4b-cf0c-46c2-9e68-ead6bd481742
parent: NULL
created: 2013-07-11 14:51:18
removed: NULL
total_size: NULL
used_bytes: NULL
*************************** 2. row ***************************
id: 4
name: nfs://10.147.28.7/export/home/sanjeev/sec_kvm_os
image_provider_name: NFS
protocol: nfs
url: nfs://10.147.28.7/export/home/sanjeev/sec_kvm_os
data_center_id: 2
scope: ZONE
role: ImageCache
uuid: b42417d4-3e97-4f75-9439-25f97badcaa1
parent: NULL
created: 2013-07-11 16:41:51
removed: NULL
total_size: NULL
used_bytes: NULL
2 rows in set (0.00 sec)
mysql> select * from data_center;
+----+---------+--------------------------------------+-------------+---------------+------+---------------+---------------+---------+---------+--------------------+-------------+--------------------+--------+-----------+-------------+---------------+------------------+-------------------+---------------+-----------------------+---------------+-------------------+------------------+--------------------------------------+---------------------------+--------------------------+---------------------+-------+---------+-------------+--------------+----------+----------+
| id | name | uuid | description | dns1
| dns2 | internal_dns1 | internal_dns2 | gateway | netmask |
router_mac_address | mac_address | guest_network_cidr | domain | domain_id |
networktype | dns_provider | gateway_provider | firewall_provider |
dhcp_provider | lb_provider | vpn_provider | userdata_provider |
allocation_state | zone_token |
is_security_group_enabled | is_local_storage_enabled | removed |
owner | created | lastUpdated | engine_state | ip6_dns1 | ip6_dns2 |
+----+---------+--------------------------------------+-------------+---------------+------+---------------+---------------+---------+---------+--------------------+-------------+--------------------+--------+-----------+-------------+---------------+------------------+-------------------+---------------+-----------------------+---------------+-------------------+------------------+--------------------------------------+---------------------------+--------------------------+---------------------+-------+---------+-------------+--------------+----------+----------+
| 1 | Basic | d234e7a5-5cd0-4029-9b80-07a47f470e79 | NULL |
10.103.128.16 | NULL | 10.103.128.16 | NULL | NULL | NULL |
02:00:00:00:00:01 | 21 | NULL | NULL | NULL |
Basic | VirtualRouter | NULL | NULL |
VirtualRouter | ElasticLoadBalancerVm | NULL | VirtualRouter |
Enabled | 64c45795-5345-36e5-b52d-5530cd602a55 |
1 | 0 | NULL | NULL | NULL | NULL
| Disabled | NULL | NULL |
| 2 | NULL | 45072eac-10d8-499d-9325-d4ee79e45026 | NULL |
10.103.128.16 | NULL | 10.103.128.16 | NULL | NULL | NULL |
02:00:00:00:00:01 | 19 | 10.1.1.0/24 | NULL | NULL |
Advanced | VirtualRouter | VirtualRouter | VirtualRouter |
VirtualRouter | VirtualRouter | VirtualRouter | VirtualRouter |
Disabled | b2359ba6-39c7-3dc6-a2ca-ac9091a6a06b |
0 | 1 | 2013-07-12 10:09:14 | NULL | NULL | NULL
| Disabled | NULL | NULL |
| 3 | Adv-kvm | 37b35302-af3c-43b9-bc6c-acb17642b0d3 | NULL |
10.103.128.16 | NULL | 10.103.128.16 | NULL | NULL | NULL |
02:00:00:00:00:01 | 21 | 10.1.1.0/24 | NULL | NULL |
Advanced | VirtualRouter | VirtualRouter | VirtualRouter |
VirtualRouter | VirtualRouter | VirtualRouter | VirtualRouter |
Disabled | 8f2a7a84-dbfc-35e0-9201-f3066c65b414 |
0 | 1 | NULL | NULL | NULL | NULL
| Disabled | NULL | NULL |
+----+---------+--------------------------------------+-------------+---------------+------+---------------+---------------+---------+---------+--------------------+-------------+--------------------+--------+-----------+-------------+---------------+------------------+-------------------+---------------+-----------------------+---------------+-------------------+------------------+--------------------------------------+---------------------------+--------------------------+---------------------+-------+---------+-------------+--------------+----------+----------+
3 rows in set (0.00 sec)
1.Data_Center 2 was removed from the setup, however we can see the image store
with role image_cache for data_center 2 in the image_store table.
2. Data_Center 3 was created after deleting DC 2 , and provided the same
image_store details as DC2, hence there is no image store with role image_cache
for DC3.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
