Sanjeev N created CLOUDSTACK-2674: ------------------------------------- Summary: [Object_Store_Refactor]Secondary Storage garbage collector failed with NPE in case of S3 storage provider Key: CLOUDSTACK-2674 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-2674 Project: CloudStack Issue Type: Bug Security Level: Public (Anyone can view this level - this is the default.) Components: Management Server Affects Versions: 4.2.0 Environment: Latest build from object_store branch Reporter: Sanjeev N Priority: Critical Fix For: 4.2.0
Secondary Storage garbage collector failed with NPE in case of S3 storage provider Steps to Reproduce: ================ 1.Bring up CS in advanced zone with S3 as the secondary storage provider 2.After SSVM is up and running register ISO to CS 3.When the ISO is in ready state delete the iso 4.Wait for the Storage scavenger thread to run Observations: ============ After deleting the ISO , it is marked as destroyed in template_store_ref table and also got deleted from S3 object store. But when secondary storage garbage collector ran it found 1 template to clean up on secondary storage(Image Store not on ImageCache store) since the iso state is in destroyed state and tried to clean it up. Found NPE while trying to clean up from Image store: 013-05-24 14:11:01,976 DEBUG [cloud.storage.StorageManagerImpl] (StorageManager-Scavenger-1:null) Storage pool garbage collector found 0 templates to clean up in storage pool: pri_xen_os 2013-05-24 14:11:01,979 DEBUG [cloud.storage.StorageManagerImpl] (StorageManager-Scavenger-1:null) Storage pool garbage collector found 0 templates to clean up in storage pool: pri_xen_os2 2013-05-24 14:11:01,993 DEBUG [cloud.storage.StorageManagerImpl] (StorageManager-Scavenger-1:null) Secondary storage garbage collector found 1 templates to cleanup on secondary storage host: 9c5f76a7-507a-4ae1-b411-702e816bbf27 2013-05-24 14:11:01,999 WARN [cloud.storage.StorageManagerImpl] (StorageManager-Scavenger-1:null) problem cleaning up templates in secondary storage store 9c5f76a7-507a-4ae1-b411-702e816bbf27 java.lang.NullPointerException at com.cloud.template.TemplateManagerImpl.templateIsDeleteable(TemplateManagerImpl.java:952) at com.cloud.storage.StorageManagerImpl.cleanupSecondaryStorage(StorageManagerImpl.java:1136) at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125) at com.cloud.storage.StorageManagerImpl.cleanupStorage(StorageManagerImpl.java:1049) at com.cloud.storage.StorageManagerImpl$StorageGarbageCollector.run(StorageManagerImpl.java:1354) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:679) 2013-05-24 14:11:02,004 DEBUG [cloud.storage.StorageManagerImpl] (StorageManager-Scavenger-1:null) Secondary storage garbage collector found 0 volumes to cleanup on secondary storage host: 9c5f76a7-507a-4ae1-b411-702e816bbf27 mysql> select * from image_store\G; *************************** 1. row *************************** id: 1 name: nfs://10.147.28.7/export/home/sanjeev/sec_xen_os image_provider_name: NFS protocol: nfs url: nfs://10.147.28.7/export/home/sanjeev/sec_xen_os data_center_id: 1 scope: ZONE role: ImageCache uuid: f521e766-fb84-4aeb-a0ab-83c2ccdfd56a parent: NULL created: 2013-05-24 12:09:21 removed: NULL total_size: NULL *************************** 2. row *************************** id: 2 name: 9c5f76a7-507a-4ae1-b411-702e816bbf27 image_provider_name: S3 protocol: http url: NULL data_center_id: NULL scope: REGION role: Image uuid: 9c5f76a7-507a-4ae1-b411-702e816bbf27 parent: NULL created: 2013-05-24 12:09:21 removed: NULL total_size: NULL 2 rows in set (0.00 sec) ERROR: No query specified mysql> select * from template_store_ref where id=6\G; *************************** 1. row *************************** id: 6 store_id: 2 template_id: 202 created: 2013-05-24 16:45:43 last_updated: 2013-05-24 18:03:38 job_id: eeb1e5b0-b87f-4b8c-8452-3fe040ed3c86 download_pct: 100 size: 2551154688 store_role: Image physical_size: 2551154688 download_state: DOWNLOADED error_str: local_path: template/tmpl/2/202/202-2-72631a94-d107-336f-9bca-2901e767a622/WIN8_RTM.120725-1247_X86FRE_ENTERPRISE_EN-US_VL-HRM_CENA_X86FREV_EN-US_DV5.ISO install_path: template/tmpl/2/202/202-2-72631a94-d107-336f-9bca-2901e767a622 url: NULL state: Destroyed destroyed: 1 is_copy: 0 update_count: 5 updated: 2013-05-24 18:05:47 1 row in set (0.00 sec) ERROR: No query specified Storage garbage collector should only try to clean up the storage artifacts from Cache storage but not from the Image store. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira