Public bug reported: When glance is using rbd store, during image download, if delete call for the same image is attempted, delete fails with InUseByStore raised from the store but image data also gets deleted from the store and still image remains in active state.
Steps to reproduce: 1. Create image: $ glance image-create --name test-new-image --file gentoo_root.img --disk-format raw --container-format bare 2. Download Image and simultaneously delete that image in other terminal: $ glance image-download 1e1f9830-29e0-4901-8eff-ef05b7faf725 --file hhh.txt [Errno 32] Corrupt image download. Hash was bc0b615f1a34f1bde273f496fa6246d33dbd760ac20e7687e55c1d06ab070907bfe087b2a25bceca3cd4ba819b096286d6e4f28a5f27398ed7afb0b0ba18721a expected e4f21997407b9cb0df347f6eba2feaeb14c19f15cf784da06b78e1d5ff776a419535c894dea10a859fa72bcb234e94ada0fc86de0ff127bf9280eede8d473edb $ glance image-delete 1e1f9830-29e0-4901-8eff-ef05b7faf725 Unable to delete image '1e1f9830-29e0-4901-8eff-ef05b7faf725' because it is in use. 3. Try to Download the same Image again $ glance image-download 1e1f9830-29e0-4901-8eff-ef05b7faf725 --file hhh.txt Image 1e1f9830-29e0-4901-8eff-ef05b7faf725 has no data. Log: WARNING glance_store._drivers.rbd [None req-9d5aeaae-efc9-4d43-8496-2c742a52a4da admin admin] Remove image c0fbdea3-5f53-40d0-a5f9-f9ade42dbbd4 failed. It is in use.: rbd.ImageBusy: [errno 16] RBD image is busy (error removing image) Dec 06 11:40:41 pdeore-bobdev [email protected][813992]: WARNING glance.api.v2.images [None req-9d5aeaae-efc9-4d43-8496-2c742a52a4da admin admin] Image c0fbdea3-5f53-40d0-a5f9-f9ade42dbbd4 could not be deleted because it is in use: The image cannot be deleted because it is in use through the backend store outside of Glance.: glance_store.exceptions.InUseByStore: The image cannot be deleted because it is in use through the backend store outside of Glance. Dec 06 11:40:44 pdeore-bobdev [email protected][813991]: WARNING glance.image_cache.drivers.sqlite [None req-e71cfe15-f144-418b-b9f7-15afbece570b admin admin] ("Fetch of cache file failed (%(e)s), rolling back by moving '%(incomplete_path)s' to '%(invalid_path)s'", {'e': NotFound('RBD image c0fbdea3-5f53-40d0-a5f9-f9ade42dbbd4 does not exist'), 'incomplete_path': '/opt/stack/data/glance/cache/incomplete/c0fbdea3-5f53-40d0-a5f9-f9ade42dbbd4', 'invalid_path': '/opt/stack/data/glance/cache/invalid/c0fbdea3-5f53-40d0-a5f9-f9ade42dbbd4'}): glance_store.exceptions.NotFound: RBD image c0fbdea3-5f53-40d0-a5f9-f9ade42dbbd4 does not exist Dec 06 11:40:44 pdeore-bobdev [email protected][813991]: ERROR glance.image_cache [None req-e71cfe15-f144-418b-b9f7-15afbece570b admin admin] Exception encountered while tee'ing image 'c0fbdea3-5f53-40d0-a5f9-f9ade42dbbd4' into cache: RBD image c0fbdea3-5f53-40d0-a5f9-f9ade42dbbd4 does not exist. Continuing with response.: glance_store.exceptions.NotFound: RBD image c0fbdea3-5f53-40d0-a5f9-f9ade42dbbd4 does not exist Dec 06 11:40:44 pdeore-bobdev [email protected][813991]: ERROR glance.image_cache Traceback (most recent call last): Dec 06 11:40:44 pdeore-bobdev [email protected][813991]: ERROR glance.image_cache File "/usr/local/lib/python3.10/dist-packages/glance_store/_drivers/rbd.py", line 269, in __iter__ Dec 06 11:40:44 pdeore-bobdev [email protected][813991]: ERROR glance.image_cache with rbd.Image(ioctx, self.name, Dec 06 11:40:44 pdeore-bobdev [email protected][813991]: ERROR glance.image_cache File "rbd.pyx", line 2894, in rbd.Image.__init__ Dec 06 11:40:44 pdeore-bobdev [email protected][813991]: ERROR glance.image_cache rbd.ImageNotFound: [errno 2] RBD image not found (error opening image b'c0fbdea3-5f53-40d0-a5f9-f9ade42dbbd4' at snapshot b'snap') Dec 06 11:40:44 pdeore-bobdev [email protected][813991]: ERROR glance.image_cache Dec 06 11:40:44 pdeore-bobdev [email protected][813991]: ERROR glance.image_cache During handling of the above exception, another exception occurred: Dec 06 11:40:44 pdeore-bobdev [email protected][813991]: ERROR glance.image_cache Dec 06 11:40:44 pdeore-bobdev [email protected][813991]: ERROR glance.image_cache Traceback (most recent call last): Dec 06 11:40:44 pdeore-bobdev [email protected][813991]: ERROR glance.image_cache File "/opt/stack/glance/glance/image_cache/__init__.py", line 352, in cache_tee_iter Dec 06 11:40:44 pdeore-bobdev [email protected][813991]: ERROR glance.image_cache for chunk in image_iter: Dec 06 11:40:44 pdeore-bobdev [email protected][813991]: ERROR glance.image_cache File "/opt/stack/glance/glance/notifier.py", line 412, in _get_chunk_data_iterator Dec 06 11:40:44 pdeore-bobdev [email protected][813991]: ERROR glance.image_cache for chunk in data: Dec 06 11:40:44 pdeore-bobdev [email protected][813991]: ERROR glance.image_cache File "/usr/local/lib/python3.10/dist-packages/glance_store/_drivers/rbd.py", line 282, in __iter__ Dec 06 11:40:44 pdeore-bobdev [email protected][813991]: ERROR glance.image_cache raise exceptions.NotFound( Dec 06 11:40:44 pdeore-bobdev [email protected][813991]: ERROR glance.image_cache glance_store.exceptions.NotFound: RBD image c0fbdea3-5f53-40d0-a5f9-f9ade42dbbd4 does not exist Dec 06 11:40:44 pdeore-bobdev [email protected][813991]: ERROR glance.image_cache ** Affects: glance Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to Glance. https://bugs.launchpad.net/bugs/2045769 Title: Image is not getting deleted even after image data is deleted from the rbd store Status in Glance: New Bug description: When glance is using rbd store, during image download, if delete call for the same image is attempted, delete fails with InUseByStore raised from the store but image data also gets deleted from the store and still image remains in active state. Steps to reproduce: 1. Create image: $ glance image-create --name test-new-image --file gentoo_root.img --disk-format raw --container-format bare 2. Download Image and simultaneously delete that image in other terminal: $ glance image-download 1e1f9830-29e0-4901-8eff-ef05b7faf725 --file hhh.txt [Errno 32] Corrupt image download. Hash was bc0b615f1a34f1bde273f496fa6246d33dbd760ac20e7687e55c1d06ab070907bfe087b2a25bceca3cd4ba819b096286d6e4f28a5f27398ed7afb0b0ba18721a expected e4f21997407b9cb0df347f6eba2feaeb14c19f15cf784da06b78e1d5ff776a419535c894dea10a859fa72bcb234e94ada0fc86de0ff127bf9280eede8d473edb $ glance image-delete 1e1f9830-29e0-4901-8eff-ef05b7faf725 Unable to delete image '1e1f9830-29e0-4901-8eff-ef05b7faf725' because it is in use. 3. Try to Download the same Image again $ glance image-download 1e1f9830-29e0-4901-8eff-ef05b7faf725 --file hhh.txt Image 1e1f9830-29e0-4901-8eff-ef05b7faf725 has no data. Log: WARNING glance_store._drivers.rbd [None req-9d5aeaae-efc9-4d43-8496-2c742a52a4da admin admin] Remove image c0fbdea3-5f53-40d0-a5f9-f9ade42dbbd4 failed. It is in use.: rbd.ImageBusy: [errno 16] RBD image is busy (error removing image) Dec 06 11:40:41 pdeore-bobdev [email protected][813992]: WARNING glance.api.v2.images [None req-9d5aeaae-efc9-4d43-8496-2c742a52a4da admin admin] Image c0fbdea3-5f53-40d0-a5f9-f9ade42dbbd4 could not be deleted because it is in use: The image cannot be deleted because it is in use through the backend store outside of Glance.: glance_store.exceptions.InUseByStore: The image cannot be deleted because it is in use through the backend store outside of Glance. Dec 06 11:40:44 pdeore-bobdev [email protected][813991]: WARNING glance.image_cache.drivers.sqlite [None req-e71cfe15-f144-418b-b9f7-15afbece570b admin admin] ("Fetch of cache file failed (%(e)s), rolling back by moving '%(incomplete_path)s' to '%(invalid_path)s'", {'e': NotFound('RBD image c0fbdea3-5f53-40d0-a5f9-f9ade42dbbd4 does not exist'), 'incomplete_path': '/opt/stack/data/glance/cache/incomplete/c0fbdea3-5f53-40d0-a5f9-f9ade42dbbd4', 'invalid_path': '/opt/stack/data/glance/cache/invalid/c0fbdea3-5f53-40d0-a5f9-f9ade42dbbd4'}): glance_store.exceptions.NotFound: RBD image c0fbdea3-5f53-40d0-a5f9-f9ade42dbbd4 does not exist Dec 06 11:40:44 pdeore-bobdev [email protected][813991]: ERROR glance.image_cache [None req-e71cfe15-f144-418b-b9f7-15afbece570b admin admin] Exception encountered while tee'ing image 'c0fbdea3-5f53-40d0-a5f9-f9ade42dbbd4' into cache: RBD image c0fbdea3-5f53-40d0-a5f9-f9ade42dbbd4 does not exist. Continuing with response.: glance_store.exceptions.NotFound: RBD image c0fbdea3-5f53-40d0-a5f9-f9ade42dbbd4 does not exist Dec 06 11:40:44 pdeore-bobdev [email protected][813991]: ERROR glance.image_cache Traceback (most recent call last): Dec 06 11:40:44 pdeore-bobdev [email protected][813991]: ERROR glance.image_cache File "/usr/local/lib/python3.10/dist-packages/glance_store/_drivers/rbd.py", line 269, in __iter__ Dec 06 11:40:44 pdeore-bobdev [email protected][813991]: ERROR glance.image_cache with rbd.Image(ioctx, self.name, Dec 06 11:40:44 pdeore-bobdev [email protected][813991]: ERROR glance.image_cache File "rbd.pyx", line 2894, in rbd.Image.__init__ Dec 06 11:40:44 pdeore-bobdev [email protected][813991]: ERROR glance.image_cache rbd.ImageNotFound: [errno 2] RBD image not found (error opening image b'c0fbdea3-5f53-40d0-a5f9-f9ade42dbbd4' at snapshot b'snap') Dec 06 11:40:44 pdeore-bobdev [email protected][813991]: ERROR glance.image_cache Dec 06 11:40:44 pdeore-bobdev [email protected][813991]: ERROR glance.image_cache During handling of the above exception, another exception occurred: Dec 06 11:40:44 pdeore-bobdev [email protected][813991]: ERROR glance.image_cache Dec 06 11:40:44 pdeore-bobdev [email protected][813991]: ERROR glance.image_cache Traceback (most recent call last): Dec 06 11:40:44 pdeore-bobdev [email protected][813991]: ERROR glance.image_cache File "/opt/stack/glance/glance/image_cache/__init__.py", line 352, in cache_tee_iter Dec 06 11:40:44 pdeore-bobdev [email protected][813991]: ERROR glance.image_cache for chunk in image_iter: Dec 06 11:40:44 pdeore-bobdev [email protected][813991]: ERROR glance.image_cache File "/opt/stack/glance/glance/notifier.py", line 412, in _get_chunk_data_iterator Dec 06 11:40:44 pdeore-bobdev [email protected][813991]: ERROR glance.image_cache for chunk in data: Dec 06 11:40:44 pdeore-bobdev [email protected][813991]: ERROR glance.image_cache File "/usr/local/lib/python3.10/dist-packages/glance_store/_drivers/rbd.py", line 282, in __iter__ Dec 06 11:40:44 pdeore-bobdev [email protected][813991]: ERROR glance.image_cache raise exceptions.NotFound( Dec 06 11:40:44 pdeore-bobdev [email protected][813991]: ERROR glance.image_cache glance_store.exceptions.NotFound: RBD image c0fbdea3-5f53-40d0-a5f9-f9ade42dbbd4 does not exist Dec 06 11:40:44 pdeore-bobdev [email protected][813991]: ERROR glance.image_cache To manage notifications about this bug go to: https://bugs.launchpad.net/glance/+bug/2045769/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
