Public bug reported: Bug Description:
Glance over Cinder/NFS configuration randomly fails with the following message when create/delete image operations are performed concurrently. This happens because the current lock mechanism in glance_store that prevents race conditions on volume mount/unmount [1][2] is local to the process, and fails when multiple Glance workers run concurrently. [1] https://opendev.org/openstack/glance_store/src/commit/9a5271ccddbe01256be0819c1a74fbfc93ab0a2d/glance_store/common/fs_mount.py#L181 [2] https://opendev.org/openstack/glance_store/src/commit/8ac98ddf6170f66d4bdefac159b633c713a75dd5/glance_store/common/fs_mount.py#L242 Jun 03 19:37:41.316051 np0041005539 [email protected][90321]: WARNING glance.image_cache.drivers.centralized_db [None req-3aff3146-9e2a-4de4-9f7b-4d8d622c03da tempest-ImageDependencyTests-1910387855 tempest-ImageDependencyTests-1910387855-project-member] ("Fetch of cache file failed (%(e)s), rolling back by moving '%(incomplete_path)s' to '%(invalid_path)s'", {'e': FileExistsError(17, 'File exists'), 'incomplete_path': '/opt/stack/data/glance/cache/incomplete/bc2f9475-0d11-4f96-b8e6-b22fdce5cace', 'invalid_path': '/opt/stack/data/glance/cache/invalid/bc2f9475-0d11-4f96-b8e6-b22fdce5cace'}): FileExistsError: [Errno 17] File exists: '/opt/stack/data/glance/images/mnt/8db6edfa5db8c4a4b76e2ad4526a0f97806edb8c809ce63ffcdb8d1a551aa57a' Jun 03 19:37:41.348228 np0041005539 [email protected][90321]: ERROR glance.image_cache [None req-3aff3146-9e2a-4de4-9f7b-4d8d622c03da tempest-ImageDependencyTests-1910387855 tempest-ImageDependencyTests-1910387855-project-member] Exception encountered while tee'ing image 'bc2f9475-0d11-4f96-b8e6-b22fdce5cace' into cache: [Errno 17] File exists: '/opt/stack/data/glance/images/mnt/8db6edfa5db8c4a4b76e2ad4526a0f97806edb8c809ce63ffcdb8d1a551aa57a'. Continuing with response.: FileExistsError: [Errno 17] File exists: '/opt/stack/data/glance/images/mnt/8db6edfa5db8c4a4b76e2ad4526a0f97806edb8c809ce63ffcdb8d1a551aa57a' Jun 03 19:37:41.348228 np0041005539 [email protected][90321]: ERROR glance.image_cache Traceback (most recent call last): Jun 03 19:37:41.348228 np0041005539 [email protected][90321]: ERROR glance.image_cache File "/opt/stack/glance/glance/image_cache/__init__.py", line 360, in cache_tee_iter Jun 03 19:37:41.348228 np0041005539 [email protected][90321]: ERROR glance.image_cache for chunk in image_iter: Jun 03 19:37:41.348228 np0041005539 [email protected][90321]: ERROR glance.image_cache File "/opt/stack/glance/glance/notifier.py", line 412, in _get_chunk_data_iterator Jun 03 19:37:41.348228 np0041005539 [email protected][90321]: ERROR glance.image_cache for chunk in data: Jun 03 19:37:41.348228 np0041005539 [email protected][90321]: ERROR glance.image_cache File "/opt/stack/data/venv/lib/python3.12/site-packages/glance_store/_drivers/cinder/store.py", line 813, in _cinder_volume_data_iterator Jun 03 19:37:41.348228 np0041005539 [email protected][90321]: ERROR glance.image_cache with self._open_cinder_volume(client, volume, 'rb') as fp: Jun 03 19:37:41.348228 np0041005539 [email protected][90321]: ERROR glance.image_cache File "/usr/lib/python3.12/contextlib.py", line 137, in __enter__ Jun 03 19:37:41.348228 np0041005539 [email protected][90321]: ERROR glance.image_cache return next(self.gen) Jun 03 19:37:41.348228 np0041005539 [email protected][90321]: ERROR glance.image_cache ^^^^^^^^^^^^^^ Jun 03 19:37:41.348228 np0041005539 [email protected][90321]: ERROR glance.image_cache File "/opt/stack/data/venv/lib/python3.12/site-packages/glance_store/_drivers/cinder/store.py", line 770, in _open_cinder_volume Jun 03 19:37:41.348228 np0041005539 [email protected][90321]: ERROR glance.image_cache device = conn.connect_volume(volume) Jun 03 19:37:41.348228 np0041005539 [email protected][90321]: ERROR glance.image_cache ^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 03 19:37:41.348228 np0041005539 [email protected][90321]: ERROR glance.image_cache File "/opt/stack/data/venv/lib/python3.12/site-packages/glance_store/_drivers/cinder/nfs.py", line 94, in connect_volume Jun 03 19:37:41.348840 np0041005539 [email protected][90321]: ERROR glance.image_cache device = connect_volume_nfs() Jun 03 19:37:41.348840 np0041005539 [email protected][90321]: ERROR glance.image_cache ^^^^^^^^^^^^^^^^^^^^ Jun 03 19:37:41.348840 np0041005539 [email protected][90321]: ERROR glance.image_cache File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_concurrency/lockutils.py", line 415, in inner Jun 03 19:37:41.348840 np0041005539 [email protected][90321]: ERROR glance.image_cache return f(*args, **kwargs) Jun 03 19:37:41.348840 np0041005539 [email protected][90321]: ERROR glance.image_cache ^^^^^^^^^^^^^^^^^^ Jun 03 19:37:41.348840 np0041005539 [email protected][90321]: ERROR glance.image_cache File "/opt/stack/data/venv/lib/python3.12/site-packages/glance_store/_drivers/cinder/nfs.py", line 89, in connect_volume_nfs Jun 03 19:37:41.348840 np0041005539 [email protected][90321]: ERROR glance.image_cache mount.mount( Jun 03 19:37:41.348840 np0041005539 [email protected][90321]: ERROR glance.image_cache File "/opt/stack/data/venv/lib/python3.12/site-packages/glance_store/common/fs_mount.py", line 358, in mount Jun 03 19:37:41.348840 np0041005539 [email protected][90321]: ERROR glance.image_cache mount_state.mount(fstype, export, vol_name, mountpoint, host, Jun 03 19:37:41.348840 np0041005539 [email protected][90321]: ERROR glance.image_cache File "/opt/stack/data/venv/lib/python3.12/site-packages/glance_store/common/fs_mount.py", line 247, in mount Jun 03 19:37:41.348840 np0041005539 [email protected][90321]: ERROR glance.image_cache os.makedirs(mountpoint) Jun 03 19:37:41.348840 np0041005539 [email protected][90321]: ERROR glance.image_cache File "<frozen os>", line 225, in makedirs Jun 03 19:37:41.348840 np0041005539 [email protected][90321]: ERROR glance.image_cache FileExistsError: [Errno 17] File exists: '/opt/stack/data/glance/images/mnt/8db6edfa5db8c4a4b76e2ad4526a0f97806edb8c809ce63ffcdb8d1a551aa57a' Jun 03 19:37:41.348840 np0041005539 [email protected][90321]: ERROR glance.image_cache Steps to reproduce the issue: Try to run multiple create/delete image operations concurrently in an environment configured with Glance over Cinder/NFS. Expected behavior: All image create/delete operations should complete without errors. Bug impact: * Tempest tests are failing; * Image create behavior is unreliable for environments running Glance over Cinder/NFS Known workaround: For the Glance service, set the number of workers to 1. This has a significant performance impact, but currently it is the only way to mitigate the issue. ** Affects: glance Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to Glance. https://bugs.launchpad.net/bugs/2115906 Title: Glance image import fails on Glance over Cinder/NFS scenario Status in Glance: New Bug description: Bug Description: Glance over Cinder/NFS configuration randomly fails with the following message when create/delete image operations are performed concurrently. This happens because the current lock mechanism in glance_store that prevents race conditions on volume mount/unmount [1][2] is local to the process, and fails when multiple Glance workers run concurrently. [1] https://opendev.org/openstack/glance_store/src/commit/9a5271ccddbe01256be0819c1a74fbfc93ab0a2d/glance_store/common/fs_mount.py#L181 [2] https://opendev.org/openstack/glance_store/src/commit/8ac98ddf6170f66d4bdefac159b633c713a75dd5/glance_store/common/fs_mount.py#L242 Jun 03 19:37:41.316051 np0041005539 [email protected][90321]: WARNING glance.image_cache.drivers.centralized_db [None req-3aff3146-9e2a-4de4-9f7b-4d8d622c03da tempest-ImageDependencyTests-1910387855 tempest-ImageDependencyTests-1910387855-project-member] ("Fetch of cache file failed (%(e)s), rolling back by moving '%(incomplete_path)s' to '%(invalid_path)s'", {'e': FileExistsError(17, 'File exists'), 'incomplete_path': '/opt/stack/data/glance/cache/incomplete/bc2f9475-0d11-4f96-b8e6-b22fdce5cace', 'invalid_path': '/opt/stack/data/glance/cache/invalid/bc2f9475-0d11-4f96-b8e6-b22fdce5cace'}): FileExistsError: [Errno 17] File exists: '/opt/stack/data/glance/images/mnt/8db6edfa5db8c4a4b76e2ad4526a0f97806edb8c809ce63ffcdb8d1a551aa57a' Jun 03 19:37:41.348228 np0041005539 [email protected][90321]: ERROR glance.image_cache [None req-3aff3146-9e2a-4de4-9f7b-4d8d622c03da tempest-ImageDependencyTests-1910387855 tempest-ImageDependencyTests-1910387855-project-member] Exception encountered while tee'ing image 'bc2f9475-0d11-4f96-b8e6-b22fdce5cace' into cache: [Errno 17] File exists: '/opt/stack/data/glance/images/mnt/8db6edfa5db8c4a4b76e2ad4526a0f97806edb8c809ce63ffcdb8d1a551aa57a'. Continuing with response.: FileExistsError: [Errno 17] File exists: '/opt/stack/data/glance/images/mnt/8db6edfa5db8c4a4b76e2ad4526a0f97806edb8c809ce63ffcdb8d1a551aa57a' Jun 03 19:37:41.348228 np0041005539 [email protected][90321]: ERROR glance.image_cache Traceback (most recent call last): Jun 03 19:37:41.348228 np0041005539 [email protected][90321]: ERROR glance.image_cache File "/opt/stack/glance/glance/image_cache/__init__.py", line 360, in cache_tee_iter Jun 03 19:37:41.348228 np0041005539 [email protected][90321]: ERROR glance.image_cache for chunk in image_iter: Jun 03 19:37:41.348228 np0041005539 [email protected][90321]: ERROR glance.image_cache File "/opt/stack/glance/glance/notifier.py", line 412, in _get_chunk_data_iterator Jun 03 19:37:41.348228 np0041005539 [email protected][90321]: ERROR glance.image_cache for chunk in data: Jun 03 19:37:41.348228 np0041005539 [email protected][90321]: ERROR glance.image_cache File "/opt/stack/data/venv/lib/python3.12/site-packages/glance_store/_drivers/cinder/store.py", line 813, in _cinder_volume_data_iterator Jun 03 19:37:41.348228 np0041005539 [email protected][90321]: ERROR glance.image_cache with self._open_cinder_volume(client, volume, 'rb') as fp: Jun 03 19:37:41.348228 np0041005539 [email protected][90321]: ERROR glance.image_cache File "/usr/lib/python3.12/contextlib.py", line 137, in __enter__ Jun 03 19:37:41.348228 np0041005539 [email protected][90321]: ERROR glance.image_cache return next(self.gen) Jun 03 19:37:41.348228 np0041005539 [email protected][90321]: ERROR glance.image_cache ^^^^^^^^^^^^^^ Jun 03 19:37:41.348228 np0041005539 [email protected][90321]: ERROR glance.image_cache File "/opt/stack/data/venv/lib/python3.12/site-packages/glance_store/_drivers/cinder/store.py", line 770, in _open_cinder_volume Jun 03 19:37:41.348228 np0041005539 [email protected][90321]: ERROR glance.image_cache device = conn.connect_volume(volume) Jun 03 19:37:41.348228 np0041005539 [email protected][90321]: ERROR glance.image_cache ^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 03 19:37:41.348228 np0041005539 [email protected][90321]: ERROR glance.image_cache File "/opt/stack/data/venv/lib/python3.12/site-packages/glance_store/_drivers/cinder/nfs.py", line 94, in connect_volume Jun 03 19:37:41.348840 np0041005539 [email protected][90321]: ERROR glance.image_cache device = connect_volume_nfs() Jun 03 19:37:41.348840 np0041005539 [email protected][90321]: ERROR glance.image_cache ^^^^^^^^^^^^^^^^^^^^ Jun 03 19:37:41.348840 np0041005539 [email protected][90321]: ERROR glance.image_cache File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_concurrency/lockutils.py", line 415, in inner Jun 03 19:37:41.348840 np0041005539 [email protected][90321]: ERROR glance.image_cache return f(*args, **kwargs) Jun 03 19:37:41.348840 np0041005539 [email protected][90321]: ERROR glance.image_cache ^^^^^^^^^^^^^^^^^^ Jun 03 19:37:41.348840 np0041005539 [email protected][90321]: ERROR glance.image_cache File "/opt/stack/data/venv/lib/python3.12/site-packages/glance_store/_drivers/cinder/nfs.py", line 89, in connect_volume_nfs Jun 03 19:37:41.348840 np0041005539 [email protected][90321]: ERROR glance.image_cache mount.mount( Jun 03 19:37:41.348840 np0041005539 [email protected][90321]: ERROR glance.image_cache File "/opt/stack/data/venv/lib/python3.12/site-packages/glance_store/common/fs_mount.py", line 358, in mount Jun 03 19:37:41.348840 np0041005539 [email protected][90321]: ERROR glance.image_cache mount_state.mount(fstype, export, vol_name, mountpoint, host, Jun 03 19:37:41.348840 np0041005539 [email protected][90321]: ERROR glance.image_cache File "/opt/stack/data/venv/lib/python3.12/site-packages/glance_store/common/fs_mount.py", line 247, in mount Jun 03 19:37:41.348840 np0041005539 [email protected][90321]: ERROR glance.image_cache os.makedirs(mountpoint) Jun 03 19:37:41.348840 np0041005539 [email protected][90321]: ERROR glance.image_cache File "<frozen os>", line 225, in makedirs Jun 03 19:37:41.348840 np0041005539 [email protected][90321]: ERROR glance.image_cache FileExistsError: [Errno 17] File exists: '/opt/stack/data/glance/images/mnt/8db6edfa5db8c4a4b76e2ad4526a0f97806edb8c809ce63ffcdb8d1a551aa57a' Jun 03 19:37:41.348840 np0041005539 [email protected][90321]: ERROR glance.image_cache Steps to reproduce the issue: Try to run multiple create/delete image operations concurrently in an environment configured with Glance over Cinder/NFS. Expected behavior: All image create/delete operations should complete without errors. Bug impact: * Tempest tests are failing; * Image create behavior is unreliable for environments running Glance over Cinder/NFS Known workaround: For the Glance service, set the number of workers to 1. This has a significant performance impact, but currently it is the only way to mitigate the issue. To manage notifications about this bug go to: https://bugs.launchpad.net/glance/+bug/2115906/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
