Public bug reported: With glance v2 and swift as the store, glance is unable to delete images. Tested with Queens and Rocky.
Code snippit: import glanceclient.v2 as glance_clientv2 glance = glance_clientv2.Client("2", session=keystone.session) image = glance.images.create(name=image_name, disk_format="qcow2", visibility="public", container_format="bare") glance.images.upload(image.id, open(local_path, 'rb')) glance.images.delete(image.id) Throws: glanceclient.exc.HTTPInternalServerError: 500 Internal Server Error: The server has either erred or is incapable of performing the requested operation. (HTTP 500) Leads to the following error in glance-api.log 2018-08-29 21:49:03.367 21964 INFO swiftclient [req-4f5b43a9-c3f1-4c71-b351-d08c2d221860 094897deb7424a4da2caccacc8c9d4ff ce38b0769f8c4037b641f8296bfd858b - bbda55ddd16e4faca419e2a13a113176 bbda55ddd16e4faca419e2a13a113176] REQ: curl -i http://10.5.0.34:8080/v1/AUTH_d213d2090cf948a0ba07e14491d58fc7/glance/02a6c228-249c-41f4-b8e3-04220a1d7529 -I -H "X-Auth-Token: gAAAAABbhxTOAmCn..." 2018-08-29 21:49:03.368 21964 INFO swiftclient [req-4f5b43a9-c3f1-4c71-b351-d08c2d221860 094897deb7424a4da2caccacc8c9d4ff ce38b0769f8c4037b641f8296bfd858b - bbda55ddd16e4faca419e2a13a113176 bbda55ddd16e4faca419e2a13a113176] RESP STATUS: 403 Forbidden 2018-08-29 21:49:03.368 21964 INFO swiftclient [req-4f5b43a9-c3f1-4c71-b351-d08c2d221860 094897deb7424a4da2caccacc8c9d4ff ce38b0769f8c4037b641f8296bfd858b - bbda55ddd16e4faca419e2a13a113176 bbda55ddd16e4faca419e2a13a113176] RESP HEADERS: {u'Date': u'Wed, 29 Aug 2018 21:49:03 GMT', u'Content-Length': u'73', u'Content-Type': u'text/html; charset=UTF-8', u'X-Openstack-Request-Id': u'tx3c78666a736e42a4b32a3-005b8714ce', u'X-Trans-Id': u'tx3c78666a736e42a4b32a3-005b8714ce'} 2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi [req-4f5b43a9-c3f1-4c71-b351-d08c2d221860 094897deb7424a4da2caccacc8c9d4ff ce38b0769f8c4037b641f8296bfd858b - bbda55ddd16e4faca419e2a13a113176 bbda55ddd16e4faca419e2a13a113176] Caught error: Object HEAD failed: http://10.5.0.34:8080/v1/AUTH_d213d2090cf948a0ba07e14491d58fc7/glance/02a6c228-249c-41f4-b8e3-04220a1d7529 403 Forbidden: ClientException: Object HEAD failed: http://10.5.0.34:8080/v1/AUTH_d213d2090cf948a0ba07e14491d58fc7/glance/02a6c228-249c-41f4-b8e3-04220a1d7529 403 Forbidden 2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi Traceback (most recent call last): 2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi File "/usr/lib/python2.7/dist-packages/glance/common/wsgi.py", line 1274, in __call__ 2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi request, **action_args) 2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi File "/usr/lib/python2.7/dist-packages/glance/common/wsgi.py", line 1317, in dispatch 2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi return method(*args, **kwargs) 2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi File "/usr/lib/python2.7/dist-packages/glance/common/utils.py", line 417, in wrapped 2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi return func(self, req, *args, **kwargs) 2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi File "/usr/lib/python2.7/dist-packages/glance/api/v2/images.py", line 343, in delete 2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi image.delete() 2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi File "/usr/lib/python2.7/dist-packages/glance/domain/proxy.py", line 189, in delete 2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi self.base.delete() 2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi File "/usr/lib/python2.7/dist-packages/glance/domain/proxy.py", line 189, in delete 2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi self.base.delete() 2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi File "/usr/lib/python2.7/dist-packages/glance/api/policy.py", line 172, in delete 2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi return self.image.delete() 2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi File "/usr/lib/python2.7/dist-packages/glance/domain/proxy.py", line 189, in delete 2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi self.base.delete() 2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi File "/usr/lib/python2.7/dist-packages/glance/location.py", line 431, in delete 2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi location) 2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi File "/usr/lib/python2.7/dist-packages/glance/common/store_utils.py", line 123, in delete_image_location_from_backend 2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi safe_delete_from_backend(context, image_id, location) 2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi File "/usr/lib/python2.7/dist-packages/glance/common/store_utils.py", line 56, in safe_delete_from_backend 2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi context=context) 2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi File "/usr/lib/python2.7/dist-packages/glance_store/backend.py", line 409, in delete_from_backend 2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi return store.delete(loc, context=context) 2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi File "/usr/lib/python2.7/dist-packages/glance_store/capabilities.py", line 225, in op_checker 2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi return store_op_fun(store, *args, **kwargs) 2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi File "/usr/lib/python2.7/dist-packages/glance_store/_drivers/swift/store.py", line 1093, in delete 2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi location.container, location.obj) 2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1826, in head_object 2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi query_string=query_string) 2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1722, in _retry 2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi service_token=self.service_token, **kwargs) 2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1252, in head_object 2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi raise ClientException.from_response(resp, 'Object HEAD failed', body) 2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi ClientException: Object HEAD failed: http://10.5.0.34:8080/v1/AUTH_d213d2090cf948a0ba07e14491d58fc7/glance/02a6c228-249c-41f4-b8e3-04220a1d7529 403 Forbidden 2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi 2018-08-29 21:49:03.376 21964 INFO eventlet.wsgi.server [req-4f5b43a9-c3f1-4c71-b351-d08c2d221860 094897deb7424a4da2caccacc8c9d4ff ce38b0769f8c4037b641f8296bfd858b - bbda55ddd16e4faca419e2a13a113176 bbda55ddd16e4faca419e2a13a113176] 10.5.0.12 - - [29/Aug/2018 21:49:03] "DELETE /v2/images/02a6c228-249c-41f4-b8e3-04220a1d7529 HTTP/1.1" 500 454 2.932388 Tested with openstack client for create and delete with same results. As the glance user using the swift client directly, the initial image part is not readable: $ swift list glance 02a6c228-249c-41f4-b8e3-04220a1d7529 02a6c228-249c-41f4-b8e3-04220a1d7529-00001 $ swift stat glance 02a6c228-249c-41f4-b8e3-04220a1d7529-00001 Account: AUTH_d213d2090cf948a0ba07e14491d58fc7 Container: glance Object: 02a6c228-249c-41f4-b8e3-04220a1d7529-00001 Content Type: application/octet-stream Content Length: 12716032 Last Modified: Wed, 29 Aug 2018 21:49:00 GMT ETag: 443b7623e27ecf03dc9e01ee93f67afe Accept-Ranges: bytes X-Timestamp: 1535579339.25431 X-Trans-Id: txd57d93cc856846f98015f-005b87155d X-Openstack-Request-Id: txd57d93cc856846f98015f-005b87155d $ swift stat glance 02a6c228-249c-41f4-b8e3-04220a1d7529 Object HEAD failed: http://10.5.0.34:8080/v1/AUTH_d213d2090cf948a0ba07e14491d58fc7/glance/02a6c228-249c-41f4-b8e3-04220a1d7529 403 Forbidden Failed Transaction ID: tx088744539df247ec8e2c0-005b871566 Note the glance container has no ACLs set: $ swift stat glance Account: AUTH_d213d2090cf948a0ba07e14491d58fc7 Container: glance Objects: 2 Bytes: 12716032 Read ACL: Write ACL: Sync To: Sync Key: Accept-Ranges: bytes X-Storage-Policy: Policy-0 Last-Modified: Wed, 29 Aug 2018 21:49:00 GMT X-Timestamp: 1535579339.45744 X-Trans-Id: txdc539510bdc84b7f84f28-005b87154b Content-Type: application/json; charset=utf-8 X-Openstack-Request-Id: txdc539510bdc84b7f84f28-005b87154b After adding read ACLs to the container, the image is deleteable swift post glance --read-acl ".r:*,.rlistings" Glance v2 should create its swift store container with correct read ACLS. ** Affects: glance Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to Glance. https://bugs.launchpad.net/bugs/1789748 Title: Glance should set container ACLs when using swift store Status in Glance: New Bug description: With glance v2 and swift as the store, glance is unable to delete images. Tested with Queens and Rocky. Code snippit: import glanceclient.v2 as glance_clientv2 glance = glance_clientv2.Client("2", session=keystone.session) image = glance.images.create(name=image_name, disk_format="qcow2", visibility="public", container_format="bare") glance.images.upload(image.id, open(local_path, 'rb')) glance.images.delete(image.id) Throws: glanceclient.exc.HTTPInternalServerError: 500 Internal Server Error: The server has either erred or is incapable of performing the requested operation. (HTTP 500) Leads to the following error in glance-api.log 2018-08-29 21:49:03.367 21964 INFO swiftclient [req-4f5b43a9-c3f1-4c71-b351-d08c2d221860 094897deb7424a4da2caccacc8c9d4ff ce38b0769f8c4037b641f8296bfd858b - bbda55ddd16e4faca419e2a13a113176 bbda55ddd16e4faca419e2a13a113176] REQ: curl -i http://10.5.0.34:8080/v1/AUTH_d213d2090cf948a0ba07e14491d58fc7/glance/02a6c228-249c-41f4-b8e3-04220a1d7529 -I -H "X-Auth-Token: gAAAAABbhxTOAmCn..." 2018-08-29 21:49:03.368 21964 INFO swiftclient [req-4f5b43a9-c3f1-4c71-b351-d08c2d221860 094897deb7424a4da2caccacc8c9d4ff ce38b0769f8c4037b641f8296bfd858b - bbda55ddd16e4faca419e2a13a113176 bbda55ddd16e4faca419e2a13a113176] RESP STATUS: 403 Forbidden 2018-08-29 21:49:03.368 21964 INFO swiftclient [req-4f5b43a9-c3f1-4c71-b351-d08c2d221860 094897deb7424a4da2caccacc8c9d4ff ce38b0769f8c4037b641f8296bfd858b - bbda55ddd16e4faca419e2a13a113176 bbda55ddd16e4faca419e2a13a113176] RESP HEADERS: {u'Date': u'Wed, 29 Aug 2018 21:49:03 GMT', u'Content-Length': u'73', u'Content-Type': u'text/html; charset=UTF-8', u'X-Openstack-Request-Id': u'tx3c78666a736e42a4b32a3-005b8714ce', u'X-Trans-Id': u'tx3c78666a736e42a4b32a3-005b8714ce'} 2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi [req-4f5b43a9-c3f1-4c71-b351-d08c2d221860 094897deb7424a4da2caccacc8c9d4ff ce38b0769f8c4037b641f8296bfd858b - bbda55ddd16e4faca419e2a13a113176 bbda55ddd16e4faca419e2a13a113176] Caught error: Object HEAD failed: http://10.5.0.34:8080/v1/AUTH_d213d2090cf948a0ba07e14491d58fc7/glance/02a6c228-249c-41f4-b8e3-04220a1d7529 403 Forbidden: ClientException: Object HEAD failed: http://10.5.0.34:8080/v1/AUTH_d213d2090cf948a0ba07e14491d58fc7/glance/02a6c228-249c-41f4-b8e3-04220a1d7529 403 Forbidden 2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi Traceback (most recent call last): 2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi File "/usr/lib/python2.7/dist-packages/glance/common/wsgi.py", line 1274, in __call__ 2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi request, **action_args) 2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi File "/usr/lib/python2.7/dist-packages/glance/common/wsgi.py", line 1317, in dispatch 2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi return method(*args, **kwargs) 2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi File "/usr/lib/python2.7/dist-packages/glance/common/utils.py", line 417, in wrapped 2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi return func(self, req, *args, **kwargs) 2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi File "/usr/lib/python2.7/dist-packages/glance/api/v2/images.py", line 343, in delete 2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi image.delete() 2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi File "/usr/lib/python2.7/dist-packages/glance/domain/proxy.py", line 189, in delete 2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi self.base.delete() 2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi File "/usr/lib/python2.7/dist-packages/glance/domain/proxy.py", line 189, in delete 2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi self.base.delete() 2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi File "/usr/lib/python2.7/dist-packages/glance/api/policy.py", line 172, in delete 2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi return self.image.delete() 2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi File "/usr/lib/python2.7/dist-packages/glance/domain/proxy.py", line 189, in delete 2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi self.base.delete() 2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi File "/usr/lib/python2.7/dist-packages/glance/location.py", line 431, in delete 2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi location) 2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi File "/usr/lib/python2.7/dist-packages/glance/common/store_utils.py", line 123, in delete_image_location_from_backend 2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi safe_delete_from_backend(context, image_id, location) 2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi File "/usr/lib/python2.7/dist-packages/glance/common/store_utils.py", line 56, in safe_delete_from_backend 2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi context=context) 2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi File "/usr/lib/python2.7/dist-packages/glance_store/backend.py", line 409, in delete_from_backend 2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi return store.delete(loc, context=context) 2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi File "/usr/lib/python2.7/dist-packages/glance_store/capabilities.py", line 225, in op_checker 2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi return store_op_fun(store, *args, **kwargs) 2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi File "/usr/lib/python2.7/dist-packages/glance_store/_drivers/swift/store.py", line 1093, in delete 2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi location.container, location.obj) 2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1826, in head_object 2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi query_string=query_string) 2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1722, in _retry 2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi service_token=self.service_token, **kwargs) 2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1252, in head_object 2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi raise ClientException.from_response(resp, 'Object HEAD failed', body) 2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi ClientException: Object HEAD failed: http://10.5.0.34:8080/v1/AUTH_d213d2090cf948a0ba07e14491d58fc7/glance/02a6c228-249c-41f4-b8e3-04220a1d7529 403 Forbidden 2018-08-29 21:49:03.369 21964 ERROR glance.common.wsgi 2018-08-29 21:49:03.376 21964 INFO eventlet.wsgi.server [req-4f5b43a9-c3f1-4c71-b351-d08c2d221860 094897deb7424a4da2caccacc8c9d4ff ce38b0769f8c4037b641f8296bfd858b - bbda55ddd16e4faca419e2a13a113176 bbda55ddd16e4faca419e2a13a113176] 10.5.0.12 - - [29/Aug/2018 21:49:03] "DELETE /v2/images/02a6c228-249c-41f4-b8e3-04220a1d7529 HTTP/1.1" 500 454 2.932388 Tested with openstack client for create and delete with same results. As the glance user using the swift client directly, the initial image part is not readable: $ swift list glance 02a6c228-249c-41f4-b8e3-04220a1d7529 02a6c228-249c-41f4-b8e3-04220a1d7529-00001 $ swift stat glance 02a6c228-249c-41f4-b8e3-04220a1d7529-00001 Account: AUTH_d213d2090cf948a0ba07e14491d58fc7 Container: glance Object: 02a6c228-249c-41f4-b8e3-04220a1d7529-00001 Content Type: application/octet-stream Content Length: 12716032 Last Modified: Wed, 29 Aug 2018 21:49:00 GMT ETag: 443b7623e27ecf03dc9e01ee93f67afe Accept-Ranges: bytes X-Timestamp: 1535579339.25431 X-Trans-Id: txd57d93cc856846f98015f-005b87155d X-Openstack-Request-Id: txd57d93cc856846f98015f-005b87155d $ swift stat glance 02a6c228-249c-41f4-b8e3-04220a1d7529 Object HEAD failed: http://10.5.0.34:8080/v1/AUTH_d213d2090cf948a0ba07e14491d58fc7/glance/02a6c228-249c-41f4-b8e3-04220a1d7529 403 Forbidden Failed Transaction ID: tx088744539df247ec8e2c0-005b871566 Note the glance container has no ACLs set: $ swift stat glance Account: AUTH_d213d2090cf948a0ba07e14491d58fc7 Container: glance Objects: 2 Bytes: 12716032 Read ACL: Write ACL: Sync To: Sync Key: Accept-Ranges: bytes X-Storage-Policy: Policy-0 Last-Modified: Wed, 29 Aug 2018 21:49:00 GMT X-Timestamp: 1535579339.45744 X-Trans-Id: txdc539510bdc84b7f84f28-005b87154b Content-Type: application/json; charset=utf-8 X-Openstack-Request-Id: txdc539510bdc84b7f84f28-005b87154b After adding read ACLs to the container, the image is deleteable swift post glance --read-acl ".r:*,.rlistings" Glance v2 should create its swift store container with correct read ACLS. To manage notifications about this bug go to: https://bugs.launchpad.net/glance/+bug/1789748/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp