[Yahoo-eng-team] [Bug 1511061] Re: Images in inconsistent state when calls to registry fail during image deletion
v1 API is deprecated. ** Changed in: glance Status: In Progress => Invalid -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to Glance. https://bugs.launchpad.net/bugs/1511061 Title: Images in inconsistent state when calls to registry fail during image deletion Status in Glance: Invalid Status in Glance juno series: New Status in Glance kilo series: New Status in Glance liberty series: New Status in OpenStack Security Advisory: Won't Fix Bug description: [0] shows a sample image that was left in an inconsistent state when a call to registry failed during image deletion. Glance v1 API makes two registry calls when deleting an image. The first call [1] is made to to set the status of an image to deleted/pending_delete. And, the other [2], to delete the rest of the metadata, which sets 'deleted_at' and 'deleted' fields in the db. If the first call fails, the image deletion request fails and the image is left intact in it's previous status. However, if the first call succeeds and the second one fails, the image is left in an inconsistent status where it's status is set to pending_delete/deleted but it's 'deleted_at' and 'deleted' fields are not set. If delayed delete is turned on, these images are never collected by the scrubber as they won't appear as deleted images because their deleted field is not set. So, these images will continue to occupy storage in the backend. Also, further attempts at deleting these images will fail with a 404 because the status is already set to pending_delete/deleted. [0] http://paste.openstack.org/show/477577/ [1]: https://github.com/openstack/glance/blob/master/glance/api/v1/images.py#L1115-L1116 [2]: https://github.com/openstack/glance/blob/master/glance/api/v1/images.py#L1132 To manage notifications about this bug go to: https://bugs.launchpad.net/glance/+bug/1511061/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1511061] Re: Images in inconsistent state when calls to registry fail during image deletion
1. I agree, the image deletion operation should be atomic. 2. Image data left behind means there is a potential risk of filling up storage quota and resulting into a DoS; be mindful that it's a denial of service but NOT a exploit as it is dependent on the operators' failure scenarios of g-api <-> reg communication. 3. The original description has information related to failure scenarios for only v1. So, a check is needed for the v2 as applicable. ** Description changed: [0] shows a sample image that was left in an inconsistent state when a call to registry failed during image deletion. - Glance API makes two registry calls when deleting an image. + Glance v1 API makes two registry calls when deleting an image. The first call [1] is made to to set the status of an image to deleted/pending_delete. And, the other [2], to delete the rest of the metadata, which sets 'deleted_at' and 'deleted' fields in the db. If the first call fails, the image deletion request fails and the image is left intact in it's previous status. However, if the first call succeeds and the second one fails, the image is left in an inconsistent status where it's status is set to pending_delete/deleted but it's 'deleted_at' and 'deleted' fields are not set. If delayed delete is turned on, these images are never collected by the scrubber as they won't appear as deleted images because their deleted field is not set. So, these images will continue to occupy storage in the backend. Also, further attempts at deleting these images will fail with a 404 because the status is already set to pending_delete/deleted. [0] http://paste.openstack.org/show/477577/ [1]: https://github.com/openstack/glance/blob/master/glance/api/v1/images.py#L1115-L1116 [2]: https://github.com/openstack/glance/blob/master/glance/api/v1/images.py#L1132 ** Changed in: glance Status: New => Triaged ** Changed in: glance Importance: Undecided => Critical ** Changed in: glance Milestone: None => mitaka-1 ** Changed in: glance Assignee: (unassigned) => Hemanth Makkapati (hemanth-makkapati) ** Also affects: glance/kilo Importance: Undecided Status: New ** Also affects: glance/juno Importance: Undecided Status: New ** Also affects: glance/liberty Importance: Undecided Status: New ** Information type changed from Public to Public Security -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to Glance. https://bugs.launchpad.net/bugs/1511061 Title: Images in inconsistent state when calls to registry fail during image deletion Status in Glance: Triaged Status in Glance juno series: New Status in Glance kilo series: New Status in Glance liberty series: New Bug description: [0] shows a sample image that was left in an inconsistent state when a call to registry failed during image deletion. Glance v1 API makes two registry calls when deleting an image. The first call [1] is made to to set the status of an image to deleted/pending_delete. And, the other [2], to delete the rest of the metadata, which sets 'deleted_at' and 'deleted' fields in the db. If the first call fails, the image deletion request fails and the image is left intact in it's previous status. However, if the first call succeeds and the second one fails, the image is left in an inconsistent status where it's status is set to pending_delete/deleted but it's 'deleted_at' and 'deleted' fields are not set. If delayed delete is turned on, these images are never collected by the scrubber as they won't appear as deleted images because their deleted field is not set. So, these images will continue to occupy storage in the backend. Also, further attempts at deleting these images will fail with a 404 because the status is already set to pending_delete/deleted. [0] http://paste.openstack.org/show/477577/ [1]: https://github.com/openstack/glance/blob/master/glance/api/v1/images.py#L1115-L1116 [2]: https://github.com/openstack/glance/blob/master/glance/api/v1/images.py#L1132 To manage notifications about this bug go to: https://bugs.launchpad.net/glance/+bug/1511061/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1511061] Re: Images in inconsistent state when calls to registry fail during image deletion
Since this report concerns a possible security risk, an incomplete security advisory task has been added while the core security reviewers for the affected project or projects confirm the bug and discuss the scope of any vulnerability along with potential solutions. Can user make the image deletion to fail ? e.g., can this be abused trivially ? ** Also affects: ossa Importance: Undecided Status: New ** Changed in: ossa Status: New => Incomplete -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to Glance. https://bugs.launchpad.net/bugs/1511061 Title: Images in inconsistent state when calls to registry fail during image deletion Status in Glance: Triaged Status in Glance juno series: New Status in Glance kilo series: New Status in Glance liberty series: New Status in OpenStack Security Advisory: Incomplete Bug description: [0] shows a sample image that was left in an inconsistent state when a call to registry failed during image deletion. Glance v1 API makes two registry calls when deleting an image. The first call [1] is made to to set the status of an image to deleted/pending_delete. And, the other [2], to delete the rest of the metadata, which sets 'deleted_at' and 'deleted' fields in the db. If the first call fails, the image deletion request fails and the image is left intact in it's previous status. However, if the first call succeeds and the second one fails, the image is left in an inconsistent status where it's status is set to pending_delete/deleted but it's 'deleted_at' and 'deleted' fields are not set. If delayed delete is turned on, these images are never collected by the scrubber as they won't appear as deleted images because their deleted field is not set. So, these images will continue to occupy storage in the backend. Also, further attempts at deleting these images will fail with a 404 because the status is already set to pending_delete/deleted. [0] http://paste.openstack.org/show/477577/ [1]: https://github.com/openstack/glance/blob/master/glance/api/v1/images.py#L1115-L1116 [2]: https://github.com/openstack/glance/blob/master/glance/api/v1/images.py#L1132 To manage notifications about this bug go to: https://bugs.launchpad.net/glance/+bug/1511061/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1511061] Re: Images in inconsistent state when calls to registry fail during image deletion
Thanks Erno, I've removed the OSSA task ** Changed in: ossa Status: Incomplete => Won't Fix -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to Glance. https://bugs.launchpad.net/bugs/1511061 Title: Images in inconsistent state when calls to registry fail during image deletion Status in Glance: Triaged Status in Glance juno series: New Status in Glance kilo series: New Status in Glance liberty series: New Status in OpenStack Security Advisory: Won't Fix Bug description: [0] shows a sample image that was left in an inconsistent state when a call to registry failed during image deletion. Glance v1 API makes two registry calls when deleting an image. The first call [1] is made to to set the status of an image to deleted/pending_delete. And, the other [2], to delete the rest of the metadata, which sets 'deleted_at' and 'deleted' fields in the db. If the first call fails, the image deletion request fails and the image is left intact in it's previous status. However, if the first call succeeds and the second one fails, the image is left in an inconsistent status where it's status is set to pending_delete/deleted but it's 'deleted_at' and 'deleted' fields are not set. If delayed delete is turned on, these images are never collected by the scrubber as they won't appear as deleted images because their deleted field is not set. So, these images will continue to occupy storage in the backend. Also, further attempts at deleting these images will fail with a 404 because the status is already set to pending_delete/deleted. [0] http://paste.openstack.org/show/477577/ [1]: https://github.com/openstack/glance/blob/master/glance/api/v1/images.py#L1115-L1116 [2]: https://github.com/openstack/glance/blob/master/glance/api/v1/images.py#L1132 To manage notifications about this bug go to: https://bugs.launchpad.net/glance/+bug/1511061/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
