[ceph-users] Re: [EXTERNAL] Re: RGW Bucket Notifications and MultiPart Uploads
Hi Mark, It is always good to move forward with the ceph versions :-) However, this was also backported to pacific [1]. So, if you want to wait for the next pacific release, the fix should be there. Yuval [1] https://github.com/ceph/ceph/pull/47175 On Thu, Jul 21, 2022 at 11:10 PM Mark Selby wrote: > “Is there a usecase for sending a notification when the upload starts?” – > Not for me. Only having to watch for ObjectCreated:CompleteMultipartUpload > and ObjectCreated:Put works for me. Quincy here I come. Thanks! > > > > > > -- > > Mark Selby > > Sr Linux Administrator, The Voleon Group > > mse...@voleon.com > > > > This email is subject to important conditions and disclosures that are > listed on this web page: https://voleon.com/disclaimer/. > > > > > > *From: *Yuval Lifshitz > *Date: *Thursday, July 21, 2022 at 12:21 AM > *To: *Mark Selby > *Cc: *"d...@redhat.com" , "ceph-users@ceph.io" < > ceph-users@ceph.io> > *Subject: *Re: [ceph-users] Re: [EXTERNAL] Re: RGW Bucket Notifications > and MultiPart Uploads > > > > Hi Mark, > > Starting from quincy, we send 1 notification, > "ObjectCreated:CompleteMultipartUpload", when the upload is complete. See: > https://docs.ceph.com/en/quincy/radosgw/s3-notification-compatibility/ > <https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.ceph.com%2Fen%2Fquincy%2Fradosgw%2Fs3-notification-compatibility%2F=05%7C01%7Cmselby%40voleon.com%7C575f0827f7d24c11344108da6ae9932f%7C45212fd85f544a19a6ba493ff6e072b1%7C0%7C0%7C637939848683524799%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C=9us%2BsEKZqY2fxYg1OhEw2frGMOrLtvxZRK7OEXt1lR8%3D=0> > > We don't send the "ObjectCreated:Post" notification when the upload starts > as it will be confusing with other objects "POST" uploads. > > Is there a usecase for sending a notification when the upload starts? > > > > Yuval > > > > > > Yuval > > > > On Thu, Jul 21, 2022 at 12:53 AM Mark Selby wrote: > > I have not tested with Quincy/17.x yet so I do not know which > notifications are sent for Multipart uploads in this release set. > > I know that for Pacific.16.x I needed to add some code/logic to only act > on notifications that represented the end state of an Object creation. > > My tests show that when a multipart upload is in progress if you perform a > head on the object before the final part is uploaded that you will get back > a 200 with the size that the object eventually will be. The multiple > noitifications that occur with the Multipart upload have size set to the > chunk that is being uploaded, not the total size. > > In order to get the behavior that I wanted, act on an object after it has > been uploaded I had to code the following: > > - get a notification and get it's size > - head the object in RGW and get it's size > - if the sizes do not match then do nothing. > - When the size in the notification matches the size on the head request > the event type is ObjectCreated:CompleteMultipartUpload and we know the > upload is complete. > > This is a bunch of extra code and round trips that.I wish that I did not > have to make. > > Hopefully Quincy only sends 2 notifications for a multipart upload (1) The > initial Post and (2) The find Put. > > > > -- > > > Mark Selby > Sr Linux Administrator, The Voleon Group > mse...@voleon.com > > This email is subject to important conditions and disclosures that are > listed on this web page: https://voleon.com/disclaimer/ > <https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fvoleon.com%2Fdisclaimer%2F=05%7C01%7Cmselby%40voleon.com%7C575f0827f7d24c11344108da6ae9932f%7C45212fd85f544a19a6ba493ff6e072b1%7C0%7C0%7C637939848683524799%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C=3hPSckiUcDuychmFpyiqXKEYmc8ZXEbaFGGGlehFsv0%3D=0> > . > > > On 7/20/22, 5:57 AM, "Daniel Gryniewicz" wrote: > > Seems like the notification for a multipart upload should look > different > to a normal upload? > > Daniel > > On 7/20/22 08:53, Yehuda Sadeh-Weinraub wrote: > > Can maybe leverage one of the other calls to check for upload > completion: > > list multipart uploads and/or list parts. The latter should work if > you > > have the upload id at hand. > > > > Yehuda > > > > On Wed, Jul 20, 2022, 8:40 AM Casey Bodley > wrote: > > > >> On Wed, Jul 20, 2022 at 12:57 AM Yuval Lifshitz < &g
[ceph-users] Re: [EXTERNAL] Re: RGW Bucket Notifications and MultiPart Uploads
“Is there a usecase for sending a notification when the upload starts?” – Not for me. Only having to watch for ObjectCreated:CompleteMultipartUpload and ObjectCreated:Put works for me. Quincy here I come. Thanks! -- Mark Selby Sr Linux Administrator, The Voleon Group mse...@voleon.com This email is subject to important conditions and disclosures that are listed on this web page: https://voleon.com/disclaimer/. From: Yuval Lifshitz Date: Thursday, July 21, 2022 at 12:21 AM To: Mark Selby Cc: "d...@redhat.com" , "ceph-users@ceph.io" Subject: Re: [ceph-users] Re: [EXTERNAL] Re: RGW Bucket Notifications and MultiPart Uploads Hi Mark, Starting from quincy, we send 1 notification, "ObjectCreated:CompleteMultipartUpload", when the upload is complete. See: https://docs.ceph.com/en/quincy/radosgw/s3-notification-compatibility/ We don't send the "ObjectCreated:Post" notification when the upload starts as it will be confusing with other objects "POST" uploads. Is there a usecase for sending a notification when the upload starts? Yuval Yuval On Thu, Jul 21, 2022 at 12:53 AM Mark Selby wrote: I have not tested with Quincy/17.x yet so I do not know which notifications are sent for Multipart uploads in this release set. I know that for Pacific.16.x I needed to add some code/logic to only act on notifications that represented the end state of an Object creation. My tests show that when a multipart upload is in progress if you perform a head on the object before the final part is uploaded that you will get back a 200 with the size that the object eventually will be. The multiple noitifications that occur with the Multipart upload have size set to the chunk that is being uploaded, not the total size. In order to get the behavior that I wanted, act on an object after it has been uploaded I had to code the following: - get a notification and get it's size - head the object in RGW and get it's size - if the sizes do not match then do nothing. - When the size in the notification matches the size on the head request the event type is ObjectCreated:CompleteMultipartUpload and we know the upload is complete. This is a bunch of extra code and round trips that.I wish that I did not have to make. Hopefully Quincy only sends 2 notifications for a multipart upload (1) The initial Post and (2) The find Put. -- Mark Selby Sr Linux Administrator, The Voleon Group mse...@voleon.com This email is subject to important conditions and disclosures that are listed on this web page: https://voleon.com/disclaimer/. On 7/20/22, 5:57 AM, "Daniel Gryniewicz" wrote: Seems like the notification for a multipart upload should look different to a normal upload? Daniel On 7/20/22 08:53, Yehuda Sadeh-Weinraub wrote: > Can maybe leverage one of the other calls to check for upload completion: > list multipart uploads and/or list parts. The latter should work if you > have the upload id at hand. > > Yehuda > > On Wed, Jul 20, 2022, 8:40 AM Casey Bodley wrote: > >> On Wed, Jul 20, 2022 at 12:57 AM Yuval Lifshitz >> wrote: >>> >>> yes, that would work. you would get a "404" until the object is fully >>> uploaded. >> >> just note that you won't always get 404 before multipart complete, >> because multipart uploads can overwrite existing objects >> >> ___ >> ceph-users mailing list -- ceph-users@ceph.io >> To unsubscribe send an email to ceph-users-le...@ceph.io >> >> > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io > ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: [EXTERNAL] Re: RGW Bucket Notifications and MultiPart Uploads
Hi Mark, Starting from quincy, we send 1 notification, "ObjectCreated:CompleteMultipartUpload", when the upload is complete. See: https://docs.ceph.com/en/quincy/radosgw/s3-notification-compatibility/ We don't send the "ObjectCreated:Post" notification when the upload starts as it will be confusing with other objects "POST" uploads. Is there a usecase for sending a notification when the upload starts? Yuval Yuval On Thu, Jul 21, 2022 at 12:53 AM Mark Selby wrote: > I have not tested with Quincy/17.x yet so I do not know which > notifications are sent for Multipart uploads in this release set. > > I know that for Pacific.16.x I needed to add some code/logic to only act > on notifications that represented the end state of an Object creation. > > My tests show that when a multipart upload is in progress if you perform a > head on the object before the final part is uploaded that you will get back > a 200 with the size that the object eventually will be. The multiple > noitifications that occur with the Multipart upload have size set to the > chunk that is being uploaded, not the total size. > > In order to get the behavior that I wanted, act on an object after it has > been uploaded I had to code the following: > > - get a notification and get it's size > - head the object in RGW and get it's size > - if the sizes do not match then do nothing. > - When the size in the notification matches the size on the head request > the event type is ObjectCreated:CompleteMultipartUpload and we know the > upload is complete. > > This is a bunch of extra code and round trips that.I wish that I did not > have to make. > > Hopefully Quincy only sends 2 notifications for a multipart upload (1) The > initial Post and (2) The find Put. > > > > -- > > > Mark Selby > Sr Linux Administrator, The Voleon Group > mse...@voleon.com > > This email is subject to important conditions and disclosures that are > listed on this web page: https://voleon.com/disclaimer/. > > > On 7/20/22, 5:57 AM, "Daniel Gryniewicz" wrote: > > Seems like the notification for a multipart upload should look > different > to a normal upload? > > Daniel > > On 7/20/22 08:53, Yehuda Sadeh-Weinraub wrote: > > Can maybe leverage one of the other calls to check for upload > completion: > > list multipart uploads and/or list parts. The latter should work if > you > > have the upload id at hand. > > > > Yehuda > > > > On Wed, Jul 20, 2022, 8:40 AM Casey Bodley > wrote: > > > >> On Wed, Jul 20, 2022 at 12:57 AM Yuval Lifshitz < > ylifs...@redhat.com> > >> wrote: > >>> > >>> yes, that would work. you would get a "404" until the object is > fully > >>> uploaded. > >> > >> just note that you won't always get 404 before multipart complete, > >> because multipart uploads can overwrite existing objects > >> > >> ___ > >> ceph-users mailing list -- ceph-users@ceph.io > >> To unsubscribe send an email to ceph-users-le...@ceph.io > >> > >> > > ___ > > ceph-users mailing list -- ceph-users@ceph.io > > To unsubscribe send an email to ceph-users-le...@ceph.io > > > > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io > ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: [EXTERNAL] Re: RGW Bucket Notifications and MultiPart Uploads
I have not tested with Quincy/17.x yet so I do not know which notifications are sent for Multipart uploads in this release set. I know that for Pacific.16.x I needed to add some code/logic to only act on notifications that represented the end state of an Object creation. My tests show that when a multipart upload is in progress if you perform a head on the object before the final part is uploaded that you will get back a 200 with the size that the object eventually will be. The multiple noitifications that occur with the Multipart upload have size set to the chunk that is being uploaded, not the total size. In order to get the behavior that I wanted, act on an object after it has been uploaded I had to code the following: - get a notification and get it's size - head the object in RGW and get it's size - if the sizes do not match then do nothing. - When the size in the notification matches the size on the head request the event type is ObjectCreated:CompleteMultipartUpload and we know the upload is complete. This is a bunch of extra code and round trips that.I wish that I did not have to make. Hopefully Quincy only sends 2 notifications for a multipart upload (1) The initial Post and (2) The find Put. -- Mark Selby Sr Linux Administrator, The Voleon Group mse...@voleon.com This email is subject to important conditions and disclosures that are listed on this web page: https://voleon.com/disclaimer/. On 7/20/22, 5:57 AM, "Daniel Gryniewicz" wrote: Seems like the notification for a multipart upload should look different to a normal upload? Daniel On 7/20/22 08:53, Yehuda Sadeh-Weinraub wrote: > Can maybe leverage one of the other calls to check for upload completion: > list multipart uploads and/or list parts. The latter should work if you > have the upload id at hand. > > Yehuda > > On Wed, Jul 20, 2022, 8:40 AM Casey Bodley wrote: > >> On Wed, Jul 20, 2022 at 12:57 AM Yuval Lifshitz >> wrote: >>> >>> yes, that would work. you would get a "404" until the object is fully >>> uploaded. >> >> just note that you won't always get 404 before multipart complete, >> because multipart uploads can overwrite existing objects >> >> ___ >> ceph-users mailing list -- ceph-users@ceph.io >> To unsubscribe send an email to ceph-users-le...@ceph.io >> >> > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io > ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: [EXTERNAL] Re: RGW Bucket Notifications and MultiPart Uploads
Seems like the notification for a multipart upload should look different to a normal upload? Daniel On 7/20/22 08:53, Yehuda Sadeh-Weinraub wrote: Can maybe leverage one of the other calls to check for upload completion: list multipart uploads and/or list parts. The latter should work if you have the upload id at hand. Yehuda On Wed, Jul 20, 2022, 8:40 AM Casey Bodley wrote: On Wed, Jul 20, 2022 at 12:57 AM Yuval Lifshitz wrote: yes, that would work. you would get a "404" until the object is fully uploaded. just note that you won't always get 404 before multipart complete, because multipart uploads can overwrite existing objects ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: [EXTERNAL] Re: RGW Bucket Notifications and MultiPart Uploads
Can maybe leverage one of the other calls to check for upload completion: list multipart uploads and/or list parts. The latter should work if you have the upload id at hand. Yehuda On Wed, Jul 20, 2022, 8:40 AM Casey Bodley wrote: > On Wed, Jul 20, 2022 at 12:57 AM Yuval Lifshitz > wrote: > > > > yes, that would work. you would get a "404" until the object is fully > > uploaded. > > just note that you won't always get 404 before multipart complete, > because multipart uploads can overwrite existing objects > > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io > > ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: [EXTERNAL] Re: RGW Bucket Notifications and MultiPart Uploads
On Wed, Jul 20, 2022 at 12:57 AM Yuval Lifshitz wrote: > > yes, that would work. you would get a "404" until the object is fully > uploaded. just note that you won't always get 404 before multipart complete, because multipart uploads can overwrite existing objects ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: [EXTERNAL] Re: RGW Bucket Notifications and MultiPart Uploads
yes, that would work. you would get a "404" until the object is fully uploaded. On Tue, Jul 19, 2022 at 6:00 PM Mark Selby wrote: > If you can that would be great. As it is likely to be a while before we > make the 16 -> 17 switch. > > > > Question: If I receive the 1st put notification for the partial object > and I check RGW to see if the object actually exists – do you know if RGW > will tell me that it is missing until the > ObjectCreated:CompleteMultipartUpload is sent? > > > > I was thinking as a stop gap I could add some code to just check for the > objects existence in RGW before taking any action? > > > > Thanks very much for taking the time out to answer this. > > > > -- > > Mark Selby > > Sr Linux Administrator, The Voleon Group > > mse...@voleon.com > > > > This email is subject to important conditions and disclosures that are > listed on this web page: https://voleon.com/disclaimer/. > > > > > > *From: *Yuval Lifshitz > *Date: *Monday, July 18, 2022 at 9:33 PM > *To: *Mark Selby > *Cc: *"ceph-users@ceph.io" > *Subject: *[EXTERNAL] Re: [ceph-users] RGW Bucket Notifications and > MultiPart Uploads > > > > *CAUTION:* This email originated from outside of the organization. Use > caution when opening attachments or links. > > > > Hi Mark, > > It is in quincy but wasn't backported to pacific yet. > > I can do this backport, but I'm not sure when is the next pacific release. > > > > Yuval > > > > On Tue, Jul 19, 2022 at 5:04 AM Mark Selby wrote: > > I am trying to use RGW Bucket Notifications to trigger events on object > creation and have into a bit of an issue when multipart uploads come into > play for large objects. > > > > With a small object only a single notification is generated -> > ObjectCreated:Put > > > > When a multipart upload is performed a string of Notifications are sent: > > ObjectCreated:Post > > ObjectCreated:Put > > ObjectCreated:Put > > ObjectCreated:Put > > … > > ObjectCreated:CompleteMultipartUpload > > > > I can ignore the Post, but all of the Put notifications look the same as a > single part upload message but the object will not actually be created > until the CompleteMultipartUpload notification happens. > > > > There is this is https://tracker.ceph.com/issues/51520 > <https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftracker.ceph.com%2Fissues%2F51520=05%7C01%7Cmselby%40voleon.com%7Cc79d203d6c6e42084ef408da693fc775%7C45212fd85f544a19a6ba493ff6e072b1%7C0%7C0%7C637938019913112991%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C=Hyc5Y3Xl5HdFr2mmXHHMa1df1gNC4tvMIkeGhW2lOlY%3D=0> > that seems to fix this issue – I can not tell if this was actually > backported or not. Does anyone know if this actually was backported? > > > > Thanks! > > > > -- > > Mark Selby > > Sr Linux Administrator, The Voleon Group > > mse...@voleon.com > > > > This email is subject to important conditions and disclosures that are > listed on this web page: https://voleon.com/disclaimer/ > <https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fvoleon.com%2Fdisclaimer%2F=05%7C01%7Cmselby%40voleon.com%7Cc79d203d6c6e42084ef408da693fc775%7C45212fd85f544a19a6ba493ff6e072b1%7C0%7C0%7C637938019913271960%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C=o%2Bujqo%2BTJJZhUQ8NJv%2BxDU2M2czwxBbkf4wXCIfMBQk%3D=0> > . > > > > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io > > ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: [EXTERNAL] Re: RGW Bucket Notifications and MultiPart Uploads
If you can that would be great. As it is likely to be a while before we make the 16 -> 17 switch. Question: If I receive the 1st put notification for the partial object and I check RGW to see if the object actually exists – do you know if RGW will tell me that it is missing until the ObjectCreated:CompleteMultipartUpload is sent? I was thinking as a stop gap I could add some code to just check for the objects existence in RGW before taking any action? Thanks very much for taking the time out to answer this. -- Mark Selby Sr Linux Administrator, The Voleon Group mse...@voleon.com This email is subject to important conditions and disclosures that are listed on this web page: https://voleon.com/disclaimer/. From: Yuval Lifshitz Date: Monday, July 18, 2022 at 9:33 PM To: Mark Selby Cc: "ceph-users@ceph.io" Subject: [EXTERNAL] Re: [ceph-users] RGW Bucket Notifications and MultiPart Uploads CAUTION: This email originated from outside of the organization. Use caution when opening attachments or links. Hi Mark, It is in quincy but wasn't backported to pacific yet. I can do this backport, but I'm not sure when is the next pacific release. Yuval On Tue, Jul 19, 2022 at 5:04 AM Mark Selby wrote: I am trying to use RGW Bucket Notifications to trigger events on object creation and have into a bit of an issue when multipart uploads come into play for large objects. With a small object only a single notification is generated -> ObjectCreated:Put When a multipart upload is performed a string of Notifications are sent: ObjectCreated:Post ObjectCreated:Put ObjectCreated:Put ObjectCreated:Put … ObjectCreated:CompleteMultipartUpload I can ignore the Post, but all of the Put notifications look the same as a single part upload message but the object will not actually be created until the CompleteMultipartUpload notification happens. There is this is https://tracker.ceph.com/issues/51520 that seems to fix this issue – I can not tell if this was actually backported or not. Does anyone know if this actually was backported? Thanks! -- Mark Selby Sr Linux Administrator, The Voleon Group mse...@voleon.com This email is subject to important conditions and disclosures that are listed on this web page: https://voleon.com/disclaimer/. ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
