[ceph-users] Re: S3 user with more than 1000 buckets

2023-10-03 Thread Thomas Bennett 
Thanks for all the responses, much appreciated.

Upping the chunk size fixes my problem in the short term but I upgrade to
17.2.6 :)

Kind regards,
Tom

On Tue, 3 Oct 2023 at 15:28, Matt Benjamin  wrote:

> Hi Thomas,
>
> If I'm not mistaken, the RGW will paginate ListBuckets essentially like
> ListObjectsv1 if the S3 client provides the appropriate "marker" parameter
> values.  COS does this too, I noticed.  I'm not sure which S3 clients can
> be relied on to do this, though.
>
> Matt
>
> On Tue, Oct 3, 2023 at 9:06 AM Thomas Bennett  wrote:
>
>> Hi Jonas,
>>
>> Thanks :) that solved my issue.
>>
>> It would seem to me that this is heading towards something that the
>> clients
>> s3 should paginate, but I couldn't find any documentation on how to
>> paginate bucket listings. All the information points to paginating object
>> listing - which makes sense.
>>
>> Just for competition of this thread:
>>
>> The rgw parameters are found at: Quincy radosgw config ref
>> 
>>
>> I ran the following command to update the parameter for all running rgw
>> daemons:
>> ceph config set client.rgw rgw_list_buckets_max_chunk 1
>>
>> And then confirmed the running daemons were configured:
>> ceph daemon /var/run/ceph/ceph-client.rgw.xxx.xxx.asok config show | grep
>> rgw_list_buckets_max_chunk
>> "rgw_list_buckets_max_chunk": "1",
>>
>> Kind regards,
>> Tom
>>
>> On Tue, 3 Oct 2023 at 13:30, Jonas Nemeiksis 
>> wrote:
>>
>> > Hi,
>> >
>> > You should increase these default settings:
>> >
>> > rgw_list_buckets_max_chunk // for buckets
>> > rgw_max_listing_results // for objects
>> >
>> > On Tue, Oct 3, 2023 at 12:59 PM Thomas Bennett  wrote:
>> >
>> >> Hi,
>> >>
>> >> I'm running a Ceph 17.2.5 Rados Gateway and I have a user with more
>> than
>> >> 1000 buckets.
>> >>
>> >> When the client tries to list all their buckets using s3cmd, rclone and
>> >> python boto3, they all three only ever return the first 1000 bucket
>> names.
>> >> I can confirm the buckets are all there (and more than 1000) by
>> checking
>> >> with the radosgw-admin command.
>> >>
>> >> Have I missed a pagination limit for listing user buckets in the rados
>> >> gateway?
>> >>
>> >> Thanks,
>> >> Tom
>> >> ___
>> >> ceph-users mailing list -- ceph-users@ceph.io
>> >> To unsubscribe send an email to ceph-users-le...@ceph.io
>> >>
>> >
>> >
>> > --
>> > Jonas
>> >
>> ___
>> ceph-users mailing list -- ceph-users@ceph.io
>> To unsubscribe send an email to ceph-users-le...@ceph.io
>>
>
>
> --
>
> Matt Benjamin
> Red Hat, Inc.
> 315 West Huron Street, Suite 140A
> Ann Arbor, Michigan 48103
>
> http://www.redhat.com/en/technologies/storage
>
> tel.  734-821-5101
> fax.  734-769-8938
> cel.  734-216-5309
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: S3 user with more than 1000 buckets

2023-10-03 Thread Matt Benjamin 
Hi Thomas,

If I'm not mistaken, the RGW will paginate ListBuckets essentially like
ListObjectsv1 if the S3 client provides the appropriate "marker" parameter
values.  COS does this too, I noticed.  I'm not sure which S3 clients can
be relied on to do this, though.

Matt

On Tue, Oct 3, 2023 at 9:06 AM Thomas Bennett  wrote:

> Hi Jonas,
>
> Thanks :) that solved my issue.
>
> It would seem to me that this is heading towards something that the clients
> s3 should paginate, but I couldn't find any documentation on how to
> paginate bucket listings. All the information points to paginating object
> listing - which makes sense.
>
> Just for competition of this thread:
>
> The rgw parameters are found at: Quincy radosgw config ref
> 
>
> I ran the following command to update the parameter for all running rgw
> daemons:
> ceph config set client.rgw rgw_list_buckets_max_chunk 1
>
> And then confirmed the running daemons were configured:
> ceph daemon /var/run/ceph/ceph-client.rgw.xxx.xxx.asok config show | grep
> rgw_list_buckets_max_chunk
> "rgw_list_buckets_max_chunk": "1",
>
> Kind regards,
> Tom
>
> On Tue, 3 Oct 2023 at 13:30, Jonas Nemeiksis  wrote:
>
> > Hi,
> >
> > You should increase these default settings:
> >
> > rgw_list_buckets_max_chunk // for buckets
> > rgw_max_listing_results // for objects
> >
> > On Tue, Oct 3, 2023 at 12:59 PM Thomas Bennett  wrote:
> >
> >> Hi,
> >>
> >> I'm running a Ceph 17.2.5 Rados Gateway and I have a user with more than
> >> 1000 buckets.
> >>
> >> When the client tries to list all their buckets using s3cmd, rclone and
> >> python boto3, they all three only ever return the first 1000 bucket
> names.
> >> I can confirm the buckets are all there (and more than 1000) by checking
> >> with the radosgw-admin command.
> >>
> >> Have I missed a pagination limit for listing user buckets in the rados
> >> gateway?
> >>
> >> Thanks,
> >> Tom
> >> ___
> >> ceph-users mailing list -- ceph-users@ceph.io
> >> To unsubscribe send an email to ceph-users-le...@ceph.io
> >>
> >
> >
> > --
> > Jonas
> >
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>


-- 

Matt Benjamin
Red Hat, Inc.
315 West Huron Street, Suite 140A
Ann Arbor, Michigan 48103

http://www.redhat.com/en/technologies/storage

tel.  734-821-5101
fax.  734-769-8938
cel.  734-216-5309
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: S3 user with more than 1000 buckets

2023-10-03 Thread Casey Bodley 
On Tue, Oct 3, 2023 at 9:06 AM Thomas Bennett  wrote:
>
> Hi Jonas,
>
> Thanks :) that solved my issue.
>
> It would seem to me that this is heading towards something that the clients
> s3 should paginate, but I couldn't find any documentation on how to
> paginate bucket listings.

the s3 ListBuckets API
(https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListBuckets.html)
doesn't support pagination, so there's no way for clients to do that

but rgw itself should be able to paginate over the 'chunks' to return
more than rgw_list_buckets_max_chunk entries in a single ListBuckets
request. i opened a bug report for this at
https://tracker.ceph.com/issues/63080

> All the information points to paginating object
> listing - which makes sense.
>
> Just for competition of this thread:
>
> The rgw parameters are found at: Quincy radosgw config ref
> 
>
> I ran the following command to update the parameter for all running rgw
> daemons:
> ceph config set client.rgw rgw_list_buckets_max_chunk 1
>
> And then confirmed the running daemons were configured:
> ceph daemon /var/run/ceph/ceph-client.rgw.xxx.xxx.asok config show | grep
> rgw_list_buckets_max_chunk
> "rgw_list_buckets_max_chunk": "1",
>
> Kind regards,
> Tom
>
> On Tue, 3 Oct 2023 at 13:30, Jonas Nemeiksis  wrote:
>
> > Hi,
> >
> > You should increase these default settings:
> >
> > rgw_list_buckets_max_chunk // for buckets
> > rgw_max_listing_results // for objects
> >
> > On Tue, Oct 3, 2023 at 12:59 PM Thomas Bennett  wrote:
> >
> >> Hi,
> >>
> >> I'm running a Ceph 17.2.5 Rados Gateway and I have a user with more than
> >> 1000 buckets.
> >>
> >> When the client tries to list all their buckets using s3cmd, rclone and
> >> python boto3, they all three only ever return the first 1000 bucket names.
> >> I can confirm the buckets are all there (and more than 1000) by checking
> >> with the radosgw-admin command.
> >>
> >> Have I missed a pagination limit for listing user buckets in the rados
> >> gateway?
> >>
> >> Thanks,
> >> Tom
> >> ___
> >> ceph-users mailing list -- ceph-users@ceph.io
> >> To unsubscribe send an email to ceph-users-le...@ceph.io
> >>
> >
> >
> > --
> > Jonas
> >
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: S3 user with more than 1000 buckets

2023-10-03 Thread Janne Johansson 
Den tis 3 okt. 2023 kl 11:59 skrev Thomas Bennett :

> Hi,
>
> I'm running a Ceph 17.2.5 Rados Gateway and I have a user with more than
> 1000 buckets.
>
> When the client tries to list all their buckets using s3cmd, rclone and
> python boto3, they all three only ever return the first 1000 bucket names.
> I can confirm the buckets are all there (and more than 1000) by checking
> with the radosgw-admin command.
>
> Have I missed a pagination limit for listing user buckets in the rados
> gateway?
>
>
There is/was this bug that made the list not tell clients that there are
more than 1000 buckets, so the clients would not ask for next list of
pagination:

https://tracker.ceph.com/issues/57901

For Quincy, it was in 17.2.6 so upgrading to that version would also fix it.

https://docs.ceph.com/en/latest/releases/quincy/
search for

   -

   rgw: Fix truncated ListBuckets response (pr#49525
   , Joshua Baergen)


-- 
May the most significant bit of your life be positive.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: S3 user with more than 1000 buckets

2023-10-03 Thread Thomas Bennett 
Hi Jonas,

Thanks :) that solved my issue.

It would seem to me that this is heading towards something that the clients
s3 should paginate, but I couldn't find any documentation on how to
paginate bucket listings. All the information points to paginating object
listing - which makes sense.

Just for competition of this thread:

The rgw parameters are found at: Quincy radosgw config ref


I ran the following command to update the parameter for all running rgw
daemons:
ceph config set client.rgw rgw_list_buckets_max_chunk 1

And then confirmed the running daemons were configured:
ceph daemon /var/run/ceph/ceph-client.rgw.xxx.xxx.asok config show | grep
rgw_list_buckets_max_chunk
"rgw_list_buckets_max_chunk": "1",

Kind regards,
Tom

On Tue, 3 Oct 2023 at 13:30, Jonas Nemeiksis  wrote:

> Hi,
>
> You should increase these default settings:
>
> rgw_list_buckets_max_chunk // for buckets
> rgw_max_listing_results // for objects
>
> On Tue, Oct 3, 2023 at 12:59 PM Thomas Bennett  wrote:
>
>> Hi,
>>
>> I'm running a Ceph 17.2.5 Rados Gateway and I have a user with more than
>> 1000 buckets.
>>
>> When the client tries to list all their buckets using s3cmd, rclone and
>> python boto3, they all three only ever return the first 1000 bucket names.
>> I can confirm the buckets are all there (and more than 1000) by checking
>> with the radosgw-admin command.
>>
>> Have I missed a pagination limit for listing user buckets in the rados
>> gateway?
>>
>> Thanks,
>> Tom
>> ___
>> ceph-users mailing list -- ceph-users@ceph.io
>> To unsubscribe send an email to ceph-users-le...@ceph.io
>>
>
>
> --
> Jonas
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: S3 user with more than 1000 buckets

2023-10-03 Thread Jonas Nemeiksis 
Hi,

You should increase these default settings:

rgw_list_buckets_max_chunk // for buckets
rgw_max_listing_results // for objects

On Tue, Oct 3, 2023 at 12:59 PM Thomas Bennett  wrote:

> Hi,
>
> I'm running a Ceph 17.2.5 Rados Gateway and I have a user with more than
> 1000 buckets.
>
> When the client tries to list all their buckets using s3cmd, rclone and
> python boto3, they all three only ever return the first 1000 bucket names.
> I can confirm the buckets are all there (and more than 1000) by checking
> with the radosgw-admin command.
>
> Have I missed a pagination limit for listing user buckets in the rados
> gateway?
>
> Thanks,
> Tom
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>


-- 
Jonas
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io