Hi,
> On 15 Mar 2024, at 01:07, Ondřej Kukla wrote:
>
> Hello I’m looking for suggestions how to track bucket creation over s3 api
> and bucket usage (num of objects and size) of all buckets in time.
>
> In our RGW setup, we have a custom client panel, where like 85% percent of
> buckets are created which is easy for us to then track the newly created
> buckets and assigned them to our clients, however we also enable them to
> create the buckets using the s3 api and there the tracking bucket to client
> is harder. We also want to periodically track the bucket usage stats (num of
> objects and size) of all buckets in time.
>
> Now we are using the GetBucketInfo from the AdminOPS api -
> https://docs.ceph.com/en/quincy/radosgw/adminops/#id44 with the stats=true
> option GET /admin/bucket?stats=1 which returns all buckets with the number of
> objects and size we then parse. We also use it for the tracking of newly
> created buckets as it gives you all of the buckets with their owners at one
> place, but we are not using the metadata in that case.
>
> This however in our environment, where we have close to ~400 buckets isn’t
> scalable and the calls can take from 10 to 400 to even 6000s to complete.
>
> So I’m looking for suggestions on how to get this data in time
> better/smarter. I know there is a bunch of prometheus exporters like
> https://github.com/pcuzner/rgw-exporter but they are usually just using the
> same AdminOPS api so that doesn’t seem like a better solution.
>
> I’m also working on sending all of our RGW ops logs to elastic which would
> give me the ability to easily query the CreateBucket calls, but that only
> solves the bucket creation and not the usage stats.
>
> How would you handle this? Do you have any suggestions?
Do you try this [1] exporter? May be this code performance is better for you?
k
[1] https://github.com/blemmenes/radosgw_usage_exporter
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
