Hi, > On 15 Mar 2024, at 01:07, Ondřej Kukla <ond...@kuuk.la> wrote: > > Hello I’m looking for suggestions how to track bucket creation over s3 api > and bucket usage (num of objects and size) of all buckets in time. > > In our RGW setup, we have a custom client panel, where like 85% percent of > buckets are created which is easy for us to then track the newly created > buckets and assigned them to our clients, however we also enable them to > create the buckets using the s3 api and there the tracking bucket to client > is harder. We also want to periodically track the bucket usage stats (num of > objects and size) of all buckets in time. > > Now we are using the GetBucketInfo from the AdminOPS api - > https://docs.ceph.com/en/quincy/radosgw/adminops/#id44 with the stats=true > option GET /admin/bucket?stats=1 which returns all buckets with the number of > objects and size we then parse. We also use it for the tracking of newly > created buckets as it gives you all of the buckets with their owners at one > place, but we are not using the metadata in that case. > > This however in our environment, where we have close to ~400 buckets isn’t > scalable and the calls can take from 10 to 400 to even 6000s to complete. > > So I’m looking for suggestions on how to get this data in time > better/smarter. I know there is a bunch of prometheus exporters like > https://github.com/pcuzner/rgw-exporter but they are usually just using the > same AdminOPS api so that doesn’t seem like a better solution. > > I’m also working on sending all of our RGW ops logs to elastic which would > give me the ability to easily query the CreateBucket calls, but that only > solves the bucket creation and not the usage stats. > > How would you handle this? Do you have any suggestions?
Do you try this [1] exporter? May be this code performance is better for you? k [1] https://github.com/blemmenes/radosgw_usage_exporter _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
