[ceph-users] Re: Workload that delete 100 M object daily via lifecycle
Enabling debug lc will execute more often the LC, but, please mind that might not respect expiration time set. By design it will consider a day the time set in interval. So, if will run more often, you will end up removing objects sooner than 365 days (as an example) if set to do so. Please test using: rgw_lifecycle_work_time 00:00-23:59 to run all day, and rgw_lc_debug_interval 86400 Meaning it will run every 4h. Paul On Wed, Jul 19, 2023 at 5:04 AM Anthony D'Atri wrote: > Indeed that's very useful. I improved the documentation for that not long > ago, took a while to sort out exactly what it was about. > > Normally LC only runs once a day as I understand it, there's a debug > option that compresses time so that it'll run more frequently, as having to > wait for a day to see the effect of changes harks back to the uucp days ;) > > > On Jul 18, 2023, at 21:37, Hoan Nguyen Van wrote: > > > > You can enable debug lc to test and tuning rgw lc parameter. > > ___ > > ceph-users mailing list -- ceph-users@ceph.io > > To unsubscribe send an email to ceph-users-le...@ceph.io > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io > ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Workload that delete 100 M object daily via lifecycle
Indeed that's very useful. I improved the documentation for that not long ago, took a while to sort out exactly what it was about. Normally LC only runs once a day as I understand it, there's a debug option that compresses time so that it'll run more frequently, as having to wait for a day to see the effect of changes harks back to the uucp days ;) > On Jul 18, 2023, at 21:37, Hoan Nguyen Van wrote: > > You can enable debug lc to test and tuning rgw lc parameter. > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Workload that delete 100 M object daily via lifecycle
You can enable debug lc to test and tuning rgw lc parameter. ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Workload that delete 100 M object daily via lifecycle
Index pool on Aerospike? Building OSDs on PRAM might be a lot less work than trying to ensure consistency on backing storage while still servicing out of RAM and not syncing every transaction. > On Jul 18, 2023, at 14:31, Peter Grandi wrote: > > [...] S3 workload, that will need to delete 100M file > daily [...] > >>> [...] average (what about peaks?) around 1,200 committed >>> deletions per second (across the traditional 3 metadata >>> OSDs) sustained, that may not leave a lot of time for file >> creation, writing or reading. :-)[...] > [...] So many people seem to think that distributed (or even local) filesystems (and in particular their metadata servers) can sustain the same workload as high volume transactional DBMSes. [...] > >> Index pool distributed over a large number of NVMe OSDs? >> Multiple, dedicated RGW instances that only run LC? > > As long as that guarantees a total maximum network+write > latency of well below 800µs across all of them that might > result in a committed rate of a deletion every 800µs (and there > are no peaks and the metadata server only does deletions and > does not do creations or opens or any "maintenance" operations > like checks and backups). :-) > > Sometimes I suggest somewhat seriously entirely RAM based > metadata OSDs, which given a suitable environment may be > feasible. But I still wonder why "So many people seem to think > ... can sustain the same workload as high volume transactional > DBMSes" :-). > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Workload that delete 100 M object daily via lifecycle
[...] S3 workload, that will need to delete 100M file daily [...] >> [...] average (what about peaks?) around 1,200 committed >> deletions per second (across the traditional 3 metadata >> OSDs) sustained, that may not leave a lot of time for file > creation, writing or reading. :-)[...] >>> [...] So many people seem to think that distributed (or >>> even local) filesystems (and in particular their metadata >>> servers) can sustain the same workload as high volume >>> transactional DBMSes. [...] > Index pool distributed over a large number of NVMe OSDs? > Multiple, dedicated RGW instances that only run LC? As long as that guarantees a total maximum network+write latency of well below 800µs across all of them that might result in a committed rate of a deletion every 800µs (and there are no peaks and the metadata server only does deletions and does not do creations or opens or any "maintenance" operations like checks and backups). :-) Sometimes I suggest somewhat seriously entirely RAM based metadata OSDs, which given a suitable environment may be feasible. But I still wonder why "So many people seem to think ... can sustain the same workload as high volume transactional DBMSes" :-). ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Workload that delete 100 M object daily via lifecycle
Index pool distributed over a large number of NVMe OSDs? Multiple, dedicated RGW instances that only run LC? > On Jul 18, 2023, at 12:08, Peter Grandi wrote: > On Mon, 17 Jul 2023 19:19:34 +0700, Ha Nguyen Van said: > >> [...] S3 workload, that will need to delete 100M file daily [...] > > So many people seem to think that distributed (or even local) > filesystems (and in particular their metadata servers) can > sustain the same workload as high volume transactional DBMSes. > > PS 100m deletions per day given 86,400 second per day means on > average (what about peaks?) around 1,200 committed deletions per > second (across the traditional 3 metadata OSDs) sustained on the > metadata server, that may not leave a lot of time for file > creation, writing or reading. :-) > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Workload that delete 100 M object daily via lifecycle
>>> On Mon, 17 Jul 2023 19:19:34 +0700, Ha Nguyen Van >>> said: > [...] S3 workload, that will need to delete 100M file daily [...] So many people seem to think that distributed (or even local) filesystems (and in particular their metadata servers) can sustain the same workload as high volume transactional DBMSes. PS 100m deletions per day given 86,400 second per day means on average (what about peaks?) around 1,200 committed deletions per second (across the traditional 3 metadata OSDs) sustained on the metadata server, that may not leave a lot of time for file creation, writing or reading. :-) ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Workload that delete 100 M object daily via lifecycle
Hi, You may want to check out this doc: https://docs.ceph.com/en/quincy/radosgw/config-ref/#lifecycle-settings As I understand it in short: - if there are thousands of buckets, we should increase the rgw_lc_max_worker. - if there are a few buckets that have hundreds of thousands of objects, we should decrease the rgw_lc_max_wp_worker. ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
