I think if you want to delete through gc,
increase this
OPTION(rgw_gc_processor_max_time, OPT_INT, 3600) // total run time
for a single gc processor work
decrease this
OPTION(rgw_gc_processor_period, OPT_INT, 3600) // gc processor cycle time
Or , I think if there is some option to bypass the gc
On Tue, Jul 25, 2017 at 5:05 AM, Bryan Stillwell <bstillw...@godaddy.com> wrote:
> Wouldn't doing it that way cause problems since references to the objects
> wouldn't be getting removed from .rgw.buckets.index?
>
> Bryan
>
> From: Roger Brown <rogerpbr...@gmail.com>
> Date: Monday, July 24, 2017 at 2:43 PM
> To: Bryan Stillwell <bstillw...@godaddy.com>, "ceph-users@lists.ceph.com"
> <ceph-users@lists.ceph.com>
> Subject: Re: [ceph-users] Speeding up garbage collection in RGW
>
> I hope someone else can answer your question better, but in my case I found
> something like this helpful to delete objects faster than I could through the
> gateway:
>
> rados -p default.rgw.buckets.data ls | grep 'replace this with pattern
> matching files you want to delete' | xargs -d '\n' -n 200 rados -p
> default.rgw.buckets.data rm
>
>
> On Mon, Jul 24, 2017 at 2:02 PM Bryan Stillwell <bstillw...@godaddy.com>
> wrote:
> I'm in the process of cleaning up a test that an internal customer did on our
> production cluster that produced over a billion objects spread across 6000
> buckets. So far I've been removing the buckets like this:
>
> printf %s\\n bucket{1..6000} | xargs -I{} -n 1 -P 32 radosgw-admin bucket rm
> --bucket={} --purge-objects
>
> However, the disk usage doesn't seem to be getting reduced at the same rate
> the objects are being removed. From what I can tell a large number of the
> objects are waiting for garbage collection.
>
> When I first read the docs it sounded like the garbage collector would only
> remove 32 objects every hour, but after looking through the logs I'm seeing
> about 55,000 objects removed every hour. That's about 1.3 million a day, so
> at this rate it'll take a couple years to clean up the rest! For comparison,
> the purge-objects command above is removing (but not GC'ing) about 30 million
> objects a day, so a much more manageable 33 days to finish.
>
> I've done some digging and it appears like I should be changing these
> configuration options:
>
> rgw gc max objs (default: 32)
> rgw gc obj min wait (default: 7200)
> rgw gc processor max time (default: 3600)
> rgw gc processor period (default: 3600)
>
> A few questions I have though are:
>
> Should 'rgw gc processor max time' and 'rgw gc processor period' always be
> set to the same value?
>
> Which would be better, increasing 'rgw gc max objs' to something like 1024,
> or reducing the 'rgw gc processor' times to something like 60 seconds?
>
> Any other guidance on the best way to adjust these values?
>
> Thanks,
> Bryan
>
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
