For mailing list archive readers in the future: On Tue, Jul 9, 2019 at 1:22 PM Paul Emmerich <paul.emmer...@croit.io> wrote:
> Try to add "--inconsistent-index" (caution: will obviously leave your > bucket in a broken state during the deletion, so don't try to use the > bucket) > this was bad advice as long as https://tracker.ceph.com/issues/40700 is not fixed, don't do that. > > You can also speed up the deletion with "--max-concurrent-ios" (default > 32). The documentation incorrectly claims that "--max-concurrent-ios" is > only for other operations but that's wrong, it is used for most bucket > operations including deleteion. > this, however, is a good idea to speed up deletion of large buckets. Try to combine the deletion command with timeout or something to not run into OOM all the time affecting other services. (or use cgroups to limit RAM) Paul > > > Paul > > -- > Paul Emmerich > > Looking for help with your Ceph cluster? Contact us at https://croit.io > > croit GmbH > Freseniusstr. 31h > 81247 München > www.croit.io > Tel: +49 89 1896585 90 > > > On Tue, Jul 9, 2019 at 1:11 PM Harald Staub <harald.st...@switch.ch> > wrote: > >> Currently removing a bucket with a lot of objects: >> radosgw-admin bucket rm --bucket=$BUCKET --bypass-gc --purge-objects >> >> This process was killed by the out-of-memory killer. Then looking at the >> graphs, we see a continuous increase of memory usage for this process, >> about +24 GB per day. Removal rate is about 3 M objects per day. >> >> It is not the fastest hardware, and this index pool is still without >> SSDs. The bucket is sharded, 1024 shards. We are on Nautilus 14.2.1, now >> about 500 OSDs. >> >> So with this bucket with 60 M objects, we would need about 480 GB of RAM >> to come through. Or is there a workaround? Should I open a tracker issue? >> >> The killed remove command can just be called again, but it will be >> killed again before it finishes. Also, it has to run some time until it >> continues to actually remove objects. This "wait time" is also >> increasing. Last time, after about 16 M objects already removed, the >> wait time was nearly 9 hours. Also during this time, there is a memory >> ramp, but not so steep. >> >> BTW it feels strange that the removal of objects is slower (about 3 >> times) than adding objects. >> >> Harry >> _______________________________________________ >> ceph-users mailing list >> ceph-users@lists.ceph.com >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> >
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
