Re: [ceph-users] rgw S3 lifecycle cannot keep up

2019-10-03 Thread Christian Pedersen
Thank you Robin. Looking at the video it doesn't seem like a fix is anywhere near ready. Am I correct in concluding that Ceph is not the right tool for my use-case? Cheers, Christian On Oct 3 2019, at 6:07 am, Robin H. Johnson wrote: > On Wed, Oct 02, 2019 at 01:48:40PM +0200, Christian

Re: [ceph-users] rgw S3 lifecycle cannot keep up

2019-10-02 Thread Robin H. Johnson
On Wed, Oct 02, 2019 at 01:48:40PM +0200, Christian Pedersen wrote: > Hi Martin, > > Even before adding cold storage on HDD, I had the cluster with SSD only. That > also could not keep up with deleting the files. > I am no where near I/O exhaustion on the SSDs or even the HDDs. Please see my

Re: [ceph-users] rgw S3 lifecycle cannot keep up

2019-10-02 Thread Christian Pedersen
Hi Martin, Even before adding cold storage on HDD, I had the cluster with SSD only. That also could not keep up with deleting the files. I am no where near I/O exhaustion on the SSDs or even the HDDs. Cheers, Christian On Oct 2 2019, at 1:23 pm, Martin Verges wrote: > Hello Christian, > > the

Re: [ceph-users] rgw S3 lifecycle cannot keep up

2019-10-02 Thread Martin Verges
Hello Christian, the problem is, that HDD is not capable of providing lots of IOs required for "~4 million small files". -- Martin Verges Managing director Mobile: +49 174 9335695 E-Mail: martin.ver...@croit.io Chat: https://t.me/MartinVerges croit GmbH, Freseniusstr. 31h, 81247 Munich CEO:

[ceph-users] rgw S3 lifecycle cannot keep up

2019-10-02 Thread Christian Pedersen
Hi, Using the S3 gateway I store ~4 million small files in my cluster every day. I have a lifecycle setup to move these files to cold storage after a day and delete them after two days. The default storage is SSD based and the cold storage is HDD. However the rgw lifecycle process cannot keep