[ceph-users] Re: Ceph performance optimization with SSDs

[Szabo, Istvan (Agoda)]

Be careful when you are designing, if you are planning to have billions of 
objects, you need to have more than 2-4% for rocksdb+wal to avoid spillover.

Istvan Szabo
Senior Infrastructure Engineer
---
Agoda Services Co., Ltd.
e: istvan.sz...@agoda.com
---

On 2021. Oct 22., at 14:47, Peter Sabaini  wrote:

Email received from the internet. If in doubt, don't click any link nor open 
any attachment !


On 22.10.21 11:29, Mevludin Blazevic wrote:
Dear Ceph users,

I have a small Ceph cluster where each host consist of a small amount of SSDs 
and a larger number of HDDs. Is there a way to use the SSDs as performance 
optimization such as putting OSD Journals to the SSDs and/or using SSDs for 
caching?


Hi,

yes, SSDs can be put to good use as journal devices[0] for OSDs, or you could 
use them as caching devices for bcache[1]. This is actually a pretty widespread 
setup, theres docs for various scenarios[2][3]

But be aware that OSD journals are pretty write-intense, so be sure to use 
fast, reliable SSDs (or NVMes). I have actually seen OSD performance (esp. 
latency and jitter) actually worsen with (prosumer-grade) SSDs that could not 
support the sustained write load.

If at all in doubt run tests before putting prod on it


[0] https://docs.ceph.com/en/latest/start/hardware-recommendations/
[1] https://bcache.evilpiepirate.org/
[2] https://charmhub.io/ceph-osd
[3] 
https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/4/html/object_gateway_for_production_guide/using-nvme-with-lvm-optimally

Best regards,
Mevludin

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Ceph performance optimization with SSDs

[MERZOUKI, HAMID]

Hello Mevludin Blazevic,

Yes it is quite possible, I did it on my cluster.
I put the cephDB (WAL + RocksDB) hard drives on the SSD drives.
That increases the performance of HDD.

Regards

-Message d'origine-
De : Mevludin Blazevic  
Envoyé : vendredi 22 octobre 2021 11:30
À : Ceph Users 
Objet : [ceph-users] Ceph performance optimization with SSDs

Caution! External email. Do not open attachments or click links, unless this 
email comes from a known sender and you know the content is safe.

Dear Ceph users,

I have a small Ceph cluster where each host consist of a small amount of SSDs 
and a larger number of HDDs. Is there a way to use the SSDs as performance 
optimization such as putting OSD Journals to the SSDs and/or using SSDs for 
caching?

Best regards,
Mevludin

--
Mevludin Blazevic, M.Sc.

University of Koblenz-Landau
Computing Centre (GHRKO)
Universitaetsstrasse 1
D-56070 Koblenz, Germany
Room A023
Tel: +49 261/287-1326

___
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to 
ceph-users-le...@ceph.io

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Ceph performance optimization with SSDs

[Peter Sabaini]

On 22.10.21 11:29, Mevludin Blazevic wrote:
> Dear Ceph users,
> 
> I have a small Ceph cluster where each host consist of a small amount of SSDs 
> and a larger number of HDDs. Is there a way to use the SSDs as performance 
> optimization such as putting OSD Journals to the SSDs and/or using SSDs for 
> caching?


Hi,

yes, SSDs can be put to good use as journal devices[0] for OSDs, or you could 
use them as caching devices for bcache[1]. This is actually a pretty widespread 
setup, theres docs for various scenarios[2][3]

But be aware that OSD journals are pretty write-intense, so be sure to use 
fast, reliable SSDs (or NVMes). I have actually seen OSD performance (esp. 
latency and jitter) actually worsen with (prosumer-grade) SSDs that could not 
support the sustained write load.

If at all in doubt run tests before putting prod on it


[0] https://docs.ceph.com/en/latest/start/hardware-recommendations/
[1] https://bcache.evilpiepirate.org/
[2] https://charmhub.io/ceph-osd
[3] 
https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/4/html/object_gateway_for_production_guide/using-nvme-with-lvm-optimally

> Best regards,
> Mevludin
> 
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Ceph performance optimization with SSDs

[Mevludin Blazevic]

Hi,

thank you very much for your quick response! Ok, so putting the DB/WAL 
on the NVMe drives could be beneficial. What happens if your SSD with 
the DB/WAL on it breaks? I assume you configured some type of 
(double/triple, etc.) replication?


Regards,

Mevludin

Am 22.10.2021 um 11:47 schrieb Zakhar Kirpichenko:

Hi,

It is my experience with Bluestore that even if you put DB/WAL on a 
very fast SSD (NVME in my case), the OSD will still refuse to write 
faster than the storage device (HDD) can write on average. This is a 
bummer because the behavior is very different from Filestore, which 
could buffer write bursts on SSD, defer and write them to HDD later 
on. Having DB/WAL on SSD still helps with general latency, though.


With regards to SSD cache, putting an SSD cache tier with Ceph is 
possible, but be prepared that it doesn't quite work the way you would 
expect it to, that its performance is very highly dependent on your 
workload, and that it may cause sudden issues: for example, with 
16.2.x my VMs would randomly lose access to their RBD images during 
intense writes through a cache tier. Putting a cache tier for an OSD 
device with bcache, dm-cache, etc may be a better and more stable 
option, although it is unlikely that you will be able to automate this 
with Ceph toolset.


Best regards,
Z

On Fri, Oct 22, 2021 at 12:30 PM Mevludin Blazevic 
mailto:mblaze...@uni-koblenz.de>> wrote:


Dear Ceph users,

I have a small Ceph cluster where each host consist of a small
amount of
SSDs and a larger number of HDDs. Is there a way to use the SSDs as
performance optimization such as putting OSD Journals to the SSDs
and/or
using SSDs for caching?

Best regards,
Mevludin

-- 
Mevludin Blazevic, M.Sc.


University of Koblenz-Landau
Computing Centre (GHRKO)
Universitaetsstrasse 1
D-56070 Koblenz, Germany
Room A023
Tel: +49 261/287-1326

___
ceph-users mailing list -- ceph-users@ceph.io

To unsubscribe send an email to ceph-users-le...@ceph.io



--
Mevludin Blazevic, M.Sc.

University of Koblenz-Landau
Computing Centre (GHRKO)
Universitaetsstrasse 1
D-56070 Koblenz, Germany
Room A023
Tel: +49 261/287-1326

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Ceph performance optimization with SSDs

[Zakhar Kirpichenko]

Hi,

It is my experience with Bluestore that even if you put DB/WAL on a very
fast SSD (NVME in my case), the OSD will still refuse to write faster than
the storage device (HDD) can write on average. This is a bummer because the
behavior is very different from Filestore, which could buffer write bursts
on SSD, defer and write them to HDD later on. Having DB/WAL on SSD still
helps with general latency, though.

With regards to SSD cache, putting an SSD cache tier with Ceph is possible,
but be prepared that it doesn't quite work the way you would expect it to,
that its performance is very highly dependent on your workload, and that it
may cause sudden issues: for example, with 16.2.x my VMs would randomly
lose access to their RBD images during intense writes through a cache tier.
Putting a cache tier for an OSD device with bcache, dm-cache, etc may be a
better and more stable option, although it is unlikely that you will be
able to automate this with Ceph toolset.

Best regards,
Z

On Fri, Oct 22, 2021 at 12:30 PM Mevludin Blazevic 
wrote:

> Dear Ceph users,
>
> I have a small Ceph cluster where each host consist of a small amount of
> SSDs and a larger number of HDDs. Is there a way to use the SSDs as
> performance optimization such as putting OSD Journals to the SSDs and/or
> using SSDs for caching?
>
> Best regards,
> Mevludin
>
> --
> Mevludin Blazevic, M.Sc.
>
> University of Koblenz-Landau
> Computing Centre (GHRKO)
> Universitaetsstrasse 1
> D-56070 Koblenz, Germany
> Room A023
> Tel: +49 261/287-1326
>
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io