[ceph-users] Re: RocksDB options for HDD, SSD, NVME Mixed productions

It's been discussed a few times on the list but RocksDB levels essentially grow by a factor of 10 (max_bytes_for_level_multiplier) by default and you need (level-1)*10 space for the next level on your drive to avoid spill over So the sequence (by default) is 256MB -> 2.56GB -> 25.6GB -> 256GB GB a

[ceph-users] Re: RocksDB options for HDD, SSD, NVME Mixed productions

Let me join, having 11 bluefs spillover in my cluster. Where this settings coming from? Istvan Szabo Senior Infrastructure Engineer --- Agoda Services Co., Ltd. e: istvan.sz...@agoda.com ---

[ceph-users] Re: RocksDB options for HDD, SSD, NVME Mixed productions

Thanks for the explanation. Then the first thing I did wrong I didn't add levels to reach total space. I didn't know that and I've set : max_bytes_for_level_base=536870912 and max_bytes_for_level_multiplier=10 536870912*10*10=50Gb I have space on Nvme's. I think I can resize the partitions. 1- Set

[ceph-users] Re: RocksDB options for HDD, SSD, NVME Mixed productions

On Wed, 22 Sept 2021 at 00:54, mhnx wrote: > > Thanks for the explanation. Then the first thing I did wrong I didn't add > levels to reach total space. I didn't know that and I've set : > max_bytes_for_level_base=536870912 and max_bytes_for_level_multiplier=10 > 536870912*10*10=50Gb > > I have sp

[ceph-users] Re: RocksDB options for HDD, SSD, NVME Mixed productions

On Wed, 22 Sept 2021 at 05:54, Szabo, Istvan (Agoda) wrote: > > Sorry to steal it, so if I have 500GB and 700GB mixed wal+rocksdb on nvme > the number should be the level base 50 and 70? Or needs to be > power of 2? Generally the sum of all levels (up to the max of your metadat

[ceph-users] Re: RocksDB options for HDD, SSD, NVME Mixed productions

Sorry to steal it, so if I have 500GB and 700GB mixed wal+rocksdb on nvme the number should be the level base 50 and 70? Or needs to be power of 2? Istvan Szabo Senior Infrastructure Engineer --- Agoda Services Co., Ltd. e: istvan.s

[ceph-users] Re: RocksDB options for HDD, SSD, NVME Mixed productions

Sorry to steal it, so if I have 500GB and 700GB mixed wal+rocksdb on nvme the number should be the level base 50 and 70? Or needs to be power of 2? Istvan Szabo Senior Infrastructure Engineer --- Agoda Services Co., Ltd. e: istvan.

[ceph-users] Re: RocksDB options for HDD, SSD, NVME Mixed productions

On Wed, 22 Sept 2021 at 07:07, Szabo, Istvan (Agoda) wrote: > > Increasing day by day, this is the current situation: (1 server has 6x 15.3TB > SAS ssds, 3x ssds are using 1x 1.92TB nvme for db+wal. > > WRN] BLUEFS_SPILLOVER: 13 OSD(s) experiencing BlueFS spillover > osd.1 spilled over 56 Gi

[ceph-users] Re: RocksDB options for HDD, SSD, NVME Mixed productions

Increasing day by day, this is the current situation: (1 server has 6x 15.3TB SAS ssds, 3x ssds are using 1x 1.92TB nvme for db+wal. WRN] BLUEFS_SPILLOVER: 13 OSD(s) experiencing BlueFS spillover osd.1 spilled over 56 GiB metadata from 'db' device (318 GiB used of 596 GiB) to slow device