Re: [ceph-users] block db sizing and calculation
Hi Konstantin! Quoting Konstantin Shalygin (k0...@k0ste.ru): > >Is there any recommandation of how many osds a single flash device can > >serve? The optane ones can do 2000MB/s write + 500.000 iop/s. > > Any sizes of db, except 3/30/300 is useless. I have this from Mattia Belluco in my notes which suggests that twice the amount is best: > Following some discussions we had at the past Cephalocon I beg to differ > on this point: when RocksDB needs to compact a layer it rewrites it > *before* deleting the old data; if you'd like to be sure you db does not > spill over to the spindle you should allocate twice the size of the > biggest layer to allow for compaction. I guess ~60 GB would be the sweet > spot assuming you don't plan to mess with size and multiplier of the > rocksDB layers and don't want to go all the way to 600 GB (300 GB x2) Source is http://lists.ceph.com/pipermail/ceph-users-ceph.com/2019-May/035086.html. And apart from the RocksDB pecularities the actual use case also needs to be considered. Lots of small files on a CephFS will require more DB space than mainly big files as Paul states in the same thread. Cheers, LF. -- Lars Fenneberg, l...@elemental.net ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] block db sizing and calculation
i'm plannung to split the block db to a seperate flash device which i also would like to use as an OSD for erasure coding metadata for rbd devices. If i want to use 14x 14TB HDDs per Node https://docs.ceph.com/docs/master/rados/configuration/bluestore-config-ref/#sizing recommends a minimum size of 140GB per 14TB HDD. Is there any recommandation of how many osds a single flash device can serve? The optane ones can do 2000MB/s write + 500.000 iop/s. Any sizes of db, except 3/30/300 is useless. How much OSD's per NVMe - quantity of OSD's that you can lose once at time. k ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] block db sizing and calculation
One tricky thing is each layer of RocksDB is 100% on SSD or 100% on HDD, so either you need to tweak the rocksdb configuration , or there will be a huge waste, e.g 20GB DB partition makes no difference compared to a 3GB one (under default rocksdb configuration) Janne Johansson 于2020年1月14日周二 下午4:43写道： > (sorry for empty mail just before) > > >> i'm plannung to split the block db to a seperate flash device which i >>> also would like to use as an OSD for erasure coding metadata for rbd >>> devices. >>> >>> If i want to use 14x 14TB HDDs per Node >>> >>> https://docs.ceph.com/docs/master/rados/configuration/bluestore-config-ref/#sizing >>> >>> recommends a minimum size of 140GB per 14TB HDD. >>> >>> Is there any recommandation of how many osds a single flash device can >>> serve? The optane ones can do 2000MB/s write + 500.000 iop/s. >>> >> >> > I think many ceph admins are more concerned with having many drives > co-using the same DB drive, since if the DB drive fails, it also means all > OSDs are lost at the same time. > Optanes and decent NVMEs are probably capable of handling tons of HDDs, so > that the bottleneck ends up being somewhere else, but the failure scenarios > are a bit scary if the whole host is lost just by that one DB device acting > up. > > -- > May the most significant bit of your life be positive. > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] block db sizing and calculation
(sorry for empty mail just before) > i'm plannung to split the block db to a seperate flash device which i >> also would like to use as an OSD for erasure coding metadata for rbd >> devices. >> >> If i want to use 14x 14TB HDDs per Node >> >> https://docs.ceph.com/docs/master/rados/configuration/bluestore-config-ref/#sizing >> >> recommends a minimum size of 140GB per 14TB HDD. >> >> Is there any recommandation of how many osds a single flash device can >> serve? The optane ones can do 2000MB/s write + 500.000 iop/s. >> > > I think many ceph admins are more concerned with having many drives co-using the same DB drive, since if the DB drive fails, it also means all OSDs are lost at the same time. Optanes and decent NVMEs are probably capable of handling tons of HDDs, so that the bottleneck ends up being somewhere else, but the failure scenarios are a bit scary if the whole host is lost just by that one DB device acting up. -- May the most significant bit of your life be positive. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] block db sizing and calculation
Den mån 13 jan. 2020 kl 08:09 skrev Stefan Priebe - Profihost AG < s.pri...@profihost.ag>: > Hello, > > i'm plannung to split the block db to a seperate flash device which i > also would like to use as an OSD for erasure coding metadata for rbd > devices. > > If i want to use 14x 14TB HDDs per Node > > https://docs.ceph.com/docs/master/rados/configuration/bluestore-config-ref/#sizing > > recommends a minimum size of 140GB per 14TB HDD. > > Is there any recommandation of how many osds a single flash device can > serve? The optane ones can do 2000MB/s write + 500.000 iop/s. > > Greets, > Stefan > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > -- May the most significant bit of your life be positive. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] block db sizing and calculation
Hello, does anybody have real live experience with externel block db? Greets, Stefan Am 13.01.20 um 08:09 schrieb Stefan Priebe - Profihost AG: > Hello, > > i'm plannung to split the block db to a seperate flash device which i > also would like to use as an OSD for erasure coding metadata for rbd > devices. > > If i want to use 14x 14TB HDDs per Node > https://docs.ceph.com/docs/master/rados/configuration/bluestore-config-ref/#sizing > > recommends a minimum size of 140GB per 14TB HDD. > > Is there any recommandation of how many osds a single flash device can > serve? The optane ones can do 2000MB/s write + 500.000 iop/s. > > Greets, > Stefan > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] block db sizing and calculation
Hello, i'm plannung to split the block db to a seperate flash device which i also would like to use as an OSD for erasure coding metadata for rbd devices. If i want to use 14x 14TB HDDs per Node https://docs.ceph.com/docs/master/rados/configuration/bluestore-config-ref/#sizing recommends a minimum size of 140GB per 14TB HDD. Is there any recommandation of how many osds a single flash device can serve? The optane ones can do 2000MB/s write + 500.000 iop/s. Greets, Stefan ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
