Re: [ceph-users] Luminous Bluestore performance, bcache
Thanks Richard, That sounds impressive, especially the around 30% hit ratio. That would be ideal for me, but we were only getting single digit results during my trials. I think around 5% was the figure if I remember correctly. However, most of our vms were created a bit chaotically (not using predefined vm templates), which could be the reason for the low cache hit rate. If most of your vms are created from templates, you should get a far better cache performance figures I think. But the main factor is the data type, which is very random for our case as we have different applications and services. Would be interested to learn how you are achieving your 30-35% hit ratio on 700 vms. Cheers - Original Message - > From: "Richard Bade" > To: "ceph-users" > Sent: Thursday, 28 June, 2018 23:56:59 > Subject: Re: [ceph-users] Luminous Bluestore performance, bcache > Hi Andrei, > These are good questions. We have another cluster with filestore and > bcache but for this particular one I was interested in testing out > bluestore. So I have used bluestore both with and without bcache. > For my synthetic load on the vm's I'm using this fio command: > fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 > --name=test --filename=test --bs=4k --iodepth=64 --size=4G > --readwrite=randwrite --rate_iops=50 > > Currently on bluestore with my synthetic load I'm getting 7% hit ratio > (cat /sys/block/bcache*/bcache/stats_total/cache_hit_ratio) > On our filestore cluster with ~700 vm's of varied workload we're > geting about 30-35% hit ratio. > In the hourly hit ratio I have as high as 50% on some osd's in our > filestore cluster. Only 25% on my synthetic load on bluestore so far, > but I hadn't actually been checking this stat until now. > > I hope that helps. > Regards, > Richard > >> Hi Richard, >> It is an interesting test for me too as I am planning to migrate to >> Bluestore storage and was considering repurposing the ssd disks >> that we currently use for journals. >> I was wondering if you are using the Filestore or the bluestone >> for the osds? >> Also, when you perform your testing, how good is the hit ratio >> that you have on the bcache? >> Are you using a lot of random data for your benchmarks? How >> large is your test file for each vm? >> We have been playing around with a few caching scenarios a >> few years back (enchanceio and a few more which I can't >> remember now) and we have seen a very poor hit ratio on the >> caching system. Was wondering if you see a different picture? >> Cheers > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Luminous Bluestore performance, bcache
Hi Andrei, These are good questions. We have another cluster with filestore and bcache but for this particular one I was interested in testing out bluestore. So I have used bluestore both with and without bcache. For my synthetic load on the vm's I'm using this fio command: fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=test --bs=4k --iodepth=64 --size=4G --readwrite=randwrite --rate_iops=50 Currently on bluestore with my synthetic load I'm getting 7% hit ratio (cat /sys/block/bcache*/bcache/stats_total/cache_hit_ratio) On our filestore cluster with ~700 vm's of varied workload we're geting about 30-35% hit ratio. In the hourly hit ratio I have as high as 50% on some osd's in our filestore cluster. Only 25% on my synthetic load on bluestore so far, but I hadn't actually been checking this stat until now. I hope that helps. Regards, Richard > Hi Richard, > It is an interesting test for me too as I am planning to migrate to > Bluestore storage and was considering repurposing the ssd disks > that we currently use for journals. > I was wondering if you are using the Filestore or the bluestone > for the osds? > Also, when you perform your testing, how good is the hit ratio > that you have on the bcache? > Are you using a lot of random data for your benchmarks? How > large is your test file for each vm? > We have been playing around with a few caching scenarios a > few years back (enchanceio and a few more which I can't > remember now) and we have seen a very poor hit ratio on the > caching system. Was wondering if you see a different picture? > Cheers ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Luminous Bluestore performance, bcache
Hi Richard, It is an interesting test for me too as I am planning to migrate to Bluestore storage and was considering repurposing the ssd disks that we currently use for journals. I was wondering if you are using the Filestore or the bluestone for the osds? Also, when you perform your testing, how good is the hit ratio that you have on the bcache? Are you using a lot of random data for your benchmarks? How large is your test file for each vm? We have been playing around with a few caching scenarios a few years back (enchanceio and a few more which I can't remember now) and we have seen a very poor hit ratio on the caching system. Was wondering if you see a different picture? Cheers - Original Message - > From: "Richard Bade" > To: "ceph-users" > Sent: Thursday, 28 June, 2018 05:42:34 > Subject: [ceph-users] Luminous Bluestore performance, bcache > Hi Everyone, > There's been a few threads go past around this but I haven't seen any > that pointed me in the right direction. > We've recently set up a new luminous (12.2.5) cluster with 5 hosts > each with 12 4TB Seagate Constellation ES spinning disks for osd's. We > also have 2x 400GB Intel DC P3700's per node. We're using this for rbd > storage for VM's running under Proxmox VE. > I firstly set these up with DB partition (approx 60GB per osd) on nvme > and data directly onto the spinning disk using ceph-deploy create. > This worked great and was very simple. > However performance wasn't great. I fired up 20vm's each running fio > trying to attain 50 iops. Ceph was only just able to keep up with the > 1000iops this generated and vm's started to have trouble hitting their > 50iops target. > So I rebuilt all the osd's halving the DB space (~30GB per osd) and > adding a 200GB BCache partition shared between 6 osd's. Again this > worked great with ceph-deploy create and was very simple. > I have had a vast improvement with my synthetic test. I can now run > 100 50iops test vm's generating a constant 5000iops load and each one > can keep up without any trouble. > > The question I have is if the poor performance out of the box is > expected? Or is there some kind of tweaking I should be doing to make > this usable for rbd images? Are others able to work ok with this kind > of config at a small scale like my 60osd's? Or is it only workable at > a larger scale? > > Regards, > Rich > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Luminous Bluestore performance, bcache
Hi Everyone, There's been a few threads go past around this but I haven't seen any that pointed me in the right direction. We've recently set up a new luminous (12.2.5) cluster with 5 hosts each with 12 4TB Seagate Constellation ES spinning disks for osd's. We also have 2x 400GB Intel DC P3700's per node. We're using this for rbd storage for VM's running under Proxmox VE. I firstly set these up with DB partition (approx 60GB per osd) on nvme and data directly onto the spinning disk using ceph-deploy create. This worked great and was very simple. However performance wasn't great. I fired up 20vm's each running fio trying to attain 50 iops. Ceph was only just able to keep up with the 1000iops this generated and vm's started to have trouble hitting their 50iops target. So I rebuilt all the osd's halving the DB space (~30GB per osd) and adding a 200GB BCache partition shared between 6 osd's. Again this worked great with ceph-deploy create and was very simple. I have had a vast improvement with my synthetic test. I can now run 100 50iops test vm's generating a constant 5000iops load and each one can keep up without any trouble. The question I have is if the poor performance out of the box is expected? Or is there some kind of tweaking I should be doing to make this usable for rbd images? Are others able to work ok with this kind of config at a small scale like my 60osd's? Or is it only workable at a larger scale? Regards, Rich ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
