Re: [ceph-users] Hardware cache settings recomendation
Those are strange numbers, where are you getting them from? Test the drives directly with fio with every combination, that’s should tell you what’s happening Jan On 18 Jun 2015, at 07:52, Mateusz Skała mateusz.sk...@budikom.net wrote: Thanks for answer, I made some test, first leave dwc=enabled and caching on journal drive disabled. Latency grows from 20ms to 90ms on this drive. Next I enabled cache on journal drive and disabled all cache on data drives. Latency on data drives grows from 30 – 50ms to 1500 – 2000ms. Test made only on one osd host with P410i controller, with SATA drives ST1000LM014-1EJ1 for data and for journal SSD INTEL SSDSC2BW12. Regards, Mateusz From: Jan Schermer [mailto:j...@schermer.cz] Sent: Wednesday, June 17, 2015 9:41 AM To: Mateusz Skała Cc: ceph-users@lists.ceph.com Subject: Re: [ceph-users] Hardware cache settings recomendation Cache on top of the data drives (not journal) will not help in most cases, those writes are already buffered in the OS - so unless your OS is very light on memory and flushing constantly it will have no effect, it just adds overhead in case a flush comes. I haven’t tested this extensively with Ceph, though. Cache enabled on journal drive _could_ help if your SSD is very slow (or if you don’t have SSD for journal at all), and if it is large enough (more than the active journal size) it could prolong the life of your SSD - depending on how and when the cache starts to flush. I know from experience that write cache on Areca controller didn't flush at all until it hit a watermark (50% capacity default or something) and it will be faster than some SSDon their own. Some SSD have higher IOPS than the cache can achieve, but you likely won’t saturate that with Ceph. Another thing is write cache on the drives themselves - I’d leave that on disabled (which is probably the default) unless the drive in question has capacitors to flush the cache in case of power failure. Controllers usually have a whitelist of devices that respect flushes on which the write cache is default=enabled, but in case of for example Dell Perc you would need to have Dell original drives or enable it manually. YMMV - i’ve hit the controller cache IOPS limit in the past with cheap Dell Perc (H310 was it?) that did ~20K IOPS top on one SSD drive, while the drive itself did close to 40K. On my SSDs, disabling write cache helps latency (good for journal) bud could be troubling for the SSD lifetime. In any case I don’t think you would saturate either with Ceph, so I recommend you just test the latency with write cache enabled/disabled on the controller and pick the one that gives the best numbers this is basically how: http://www.sebastien-han.fr/blog/2014/10/10/ceph-how-to-test-if-your-ssd-is-suitable-as-a-journal-device/ Ceph recommended way is to use everything as passthrough (initiator/target mode) or JBOD (RAID0 with single drives on some controllers), so I’d stick with that. Jan On 17 Jun 2015, at 08:01, Mateusz Skała mateusz.sk...@budikom.net wrote: Yes, all disk are in single drive raid 0. Now cache is enabled for all drives, should I disable cache for SSD drives? Regards, Mateusz From: Tyler Bishop [mailto:tyler.bis...@beyondhosting.net] Sent: Thursday, June 11, 2015 7:30 PM To: Mateusz Skała Cc: ceph-users@lists.ceph.com Subject: Re: [ceph-users] Hardware cache settings recomendation You want write cache to disk, no write cache for SSD. I assume all of your data disk are single drive raid 0? Tyler Bishop Chief Executive Officer 513-299-7108 x10 tyler.bis...@beyondhosting.net If you are not the intended recipient of this transmission you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited. From: Mateusz Skała mateusz.sk...@budikom.net To: ceph-users@lists.ceph.com Sent: Saturday, June 6, 2015 4:09:59 AM Subject: [ceph-users] Hardware cache settings recomendation Hi, Please help me with hardware cache settings on controllers for ceph rbd best performance. All Ceph hosts have one SSD drive for journal. We are using 4 different controllers, all with BBU: • HP Smart Array P400 • HP Smart Array P410i • Dell PERC 6/i • Dell PERC H700 I have to set cache policy, on Dell settings are: • Read Policy o Read-Ahead (current) o No-Read-Ahead o Adaptive Read-Ahead • Write Policy o Write-Back (current) o Write-Through • Cache Policy o Cache I/O o Direct I/O (current) • Disk Cache Policy o Default (current) o Enabled o Disabled On HP controllers: • Cache Ratio (current: 25% Read / 75% Write) • Drive Write Cache o Enabled (current) o Disabled And there is one more setting
Re: [ceph-users] Hardware cache settings recomendation
Thanks for answer, I made some test, first leave dwc=enabled and caching on journal drive disabled. Latency grows from 20ms to 90ms on this drive. Next I enabled cache on journal drive and disabled all cache on data drives. Latency on data drives grows from 30 – 50ms to 1500 – 2000ms. Test made only on one osd host with P410i controller, with SATA drives ST1000LM014-1EJ1 for data and for journal SSD INTEL SSDSC2BW12. Regards, Mateusz From: Jan Schermer [mailto:j...@schermer.cz] Sent: Wednesday, June 17, 2015 9:41 AM To: Mateusz Skała Cc: ceph-users@lists.ceph.com Subject: Re: [ceph-users] Hardware cache settings recomendation Cache on top of the data drives (not journal) will not help in most cases, those writes are already buffered in the OS - so unless your OS is very light on memory and flushing constantly it will have no effect, it just adds overhead in case a flush comes. I haven’t tested this extensively with Ceph, though. Cache enabled on journal drive _could_ help if your SSD is very slow (or if you don’t have SSD for journal at all), and if it is large enough (more than the active journal size) it could prolong the life of your SSD - depending on how and when the cache starts to flush. I know from experience that write cache on Areca controller didn't flush at all until it hit a watermark (50% capacity default or something) and it will be faster than some SSDon their own. Some SSD have higher IOPS than the cache can achieve, but you likely won’t saturate that with Ceph. Another thing is write cache on the drives themselves - I’d leave that on disabled (which is probably the default) unless the drive in question has capacitors to flush the cache in case of power failure. Controllers usually have a whitelist of devices that respect flushes on which the write cache is default=enabled, but in case of for example Dell Perc you would need to have Dell original drives or enable it manually. YMMV - i’ve hit the controller cache IOPS limit in the past with cheap Dell Perc (H310 was it?) that did ~20K IOPS top on one SSD drive, while the drive itself did close to 40K. On my SSDs, disabling write cache helps latency (good for journal) bud could be troubling for the SSD lifetime. In any case I don’t think you would saturate either with Ceph, so I recommend you just test the latency with write cache enabled/disabled on the controller and pick the one that gives the best numbers this is basically how: http://www.sebastien-han.fr/blog/2014/10/10/ceph-how-to-test-if-your-ssd-is-suitable-as-a-journal-device/ Ceph recommended way is to use everything as passthrough (initiator/target mode) or JBOD (RAID0 with single drives on some controllers), so I’d stick with that. Jan On 17 Jun 2015, at 08:01, Mateusz Skała mateusz.sk...@budikom.net wrote: Yes, all disk are in single drive raid 0. Now cache is enabled for all drives, should I disable cache for SSD drives? Regards, Mateusz From: Tyler Bishop [mailto:tyler.bis...@beyondhosting.net] Sent: Thursday, June 11, 2015 7:30 PM To: Mateusz Skała Cc: ceph-users@lists.ceph.com Subject: Re: [ceph-users] Hardware cache settings recomendation You want write cache to disk, no write cache for SSD. I assume all of your data disk are single drive raid 0? Tyler Bishop Chief Executive Officer 513-299-7108 x10 tyler.bis...@beyondhosting.net If you are not the intended recipient of this transmission you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited. From: Mateusz Skała mateusz.sk...@budikom.net To: ceph-users@lists.ceph.com Sent: Saturday, June 6, 2015 4:09:59 AM Subject: [ceph-users] Hardware cache settings recomendation Hi, Please help me with hardware cache settings on controllers for ceph rbd best performance. All Ceph hosts have one SSD drive for journal. We are using 4 different controllers, all with BBU: • HP Smart Array P400 • HP Smart Array P410i • Dell PERC 6/i • Dell PERC H700 I have to set cache policy, on Dell settings are: • Read Policy o Read-Ahead (current) o No-Read-Ahead o Adaptive Read-Ahead • Write Policy o Write-Back (current) o Write-Through • Cache Policy o Cache I/O o Direct I/O (current) • Disk Cache Policy o Default (current) o Enabled o Disabled On HP controllers: • Cache Ratio (current: 25% Read / 75% Write) • Drive Write Cache o Enabled (current) o Disabled And there is one more setting in LogicalDrive option: • Caching: o Enabled (current) o Disabled Please verify my settings and give me some recomendations. Best regards, Mateusz ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Hardware cache settings recomendation
Yes, all disk are in single drive raid 0. Now cache is enabled for all drives, should I disable cache for SSD drives? Regards, Mateusz From: Tyler Bishop [mailto:tyler.bis...@beyondhosting.net] Sent: Thursday, June 11, 2015 7:30 PM To: Mateusz Skała Cc: ceph-users@lists.ceph.com Subject: Re: [ceph-users] Hardware cache settings recomendation You want write cache to disk, no write cache for SSD. I assume all of your data disk are single drive raid 0? http://static.beyondhosting.net/img/bh-small.png Tyler Bishop Chief Executive Officer 513-299-7108 x10 tyler.bis...@beyondhosting.net mailto:tyler.bis...@beyondhosting.net If you are not the intended recipient of this transmission you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited. _ From: Mateusz Skała mateusz.sk...@budikom.net mailto:mateusz.sk...@budikom.net To: ceph-users@lists.ceph.com mailto:ceph-users@lists.ceph.com Sent: Saturday, June 6, 2015 4:09:59 AM Subject: [ceph-users] Hardware cache settings recomendation Hi, Please help me with hardware cache settings on controllers for ceph rbd best performance. All Ceph hosts have one SSD drive for journal. We are using 4 different controllers, all with BBU: * HP Smart Array P400 * HP Smart Array P410i * Dell PERC 6/i * Dell PERC H700 I have to set cache policy, on Dell settings are: * Read Policy o Read-Ahead (current) o No-Read-Ahead o Adaptive Read-Ahead * Write Policy o Write-Back (current) o Write-Through * Cache Policy o Cache I/O o Direct I/O (current) * Disk Cache Policy o Default (current) o Enabled o Disabled On HP controllers: * Cache Ratio (current: 25% Read / 75% Write) * Drive Write Cache o Enabled (current) o Disabled And there is one more setting in LogicalDrive option: * Caching: o Enabled (current) o Disabled Please verify my settings and give me some recomendations. Best regards, Mateusz ___ ceph-users mailing list ceph-users@lists.ceph.com mailto:ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Hardware cache settings recomendation
Cache on top of the data drives (not journal) will not help in most cases, those writes are already buffered in the OS - so unless your OS is very light on memory and flushing constantly it will have no effect, it just adds overhead in case a flush comes. I haven’t tested this extensively with Ceph, though. Cache enabled on journal drive _could_ help if your SSD is very slow (or if you don’t have SSD for journal at all), and if it is large enough (more than the active journal size) it could prolong the life of your SSD - depending on how and when the cache starts to flush. I know from experience that write cache on Areca controller didn't flush at all until it hit a watermark (50% capacity default or something) and it will be faster than some SSDon their own. Some SSD have higher IOPS than the cache can achieve, but you likely won’t saturate that with Ceph. Another thing is write cache on the drives themselves - I’d leave that on disabled (which is probably the default) unless the drive in question has capacitors to flush the cache in case of power failure. Controllers usually have a whitelist of devices that respect flushes on which the write cache is default=enabled, but in case of for example Dell Perc you would need to have Dell original drives or enable it manually. YMMV - i’ve hit the controller cache IOPS limit in the past with cheap Dell Perc (H310 was it?) that did ~20K IOPS top on one SSD drive, while the drive itself did close to 40K. On my SSDs, disabling write cache helps latency (good for journal) bud could be troubling for the SSD lifetime. In any case I don’t think you would saturate either with Ceph, so I recommend you just test the latency with write cache enabled/disabled on the controller and pick the one that gives the best numbers this is basically how: http://www.sebastien-han.fr/blog/2014/10/10/ceph-how-to-test-if-your-ssd-is-suitable-as-a-journal-device/ http://www.sebastien-han.fr/blog/2014/10/10/ceph-how-to-test-if-your-ssd-is-suitable-as-a-journal-device/ Ceph recommended way is to use everything as passthrough (initiator/target mode) or JBOD (RAID0 with single drives on some controllers), so I’d stick with that. Jan On 17 Jun 2015, at 08:01, Mateusz Skała mateusz.sk...@budikom.net wrote: Yes, all disk are in single drive raid 0. Now cache is enabled for all drives, should I disable cache for SSD drives? Regards, Mateusz From: Tyler Bishop [mailto:tyler.bis...@beyondhosting.net mailto:tyler.bis...@beyondhosting.net] Sent: Thursday, June 11, 2015 7:30 PM To: Mateusz Skała Cc: ceph-users@lists.ceph.com mailto:ceph-users@lists.ceph.com Subject: Re: [ceph-users] Hardware cache settings recomendation You want write cache to disk, no write cache for SSD. I assume all of your data disk are single drive raid 0? Tyler Bishop Chief Executive Officer 513-299-7108 x10 tyler.bis...@beyondhosting.net mailto:tyler.bis...@beyondhosting.net If you are not the intended recipient of this transmission you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited. From: Mateusz Skała mateusz.sk...@budikom.net mailto:mateusz.sk...@budikom.net To: ceph-users@lists.ceph.com mailto:ceph-users@lists.ceph.com Sent: Saturday, June 6, 2015 4:09:59 AM Subject: [ceph-users] Hardware cache settings recomendation Hi, Please help me with hardware cache settings on controllers for ceph rbd best performance. All Ceph hosts have one SSD drive for journal. We are using 4 different controllers, all with BBU: · HP Smart Array P400 · HP Smart Array P410i · Dell PERC 6/i · Dell PERC H700 I have to set cache policy, on Dell settings are: · Read Policy o Read-Ahead (current) o No-Read-Ahead o Adaptive Read-Ahead · Write Policy o Write-Back (current) o Write-Through · Cache Policy o Cache I/O o Direct I/O (current) · Disk Cache Policy o Default (current) o Enabled o Disabled On HP controllers: · Cache Ratio (current: 25% Read / 75% Write) · Drive Write Cache o Enabled (current) o Disabled And there is one more setting in LogicalDrive option: · Caching: o Enabled (current) o Disabled Please verify my settings and give me some recomendations. Best regards, Mateusz ___ ceph-users mailing list ceph-users@lists.ceph.com mailto:ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com___ ceph-users mailing list ceph-users@lists.ceph.com mailto:ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Hardware cache settings recomendation
You want write cache to disk, no write cache for SSD. I assume all of your data disk are single drive raid 0? Tyler Bishop Chief Executive Officer 513-299-7108 x10 tyler.bis...@beyondhosting.net If you are not the intended recipient of this transmission you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited. From: Mateusz Skała mateusz.sk...@budikom.net To: ceph-users@lists.ceph.com Sent: Saturday, June 6, 2015 4:09:59 AM Subject: [ceph-users] Hardware cache settings recomendation Hi, Please help me with hardware cache settings on controllers for ceph rbd best performance. All Ceph hosts have one SSD drive for journal. We are using 4 different controllers, all with BBU: · HP Smart Array P400 · HP Smart Array P410i · Dell PERC 6/i · Dell PERC H700 I have to set cache policy, on Dell settings are: · Read Policy o Read-Ahead (current) o No-Read-Ahead o Adaptive Read-Ahead · Write Policy o Write-Back (current) o Write-Through · Cache Policy o Cache I/O o Direct I/O (current) · Disk Cache Policy o Default (current) o Enabled o Disabled On HP controllers: · Cache Ratio (current: 25% Read / 75% Write) · Drive Write Cache o Enabled (current) o Disabled And there is one more setting in LogicalDrive option: · Caching: o Enabled (current) o Disabled Please verify my settings and give me some recomendations. Best regards, Mateusz ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
