Re: [ceph-users] Ceph cluster with SSDs

2017-09-12 Thread Christian Balzer 


Please don't remove the ML.
I'm not a support channel and if I reply to mails it is so that
others hopefully will learn from that. 
ML re-added.

On Mon, 11 Sep 2017 16:30:18 +0530 M Ranga Swami Reddy wrote:

> >>> >> Here I have NVMes from Intel. but as the support of these NVMes not
> >>> >> there from Intel, we decided not to use these NVMes as a journal.  
> >>> >
> >>> > You again fail to provide with specific model numbers...  
> >>>
> >>> NEMe - Intel DC P3608  - 1.6TB  
> >>
> >> 3DWPD, so you could put this in front (journal~ of 30 or so of those
> >> Samsungs and it still would last longer.  
> >
> >
> > Sure, I will try this and update the results.
> >
> > Btw,  the "osd bench" showing very number with these SSD based (as
> > compared with HDD baed) OSDs). For ex: HDD based OSDs showing around
> > 500 MB/s and SSD based OSDs showing < 300 MB/s. Its strage to see this
> > results.
> > Any thing do I miss here?  
> 
OSD bench still is the wrong test tool for pretty much anything.
There are no HDDs that write 500MB/s.
So this is either a RAID or something behind a controller with HW cache,
not the 140MB/s or so I'd expect to see with a directly connected HDD.
OSD bench also only writes 1GB by default, something that's easily cached
in such a setup.

The 300MB/s for your EVO SSDs could be the result of how OSD bench works
(sync writes, does it use the journal?) or something simple and silly as
these SSDs hooked up to SATA-2 (3Gb/s aka 300MB/s) ports.

> 
> After adding the NVMe drives ad journal, I could see the osd bench
> improved results (showing > 600 MB/s (without NVMe < 300MB/s)..
> 
What exactly did you do?
How many journals per NVMe?
6 or 7 of the EVOs (at 300 MB/s) will saturate the P3608 in a bandwidth
test. 
4 of the EVOs if their 500MB/s write speed can be achieved.

And since you never mentioned any other details, your cluster could also
be network or CPU bound for all we know.

> But  volumes created from SSD pool, not showing any performance
> improvements (like dd o/p, fio, rbd map, rados bench etc)..

If you're using fio with the rbd io module, I found it to be horribly
buggy.
Best real life test is a fio with a VM.
And to test for IOPS (4k ones), bandwidth is most likely NOT what you will
lack in production. 

> Do I miss any ceph config. setting to above good performacne numbers?
>
Not likely, no.
 
> Thanks
> Swami
> 
> 
> >>> > No support from Intel suggests that these may be consumer models again.
> >>> >
> >>> > Samsung also makes DC grade SSDs and NVMEs, as Adrian pointed out.
> >>> >  
> >>> >> Btw, if we split this SSD with multiple OSD (for ex: 1 SSD with 4 or 2
> >>> >> OSDs), is  this help any performance numbers?
> >>> >>  
> >>> > Of course not, if anything it will make it worse due to the overhead
> >>> > outside the SSD itself.
> >>> >
> >>> > Christian
> >>> >  
> >>> >> On Sun, Aug 20, 2017 at 9:33 AM, Christian Balzer  
> >>> >> wrote:  
> >>> >> >
> >>> >> > Hello,
> >>> >> >
> >>> >> > On Sat, 19 Aug 2017 23:22:11 +0530 M Ranga Swami Reddy wrote:
> >>> >> >  
> >>> >> >> SSD make details : SSD 850 EVO 2.5" SATA III 4TB Memory & Storage -
> >>> >> >> MZ-75E4T0B/AM | Samsung
> >>> >> >>  
> >>> >> > And there's your answer.
> >>> >> >
> >>> >> > A bit of googling in the archives here would have shown you that 
> >>> >> > these are
> >>> >> > TOTALLY unsuitable for use with Ceph.
> >>> >> > Not only because of the horrid speed when used with/for Ceph 
> >>> >> > journaling
> >>> >> > (direct/sync I/O) but also their abysmal endurance of 0.04 DWPD over 
> >>> >> > 5
> >>> >> > years.
> >>> >> > Or in other words 160GB/day, which after the Ceph journal double 
> >>> >> > writes
> >>> >> > and FS journals, other overhead and write amplification in general
> >>> >> > probably means less that effective 40GB/day.
> >>> >> >
> >>> >> > In contrast the lowest endurance DC grade SSDs tend to be 0.3 DWPD 
> >>> >> > and
> >>> >> > more commonly 1 DWPD.
> >>> >> > And I'm not buying anything below 3 DWPD for use with Ceph.
> >>> >> >
> >>> >> > Your only chance to improve the speed here is to take the journals 
> >>> >> > off
> >>> >> > them and put them onto fast and durable enough NVMes like the Intel 
> >>> >> > DC P
> >>> >> > 3700 or at worst 3600 types.
> >>> >> >
> >>> >> > That still leaves you with their crappy endurance, only twice as 
> >>> >> > high than
> >>> >> > before with the journals offloaded.
> >>> >> >
> >>> >> > Christian
> >>> >> >  
> >>> >> >> On Sat, Aug 19, 2017 at 10:44 PM, M Ranga Swami Reddy
> >>> >> >>  wrote:  
> >>> >> >> > Yes, Its in production and used the pg count as per the pg 
> >>> >> >> > calcuator @ ceph.com.
> >>> >> >> >
> >>> >> >> > On Fri, Aug 18, 2017 at 3:30 AM, Mehmet  wrote:  
> >>> >> >> >> Which ssds are used? Are they in production? If so how is your 
> >>> >> >> >> PG Count?
> >>> >> >> >>
> >>> >> >> >> Am 17. August 2017 20:04:25 MESZ schrieb M Ranga Swami Reddy
>

Re: [ceph-users] Ceph cluster with SSDs

2017-08-23 Thread Christian Balzer 
On Wed, 23 Aug 2017 16:48:12 +0530 M Ranga Swami Reddy wrote:

> On Mon, Aug 21, 2017 at 5:37 PM, Christian Balzer  wrote:
> > On Mon, 21 Aug 2017 17:13:10 +0530 M Ranga Swami Reddy wrote:
> >  
> >> Thank you.
> >> Here I have NVMes from Intel. but as the support of these NVMes not
> >> there from Intel, we decided not to use these NVMes as a journal.  
> >
> > You again fail to provide with specific model numbers...  
> 
> NEMe - Intel DC P3608  - 1.6TB

3DWPD, so you could put this in front (journal~ of 30 or so of those
Samsungs and it still would last longer.

Christian

> 
> Thanks
> Swami
> 
> > No support from Intel suggests that these may be consumer models again.
> >
> > Samsung also makes DC grade SSDs and NVMEs, as Adrian pointed out.
> >  
> >> Btw, if we split this SSD with multiple OSD (for ex: 1 SSD with 4 or 2
> >> OSDs), is  this help any performance numbers?
> >>  
> > Of course not, if anything it will make it worse due to the overhead
> > outside the SSD itself.
> >
> > Christian
> >  
> >> On Sun, Aug 20, 2017 at 9:33 AM, Christian Balzer  wrote:  
> >> >
> >> > Hello,
> >> >
> >> > On Sat, 19 Aug 2017 23:22:11 +0530 M Ranga Swami Reddy wrote:
> >> >  
> >> >> SSD make details : SSD 850 EVO 2.5" SATA III 4TB Memory & Storage -
> >> >> MZ-75E4T0B/AM | Samsung
> >> >>  
> >> > And there's your answer.
> >> >
> >> > A bit of googling in the archives here would have shown you that these 
> >> > are
> >> > TOTALLY unsuitable for use with Ceph.
> >> > Not only because of the horrid speed when used with/for Ceph journaling
> >> > (direct/sync I/O) but also their abysmal endurance of 0.04 DWPD over 5
> >> > years.
> >> > Or in other words 160GB/day, which after the Ceph journal double writes
> >> > and FS journals, other overhead and write amplification in general
> >> > probably means less that effective 40GB/day.
> >> >
> >> > In contrast the lowest endurance DC grade SSDs tend to be 0.3 DWPD and
> >> > more commonly 1 DWPD.
> >> > And I'm not buying anything below 3 DWPD for use with Ceph.
> >> >
> >> > Your only chance to improve the speed here is to take the journals off
> >> > them and put them onto fast and durable enough NVMes like the Intel DC P
> >> > 3700 or at worst 3600 types.
> >> >
> >> > That still leaves you with their crappy endurance, only twice as high 
> >> > than
> >> > before with the journals offloaded.
> >> >
> >> > Christian
> >> >  
> >> >> On Sat, Aug 19, 2017 at 10:44 PM, M Ranga Swami Reddy
> >> >>  wrote:  
> >> >> > Yes, Its in production and used the pg count as per the pg calcuator 
> >> >> > @ ceph.com.
> >> >> >
> >> >> > On Fri, Aug 18, 2017 at 3:30 AM, Mehmet  wrote:  
> >> >> >> Which ssds are used? Are they in production? If so how is your PG 
> >> >> >> Count?
> >> >> >>
> >> >> >> Am 17. August 2017 20:04:25 MESZ schrieb M Ranga Swami Reddy
> >> >> >> :  
> >> >> >>>
> >> >> >>> Hello,
> >> >> >>> I am using the Ceph cluster with HDDs and SSDs. Created separate 
> >> >> >>> pool for
> >> >> >>> each.
> >> >> >>> Now, when I ran the "ceph osd bench", HDD's OSDs show around 500 
> >> >> >>> MB/s
> >> >> >>> and SSD's OSD show around 280MB/s.
> >> >> >>>
> >> >> >>> Ideally, what I expected was - SSD's OSDs should be at-least 40% 
> >> >> >>> high
> >> >> >>> as compared with HDD's OSD bench.
> >> >> >>>
> >> >> >>> Did I miss anything here? Any hint is appreciated.
> >> >> >>>
> >> >> >>> Thanks
> >> >> >>> Swami
> >> >> >>> 
> >> >> >>>
> >> >> >>> ceph-users mailing list
> >> >> >>> ceph-users@lists.ceph.com
> >> >> >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com  
> >> >> >>
> >> >> >>
> >> >> >> ___
> >> >> >> ceph-users mailing list
> >> >> >> ceph-users@lists.ceph.com
> >> >> >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >> >> >>  
> >> >> ___
> >> >> ceph-users mailing list
> >> >> ceph-users@lists.ceph.com
> >> >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >> >>  
> >> >
> >> >
> >> > --
> >> > Christian BalzerNetwork/Systems Engineer
> >> > ch...@gol.com   Rakuten Communications  
> >>  
> >
> >
> > --
> > Christian BalzerNetwork/Systems Engineer
> > ch...@gol.com   Rakuten Communications  
> 


-- 
Christian BalzerNetwork/Systems Engineer
ch...@gol.com   Rakuten Communications
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph cluster with SSDs

2017-08-23 Thread M Ranga Swami Reddy 
On Mon, Aug 21, 2017 at 5:37 PM, Christian Balzer  wrote:
> On Mon, 21 Aug 2017 17:13:10 +0530 M Ranga Swami Reddy wrote:
>
>> Thank you.
>> Here I have NVMes from Intel. but as the support of these NVMes not
>> there from Intel, we decided not to use these NVMes as a journal.
>
> You again fail to provide with specific model numbers...

NEMe - Intel DC P3608  - 1.6TB


Thanks
Swami

> No support from Intel suggests that these may be consumer models again.
>
> Samsung also makes DC grade SSDs and NVMEs, as Adrian pointed out.
>
>> Btw, if we split this SSD with multiple OSD (for ex: 1 SSD with 4 or 2
>> OSDs), is  this help any performance numbers?
>>
> Of course not, if anything it will make it worse due to the overhead
> outside the SSD itself.
>
> Christian
>
>> On Sun, Aug 20, 2017 at 9:33 AM, Christian Balzer  wrote:
>> >
>> > Hello,
>> >
>> > On Sat, 19 Aug 2017 23:22:11 +0530 M Ranga Swami Reddy wrote:
>> >
>> >> SSD make details : SSD 850 EVO 2.5" SATA III 4TB Memory & Storage -
>> >> MZ-75E4T0B/AM | Samsung
>> >>
>> > And there's your answer.
>> >
>> > A bit of googling in the archives here would have shown you that these are
>> > TOTALLY unsuitable for use with Ceph.
>> > Not only because of the horrid speed when used with/for Ceph journaling
>> > (direct/sync I/O) but also their abysmal endurance of 0.04 DWPD over 5
>> > years.
>> > Or in other words 160GB/day, which after the Ceph journal double writes
>> > and FS journals, other overhead and write amplification in general
>> > probably means less that effective 40GB/day.
>> >
>> > In contrast the lowest endurance DC grade SSDs tend to be 0.3 DWPD and
>> > more commonly 1 DWPD.
>> > And I'm not buying anything below 3 DWPD for use with Ceph.
>> >
>> > Your only chance to improve the speed here is to take the journals off
>> > them and put them onto fast and durable enough NVMes like the Intel DC P
>> > 3700 or at worst 3600 types.
>> >
>> > That still leaves you with their crappy endurance, only twice as high than
>> > before with the journals offloaded.
>> >
>> > Christian
>> >
>> >> On Sat, Aug 19, 2017 at 10:44 PM, M Ranga Swami Reddy
>> >>  wrote:
>> >> > Yes, Its in production and used the pg count as per the pg calcuator @ 
>> >> > ceph.com.
>> >> >
>> >> > On Fri, Aug 18, 2017 at 3:30 AM, Mehmet  wrote:
>> >> >> Which ssds are used? Are they in production? If so how is your PG 
>> >> >> Count?
>> >> >>
>> >> >> Am 17. August 2017 20:04:25 MESZ schrieb M Ranga Swami Reddy
>> >> >> :
>> >> >>>
>> >> >>> Hello,
>> >> >>> I am using the Ceph cluster with HDDs and SSDs. Created separate pool 
>> >> >>> for
>> >> >>> each.
>> >> >>> Now, when I ran the "ceph osd bench", HDD's OSDs show around 500 MB/s
>> >> >>> and SSD's OSD show around 280MB/s.
>> >> >>>
>> >> >>> Ideally, what I expected was - SSD's OSDs should be at-least 40% high
>> >> >>> as compared with HDD's OSD bench.
>> >> >>>
>> >> >>> Did I miss anything here? Any hint is appreciated.
>> >> >>>
>> >> >>> Thanks
>> >> >>> Swami
>> >> >>> 
>> >> >>>
>> >> >>> ceph-users mailing list
>> >> >>> ceph-users@lists.ceph.com
>> >> >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>> >> >>
>> >> >>
>> >> >> ___
>> >> >> ceph-users mailing list
>> >> >> ceph-users@lists.ceph.com
>> >> >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>> >> >>
>> >> ___
>> >> ceph-users mailing list
>> >> ceph-users@lists.ceph.com
>> >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>> >>
>> >
>> >
>> > --
>> > Christian BalzerNetwork/Systems Engineer
>> > ch...@gol.com   Rakuten Communications
>>
>
>
> --
> Christian BalzerNetwork/Systems Engineer
> ch...@gol.com   Rakuten Communications
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph cluster with SSDs

2017-08-21 Thread Christian Balzer 
On Mon, 21 Aug 2017 17:13:10 +0530 M Ranga Swami Reddy wrote:

> Thank you.
> Here I have NVMes from Intel. but as the support of these NVMes not
> there from Intel, we decided not to use these NVMes as a journal.

You again fail to provide with specific model numbers...
No support from Intel suggests that these may be consumer models again.

Samsung also makes DC grade SSDs and NVMEs, as Adrian pointed out.

> Btw, if we split this SSD with multiple OSD (for ex: 1 SSD with 4 or 2
> OSDs), is  this help any performance numbers?
> 
Of course not, if anything it will make it worse due to the overhead
outside the SSD itself.

Christian

> On Sun, Aug 20, 2017 at 9:33 AM, Christian Balzer  wrote:
> >
> > Hello,
> >
> > On Sat, 19 Aug 2017 23:22:11 +0530 M Ranga Swami Reddy wrote:
> >  
> >> SSD make details : SSD 850 EVO 2.5" SATA III 4TB Memory & Storage -
> >> MZ-75E4T0B/AM | Samsung
> >>  
> > And there's your answer.
> >
> > A bit of googling in the archives here would have shown you that these are
> > TOTALLY unsuitable for use with Ceph.
> > Not only because of the horrid speed when used with/for Ceph journaling
> > (direct/sync I/O) but also their abysmal endurance of 0.04 DWPD over 5
> > years.
> > Or in other words 160GB/day, which after the Ceph journal double writes
> > and FS journals, other overhead and write amplification in general
> > probably means less that effective 40GB/day.
> >
> > In contrast the lowest endurance DC grade SSDs tend to be 0.3 DWPD and
> > more commonly 1 DWPD.
> > And I'm not buying anything below 3 DWPD for use with Ceph.
> >
> > Your only chance to improve the speed here is to take the journals off
> > them and put them onto fast and durable enough NVMes like the Intel DC P
> > 3700 or at worst 3600 types.
> >
> > That still leaves you with their crappy endurance, only twice as high than
> > before with the journals offloaded.
> >
> > Christian
> >  
> >> On Sat, Aug 19, 2017 at 10:44 PM, M Ranga Swami Reddy
> >>  wrote:  
> >> > Yes, Its in production and used the pg count as per the pg calcuator @ 
> >> > ceph.com.
> >> >
> >> > On Fri, Aug 18, 2017 at 3:30 AM, Mehmet  wrote:  
> >> >> Which ssds are used? Are they in production? If so how is your PG Count?
> >> >>
> >> >> Am 17. August 2017 20:04:25 MESZ schrieb M Ranga Swami Reddy
> >> >> :  
> >> >>>
> >> >>> Hello,
> >> >>> I am using the Ceph cluster with HDDs and SSDs. Created separate pool 
> >> >>> for
> >> >>> each.
> >> >>> Now, when I ran the "ceph osd bench", HDD's OSDs show around 500 MB/s
> >> >>> and SSD's OSD show around 280MB/s.
> >> >>>
> >> >>> Ideally, what I expected was - SSD's OSDs should be at-least 40% high
> >> >>> as compared with HDD's OSD bench.
> >> >>>
> >> >>> Did I miss anything here? Any hint is appreciated.
> >> >>>
> >> >>> Thanks
> >> >>> Swami
> >> >>> 
> >> >>>
> >> >>> ceph-users mailing list
> >> >>> ceph-users@lists.ceph.com
> >> >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com  
> >> >>
> >> >>
> >> >> ___
> >> >> ceph-users mailing list
> >> >> ceph-users@lists.ceph.com
> >> >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >> >>  
> >> ___
> >> ceph-users mailing list
> >> ceph-users@lists.ceph.com
> >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >>  
> >
> >
> > --
> > Christian BalzerNetwork/Systems Engineer
> > ch...@gol.com   Rakuten Communications  
> 


-- 
Christian BalzerNetwork/Systems Engineer
ch...@gol.com   Rakuten Communications
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph cluster with SSDs

2017-08-21 Thread M Ranga Swami Reddy 
Thank you.
Here I have NVMes from Intel. but as the support of these NVMes not
there from Intel, we decided not to use these NVMes as a journal.
Btw, if we split this SSD with multiple OSD (for ex: 1 SSD with 4 or 2
OSDs), is  this help any performance numbers?

On Sun, Aug 20, 2017 at 9:33 AM, Christian Balzer  wrote:
>
> Hello,
>
> On Sat, 19 Aug 2017 23:22:11 +0530 M Ranga Swami Reddy wrote:
>
>> SSD make details : SSD 850 EVO 2.5" SATA III 4TB Memory & Storage -
>> MZ-75E4T0B/AM | Samsung
>>
> And there's your answer.
>
> A bit of googling in the archives here would have shown you that these are
> TOTALLY unsuitable for use with Ceph.
> Not only because of the horrid speed when used with/for Ceph journaling
> (direct/sync I/O) but also their abysmal endurance of 0.04 DWPD over 5
> years.
> Or in other words 160GB/day, which after the Ceph journal double writes
> and FS journals, other overhead and write amplification in general
> probably means less that effective 40GB/day.
>
> In contrast the lowest endurance DC grade SSDs tend to be 0.3 DWPD and
> more commonly 1 DWPD.
> And I'm not buying anything below 3 DWPD for use with Ceph.
>
> Your only chance to improve the speed here is to take the journals off
> them and put them onto fast and durable enough NVMes like the Intel DC P
> 3700 or at worst 3600 types.
>
> That still leaves you with their crappy endurance, only twice as high than
> before with the journals offloaded.
>
> Christian
>
>> On Sat, Aug 19, 2017 at 10:44 PM, M Ranga Swami Reddy
>>  wrote:
>> > Yes, Its in production and used the pg count as per the pg calcuator @ 
>> > ceph.com.
>> >
>> > On Fri, Aug 18, 2017 at 3:30 AM, Mehmet  wrote:
>> >> Which ssds are used? Are they in production? If so how is your PG Count?
>> >>
>> >> Am 17. August 2017 20:04:25 MESZ schrieb M Ranga Swami Reddy
>> >> :
>> >>>
>> >>> Hello,
>> >>> I am using the Ceph cluster with HDDs and SSDs. Created separate pool for
>> >>> each.
>> >>> Now, when I ran the "ceph osd bench", HDD's OSDs show around 500 MB/s
>> >>> and SSD's OSD show around 280MB/s.
>> >>>
>> >>> Ideally, what I expected was - SSD's OSDs should be at-least 40% high
>> >>> as compared with HDD's OSD bench.
>> >>>
>> >>> Did I miss anything here? Any hint is appreciated.
>> >>>
>> >>> Thanks
>> >>> Swami
>> >>> 
>> >>>
>> >>> ceph-users mailing list
>> >>> ceph-users@lists.ceph.com
>> >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>> >>
>> >>
>> >> ___
>> >> ceph-users mailing list
>> >> ceph-users@lists.ceph.com
>> >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>> >>
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>
>
> --
> Christian BalzerNetwork/Systems Engineer
> ch...@gol.com   Rakuten Communications
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph cluster with SSDs

2017-08-20 Thread Christian Balzer 
On Mon, 21 Aug 2017 01:48:49 + Adrian Saul wrote:

> > SSD make details : SSD 850 EVO 2.5" SATA III 4TB Memory & Storage - MZ-
> > 75E4T0B/AM | Samsung  
> 
> The performance difference between these and the SM or PM863 range is night 
> and day.  I would not use these for anything you care about with performance, 
> particularly IOPS or latency.
> Their write latency is highly variable and even at best is still 5x higher 
> than what the SM863 range does.  When we compared them we could not get them 
> below 6ms and they frequently spiked to much higher values (25-30ms).  With 
> the SM863s they were a constant sub 1ms and didn't fluctuate.  I believe it 
> was the garbage collection on the Evos that causes the issue.  Here was the 
> difference in average latencies from a pool made of half Evo and half SM863:
> 
> Write latency - Evo 7.64ms - SM863 0.55ms
> Read Latency - Evo 2.56ms - SM863  0.16ms
> 
Yup, you get these unpredictable (and thus unsuitable) randomness and
generally higher latency with nearly all consumer SSDs.
And yes, typically GC related.

The reason it's so slow with sync writes is with near certainty that their
large DRAM cache is useless with these, as said cache isn't protected
against power failures and thus needs to be bypassed. 
Other consumer SSDs (IIRC Intel 510s amongst them) used to blatantly lie
about sync writes and thus appeared fast while putting your data at
significant risk.

Christian

> Add to that Christian's remarks on the write endurance and they are only good 
> for desktops that wont exercise them that much.   You are far better 
> investing in DC/Enterprise grade devices.
> 
> 
> 
> 
> >
> > On Sat, Aug 19, 2017 at 10:44 PM, M Ranga Swami Reddy
> >  wrote:  
> > > Yes, Its in production and used the pg count as per the pg calcuator @  
> > ceph.com.  
> > >
> > > On Fri, Aug 18, 2017 at 3:30 AM, Mehmet  wrote:  
> > >> Which ssds are used? Are they in production? If so how is your PG Count?
> > >>
> > >> Am 17. August 2017 20:04:25 MESZ schrieb M Ranga Swami Reddy
> > >> :  
> > >>>
> > >>> Hello,
> > >>> I am using the Ceph cluster with HDDs and SSDs. Created separate
> > >>> pool for each.
> > >>> Now, when I ran the "ceph osd bench", HDD's OSDs show around 500
> > >>> MB/s and SSD's OSD show around 280MB/s.
> > >>>
> > >>> Ideally, what I expected was - SSD's OSDs should be at-least 40%
> > >>> high as compared with HDD's OSD bench.
> > >>>
> > >>> Did I miss anything here? Any hint is appreciated.
> > >>>
> > >>> Thanks
> > >>> Swami
> > >>> 
> > >>>
> > >>> ceph-users mailing list
> > >>> ceph-users@lists.ceph.com
> > >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com  
> > >>
> > >>
> > >> ___
> > >> ceph-users mailing list
> > >> ceph-users@lists.ceph.com
> > >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> > >>  
> > ___
> > ceph-users mailing list
> > ceph-users@lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com  
> Confidentiality: This email and any attachments are confidential and may be 
> subject to copyright, legal or some other professional privilege. They are 
> intended solely for the attention and use of the named addressee(s). They may 
> only be copied, distributed or disclosed with the consent of the copyright 
> owner. If you have received this email by mistake or by breach of the 
> confidentiality clause, please notify the sender immediately by return email 
> and delete or destroy all copies of the email. Any confidentiality, privilege 
> or copyright is not waived or lost because this email has been sent to you by 
> mistake.
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 


-- 
Christian BalzerNetwork/Systems Engineer
ch...@gol.com   Rakuten Communications
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph cluster with SSDs

2017-08-20 Thread Adrian Saul 
> SSD make details : SSD 850 EVO 2.5" SATA III 4TB Memory & Storage - MZ-
> 75E4T0B/AM | Samsung

The performance difference between these and the SM or PM863 range is night and 
day.  I would not use these for anything you care about with performance, 
particularly IOPS or latency.
Their write latency is highly variable and even at best is still 5x higher than 
what the SM863 range does.  When we compared them we could not get them below 
6ms and they frequently spiked to much higher values (25-30ms).  With the 
SM863s they were a constant sub 1ms and didn't fluctuate.  I believe it was the 
garbage collection on the Evos that causes the issue.  Here was the difference 
in average latencies from a pool made of half Evo and half SM863:

Write latency - Evo 7.64ms - SM863 0.55ms
Read Latency - Evo 2.56ms - SM863  0.16ms

Add to that Christian's remarks on the write endurance and they are only good 
for desktops that wont exercise them that much.   You are far better investing 
in DC/Enterprise grade devices.




>
> On Sat, Aug 19, 2017 at 10:44 PM, M Ranga Swami Reddy
>  wrote:
> > Yes, Its in production and used the pg count as per the pg calcuator @
> ceph.com.
> >
> > On Fri, Aug 18, 2017 at 3:30 AM, Mehmet  wrote:
> >> Which ssds are used? Are they in production? If so how is your PG Count?
> >>
> >> Am 17. August 2017 20:04:25 MESZ schrieb M Ranga Swami Reddy
> >> :
> >>>
> >>> Hello,
> >>> I am using the Ceph cluster with HDDs and SSDs. Created separate
> >>> pool for each.
> >>> Now, when I ran the "ceph osd bench", HDD's OSDs show around 500
> >>> MB/s and SSD's OSD show around 280MB/s.
> >>>
> >>> Ideally, what I expected was - SSD's OSDs should be at-least 40%
> >>> high as compared with HDD's OSD bench.
> >>>
> >>> Did I miss anything here? Any hint is appreciated.
> >>>
> >>> Thanks
> >>> Swami
> >>> 
> >>>
> >>> ceph-users mailing list
> >>> ceph-users@lists.ceph.com
> >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >>
> >>
> >> ___
> >> ceph-users mailing list
> >> ceph-users@lists.ceph.com
> >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Confidentiality: This email and any attachments are confidential and may be 
subject to copyright, legal or some other professional privilege. They are 
intended solely for the attention and use of the named addressee(s). They may 
only be copied, distributed or disclosed with the consent of the copyright 
owner. If you have received this email by mistake or by breach of the 
confidentiality clause, please notify the sender immediately by return email 
and delete or destroy all copies of the email. Any confidentiality, privilege 
or copyright is not waived or lost because this email has been sent to you by 
mistake.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph cluster with SSDs

2017-08-20 Thread Christian Balzer 
On Sun, 20 Aug 2017 08:38:54 +0200 Sinan Polat wrote:

> What has DWPD to do with performance / IOPS? The SSD will just fail earlier, 
> but it should not have any affect on the performance, right?
> 
Nothing, I listed BOTH reasons why these are unsuitable.

You just don't buy something huge like 4TB SSDs and expect to just write
40GB/day to them in.

> Correct me if I am wrong, just want to learn.
>
Learning is easy if you're will to make a little effort.
Just like the OP you failed to search for "Samsung Evo Ceph" and find all
the bad news like in this result:

https://forum.proxmox.com/threads/slow-ceph-journal-on-samsung-850-pro.27733/
 
Christian
> 
> > Op 20 aug. 2017 om 06:03 heeft Christian Balzer  het 
> > volgende geschreven:
> > 
> > DWPD  
> 
> 


-- 
Christian BalzerNetwork/Systems Engineer
ch...@gol.com   Rakuten Communications
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph cluster with SSDs

2017-08-20 Thread Sinan Polat 
What has DWPD to do with performance / IOPS? The SSD will just fail earlier, 
but it should not have any affect on the performance, right?

Correct me if I am wrong, just want to learn.


> Op 20 aug. 2017 om 06:03 heeft Christian Balzer  het volgende 
> geschreven:
> 
> DWPD

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph cluster with SSDs

2017-08-19 Thread Christian Balzer 

Hello,

On Sat, 19 Aug 2017 23:22:11 +0530 M Ranga Swami Reddy wrote:

> SSD make details : SSD 850 EVO 2.5" SATA III 4TB Memory & Storage -
> MZ-75E4T0B/AM | Samsung
>
And there's your answer.

A bit of googling in the archives here would have shown you that these are
TOTALLY unsuitable for use with Ceph.
Not only because of the horrid speed when used with/for Ceph journaling
(direct/sync I/O) but also their abysmal endurance of 0.04 DWPD over 5
years.
Or in other words 160GB/day, which after the Ceph journal double writes
and FS journals, other overhead and write amplification in general
probably means less that effective 40GB/day.

In contrast the lowest endurance DC grade SSDs tend to be 0.3 DWPD and
more commonly 1 DWPD.
And I'm not buying anything below 3 DWPD for use with Ceph.

Your only chance to improve the speed here is to take the journals off
them and put them onto fast and durable enough NVMes like the Intel DC P
3700 or at worst 3600 types.

That still leaves you with their crappy endurance, only twice as high than
before with the journals offloaded.
 
Christian

> On Sat, Aug 19, 2017 at 10:44 PM, M Ranga Swami Reddy
>  wrote:
> > Yes, Its in production and used the pg count as per the pg calcuator @ 
> > ceph.com.
> >
> > On Fri, Aug 18, 2017 at 3:30 AM, Mehmet  wrote:  
> >> Which ssds are used? Are they in production? If so how is your PG Count?
> >>
> >> Am 17. August 2017 20:04:25 MESZ schrieb M Ranga Swami Reddy
> >> :  
> >>>
> >>> Hello,
> >>> I am using the Ceph cluster with HDDs and SSDs. Created separate pool for
> >>> each.
> >>> Now, when I ran the "ceph osd bench", HDD's OSDs show around 500 MB/s
> >>> and SSD's OSD show around 280MB/s.
> >>>
> >>> Ideally, what I expected was - SSD's OSDs should be at-least 40% high
> >>> as compared with HDD's OSD bench.
> >>>
> >>> Did I miss anything here? Any hint is appreciated.
> >>>
> >>> Thanks
> >>> Swami
> >>> 
> >>>
> >>> ceph-users mailing list
> >>> ceph-users@lists.ceph.com
> >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com  
> >>
> >>
> >> ___
> >> ceph-users mailing list
> >> ceph-users@lists.ceph.com
> >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >>  
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 


-- 
Christian BalzerNetwork/Systems Engineer
ch...@gol.com   Rakuten Communications
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph cluster with SSDs

2017-08-19 Thread M Ranga Swami Reddy 
SSD make details : SSD 850 EVO 2.5" SATA III 4TB Memory & Storage -
MZ-75E4T0B/AM | Samsung

On Sat, Aug 19, 2017 at 10:44 PM, M Ranga Swami Reddy
 wrote:
> Yes, Its in production and used the pg count as per the pg calcuator @ 
> ceph.com.
>
> On Fri, Aug 18, 2017 at 3:30 AM, Mehmet  wrote:
>> Which ssds are used? Are they in production? If so how is your PG Count?
>>
>> Am 17. August 2017 20:04:25 MESZ schrieb M Ranga Swami Reddy
>> :
>>>
>>> Hello,
>>> I am using the Ceph cluster with HDDs and SSDs. Created separate pool for
>>> each.
>>> Now, when I ran the "ceph osd bench", HDD's OSDs show around 500 MB/s
>>> and SSD's OSD show around 280MB/s.
>>>
>>> Ideally, what I expected was - SSD's OSDs should be at-least 40% high
>>> as compared with HDD's OSD bench.
>>>
>>> Did I miss anything here? Any hint is appreciated.
>>>
>>> Thanks
>>> Swami
>>> 
>>>
>>> ceph-users mailing list
>>> ceph-users@lists.ceph.com
>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>>
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph cluster with SSDs

2017-08-19 Thread M Ranga Swami Reddy 
I did not only "osd bench". Performed rbd image mapped and DD test on
it... here also got very less number with image on SSD pool as
compared with image on HDD pool.
As per SSD datasheet - they claim 500 MB/s, but I am getting somewhat
near 50 MB/s with dd cmd.


On Fri, Aug 18, 2017 at 6:32 AM, Christian Balzer  wrote:
>
> Hello,
>
> On Fri, 18 Aug 2017 00:00:09 +0200 Mehmet wrote:
>
>> Which ssds are used? Are they in production? If so how is your PG Count?
>>
> What he wrote.
> W/o knowing which apples you're comparing to what oranges, this is
> pointless.
>
> Also testing osd bench is the LEAST relevant test you can do, as it only
> deals with local bandwidth, while what people nearly always want/need in
> the end is IOPS and low latency.
> Which you test best from a real client perspective.
>
> Christian
>
>> Am 17. August 2017 20:04:25 MESZ schrieb M Ranga Swami Reddy 
>> :
>> >Hello,
>> >I am using the Ceph cluster with HDDs and SSDs. Created separate pool
>> >for each.
>> >Now, when I ran the "ceph osd bench", HDD's OSDs show around 500 MB/s
>> >and SSD's OSD show around 280MB/s.
>> >
>> >Ideally, what I expected was - SSD's OSDs should be at-least 40% high
>> >as compared with HDD's OSD bench.
>> >
>> >Did I miss anything here? Any hint is appreciated.
>> >
>> >Thanks
>> >Swami
>> >___
>> >ceph-users mailing list
>> >ceph-users@lists.ceph.com
>> >http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
> --
> Christian BalzerNetwork/Systems Engineer
> ch...@gol.com   Rakuten Communications
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph cluster with SSDs

2017-08-19 Thread M Ranga Swami Reddy 
Yes, Its in production and used the pg count as per the pg calcuator @ ceph.com.

On Fri, Aug 18, 2017 at 3:30 AM, Mehmet  wrote:
> Which ssds are used? Are they in production? If so how is your PG Count?
>
> Am 17. August 2017 20:04:25 MESZ schrieb M Ranga Swami Reddy
> :
>>
>> Hello,
>> I am using the Ceph cluster with HDDs and SSDs. Created separate pool for
>> each.
>> Now, when I ran the "ceph osd bench", HDD's OSDs show around 500 MB/s
>> and SSD's OSD show around 280MB/s.
>>
>> Ideally, what I expected was - SSD's OSDs should be at-least 40% high
>> as compared with HDD's OSD bench.
>>
>> Did I miss anything here? Any hint is appreciated.
>>
>> Thanks
>> Swami
>> 
>>
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph cluster with SSDs

2017-08-17 Thread Christian Balzer 

Hello,

On Fri, 18 Aug 2017 00:00:09 +0200 Mehmet wrote:

> Which ssds are used? Are they in production? If so how is your PG Count?
>
What he wrote.
W/o knowing which apples you're comparing to what oranges, this is
pointless.

Also testing osd bench is the LEAST relevant test you can do, as it only
deals with local bandwidth, while what people nearly always want/need in
the end is IOPS and low latency.
Which you test best from a real client perspective.

Christian
 
> Am 17. August 2017 20:04:25 MESZ schrieb M Ranga Swami Reddy 
> :
> >Hello,
> >I am using the Ceph cluster with HDDs and SSDs. Created separate pool
> >for each.
> >Now, when I ran the "ceph osd bench", HDD's OSDs show around 500 MB/s
> >and SSD's OSD show around 280MB/s.
> >
> >Ideally, what I expected was - SSD's OSDs should be at-least 40% high
> >as compared with HDD's OSD bench.
> >
> >Did I miss anything here? Any hint is appreciated.
> >
> >Thanks
> >Swami
> >___
> >ceph-users mailing list
> >ceph-users@lists.ceph.com
> >http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com  


-- 
Christian BalzerNetwork/Systems Engineer
ch...@gol.com   Rakuten Communications
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph cluster with SSDs

2017-08-17 Thread Mehmet 
Which ssds are used? Are they in production? If so how is your PG Count?

Am 17. August 2017 20:04:25 MESZ schrieb M Ranga Swami Reddy 
:
>Hello,
>I am using the Ceph cluster with HDDs and SSDs. Created separate pool
>for each.
>Now, when I ran the "ceph osd bench", HDD's OSDs show around 500 MB/s
>and SSD's OSD show around 280MB/s.
>
>Ideally, what I expected was - SSD's OSDs should be at-least 40% high
>as compared with HDD's OSD bench.
>
>Did I miss anything here? Any hint is appreciated.
>
>Thanks
>Swami
>___
>ceph-users mailing list
>ceph-users@lists.ceph.com
>http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Ceph cluster with SSDs

2017-08-17 Thread M Ranga Swami Reddy 
Hello,
I am using the Ceph cluster with HDDs and SSDs. Created separate pool for each.
Now, when I ran the "ceph osd bench", HDD's OSDs show around 500 MB/s
and SSD's OSD show around 280MB/s.

Ideally, what I expected was - SSD's OSDs should be at-least 40% high
as compared with HDD's OSD bench.

Did I miss anything here? Any hint is appreciated.

Thanks
Swami
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com