Re: [ceph-users] performance in a small cluster

2019-06-07 Thread Paul Emmerich
On Tue, Jun 4, 2019 at 1:39 PM wrote: > >> Basically they max out at around 1000 IOPS and report 100% > >> utilization and feel slow. > >> > >> Haven't seen the 5200 yet. > > Micron 5100s performs wonderfully! > > You have to just turn its write cache off: > > hdparm -W 0 /dev/sdX > > 1000 IOPS m

Re: [ceph-users] performance in a small cluster

2019-06-04 Thread vitalif
Basically they max out at around 1000 IOPS and report 100% utilization and feel slow. Haven't seen the 5200 yet. Micron 5100s performs wonderfully! You have to just turn its write cache off: hdparm -W 0 /dev/sdX 1000 IOPS means you haven't done it. Although even with write cache enabled I o

Re: [ceph-users] performance in a small cluster

2019-05-31 Thread Reed Dier
Is there any other evidence of this? I have 20 5100 MAX (MTFDDAK1T9TCC) and have not experienced any real issues with them. I would pick my Samsung SM863a's or any of my Intel's over the Micron's, but I haven't seen the Micron's cause any issues for me. For what its worth, they are all FW D0MU02

Re: [ceph-users] performance in a small cluster

2019-05-29 Thread Paul Emmerich
On Wed, May 29, 2019 at 11:37 AM Robert Sander wrote: > Hi, > > Am 29.05.19 um 11:19 schrieb Martin Verges: > > > > We have identified the performance settings in the BIOS as a major > > factor > > > > could you share your insights what options you changed to increase > > performance and

Re: [ceph-users] performance in a small cluster

2019-05-29 Thread Paul Emmerich
On Wed, May 29, 2019 at 9:36 AM Robert Sander wrote: > Am 24.05.19 um 14:43 schrieb Paul Emmerich: > > * SSD model? Lots of cheap SSDs simply can't handle more than that > > The customer currently has 12 Micron 5100 1,92TB (Micron_5100_MTFDDAK1) > SSDs and will get a batch of Micron 5200 in the n

Re: [ceph-users] performance in a small cluster

2019-05-29 Thread Robert Sander
Hi, Am 29.05.19 um 11:19 schrieb Martin Verges: > > We have identified the performance settings in the BIOS as a major > factor > > could you share your insights what options you changed to increase > performance and could you provide numbers to it? Most default perfomance settings nowa

Re: [ceph-users] performance in a small cluster

2019-05-29 Thread Andrei Mikhailovsky
It would be interesting to learn the improvements types and the BIOS changes that helped you. Thanks > From: "Martin Verges" > To: "Robert Sander" > Cc: "ceph-users" > Sent: Wednesday, 29 May, 2019 10:19:09 > Subject: Re: [ceph-users] performan

Re: [ceph-users] performance in a small cluster

2019-05-29 Thread Martin Verges
Hello Robert, We have identified the performance settings in the BIOS as a major factor > could you share your insights what options you changed to increase performance and could you provide numbers to it? Many thanks in advance -- Martin Verges Managing director Mobile: +49 174 9335695 E-Mail

Re: [ceph-users] performance in a small cluster

2019-05-29 Thread Robert Sander
Am 24.05.19 um 14:43 schrieb Paul Emmerich: > * SSD model? Lots of cheap SSDs simply can't handle more than that The customer currently has 12 Micron 5100 1,92TB (Micron_5100_MTFDDAK1) SSDs and will get a batch of Micron 5200 in the next days We have identified the performance settings in the BIO

Re: [ceph-users] performance in a small cluster

2019-05-27 Thread Stefan Kooman
Quoting Robert Sander (r.san...@heinlein-support.de): > Hi, > > we have a small cluster at a customer's site with three nodes and 4 SSD-OSDs > each. > Connected with 10G the system is supposed to perform well. > > rados bench shows ~450MB/s write and ~950MB/s read speeds with 4MB objects > but on

Re: [ceph-users] performance in a small cluster

2019-05-25 Thread Marc Roos
+--+--+ -Original Message- From: Robert Sander [mailto:r.san...@heinlein-support.de] Sent: vrijdag 24 mei 2019 15:26 To: ceph-users Subject: Re: [ceph-users] performance in a small cluster Am 24.05.19 um 14:43 schrieb Paul Emmerich: > 20 MB/s at 4K blocks is ~5000 i

Re: [ceph-users] performance in a small cluster

2019-05-25 Thread Marc Schöchlin
Hello Robert, probably the following tool provides deeper insights whats happening on your osds: https://github.com/scoopex/ceph/blob/master/src/tools/histogram_dump.py https://github.com/ceph/ceph/pull/28244 https://user-images.githubusercontent.com/288876/58368661-410afa00-7ef0-11e9-9aca-b09d9

Re: [ceph-users] performance in a small cluster

2019-05-24 Thread Maged Mokhtar
Hi Robert 1) Can you specify how many threads were used in the 4k write rados test ? i suspect that only 16 threads were used, this is because it is the default + also the average latency was 2.9 ms giving average of 344 iops per thread, your average iops were 5512 divide this by 344 we get 16

Re: [ceph-users] performance in a small cluster

2019-05-24 Thread Paul Emmerich
On Sat, May 25, 2019 at 12:30 AM Mark Lehrer wrote: > > but only 20MB/s write and 95MB/s read with 4KB objects. > > There is copy-on-write overhead for each block, so 4K performance is > going to be limited no matter what. > no snapshots are involved and he's using rados bench which operates on

Re: [ceph-users] performance in a small cluster

2019-05-24 Thread Mark Lehrer
> but only 20MB/s write and 95MB/s read with 4KB objects. There is copy-on-write overhead for each block, so 4K performance is going to be limited no matter what. However, if your system is like mine the main problem you will run into is that Ceph was designed for spinning disks. Therefore, its

Re: [ceph-users] performance in a small cluster

2019-05-24 Thread Paul Emmerich
On Fri, May 24, 2019 at 3:27 PM Robert Sander wrote: > Am 24.05.19 um 14:43 schrieb Paul Emmerich: > > 20 MB/s at 4K blocks is ~5000 iops, that's 1250 IOPS per SSD (assuming > > replica 3). > > > > What we usually check in scenarios like these: > > > > * SSD model? Lots of cheap SSDs simply can't

Re: [ceph-users] performance in a small cluster

2019-05-24 Thread Robert LeBlanc
On Fri, May 24, 2019 at 6:26 AM Robert Sander wrote: > Am 24.05.19 um 14:43 schrieb Paul Emmerich: > > 20 MB/s at 4K blocks is ~5000 iops, that's 1250 IOPS per SSD (assuming > > replica 3). > > > > What we usually check in scenarios like these: > > > > * SSD model? Lots of cheap SSDs simply can't

Re: [ceph-users] performance in a small cluster

2019-05-24 Thread Robert Sander
Am 24.05.19 um 14:43 schrieb Paul Emmerich: > 20 MB/s at 4K blocks is ~5000 iops, that's 1250 IOPS per SSD (assuming > replica 3). > > What we usually check in scenarios like these: > > * SSD model? Lots of cheap SSDs simply can't handle more than that The system has been newly created and is no

Re: [ceph-users] performance in a small cluster

2019-05-24 Thread Paul Emmerich
20 MB/s at 4K blocks is ~5000 iops, that's 1250 IOPS per SSD (assuming replica 3). What we usually check in scenarios like these: * SSD model? Lots of cheap SSDs simply can't handle more than that * Get some proper statistics such as OSD latencies, disk IO utilization, etc. A benchmark without de

[ceph-users] performance in a small cluster

2019-05-24 Thread Robert Sander
Hi, we have a small cluster at a customer's site with three nodes and 4 SSD-OSDs each. Connected with 10G the system is supposed to perform well. rados bench shows ~450MB/s write and ~950MB/s read speeds with 4MB objects but only 20MB/s write and 95MB/s read with 4KB objects. This is a litt