Re: [ceph-users] CPU use for OSD daemon

[Christian Balzer]

Hello,

re-adding list.

On Tue, 28 Jun 2016 20:52:51 +0300 George Shuklin wrote:

> On 06/28/2016 06:46 PM, Christian Balzer wrote:
> > Hello,
> >
> > On Tue, 28 Jun 2016 18:23:02 +0300 George Shuklin wrote:
> >
> >> Hello.
> >>
> >> I'm testing different configuration for Ceph.
> > What version...
> jewel.
>
That should be pretty fast by itself, after that the optimizations
Alexandre mentioned. 
 
> >
> >> I found that osd are
> >> REALLY hungry for cpu.
> >>
> > They can be, but unlikely in your case.
> >
> >> I've created a tiny pool with size 1 with single OSD made of fast
> >> intel SSD (2500-series), on old dell server (R210),  Xeon E3-1230 V2
> >> @ 3.30GHz.
> >>
> > At a replication size of 1, a totally unrealistic test scenario.
> >
> > Ignoring that, an Intel SSD PRO 2500 is a consumer SSD and as such with
> > near certainty ill suited for usage with Ceph, especially when it
> > comes to journals.
> > Check/google the countless threads about what constitutes SSDs
> > suitable for Ceph usage.
> 
> I understand that, but the point is that it was stuck at cpu, not IO on 
> SSD (disk utilization was < 5% according to atop).
> 
That makes little to no sense.

> >> I see some horribly-low performance and clear
> >> bottleneck at ceph-osd process: it consumes about 110% of CPU and
> >> giving
> > 110% actual CPU usage?
> > I'd wager a significant amount of that is IOWAIT...
> No, it was clear computation, not IO.
> 
> It was somehow badly created OSD. I've recreated it
Any details on that?
So people in the future searching for a problem like this can avoid it.

>, and now I'm hitting 
> limits of SSD performance with ~900 IOPS (with 99% utilization of SSD 
> and 23% utilization of CPU by ceph-osd).
> 
That ratio and performance sounds more like it, given your SSD model.

Christian
-- 
Christian BalzerNetwork/Systems Engineer
ch...@gol.com   Global OnLine Japan/Rakuten Communications
http://www.gol.com/
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] CPU use for OSD daemon

[Alexandre DERUMIER]

>>And when I benchmark it I see some horribly-low performance and clear
>>bottleneck at ceph-osd process: it consumes about 110% of CPU and giving
>>me following results: 127 iops in fio benchmark (4k randwrite) for rbd
>>device, rados benchmark gives me ~21 IOPS and 76Mb/s (write).

on a 2x xeon 3,1ghz  10 cores (20 cores total), I can reach around 40 iops 
4k read (80% total cpu), or 7 iops 4k write (1x repli) 100% total cpu

this is with jemalloc, debug and cephx are disabled.


- Mail original -
De: "George Shuklin" 
À: "ceph-users" 
Envoyé: Mardi 28 Juin 2016 17:23:02
Objet: [ceph-users] CPU use for OSD daemon

Hello. 

I'm testing different configuration for Ceph. I found that osd are 
REALLY hungry for cpu. 

I've created a tiny pool with size 1 with single OSD made of fast intel 
SSD (2500-series), on old dell server (R210), Xeon E3-1230 V2 @ 3.30GHz. 

And when I benchmark it I see some horribly-low performance and clear 
bottleneck at ceph-osd process: it consumes about 110% of CPU and giving 
me following results: 127 iops in fio benchmark (4k randwrite) for rbd 
device, rados benchmark gives me ~21 IOPS and 76Mb/s (write). 

It this a normal CPU utilization for osd daemon for such tiny performance? 

Relevant part of the crush map: 

rule rule_fast { 
ruleset 1 
type replicated 
min_size 1 
max_size 10 
step take fast 
step chooseleaf firstn 0 type osd 
step emit 
} 
root fast2500 { 
id -17 
alg straw 
hash 0 # rjenkins1 
item pp7 weight 1.0 
} 

host pp7 { 
id -11 
alg straw 
hash 0 # rjenkins1 
item osd.5 weight 1.0 
} 


host pp7 { 
id -11 
alg straw 
hash 0 # rjenkins1 
item osd.5 weight 1.0 
} 

device 5 osd.5 
___ 
ceph-users mailing list 
ceph-users@lists.ceph.com 
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com 

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] CPU use for OSD daemon

[Christian Balzer]

Hello,

On Tue, 28 Jun 2016 18:23:02 +0300 George Shuklin wrote:

> Hello.
> 
> I'm testing different configuration for Ceph. 

What version...

> I found that osd are 
> REALLY hungry for cpu.
> 
They can be, but unlikely in your case.

> I've created a tiny pool with size 1 with single OSD made of fast intel 
> SSD (2500-series), on old dell server (R210),  Xeon E3-1230 V2 @ 3.30GHz.
> 
At a replication size of 1, a totally unrealistic test scenario.

Ignoring that, an Intel SSD PRO 2500 is a consumer SSD and as such with
near certainty ill suited for usage with Ceph, especially when it comes to
journals. 
Check/google the countless threads about what constitutes SSDs suitable for
Ceph usage. 

> And when I benchmark it 
How? 
Fio, we can gather, but whether against a RBD image, with user or kernel
client, with the fio RBD engine...

>I see some horribly-low performance and clear 
> bottleneck at ceph-osd process: it consumes about 110% of CPU and giving 
110% actual CPU usage?
I'd wager a significant amount of that is IOWAIT...

> me following results: 127 iops in fio benchmark (4k randwrite) for rbd 
> device, rados benchmark gives me ~21 IOPS and 76Mb/s (write).
> 
Pretty clear indication that the SSD isn't handling sync writes well,
lacking further info.
 
> It this a normal CPU utilization for osd daemon for such tiny
> performance?
> 
> Relevant part of the crush map:
> 
Irrelevant in this context really.


Christian
-- 
Christian BalzerNetwork/Systems Engineer
ch...@gol.com   Global OnLine Japan/Rakuten Communications
http://www.gol.com/
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com