HI,

Ceph.conf:
       osd journal size = 15360
       rbd cache = true
        rbd cache size = 2147483648
        rbd cache max dirty = 1073741824
        rbd cache max dirty age = 100
        osd recovery max active = 1
         osd max backfills = 1
         osd mkfs options xfs = "-f -i size=2048"
         osd mount options xfs = 
"rw,noatime,nobarrier,logbsize=256k,logbufs=8,inode64,allocsize=4M"
         osd op threads = 8

so it should be 8 threads?

All 3 machines have more or less the same disk load at the same time.
also the disks:
sdb              35.56        87.10      6849.09     617310   48540806
sdc              26.75        72.62      5148.58     514701   36488992
sdd              35.15        53.48      6802.57     378993   48211141
sde              31.04        79.04      6208.48     560141   44000710
sdf              32.79        38.35      6238.28     271805   44211891
sdg              31.67        77.84      5987.45     551680   42434167
sdh              32.95        51.29      6315.76     363533   44761001
sdi              31.67        56.93      5956.29     403478   42213336
sdj              35.83        77.82      6929.31     551501   49109354
sdk              36.86        73.84      7291.00     523345   51672704
sdl              36.02       112.90      7040.47     800177   49897132
sdm              33.25        38.02      6455.05     269446   45748178
sdn              33.52        39.10      6645.19     277101   47095696
sdo              33.26        46.22      6388.20     327541   45274394
sdp              33.38        74.12      6480.62     525325   45929369


the question is: is this a poor performance to get max 500mb/write with 45 
disks and replica 2 or should I expect this?


-----Ursprüngliche Nachricht-----
Von: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] Im Auftrag von Wido 
den Hollander
Gesendet: Donnerstag, 03. Juli 2014 15:22
An: ceph-users@lists.ceph.com
Betreff: Re: [ceph-users] write performance per disk

On 07/03/2014 03:11 PM, VELARTIS Philipp Dürhammer wrote:
> Hi,
>
> I have a ceph cluster setup (with 45 sata disk journal on disks) and 
> get only 450mb/sec writes seq (maximum playing around with threads in 
> rados
> bench) with replica of 2
>

How many threads?

> Which is about ~20Mb writes per disk (what y see in atop also) 
> theoretically with replica2 and having journals on disk should be 45 X 
> 100mb (sata) / 2 (replica) / 2 (journal writes) which makes it 1125 
> satas in reality have 120mb/sec so the theoretical output should be more.
>
> I would expect to have between 40-50mb/sec for each sata disk
>
> Can somebody confirm that he can reach this speed with a setup with 
> journals on the satas (with journals on ssd speed should be 100mb per disk)?
> or does ceph only give about ¼ of the speed for a disk? (and not the ½ 
> as expected because of journals)
>

Did you verify how much each machine is doing? It could be that the data is not 
distributed evenly and that on a certain machine the drives are doing 50MB/sec.

> My setup is 3 servers with: 2 x 2.6ghz xeons, 128gb ram 15 satas for 
> ceph (and ssds for system) 1 x 10gig for external traffic, 1 x 10gig 
> for osd traffic with reads I can saturate the network but writes is 
> far away. And I would expect at least to saturate the 10gig with 
> sequential writes also
>

Should be possible, but with 3 servers the data distribution might not be 
optimal causing a lower write performance.

I've seen 10Gbit write performance on multiple clusters without any problems.

> Thank you
>
>
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>


--
Wido den Hollander
Ceph consultant and trainer
42on B.V.

Phone: +31 (0)20 700 9902
Skype: contact42on
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Reply via email to