Hi to all,

I need your help to understand how to tune my ceph configuration to achieve 
some performance result.

my installation is built as follow:

5 server with 16gb ram and 8 core
5 client (same machine)

Each computer is connected with a 1gb/s ethernet to the same switch.
The device storage export 20 luns with 7.2TB of capacity.

Every server is connected to a device storace with a FC connection at 4Gb/s  
and export 2 osds.

I always has a healthy ceph cluster up and running before start any test.

I have tried 3 different configuration of storage to achieve the best result.

The test for the performance is always the same: 5 dd from /dev/zero to a file 
the dimension of the dd is 25GB, each dd is run from a different client 
concurrently and the result file is stored intoceph posix mounted folder.

1) first configuration 
every luns is been partitioned with data and journal, every 
/var/lib/caph/osd/[osd folder]/journal  was linked to the partition journal (on 
the same lun).

In this configuration I saw that the san storage was stormed by a lot of iops 
and the performance was very slow, I investigated and I discovered that a 
possible reason could be the journal of every osd,
http://pastebin.com/GdkngHTd

so i decide to change the configuration as follow.

2) second 

Every server use 4 luns one for data and one for journal, changing the link of 
journal dir into the var/lib/caph/osd/[osd fold]/journal. Obviously I have 
flushed and recreated journal before start the cluster again.

again this configuration was not optimal and the performance was a little 
better from the previous experiment but again very far from the nominal 
possibility of the storage... i haven't ssds where to put the journal... so I 
decided to change the configuration again.
http://pastebin.com/VhmAt7Sf

3) third
Every server use a ram disk for journal.

I thought that since it is a test configuration I could use a ram disk that is 
the fastest device that I have and put there the journal... so i did. I launch 
again the same test... but the performance is still the same...

http://pastebin.com/tdAdGGnM


What am I doing wrong?

Could anyone help me to understand what I can do to achieve the best 
performance from this device?

ps. this device with the gpfs filesystem reached 500/600 MB/s in write and 
600/700MB/s in read with ceph only 250/300 in almost all configurations...

pps. i know that with 5 servers and 5 client I will never reach the same 
performance but I will expect to saturate the ethernet bandwidth.

thanks a lot 
Matteo


ceph.conf: http://pastebin.com/ZQsA1i7A
ceph -w during last configuration test: http://pastebin.com/zzJ6atj1
ceph-osd tree: http://pastebin.com/UxzqmSN8


speed during test inboud from server: http://pastebin.com/d8FT8UZE
speed during test outbound from client: http://pastebin.com/2W1DWaPP







---
Matteo Favaro
mail: matteo.favaro (at) cnaf.infn.it
alias: mfavaro (at) cnaf.infn.it

Attachment: smime.p7s
Description: S/MIME cryptographic signature

_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Reply via email to