Re: [ceph-users] How to improve performance of ceph objcect storage cluster
I've never worked enough with rbd to be sure. I know for files, when I turned on striping, I got far better performance. It seems like for RBD, the default is: Just to see if it helps with rbd, I would try stripe_count=4, stripe_unit=1mb... or something like that. If you tinker with these params, and they improve performance, let me know. -Original Message- From: Christian Kauhaus [mailto:k...@gocept.com] Sent: Friday, June 27, 2014 3:35 AM To: Aronesty, Erik; Udo Lembke; ceph-users@lists.ceph.com Subject: Re: [ceph-users] How to improve performance of ceph objcect storage cluster Am 26.06.2014 20:05, schrieb Aronesty, Erik: Well, it's the same for rbd, what's your stripe count set to? For a small system, it should be at least the # of nodes in your system.As systems get larger, there's limited returns... I would imagine there would be some OSD caching advantage to keeping the number limited (IE: more requests of the same device = more likely the device has the next stripe unit prefetched). I'm trying to make sure I understand this: usually you can't set the stripe count directly, but you can set the default stripe size of RBD volumes. So in consequence, does this mean to go with a larger RBD object size than the default (4MiB)? Regards Christian -- Dipl.-Inf. Christian Kauhaus · k...@gocept.com · systems administration gocept gmbh co. kg · Forsterstraße 29 · 06112 Halle (Saale) · Germany http://gocept.com · tel +49 345 219401-11 Python, Pyramid, Plone, Zope · consulting, development, hosting, operations ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] How to improve performance of ceph objcect storage cluster
Am 26.06.2014 20:05, schrieb Aronesty, Erik: Well, it's the same for rbd, what's your stripe count set to? For a small system, it should be at least the # of nodes in your system.As systems get larger, there's limited returns... I would imagine there would be some OSD caching advantage to keeping the number limited (IE: more requests of the same device = more likely the device has the next stripe unit prefetched). I'm trying to make sure I understand this: usually you can't set the stripe count directly, but you can set the default stripe size of RBD volumes. So in consequence, does this mean to go with a larger RBD object size than the default (4MiB)? Regards Christian -- Dipl.-Inf. Christian Kauhaus · k...@gocept.com · systems administration gocept gmbh co. kg · Forsterstraße 29 · 06112 Halle (Saale) · Germany http://gocept.com · tel +49 345 219401-11 Python, Pyramid, Plone, Zope · consulting, development, hosting, operations ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] How to improve performance of ceph objcect storage cluster
Hi, Am 25.06.2014 16:48, schrieb Aronesty, Erik: I'm assuming you're testing the speed of cephfs (the file system) and not ceph object storage. for my part I mean object storage (VM disk via rbd). Udo ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] How to improve performance of ceph objcect storage cluster
Well, it's the same for rbd, what's your stripe count set to? For a small system, it should be at least the # of nodes in your system.As systems get larger, there's limited returns... I would imagine there would be some OSD caching advantage to keeping the number limited (IE: more requests of the same device = more likely the device has the next stripe unit prefetched). -Original Message- From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Udo Lembke Sent: Thursday, June 26, 2014 2:03 AM To: ceph-users@lists.ceph.com Subject: Re: [ceph-users] How to improve performance of ceph objcect storage cluster Hi, Am 25.06.2014 16:48, schrieb Aronesty, Erik: I'm assuming you're testing the speed of cephfs (the file system) and not ceph object storage. for my part I mean object storage (VM disk via rbd). Udo ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] How to improve performance of ceph objcect storage cluster
Cern noted that they need to reformat to put the Journal in a partition
rather than on the OSD's filesystem like you did. See
http://www.slideshare.net/Inktank_Ceph/scaling-ceph-at-cern, slide 24.
When I saw that ceph disk prepare created a journal partition, I thought it
was stupid to force a seek like that. (This was before I saw Cern's
slides). I really should've known better, there's a reason it's the
default behavior. I didn't even benchmark the two. *hangs head in shame*
I really can't tell you why it's bad idea, but I can say that my recoveries
are extremely painful. I'm using RadosGW, and I only care about seconds of
latency. During large recoveries (like adding new nodes), people complain
about how slow the cluster is.
I'm in the middle of rolling out SSD journals to all machines.
On Tue, Jun 24, 2014 at 11:52 PM, wsnote wsn...@163.com wrote:
OS: CentOS 6.5
Version: Ceph 0.79
Hi, everybody!
I have installed a ceph cluster with 10 servers.
I test the throughput of ceph cluster in the same datacenter.
Upload files of 1GB from one server or several servers to one server or
several servers, the total is about 30MB/s.
That is to say, there is no difference between one server or one cluster
about throughput when uploading files.
How to optimize the performance of ceph object storage?
Thanks!
Info about ceph cluster:
4 MONs in the first 4 nodes in the cluster.
11 OSDs in each server, 109 OSDs in total (one disk was bad).
4TB each disk, 391TB in total (109*4-391=45TB.Where did 45TB space?)
1 RGW in each server, 10 RGWs in total.That is to say, I can use S3 API in
each Server.
ceph.conf:
[global]
auth supported = none
;auth_service_required = cephx
;auth_client_required = cephx
;auth_cluster_required = cephx
filestore_xattr_use_omap = true
max open files = 131072
log file = /var/log/ceph/$name.log
pid file = /var/run/ceph/$name.pid
keyring = /etc/ceph/keyring.admin
mon_clock_drift_allowed = 2 ;clock skew detected
[mon]
mon data = /data/mon$id
keyring = /etc/ceph/keyring.$name
[osd]
osd data = /data/osd$id
osd journal = /data/osd$id/journal
osd journal size = 1024;
keyring = /etc/ceph/keyring.$name
osd mkfs type = xfs
osd mount options xfs = rw,noatime
osd mkfs options xfs = -f
[client.radosgw.cn-bj-1]
rgw region = cn
rgw region root pool = .cn.rgw.root
rgw zone = cn-bj
rgw zone root pool = .cn-wz.rgw.root
host = yun168
public_addr = 192.168.10.115
rgw dns name = s3.domain.com
keyring = /etc/ceph/ceph.client.radosgw.keyring
rgw socket path = /var/run/ceph/$name.sock
log file = /var/log/ceph/radosgw.log
debug rgw = 20
rgw print continue = true
rgw should log = true
[root@yun168 ~]# ceph -s
cluster e48b0d5b-ff08-4a8e-88aa-4acd3f5a6204
health HEALTH_OK
monmap e7: 4 mons at {... ... ...}, election epoch 78, quorum
0,1,2,3 0,1,2,3
mdsmap e49: 0/0/1 up
osdmap e3722: 109 osds: 109 up, 109 in
pgmap v106768: 29432 pgs, 19 pools, 12775 GB data, 12786 kobjects
640 GB used, 390 TB / 391 TB avail
29432 active+clean
client io 1734 kB/s rd, 29755 kB/s wr, 443 op/s
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] How to improve performance of ceph objcect storage cluster
OS: CentOS 6.5
Version: Ceph 0.79
Hi, everybody!
I have installed a ceph cluster with 10 servers.
I test the throughput of ceph cluster in the same datacenter.
Upload files of 1GB from one server or several servers to one server or several
servers, the total is about 30MB/s.
That is to say, there is no difference between one server or one cluster about
throughput when uploading files.
How to optimize the performance of ceph object storage?
Thanks!
Info about ceph cluster:
4 MONs in the first 4 nodes in the cluster.
11 OSDs in each server, 109 OSDs in total (one disk was bad).
4TB each disk, 391TB in total (109*4-391=45TB.Where did 45TB space?)
1 RGW in each server, 10 RGWs in total.That is to say, I can use S3 API in each
Server.
ceph.conf:
[global]
auth supported = none
;auth_service_required = cephx
;auth_client_required = cephx
;auth_cluster_required = cephx
filestore_xattr_use_omap = true
max open files = 131072
log file = /var/log/ceph/$name.log
pid file = /var/run/ceph/$name.pid
keyring = /etc/ceph/keyring.admin
mon_clock_drift_allowed = 2 ;clock skew detected
[mon]
mon data = /data/mon$id
keyring = /etc/ceph/keyring.$name
[osd]
osd data = /data/osd$id
osd journal = /data/osd$id/journal
osd journal size = 1024;
keyring = /etc/ceph/keyring.$name
osd mkfs type = xfs
osd mount options xfs = rw,noatime
osd mkfs options xfs = -f
[client.radosgw.cn-bj-1]
rgw region = cn
rgw region root pool = .cn.rgw.root
rgw zone = cn-bj
rgw zone root pool = .cn-wz.rgw.root
host = yun168
public_addr = 192.168.10.115
rgw dns name = s3.domain.com
keyring = /etc/ceph/ceph.client.radosgw.keyring
rgw socket path = /var/run/ceph/$name.sock
log file = /var/log/ceph/radosgw.log
debug rgw = 20
rgw print continue = true
rgw should log = true
[root@yun168 ~]# ceph -s
cluster e48b0d5b-ff08-4a8e-88aa-4acd3f5a6204
health HEALTH_OK
monmap e7: 4 mons at {... ... ...}, election epoch 78, quorum 0,1,2,3
0,1,2,3
mdsmap e49: 0/0/1 up
osdmap e3722: 109 osds: 109 up, 109 in
pgmap v106768: 29432 pgs, 19 pools, 12775 GB data, 12786 kobjects
640 GB used, 390 TB / 391 TB avail
29432 active+clean
client io 1734 kB/s rd, 29755 kB/s wr, 443 op/s
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] How to improve performance of ceph objcect storage cluster
Hi, I am also searching for tuning the single thread performance. You can try following parameters: [osd] osd mount options xfs = rw,noatime,inode64,logbsize=256k,delaylog,allocsize=4M osd_op_threads = 4 osd_disk_threads = 4 Udo Am 25.06.2014 08:52, schrieb wsnote: OS: CentOS 6.5 Version: Ceph 0.79 Hi, everybody! I have installed a ceph cluster with 10 servers. I test the throughput of ceph cluster in the same datacenter. Upload files of 1GB from one server or several servers to one server or several servers, the total is about 30MB/s. That is to say, there is no difference between one server or one cluster about throughput when uploading files. How to optimize the performance of ceph object storage? Thanks! ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] How to improve performance of ceph objcect storage cluster
I'm assuming you're testing the speed of cephfs (the file system) and not ceph object storage. In my recent experience the primary thing that sped cephfs up was turning on striping. That way the client should be able to pull down data from all 10 nodes at once, and writes should, also, be written in parallel as well. Setting the various cache, stripe, etc sizes is particularly important and depends heavily on the type of load you are testing. -Original Message- From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Udo Lembke Sent: Wednesday, June 25, 2014 7:09 AM To: ceph-users@lists.ceph.com Subject: Re: [ceph-users] How to improve performance of ceph objcect storage cluster Hi, I am also searching for tuning the single thread performance. You can try following parameters: [osd] osd mount options xfs = rw,noatime,inode64,logbsize=256k,delaylog,allocsize=4M osd_op_threads = 4 osd_disk_threads = 4 Udo Am 25.06.2014 08:52, schrieb wsnote: OS: CentOS 6.5 Version: Ceph 0.79 Hi, everybody! I have installed a ceph cluster with 10 servers. I test the throughput of ceph cluster in the same datacenter. Upload files of 1GB from one server or several servers to one server or several servers, the total is about 30MB/s. That is to say, there is no difference between one server or one cluster about throughput when uploading files. How to optimize the performance of ceph object storage? Thanks! ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
