Hi Mark, It's a test cluster and I will try with the new release. As I mentioned in the mail, I think number of rados client instance is the limitation. Could you please let me know how many rados client instance the radosgw daemon is instantiating ? Is it configurable somehow ?
Thanks & Regards Somnath -----Original Message----- From: Mark Nelson [mailto:mark.nel...@inktank.com] Sent: Friday, September 20, 2013 4:02 PM To: Somnath Roy Cc: ceph-users@lists.ceph.com Subject: Re: [ceph-users] Scaling radosgw module On 09/20/2013 05:49 PM, Somnath Roy wrote: > Hi Mark, > Thanks for your quick response. > I tried adding the 'num_container = 100' in the job file and found that the > performance actually decreasing with that option. I am getting around 1K less > iops after putting this. Another observation is that in order to get back the > earlier iops I need to restart the radosgw service. Just removing the > num_container option from the job file and running swift-bench again is not > helping. It seems something radosgw service is caching here. Interesting, that means you aren't being limited by a single container index only residing on 1 OSD. Eventually that might be a limitation, but not here apparently. > > Regarding object size, I have tried with larger object size as well but iops > are much lower in those cases. Yeah, the larger the object size the lower the iops, but potentially the higher the MB/s throughput. > > Regarding moving it to the ceph wip branch, can I just upgrade from dumpling ? Yes, it's actually just dumpling with a minor code change, however given that it's development code I would not recommend doing this if the cluster is in production. > > Thanks & Regards > Somnath > > -----Original Message----- > From: ceph-users-boun...@lists.ceph.com > [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Mark Nelson > Sent: Friday, September 20, 2013 3:03 PM > To: ceph-users@lists.ceph.com > Subject: Re: [ceph-users] Scaling radosgw module > > Hi, > > A couple of things that might be worth trying: > > use multiple containers in swift-bench. Newer versions should support this. > Also, if this is a test cluster, you may want to try the ceph > wip-6286 branch as we have a rather major performance improvement in it when > dealing with small objects. > > Beyond that, we are currently investigating performance slowdowns due to OSD > directory splitting behavior that can crop up with many (millions) of > objects. This we think has potentially been hitting a couple of folks that > have very large object collections. > > Thanks, > Mark > > On 09/20/2013 04:57 PM, Somnath Roy wrote: >> Hi, >> I am running Ceph on a 3 node cluster and each of my server node is running >> 10 OSDs, one for each disk. I have one admin node and all the nodes are >> connected with 2 X 10G network. One network is for cluster and other one >> configured as public network. >> >> All the OSD journals are on SSDs. >> >> I started with rados bench command to benchmark the read performance of this >> Cluster on a large pool (~10K PGs) and found that each rados client has a >> limitation. Each client can only drive up to a certain mark. Each server >> node cpu utilization shows it is around 85-90% idle and the admin node >> (from where rados client is running) is around ~80-85% idle. I am trying >> with 4K object size. >> >> I started running more clients on the admin node and the performance is >> scaling till it hits the client cpu limit. Server still has the cpu of >> 30-35% idle. >> >> Now, I am behind radosgw and in one of the server node I installed the >> required modules like apache, fastcgi, radosgw etc. I configured swift >> bench and started benchmarking. Here is my swift-bench job script. >> >> [bench] >> auth = http://<my-server>/auth >> user = somroy:swift >> key = UbJl9o+OPnzGaRbgqkS9OtPQ01TkAXAeA9RmVzVt >> concurrency = 64 >> object_size = 4096 >> num_objects = 1000 >> num_gets = 200000 >> delete = yes >> auth_version = 1.0 >> >> >> First of all, the read performance I am getting with one radosgw is more >> than 5x slower than what I am getting with one rbd client or one rados bench >> client. Is this expected ? Here is my ceph.conf radosgw config option. >> >> [client.radosgw.gateway] >> host = emsserver1 >> keyring = /etc/ceph/keyring.radosgw.gateway rgw_socket_path = >> /tmp/radosgw.sock log_file = /var/log/ceph/radosgw.log rgw_dns_name = >> <ip> rgw_ops_log_rados = false debug_rgw = 0 rgw_thread_pool_size = >> 300 >> >> The server node (where radosgw is also present) avg cpu utilization is very >> low (~75-80% idle). Out of the ~20% consumption, I saw radosgw is consuming >> bulk of the cpu in the node and ceph-osds are not much. The other two server >> node is ~95% idle ; 10 ceph-osds are consuming this of total 5% of cpu !! >> >> So, clearly, I am not able to generate much load on the cluster. >> So, I tried to run multiple swift-bench instances with the same job , all >> hitting the single radosgw instance. I saw no improvement on the >> performance, each instance iops is almost now = (single instance iop/number >> of swift-bench instance). The aggregated iops is remaining almost same as of >> single instance. >> >> This means we are hitting the single client instance limit here too. >> My question is, for all the requests radosgw is opening only single client >> connection to the object store ? >> If so, is there any configuration like 'noshare' option in case of rbd that >> Josh pointed out in my earlier mail ? >> >> If not, how a single radosgw instance will scale ? >> >> Appreciate, if anybody can help me on this. >> >> Thanks & Regards >> Somnath >> >> ________________________________ >> >> PLEASE NOTE: The information contained in this electronic mail message is >> intended only for the use of the designated recipient(s) named above. If the >> reader of this message is not the intended recipient, you are hereby >> notified that you have received this message in error and that any review, >> dissemination, distribution, or copying of this message is strictly >> prohibited. If you have received this communication in error, please notify >> the sender by telephone or e-mail (as shown above) immediately and destroy >> any and all copies of this message in your possession (whether hard copies >> or electronically stored copies). >> >> >> _______________________________________________ >> ceph-users mailing list >> ceph-users@lists.ceph.com >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> > > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
