Hello, There is a known limitation of bucket scalability, and there is a blueprint tracking it - https://wiki.ceph.com/Planning/Blueprints/Submissions/rgw%3A_bucket_index_scalability.
At time being, I would recommend to do sharding at application level (create multiple buckets) to workaround this limitation. Thanks, Guang On Jun 30, 2014, at 2:54 PM, baijia...@126.com wrote: > > hello, everyone! > > when I user rest bench test RGW performance and the cmd is: > ./rest-bench --access-key=ak --secret=sk --bucket=bucket_name --seconds=600 > -t 200 -b 524288 -no-cleanup write > > test result: > Total time run: 362.962324 T > otal writes made: 48189 > Write size: 524288 > Bandwidth (MB/sec): 66.383 > Stddev Bandwidth: 40.7776 > Max bandwidth (MB/sec): 173 > Min bandwidth (MB/sec): 0 > Average Latency: 1.50435 > Stddev Latency: 0.910731 > Max latency: 9.12276 > Min latency: 0.19867 > > my environment is 4 host and 40 disk(osd)。 but test result is very bad, > average latency is 1.5 seconds 。and I find write obj metadate is very > slowly。because it puts so many object to one bucket, we know writing object > metadate can call method “bucket_prepare_op”,and test find this op is very > slowly。 I find the osd which contain bucket-obj。and see the > “bucket_prepare_op”by dump_historic_ops : > { "description": "osd_op(client.4742.0:87613 .dir.default.4243.3 [call > rgw.bucket_prepare_op] 3.3670fe74 e317)", > "received_at": "2014-06-30 13:35:55.409597", > "age": "51.148026", > "duration": "4.130137", > "type_data": [ > "commit sent; apply or cleanup", > { "client": "client.4742", > "tid": 87613}, > [ > { "time": "2014-06-30 13:35:55.409660", > "event": "waiting_for_osdmap"}, > { "time": "2014-06-30 13:35:55.409669", > "event": "queue op_wq"}, > { "time": "2014-06-30 13:35:55.896766", > "event": "reached_pg"}, > { "time": "2014-06-30 13:35:55.896793", > "event": "started"}, > { "time": "2014-06-30 13:35:55.896796", > "event": "started"}, > { "time": "2014-06-30 13:35:55.899450", > "event": "waiting for subops from [40,43]"}, > { "time": "2014-06-30 13:35:55.899757", > "event": "commit_queued_for_journal_write"}, > { "time": "2014-06-30 13:35:55.899799", > "event": "write_thread_in_journal_buffer"}, > { "time": "2014-06-30 13:35:55.899910", > "event": "journaled_completion_queued"}, > { "time": "2014-06-30 13:35:55.899936", > "event": "journal first callback"}, > { "time": "2014-06-30 13:35:55.899944", > "event": "queuing ondisk"}, > { "time": "2014-06-30 13:35:56.142104", > "event": "sub_op_commit_rec"}, > { "time": "2014-06-30 13:35:56.176950", > "event": "sub_op_commit_rec"}, > { "time": "2014-06-30 13:35:59.535301", > "event": "op_commit"}, > { "time": "2014-06-30 13:35:59.535331", > "event": "commit_sent"}, > { "time": "2014-06-30 13:35:59.539723", > "event": "op_applied"}, > { "time": "2014-06-30 13:35:59.539734", > "event": "done"}]]}, > > so why from "journaled_completion_queued" to "op_commit" is very slowly, and > what happened? > thanks > > baijia...@126.com > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com