[ceph-users] Radosgw scaling recommendation?

2017-02-09 Thread Benjeman Meekhof
Hi all, We're doing some stress testing with clients hitting our rados gw nodes with simultaneous connections. When the number of client connections exceeds about 5400 we start seeing 403 forbidden errors and log messages like the following: 2017-02-09 08:53:16.915536 7f8c667bc700 0 NOTICE: requ

Re: [ceph-users] Radosgw scaling recommendation?

2017-02-09 Thread Mark Nelson
I'm not really an RGW expert, but I'd suggest increasing the "rgw_thread_pool_size" option to something much higher than the default 100 threads if you haven't already. RGW requires at least 1 thread per client connection, so with many concurrent connections some of them might end up timing ou

Re: [ceph-users] Radosgw scaling recommendation?

2017-02-09 Thread Wido den Hollander
> Op 9 februari 2017 om 19:34 schreef Mark Nelson : > > > I'm not really an RGW expert, but I'd suggest increasing the > "rgw_thread_pool_size" option to something much higher than the default > 100 threads if you haven't already. RGW requires at least 1 thread per > client connection, so wi

Re: [ceph-users] Radosgw scaling recommendation?

2017-02-09 Thread Ben Hines
I'm curious how does the num_threads option to civetweb relate to the 'rgw thread pool size'? Should i make them equal? ie: rgw frontends = civetweb enable_keep_alive=yes port=80 num_threads=125 error_log_file=/var/log/ceph/civetweb.error.log access_log_file=/var/log/ceph/civetweb.access.log -

Re: [ceph-users] Radosgw scaling recommendation?

2017-02-14 Thread Benjeman Meekhof
Thanks everyone for the suggestions, playing with all three of the tuning knobs mentioned has greatly increased the number of client connections an instance can deal with. We're still experimenting to find the max values to saturate our hardware. With values as below we'd see something around 50