Could you please check the objsize stats on every Riak node? If they are all zero then ... ???? -- Luke Bakken Engineer lbak...@basho.com
On Mon, May 2, 2016 at 8:26 AM, Guillaume Boddaert <guilla...@lighthouse-analytics.co> wrote: > My clients are working through an haproxy box configured on round-robin. > > I've switched from PBC to HTTP to provide you this: > > May 2 15:24:12 intrabalancer haproxy[29677]: my_daemon_box:53456 > [02/May/2016:15:24:12.390] riak_rest riak_rest_backend/riak2 6/0/1/54/61 503 > 222 - - ---- 5/4/2/1/0 0/0 "PUT > /types/activity_fr/buckets/twitter/keys/727126697699594240?dw=1&timeout=50&returnbody=true > HTTP/1.1" > May 2 15:24:12 intrabalancer haproxy[29677]: my_daemon_box:53455 > [02/May/2016:15:24:12.339] riak_rest riak_rest_backend/riak3 57/0/0/54/111 > 503 222 - - ---- 5/4/1/1/0 0/0 "PUT > /types/activity_fr/buckets/twitter/keys/727126698291023872?dw=1&timeout=50&returnbody=true > HTTP/1.1" > May 2 15:24:12 intrabalancer haproxy[29677]: my_daemon_box:53454 > [02/May/2016:15:24:12.285] riak_rest riak_rest_backend/riak5 63/0/0/106/169 > 503 222 - - ---- 5/4/0/1/0 0/0 "PUT > /types/activity_fr/buckets/twitter/keys/727126698249064449?dw=1&timeout=50&returnbody=true > HTTP/1.1" > May 2 15:24:12 intrabalancer haproxy[29677]: my_daemon_box:53455 > [02/May/2016:15:24:12.451] riak_rest riak_rest_backend/riak5 13/0/1/58/72 > 503 222 - - ---- 5/4/3/1/0 0/0 "PUT > /types/activity_fr/buckets/twitter/keys/727126696256802816?dw=1&timeout=50&returnbody=true > HTTP/1.1" > May 2 15:24:12 intrabalancer haproxy[29677]: my_daemon_box:53456 > [02/May/2016:15:24:12.451] riak_rest riak_rest_backend/riak4 10/0/1/62/73 > 503 222 - - ---- 5/4/2/1/0 0/0 "PUT > /types/activity_fr/buckets/twitter/keys/727126698425212929?dw=1&timeout=50&returnbody=true > HTTP/1.1" > May 2 15:24:12 intrabalancer haproxy[29677]: my_daemon_box:53453 > [02/May/2016:15:24:12.397] riak_rest riak_rest_backend/riak1 66/0/0/63/129 > 503 222 - - ---- 5/4/1/1/0 0/0 "PUT > /types/activity_fr/buckets/twitter/keys/727126698215485440?dw=1&timeout=50&returnbody=true > HTTP/1.1" > > Yet objsize is still 0 on puts: > admin@riak1:~$ sudo riak-admin status | grep -e 'objsize' > consistent_get_objsize_100 : 0 > consistent_get_objsize_95 : 0 > consistent_get_objsize_99 : 0 > consistent_get_objsize_mean : 0 > consistent_get_objsize_median : 0 > consistent_put_objsize_100 : 0 > consistent_put_objsize_95 : 0 > consistent_put_objsize_99 : 0 > consistent_put_objsize_mean : 0 > consistent_put_objsize_median : 0 > node_get_fsm_counter_objsize_100 : 0 > node_get_fsm_counter_objsize_95 : 0 > node_get_fsm_counter_objsize_99 : 0 > node_get_fsm_counter_objsize_mean : 0 > node_get_fsm_counter_objsize_median : 0 > node_get_fsm_map_objsize_100 : 0 > node_get_fsm_map_objsize_95 : 0 > node_get_fsm_map_objsize_99 : 0 > node_get_fsm_map_objsize_mean : 0 > node_get_fsm_map_objsize_median : 0 > node_get_fsm_objsize_100 : 10916 > node_get_fsm_objsize_95 : 7393 > node_get_fsm_objsize_99 : 8845 > node_get_fsm_objsize_mean : 4098 > node_get_fsm_objsize_median : 3891 > node_get_fsm_set_objsize_100 : 0 > node_get_fsm_set_objsize_95 : 0 > node_get_fsm_set_objsize_99 : 0 > node_get_fsm_set_objsize_mean : 0 > node_get_fsm_set_objsize_median : 0 > write_once_put_objsize_100 : 0 > write_once_put_objsize_95 : 0 > write_once_put_objsize_99 : 0 > write_once_put_objsize_mean : 0 > write_once_put_objsize_median : 0 > > > > G. > > > On 02/05/2016 17:21, Luke Bakken wrote: >> >> Which Riak client are you using? Do you have it configured to connect >> to all nodes in your cluster or just one? >> -- >> Luke Bakken >> Engineer >> lbak...@basho.com >> >> >> On Mon, May 2, 2016 at 7:40 AM, Guillaume Boddaert >> <guilla...@lighthouse-analytics.co> wrote: >>> >>> Hi Luke, >>> >>> Well objsize seems to be 0, that's very troubling. >>> I can assure you that i am writing 75 items per second at the moment and >>> that I can pull data from the cluster. >>> >>> >>> admin@riak3:~$ sudo riak-admin status | grep -e 'objsize' >>> >>> consistent_get_objsize_100 : 0 >>> consistent_get_objsize_95 : 0 >>> consistent_get_objsize_99 : 0 >>> consistent_get_objsize_mean : 0 >>> consistent_get_objsize_median : 0 >>> consistent_put_objsize_100 : 0 >>> consistent_put_objsize_95 : 0 >>> consistent_put_objsize_99 : 0 >>> consistent_put_objsize_mean : 0 >>> consistent_put_objsize_median : 0 >>> node_get_fsm_counter_objsize_100 : 0 >>> node_get_fsm_counter_objsize_95 : 0 >>> node_get_fsm_counter_objsize_99 : 0 >>> node_get_fsm_counter_objsize_mean : 0 >>> node_get_fsm_counter_objsize_median : 0 >>> node_get_fsm_map_objsize_100 : 0 >>> node_get_fsm_map_objsize_95 : 0 >>> node_get_fsm_map_objsize_99 : 0 >>> node_get_fsm_map_objsize_mean : 0 >>> node_get_fsm_map_objsize_median : 0 >>> node_get_fsm_objsize_100 : 0 >>> node_get_fsm_objsize_95 : 0 >>> node_get_fsm_objsize_99 : 0 >>> node_get_fsm_objsize_mean : 0 >>> node_get_fsm_objsize_median : 0 >>> node_get_fsm_set_objsize_100 : 0 >>> node_get_fsm_set_objsize_95 : 0 >>> node_get_fsm_set_objsize_99 : 0 >>> node_get_fsm_set_objsize_mean : 0 >>> node_get_fsm_set_objsize_median : 0 >>> write_once_put_objsize_100 : 0 >>> write_once_put_objsize_95 : 0 >>> write_once_put_objsize_99 : 0 >>> write_once_put_objsize_mean : 0 >>> write_once_put_objsize_median : 0 >>> >>> >>> >>> >>> On 02/05/2016 16:13, Luke Bakken wrote: >>>> >>>> Hi Guillaume - >>>> >>>> What are the "objsize" stats for your cluster? >>>> -- >>>> Luke Bakken >>>> Engineer >>>> lbak...@basho.com >>>> >>>> >>>> On Mon, May 2, 2016 at 4:45 AM, Guillaume Boddaert >>>> <guilla...@lighthouse-analytics.co> wrote: >>>>> >>>>> Hi, >>>>> >>>>> I'm trying to setup a production environment with Riak as backend. >>>>> Unfortunately I have very slow write times that bottleneck my whole >>>>> system. >>>>> >>>>> Here is a sample of one of my node (riak-admin status | grep -e >>>>> '^node_put_fsm_time'): >>>>> node_put_fsm_time_100 : 3305516 >>>>> node_put_fsm_time_95 : 230589 >>>>> node_put_fsm_time_99 : 1694593 >>>>> node_put_fsm_time_mean : 79864 >>>>> node_put_fsm_time_median : 14973 >>>>> >>>>> As you can see, I have really good times for most of my writes, yet the >>>>> mean >>>>> time is not so good because a few writes are taking long (Up to 3 >>>>> seconds) >>>>> How can I get rid of those slow insert ? Is that intended/normal ? >>>>> >>>>> My setup is the following: >>>>> 5 hosts (2CPU, %Cpu(s): 47,1 us, 1,3 sy, 0,0 ni, 51,3 id, 0,0 wa, >>>>> 0,0 >>>>> hi, 0,2 si, 0,0 st) , ring_size: 128, aae disabled. >>>>> Writes are w=1 dw=0 >>>>> each host has 32go of ram, that is almost used for system caching only. >>>>> My data are stored on an openstack volume that support up to 3000IOPS. >>>>> >>>>> Here is an iostat sample for 1 minute: >>>>> avg-cpu: %user %nice %system %iowait %steal %idle >>>>> 38.00 0.00 1.81 0.03 0.08 60.07 >>>>> >>>>> Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn >>>>> vda 0.37 0.00 2.27 0 136 >>>>> vdb 9.60 0.00 294.53 0 17672 >>>>> >>>>> Thanks, >>>>> >>>>> Guilaume BODDAERT >>>>> >>>>> _______________________________________________ >>>>> riak-users mailing list >>>>> riak-users@lists.basho.com >>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >>> >>> >>> >>> _______________________________________________ >>> riak-users mailing list >>> riak-users@lists.basho.com >>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > > _______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
