I believe you should be looking for the get_fsm_objsize stats listed here: http://docs.basho.com/riak/kv/2.1.4/using/cluster-operations/inspecting-node/#get-fsm-objsize . Unless you are using consistent bucket types or write once bucket types.
-Alexander Alexander Sicular Solutions Architect Basho Technologies 9175130679 @siculars On Mon, May 2, 2016 at 10:43 AM, Guillaume Boddaert < guilla...@lighthouse-analytics.co> wrote: > Here we go for a complete round of my hosts, all are objsize : 0 > > Here is a sample answer (headers only, that are followed by the full set > of JSON content) from the RIAK5 host > > HTTP/1.1 200 OK > X-Riak-Vclock: a85hYGBgzGDKBVI8xTxKnGbpn7QYuPafyWBKZMxjZXjyYfYFviwA > Vary: Accept-Encoding > Server: MochiWeb/1.1 WebMachine/1.10.8 (that head fake, tho) > Link: </buckets/twitter>; rel="up" > Last-Modified: Mon, 02 May 2016 15:40:20 GMT > ETag: "2l2QODpewyBZQFqDnyEy3F" > Date: Mon, 02 May 2016 15:40:20 GMT > Content-Type: application/json > Content-Length: 10722 > > Below the riak-admin status output. > > > admin@riak1:~$ sudo riak-admin status | grep -e 'objsize' | grep 'put' > consistent_put_objsize_100 : 0 > consistent_put_objsize_95 : 0 > consistent_put_objsize_99 : 0 > consistent_put_objsize_mean : 0 > consistent_put_objsize_median : 0 > write_once_put_objsize_100 : 0 > write_once_put_objsize_95 : 0 > write_once_put_objsize_99 : 0 > write_once_put_objsize_mean : 0 > write_once_put_objsize_median : 0 > > admin@riak2:~$ sudo riak-admin status | grep -e 'objsize' | grep 'put' > consistent_put_objsize_100 : 0 > consistent_put_objsize_95 : 0 > consistent_put_objsize_99 : 0 > consistent_put_objsize_mean : 0 > consistent_put_objsize_median : 0 > write_once_put_objsize_100 : 0 > write_once_put_objsize_95 : 0 > write_once_put_objsize_99 : 0 > write_once_put_objsize_mean : 0 > write_once_put_objsize_median : 0 > > admin@riak3:~$ sudo riak-admin status | grep -e 'objsize' | grep 'put' > consistent_put_objsize_100 : 0 > consistent_put_objsize_95 : 0 > consistent_put_objsize_99 : 0 > consistent_put_objsize_mean : 0 > consistent_put_objsize_median : 0 > write_once_put_objsize_100 : 0 > write_once_put_objsize_95 : 0 > write_once_put_objsize_99 : 0 > write_once_put_objsize_mean : 0 > write_once_put_objsize_median : 0 > > admin@riak4:~$ sudo riak-admin status | grep -e 'objsize' | grep 'put' > > consistent_put_objsize_100 : 0 > consistent_put_objsize_95 : 0 > consistent_put_objsize_99 : 0 > consistent_put_objsize_mean : 0 > consistent_put_objsize_median : 0 > write_once_put_objsize_100 : 0 > write_once_put_objsize_95 : 0 > write_once_put_objsize_99 : 0 > write_once_put_objsize_mean : 0 > write_once_put_objsize_median : 0 > > admin@riak5:~$ sudo riak-admin status | grep -e 'objsize' | grep 'put' > > consistent_put_objsize_100 : 0 > consistent_put_objsize_95 : 0 > consistent_put_objsize_99 : 0 > consistent_put_objsize_mean : 0 > consistent_put_objsize_median : 0 > write_once_put_objsize_100 : 0 > write_once_put_objsize_95 : 0 > write_once_put_objsize_99 : 0 > write_once_put_objsize_mean : 0 > write_once_put_objsize_median : 0 > > On 02/05/2016 17:32, Luke Bakken wrote: > >> Could you please check the objsize stats on every Riak node? If they >> are all zero then ... ???? >> -- >> Luke Bakken >> Engineer >> lbak...@basho.com >> >> >> On Mon, May 2, 2016 at 8:26 AM, Guillaume Boddaert >> <guilla...@lighthouse-analytics.co> wrote: >> >>> My clients are working through an haproxy box configured on round-robin. >>> >>> I've switched from PBC to HTTP to provide you this: >>> >>> May 2 15:24:12 intrabalancer haproxy[29677]: my_daemon_box:53456 >>> [02/May/2016:15:24:12.390] riak_rest riak_rest_backend/riak2 6/0/1/54/61 >>> 503 >>> 222 - - ---- 5/4/2/1/0 0/0 "PUT >>> >>> /types/activity_fr/buckets/twitter/keys/727126697699594240?dw=1&timeout=50&returnbody=true >>> HTTP/1.1" >>> May 2 15:24:12 intrabalancer haproxy[29677]: my_daemon_box:53455 >>> [02/May/2016:15:24:12.339] riak_rest riak_rest_backend/riak3 >>> 57/0/0/54/111 >>> 503 222 - - ---- 5/4/1/1/0 0/0 "PUT >>> >>> /types/activity_fr/buckets/twitter/keys/727126698291023872?dw=1&timeout=50&returnbody=true >>> HTTP/1.1" >>> May 2 15:24:12 intrabalancer haproxy[29677]: my_daemon_box:53454 >>> [02/May/2016:15:24:12.285] riak_rest riak_rest_backend/riak5 >>> 63/0/0/106/169 >>> 503 222 - - ---- 5/4/0/1/0 0/0 "PUT >>> >>> /types/activity_fr/buckets/twitter/keys/727126698249064449?dw=1&timeout=50&returnbody=true >>> HTTP/1.1" >>> May 2 15:24:12 intrabalancer haproxy[29677]: my_daemon_box:53455 >>> [02/May/2016:15:24:12.451] riak_rest riak_rest_backend/riak5 13/0/1/58/72 >>> 503 222 - - ---- 5/4/3/1/0 0/0 "PUT >>> >>> /types/activity_fr/buckets/twitter/keys/727126696256802816?dw=1&timeout=50&returnbody=true >>> HTTP/1.1" >>> May 2 15:24:12 intrabalancer haproxy[29677]: my_daemon_box:53456 >>> [02/May/2016:15:24:12.451] riak_rest riak_rest_backend/riak4 10/0/1/62/73 >>> 503 222 - - ---- 5/4/2/1/0 0/0 "PUT >>> >>> /types/activity_fr/buckets/twitter/keys/727126698425212929?dw=1&timeout=50&returnbody=true >>> HTTP/1.1" >>> May 2 15:24:12 intrabalancer haproxy[29677]: my_daemon_box:53453 >>> [02/May/2016:15:24:12.397] riak_rest riak_rest_backend/riak1 >>> 66/0/0/63/129 >>> 503 222 - - ---- 5/4/1/1/0 0/0 "PUT >>> >>> /types/activity_fr/buckets/twitter/keys/727126698215485440?dw=1&timeout=50&returnbody=true >>> HTTP/1.1" >>> >>> Yet objsize is still 0 on puts: >>> admin@riak1:~$ sudo riak-admin status | grep -e 'objsize' >>> consistent_get_objsize_100 : 0 >>> consistent_get_objsize_95 : 0 >>> consistent_get_objsize_99 : 0 >>> consistent_get_objsize_mean : 0 >>> consistent_get_objsize_median : 0 >>> consistent_put_objsize_100 : 0 >>> consistent_put_objsize_95 : 0 >>> consistent_put_objsize_99 : 0 >>> consistent_put_objsize_mean : 0 >>> consistent_put_objsize_median : 0 >>> node_get_fsm_counter_objsize_100 : 0 >>> node_get_fsm_counter_objsize_95 : 0 >>> node_get_fsm_counter_objsize_99 : 0 >>> node_get_fsm_counter_objsize_mean : 0 >>> node_get_fsm_counter_objsize_median : 0 >>> node_get_fsm_map_objsize_100 : 0 >>> node_get_fsm_map_objsize_95 : 0 >>> node_get_fsm_map_objsize_99 : 0 >>> node_get_fsm_map_objsize_mean : 0 >>> node_get_fsm_map_objsize_median : 0 >>> node_get_fsm_objsize_100 : 10916 >>> node_get_fsm_objsize_95 : 7393 >>> node_get_fsm_objsize_99 : 8845 >>> node_get_fsm_objsize_mean : 4098 >>> node_get_fsm_objsize_median : 3891 >>> node_get_fsm_set_objsize_100 : 0 >>> node_get_fsm_set_objsize_95 : 0 >>> node_get_fsm_set_objsize_99 : 0 >>> node_get_fsm_set_objsize_mean : 0 >>> node_get_fsm_set_objsize_median : 0 >>> write_once_put_objsize_100 : 0 >>> write_once_put_objsize_95 : 0 >>> write_once_put_objsize_99 : 0 >>> write_once_put_objsize_mean : 0 >>> write_once_put_objsize_median : 0 >>> >>> >>> >>> G. >>> >>> >>> On 02/05/2016 17:21, Luke Bakken wrote: >>> >>>> Which Riak client are you using? Do you have it configured to connect >>>> to all nodes in your cluster or just one? >>>> -- >>>> Luke Bakken >>>> Engineer >>>> lbak...@basho.com >>>> >>>> >>>> On Mon, May 2, 2016 at 7:40 AM, Guillaume Boddaert >>>> <guilla...@lighthouse-analytics.co> wrote: >>>> >>>>> Hi Luke, >>>>> >>>>> Well objsize seems to be 0, that's very troubling. >>>>> I can assure you that i am writing 75 items per second at the moment >>>>> and >>>>> that I can pull data from the cluster. >>>>> >>>>> >>>>> admin@riak3:~$ sudo riak-admin status | grep -e 'objsize' >>>>> >>>>> consistent_get_objsize_100 : 0 >>>>> consistent_get_objsize_95 : 0 >>>>> consistent_get_objsize_99 : 0 >>>>> consistent_get_objsize_mean : 0 >>>>> consistent_get_objsize_median : 0 >>>>> consistent_put_objsize_100 : 0 >>>>> consistent_put_objsize_95 : 0 >>>>> consistent_put_objsize_99 : 0 >>>>> consistent_put_objsize_mean : 0 >>>>> consistent_put_objsize_median : 0 >>>>> node_get_fsm_counter_objsize_100 : 0 >>>>> node_get_fsm_counter_objsize_95 : 0 >>>>> node_get_fsm_counter_objsize_99 : 0 >>>>> node_get_fsm_counter_objsize_mean : 0 >>>>> node_get_fsm_counter_objsize_median : 0 >>>>> node_get_fsm_map_objsize_100 : 0 >>>>> node_get_fsm_map_objsize_95 : 0 >>>>> node_get_fsm_map_objsize_99 : 0 >>>>> node_get_fsm_map_objsize_mean : 0 >>>>> node_get_fsm_map_objsize_median : 0 >>>>> node_get_fsm_objsize_100 : 0 >>>>> node_get_fsm_objsize_95 : 0 >>>>> node_get_fsm_objsize_99 : 0 >>>>> node_get_fsm_objsize_mean : 0 >>>>> node_get_fsm_objsize_median : 0 >>>>> node_get_fsm_set_objsize_100 : 0 >>>>> node_get_fsm_set_objsize_95 : 0 >>>>> node_get_fsm_set_objsize_99 : 0 >>>>> node_get_fsm_set_objsize_mean : 0 >>>>> node_get_fsm_set_objsize_median : 0 >>>>> write_once_put_objsize_100 : 0 >>>>> write_once_put_objsize_95 : 0 >>>>> write_once_put_objsize_99 : 0 >>>>> write_once_put_objsize_mean : 0 >>>>> write_once_put_objsize_median : 0 >>>>> >>>>> >>>>> >>>>> >>>>> On 02/05/2016 16:13, Luke Bakken wrote: >>>>> >>>>>> Hi Guillaume - >>>>>> >>>>>> What are the "objsize" stats for your cluster? >>>>>> -- >>>>>> Luke Bakken >>>>>> Engineer >>>>>> lbak...@basho.com >>>>>> >>>>>> >>>>>> On Mon, May 2, 2016 at 4:45 AM, Guillaume Boddaert >>>>>> <guilla...@lighthouse-analytics.co> wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> I'm trying to setup a production environment with Riak as backend. >>>>>>> Unfortunately I have very slow write times that bottleneck my whole >>>>>>> system. >>>>>>> >>>>>>> Here is a sample of one of my node (riak-admin status | grep -e >>>>>>> '^node_put_fsm_time'): >>>>>>> node_put_fsm_time_100 : 3305516 >>>>>>> node_put_fsm_time_95 : 230589 >>>>>>> node_put_fsm_time_99 : 1694593 >>>>>>> node_put_fsm_time_mean : 79864 >>>>>>> node_put_fsm_time_median : 14973 >>>>>>> >>>>>>> As you can see, I have really good times for most of my writes, yet >>>>>>> the >>>>>>> mean >>>>>>> time is not so good because a few writes are taking long (Up to 3 >>>>>>> seconds) >>>>>>> How can I get rid of those slow insert ? Is that intended/normal ? >>>>>>> >>>>>>> My setup is the following: >>>>>>> 5 hosts (2CPU, %Cpu(s): 47,1 us, 1,3 sy, 0,0 ni, 51,3 id, 0,0 wa, >>>>>>> 0,0 >>>>>>> hi, 0,2 si, 0,0 st) , ring_size: 128, aae disabled. >>>>>>> Writes are w=1 dw=0 >>>>>>> each host has 32go of ram, that is almost used for system caching >>>>>>> only. >>>>>>> My data are stored on an openstack volume that support up to >>>>>>> 3000IOPS. >>>>>>> >>>>>>> Here is an iostat sample for 1 minute: >>>>>>> avg-cpu: %user %nice %system %iowait %steal %idle >>>>>>> 38.00 0.00 1.81 0.03 0.08 60.07 >>>>>>> >>>>>>> Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn >>>>>>> vda 0.37 0.00 2.27 0 136 >>>>>>> vdb 9.60 0.00 294.53 0 17672 >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Guilaume BODDAERT >>>>>>> >>>>>>> _______________________________________________ >>>>>>> riak-users mailing list >>>>>>> riak-users@lists.basho.com >>>>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >>>>>>> >>>>>> >>>>> >>>>> _______________________________________________ >>>>> riak-users mailing list >>>>> riak-users@lists.basho.com >>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >>>>> >>>> >>> > > _______________________________________________ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >
_______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
