Here we go for a complete round of my hosts, all are objsize : 0
Here is a sample answer (headers only, that are followed by the full set
of JSON content) from the RIAK5 host
HTTP/1.1 200 OK
X-Riak-Vclock: a85hYGBgzGDKBVI8xTxKnGbpn7QYuPafyWBKZMxjZXjyYfYFviwA
Vary: Accept-Encoding
Server: MochiWeb/1.1 WebMachine/1.10.8 (that head fake, tho)
Link: </buckets/twitter>; rel="up"
Last-Modified: Mon, 02 May 2016 15:40:20 GMT
ETag: "2l2QODpewyBZQFqDnyEy3F"
Date: Mon, 02 May 2016 15:40:20 GMT
Content-Type: application/json
Content-Length: 10722
Below the riak-admin status output.
admin@riak1:~$ sudo riak-admin status | grep -e 'objsize' | grep 'put'
consistent_put_objsize_100 : 0
consistent_put_objsize_95 : 0
consistent_put_objsize_99 : 0
consistent_put_objsize_mean : 0
consistent_put_objsize_median : 0
write_once_put_objsize_100 : 0
write_once_put_objsize_95 : 0
write_once_put_objsize_99 : 0
write_once_put_objsize_mean : 0
write_once_put_objsize_median : 0
admin@riak2:~$ sudo riak-admin status | grep -e 'objsize' | grep 'put'
consistent_put_objsize_100 : 0
consistent_put_objsize_95 : 0
consistent_put_objsize_99 : 0
consistent_put_objsize_mean : 0
consistent_put_objsize_median : 0
write_once_put_objsize_100 : 0
write_once_put_objsize_95 : 0
write_once_put_objsize_99 : 0
write_once_put_objsize_mean : 0
write_once_put_objsize_median : 0
admin@riak3:~$ sudo riak-admin status | grep -e 'objsize' | grep 'put'
consistent_put_objsize_100 : 0
consistent_put_objsize_95 : 0
consistent_put_objsize_99 : 0
consistent_put_objsize_mean : 0
consistent_put_objsize_median : 0
write_once_put_objsize_100 : 0
write_once_put_objsize_95 : 0
write_once_put_objsize_99 : 0
write_once_put_objsize_mean : 0
write_once_put_objsize_median : 0
admin@riak4:~$ sudo riak-admin status | grep -e 'objsize' | grep 'put'
consistent_put_objsize_100 : 0
consistent_put_objsize_95 : 0
consistent_put_objsize_99 : 0
consistent_put_objsize_mean : 0
consistent_put_objsize_median : 0
write_once_put_objsize_100 : 0
write_once_put_objsize_95 : 0
write_once_put_objsize_99 : 0
write_once_put_objsize_mean : 0
write_once_put_objsize_median : 0
admin@riak5:~$ sudo riak-admin status | grep -e 'objsize' | grep 'put'
consistent_put_objsize_100 : 0
consistent_put_objsize_95 : 0
consistent_put_objsize_99 : 0
consistent_put_objsize_mean : 0
consistent_put_objsize_median : 0
write_once_put_objsize_100 : 0
write_once_put_objsize_95 : 0
write_once_put_objsize_99 : 0
write_once_put_objsize_mean : 0
write_once_put_objsize_median : 0
On 02/05/2016 17:32, Luke Bakken wrote:
Could you please check the objsize stats on every Riak node? If they
are all zero then ... ????
--
Luke Bakken
Engineer
lbak...@basho.com
On Mon, May 2, 2016 at 8:26 AM, Guillaume Boddaert
<guilla...@lighthouse-analytics.co> wrote:
My clients are working through an haproxy box configured on round-robin.
I've switched from PBC to HTTP to provide you this:
May 2 15:24:12 intrabalancer haproxy[29677]: my_daemon_box:53456
[02/May/2016:15:24:12.390] riak_rest riak_rest_backend/riak2 6/0/1/54/61 503
222 - - ---- 5/4/2/1/0 0/0 "PUT
/types/activity_fr/buckets/twitter/keys/727126697699594240?dw=1&timeout=50&returnbody=true
HTTP/1.1"
May 2 15:24:12 intrabalancer haproxy[29677]: my_daemon_box:53455
[02/May/2016:15:24:12.339] riak_rest riak_rest_backend/riak3 57/0/0/54/111
503 222 - - ---- 5/4/1/1/0 0/0 "PUT
/types/activity_fr/buckets/twitter/keys/727126698291023872?dw=1&timeout=50&returnbody=true
HTTP/1.1"
May 2 15:24:12 intrabalancer haproxy[29677]: my_daemon_box:53454
[02/May/2016:15:24:12.285] riak_rest riak_rest_backend/riak5 63/0/0/106/169
503 222 - - ---- 5/4/0/1/0 0/0 "PUT
/types/activity_fr/buckets/twitter/keys/727126698249064449?dw=1&timeout=50&returnbody=true
HTTP/1.1"
May 2 15:24:12 intrabalancer haproxy[29677]: my_daemon_box:53455
[02/May/2016:15:24:12.451] riak_rest riak_rest_backend/riak5 13/0/1/58/72
503 222 - - ---- 5/4/3/1/0 0/0 "PUT
/types/activity_fr/buckets/twitter/keys/727126696256802816?dw=1&timeout=50&returnbody=true
HTTP/1.1"
May 2 15:24:12 intrabalancer haproxy[29677]: my_daemon_box:53456
[02/May/2016:15:24:12.451] riak_rest riak_rest_backend/riak4 10/0/1/62/73
503 222 - - ---- 5/4/2/1/0 0/0 "PUT
/types/activity_fr/buckets/twitter/keys/727126698425212929?dw=1&timeout=50&returnbody=true
HTTP/1.1"
May 2 15:24:12 intrabalancer haproxy[29677]: my_daemon_box:53453
[02/May/2016:15:24:12.397] riak_rest riak_rest_backend/riak1 66/0/0/63/129
503 222 - - ---- 5/4/1/1/0 0/0 "PUT
/types/activity_fr/buckets/twitter/keys/727126698215485440?dw=1&timeout=50&returnbody=true
HTTP/1.1"
Yet objsize is still 0 on puts:
admin@riak1:~$ sudo riak-admin status | grep -e 'objsize'
consistent_get_objsize_100 : 0
consistent_get_objsize_95 : 0
consistent_get_objsize_99 : 0
consistent_get_objsize_mean : 0
consistent_get_objsize_median : 0
consistent_put_objsize_100 : 0
consistent_put_objsize_95 : 0
consistent_put_objsize_99 : 0
consistent_put_objsize_mean : 0
consistent_put_objsize_median : 0
node_get_fsm_counter_objsize_100 : 0
node_get_fsm_counter_objsize_95 : 0
node_get_fsm_counter_objsize_99 : 0
node_get_fsm_counter_objsize_mean : 0
node_get_fsm_counter_objsize_median : 0
node_get_fsm_map_objsize_100 : 0
node_get_fsm_map_objsize_95 : 0
node_get_fsm_map_objsize_99 : 0
node_get_fsm_map_objsize_mean : 0
node_get_fsm_map_objsize_median : 0
node_get_fsm_objsize_100 : 10916
node_get_fsm_objsize_95 : 7393
node_get_fsm_objsize_99 : 8845
node_get_fsm_objsize_mean : 4098
node_get_fsm_objsize_median : 3891
node_get_fsm_set_objsize_100 : 0
node_get_fsm_set_objsize_95 : 0
node_get_fsm_set_objsize_99 : 0
node_get_fsm_set_objsize_mean : 0
node_get_fsm_set_objsize_median : 0
write_once_put_objsize_100 : 0
write_once_put_objsize_95 : 0
write_once_put_objsize_99 : 0
write_once_put_objsize_mean : 0
write_once_put_objsize_median : 0
G.
On 02/05/2016 17:21, Luke Bakken wrote:
Which Riak client are you using? Do you have it configured to connect
to all nodes in your cluster or just one?
--
Luke Bakken
Engineer
lbak...@basho.com
On Mon, May 2, 2016 at 7:40 AM, Guillaume Boddaert
<guilla...@lighthouse-analytics.co> wrote:
Hi Luke,
Well objsize seems to be 0, that's very troubling.
I can assure you that i am writing 75 items per second at the moment and
that I can pull data from the cluster.
admin@riak3:~$ sudo riak-admin status | grep -e 'objsize'
consistent_get_objsize_100 : 0
consistent_get_objsize_95 : 0
consistent_get_objsize_99 : 0
consistent_get_objsize_mean : 0
consistent_get_objsize_median : 0
consistent_put_objsize_100 : 0
consistent_put_objsize_95 : 0
consistent_put_objsize_99 : 0
consistent_put_objsize_mean : 0
consistent_put_objsize_median : 0
node_get_fsm_counter_objsize_100 : 0
node_get_fsm_counter_objsize_95 : 0
node_get_fsm_counter_objsize_99 : 0
node_get_fsm_counter_objsize_mean : 0
node_get_fsm_counter_objsize_median : 0
node_get_fsm_map_objsize_100 : 0
node_get_fsm_map_objsize_95 : 0
node_get_fsm_map_objsize_99 : 0
node_get_fsm_map_objsize_mean : 0
node_get_fsm_map_objsize_median : 0
node_get_fsm_objsize_100 : 0
node_get_fsm_objsize_95 : 0
node_get_fsm_objsize_99 : 0
node_get_fsm_objsize_mean : 0
node_get_fsm_objsize_median : 0
node_get_fsm_set_objsize_100 : 0
node_get_fsm_set_objsize_95 : 0
node_get_fsm_set_objsize_99 : 0
node_get_fsm_set_objsize_mean : 0
node_get_fsm_set_objsize_median : 0
write_once_put_objsize_100 : 0
write_once_put_objsize_95 : 0
write_once_put_objsize_99 : 0
write_once_put_objsize_mean : 0
write_once_put_objsize_median : 0
On 02/05/2016 16:13, Luke Bakken wrote:
Hi Guillaume -
What are the "objsize" stats for your cluster?
--
Luke Bakken
Engineer
lbak...@basho.com
On Mon, May 2, 2016 at 4:45 AM, Guillaume Boddaert
<guilla...@lighthouse-analytics.co> wrote:
Hi,
I'm trying to setup a production environment with Riak as backend.
Unfortunately I have very slow write times that bottleneck my whole
system.
Here is a sample of one of my node (riak-admin status | grep -e
'^node_put_fsm_time'):
node_put_fsm_time_100 : 3305516
node_put_fsm_time_95 : 230589
node_put_fsm_time_99 : 1694593
node_put_fsm_time_mean : 79864
node_put_fsm_time_median : 14973
As you can see, I have really good times for most of my writes, yet the
mean
time is not so good because a few writes are taking long (Up to 3
seconds)
How can I get rid of those slow insert ? Is that intended/normal ?
My setup is the following:
5 hosts (2CPU, %Cpu(s): 47,1 us, 1,3 sy, 0,0 ni, 51,3 id, 0,0 wa,
0,0
hi, 0,2 si, 0,0 st) , ring_size: 128, aae disabled.
Writes are w=1 dw=0
each host has 32go of ram, that is almost used for system caching only.
My data are stored on an openstack volume that support up to 3000IOPS.
Here is an iostat sample for 1 minute:
avg-cpu: %user %nice %system %iowait %steal %idle
38.00 0.00 1.81 0.03 0.08 60.07
Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn
vda 0.37 0.00 2.27 0 136
vdb 9.60 0.00 294.53 0 17672
Thanks,
Guilaume BODDAERT
_______________________________________________
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
_______________________________________________
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
_______________________________________________
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
