Guillaume -

You said earlier "My data are stored on an openstack volume that
support up to 3000IOPS". There is a likelihood that your write load is
exceeding the capacity of your virtual environment, especially if some
Riak nodes are sharing physical disk or server infrastructure.

Some suggestions:

* If you're not using Riak Search, set "search = off" in riak.conf

* Be sure to carefully read and apply all tunings:
http://docs.basho.com/riak/kv/2.1.4/using/performance/

* You may wish to increase the memory dedicated to leveldb:
http://docs.basho.com/riak/kv/2.1.4/configuring/backend/#leveldb

--
Luke Bakken
Engineer
lbak...@basho.com


On Tue, May 3, 2016 at 7:33 AM, Guillaume Boddaert
<guilla...@lighthouse-analytics.co> wrote:
> Hi,
>
> Sorry for the delay, I've spent a lot of time trying to understand if the
> problem was elsewhere.
> I've simplified my infrastructure and got a simple layout that don't rely
> anymore on loadbalancer and also corrected some minor performance issue on
> my workers.
>
> At the moment, i have up to 32 workers that are calling riak for writes,
> each of them are set to :
> w=1
> dw=0
> timeout=1000
> using protobuf
> a timeouted attempt is rerun 180s later
>
> From my application server perspective, 23% of the calls are rejected by
> timeout (75446 tries, 57564 success, 17578 timeout).
>
> Here is a sample riak-admin stat for one of my 5 hosts:
>
> node_put_fsm_time_100 : 999331
> node_put_fsm_time_95 : 773682
> node_put_fsm_time_99 : 959444
> node_put_fsm_time_mean : 156242
> node_put_fsm_time_median : 20235
> vnode_put_fsm_time_100 : 5267527
> vnode_put_fsm_time_95 : 2437457
> vnode_put_fsm_time_99 : 4819538
> vnode_put_fsm_time_mean : 175567
> vnode_put_fsm_time_median : 6928
>
> I am using leveldb, so i can't tune bitcask backend as suggested.
>
> I've changed the vmdirty settings and enabled them:
> admin@riak1:~$ sudo sysctl -a | grep dirtyvm.dirty_background_ratio = 0
> vm.dirty_background_bytes = 209715200
> vm.dirty_ratio = 40
> vm.dirty_bytes = 0
> vm.dirty_writeback_centisecs = 100
> vm.dirty_expire_centisecs = 200
>
> I've seen less idle time between writes, iostat is showing near constant
> writes between 20 and 500 kb/s, with some surges around 4000 kb/s. That's
> better, but not that great.
>
> Here is the current configuration for my "activity_fr" bucket type and
> "tweet" bucket:
>
>
> admin@riak1:~$ http localhost:8098/types/activity_fr/props
> HTTP/1.1 200 OK
> Content-Encoding: gzip
> Content-Length: 314
> Content-Type: application/json
> Date: Tue, 03 May 2016 14:30:21 GMT
> Server: MochiWeb/1.1 WebMachine/1.10.8 (that head fake, tho)
> Vary: Accept-Encoding
> {
>     "props": {
>         "active": true,
>         "allow_mult": false,
>         "basic_quorum": false,
>         "big_vclock": 50,
>         "chash_keyfun": {
>             "fun": "chash_std_keyfun",
>             "mod": "riak_core_util"
>         },
>         "claimant": "r...@riak2.lighthouse-analytics.co",
>         "dvv_enabled": false,
>         "dw": "quorum",
>         "last_write_wins": true,
>         "linkfun": {
>             "fun": "mapreduce_linkfun",
>             "mod": "riak_kv_wm_link_walker"
>         },
>         "n_val": 3,
>         "notfound_ok": true,
>         "old_vclock": 86400,
>         "postcommit": [],
>         "pr": 0,
>         "precommit": [],
>         "pw": 0,
>         "r": "quorum",
>         "rw": "quorum",
>         "search_index": "activity_fr.20160422104506",
>         "small_vclock": 50,
>         "w": "quorum",
>         "young_vclock": 20
>     }
> }
>
> admin@riak1:~$ http localhost:8098/types/activity_fr/buckets/tweet/props
> HTTP/1.1 200 OK
> Content-Encoding: gzip
> Content-Length: 322
> Content-Type: application/json
> Date: Tue, 03 May 2016 14:30:02 GMT
> Server: MochiWeb/1.1 WebMachine/1.10.8 (that head fake, tho)
> Vary: Accept-Encoding
>
> {
>     "props": {
>         "active": true,
>         "allow_mult": false,
>         "basic_quorum": false,
>         "big_vclock": 50,
>         "chash_keyfun": {
>             "fun": "chash_std_keyfun",
>             "mod": "riak_core_util"
>         },
>         "claimant": "r...@riak2.lighthouse-analytics.co",
>         "dvv_enabled": false,
>         "dw": "quorum",
>         "last_write_wins": true,
>         "linkfun": {
>             "fun": "mapreduce_linkfun",
>             "mod": "riak_kv_wm_link_walker"
>         },
>         "n_val": 3,
>         "name": "tweet",
>         "notfound_ok": true,
>         "old_vclock": 86400,
>         "postcommit": [],
>         "pr": 0,
>         "precommit": [],
>         "pw": 0,
>         "r": "quorum",
>         "rw": "quorum",
>         "search_index": "activity_fr.20160422104506",
>         "small_vclock": 50,
>         "w": "quorum",
>         "young_vclock": 20
>     }
> }
>
> I really don't know what to do. Can you help ?
>
> Guillaume

_______________________________________________
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Reply via email to