Guillaume,
A prebuilt eleveldb 2.0.18 for Debian 7 is found here:
https://s3.amazonaws.com/downloads.basho.com/patches/eleveldb/2.0.18/eleveldb_2.0.18_debian7.tgz
<https://s3.amazonaws.com/downloads.basho.com/patches/eleveldb/2.0.18/eleveldb_2.0.18_debian7.tgz>
There are good instructions for applying an eleveldb patch here:
http://docs.basho.com/community/productadvisories/leveldbsegfault/#patch-eleveldb-so
Key points about the above web page:
- use the eleveldb patch file link in this email, NOT links on the web page
- the Debian directory listed on the web page will be slightly different than
your Riak 2.1.4 installation:
/usr/lib/riak/lib/eleveldb-<something_different>/priv/
Matthew
> On May 3, 2016, at 1:01 PM, Matthew Von-Maszewski <matth...@basho.com> wrote:
>
> Guillaume,
>
> I have reviewed the debug package for your riak1 server. There are two
> potential areas of follow-up:
>
> 1. You are running our most recent Riak 2.1.4 which has eleveldb 2.0.17. We
> have seen a case where a recent feature in eleveldb 2.0.17 caused too much
> cache flushing, impacting leveldb’s performance. A discussion is here:
>
> https://github.com/basho/leveldb/wiki/mv-timed-grooming2
> <https://github.com/basho/leveldb/wiki/mv-timed-grooming2>
>
> 2. Yokozuna search was recently updated for some timeout problems. Those
> updates are not yet in a public build. One of our other engineers is likely
> to respond to you on that topic.
>
>
> An eleveldb 2.0.18 is tagged and available via github if you want to build it
> yourself. Otherwise, Basho may be releasing prebuilt patches of eleveldb
> 2.0.18 in the near future. But no date is currently set.
>
> Matthew
>
>> On May 3, 2016, at 10:50 AM, Luke Bakken <lbak...@basho.com
>> <mailto:lbak...@basho.com>> wrote:
>>
>> Guillaume -
>>
>> You said earlier "My data are stored on an openstack volume that
>> support up to 3000IOPS". There is a likelihood that your write load is
>> exceeding the capacity of your virtual environment, especially if some
>> Riak nodes are sharing physical disk or server infrastructure.
>>
>> Some suggestions:
>>
>> * If you're not using Riak Search, set "search = off" in riak.conf
>>
>> * Be sure to carefully read and apply all tunings:
>> http://docs.basho.com/riak/kv/2.1.4/using/performance/
>> <http://docs.basho.com/riak/kv/2.1.4/using/performance/>
>>
>> * You may wish to increase the memory dedicated to leveldb:
>> http://docs.basho.com/riak/kv/2.1.4/configuring/backend/#leveldb
>>
>> --
>> Luke Bakken
>> Engineer
>> lbak...@basho.com
>>
>>
>> On Tue, May 3, 2016 at 7:33 AM, Guillaume Boddaert
>> <guilla...@lighthouse-analytics.co> wrote:
>>> Hi,
>>>
>>> Sorry for the delay, I've spent a lot of time trying to understand if the
>>> problem was elsewhere.
>>> I've simplified my infrastructure and got a simple layout that don't rely
>>> anymore on loadbalancer and also corrected some minor performance issue on
>>> my workers.
>>>
>>> At the moment, i have up to 32 workers that are calling riak for writes,
>>> each of them are set to :
>>> w=1
>>> dw=0
>>> timeout=1000
>>> using protobuf
>>> a timeouted attempt is rerun 180s later
>>>
>>> From my application server perspective, 23% of the calls are rejected by
>>> timeout (75446 tries, 57564 success, 17578 timeout).
>>>
>>> Here is a sample riak-admin stat for one of my 5 hosts:
>>>
>>> node_put_fsm_time_100 : 999331
>>> node_put_fsm_time_95 : 773682
>>> node_put_fsm_time_99 : 959444
>>> node_put_fsm_time_mean : 156242
>>> node_put_fsm_time_median : 20235
>>> vnode_put_fsm_time_100 : 5267527
>>> vnode_put_fsm_time_95 : 2437457
>>> vnode_put_fsm_time_99 : 4819538
>>> vnode_put_fsm_time_mean : 175567
>>> vnode_put_fsm_time_median : 6928
>>>
>>> I am using leveldb, so i can't tune bitcask backend as suggested.
>>>
>>> I've changed the vmdirty settings and enabled them:
>>> admin@riak1:~$ sudo sysctl -a | grep dirtyvm.dirty_background_ratio = 0
>>> vm.dirty_background_bytes = 209715200
>>> vm.dirty_ratio = 40
>>> vm.dirty_bytes = 0
>>> vm.dirty_writeback_centisecs = 100
>>> vm.dirty_expire_centisecs = 200
>>>
>>> I've seen less idle time between writes, iostat is showing near constant
>>> writes between 20 and 500 kb/s, with some surges around 4000 kb/s. That's
>>> better, but not that great.
>>>
>>> Here is the current configuration for my "activity_fr" bucket type and
>>> "tweet" bucket:
>>>
>>>
>>> admin@riak1:~$ http localhost:8098/types/activity_fr/props
>>> HTTP/1.1 200 OK
>>> Content-Encoding: gzip
>>> Content-Length: 314
>>> Content-Type: application/json
>>> Date: Tue, 03 May 2016 14:30:21 GMT
>>> Server: MochiWeb/1.1 WebMachine/1.10.8 (that head fake, tho)
>>> Vary: Accept-Encoding
>>> {
>>> "props": {
>>> "active": true,
>>> "allow_mult": false,
>>> "basic_quorum": false,
>>> "big_vclock": 50,
>>> "chash_keyfun": {
>>> "fun": "chash_std_keyfun",
>>> "mod": "riak_core_util"
>>> },
>>> "claimant": "r...@riak2.lighthouse-analytics.co",
>>> "dvv_enabled": false,
>>> "dw": "quorum",
>>> "last_write_wins": true,
>>> "linkfun": {
>>> "fun": "mapreduce_linkfun",
>>> "mod": "riak_kv_wm_link_walker"
>>> },
>>> "n_val": 3,
>>> "notfound_ok": true,
>>> "old_vclock": 86400,
>>> "postcommit": [],
>>> "pr": 0,
>>> "precommit": [],
>>> "pw": 0,
>>> "r": "quorum",
>>> "rw": "quorum",
>>> "search_index": "activity_fr.20160422104506",
>>> "small_vclock": 50,
>>> "w": "quorum",
>>> "young_vclock": 20
>>> }
>>> }
>>>
>>> admin@riak1:~$ http localhost:8098/types/activity_fr/buckets/tweet/props
>>> HTTP/1.1 200 OK
>>> Content-Encoding: gzip
>>> Content-Length: 322
>>> Content-Type: application/json
>>> Date: Tue, 03 May 2016 14:30:02 GMT
>>> Server: MochiWeb/1.1 WebMachine/1.10.8 (that head fake, tho)
>>> Vary: Accept-Encoding
>>>
>>> {
>>> "props": {
>>> "active": true,
>>> "allow_mult": false,
>>> "basic_quorum": false,
>>> "big_vclock": 50,
>>> "chash_keyfun": {
>>> "fun": "chash_std_keyfun",
>>> "mod": "riak_core_util"
>>> },
>>> "claimant": "r...@riak2.lighthouse-analytics.co",
>>> "dvv_enabled": false,
>>> "dw": "quorum",
>>> "last_write_wins": true,
>>> "linkfun": {
>>> "fun": "mapreduce_linkfun",
>>> "mod": "riak_kv_wm_link_walker"
>>> },
>>> "n_val": 3,
>>> "name": "tweet",
>>> "notfound_ok": true,
>>> "old_vclock": 86400,
>>> "postcommit": [],
>>> "pr": 0,
>>> "precommit": [],
>>> "pw": 0,
>>> "r": "quorum",
>>> "rw": "quorum",
>>> "search_index": "activity_fr.20160422104506",
>>> "small_vclock": 50,
>>> "w": "quorum",
>>> "young_vclock": 20
>>> }
>>> }
>>>
>>> I really don't know what to do. Can you help ?
>>>
>>> Guillaume
>>
>> _______________________________________________
>> riak-users mailing list
>> riak-users@lists.basho.com
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
_______________________________________________
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
