Re: Very slow acquisition time (99 percentile) while fast median times

2016-05-03 Thread Matthew Von-Maszewski
Guillaume, A prebuilt eleveldb 2.0.18 for Debian 7 is found here: https://s3.amazonaws.com/downloads.basho.com/patches/eleveldb/2.0.18/eleveldb_2.0.18_debian7.tgz There are good instructions

Re: Very slow acquisition time (99 percentile) while fast median times

2016-05-03 Thread Fred Dushin
Hi Guillaume, From your bucket properties it looks like you are using search, and I assume that is search 2.0 (i.e., yokozuna), and not the legacy Riak Search. It is true that in the current 2.0 and 2.1 trunks the indexing into Solr via Yokozuna is synchronous with the vnode -- very long times

Re: Very slow acquisition time (99 percentile) while fast median times

2016-05-03 Thread Matthew Von-Maszewski
Guillaume, I have reviewed the debug package for your riak1 server. There are two potential areas of follow-up: 1. You are running our most recent Riak 2.1.4 which has eleveldb 2.0.17. We have seen a case where a recent feature in eleveldb 2.0.17 caused too much cache flushing, impacting

Re: Very slow acquisition time (99 percentile) while fast median times

2016-05-03 Thread Luke Bakken
Guillaume - You said earlier "My data are stored on an openstack volume that support up to 3000IOPS". There is a likelihood that your write load is exceeding the capacity of your virtual environment, especially if some Riak nodes are sharing physical disk or server infrastructure. Some

Re: Very slow acquisition time (99 percentile) while fast median times

2016-05-03 Thread Guillaume Boddaert
Hi, Sorry for the delay, I've spent a lot of time trying to understand if the problem was elsewhere. I've simplified my infrastructure and got a simple layout that don't rely anymore on loadbalancer and also corrected some minor performance issue on my workers. At the moment, i have up to