Re: How you are dealing with spikes?

2014-12-31 Thread Alexander Popov
Was replaced most 2i and MapReduce calls to SOLR requests, seems not helps to much. Now Solr request has peaks sometimes looking on /stats : what is difference between search_index_latency search_query_latency - search_query_throughput_count: 364711, - search_query_throughput_one: 0, -

Re: How you are dealing with spikes?

2014-12-21 Thread Alexander Popov
Left graph show counts, Right graph show times, graphs are synchronized by time What about SOLR requests instead 2i? should it be faster? Or what you recommend to use for populating lists of users data? for example now we have files, that have 2i like owner, so when user request his files, we po

Re: How you are dealing with spikes?

2014-12-09 Thread Bryan Hunt
Hey Alexander, The times are missing from the leftmost graph (requests). We also don't recommend mapreduce and 2i for user initiated requests. You would have much more stable response times if you precalculated that stuff (just as when using a relational database). mapreduce and 2i are coverage qu

Re: How you are dealing with spikes?

2014-12-09 Thread Alexander Popov
Stats when recent spike happens for 15 minutes around it get (826) save (341) listByIndex (1161) mapReduce (621) //Input is IDs list SOLR (4294) 6 Solr requests was longer than 9sec ( all returns 0 rows ) 4 Solr requests was longer within 4-5s ( both returns 0 rows ) 11 listByIndex re

Re: How you are dealing with spikes?

2014-12-09 Thread Luke Bakken
Hi Alexander, Can you comment on the read vs. write load of this cluster/ Could you please run the following command and reply with the output? sysctl -a | fgrep vm.dirty_ We've seen cases where dirty pages get written in a synchronous manner all at once, causing latency spikes due to I/O block

How you are dealing with spikes?

2014-12-09 Thread Alexander Popov
I have Riak 2.0.1 cluster with 5 nodes ( ec2 m3-large ) with elnm in front sometimes I got spikes up to 10 seconds I can't say that I have huge load at this time, max 200 requests per second for all 5 nodes. Most expensive queries is * list by secondary index ( usually returns from 0 to 100 re