Re: Slow Response for less volume

2018-10-24 Thread Deepak Goel
Are you getting errors in Jmeter?

On Wed, 24 Oct 2018, 21:49 Amjad Khan,  wrote:

> Hi,
>
> We recently moved to Solr Cloud (Google) with 4 nodes and have very
> limited number of data.
>
> We are facing very weird issue here, solr cluster response time for query
> is high when we have less number of hit and the moment we run our test to
> hit the solr cluster hard we see better response in 10ms.
>
> Any clue will be appreciated.
>
> Thanks


Re: Slow Response for less volume

2018-10-24 Thread Walter Underwood
If your cache is 2048 entries, then every one of those 1600 queries is in cache.

Our logs typically have about a million lines, with distinct queries 
distributed according to the Zipf law. Some common queries, a long tail, that 
sort of thing.

wunder
Walter Underwood
wun...@wunderwood.org
http://observer.wunderwood.org/  (my blog)

> On Oct 24, 2018, at 10:02 AM, Amjad Khan  wrote:
> 
> Thanks Wunder for this prompt response.
> 
> We are testing with 1600 different text to search with Jmeter and that keeps 
> running continuously, and keep running continuously means cache has been 
> built and there should be better response now. Doesn’t it?
> 
> Thanks
> 
> 
> 
>> On Oct 24, 2018, at 12:20 PM, Walter Underwood  wrote:
>> 
>> Are you testing with a small number of queries? If your cache is larger than 
>> the number of queries in your benchmark, the first round will load the 
>> cache, then everything will be super fast.
>> 
>> Load testing a system with caches is hard.
>> 
>> wunder
>> Walter Underwood
>> wun...@wunderwood.org
>> http://observer.wunderwood.org/  (my blog)
>> 
>>> On Oct 24, 2018, at 9:19 AM, Amjad Khan  wrote:
>>> 
>>> Hi,
>>> 
>>> We recently moved to Solr Cloud (Google) with 4 nodes and have very limited 
>>> number of data.
>>> 
>>> We are facing very weird issue here, solr cluster response time for query 
>>> is high when we have less number of hit and the moment we run our test to 
>>> hit the solr cluster hard we see better response in 10ms.
>>> 
>>> Any clue will be appreciated.
>>> 
>>> Thanks
>> 
> 



Re: Slow Response for less volume

2018-10-24 Thread Walter Underwood
But a zero size cache doesn’t give realistic benchmarks. It makes things slower 
than they will be in production.

We do this:

1. Collect production logs.
2. Split the logs into a warming log and and a benchmark log. The warming log 
should be at least as large as the query result cache.
3. Run the warming log with four threads (unlikely to overload the system).
4. Run the benchmark with a controlled requests/minute and enough threads to 
keep up with that. Might be a few hundred with a large, slow cluster. Run for 
at least an hour.
5. Analyze the results into percentile response times for each request handler. 
Warn about any errors or a benchmark that takes too long.

Then reepeat. Oh, yeah, load the prod content first.

wunder
Walter Underwood
wun...@wunderwood.org
http://observer.wunderwood.org/  (my blog)

> On Oct 24, 2018, at 9:52 AM, Erick Erickson  wrote:
> 
> You can set your queryResultCache and filterCache "size" parameter to
> zero in solrconfig.xml to disable those caches.
> On Wed, Oct 24, 2018 at 9:21 AM Walter Underwood  
> wrote:
>> 
>> Are you testing with a small number of queries? If your cache is larger than 
>> the number of queries in your benchmark, the first round will load the 
>> cache, then everything will be super fast.
>> 
>> Load testing a system with caches is hard.
>> 
>> wunder
>> Walter Underwood
>> wun...@wunderwood.org
>> http://observer.wunderwood.org/  (my blog)
>> 
>>> On Oct 24, 2018, at 9:19 AM, Amjad Khan  wrote:
>>> 
>>> Hi,
>>> 
>>> We recently moved to Solr Cloud (Google) with 4 nodes and have very limited 
>>> number of data.
>>> 
>>> We are facing very weird issue here, solr cluster response time for query 
>>> is high when we have less number of hit and the moment we run our test to 
>>> hit the solr cluster hard we see better response in 10ms.
>>> 
>>> Any clue will be appreciated.
>>> 
>>> Thanks
>> 



Re: Slow Response for less volume

2018-10-24 Thread Amjad Khan
Thanks Erick,

But do you think that disabling the cache will increase the response time 
instead of solving the problem here.


> On Oct 24, 2018, at 12:52 PM, Erick Erickson  wrote:
> 
> queryResultCache



Re: Slow Response for less volume

2018-10-24 Thread Amjad Khan
Thanks Wunder for this prompt response.

We are testing with 1600 different text to search with Jmeter and that keeps 
running continuously, and keep running continuously means cache has been built 
and there should be better response now. Doesn’t it?

Thanks



> On Oct 24, 2018, at 12:20 PM, Walter Underwood  wrote:
> 
> Are you testing with a small number of queries? If your cache is larger than 
> the number of queries in your benchmark, the first round will load the cache, 
> then everything will be super fast.
> 
> Load testing a system with caches is hard.
> 
> wunder
> Walter Underwood
> wun...@wunderwood.org
> http://observer.wunderwood.org/  (my blog)
> 
>> On Oct 24, 2018, at 9:19 AM, Amjad Khan  wrote:
>> 
>> Hi,
>> 
>> We recently moved to Solr Cloud (Google) with 4 nodes and have very limited 
>> number of data.
>> 
>> We are facing very weird issue here, solr cluster response time for query is 
>> high when we have less number of hit and the moment we run our test to hit 
>> the solr cluster hard we see better response in 10ms.
>> 
>> Any clue will be appreciated.
>> 
>> Thanks
> 



Re: Slow Response for less volume

2018-10-24 Thread Erick Erickson
You can set your queryResultCache and filterCache "size" parameter to
zero in solrconfig.xml to disable those caches.
On Wed, Oct 24, 2018 at 9:21 AM Walter Underwood  wrote:
>
> Are you testing with a small number of queries? If your cache is larger than 
> the number of queries in your benchmark, the first round will load the cache, 
> then everything will be super fast.
>
> Load testing a system with caches is hard.
>
> wunder
> Walter Underwood
> wun...@wunderwood.org
> http://observer.wunderwood.org/  (my blog)
>
> > On Oct 24, 2018, at 9:19 AM, Amjad Khan  wrote:
> >
> > Hi,
> >
> > We recently moved to Solr Cloud (Google) with 4 nodes and have very limited 
> > number of data.
> >
> > We are facing very weird issue here, solr cluster response time for query 
> > is high when we have less number of hit and the moment we run our test to 
> > hit the solr cluster hard we see better response in 10ms.
> >
> > Any clue will be appreciated.
> >
> > Thanks
>


Re: Slow Response for less volume

2018-10-24 Thread Walter Underwood
Are you testing with a small number of queries? If your cache is larger than 
the number of queries in your benchmark, the first round will load the cache, 
then everything will be super fast.

Load testing a system with caches is hard.

wunder
Walter Underwood
wun...@wunderwood.org
http://observer.wunderwood.org/  (my blog)

> On Oct 24, 2018, at 9:19 AM, Amjad Khan  wrote:
> 
> Hi,
> 
> We recently moved to Solr Cloud (Google) with 4 nodes and have very limited 
> number of data.
> 
> We are facing very weird issue here, solr cluster response time for query is 
> high when we have less number of hit and the moment we run our test to hit 
> the solr cluster hard we see better response in 10ms.
> 
> Any clue will be appreciated.
> 
> Thanks



Slow Response for less volume

2018-10-24 Thread Amjad Khan
Hi,

We recently moved to Solr Cloud (Google) with 4 nodes and have very limited 
number of data.

We are facing very weird issue here, solr cluster response time for query is 
high when we have less number of hit and the moment we run our test to hit the 
solr cluster hard we see better response in 10ms.

Any clue will be appreciated.

Thanks