Re: SOLR query times
: If the latency is caused by loading and caching of Lucene's segments, : is there a way to force Lucene's index to preload this? This seems to configure some static queries in newSearcher and firstSearcher listeners ... it doesn't really matter what the queries are as long as they force some access to the new index ... but if you do any sorting you should definitely make sure that is represented in the queries as well. -Hoss
Re: SOLR query times
Hi Grant, Thanks for your response. I'm trying to simulate our production environment's search traffic which has very low cache hit rate. Turning off the caches can help us better understand query times and the load of the slave's when distribution occurs with a small list of pre-canned queries. If the latency is caused by loading and caching of Lucene's segments, is there a way to force Lucene's index to preload this? This seems to be the case in our production environment, when SOLR restarts the load spikes and it takes a couple of hours before it settles down. Also, are there general acceptable ways of doing scalability and performance characterization? Thanks, Sammy. On Sun, Oct 12, 2008 at 8:17 AM, Grant Ingersoll [EMAIL PROTECTED] wrote: This is pretty typical. The first query is always more expensive, as Lucene lazily loads some pieces of the index into memory and you may see the FieldCache in action, depending on sorting, not to mention you are also seeing operating system caching take place. Is there some reason you don't want these or are you just trying to understand the why? -Grant On Oct 10, 2008, at 6:25 PM, Sammy Yu wrote: Hi, I'm using SOLR 1.3 on a index with approximately 8 million documents. I would like to disable SOLR's cache so that it is easier for me to test the scenario when there is a small likelihood of cache hits. I've disabled caching by commenting out the filterCache, queryResultCache, and documentCache section in solrconfig.xml as suggested by the Wiki. It seems disabled because the admin interface no longer shows any entries in the Cache section. However, it appears that there is still some sort caching taking place. The first time I make specific query it would take around 100 msec, subsequent queries would take around 15 msec. Is there some sort of caching happening at Lucene level? Thanks for your help, Sammy Yu -- Grant Ingersoll Lucene Boot Camp Training Nov. 3-4, 2008, ApacheCon US New Orleans. http://www.lucenebootcamp.com Lucene Helpful Hints: http://wiki.apache.org/lucene-java/BasicsOfPerformance http://wiki.apache.org/lucene-java/LuceneFAQ
Re: SOLR query times
This is pretty typical. The first query is always more expensive, as Lucene lazily loads some pieces of the index into memory and you may see the FieldCache in action, depending on sorting, not to mention you are also seeing operating system caching take place. Is there some reason you don't want these or are you just trying to understand the why? -Grant On Oct 10, 2008, at 6:25 PM, Sammy Yu wrote: Hi, I'm using SOLR 1.3 on a index with approximately 8 million documents. I would like to disable SOLR's cache so that it is easier for me to test the scenario when there is a small likelihood of cache hits. I've disabled caching by commenting out the filterCache, queryResultCache, and documentCache section in solrconfig.xml as suggested by the Wiki. It seems disabled because the admin interface no longer shows any entries in the Cache section. However, it appears that there is still some sort caching taking place. The first time I make specific query it would take around 100 msec, subsequent queries would take around 15 msec. Is there some sort of caching happening at Lucene level? Thanks for your help, Sammy Yu -- Grant Ingersoll Lucene Boot Camp Training Nov. 3-4, 2008, ApacheCon US New Orleans. http://www.lucenebootcamp.com Lucene Helpful Hints: http://wiki.apache.org/lucene-java/BasicsOfPerformance http://wiki.apache.org/lucene-java/LuceneFAQ
SOLR query times
Hi, I'm using SOLR 1.3 on a index with approximately 8 million documents. I would like to disable SOLR's cache so that it is easier for me to test the scenario when there is a small likelihood of cache hits. I've disabled caching by commenting out the filterCache, queryResultCache, and documentCache section in solrconfig.xml as suggested by the Wiki. It seems disabled because the admin interface no longer shows any entries in the Cache section. However, it appears that there is still some sort caching taking place. The first time I make specific query it would take around 100 msec, subsequent queries would take around 15 msec. Is there some sort of caching happening at Lucene level? Thanks for your help, Sammy Yu
