Hi, Have you seen https://wiki.apache.org/solr/CollapsingQParserPlugin ? May help with the field collapsing queries.
Otis -- Performance Monitoring * Log Analytics * Search Analytics Solr & Elasticsearch Support * http://sematext.com/ On Tue, Jun 3, 2014 at 8:41 AM, Jean-Sebastien Vachon < jean-sebastien.vac...@wantedanalytics.com> wrote: > Hi Otis, > > We saw some improvement when increasing the size of the caches. Since > then, we followed Shawn advice on the filterCache and gave some additional > RAM to the JVM in order to reduce GC. The performance is very good right > now but we are still experiencing some instability but not at the same > level as before. > With our current settings the number of evictions is actually very low so > we might be able to reduce some caches to free up some additional memory > for the JVM to use. > > As for the queries, it is a set of 5 million queries taken from our logs > so they vary a lot. All I can say is that all queries involve either > grouping/field collapsing and/or radius search around a point. Our largest > customer is using a set of 8-10 filters that are translated as fq > parameters. The collection contains around 13 million documents distributed > on 5 shards with 2 replicas. The second collection has the same > configuration and is used for indexing or as a fail-over index in case the > first one falls. > > We`ll keep making adjustments today but we are pretty close of having > something that performs while being stable. > > Thanks all for your help. > > > > > -----Original Message----- > > From: Otis Gospodnetic [mailto:otis.gospodne...@gmail.com] > > Sent: June-03-14 12:17 AM > > To: solr-user@lucene.apache.org > > Subject: Re: Strange behaviour when tuning the caches > > > > Hi Jean-Sebastien, > > > > One thing you didn't mention is whether as you are increasing(I assume) > > cache sizes you actually see performance improve? If not, then maybe > there > > is no value increasing cache sizes. > > > > I assume you changed only one cache at a time? Were you able to get any > > one of them to the point where there were no evictions without things > > breaking? > > > > What are your queries like, can you share a few examples? > > > > Otis > > -- > > Performance Monitoring * Log Analytics * Search Analytics Solr & > > Elasticsearch Support * http://sematext.com/ > > > > > > On Mon, Jun 2, 2014 at 11:09 AM, Jean-Sebastien Vachon < jean- > > sebastien.vac...@wantedanalytics.com> wrote: > > > > > Thanks for your quick response. > > > > > > Our JVM is configured with a heap of 8GB. So we are pretty close of > > > the "optimal" configuration you are mentioning. The only other > > > programs running is Zookeeper (which has its own storage device) and a > > > proprietary API (with a heap of 1GB) we have on top of Solr to server > our > > customer`s requests. > > > > > > I will look into the filterCache to see if we can better use it. > > > > > > Thanks for your help > > > > > > > -----Original Message----- > > > > From: Shawn Heisey [mailto:s...@elyograg.org] > > > > Sent: June-02-14 10:48 AM > > > > To: solr-user@lucene.apache.org > > > > Subject: Re: Strange behaviour when tuning the caches > > > > > > > > On 6/2/2014 8:24 AM, Jean-Sebastien Vachon wrote: > > > > > We have yet to determine where the exact breaking point is. > > > > > > > > > > The two patterns we are seeing are: > > > > > > > > > > - less cache (around 20-30% hit/ratio), poor performance > but > > > > > overall good stability > > > > > > > > When caches are too small, a low hit ratio is expected. Increasing > > > > them > > > is a > > > > good idea, but only increase them a little bit at a time. The > > > filterCache in > > > > particular should not be increased dramatically, especially the > > > > autowarmCount value. Filters can take a very long time to execute, > > > > so a > > > high > > > > autowarmCount can result in commits taking forever. > > > > > > > > Each filter entry can take up a lot of heap memory -- in terms of > > > > bytes, > > > it is > > > > the number of documents in the core divided by 8. This means that > > > > if the core has 10 million documents, each filter entry (for JUST > > > > that > > > > core) will take over a megabyte of RAM. > > > > > > > > > - more cache (over 90% hit/ratio), improved performance > but > > > > > almost no stability. In that case, we start seeing messages such > > > > > as "No shards hosting shard X" or "cancelElection did not find > > > > > election node to remove" > > > > > > > > This would not be a direct result of increasing the cache size, > > > > unless > > > perhaps > > > > you've increased them so they are *REALLY* big and you're running > > > > out of RAM for the heap or OS disk cache. > > > > > > > > > Anyone, has any advice on what could cause this? I am beginning to > > > > > suspect the JVM version, is there any minimal requirements > > > > > regarding the JVM? > > > > > > > > Oracle Java 7 is recommended for all releases, and required for Solr > > > 4.8. You > > > > just need to stay away from 7u40, 7u45, and 7u51 because of bugs in > > > > Java itself. Right now, the latest release is recommended, which is > 7u60. > > > The > > > > 7u21 release that you are running should be perfectly fine. > > > > > > > > With six 9.4GB cores per node, you'll achieve the best performance > > > > if you have about 60GB of RAM left over for the OS disk cache to use > > > > -- the > > > size of > > > > your index data on disk. You did mention that you have 92GB of RAM > > > > per node, but you have not said how big your Java heap is, or > > > > whether there > > > is > > > > other software on the machine that may be eating up RAM for its heap > > > > or data. > > > > > > > > http://wiki.apache.org/solr/SolrPerformanceProblems > > > > > > > > Thanks, > > > > Shawn > > > > > > > > ----- > > > > Aucun virus trouvé dans ce message. > > > > Analyse effectuée par AVG - www.avg.fr > > > > Version: 2014.0.4570 / Base de données virale: 3950/7571 - Date: > > > > 27/05/2014 > > > > > > > ----- > > Aucun virus trouvé dans ce message. > > Analyse effectuée par AVG - www.avg.fr > > Version: 2014.0.4570 / Base de données virale: 3950/7571 - Date: > > 27/05/2014 La Base de données des virus a expiré. >
