What are your cache sizes? Max doc? Also, what GC settings are you using? 6GB isn't all that much for a memory-intensive app like Solr, esp. given the number of facet fields you have. Lastly, are you using docvalues for your facet fields? That should help reduce the amount of heap needed to compute facets.
On Tue, Jul 14, 2015 at 2:33 PM, Yael Gurevich <yae...@gmail.com> wrote: > Hi, > > We're running Solr 4.10.1 on Linux using Tomcat. Distributed environment, > 40 virtual servers with high resources. Concurrent queries that are quite > complex (may be hundreds of terms), NRT indexing and a few hundreds of > facet fields which might have many (hundreds of thousands) distinct values. > > We've configured a 6GB JVM heap, and after quite a bit of work, it seems to > be pretty well configured GC parameter-wise (we're using CMS and ParNew). > > The following problem occurs - > Once every couple of hours, suddenly start getting > "concurrent-mode-failure" on one or more servers, the memory starts > climbing up further and further and "concurrent-mode-failure" continues. > Naturally, during this time, SOLR is unresponsive and the queries are > timed-out. Eventually it might pass (GC will succeed), after 5-10 minutes. > Sometimes this phenomenon can occur for a great deal of time, one server > goes up and then another and so forth. > > Memory dumps point to ConcurrentLRUCache (used in filterCache and > fieldValueCache). Mathematically speaking, the sizes I see in the dumps do > not make sense. The configured sizes shouldn't take up more than a few > hunderds of MBs. > > Any ideas? Anyone seen this kind of problem?
