What are your cache sizes? Max doc?
Also, what GC settings are you using? 6GB isn't all that much for a
memory-intensive app like Solr, esp. given the number of facet fields
you have. Lastly, are you using docvalues for your facet fields? That
should help reduce the amount of heap needed to compute facets.
On Tue, Jul 14, 2015 at 2:33 PM, Yael Gurevich yae...@gmail.com wrote:
Hi,
We're running Solr 4.10.1 on Linux using Tomcat. Distributed environment,
40 virtual servers with high resources. Concurrent queries that are quite
complex (may be hundreds of terms), NRT indexing and a few hundreds of
facet fields which might have many (hundreds of thousands) distinct values.
We've configured a 6GB JVM heap, and after quite a bit of work, it seems to
be pretty well configured GC parameter-wise (we're using CMS and ParNew).
The following problem occurs -
Once every couple of hours, suddenly start getting
concurrent-mode-failure on one or more servers, the memory starts
climbing up further and further and concurrent-mode-failure continues.
Naturally, during this time, SOLR is unresponsive and the queries are
timed-out. Eventually it might pass (GC will succeed), after 5-10 minutes.
Sometimes this phenomenon can occur for a great deal of time, one server
goes up and then another and so forth.
Memory dumps point to ConcurrentLRUCache (used in filterCache and
fieldValueCache). Mathematically speaking, the sizes I see in the dumps do
not make sense. The configured sizes shouldn't take up more than a few
hunderds of MBs.
Any ideas? Anyone seen this kind of problem?
