Solr hangs / LRU operations are heavy on cpu

[Sergey Shvets]

Hi,

we have quite a problem with Solr. We are running it in a config 6x3, and
suddenly solr started to hang, taking all the available cpu on the nodes.

In the threads dump noticed things like this can eat lot of CPU time


   - org.apache.solr.search.LRUCache.put​(LRUCache.java:116)
   -
   org.apache.solr.search.SolrIndexSearcher.doc​(SolrIndexSearcher.java:705)
   -
   
org.apache.solr.response.BinaryResponseWriter$Resolver.writeResultsBody​(BinaryResponseWriter.java:155)
   -
   
org.apache.solr.response.BinaryResponseWriter$Resolver.writeResults​(BinaryResponseWriter.java:183)
   -
   
org.apache.solr.response.BinaryResponseWriter$Resolver.resolve​(BinaryResponseWriter.java:88)
   -
   org.apache.solr.common.util.JavaBinCodec.writeVal​(JavaBinCodec.java:158)
   -
   
org.apache.solr.common.util.JavaBinCodec.writeNamedList​(JavaBinCodec.java:148)
   -
   
org.apache.solr.common.util.JavaBinCodec.writeKnownType​(JavaBinCodec.java:242)
   -
   org.apache.solr.common.util.JavaBinCodec.writeVal​(JavaBinCodec.java:153)
   - org.apache.solr.common.util.JavaBinCodec.marshal​(JavaBinCodec.java:96)
   -
   
org.apache.solr.response.BinaryResponseWriter.write​(BinaryResponseWriter.java:52)
   -
   
org.apache.solr.servlet.SolrDispatchFilter.writeResponse​(SolrDispatchFilter.java:758)
   -
   
org.apache.solr.servlet.SolrDispatchFilter.doFilter​(SolrDispatchFilter.java:426)
   -
   
org.apache.solr.servlet.SolrDispatchFilter.doFilter​(SolrDispatchFilter.java:207)
   -
   
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter​(ApplicationFilterChain.java:241)
   -
   
org.apache.catalina.core.ApplicationFilterChain.doFilter​(ApplicationFilterChain.java:208)
   -
   
org.apache.catalina.core.StandardWrapperValve.invoke​(StandardWrapperValve.java:220)
   -
   
org.apache.catalina.core.StandardContextValve.invoke​(StandardContextValve.java:122)
   -
   
org.apache.catalina.core.StandardHostValve.invoke​(StandardHostValve.java:170)
   -
   
org.apache.catalina.valves.ErrorReportValve.invoke​(ErrorReportValve.java:103)
   -
   org.apache.catalina.valves.AccessLogValve.invoke​(AccessLogValve.java:950)
   -
   
org.apache.catalina.core.StandardEngineValve.invoke​(StandardEngineValve.java:116)


The cache itself is very minimalistic


  <filterCache class="solr.FastLRUCache" size="512" initialSize="512"
autowarmCount="0"/>
    <queryResultCache class="solr.LRUCache" size="512"
initialSize="512" autowarmCount="0"/>
    <documentCache class="solr.LRUCache" size="512" initialSize="512"
autowarmCount="0"/>
    <fieldValueCache class="solr.FastLRUCache" size="1024"
autowarmCount="256" showItems="10" />
    <cache name="perSegFilter" class="solr.search.LRUCache" size="10"
initialSize="0" autowarmCount="10"
regenerator="solr.NoOpRegenerator"/>
    <enableLazyFieldLoading>true</enableLazyFieldLoading>
    <queryResultWindowSize>20</queryResultWindowSize>
    <queryResultMaxDocsCached>200</queryResultMaxDocsCached>

Solr version is 4.10.3

Any of help is appreciated!

sergey