Hello list,
in the last couple of weeks one of my machines is experiencing
OutOfMemoryError: Java heap space errors. In a couple of hours after
starting the SOLR instance queries with execution times of unter 100ms
need more than 10s to execute and many Java heap space erros appear in
the logs. I would be grateful for any hints, how to localize/fix the
problem.
The system is a single shard SOLR 4.6.1 instance with two cores in the
default jetty container. The CORE1 has ~45M documents (disk size of
32GB) with 40 fields (all stored and indexed). CORE2 has ~20M documents
(disk size of 18GB) with 60 fields (both stored and indexed also). All
the fields are relatively short with a maximal length of 100 characters.
In both cores 20 of the fields are used for faceting, have a cache
populating query on "newSearcher" event and the following cache
configurations:
CORE1
<filterCache class="solr.FastLRUCache" size="128" initialSize="0"
autowarmCount="0"/>
<queryResultCache class="solr.LRUCache" size="128" initialSize="0"
autowarmCount="0"/>
<fieldValueCache class="solr.FastLRUCache" size="128" autowarmCount="0"
showItems="0" />
CORE2
<filterCache class="solr.FastLRUCache" size="32" initialSize="0"
autowarmCount="0"/>
<fieldValueCache class="solr.FastLRUCache" size="32" autowarmCount="0"
showItems="0" />
The OS in use is a 64bit linux with an OpenJDK 1.7 Java with 48G RAM.
The JVM gets the following parameters to start the jetty container:
-XX:+UseCompressedOops
-XX:+UseCompressedStrings
-XX:+OptimizeStringConcat
-XX:+UseStringCache
-XX:+UseCMSInitiatingOccupancyOnly
-XX:CMSInitiatingOccupancyFraction=75
-XX:MaxTenuringThreshold=1
-XX:SurvivorRatio=8
-XX:+CMSParallelRemarkEnabled
-XX:+UseConcMarkSweepGC
-XX:+UseParNewGC
Cheers
Angel
P.S.: Here the complete error OutOfMemoryError message:
org.apache.solr.common.SolrException; null:java.lang.RuntimeException:
java.lang.OutOfMemoryError: Java heap space
at
org.apache.solr.servlet.SolrDispatchFilter.sendError(SolrDispatchFilter.java:735)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:438)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:201)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
at org.eclipse.jetty.server.Server.handle(Server.java:368)
at
org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
at
org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
at
org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:953)
at
org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1014)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:861)
at
org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
at
org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
at
org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:724)
Caused by: java.lang.OutOfMemoryError: Java heap space
