> Can you give more info? How much RAM are you giving Solr with Xmx?
RHEL 5.4
Java(TM) SE Runtime Environment (build 1.6.0_24-b07)
Java HotSpot(TM) 64-Bit Server VM (build 19.1-b02, mixed mode)
java -Xmx16384m -Xms8192m
SOLR shards=6, each with a replica (i.e. total 12 JVMs)
Solrj multi-threaded client sends several 1,000 docs/sec with auto commit or
commit within
Same client with explicit client commit tops out around 1,000 / sec
> How often are you trying to commit? With the client? With auto commit?
With auto commit or commit within, I've tried things in the range of
10,000-60,000 ms. I have tried both with and without soft commit. When I
tried soft, it was in the range of 500-5,000 ms. With explicit client commit I
have tried in the range of 10,000 to 200,000 documents.
>Can you be more specific about the behavior you are seeing with auto commit vs
>client commit?
With auto commit, the client threads run without pausing and quickly ramp up to
several 1,000s docs per second. After a variable number of minutes, but seldom
as long as an hour, the server(s) to which the client(s) are attached get
exceptions. I have included one stack trace below. Clients continue to run
without error.
With explicit client commit, the client will pause for a few seconds at the
commit with a single thread and hit overall index rates around 500/sec. With a
second thread, the commit is around 5 seconds and overall rate is around
1,000/sec. Adding a third thread causes the commit time to increase to a
minute or more. Overall rate increases only slightly with the third and
subsequent threads.
> Anything else you can add?
I have seen this with a configuration as small as a single shard and a replica,
but I've been always working with SolrCloud. When there is a single shard
without a replica, I have not seen the problem.
INFO: [shipment] webapp=/solr path=/update
params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2
Jul 14, 2012 9:20:57 PM org.apache.solr.core.SolrCore execute
INFO: [shipment] webapp=/solr path=/update
params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3
Jul 14, 2012 9:20:57 PM org.apache.solr.common.SolrException log
SEVERE: null:java.lang.RuntimeException: java.lang.OutOfMemoryError: unable to
create new native thread
at
org.apache.solr.servlet.SolrDispatchFilter.sendError(SolrDispatchFilter.java:456)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:284)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1337)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:484)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:119)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:233)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1065)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:413)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:192)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:999)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:250)
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:149)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111)
at org.eclipse.jetty.server.Server.handle(Server.java:351)
at
org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:454)
at
org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:47)
at
org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:900)
at
org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:954)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:952)
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
at
org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:66)
at
org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:254)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:640)
at
java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(ThreadPoolExecutor.java:727)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:657)
at
java.util.concurrent.ExecutorCompletionService.submit(ExecutorCompletionService.java:152)
at
org.apache.solr.update.SolrCmdDistributor.submit(SolrCmdDistributor.java:340)
at
org.apache.solr.update.SolrCmdDistributor.submit(SolrCmdDistributor.java:296)
at
org.apache.solr.update.SolrCmdDistributor.flushAdds(SolrCmdDistributor.java:228)
at
org.apache.solr.update.SolrCmdDistributor.finish(SolrCmdDistributor.java:101)
at
org.apache.solr.update.processor.DistributedUpdateProcessor.doFinish(DistributedUpdateProcessor.java:329)
at
org.apache.solr.update.processor.DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:952)
at
org.apache.solr.update.processor.LogUpdateProcessor.finish(LogUpdateProcessorFactory.java:176)
at
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:83)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1561)
at
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:442)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:263)
... 25 more
Jul 14, 2012 9:20:57 PM org.apache.solr.common.SolrException log
-----Original Message-----
From: Mark Miller [mailto:markrmil...@gmail.com]
Sent: Saturday, July 14, 2012 2:44 PM
To: solr-user@lucene.apache.org
Subject: Re: SOLR 4 Alpha Out Of Mem Err
Can you give more info? How much RAM are you giving Solr with Xmx?
Can you be more specific about the behavior you are seeing with auto commit vs
client commit?
How often are you trying to commit? With the client? With auto commit?
Are you doing soft commits? Std commits? A mix?
What's the stack trace for the OOM?
What OS are you using?
Anything else you can add?
--
Mark Miller
On Saturday, July 14, 2012 at 4:21 PM, Nick Koton wrote:
> I have been experiencing out of memory errors when indexing via solrj into a
> 4 alpha cluster. It seems when I delegate commits to the server (either
> auto commit or commit within) there is nothing to throttle the solrj clients
> and the server struggles to fan out the work. However, when I handle
> commits entirely within the client, the indexing rate is very restricted.
>
> Any suggestions would be appreciated
>
> Nick Cotton
> nick.ko...@gmail.com
>
>
