After trying a number of things, I am successful in allowing the server to auto commit and without having it hit thread/memory errors. I have isolated the required client change to replacing ConcurrentUpdateSolrServer with HttpSolrServer. I am able to maintain index rates of 3,000 documents/sec with 6 shards and two servers per shard. The servers receiving the index requests hit steady state with approximately 800 threads per server.
So could there be something amiss in the server side implementation of ConcurrentUpdateSolrServer? Best regards, Nick -----Original Message----- From: Nick Koton [mailto:nick.ko...@gmail.com] Sent: Monday, July 16, 2012 5:53 PM To: 'solr-user@lucene.apache.org' Subject: RE: SOLR 4 Alpha Out Of Mem Err > That suggests you're running out of threads Michael, Thanks for this useful observation. What I found just prior to the "problem situation" was literally thousands of threads in the server JVM. I have pasted a few samples below obtained from the admin GUI. I spent some time today using this barometer, but I don't have enough to share right now. I'm looking at the difference between ConcurrentUpdateSolrServer and HttpSolrServer and how my client may be misusing them. I'll assume my client is misbehaving and driving the server crazy for now. If I figure out how, I will share it so perhaps a safe guard can be put in place. Nick Server threads - very roughly 0.1 %: cmdDistribExecutor-9-thread-7161 (10096) java.util.concurrent.SynchronousQueue$TransferStack@17b90c55 . sun.misc.Unsafe.park(Native Method) . java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) . java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Synchronous Queue.java:424) . java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueu e.java:323) . java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874) . java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945) . java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9 07) . java.lang.Thread.run(Thread.java:662) -0.0000ms -0.0000ms cmdDistribExecutor-9-thread-7160 (10086) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5509b5 6 . sun.misc.Unsafe.park(Native Method) . java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) . java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await( AbstractQueuedSynchronizer.java:1987) . org.apache.http.impl.conn.tsccm.WaitingThread.await(WaitingThread.java:158) . org.apache.http.impl.conn.tsccm.ConnPoolByRoute.getEntryBlocking(ConnPoolByR oute.java:403) . org.apache.http.impl.conn.tsccm.ConnPoolByRoute$1.getPoolEntry(ConnPoolByRou te.java:300) . org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager$1.getConnection( ThreadSafeClientConnManager.java:224) . org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDir ector.java:401) . org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.ja va:820) . org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.ja va:754) . org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.ja va:732) . org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java :351) . org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java :182) . org.apache.solr.update.SolrCmdDistributor$1.call(SolrCmdDistributor.java:325 ) . org.apache.solr.update.SolrCmdDistributor$1.call(SolrCmdDistributor.java:306 ) . java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) . java.util.concurrent.FutureTask.run(FutureTask.java:138) . java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) . java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) . java.util.concurrent.FutureTask.run(FutureTask.java:138) . java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja va:886) . java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9 08) . java.lang.Thread.run(Thread.java:662) 20.0000ms 20.0000ms cmdDistribExecutor-9-thread-7159 (10085) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6f062d d3 . sun.misc.Unsafe.park(Native Method) . java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) . java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await( AbstractQueuedSynchronizer.java:1987) . org.apache.http.impl.conn.tsccm.WaitingThread.await(WaitingThread.java:158) . org.apache.http.impl.conn.tsccm.ConnPoolByRoute.getEntryBlocking(ConnPoolByR oute.java:403) . org.apache.http.impl.conn.tsccm.ConnPoolByRoute$1.getPoolEntry(ConnPoolByRou te.java:300) . org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager$1.getConnection( ThreadSafeClientConnManager.java:224) . org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDir ector.java:401) . org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.ja va:820) . org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.ja va:754) . org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.ja va:732) . org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java :351) . org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java :182) . org.apache.solr.update.SolrCmdDistributor$1.call(SolrCmdDistributor.java:325 ) . org.apache.solr.update.SolrCmdDistributor$1.call(SolrCmdDistributor.java:306 ) . java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) . java.util.concurrent.FutureTask.run(FutureTask.java:138) . java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) . java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) . java.util.concurrent.FutureTask.run(FutureTask.java:138) . java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja va:886) . java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9 08) . java.lang.Thread.run(Thread.java:662) -----Original Message----- From: Michael Della Bitta [mailto:michael.della.bi...@appinions.com] Sent: Sunday, July 15, 2012 4:46 PM To: solr-user@lucene.apache.org Subject: Re: SOLR 4 Alpha Out Of Mem Err "unable to create new native thread" That suggests you're running out of threads, not RAM. Possibly you're using a multithreaded collector, and it's pushing you over the top of how many threads your OS lets a single process allocate? Or somehow the thread stack size is set too high? More here: http://stackoverflow.com/questions/763579/how-many-threads-can-a-java-vm-sup port Michael Della Bitta ------------------------------------------------ Appinions, Inc. -- Where Influence Isn't a Game. http://www.appinions.com On Sun, Jul 15, 2012 at 2:45 PM, Nick Koton <nick.ko...@gmail.com> wrote: >>> Solrj multi-threaded client sends several 1,000 docs/sec > >>Can you expand on that? How many threads at once are sending docs to solr? > Is each request a single doc or multiple? > I realize, after the fact, that my solrj client is much like > org.apache.solr.client.solrj.LargeVolumeTestBase. The number of > threads is configurable at run time as are the various commit > parameters. Most of the test have been in the 4-16 threads range. > Most of my testing has been with the single document > SolrServer::add(SolrInputDocument doc )method. When I realized what > LargeVolumeTestBase is doing, I converted my program to use the > SolrServer::add(Collection<SolrInputDocument> docs) method with 100 > documents in each add batch. Unfortunately, the out of memory errors still occur without client side commits. > > If you agree my three approaches to committing are logical, would it > be useful for me to try to reproduce this with "example" schema in a > small cloud configuration using LargeVolumeTestBase or the like? It > will take me a couple days to work it in. Or perhaps this sort of test is already run? > > Best > Nick > > -----Original Message----- > From: ysee...@gmail.com [mailto:ysee...@gmail.com] On Behalf Of Yonik > Seeley > Sent: Sunday, July 15, 2012 11:05 AM > To: Nick Koton > Cc: solr-user@lucene.apache.org > Subject: Re: SOLR 4 Alpha Out Of Mem Err > > On Sun, Jul 15, 2012 at 11:52 AM, Nick Koton <nick.ko...@gmail.com> wrote: >>> Do you have the following hard autoCommit in your config (as the >>> stock >> server does)? >>> <autoCommit> >>> <maxTime>15000</maxTime> >>> <openSearcher>false</openSearcher> >>> </autoCommit> >> >> I have tried with and without that setting. When I described running >> with auto commit, that setting is what I mean. > > OK cool. You should be able to run the stock server (i.e. with this > autocommit) and blast in updates all day long - it looks like you have > more than enough memory. If you can't, we need to fix something. You > shouldn't need explicit commits unless you want the docs to be > searchable at that point. > >> Solrj multi-threaded client sends several 1,000 docs/sec > > Can you expand on that? How many threads at once are sending docs to solr? > Is each request a single doc or multiple? > > -Yonik > http://lucidimagination.com >
