I experienced the hang described with the Solr 1.4.0 build. Yonik - I also thought the streaming updater was blocking on commits but updates never resumed.
To be honest I was in a bit of a rush to meet a deadline so after spending a day or so tinkering I bailed out and just wrote a component by hand. I have not tried to reproduce this using the current trunk. I was using the 32-bit Sun JRE on a Red Hat EL 5 HP server. I'm not sure if the following enriches this thread, but I'll include it anyways: write a document generator and start adding a ton of 'em to a Solr server instance using the streaming updater. You *should* experience the hang. HTH, Rich On Fri, Apr 16, 2010 at 1:34 PM, Sascha Szott <sz...@zib.de> wrote: > Hi Yonik, > > thanks for your fast reply. > > > Yonik Seeley wrote: > >> Thanks for the report Sascha. >> So after the hang, it never recovers? Some amount of hanging could be >> visible if there was a commit on the Solr server or something else to >> cause the solr requests to block for a while... but it should return >> to normal on it's own... >> > In my case the whole application hangs and never recovers (CPU utilization > goes down to near 0%). Interestingly, the problem reproducibly occurs only > if SUSS is created with *more than 2* threads. > > > Looking at the stack trace, it looks like threads are blocked waiting >> to get an http connection. >> > I forgot to mention that my index app has exclusive access to the Solr > instance. Therefore, concurrent searches against the same Solr instance > while indexing are excluded. > > > I'm traveling all next week, but I'll open a JIRA issue for this now. >> > Thank you! > > > Anything that would help us reproduce this is much appreciated. >> > Are there any other who have experienced the same problem? > > -Sascha > > > >> On Fri, Apr 16, 2010 at 8:57 AM, Sascha Szott<sz...@zib.de> wrote: >> >>> Hi Yonik, >>> >>> Yonik Seeley wrote: >>> >>>> >>>> Stephen, were you running stock Solr 1.4, or did you apply any of the >>>> SolrJ patches? >>>> I'm trying to figure out if anyone still has any problems, or if this >>>> was fixed with SOLR-1711: >>>> >>> >>> I'm using the latest trunk version (rev. 934846) and constantly running >>> into >>> the same problem. I'm using StreamingUpdateSolrServer with 3 treads and a >>> queue size of 20 (not really knowing if this configuration is optimal). >>> My >>> multi-threaded application indexes 200k data items (bibliographic >>> metadata >>> in Dublin Core format) and constantly hangs after running for some time. >>> >>> Below you can find the thread dump of one of my index threads (after the >>> app >>> hangs all dumps are the same) >>> >>> "thread 19" prio=10 tid=0x00007fe8c0415800 nid=0x277d waiting on >>> condition >>> [0x0000000042d05000] >>> java.lang.Thread.State: WAITING (parking) >>> at sun.misc.Unsafe.park(Native Method) >>> - parking to wait for<0x00007fe8cdcb7598> (a >>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) >>> at >>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) >>> at >>> >>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) >>> at >>> >>> java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:254) >>> at >>> >>> org.apache.solr.client.solrj.impl.StreamingUpdateSolrServer.request(StreamingUpdateSolrServer.java:216) >>> at >>> >>> org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:105) >>> at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:64) >>> at >>> >>> de.kobv.ked.index.SolrIndexWriter.addIndexDocument(SolrIndexWriter.java:29) >>> at >>> >>> de.kobv.ked.index.SolrIndexWriter.addIndexDocument(SolrIndexWriter.java:10) >>> at >>> >>> de.kobv.ked.index.AbstractIndexThread.addIndexDocument(AbstractIndexThread.java:59) >>> at de.kobv.ked.rss.RssThread.indiziere(RssThread.java:30) >>> at de.kobv.ked.rss.RssThread.run(RssThread.java:58) >>> >>> >>> >>> and of the three SUSS threads: >>> >>> "pool-1-thread-3" prio=10 tid=0x00007fe8c7b7f000 nid=0x2780 in >>> Object.wait() >>> [0x00000000409ac000] >>> java.lang.Thread.State: WAITING (on object monitor) >>> at java.lang.Object.wait(Native Method) >>> - waiting on<0x00007fe8cdcb6f10> (a >>> >>> org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ConnectionPool) >>> at >>> >>> org.apache.commons.httpclient.MultiThreadedHttpConnectionManager.doGetConnection(MultiThreadedHttpConnectionManager.java:518) >>> - locked<0x00007fe8cdcb6f10> (a >>> >>> org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ConnectionPool) >>> at >>> >>> org.apache.commons.httpclient.MultiThreadedHttpConnectionManager.getConnectionWithTimeout(MultiThreadedHttpConnectionManager.java:416) >>> at >>> >>> org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:153) >>> at >>> >>> org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) >>> at >>> >>> org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) >>> at >>> >>> org.apache.solr.client.solrj.impl.StreamingUpdateSolrServer$Runner.run(StreamingUpdateSolrServer.java:153) >>> at >>> >>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >>> at >>> >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >>> at java.lang.Thread.run(Thread.java:619) >>> >>> "pool-1-thread-2" prio=10 tid=0x00007fe8c7afa000 nid=0x277f in >>> Object.wait() >>> [0x0000000040209000] >>> java.lang.Thread.State: WAITING (on object monitor) >>> at java.lang.Object.wait(Native Method) >>> - waiting on<0x00007fe8cdcb6f10> (a >>> >>> org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ConnectionPool) >>> at >>> >>> org.apache.commons.httpclient.MultiThreadedHttpConnectionManager.doGetConnection(MultiThreadedHttpConnectionManager.java:518) >>> - locked<0x00007fe8cdcb6f10> (a >>> >>> org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ConnectionPool) >>> at >>> >>> org.apache.commons.httpclient.MultiThreadedHttpConnectionManager.getConnectionWithTimeout(MultiThreadedHttpConnectionManager.java:416) >>> at >>> >>> org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:153) >>> at >>> >>> org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) >>> at >>> >>> org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) >>> at >>> >>> org.apache.solr.client.solrj.impl.StreamingUpdateSolrServer$Runner.run(StreamingUpdateSolrServer.java:153) >>> at >>> >>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >>> at >>> >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >>> at java.lang.Thread.run(Thread.java:619) >>> >>> "pool-1-thread-1" prio=10 tid=0x00007fe8c79f2800 nid=0x277e in >>> Object.wait() >>> [0x0000000042e06000] >>> java.lang.Thread.State: WAITING (on object monitor) >>> at java.lang.Object.wait(Native Method) >>> - waiting on<0x00007fe8cdcb6f10> (a >>> >>> org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ConnectionPool) >>> at >>> >>> org.apache.commons.httpclient.MultiThreadedHttpConnectionManager.doGetConnection(MultiThreadedHttpConnectionManager.java:518) >>> - locked<0x00007fe8cdcb6f10> (a >>> >>> org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ConnectionPool) >>> at >>> >>> org.apache.commons.httpclient.MultiThreadedHttpConnectionManager.getConnectionWithTimeout(MultiThreadedHttpConnectionManager.java:416) >>> at >>> >>> org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:153) >>> at >>> >>> org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) >>> at >>> >>> org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) >>> at >>> >>> org.apache.solr.client.solrj.impl.StreamingUpdateSolrServer$Runner.run(StreamingUpdateSolrServer.java:153) >>> at >>> >>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >>> at >>> >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >>> at java.lang.Thread.run(Thread.java:619) >>> >>> >>> >>> Do you have any idea how we can fix this issue or, at least, locate the >>> error? >>> >>> Best, >>> Sascha >>> >>> >>>> * SOLR-1711: SolrJ - StreamingUpdateSolrServer had a race condition that >>>> could halt the streaming of documents. (Attila Babo via yonik) >>>> >>>> Also note that people may want this patch if dealing with i18n: >>>> >>>> * SOLR-1595: StreamingUpdateSolrServer used the platform default >>>> character >>>> set when streaming updates, rather than using UTF-8 as the HTTP >>>> headers >>>> indicated, leading to an encoding mismatch. (hossman, yonik) >>>> >>>> >>>> -Yonik >>>> Apache Lucene Eurocon 2010 >>>> 18-21 May 2010 | Prague >>>> >>>> >>>> >>>> On Fri, Feb 5, 2010 at 3:20 PM, Stephen Meyer<sme...@library.wisc.edu> >>>> wrote: >>>> >>>>> >>>>> I am trying to use the StreamingUpdateSolrServer to index a bunch of >>>>> bibliographic data and it is hanging up every time I run it. Sometimes >>>>> it >>>>> hangs after about 100k records (after about 2 minutes), sometimes after >>>>> 4M >>>>> records (after about 80 minutes) and all different intervals in >>>>> between. >>>>> It >>>>> appears to be the same issue described here: >>>>> >>>>> https://issues.apache.org/jira/browse/SOLR-1543 >>>>> >>>>> The thread dump (included below) seems to indicate that a lock isn't >>>>> being >>>>> released because somewhere in the thread chain after adding a >>>>> SolrInputDocument. >>>>> >>>>> Is there some kind of Solr equivalent to closing a session like you do >>>>> in >>>>> an >>>>> ORM like Hibernate? >>>>> >>>>> Thanks, >>>>> -Steve >>>>> -- >>>>> Stephen Meyer >>>>> Library Application Developer >>>>> UW-Madison Libraries >>>>> 312F Memorial Library >>>>> 728 State St. >>>>> Madison, WI 53706 >>>>> >>>>> sme...@library.wisc.edu >>>>> 608-265-2844 (ph) >>>>> >>>>> >>>>> "Just don't let the human factor fail to be a factor at all." >>>>> - Andrew Bird, "Tables and Chairs" >>>>> >>>>> Full thread dump Java HotSpot(TM) Client VM (1.5.0_22-147 mixed mode): >>>>> >>>>> "pool-1-thread-6" prio=5 tid=0x00d26d50 nid=0x1043c00 in Object.wait() >>>>> [0xb0e0d000..0xb0e0dd90] >>>>> at java.lang.Object.wait(Native Method) >>>>> - waiting on<0x0bbe29f8> (a >>>>> >>>>> >>>>> org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ConnectionPool) >>>>> at >>>>> >>>>> >>>>> org.apache.commons.httpclient.MultiThreadedHttpConnectionManager.doGetConnection(MultiThreadedHttpConnectionManager.java:518) >>>>> - locked<0x0bbe29f8> (a >>>>> >>>>> >>>>> org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ConnectionPool) >>>>> at >>>>> >>>>> >>>>> org.apache.commons.httpclient.MultiThreadedHttpConnectionManager.getConnectionWithTimeout(MultiThreadedHttpConnectionManager.java:416) >>>>> at >>>>> >>>>> >>>>> org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:153) >>>>> at >>>>> >>>>> >>>>> org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) >>>>> at >>>>> >>>>> >>>>> org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) >>>>> at >>>>> >>>>> >>>>> org.apache.solr.client.solrj.impl.StreamingUpdateSolrServer$Runner.run(StreamingUpdateSolrServer.java:153) >>>>> at >>>>> >>>>> >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:651) >>>>> at >>>>> >>>>> >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:676) >>>>> at java.lang.Thread.run(Thread.java:613) >>>>> >>>>> "pool-1-thread-5" prio=5 tid=0x00d11530 nid=0x1042e00 in Object.wait() >>>>> [0xb0d8c000..0xb0d8cd90] >>>>> at java.lang.Object.wait(Native Method) >>>>> - waiting on<0x0bbe29f8> (a >>>>> >>>>> >>>>> org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ConnectionPool) >>>>> at >>>>> >>>>> >>>>> org.apache.commons.httpclient.MultiThreadedHttpConnectionManager.doGetConnection(MultiThreadedHttpConnectionManager.java:518) >>>>> - locked<0x0bbe29f8> (a >>>>> >>>>> >>>>> org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ConnectionPool) >>>>> at >>>>> >>>>> >>>>> org.apache.commons.httpclient.MultiThreadedHttpConnectionManager.getConnectionWithTimeout(MultiThreadedHttpConnectionManager.java:416) >>>>> at >>>>> >>>>> >>>>> org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:153) >>>>> at >>>>> >>>>> >>>>> org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) >>>>> at >>>>> >>>>> >>>>> org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) >>>>> at >>>>> >>>>> >>>>> org.apache.solr.client.solrj.impl.StreamingUpdateSolrServer$Runner.run(StreamingUpdateSolrServer.java:153) >>>>> at >>>>> >>>>> >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:651) >>>>> at >>>>> >>>>> >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:676) >>>>> at java.lang.Thread.run(Thread.java:613) >>>>> >>>>> "MultiThreadedHttpConnectionManager cleanup" daemon prio=5 >>>>> tid=0x00d13630 >>>>> nid=0x10fba00 in Object.wait() [0xb0d0b000..0xb0d0bd90] >>>>> at java.lang.Object.wait(Native Method) >>>>> - waiting on<0x0bbb0270> (a >>>>> java.lang.ref.ReferenceQueue$Lock) >>>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:120) >>>>> - locked<0x0bbb0270> (a java.lang.ref.ReferenceQueue$Lock) >>>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:136) >>>>> at >>>>> >>>>> >>>>> org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ReferenceQueueThread.run(MultiThreadedHttpConnectionManager.java:1122) >>>>> >>>>> "Low Memory Detector" daemon prio=5 tid=0x00d08dc0 nid=0x101ae00 >>>>> runnable >>>>> [0x00000000..0x00000000] >>>>> >>>>> "CompilerThread0" daemon prio=9 tid=0x00d08450 nid=0x101a000 waiting on >>>>> condition [0x00000000..0xb0b077d8] >>>>> >>>>> "Signal Dispatcher" daemon prio=9 tid=0x00d07f60 nid=0x1015400 waiting >>>>> on >>>>> condition [0x00000000..0x00000000] >>>>> >>>>> "Finalizer" daemon prio=8 tid=0x00d07690 nid=0x1019200 in Object.wait() >>>>> [0xb0a05000..0xb0a05d90] >>>>> at java.lang.Object.wait(Native Method) >>>>> - waiting on<0x0bbb06c0> (a >>>>> java.lang.ref.ReferenceQueue$Lock) >>>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:120) >>>>> - locked<0x0bbb06c0> (a java.lang.ref.ReferenceQueue$Lock) >>>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:136) >>>>> at >>>>> java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159) >>>>> >>>>> "Reference Handler" daemon prio=10 tid=0x00d072b0 nid=0x1017a00 in >>>>> Object.wait() [0xb0984000..0xb0984d90] >>>>> at java.lang.Object.wait(Native Method) >>>>> - waiting on<0x0bbb00a0> (a java.lang.ref.Reference$Lock) >>>>> at java.lang.Object.wait(Object.java:474) >>>>> at >>>>> java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116) >>>>> - locked<0x0bbb00a0> (a java.lang.ref.Reference$Lock) >>>>> >>>>> "main" prio=5 tid=0x00d013c0 nid=0xb0801000 waiting on condition >>>>> [0xb07ff000..0xb0800188] >>>>> at sun.misc.Unsafe.park(Native Method) >>>>> at >>>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:118) >>>>> at >>>>> >>>>> >>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841) >>>>> at >>>>> >>>>> >>>>> java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:253) >>>>> at >>>>> >>>>> >>>>> org.apache.solr.client.solrj.impl.StreamingUpdateSolrServer.request(StreamingUpdateSolrServer.java:216) >>>>> at >>>>> >>>>> >>>>> org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:105) >>>>> at >>>>> org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:64) >>>>> at >>>>> >>>>> >>>>> edu.wisc.library.sdg.forward.parser.MarcProcessor.nextFeedItemFromThisWeek(MarcProcessor.java:276) >>>>> at >>>>> >>>>> >>>>> edu.wisc.library.sdg.forward.parser.MarcProcessor.process(MarcProcessor.java:148) >>>>> at edu.wisc.library.sdg.forward.App.main(App.java:36) >>>>> >>>>> "VM Thread" prio=9 tid=0x00d06a10 nid=0x1016600 runnable >>>>> >>>>> "VM Periodic Task Thread" prio=9 tid=0x00d09a40 nid=0x101bc00 waiting >>>>> on >>>>> condition >>>>> >>>>> "Exception Catcher Thread" prio=10 tid=0x00d015e0 nid=0x100a800 >>>>> runnable >>>>> >>>>> >>> >>> >
