Re: StreamingUpdateSolrServer hangs
On Thu, Apr 29, 2010 at 7:51 PM, Yonik Seeley yo...@lucidimagination.com wrote: I'm trying to reproduce now... single thread adding documents to a multithreaded client, StreamingUpdateSolrServer(addr,32,4) I'm currently at the 2.5 hour mark and 100M documents - no issues so far. I let it go to 500M docs... everything works fine (this is with the current trunk). -Yonik Apache Lucene Eurocon 2010 18-21 May 2010 | Prague
Re: StreamingUpdateSolrServer hangs
On Fri, Apr 16, 2010 at 1:34 PM, Sascha Szott sz...@zib.de wrote: In my case the whole application hangs and never recovers (CPU utilization goes down to near 0%). Interestingly, the problem reproducibly occurs only if SUSS is created with *more than 2* threads. Is your application also using multiple threads when adding docs to the SUSS? FYI, I opened https://issues.apache.org/jira/browse/SOLR-1885 to track this. -Yonik Apache Lucene Eurocon 2010 18-21 May 2010 | Prague
Re: StreamingUpdateSolrServer hangs
In solrconfig.xml, there is a parameter controlling remote streaming: requestDispatcher handleSelect=true !--Make sure your system has some authentication before enabling remote streaming! -- requestParsers enableRemoteStreaming=true multipartUploadLimitInKB=2048000 / 1) Is this relevant with the SUSS? 2) It seems to be 'true' in the example default, which may not be a good idea. On Thu, Apr 29, 2010 at 2:12 PM, Yonik Seeley yo...@lucidimagination.com wrote: On Fri, Apr 16, 2010 at 1:34 PM, Sascha Szott sz...@zib.de wrote: In my case the whole application hangs and never recovers (CPU utilization goes down to near 0%). Interestingly, the problem reproducibly occurs only if SUSS is created with *more than 2* threads. Is your application also using multiple threads when adding docs to the SUSS? FYI, I opened https://issues.apache.org/jira/browse/SOLR-1885 to track this. -Yonik Apache Lucene Eurocon 2010 18-21 May 2010 | Prague -- Lance Norskog goks...@gmail.com
Re: StreamingUpdateSolrServer hangs
On Thu, Apr 29, 2010 at 6:04 PM, Lance Norskog goks...@gmail.com wrote: In solrconfig.xml, there is a parameter controlling remote streaming: requestDispatcher handleSelect=true !--Make sure your system has some authentication before enabling remote streaming! -- requestParsers enableRemoteStreaming=true multipartUploadLimitInKB=2048000 / 1) Is this relevant with the SUSS? No, this relates to solr pulling data from another source (via stream.url) -Yonik Apache Lucene Eurocon 2010 18-21 May 2010 | Prague
Re: StreamingUpdateSolrServer hangs
What is the garbage collection status when this happens? What are the open sockets in the OS when this happens? Run 'netstat -an | fgrep 8983' where 8983 is the Solr incoming port number. A side note on sockets: SUSS uses the MultiThreadedHttpConnectionManager but never calls MultiThreadedHttpConnectionManager.closeIdleConnections() on its sockets. I don't know if this is a problem, but it should do this as a matter of dotting the i's and crossing the t's. On Thu, Apr 29, 2010 at 3:25 PM, Yonik Seeley yo...@lucidimagination.com wrote: On Thu, Apr 29, 2010 at 6:04 PM, Lance Norskog goks...@gmail.com wrote: In solrconfig.xml, there is a parameter controlling remote streaming: requestDispatcher handleSelect=true !--Make sure your system has some authentication before enabling remote streaming! -- requestParsers enableRemoteStreaming=true multipartUploadLimitInKB=2048000 / 1) Is this relevant with the SUSS? No, this relates to solr pulling data from another source (via stream.url) -Yonik Apache Lucene Eurocon 2010 18-21 May 2010 | Prague -- Lance Norskog goks...@gmail.com
Re: StreamingUpdateSolrServer hangs
I'm trying to reproduce now... single thread adding documents to a multithreaded client, StreamingUpdateSolrServer(addr,32,4) I'm currently at the 2.5 hour mark and 100M documents - no issues so far. -Yonik Apache Lucene Eurocon 2010 18-21 May 2010 | Prague On Thu, Apr 29, 2010 at 5:12 PM, Yonik Seeley yo...@lucidimagination.com wrote: On Fri, Apr 16, 2010 at 1:34 PM, Sascha Szott sz...@zib.de wrote: In my case the whole application hangs and never recovers (CPU utilization goes down to near 0%). Interestingly, the problem reproducibly occurs only if SUSS is created with *more than 2* threads. Is your application also using multiple threads when adding docs to the SUSS? FYI, I opened https://issues.apache.org/jira/browse/SOLR-1885 to track this. -Yonik Apache Lucene Eurocon 2010 18-21 May 2010 | Prague
Re: StreamingUpdateSolrServer hangs
Hi Yonik, Yonik Seeley wrote: Stephen, were you running stock Solr 1.4, or did you apply any of the SolrJ patches? I'm trying to figure out if anyone still has any problems, or if this was fixed with SOLR-1711: I'm using the latest trunk version (rev. 934846) and constantly running into the same problem. I'm using StreamingUpdateSolrServer with 3 treads and a queue size of 20 (not really knowing if this configuration is optimal). My multi-threaded application indexes 200k data items (bibliographic metadata in Dublin Core format) and constantly hangs after running for some time. Below you can find the thread dump of one of my index threads (after the app hangs all dumps are the same) thread 19 prio=10 tid=0x7fe8c0415800 nid=0x277d waiting on condition [0x42d05000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for 0x7fe8cdcb7598 (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:254) at org.apache.solr.client.solrj.impl.StreamingUpdateSolrServer.request(StreamingUpdateSolrServer.java:216) at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:105) at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:64) at de.kobv.ked.index.SolrIndexWriter.addIndexDocument(SolrIndexWriter.java:29) at de.kobv.ked.index.SolrIndexWriter.addIndexDocument(SolrIndexWriter.java:10) at de.kobv.ked.index.AbstractIndexThread.addIndexDocument(AbstractIndexThread.java:59) at de.kobv.ked.rss.RssThread.indiziere(RssThread.java:30) at de.kobv.ked.rss.RssThread.run(RssThread.java:58) and of the three SUSS threads: pool-1-thread-3 prio=10 tid=0x7fe8c7b7f000 nid=0x2780 in Object.wait() [0x409ac000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on 0x7fe8cdcb6f10 (a org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ConnectionPool) at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager.doGetConnection(MultiThreadedHttpConnectionManager.java:518) - locked 0x7fe8cdcb6f10 (a org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ConnectionPool) at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager.getConnectionWithTimeout(MultiThreadedHttpConnectionManager.java:416) at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:153) at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) at org.apache.solr.client.solrj.impl.StreamingUpdateSolrServer$Runner.run(StreamingUpdateSolrServer.java:153) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) pool-1-thread-2 prio=10 tid=0x7fe8c7afa000 nid=0x277f in Object.wait() [0x40209000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on 0x7fe8cdcb6f10 (a org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ConnectionPool) at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager.doGetConnection(MultiThreadedHttpConnectionManager.java:518) - locked 0x7fe8cdcb6f10 (a org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ConnectionPool) at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager.getConnectionWithTimeout(MultiThreadedHttpConnectionManager.java:416) at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:153) at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) at org.apache.solr.client.solrj.impl.StreamingUpdateSolrServer$Runner.run(StreamingUpdateSolrServer.java:153) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) pool-1-thread-1 prio=10 tid=0x7fe8c79f2800 nid=0x277e in Object.wait() [0x42e06000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on 0x7fe8cdcb6f10 (a org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ConnectionPool) at
Re: StreamingUpdateSolrServer hangs
Thanks for the report Sascha. So after the hang, it never recovers? Some amount of hanging could be visible if there was a commit on the Solr server or something else to cause the solr requests to block for a while... but it should return to normal on it's own... Looking at the stack trace, it looks like threads are blocked waiting to get an http connection. I'm traveling all next week, but I'll open a JIRA issue for this now. Anything that would help us reproduce this is much appreciated. -Yonik Apache Lucene Eurocon 2010 18-21 May 2010 | Prague On Fri, Apr 16, 2010 at 8:57 AM, Sascha Szott sz...@zib.de wrote: Hi Yonik, Yonik Seeley wrote: Stephen, were you running stock Solr 1.4, or did you apply any of the SolrJ patches? I'm trying to figure out if anyone still has any problems, or if this was fixed with SOLR-1711: I'm using the latest trunk version (rev. 934846) and constantly running into the same problem. I'm using StreamingUpdateSolrServer with 3 treads and a queue size of 20 (not really knowing if this configuration is optimal). My multi-threaded application indexes 200k data items (bibliographic metadata in Dublin Core format) and constantly hangs after running for some time. Below you can find the thread dump of one of my index threads (after the app hangs all dumps are the same) thread 19 prio=10 tid=0x7fe8c0415800 nid=0x277d waiting on condition [0x42d05000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for 0x7fe8cdcb7598 (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:254) at org.apache.solr.client.solrj.impl.StreamingUpdateSolrServer.request(StreamingUpdateSolrServer.java:216) at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:105) at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:64) at de.kobv.ked.index.SolrIndexWriter.addIndexDocument(SolrIndexWriter.java:29) at de.kobv.ked.index.SolrIndexWriter.addIndexDocument(SolrIndexWriter.java:10) at de.kobv.ked.index.AbstractIndexThread.addIndexDocument(AbstractIndexThread.java:59) at de.kobv.ked.rss.RssThread.indiziere(RssThread.java:30) at de.kobv.ked.rss.RssThread.run(RssThread.java:58) and of the three SUSS threads: pool-1-thread-3 prio=10 tid=0x7fe8c7b7f000 nid=0x2780 in Object.wait() [0x409ac000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on 0x7fe8cdcb6f10 (a org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ConnectionPool) at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager.doGetConnection(MultiThreadedHttpConnectionManager.java:518) - locked 0x7fe8cdcb6f10 (a org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ConnectionPool) at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager.getConnectionWithTimeout(MultiThreadedHttpConnectionManager.java:416) at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:153) at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) at org.apache.solr.client.solrj.impl.StreamingUpdateSolrServer$Runner.run(StreamingUpdateSolrServer.java:153) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) pool-1-thread-2 prio=10 tid=0x7fe8c7afa000 nid=0x277f in Object.wait() [0x40209000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on 0x7fe8cdcb6f10 (a org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ConnectionPool) at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager.doGetConnection(MultiThreadedHttpConnectionManager.java:518) - locked 0x7fe8cdcb6f10 (a org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ConnectionPool) at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager.getConnectionWithTimeout(MultiThreadedHttpConnectionManager.java:416) at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:153) at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) at
Re: StreamingUpdateSolrServer hangs
Hi Yonik, thanks for your fast reply. Yonik Seeley wrote: Thanks for the report Sascha. So after the hang, it never recovers? Some amount of hanging could be visible if there was a commit on the Solr server or something else to cause the solr requests to block for a while... but it should return to normal on it's own... In my case the whole application hangs and never recovers (CPU utilization goes down to near 0%). Interestingly, the problem reproducibly occurs only if SUSS is created with *more than 2* threads. Looking at the stack trace, it looks like threads are blocked waiting to get an http connection. I forgot to mention that my index app has exclusive access to the Solr instance. Therefore, concurrent searches against the same Solr instance while indexing are excluded. I'm traveling all next week, but I'll open a JIRA issue for this now. Thank you! Anything that would help us reproduce this is much appreciated. Are there any other who have experienced the same problem? -Sascha On Fri, Apr 16, 2010 at 8:57 AM, Sascha Szottsz...@zib.de wrote: Hi Yonik, Yonik Seeley wrote: Stephen, were you running stock Solr 1.4, or did you apply any of the SolrJ patches? I'm trying to figure out if anyone still has any problems, or if this was fixed with SOLR-1711: I'm using the latest trunk version (rev. 934846) and constantly running into the same problem. I'm using StreamingUpdateSolrServer with 3 treads and a queue size of 20 (not really knowing if this configuration is optimal). My multi-threaded application indexes 200k data items (bibliographic metadata in Dublin Core format) and constantly hangs after running for some time. Below you can find the thread dump of one of my index threads (after the app hangs all dumps are the same) thread 19 prio=10 tid=0x7fe8c0415800 nid=0x277d waiting on condition [0x42d05000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for0x7fe8cdcb7598 (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:254) at org.apache.solr.client.solrj.impl.StreamingUpdateSolrServer.request(StreamingUpdateSolrServer.java:216) at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:105) at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:64) at de.kobv.ked.index.SolrIndexWriter.addIndexDocument(SolrIndexWriter.java:29) at de.kobv.ked.index.SolrIndexWriter.addIndexDocument(SolrIndexWriter.java:10) at de.kobv.ked.index.AbstractIndexThread.addIndexDocument(AbstractIndexThread.java:59) at de.kobv.ked.rss.RssThread.indiziere(RssThread.java:30) at de.kobv.ked.rss.RssThread.run(RssThread.java:58) and of the three SUSS threads: pool-1-thread-3 prio=10 tid=0x7fe8c7b7f000 nid=0x2780 in Object.wait() [0x409ac000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on0x7fe8cdcb6f10 (a org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ConnectionPool) at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager.doGetConnection(MultiThreadedHttpConnectionManager.java:518) - locked0x7fe8cdcb6f10 (a org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ConnectionPool) at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager.getConnectionWithTimeout(MultiThreadedHttpConnectionManager.java:416) at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:153) at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) at org.apache.solr.client.solrj.impl.StreamingUpdateSolrServer$Runner.run(StreamingUpdateSolrServer.java:153) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) pool-1-thread-2 prio=10 tid=0x7fe8c7afa000 nid=0x277f in Object.wait() [0x40209000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on0x7fe8cdcb6f10 (a org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ConnectionPool) at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager.doGetConnection(MultiThreadedHttpConnectionManager.java:518) - locked0x7fe8cdcb6f10 (a
Re: StreamingUpdateSolrServer hangs
I experienced the hang described with the Solr 1.4.0 build. Yonik - I also thought the streaming updater was blocking on commits but updates never resumed. To be honest I was in a bit of a rush to meet a deadline so after spending a day or so tinkering I bailed out and just wrote a component by hand. I have not tried to reproduce this using the current trunk. I was using the 32-bit Sun JRE on a Red Hat EL 5 HP server. I'm not sure if the following enriches this thread, but I'll include it anyways: write a document generator and start adding a ton of 'em to a Solr server instance using the streaming updater. You *should* experience the hang. HTH, Rich On Fri, Apr 16, 2010 at 1:34 PM, Sascha Szott sz...@zib.de wrote: Hi Yonik, thanks for your fast reply. Yonik Seeley wrote: Thanks for the report Sascha. So after the hang, it never recovers? Some amount of hanging could be visible if there was a commit on the Solr server or something else to cause the solr requests to block for a while... but it should return to normal on it's own... In my case the whole application hangs and never recovers (CPU utilization goes down to near 0%). Interestingly, the problem reproducibly occurs only if SUSS is created with *more than 2* threads. Looking at the stack trace, it looks like threads are blocked waiting to get an http connection. I forgot to mention that my index app has exclusive access to the Solr instance. Therefore, concurrent searches against the same Solr instance while indexing are excluded. I'm traveling all next week, but I'll open a JIRA issue for this now. Thank you! Anything that would help us reproduce this is much appreciated. Are there any other who have experienced the same problem? -Sascha On Fri, Apr 16, 2010 at 8:57 AM, Sascha Szottsz...@zib.de wrote: Hi Yonik, Yonik Seeley wrote: Stephen, were you running stock Solr 1.4, or did you apply any of the SolrJ patches? I'm trying to figure out if anyone still has any problems, or if this was fixed with SOLR-1711: I'm using the latest trunk version (rev. 934846) and constantly running into the same problem. I'm using StreamingUpdateSolrServer with 3 treads and a queue size of 20 (not really knowing if this configuration is optimal). My multi-threaded application indexes 200k data items (bibliographic metadata in Dublin Core format) and constantly hangs after running for some time. Below you can find the thread dump of one of my index threads (after the app hangs all dumps are the same) thread 19 prio=10 tid=0x7fe8c0415800 nid=0x277d waiting on condition [0x42d05000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for0x7fe8cdcb7598 (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:254) at org.apache.solr.client.solrj.impl.StreamingUpdateSolrServer.request(StreamingUpdateSolrServer.java:216) at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:105) at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:64) at de.kobv.ked.index.SolrIndexWriter.addIndexDocument(SolrIndexWriter.java:29) at de.kobv.ked.index.SolrIndexWriter.addIndexDocument(SolrIndexWriter.java:10) at de.kobv.ked.index.AbstractIndexThread.addIndexDocument(AbstractIndexThread.java:59) at de.kobv.ked.rss.RssThread.indiziere(RssThread.java:30) at de.kobv.ked.rss.RssThread.run(RssThread.java:58) and of the three SUSS threads: pool-1-thread-3 prio=10 tid=0x7fe8c7b7f000 nid=0x2780 in Object.wait() [0x409ac000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on0x7fe8cdcb6f10 (a org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ConnectionPool) at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager.doGetConnection(MultiThreadedHttpConnectionManager.java:518) - locked0x7fe8cdcb6f10 (a org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ConnectionPool) at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager.getConnectionWithTimeout(MultiThreadedHttpConnectionManager.java:416) at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:153) at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) at
Re: StreamingUpdateSolrServer hangs
Stephen, were you running stock Solr 1.4, or did you apply any of the SolrJ patches? I'm trying to figure out if anyone still has any problems, or if this was fixed with SOLR-1711: * SOLR-1711: SolrJ - StreamingUpdateSolrServer had a race condition that could halt the streaming of documents. (Attila Babo via yonik) Also note that people may want this patch if dealing with i18n: * SOLR-1595: StreamingUpdateSolrServer used the platform default character set when streaming updates, rather than using UTF-8 as the HTTP headers indicated, leading to an encoding mismatch. (hossman, yonik) -Yonik Apache Lucene Eurocon 2010 18-21 May 2010 | Prague On Fri, Feb 5, 2010 at 3:20 PM, Stephen Meyer sme...@library.wisc.edu wrote: I am trying to use the StreamingUpdateSolrServer to index a bunch of bibliographic data and it is hanging up every time I run it. Sometimes it hangs after about 100k records (after about 2 minutes), sometimes after 4M records (after about 80 minutes) and all different intervals in between. It appears to be the same issue described here: https://issues.apache.org/jira/browse/SOLR-1543 The thread dump (included below) seems to indicate that a lock isn't being released because somewhere in the thread chain after adding a SolrInputDocument. Is there some kind of Solr equivalent to closing a session like you do in an ORM like Hibernate? Thanks, -Steve -- Stephen Meyer Library Application Developer UW-Madison Libraries 312F Memorial Library 728 State St. Madison, WI 53706 sme...@library.wisc.edu 608-265-2844 (ph) Just don't let the human factor fail to be a factor at all. - Andrew Bird, Tables and Chairs Full thread dump Java HotSpot(TM) Client VM (1.5.0_22-147 mixed mode): pool-1-thread-6 prio=5 tid=0x00d26d50 nid=0x1043c00 in Object.wait() [0xb0e0d000..0xb0e0dd90] at java.lang.Object.wait(Native Method) - waiting on 0x0bbe29f8 (a org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ConnectionPool) at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager.doGetConnection(MultiThreadedHttpConnectionManager.java:518) - locked 0x0bbe29f8 (a org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ConnectionPool) at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager.getConnectionWithTimeout(MultiThreadedHttpConnectionManager.java:416) at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:153) at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) at org.apache.solr.client.solrj.impl.StreamingUpdateSolrServer$Runner.run(StreamingUpdateSolrServer.java:153) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:651) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:676) at java.lang.Thread.run(Thread.java:613) pool-1-thread-5 prio=5 tid=0x00d11530 nid=0x1042e00 in Object.wait() [0xb0d8c000..0xb0d8cd90] at java.lang.Object.wait(Native Method) - waiting on 0x0bbe29f8 (a org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ConnectionPool) at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager.doGetConnection(MultiThreadedHttpConnectionManager.java:518) - locked 0x0bbe29f8 (a org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ConnectionPool) at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager.getConnectionWithTimeout(MultiThreadedHttpConnectionManager.java:416) at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:153) at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) at org.apache.solr.client.solrj.impl.StreamingUpdateSolrServer$Runner.run(StreamingUpdateSolrServer.java:153) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:651) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:676) at java.lang.Thread.run(Thread.java:613) MultiThreadedHttpConnectionManager cleanup daemon prio=5 tid=0x00d13630 nid=0x10fba00 in Object.wait() [0xb0d0b000..0xb0d0bd90] at java.lang.Object.wait(Native Method) - waiting on 0x0bbb0270 (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:120) - locked 0x0bbb0270 (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:136) at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ReferenceQueueThread.run(MultiThreadedHttpConnectionManager.java:1122) Low Memory Detector daemon prio=5
