Hi Yonik,
thanks for your fast reply.
Yonik Seeley wrote:
Thanks for the report Sascha.
So after the hang, it never recovers? Some amount of hanging could be
visible if there was a commit on the Solr server or something else to
cause the solr requests to block for a while... but it should return
to normal on it's own...
In my case the whole application hangs and never recovers (CPU
utilization goes down to near 0%). Interestingly, the problem
reproducibly occurs only if SUSS is created with *more than 2* threads.
Looking at the stack trace, it looks like threads are blocked waiting
to get an http connection.
I forgot to mention that my index app has exclusive access to the Solr
instance. Therefore, concurrent searches against the same Solr instance
while indexing are excluded.
I'm traveling all next week, but I'll open a JIRA issue for this now.
Thank you!
Anything that would help us reproduce this is much appreciated.
Are there any other who have experienced the same problem?
-Sascha
On Fri, Apr 16, 2010 at 8:57 AM, Sascha Szott<sz...@zib.de> wrote:
Hi Yonik,
Yonik Seeley wrote:
Stephen, were you running stock Solr 1.4, or did you apply any of the
SolrJ patches?
I'm trying to figure out if anyone still has any problems, or if this
was fixed with SOLR-1711:
I'm using the latest trunk version (rev. 934846) and constantly running into
the same problem. I'm using StreamingUpdateSolrServer with 3 treads and a
queue size of 20 (not really knowing if this configuration is optimal). My
multi-threaded application indexes 200k data items (bibliographic metadata
in Dublin Core format) and constantly hangs after running for some time.
Below you can find the thread dump of one of my index threads (after the app
hangs all dumps are the same)
"thread 19" prio=10 tid=0x00007fe8c0415800 nid=0x277d waiting on condition
[0x0000000042d05000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for<0x00007fe8cdcb7598> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
at
java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:254)
at
org.apache.solr.client.solrj.impl.StreamingUpdateSolrServer.request(StreamingUpdateSolrServer.java:216)
at
org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:105)
at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:64)
at
de.kobv.ked.index.SolrIndexWriter.addIndexDocument(SolrIndexWriter.java:29)
at
de.kobv.ked.index.SolrIndexWriter.addIndexDocument(SolrIndexWriter.java:10)
at
de.kobv.ked.index.AbstractIndexThread.addIndexDocument(AbstractIndexThread.java:59)
at de.kobv.ked.rss.RssThread.indiziere(RssThread.java:30)
at de.kobv.ked.rss.RssThread.run(RssThread.java:58)
and of the three SUSS threads:
"pool-1-thread-3" prio=10 tid=0x00007fe8c7b7f000 nid=0x2780 in Object.wait()
[0x00000000409ac000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on<0x00007fe8cdcb6f10> (a
org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ConnectionPool)
at
org.apache.commons.httpclient.MultiThreadedHttpConnectionManager.doGetConnection(MultiThreadedHttpConnectionManager.java:518)
- locked<0x00007fe8cdcb6f10> (a
org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ConnectionPool)
at
org.apache.commons.httpclient.MultiThreadedHttpConnectionManager.getConnectionWithTimeout(MultiThreadedHttpConnectionManager.java:416)
at
org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:153)
at
org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
at
org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
at
org.apache.solr.client.solrj.impl.StreamingUpdateSolrServer$Runner.run(StreamingUpdateSolrServer.java:153)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
"pool-1-thread-2" prio=10 tid=0x00007fe8c7afa000 nid=0x277f in Object.wait()
[0x0000000040209000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on<0x00007fe8cdcb6f10> (a
org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ConnectionPool)
at
org.apache.commons.httpclient.MultiThreadedHttpConnectionManager.doGetConnection(MultiThreadedHttpConnectionManager.java:518)
- locked<0x00007fe8cdcb6f10> (a
org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ConnectionPool)
at
org.apache.commons.httpclient.MultiThreadedHttpConnectionManager.getConnectionWithTimeout(MultiThreadedHttpConnectionManager.java:416)
at
org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:153)
at
org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
at
org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
at
org.apache.solr.client.solrj.impl.StreamingUpdateSolrServer$Runner.run(StreamingUpdateSolrServer.java:153)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
"pool-1-thread-1" prio=10 tid=0x00007fe8c79f2800 nid=0x277e in Object.wait()
[0x0000000042e06000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on<0x00007fe8cdcb6f10> (a
org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ConnectionPool)
at
org.apache.commons.httpclient.MultiThreadedHttpConnectionManager.doGetConnection(MultiThreadedHttpConnectionManager.java:518)
- locked<0x00007fe8cdcb6f10> (a
org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ConnectionPool)
at
org.apache.commons.httpclient.MultiThreadedHttpConnectionManager.getConnectionWithTimeout(MultiThreadedHttpConnectionManager.java:416)
at
org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:153)
at
org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
at
org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
at
org.apache.solr.client.solrj.impl.StreamingUpdateSolrServer$Runner.run(StreamingUpdateSolrServer.java:153)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Do you have any idea how we can fix this issue or, at least, locate the
error?
Best,
Sascha
* SOLR-1711: SolrJ - StreamingUpdateSolrServer had a race condition that
could halt the streaming of documents. (Attila Babo via yonik)
Also note that people may want this patch if dealing with i18n:
* SOLR-1595: StreamingUpdateSolrServer used the platform default character
set when streaming updates, rather than using UTF-8 as the HTTP headers
indicated, leading to an encoding mismatch. (hossman, yonik)
-Yonik
Apache Lucene Eurocon 2010
18-21 May 2010 | Prague
On Fri, Feb 5, 2010 at 3:20 PM, Stephen Meyer<sme...@library.wisc.edu>
wrote:
I am trying to use the StreamingUpdateSolrServer to index a bunch of
bibliographic data and it is hanging up every time I run it. Sometimes it
hangs after about 100k records (after about 2 minutes), sometimes after
4M
records (after about 80 minutes) and all different intervals in between.
It
appears to be the same issue described here:
https://issues.apache.org/jira/browse/SOLR-1543
The thread dump (included below) seems to indicate that a lock isn't
being
released because somewhere in the thread chain after adding a
SolrInputDocument.
Is there some kind of Solr equivalent to closing a session like you do in
an
ORM like Hibernate?
Thanks,
-Steve
--
Stephen Meyer
Library Application Developer
UW-Madison Libraries
312F Memorial Library
728 State St.
Madison, WI 53706
sme...@library.wisc.edu
608-265-2844 (ph)
"Just don't let the human factor fail to be a factor at all."
- Andrew Bird, "Tables and Chairs"
Full thread dump Java HotSpot(TM) Client VM (1.5.0_22-147 mixed mode):
"pool-1-thread-6" prio=5 tid=0x00d26d50 nid=0x1043c00 in Object.wait()
[0xb0e0d000..0xb0e0dd90]
at java.lang.Object.wait(Native Method)
- waiting on<0x0bbe29f8> (a
org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ConnectionPool)
at
org.apache.commons.httpclient.MultiThreadedHttpConnectionManager.doGetConnection(MultiThreadedHttpConnectionManager.java:518)
- locked<0x0bbe29f8> (a
org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ConnectionPool)
at
org.apache.commons.httpclient.MultiThreadedHttpConnectionManager.getConnectionWithTimeout(MultiThreadedHttpConnectionManager.java:416)
at
org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:153)
at
org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
at
org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
at
org.apache.solr.client.solrj.impl.StreamingUpdateSolrServer$Runner.run(StreamingUpdateSolrServer.java:153)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:651)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:676)
at java.lang.Thread.run(Thread.java:613)
"pool-1-thread-5" prio=5 tid=0x00d11530 nid=0x1042e00 in Object.wait()
[0xb0d8c000..0xb0d8cd90]
at java.lang.Object.wait(Native Method)
- waiting on<0x0bbe29f8> (a
org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ConnectionPool)
at
org.apache.commons.httpclient.MultiThreadedHttpConnectionManager.doGetConnection(MultiThreadedHttpConnectionManager.java:518)
- locked<0x0bbe29f8> (a
org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ConnectionPool)
at
org.apache.commons.httpclient.MultiThreadedHttpConnectionManager.getConnectionWithTimeout(MultiThreadedHttpConnectionManager.java:416)
at
org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:153)
at
org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
at
org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
at
org.apache.solr.client.solrj.impl.StreamingUpdateSolrServer$Runner.run(StreamingUpdateSolrServer.java:153)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:651)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:676)
at java.lang.Thread.run(Thread.java:613)
"MultiThreadedHttpConnectionManager cleanup" daemon prio=5 tid=0x00d13630
nid=0x10fba00 in Object.wait() [0xb0d0b000..0xb0d0bd90]
at java.lang.Object.wait(Native Method)
- waiting on<0x0bbb0270> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:120)
- locked<0x0bbb0270> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:136)
at
org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ReferenceQueueThread.run(MultiThreadedHttpConnectionManager.java:1122)
"Low Memory Detector" daemon prio=5 tid=0x00d08dc0 nid=0x101ae00 runnable
[0x00000000..0x00000000]
"CompilerThread0" daemon prio=9 tid=0x00d08450 nid=0x101a000 waiting on
condition [0x00000000..0xb0b077d8]
"Signal Dispatcher" daemon prio=9 tid=0x00d07f60 nid=0x1015400 waiting on
condition [0x00000000..0x00000000]
"Finalizer" daemon prio=8 tid=0x00d07690 nid=0x1019200 in Object.wait()
[0xb0a05000..0xb0a05d90]
at java.lang.Object.wait(Native Method)
- waiting on<0x0bbb06c0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:120)
- locked<0x0bbb06c0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:136)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
"Reference Handler" daemon prio=10 tid=0x00d072b0 nid=0x1017a00 in
Object.wait() [0xb0984000..0xb0984d90]
at java.lang.Object.wait(Native Method)
- waiting on<0x0bbb00a0> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:474)
at
java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
- locked<0x0bbb00a0> (a java.lang.ref.Reference$Lock)
"main" prio=5 tid=0x00d013c0 nid=0xb0801000 waiting on condition
[0xb07ff000..0xb0800188]
at sun.misc.Unsafe.park(Native Method)
at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
at
java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:253)
at
org.apache.solr.client.solrj.impl.StreamingUpdateSolrServer.request(StreamingUpdateSolrServer.java:216)
at
org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:105)
at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:64)
at
edu.wisc.library.sdg.forward.parser.MarcProcessor.nextFeedItemFromThisWeek(MarcProcessor.java:276)
at
edu.wisc.library.sdg.forward.parser.MarcProcessor.process(MarcProcessor.java:148)
at edu.wisc.library.sdg.forward.App.main(App.java:36)
"VM Thread" prio=9 tid=0x00d06a10 nid=0x1016600 runnable
"VM Periodic Task Thread" prio=9 tid=0x00d09a40 nid=0x101bc00 waiting on
condition
"Exception Catcher Thread" prio=10 tid=0x00d015e0 nid=0x100a800 runnable
