I am trying to use the StreamingUpdateSolrServer to index a bunch of
bibliographic data and it is hanging up every time I run it. Sometimes
it hangs after about 100k records (after about 2 minutes), sometimes
after 4M records (after about 80 minutes) and all different intervals in
between. It appears to be the same issue described here:
https://issues.apache.org/jira/browse/SOLR-1543
The thread dump (included below) seems to indicate that a lock isn't
being released because somewhere in the thread chain after adding a
SolrInputDocument.
Is there some kind of Solr equivalent to closing a session like you do
in an ORM like Hibernate?
Thanks,
-Steve
--
Stephen Meyer
Library Application Developer
UW-Madison Libraries
312F Memorial Library
728 State St.
Madison, WI 53706
sme...@library.wisc.edu
608-265-2844 (ph)
"Just don't let the human factor fail to be a factor at all."
- Andrew Bird, "Tables and Chairs"
Full thread dump Java HotSpot(TM) Client VM (1.5.0_22-147 mixed mode):
"pool-1-thread-6" prio=5 tid=0x00d26d50 nid=0x1043c00 in Object.wait()
[0xb0e0d000..0xb0e0dd90]
at java.lang.Object.wait(Native Method)
- waiting on <0x0bbe29f8> (a
org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ConnectionPool)
at
org.apache.commons.httpclient.MultiThreadedHttpConnectionManager.doGetConnection(MultiThreadedHttpConnectionManager.java:518)
- locked <0x0bbe29f8> (a
org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ConnectionPool)
at
org.apache.commons.httpclient.MultiThreadedHttpConnectionManager.getConnectionWithTimeout(MultiThreadedHttpConnectionManager.java:416)
at
org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:153)
at
org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
at
org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
at
org.apache.solr.client.solrj.impl.StreamingUpdateSolrServer$Runner.run(StreamingUpdateSolrServer.java:153)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:651)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:676)
at java.lang.Thread.run(Thread.java:613)
"pool-1-thread-5" prio=5 tid=0x00d11530 nid=0x1042e00 in Object.wait()
[0xb0d8c000..0xb0d8cd90]
at java.lang.Object.wait(Native Method)
- waiting on <0x0bbe29f8> (a
org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ConnectionPool)
at
org.apache.commons.httpclient.MultiThreadedHttpConnectionManager.doGetConnection(MultiThreadedHttpConnectionManager.java:518)
- locked <0x0bbe29f8> (a
org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ConnectionPool)
at
org.apache.commons.httpclient.MultiThreadedHttpConnectionManager.getConnectionWithTimeout(MultiThreadedHttpConnectionManager.java:416)
at
org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:153)
at
org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
at
org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
at
org.apache.solr.client.solrj.impl.StreamingUpdateSolrServer$Runner.run(StreamingUpdateSolrServer.java:153)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:651)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:676)
at java.lang.Thread.run(Thread.java:613)
"MultiThreadedHttpConnectionManager cleanup" daemon prio=5
tid=0x00d13630 nid=0x10fba00 in Object.wait() [0xb0d0b000..0xb0d0bd90]
at java.lang.Object.wait(Native Method)
- waiting on <0x0bbb0270> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:120)
- locked <0x0bbb0270> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:136)
at
org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ReferenceQueueThread.run(MultiThreadedHttpConnectionManager.java:1122)
"Low Memory Detector" daemon prio=5 tid=0x00d08dc0 nid=0x101ae00
runnable [0x00000000..0x00000000]
"CompilerThread0" daemon prio=9 tid=0x00d08450 nid=0x101a000 waiting on
condition [0x00000000..0xb0b077d8]
"Signal Dispatcher" daemon prio=9 tid=0x00d07f60 nid=0x1015400 waiting
on condition [0x00000000..0x00000000]
"Finalizer" daemon prio=8 tid=0x00d07690 nid=0x1019200 in Object.wait()
[0xb0a05000..0xb0a05d90]
at java.lang.Object.wait(Native Method)
- waiting on <0x0bbb06c0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:120)
- locked <0x0bbb06c0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:136)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
"Reference Handler" daemon prio=10 tid=0x00d072b0 nid=0x1017a00 in
Object.wait() [0xb0984000..0xb0984d90]
at java.lang.Object.wait(Native Method)
- waiting on <0x0bbb00a0> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:474)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
- locked <0x0bbb00a0> (a java.lang.ref.Reference$Lock)
"main" prio=5 tid=0x00d013c0 nid=0xb0801000 waiting on condition
[0xb07ff000..0xb0800188]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
at
java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:253)
at
org.apache.solr.client.solrj.impl.StreamingUpdateSolrServer.request(StreamingUpdateSolrServer.java:216)
at
org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:105)
at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:64)
at
edu.wisc.library.sdg.forward.parser.MarcProcessor.nextFeedItemFromThisWeek(MarcProcessor.java:276)
at
edu.wisc.library.sdg.forward.parser.MarcProcessor.process(MarcProcessor.java:148)
at edu.wisc.library.sdg.forward.App.main(App.java:36)
"VM Thread" prio=9 tid=0x00d06a10 nid=0x1016600 runnable
"VM Periodic Task Thread" prio=9 tid=0x00d09a40 nid=0x101bc00 waiting on
condition
"Exception Catcher Thread" prio=10 tid=0x00d015e0 nid=0x100a800 runnable
