Hanging while indexing/digesting on multiple threads
----------------------------------------------------
Key: LUCENE-1381
URL: https://issues.apache.org/jira/browse/LUCENE-1381
Project: Lucene - Java
Issue Type: Bug
Components: Analysis
Affects Versions: 2.3.2
Environment: Java HotSpot(TM) 64-Bit Server VM (1.5.0_16-b02 mixed
mode) on 2.6.9-78.0.1.ELsmp #1 SMP x86_64 x86_64 x86_64 GNU/Linux
Reporter: David Fertig
With several older lucene projects already running and "stable", I have
recently written a multi-threading indexer using to the 2.3.2 release.
My volume is in the millions of documents indexed daily and I have been stress
testing for a while now. My current setup has 3 JVMs, each running 6 threads
indexing different documents, with 1 IndexWriter per JVM. For stability
testing, the indexer shutsdown and exits every 5-10 minutes, with a new JVM is
started again for a clean restart. At this rate, I have noticed an rare, but
eventually consistent internal hang/deadlock in all indexer threads while
parsing documents. My 'manager' thread is alive and regularly polling the
indexer threads and displaying their state variables, but the indexer threads
themselves appear not to be making progress while using up nearly 100% of
available CPU. Memory usage is relativly low and stable at 481m out of 2048m
available.
Most stack traces, and STAY in this state even after repeated inspections:
(pressing CTRL-\ in active JVM window)
----------
Full thread dump Java HotSpot(TM) 64-Bit Server VM (1.5.0_16-b02 mixed mode):
"Thread-6" prio=1 tid=0x0000002b25750920 nid=0x34f6 runnable
[0x0000000041465000..0x0000000041465db0]
at java.util.WeakHashMap.eq(WeakHashMap.java:254)
at java.util.WeakHashMap.get(WeakHashMap.java:345)
at
org.apache.commons.beanutils.MethodUtils.getMatchingAccessibleMethod(MethodUtils.java:530)
at
org.apache.commons.beanutils.MethodUtils.invokeMethod(MethodUtils.java:209)
at
org.apache.commons.digester.CallMethodRule.end(CallMethodRule.java:625)
at org.apache.commons.digester.Rule.end(Rule.java:230)
at org.apache.commons.digester.Digester.endElement(Digester.java:1130)
at
com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:633)
at
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(XMLDocumentFragmentScannerImpl.java:1241)
at
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(XMLDocumentFragmentScannerImpl.java:1685)
at
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:368)
at
com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:834)
at
com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:764)
at
com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:148)
at
com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1242)
at org.apache.commons.digester.Digester.parse(Digester.java:1685)
at
com.cymfony.dci.lucene.IndexerThread.indexFile(IndexerThread.java:199)
at
com.cymfony.dci.lucene.IndexerThread.indexDirectory(IndexerThread.java:142)
at com.cymfony.dci.lucene.IndexerThread.run(IndexerThread.java:81)
"Thread-5" prio=1 tid=0x0000002b25754eb0 nid=0x34f5 runnable
[0x0000000041364000..0x0000000041364d30]
at java.lang.String.equals(String.java:858)
at
org.apache.commons.beanutils.MethodUtils$MethodDescriptor.equals(MethodUtils.java:833)
at java.util.WeakHashMap.eq(WeakHashMap.java:254)
at java.util.WeakHashMap.get(WeakHashMap.java:345)
at
org.apache.commons.beanutils.MethodUtils.getMatchingAccessibleMethod(MethodUtils.java:530)
at
org.apache.commons.beanutils.MethodUtils.invokeMethod(MethodUtils.java:209)
at
org.apache.commons.digester.CallMethodRule.end(CallMethodRule.java:625)
at org.apache.commons.digester.Rule.end(Rule.java:230)
at org.apache.commons.digester.Digester.endElement(Digester.java:1130)
at
com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:633)
at
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(XMLDocumentFragmentScannerImpl.java:1241)
at
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(XMLDocumentFragmentScannerImpl.java:1685)
at
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:368)
at
com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:834)
at
com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:764)
at
com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:148)
at
com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1242)
at org.apache.commons.digester.Digester.parse(Digester.java:1685)
at
com.cymfony.dci.lucene.IndexerThread.indexFile(IndexerThread.java:199)
at
com.cymfony.dci.lucene.IndexerThread.indexDirectory(IndexerThread.java:142)
at com.cymfony.dci.lucene.IndexerThread.run(IndexerThread.java:81)
"Thread-4" prio=1 tid=0x0000002b25754860 nid=0x34f4 runnable
[0x0000000041263000..0x0000000041263cb0]
at java.lang.String.equals(String.java:858)
at
org.apache.commons.beanutils.MethodUtils$MethodDescriptor.equals(MethodUtils.java:833)
at java.util.WeakHashMap.eq(WeakHashMap.java:254)
at java.util.WeakHashMap.get(WeakHashMap.java:345)
at
org.apache.commons.beanutils.MethodUtils.getMatchingAccessibleMethod(MethodUtils.java:530)
at
org.apache.commons.beanutils.MethodUtils.invokeMethod(MethodUtils.java:209)
at
org.apache.commons.digester.CallMethodRule.end(CallMethodRule.java:625)
at org.apache.commons.digester.Rule.end(Rule.java:230)
at org.apache.commons.digester.Digester.endElement(Digester.java:1130)
at
com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:633)
at
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(XMLDocumentFragmentScannerImpl.java:1241)
at
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(XMLDocumentFragmentScannerImpl.java:1685)
at
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:368)
at
com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:834)
at
com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:764)
at
com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:148)
at
com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1242)
at org.apache.commons.digester.Digester.parse(Digester.java:1685)
at
com.cymfony.dci.lucene.IndexerThread.indexFile(IndexerThread.java:199)
at
com.cymfony.dci.lucene.IndexerThread.indexDirectory(IndexerThread.java:142)
at com.cymfony.dci.lucene.IndexerThread.run(IndexerThread.java:81)
"Thread-3" prio=1 tid=0x0000002b2509d360 nid=0x34f3 runnable
[0x0000000041162000..0x0000000041162c30]
at java.lang.String.equals(String.java:858)
at
org.apache.commons.beanutils.MethodUtils$MethodDescriptor.equals(MethodUtils.java:833)
at java.util.WeakHashMap.eq(WeakHashMap.java:254)
at java.util.WeakHashMap.get(WeakHashMap.java:345)
at
org.apache.commons.beanutils.MethodUtils.getMatchingAccessibleMethod(MethodUtils.java:530)
at
org.apache.commons.beanutils.MethodUtils.invokeMethod(MethodUtils.java:209)
at
org.apache.commons.digester.CallMethodRule.end(CallMethodRule.java:625)
at org.apache.commons.digester.Rule.end(Rule.java:230)
at org.apache.commons.digester.Digester.endElement(Digester.java:1130)
at
com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:633)
at
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(XMLDocumentFragmentScannerImpl.java:1241)
at
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(XMLDocumentFragmentScannerImpl.java:1685)
at
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:368)
at
com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:834)
at
com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:764)
at
com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:148)
at
com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1242)
at org.apache.commons.digester.Digester.parse(Digester.java:1685)
at
com.cymfony.dci.lucene.IndexerThread.indexFile(IndexerThread.java:199)
at
com.cymfony.dci.lucene.IndexerThread.indexDirectory(IndexerThread.java:142)
at com.cymfony.dci.lucene.IndexerThread.run(IndexerThread.java:81)
"Thread-2" prio=1 tid=0x0000002b25083e00 nid=0x34f2 runnable
[0x0000000041061000..0x0000000041061bb0]
at java.util.WeakHashMap.expungeStaleEntries(WeakHashMap.java:289)
at java.util.WeakHashMap.getTable(WeakHashMap.java:297)
at java.util.WeakHashMap.get(WeakHashMap.java:341)
at
org.apache.commons.beanutils.MethodUtils.getMatchingAccessibleMethod(MethodUtils.java:530)
at
org.apache.commons.beanutils.MethodUtils.invokeMethod(MethodUtils.java:209)
at
org.apache.commons.digester.CallMethodRule.end(CallMethodRule.java:625)
at org.apache.commons.digester.Rule.end(Rule.java:230)
at org.apache.commons.digester.Digester.endElement(Digester.java:1130)
at
com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:633)
at
com.sun.org.apache.xerces.internal.parsers.AbstractXMLDocumentParser.emptyElement(AbstractXMLDocumentParser.java:221)
at
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanStartElement(XMLDocumentFragmentScannerImpl.java:872)
at
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(XMLDocumentFragmentScannerImpl.java:1693)
at
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:368)
at
com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:834)
at
com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:764)
at
com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:148)
at
com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1242)
at org.apache.commons.digester.Digester.parse(Digester.java:1685)
at
com.cymfony.dci.lucene.IndexerThread.indexFile(IndexerThread.java:199)
at
com.cymfony.dci.lucene.IndexerThread.indexDirectory(IndexerThread.java:142)
at com.cymfony.dci.lucene.IndexerThread.run(IndexerThread.java:81)
"Thread-1" prio=1 tid=0x0000002b2509a4a0 nid=0x34f1 runnable
[0x0000000040f60000..0x0000000040f60b30]
at java.util.WeakHashMap.expungeStaleEntries(WeakHashMap.java:289)
at java.util.WeakHashMap.getTable(WeakHashMap.java:297)
at java.util.WeakHashMap.get(WeakHashMap.java:341)
at
org.apache.commons.beanutils.MethodUtils.getMatchingAccessibleMethod(MethodUtils.java:530)
at
org.apache.commons.beanutils.MethodUtils.invokeMethod(MethodUtils.java:209)
at
org.apache.commons.digester.CallMethodRule.end(CallMethodRule.java:625)
at org.apache.commons.digester.Rule.end(Rule.java:230)
at org.apache.commons.digester.Digester.endElement(Digester.java:1130)
at
com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:633)
at
com.sun.org.apache.xerces.internal.parsers.AbstractXMLDocumentParser.emptyElement(AbstractXMLDocumentParser.java:221)
at
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanStartElement(XMLDocumentFragmentScannerImpl.java:872)
at
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(XMLDocumentFragmentScannerImpl.java:1693)
at
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:368)
at
com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:834)
at
com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:764)
at
com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:148)
at
com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1242)
at org.apache.commons.digester.Digester.parse(Digester.java:1685)
at
com.cymfony.dci.lucene.IndexerThread.indexFile(IndexerThread.java:199)
at
com.cymfony.dci.lucene.IndexerThread.indexDirectory(IndexerThread.java:142)
at com.cymfony.dci.lucene.IndexerThread.run(IndexerThread.java:81)
"Low Memory Detector" daemon prio=1 tid=0x0000002b25002430 nid=0x34ef runnable
[0x0000000000000000..0x0000000000000000]
"CompilerThread1" daemon prio=1 tid=0x0000002b250009d0 nid=0x34ee waiting on
condition [0x0000000000000000..0x0000000040c5c6d0]
"CompilerThread0" daemon prio=1 tid=0x0000002b220c2ce0 nid=0x34ed waiting on
condition [0x0000000000000000..0x0000000040b5b650]
"AdapterThread" daemon prio=1 tid=0x0000002b220c1750 nid=0x34ec waiting on
condition [0x0000000000000000..0x0000000000000000]
"Signal Dispatcher" daemon prio=1 tid=0x0000002b220c02d0 nid=0x34eb runnable
[0x0000000000000000..0x0000000000000000]
"Finalizer" daemon prio=1 tid=0x0000002b220ae1e0 nid=0x34ea in Object.wait()
[0x0000000040859000..0x0000000040859bb0]
at java.lang.Object.wait(Native Method)
- waiting on <0x0000002b0a8d7408> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:120)
- locked <0x0000002b0a8d7408> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:136)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
"Reference Handler" daemon prio=1 tid=0x0000002b220ab980 nid=0x34e9 in
Object.wait() [0x0000000040758000..0x0000000040758b30]
at java.lang.Object.wait(Native Method)
- waiting on <0x0000002b0a7da3b0> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:474)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
- locked <0x0000002b0a7da3b0> (a java.lang.ref.Reference$Lock)
"main" prio=1 tid=0x0000000040115ea0 nid=0x34e3 waiting on condition
[0x0000007fbfffc000..0x0000007fbfffd280]
at java.lang.Thread.sleep(Native Method)
at
com.cymfony.dci.lucene.IndexerThreadManager.stopAndWaitForIndexerThreads(IndexerThreadManager.java:227)
at
com.cymfony.dci.lucene.IndexerThreadManager.indexDocs(IndexerThreadManager.java:167)
at com.cymfony.dci.lucene.Indexer.indexDocs(Indexer.java:48)
at com.cymfony.dci.lucene.Indexer.indexUnprocessedDocs(Indexer.java:136)
at
com.cymfony.dci.lucene.CommandLineIndexer.main(CommandLineIndexer.java:37)
"VM Thread" prio=1 tid=0x0000002b220a7480 nid=0x34e8 runnable
"GC task thread#0 (ParallelGC)" prio=1 tid=0x0000000040133fe0 nid=0x34e4
runnable
"GC task thread#1 (ParallelGC)" prio=1 tid=0x0000000040135630 nid=0x34e5
runnable
"GC task thread#2 (ParallelGC)" prio=1 tid=0x0000000040136480 nid=0x34e6
runnable
"GC task thread#3 (ParallelGC)" prio=1 tid=0x0000000040137300 nid=0x34e7
runnable
"VM Periodic Task Thread" prio=1 tid=0x0000002b25004e10 nid=0x34f0 waiting on
condition
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
