[ https://issues.apache.org/jira/browse/LUCENE-1381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12630280#action_12630280 ]
David Fertig commented on LUCENE-1381: -------------------------------------- Otis, You are correct, I should have started by joining the mailing list and working with that first, sorry. All threads share the same IndexWriter for the same index. I agree that nothing in the stack traces points to lucene and have been reviewing and upgrading the other apache libraries as well. This is fairly repeatable over time. Dave > Hanging while indexing/digesting on multiple threads > ---------------------------------------------------- > > Key: LUCENE-1381 > URL: https://issues.apache.org/jira/browse/LUCENE-1381 > Project: Lucene - Java > Issue Type: Bug > Components: Analysis > Affects Versions: 2.3.2 > Environment: Java HotSpot(TM) 64-Bit Server VM (1.5.0_16-b02 mixed > mode) on 2.6.9-78.0.1.ELsmp #1 SMP x86_64 x86_64 x86_64 GNU/Linux > Reporter: David Fertig > > With several older lucene projects already running and "stable", I have > recently written a multi-threading indexer using to the 2.3.2 release. > My volume is in the millions of documents indexed daily and I have been > stress testing for a while now. My current setup has 3 JVMs, each running 6 > threads indexing different documents, with 1 IndexWriter per JVM. For > stability testing, the indexer shutsdown and exits every 5-10 minutes, with a > new JVM is started again for a clean restart. At this rate, I have noticed an > rare, but eventually consistent internal hang/deadlock in all indexer threads > while parsing documents. My 'manager' thread is alive and regularly polling > the indexer threads and displaying their state variables, but the indexer > threads themselves appear not to be making progress while using up nearly > 100% of available CPU. Memory usage is relativly low and stable at 481m out > of 2048m available. > Most stack traces, and STAY in this state even after repeated inspections: > (pressing CTRL-\ in active JVM window) > ---------- > Full thread dump Java HotSpot(TM) 64-Bit Server VM (1.5.0_16-b02 mixed mode): > "Thread-6" prio=1 tid=0x0000002b25750920 nid=0x34f6 runnable > [0x0000000041465000..0x0000000041465db0] > at java.util.WeakHashMap.eq(WeakHashMap.java:254) > at java.util.WeakHashMap.get(WeakHashMap.java:345) > at > org.apache.commons.beanutils.MethodUtils.getMatchingAccessibleMethod(MethodUtils.java:530) > at > org.apache.commons.beanutils.MethodUtils.invokeMethod(MethodUtils.java:209) > at > org.apache.commons.digester.CallMethodRule.end(CallMethodRule.java:625) > at org.apache.commons.digester.Rule.end(Rule.java:230) > at org.apache.commons.digester.Digester.endElement(Digester.java:1130) > at > com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:633) > at > com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(XMLDocumentFragmentScannerImpl.java:1241) > at > com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(XMLDocumentFragmentScannerImpl.java:1685) > at > com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:368) > at > com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:834) > at > com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:764) > at > com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:148) > at > com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1242) > at org.apache.commons.digester.Digester.parse(Digester.java:1685) > ... > "Thread-5" prio=1 tid=0x0000002b25754eb0 nid=0x34f5 runnable > [0x0000000041364000..0x0000000041364d30] > at java.lang.String.equals(String.java:858) > at > org.apache.commons.beanutils.MethodUtils$MethodDescriptor.equals(MethodUtils.java:833) > at java.util.WeakHashMap.eq(WeakHashMap.java:254) > at java.util.WeakHashMap.get(WeakHashMap.java:345) > at > org.apache.commons.beanutils.MethodUtils.getMatchingAccessibleMethod(MethodUtils.java:530) > at > org.apache.commons.beanutils.MethodUtils.invokeMethod(MethodUtils.java:209) > at > org.apache.commons.digester.CallMethodRule.end(CallMethodRule.java:625) > at org.apache.commons.digester.Rule.end(Rule.java:230) > at org.apache.commons.digester.Digester.endElement(Digester.java:1130) > at > com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:633) > at > com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(XMLDocumentFragmentScannerImpl.java:1241) > at > com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(XMLDocumentFragmentScannerImpl.java:1685) > at > com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:368) > at > com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:834) > at > com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:764) > at > com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:148) > at > com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1242) > at org.apache.commons.digester.Digester.parse(Digester.java:1685) > ... > "Thread-4" prio=1 tid=0x0000002b25754860 nid=0x34f4 runnable > [0x0000000041263000..0x0000000041263cb0] > at java.lang.String.equals(String.java:858) > at > org.apache.commons.beanutils.MethodUtils$MethodDescriptor.equals(MethodUtils.java:833) > at java.util.WeakHashMap.eq(WeakHashMap.java:254) > at java.util.WeakHashMap.get(WeakHashMap.java:345) > at > org.apache.commons.beanutils.MethodUtils.getMatchingAccessibleMethod(MethodUtils.java:530) > at > org.apache.commons.beanutils.MethodUtils.invokeMethod(MethodUtils.java:209) > at > org.apache.commons.digester.CallMethodRule.end(CallMethodRule.java:625) > at org.apache.commons.digester.Rule.end(Rule.java:230) > at org.apache.commons.digester.Digester.endElement(Digester.java:1130) > at > com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:633) > at > com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(XMLDocumentFragmentScannerImpl.java:1241) > at > com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(XMLDocumentFragmentScannerImpl.java:1685) > at > com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:368) > at > com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:834) > at > com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:764) > at > com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:148) > at > com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1242) > at org.apache.commons.digester.Digester.parse(Digester.java:1685) > ... > "Thread-3" prio=1 tid=0x0000002b2509d360 nid=0x34f3 runnable > [0x0000000041162000..0x0000000041162c30] > at java.lang.String.equals(String.java:858) > at > org.apache.commons.beanutils.MethodUtils$MethodDescriptor.equals(MethodUtils.java:833) > at java.util.WeakHashMap.eq(WeakHashMap.java:254) > at java.util.WeakHashMap.get(WeakHashMap.java:345) > at > org.apache.commons.beanutils.MethodUtils.getMatchingAccessibleMethod(MethodUtils.java:530) > at > org.apache.commons.beanutils.MethodUtils.invokeMethod(MethodUtils.java:209) > at > org.apache.commons.digester.CallMethodRule.end(CallMethodRule.java:625) > at org.apache.commons.digester.Rule.end(Rule.java:230) > at org.apache.commons.digester.Digester.endElement(Digester.java:1130) > at > com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:633) > at > com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(XMLDocumentFragmentScannerImpl.java:1241) > at > com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(XMLDocumentFragmentScannerImpl.java:1685) > at > com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:368) > at > com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:834) > at > com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:764) > at > com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:148) > at > com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1242) > at org.apache.commons.digester.Digester.parse(Digester.java:1685) > ... > "Thread-2" prio=1 tid=0x0000002b25083e00 nid=0x34f2 runnable > [0x0000000041061000..0x0000000041061bb0] > at java.util.WeakHashMap.expungeStaleEntries(WeakHashMap.java:289) > at java.util.WeakHashMap.getTable(WeakHashMap.java:297) > at java.util.WeakHashMap.get(WeakHashMap.java:341) > at > org.apache.commons.beanutils.MethodUtils.getMatchingAccessibleMethod(MethodUtils.java:530) > at > org.apache.commons.beanutils.MethodUtils.invokeMethod(MethodUtils.java:209) > at > org.apache.commons.digester.CallMethodRule.end(CallMethodRule.java:625) > at org.apache.commons.digester.Rule.end(Rule.java:230) > at org.apache.commons.digester.Digester.endElement(Digester.java:1130) > at > com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:633) > at > com.sun.org.apache.xerces.internal.parsers.AbstractXMLDocumentParser.emptyElement(AbstractXMLDocumentParser.java:221) > at > com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanStartElement(XMLDocumentFragmentScannerImpl.java:872) > at > com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(XMLDocumentFragmentScannerImpl.java:1693) > at > com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:368) > at > com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:834) > at > com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:764) > at > com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:148) > at > com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1242) > at org.apache.commons.digester.Digester.parse(Digester.java:1685) > ... > "Thread-1" prio=1 tid=0x0000002b2509a4a0 nid=0x34f1 runnable > [0x0000000040f60000..0x0000000040f60b30] > at java.util.WeakHashMap.expungeStaleEntries(WeakHashMap.java:289) > at java.util.WeakHashMap.getTable(WeakHashMap.java:297) > at java.util.WeakHashMap.get(WeakHashMap.java:341) > at > org.apache.commons.beanutils.MethodUtils.getMatchingAccessibleMethod(MethodUtils.java:530) > at > org.apache.commons.beanutils.MethodUtils.invokeMethod(MethodUtils.java:209) > at > org.apache.commons.digester.CallMethodRule.end(CallMethodRule.java:625) > at org.apache.commons.digester.Rule.end(Rule.java:230) > at org.apache.commons.digester.Digester.endElement(Digester.java:1130) > at > com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:633) > at > com.sun.org.apache.xerces.internal.parsers.AbstractXMLDocumentParser.emptyElement(AbstractXMLDocumentParser.java:221) > at > com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanStartElement(XMLDocumentFragmentScannerImpl.java:872) > at > com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(XMLDocumentFragmentScannerImpl.java:1693) > at > com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:368) > at > com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:834) > at > com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:764) > at > com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:148) > at > com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1242) > at org.apache.commons.digester.Digester.parse(Digester.java:1685) > ... > "Low Memory Detector" daemon prio=1 tid=0x0000002b25002430 nid=0x34ef > runnable [0x0000000000000000..0x0000000000000000] > "CompilerThread1" daemon prio=1 tid=0x0000002b250009d0 nid=0x34ee waiting on > condition [0x0000000000000000..0x0000000040c5c6d0] > "CompilerThread0" daemon prio=1 tid=0x0000002b220c2ce0 nid=0x34ed waiting on > condition [0x0000000000000000..0x0000000040b5b650] > "AdapterThread" daemon prio=1 tid=0x0000002b220c1750 nid=0x34ec waiting on > condition [0x0000000000000000..0x0000000000000000] > "Signal Dispatcher" daemon prio=1 tid=0x0000002b220c02d0 nid=0x34eb runnable > [0x0000000000000000..0x0000000000000000] > "Finalizer" daemon prio=1 tid=0x0000002b220ae1e0 nid=0x34ea in Object.wait() > [0x0000000040859000..0x0000000040859bb0] > at java.lang.Object.wait(Native Method) > - waiting on <0x0000002b0a8d7408> (a > java.lang.ref.ReferenceQueue$Lock) > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:120) > - locked <0x0000002b0a8d7408> (a java.lang.ref.ReferenceQueue$Lock) > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:136) > at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159) > "Reference Handler" daemon prio=1 tid=0x0000002b220ab980 nid=0x34e9 in > Object.wait() [0x0000000040758000..0x0000000040758b30] > at java.lang.Object.wait(Native Method) > - waiting on <0x0000002b0a7da3b0> (a java.lang.ref.Reference$Lock) > at java.lang.Object.wait(Object.java:474) > at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116) > - locked <0x0000002b0a7da3b0> (a java.lang.ref.Reference$Lock) > "main" prio=1 tid=0x0000000040115ea0 nid=0x34e3 waiting on condition > [0x0000007fbfffc000..0x0000007fbfffd280] > at java.lang.Thread.sleep(Native Method) > ... > "VM Thread" prio=1 tid=0x0000002b220a7480 nid=0x34e8 runnable > "GC task thread#0 (ParallelGC)" prio=1 tid=0x0000000040133fe0 nid=0x34e4 > runnable > "GC task thread#1 (ParallelGC)" prio=1 tid=0x0000000040135630 nid=0x34e5 > runnable > "GC task thread#2 (ParallelGC)" prio=1 tid=0x0000000040136480 nid=0x34e6 > runnable > "GC task thread#3 (ParallelGC)" prio=1 tid=0x0000000040137300 nid=0x34e7 > runnable > "VM Periodic Task Thread" prio=1 tid=0x0000002b25004e10 nid=0x34f0 waiting on > condition -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]