[
https://issues.apache.org/jira/browse/LUCENE-3692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Simon Willnauer updated LUCENE-3692:
------------------------------------
Attachment: LUCENE-3692.patch
here is a patch that makes pruning the ticket queue non-blocking ie. we don't
hold the lock that is used to append to the queue when we try to publish the
head of the queue but at the same time prevent other threads from peeking at
the head and publish. I extracted all that logic into a new more readable class
and cleaned up DW.
I also added lots of new asserts in that queue - all tests pass over 300
iterations even the famous TestRealtimeGet :)
> DocumentsWriter blocks flushes when applyDeletes takes forever - memory not
> released
> ------------------------------------------------------------------------------------
>
> Key: LUCENE-3692
> URL: https://issues.apache.org/jira/browse/LUCENE-3692
> Project: Lucene - Java
> Issue Type: Bug
> Components: core/index
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Fix For: 4.0
>
> Attachments: LUCENE-3692.patch
>
>
> In DocumentsWriter we have a safety check that applies all deletes if the
> deletes consume too much RAM to prevent too-frequent flushing of a long tail
> of tiny segments. If we enter applyAllDeletes we essentially lock on IW ->
> BufferedDeletes which is fine since this usually doesn't take long and
> doesn't keep DWPTs from indexing. Yet, if that takes long and at the same
> time a semgent is flushed and subsequently published to the IW we take the
> lock on the ticket queue and the IW. Now this prevents all other threads to
> append to the ticketQueue which is done BEFORE we actually flush the segment
> concurrently and free up the RAM.
> Essentially its ok to block on the IW lock but we should not keep concurrent
> flushed from execution just because we apply deletes. The threads will block
> once they try to execute maybeMerge after the segment is flushed so we don't
> pile up subsequent memory but we should actually allow the DWPT to be flushed
> since we actually try to get rid of memory.
> I ran into this by accident due to a coding bug using delete queries instead
> of terms for each document. This thread dump show the problem:
> {noformat}
> "Application Worker Thread" prio=10 tid=0x00007fdda0238000 nid=0x3256 waiting
> for
> monitor entry [0x00007fddad3c2000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:424)
> - waiting to lock <0x00007fddb74ff990> (a
> org.apache.lucene.index.DocumentsWriter$TicketQueue)
> at
> org.apache.lucene.index.DocumentsWriter.postUpdate(DocumentsWriter.java:320)
> at
> org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:393)
> at
> org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1484)
> at
> org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1456)
> at
> org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:160)
> "Application Worker Thread" prio=10 tid=0x00007fdda0236000 nid=0x3255 waiting
> for
> monitor entry [0x00007fddad4c3000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.lucene.index.IndexWriter.updatePendingMerges(IndexWriter.java:1854)
> - waiting to lock <0x00007fddb74fe350> (a
> org.apache.solr.update.SolrIndexWriter)
> at
> org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:1848)
> at
> org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:1843)
> at
> org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1493)
> at
> org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1456)
> at
> org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:160)
> "Application Worker Thread" prio=10 tid=0x00007fdda0234000 nid=0x3254 waiting
> for
> monitor entry [0x00007fddad5c4000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.lucene.index.IndexWriter.updatePendingMerges(IndexWriter.java:1854)
> - waiting to lock <0x00007fddb74fe350> (a
> org.apache.solr.update.SolrIndexWriter)
> at
> org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:1848)
> at
> org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:1843)
> at
> org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1493)
> at
> org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1456)
> at
> org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:160)
> "Application Worker Thread" prio=10 tid=0x00007fdda0232000 nid=0x3253 waiting
> for
> monitor entry [0x00007fddad6c5000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:424)
> - waiting to lock <0x00007fddb74ff990> (a
> org.apache.lucene.index.DocumentsWriter$TicketQueue)
> at
> org.apache.lucene.index.DocumentsWriter.postUpdate(DocumentsWriter.java:320)
> at
> org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:393)
> at
> org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1484)
> at
> org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1456)
> at
> org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:160)
> "Application Worker Thread" prio=10 tid=0x00007fdda0230800 nid=0x3252 waiting
> for
> monitor entry [0x00007fddad7c6000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.lucene.index.IndexWriter.updatePendingMerges(IndexWriter.java:1854)
> - waiting to lock <0x00007fddb74fe350> (a
> org.apache.solr.update.SolrIndexWriter)
> at
> org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:1848)
> at
> org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:1843)
> at
> org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1493)
> at
> org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1456)
> at
> org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:160)
> "Application Worker Thread" prio=10 tid=0x00007fdda022e800 nid=0x3251 runnable
> [0x00007fddad8c6000]
> java.lang.Thread.State: RUNNABLE
> at java.nio.Bits.copyToArray(Bits.java:715)
> at java.nio.DirectByteBuffer.get(DirectByteBuffer.java:233)
> at
> org.apache.lucene.store.MMapDirectory$MMapIndexInput.readBytes(MMapDirectory.java:319)
> at
> org.apache.lucene.index.codecs.BlockTreeTermsReader$FieldReader$SegmentTermsEnum$Frame.loadBlock(BlockTreeTermsReader.java:2283)
> at
> org.apache.lucene.index.codecs.BlockTreeTermsReader$FieldReader$SegmentTermsEnum.seekExact(BlockTreeTermsReader.java:1600)
> at org.apache.lucene.util.TermContext.build(TermContext.java:97)
> at org.apache.lucene.search.TermQuery.createWeight(TermQuery.java:180)
> at
> org.apache.lucene.search.BooleanQuery$BooleanWeight.<init>(BooleanQuery.java:186)
> at
> org.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:423)
> at
> org.apache.lucene.search.IndexSearcher.createNormalizedWeight(IndexSearcher.java:583)
> at
> org.apache.lucene.search.QueryWrapperFilter.getDocIdSet(QueryWrapperFilter.java:55)
> at
> org.apache.lucene.index.BufferedDeletesStream.applyQueryDeletes(BufferedDeletesStream.java:431)
> at
> org.apache.lucene.index.BufferedDeletesStream.applyDeletes(BufferedDeletesStream.java:268)
> - locked <0x00007fddb751e1e8> (a
> org.apache.lucene.index.BufferedDeletesStream)
> at
> org.apache.lucene.index.IndexWriter.applyAllDeletes(IndexWriter.java:2852)
> - locked <0x00007fddb74fe350> (a
> org.apache.solr.update.SolrIndexWriter)
> at
> org.apache.lucene.index.DocumentsWriter.applyAllDeletes(DocumentsWriter.java:188)
> at
> org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:470)
> at
> org.apache.lucene.index.DocumentsWriter.postUpdate(DocumentsWriter.java:320)
> at
> org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:393)
> at
> org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1484)
> at
> org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1456)
> at
> org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:160)
>
> "Application Worker Thread" prio=10 tid=0x00007fdda022d800 nid=0x3250 waiting
> for
> monitor entry [0x00007fddad9c8000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:424)
> - waiting to lock <0x00007fddb74ff990> (a
> org.apache.lucene.index.DocumentsWriter$TicketQueue)
> at
> org.apache.lucene.index.DocumentsWriter.postUpdate(DocumentsWriter.java:320)
> at
> org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:393)
> at
> org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1484)
> at
> org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1456)
> at
> org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:160)
>
> "Application Worker Thread" prio=10 tid=0x00007fdda022d000 nid=0x324f waiting
> for
> monitor entry [0x00007fddadac9000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.lucene.index.IndexWriter.useCompoundFile(IndexWriter.java:2274)
> - waiting to lock <0x00007fddb74fe350> (a
> org.apache.solr.update.SolrIndexWriter)
> at
> org.apache.lucene.index.IndexWriter.prepareFlushedSegment(IndexWriter.java:2156)
> at
> org.apache.lucene.index.DocumentsWriter.publishFlushedSegment(DocumentsWriter.java:526)
> at
> org.apache.lucene.index.DocumentsWriter.finishFlush(DocumentsWriter.java:506)
> at
> org.apache.lucene.index.DocumentsWriter.applyFlushTickets(DocumentsWriter.java:483)
> - locked <0x00007fddb74ff990> (a
> org.apache.lucene.index.DocumentsWriter$TicketQueue)
> at
> org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:449)
> at
> org.apache.lucene.index.DocumentsWriter.postUpdate(DocumentsWriter.java:320)
> at
> org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:393)
> at
> org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1484)
> at
> org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1456)
> at
> org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:160)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
