[ https://issues.apache.org/jira/browse/LUCENE-2680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jason Rutherglen updated LUCENE-2680: ------------------------------------- Attachment: LUCENE-2680.patch Here's a check point patch before I re-add lastSegmentInfo/Index. All tests pass except for what's below. I'm guessing segments with all docs deleted, are deleted before the test expects. {code} [junit] Testcase: testCommitThreadSafety(org.apache.lucene.index.TestIndexWriter): FAILED [junit] [junit] junit.framework.AssertionFailedError: [junit] at org.apache.lucene.util.LuceneTestCase$LuceneTestCaseRunner.runChild(LuceneTestCase.java:878) [junit] at org.apache.lucene.util.LuceneTestCase$LuceneTestCaseRunner.runChild(LuceneTestCase.java:844) [junit] at org.apache.lucene.index.TestIndexWriter.testCommitThreadSafety(TestIndexWriter.java:4699) [junit] [junit] [junit] Testcase: testCommitThreadSafety(org.apache.lucene.index.TestIndexWriter): FAILED [junit] Some threads threw uncaught exceptions! [junit] junit.framework.AssertionFailedError: Some threads threw uncaught exceptions! [junit] at org.apache.lucene.util.LuceneTestCase$LuceneTestCaseRunner.runChild(LuceneTestCase.java:878) [junit] at org.apache.lucene.util.LuceneTestCase$LuceneTestCaseRunner.runChild(LuceneTestCase.java:844) [junit] at org.apache.lucene.util.LuceneTestCase.tearDown(LuceneTestCase.java:437) [junit] [junit] [junit] Tests run: 116, Failures: 2, Errors: 0, Time elapsed: 159.577 sec [junit] [junit] ------------- Standard Output --------------- [junit] NOTE: reproduce with: ant test -Dtestcase=TestIndexWriter -Dtestmethod=testCommitThreadSafety -Dtests.seed=1826133140332330367:8102643307925777745 [junit] NOTE: test params are: codec=MockFixedIntBlock(blockSize=564), locale=es_CR, timezone=Asia/Urumqi [junit] ------------- ---------------- --------------- [junit] ------------- Standard Error ----------------- [junit] The following exceptions were thrown by threads: [junit] *** Thread: Thread-1106 *** [junit] java.lang.RuntimeException: java.lang.AssertionError: term=f:0_8; r=DirectoryReader(_0:c1 _1:c1 _2:c1 _3:c1 _4:c1 _5:c1 _6:c1 _7:c2 _8:c4 ) expected:<1> but was:<0> [junit] at org.apache.lucene.index.TestIndexWriter$9.run(TestIndexWriter.java:4690) [junit] Caused by: java.lang.AssertionError: term=f:0_8; r=DirectoryReader(_0:c1 _1:c1 _2:c1 _3:c1 _4:c1 _5:c1 _6:c1 _7:c2 _8:c4 ) expected:<1> but was:<0> [junit] at org.junit.Assert.fail(Assert.java:91) [junit] at org.junit.Assert.failNotEquals(Assert.java:645) [junit] at org.junit.Assert.assertEquals(Assert.java:126) [junit] at org.junit.Assert.assertEquals(Assert.java:470) [junit] at org.apache.lucene.index.TestIndexWriter$9.run(TestIndexWriter.java:4684) [junit] NOTE: all tests run in this JVM: [junit] [TestMockAnalyzer, TestByteSlices, TestFilterIndexReader, TestIndexFileDeleter, TestIndexReaderClone, TestIndexReaderReopen, TestIndexWriter] {code} > Improve how IndexWriter flushes deletes against existing segments > ----------------------------------------------------------------- > > Key: LUCENE-2680 > URL: https://issues.apache.org/jira/browse/LUCENE-2680 > Project: Lucene - Java > Issue Type: Improvement > Reporter: Michael McCandless > Fix For: 4.0 > > Attachments: LUCENE-2680.patch, LUCENE-2680.patch, LUCENE-2680.patch, > LUCENE-2680.patch, LUCENE-2680.patch > > > IndexWriter buffers up all deletes (by Term and Query) and only > applies them if 1) commit or NRT getReader() is called, or 2) a merge > is about to kickoff. > We do this because, for a large index, it's very costly to open a > SegmentReader for every segment in the index. So we defer as long as > we can. We do it just before merge so that the merge can eliminate > the deleted docs. > But, most merges are small, yet in a big index we apply deletes to all > of the segments, which is really very wasteful. > Instead, we should only apply the buffered deletes to the segments > that are about to be merged, and keep the buffer around for the > remaining segments. > I think it's not so hard to do; we'd have to have generations of > pending deletions, because the newly merged segment doesn't need the > same buffered deletions applied again. So every time a merge kicks > off, we pinch off the current set of buffered deletions, open a new > set (the next generation), and record which segment was created as of > which generation. > This should be a very sizable gain for large indices that mix > deletes, though, less so in flex since opening the terms index is much > faster. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org