[ https://issues.apache.org/jira/browse/LUCENE-2680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jason Rutherglen updated LUCENE-2680: ------------------------------------- Attachment: LUCENE-2680.patch Here's a new patch with properly working last segment index. The trunk version of apply deletes has become applyDeletesAll and is functionally unchanged. There's a new method, DW applyDeletesToSegments called by _mergeInit for segments that are about to be merged. The deleted terms and queries for these segments are kept in hash sets because docid-uptos are not needed. Like the last patch DW maintains the last segment index. There's no need to maintain the last-segindex per BD, instead I think it's only useful per DW, and for trunk we only have one DW being used at a time. On successful merge, the last segment index is set to the segment index previous to the start segment of the merge. The merged segments deletes are coalesced into the startIndex-1's segment deletes. > Improve how IndexWriter flushes deletes against existing segments > ----------------------------------------------------------------- > > Key: LUCENE-2680 > URL: https://issues.apache.org/jira/browse/LUCENE-2680 > Project: Lucene - Java > Issue Type: Improvement > Reporter: Michael McCandless > Fix For: 4.0 > > Attachments: LUCENE-2680.patch, LUCENE-2680.patch > > > IndexWriter buffers up all deletes (by Term and Query) and only > applies them if 1) commit or NRT getReader() is called, or 2) a merge > is about to kickoff. > We do this because, for a large index, it's very costly to open a > SegmentReader for every segment in the index. So we defer as long as > we can. We do it just before merge so that the merge can eliminate > the deleted docs. > But, most merges are small, yet in a big index we apply deletes to all > of the segments, which is really very wasteful. > Instead, we should only apply the buffered deletes to the segments > that are about to be merged, and keep the buffer around for the > remaining segments. > I think it's not so hard to do; we'd have to have generations of > pending deletions, because the newly merged segment doesn't need the > same buffered deletions applied again. So every time a merge kicks > off, we pinch off the current set of buffered deletions, open a new > set (the next generation), and record which segment was created as of > which generation. > This should be a very sizable gain for large indices that mix > deletes, though, less so in flex since opening the terms index is much > faster. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org