[
https://issues.apache.org/jira/browse/LUCENE-1526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12666676#action_12666676
]
Michael McCandless commented on LUCENE-1526:
--------------------------------------------
{quote}
I'm still creating performance tests, however the system is simply
using skipto, to obtain the next deleted doc, the next deleted doc is
cached in segmenttermdocs. So there shouldn't be a slowdown?
{quote}
I was actually thinking more generally that this is the natural
tradeoff one makes with realtime search. EG flushing a new segment
every document or two will necessarily give worse indexing throughput
than bulk indexing w/ large RAM buffer, but gives much faster
turnaround on searching.
But it sounds like you're talking specifically about performance of
switching to iterator access to deleted docs. I think the larger
number of [harder-for-cpu-to-predict] if statements may be the cause
of the slowdown once %tg deletes gets high enough? Also, if the
underlying skipTo is a linear scan over a non-sparse representation
then that's further cost.
> Tombstone deletions in IndexReader
> ----------------------------------
>
> Key: LUCENE-1526
> URL: https://issues.apache.org/jira/browse/LUCENE-1526
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Affects Versions: 2.4
> Reporter: Jason Rutherglen
> Priority: Minor
> Fix For: 2.9
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> SegmentReader currently uses a BitVector to represent deleted docs.
> When performing rapid clone (see LUCENE-1314) and delete operations,
> performing a copy on write of the BitVector can become costly because
> the entire underlying byte array must be created and copied. A way to
> make this clone delete process faster is to implement tombstones, a
> term coined by Marvin Humphrey. Tombstones represent new deletions
> plus the incremental deletions from previously reopened readers in
> the current reader.
> The proposed implementation of tombstones is to accumulate deletions
> into an int array represented as a DocIdSet. With LUCENE-1476,
> SegmentTermDocs iterates over deleted docs using a DocIdSet rather
> than accessing the BitVector by calling get. This allows a BitVector
> and a set of tombstones to by ANDed together as the current reader's
> delete docs.
> A tombstone merge policy needs to be defined to determine when to
> merge tombstone DocIdSets into a new deleted docs BitVector as too
> many tombstones would eventually be detrimental to performance. A
> probable implementation will merge tombstones based on the number of
> tombstones and the total number of documents in the tombstones. The
> merge policy may be set in the clone/reopen methods or on the
> IndexReader.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-dev-h...@lucene.apache.org
