[
https://issues.apache.org/jira/browse/LUCENE-5693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14005862#comment-14005862
]
Robert Muir commented on LUCENE-5693:
-------------------------------------
{quote}
I guess what bothers me here is this apparent precedent that deleted docs are
in fact required to be present everywhere in a segment. Yes, this is the case
today, but I think it's an impl detail and should not be required, e.g.
enforced by CheckIndex, tests asserting that it's the case.
{quote}
Thats not the case. I am worried about *bugs, complexities, and slowdowns in
lucene itself*. I already mentioned my list of concerns and I think they are
all realistic.
To me, the patch is a bit naive.
Perhaps you forgot (or didn't think about) what Sorted/SortedSetDocValuesWriter
would have to do, if it wanted to filter out deleted documents? This would slow
down flushing a lot, which presumably is important to people who are deleting
documents in IndexWriter's ramBuffer. Filtering out deleted documents here
would only *hurt* the user. Better to leave this to merge.
And what about stored fields and term vectors? why wouldn't you put a TODO
there in your patch? Is it because its "ok" to have the API and system
inconsistency there, because it would be slower to buffer them in RAM?
I don't like these implicit exceptions to the rule. If we want to intentionally
make a mess, there needs to be hard justification why we are doing such a
thing. All these little unproven optimizations, API inconsistencies,
exceptional cases, they all add up. I think it would be better to only
complicate things when its a big win, otherwise the whole codebase will end out
looking like IndexWriter.java.
All that being said, as I already stated on this issue, I am fine with
filtering out the postings as an exception to the rule. I really don't like it
one bit: but i can compromise for this piece if it really brings big benefits.
Doing it for the rest of the codec api makes no sense at all.
> don't write deleted documents on flush
> --------------------------------------
>
> Key: LUCENE-5693
> URL: https://issues.apache.org/jira/browse/LUCENE-5693
> Project: Lucene - Core
> Issue Type: Improvement
> Reporter: Michael McCandless
> Attachments: LUCENE-5693.patch
>
>
> When we flush a new segment, sometimes some documents are "born deleted",
> e.g. if the app did a IW.deleteDocuments that matched some not-yet-flushed
> documents.
> We already compute the liveDocs on flush, but then we continue (wastefully)
> to send those known-deleted documents to all Codec parts.
> I started to implement this on LUCENE-5675 but it was too controversial.
> Also, I expect typically the number of deleted docs is 0, or small, so not
> writing "born deleted" docs won't be much of a win for most apps. Still it
> seems silly to write them, consuming IO/CPU in the process, only to consume
> more IO/CPU later for merging to re-delete them.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
