[
https://issues.apache.org/jira/browse/LUCENE-2047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12779093#action_12779093
]
Michael McCandless commented on LUCENE-2047:
--------------------------------------------
{quote}
bq. 1: Analyzing hits an exception for a doc, it's doc id has already been
allocated so we mark it for deletion later (on flush?) in BufferedDeletes.
So there's only one use case right now, which is only when
analyzing an individual doc fails. The update doc adds the term
to the BufferedDeletes for later application.
{quote}
No, it adds the docID for later application. This is the one case
(entirely internal to IW) where we delete by docID in the writer.
{quote}
I think we can resolve the update doc term in the foreground. I'm
wondering if we need a different doc id queue for these? I get
the hunch yes, because the other doc ids need to be applied even
on IO exception, whereas update doc id will not be applied?
{quote}
I think we can use the same queue -- whether they are applied or not
follows exactly the same logic (ie, successful flush moves all
deletesInRAM over to deletesFlushed), ie, an aborting exception
cleares the deletesInRAM.
{quote}
2: RAM Buffer writing hits an exception, we've had
updates which marked deletes in current segments, however they
haven't been applied yet because they're stored in
BufferedDeletes docids. They're applied on successful flush.
In essence we need to implement number 2?
{quote}
I'm confused -- #2 is already what's implemented in IW, today.
The changes on the table now are to:
* Immediately resolve deleted (updated) terms/queries -> docIDs
* Change how we enqueue docIDs to be per-SR, instead.
* But: you still must buffer Term/Query for the current RAM buffer,
and on flushing it to a real segment, resolve them to docIDs.
Otherwise we don't need to change what's done today (ie, keep the
deletesInRAM vs deletesFlushed)?
bq. What is an example of a non-aborting exception?
Anything hit during analysis (ie, TokenStream.incrementToken()), or
anywhere except DocInverterPerField in the indexing chain (eg if we
hit something when writing the stored fields, I don't think it'll
abort).
{quote}
I'm browsing through the applyDeletes call path, I'm tempted to
rework how we're doing this.
{quote}
I think this would be a good improvement -- it would mean we don't
need to ever remapDeletes, right?
The thing is... this has to work in non-NRT mode, too. Somehow, you
have to buffer such a deleted docID against the next segment to be
flushed (ie the current RAM buffer). And once it's flushed, we move
the docID from deletesInRAM (stored per-SR) to the SR's actual deleted
docs BV.
We would still keep the deletes partitioned, into those done during
the current RAM segment vs those successfully flushed, right?
bq. I can't find the code that handles aborts.
It's DW's abort() method, and eg in DocInverterPerField we call
DW.setAborting on exception to note that this exception is an aborting
one.
> IndexWriter should immediately resolve deleted docs to docID in
> near-real-time mode
> -----------------------------------------------------------------------------------
>
> Key: LUCENE-2047
> URL: https://issues.apache.org/jira/browse/LUCENE-2047
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Priority: Minor
> Fix For: 3.1
>
> Attachments: LUCENE-2047.patch, LUCENE-2047.patch
>
>
> Spinoff from LUCENE-1526.
> When deleteDocuments(Term) is called, we currently always buffer the
> Term and only later, when it's time to flush deletes, resolve to
> docIDs. This is necessary because we don't in general hold
> SegmentReaders open.
> But, when IndexWriter is in NRT mode, we pool the readers, and so
> deleting in the foreground is possible.
> It's also beneficial, in that in can reduce the turnaround time when
> reopening a new NRT reader by taking this resolution off the reopen
> path. And if multiple threads are used to do the deletion, then we
> gain concurrency, vs reopen which is not concurrent when flushing the
> deletes.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-dev-h...@lucene.apache.org
