[
https://issues.apache.org/jira/browse/LUCENE-2593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael McCandless updated LUCENE-2593:
---------------------------------------
Attachment: LUCENE-2593.patch
Attached patch; I ended up making a number of defensive fixes on how
IW/DW/SR handle exceptions:
* Generally I moved the error recover down lower, eg SegmentReader
now restores its SegmentInfo and deletes the partially written
file, on hitting an exception writing changed norms or deletes.
* IW's ReaderPool no longer forcefully drops changes if it hits an
exception committing an SR. The SR now remains pooled, holding
onto its changes, in case a future commit is attempted and the SR
is able to commit.
* We checkpoint with IndexFileDeleter more "finely" now, so that as
soon as a new file is referenced (eg from writing deletes), IFD
knows about it. This prevents incorrect deletion of a file eg if
a merge IncRefs and then DecRefs before we can checkpoint.
I believe this issues goes back to 2.9, when we added reader pooling
(for NRT).
> disk full can cause index corruption in certain cases
> -----------------------------------------------------
>
> Key: LUCENE-2593
> URL: https://issues.apache.org/jira/browse/LUCENE-2593
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Fix For: 2.9.4, 3.0.3, 3.1, 4.0
>
> Attachments: LUCENE-2593.patch
>
>
> Robert uncovered this nasty bug, in adding more randomness to
> oal.index tests...
> I got a standalone test to show the issue; the corruption path is
> as follows:
> * The merge hits an initial exception (eg disk full when merging the
> postings).
> * In handling this exception, IW closes all the sub-readers,
> suppressing any further exceptions.
> * If one of these sub-readers has pending deletions, which happens
> if readers are pooled in IW, it will flush them. If that flush
> hits a 2nd exception (eg disk full), then SegmentReader
> [incorrectly] leaves the SegmentInfo's delGen advanced by 1,
> referencing a corrupt file, yet the SegmentReader is still
> forcefully closed.
> * If enough disk frees up such that a later IW.commit/close
> succeeds, the resulting segments file will reference an invalid
> deletions file.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
