[ https://issues.apache.org/jira/browse/LUCENE-2593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless updated LUCENE-2593: --------------------------------------- Attachment: LUCENE-2593.patch Attached patch; I ended up making a number of defensive fixes on how IW/DW/SR handle exceptions: * Generally I moved the error recover down lower, eg SegmentReader now restores its SegmentInfo and deletes the partially written file, on hitting an exception writing changed norms or deletes. * IW's ReaderPool no longer forcefully drops changes if it hits an exception committing an SR. The SR now remains pooled, holding onto its changes, in case a future commit is attempted and the SR is able to commit. * We checkpoint with IndexFileDeleter more "finely" now, so that as soon as a new file is referenced (eg from writing deletes), IFD knows about it. This prevents incorrect deletion of a file eg if a merge IncRefs and then DecRefs before we can checkpoint. I believe this issues goes back to 2.9, when we added reader pooling (for NRT). > disk full can cause index corruption in certain cases > ----------------------------------------------------- > > Key: LUCENE-2593 > URL: https://issues.apache.org/jira/browse/LUCENE-2593 > Project: Lucene - Java > Issue Type: Bug > Components: Index > Reporter: Michael McCandless > Assignee: Michael McCandless > Fix For: 2.9.4, 3.0.3, 3.1, 4.0 > > Attachments: LUCENE-2593.patch > > > Robert uncovered this nasty bug, in adding more randomness to > oal.index tests... > I got a standalone test to show the issue; the corruption path is > as follows: > * The merge hits an initial exception (eg disk full when merging the > postings). > * In handling this exception, IW closes all the sub-readers, > suppressing any further exceptions. > * If one of these sub-readers has pending deletions, which happens > if readers are pooled in IW, it will flush them. If that flush > hits a 2nd exception (eg disk full), then SegmentReader > [incorrectly] leaves the SegmentInfo's delGen advanced by 1, > referencing a corrupt file, yet the SegmentReader is still > forcefully closed. > * If enough disk frees up such that a later IW.commit/close > succeeds, the resulting segments file will reference an invalid > deletions file. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org