[
https://issues.apache.org/jira/browse/LUCENE-5987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael McCandless updated LUCENE-5987:
---------------------------------------
Attachment: LUCENE-5987.patch
Patch. I added new AbortIndexWriterException; it is package private,
and is only used to communicate internally in IndexWriter (other
approaches I tried kept leading to deadlock). I removed the
"andReset" from DocumentsWriterPerThread.checkAndResetHasAborted: I
think it's dangerous to ever set hasAborted back to false. It's like
unplugging your carbon monoxide detector because it's making too much
noise.
This is an important behavior change.
First off, an aborting exception, which is an exception that strikes
at a "bad" time (e.g. when appending to stored fields or term vector
files) such that the entire segment is now unusable, will now
forcefully close the IndexWriter. This is so you know you lost any
uncommitted data.
Second off, when this happens, I don't think IndexWriter should try to
be so crazy about deleting every last unref'd file when disaster
strikes: if your house is burning down, you don't worry about washing
the dirty dishes.
This change made a number of tests angry (IW suddenly closing, and
also leaving unref'd files), and I did distributed beasting to try to
ferret them out, but I expect we'll have a long tail of Jenkins
failures after committing this.
> Make indexwriter a mere mortal when exceptions strike
> -----------------------------------------------------
>
> Key: LUCENE-5987
> URL: https://issues.apache.org/jira/browse/LUCENE-5987
> Project: Lucene - Core
> Issue Type: Task
> Reporter: Robert Muir
> Assignee: Michael McCandless
> Attachments: LUCENE-5987.patch
>
>
> IndexWriter's exception handling is overly complicated. Every method in
> general reads like this:
> {code}
> try {
> try {
> try {
> ...
> // lock order: COMPLICATED
> synchronized(this or that) {
> }
> ...
> } finally {
> if (!success5) {
> deleter.deleteThisFileOrThat();
> }
> ...
> }
> }
> {code}
> Part of the problem is it acts like its an invincible superhero, e.g. can
> take a disk full on merge or flush to the face and just keep on trucking, and
> you can somehow fix the root cause and then just go about making commits on
> the same instance.
> But we have a hard enough time ensuring exceptions dont do the wrong thing
> (e.g. cause corruption), and I don't think we really test this crazy behavior
> anywhere: e.g. making commits AFTER hitting disk full and so on.
> It would probably be simpler if when such things happen, IW just considered
> them "tragic" just like OOM and rolled itself back, instead of doing all
> kinds of really scary stuff to try to "keep itself healthy" (like the little
> dance it plays with IFD in mergeMiddle manually deleting CFS files).
> Besides, without something like a WAL, Indexwriter isn't really fit to be a
> superhero anyway: it can't prevent you from losing data in such situations.
> It just doesn't have the right tools for the job.
> edit: just to be clear I am referring to abort (low level exception during
> flush) and exceptions during merge. For simple non-aborting cases like
> analyzer errors, of course we can deal with this. We already made great
> progress on turning a lot of BS exceptions that would cause aborts into
> non-aborting ones recently.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
