On Wed, Apr 7, 2010 at 3:27 PM, Earwin Burrfoot <ear...@gmail.com> wrote: >> No, this doesn't make sense. The OS detects a disk full on accepting >> the write into the write cache, not [later] on flushing the write >> cache to disk. If the OS accepts the write, then disk is not full (ie >> flushing the cache will succeed, unless some other not-disk-full >> problem happens). >> >> Hmmm, at least, normally. What OS/IO system were you on when you saw >> corruption due to disk full when fsync is disabled? >> >> I'm still skeptical that disk full even with fsync disabled can lead >> to corruption.... I'd like to see some concrete proof :) > > Linux 2.6.30-1-amd64, ext3, simple scsi drive
Hmmmmm. Linux should detect disk full on the initial write. > I checked with our resident DB brainiac, he says such things are possible. > > Okay, I'm not 100% sure this is the cause of my corruptions. It just happened > that when the index got corrupted, disk space was also used up - several > times. > I had that silent-fail-to-write theory and checked it up with some > knowledgeable > people. Even if they are right, I can be mistaken and the root cause > is different. OK... if you get a more concrete case where disk full causes corruption when you disable fsync, please post details back. From what I understand this should never happen. >> You're mixing up terminology a bit here -- you can't "hold on to the >> latest commit then switch to it". A commit (as sent to the deletion >> policy) means a *real* commit (ie, IW.commit or IW.close was called). >> So I think your BG thread would simply be calling IW.commit every N >> seconds? > Under "hold on to" I meant - keep from being deleted, like SnapshotDP does. But, IW doesn't let you "hold on to" checkpoints... only to commits. Ie SnapshotDP will only "see" actual commit/close calls, not intermediate checkpoints like a random segment merge completing, a flush happening, etc. Or... maybe you would in fact call commit frequently from the main threads (but with fsync disabled), and then your DP holds onto these "fake commits", periodically picking one of them to do the "real" fsync ing? >>> I'm just playing around with stupid idea. I'd like to have NRT >>> look-alike without binding readers and writers. :) >> I see... well binding durability & visibility will always be costly. >> This is why Lucene decouples them (by making NRT readers available). > My experiments do the same, essentially. > > But after I understood that to perform deletions IW has to load term indexes > anyway, I'm almost ready to give up and go for intertwined IW/IR mess :) Hey if you really think it's a mess, post a patch that cleans it up :) >> BTW, if you know your OS/IO system always persists cached writes w/in >> N seconds, a safe way to avoid fsync is to use a by-time expiring >> deletion policy. Ie, a commit stays alive as long as its age is less >> than X... DP's unit test has such a policy. But you better really >> know for sure that the OS/IO system guarantee that :) > Yeah. I thought of it, but it is even more shady :) I agree. And even if you know you're on Linux, and that your pdflush flushes after X seconds, you still have the IO system to contend with. Best to stick with fsync, commit only for safety as needed by the app, and use NRT for fast visibility. Mike --------------------------------------------------------------------- To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org For additional commands, e-mail: java-dev-h...@lucene.apache.org
