So, I want to pump my IndexWriter hard and fast with documents. Removing fsync from FSDirectory helps. But for that I pay with possibility of index corruption, not only if my node suddenly loses power/kernelpanics, but also if it runs out of disk space (which happens more frequently).
I invented the following solution: We write a special deletion policy that resembles SnapshotDeletionPolicy. At all times it takes hold of "current synced commit" and preserves it. Once every N minutes a special thread takes latest commit, syncs it and nominates as "current synced commit". The previous one gets deleted. Now we are disastery-proof, and do fsync asynchronously from indexing threads. We pay for this with somewhat bigger transient disc usage, and probably losing a few minutes worth of updates in case of a crash, but that's acceptable. How does this sound? -- Kirill Zakharenko/Кирилл Захаренко (ear...@gmail.com) Home / Mobile: +7 (495) 683-567-4 / +7 (903) 5-888-423 ICQ: 104465785 --------------------------------------------------------------------- To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org For additional commands, e-mail: java-dev-h...@lucene.apache.org
