Re: IndexWriter flush/commit exception

2013-12-19 Thread Michael McCandless
On Wed, Dec 18, 2013 at 11:34 PM, Ravikumar Govindarajan wrote: >> You could make a custom Dir wrapper that always caches in RAM, but >> that sounds a bit terrifying :) > > This was exactly what I implemented:) I see :) > A commit-thread runs periodically > every 30 seconds, while RAM-Monitor th

Re: IndexWriter flush/commit exception

2013-12-18 Thread Ravikumar Govindarajan
> You could make a custom Dir wrapper that always caches in RAM, but > that sounds a bit terrifying :) This was exactly what I implemented:) A commit-thread runs periodically every 30 seconds, while RAM-Monitor thread runs every 5 seconds to commit data in-case sizeInBytes>=70%-of-maxCachedBytes.

Re: IndexWriter flush/commit exception

2013-12-18 Thread Michael McCandless
On Wed, Dec 18, 2013 at 3:15 AM, Ravikumar Govindarajan wrote: > Thanks Mike for a great explanation on Flush IOException You're welcome! > I was thinking on the perspective of a HDFSDirectory. In addition to the > all causes of IOException during flush you have listed, a HDFSDirectory > also ha

Re: IndexWriter flush/commit exception

2013-12-18 Thread Ravikumar Govindarajan
Thanks Mike for a great explanation on Flush IOException I was thinking on the perspective of a HDFSDirectory. In addition to the all causes of IOException during flush you have listed, a HDFSDirectory also has to deal with network issues, which is not lucene's problem at all. But I would ideally

Re: IndexWriter flush/commit exception

2013-12-17 Thread Michael McCandless
On Mon, Dec 16, 2013 at 7:33 AM, Ravikumar Govindarajan wrote: > I am trying to model a transaction-log for lucene, which creates a > transaction-log per-commit > > Things work fine during normal operations, but I cannot fathom the effect > during > > a. IOException during Index-Commit > > Will th

IndexWriter flush/commit exception

2013-12-16 Thread Ravikumar Govindarajan
I am trying to model a transaction-log for lucene, which creates a transaction-log per-commit Things work fine during normal operations, but I cannot fathom the effect during a. IOException during Index-Commit Will the index be restored to previous commit-point? Can I blindly re-try operations f