Thanks Aaron, will definitely look at moving to 0.2.2 version... There is one problem when performing commit on every mutate call, which I need your help in understanding
Our NRT reader re-opens every 10 seconds... Approx... 10 seconds worth of data is buffered in IndexWriter RAM and then flushed to disk. With 0.2.2, every mutate is auto flushed to disk because of commit... This will create a lot of tiny segments and problem gets exacerbated when MergeScheduler starts accumulating backlog... In the case of deletes/updates, the calling thread previously will return immediately, adding the operation to IW RAM-Buffer. A background NRT thread will apply these deletes. But now, the calling thread is stalled until the delete/update is successfully performed... One option is to accumulate change-sets and push it to Blur for every 5 seconds etc..., instead of pushing immediately to blur... -- Ravi On Tue, Apr 22, 2014 at 10:25 PM, Aaron McCurry <[email protected]> wrote: > I think that you are using 0.2.0. The DirectoryReferenceCounter was a > flawed piece of code and has been completely removed in 0.2.2. A much > simpler and more reliable piece of code now exists that handles the same > function. I can point you to the new way it's all implemented if you would > like. However I would suggest that you not use 0.2.0 because of these > issues as well as many other stability problems. > > Aaron > > > On Tue, Apr 22, 2014 at 4:21 AM, Ravikumar Govindarajan < > [email protected]> wrote: > > > DirectoryReferenceCounter wraps an IndexInput with an AtomicInteger for > > ref-count. > > > > I believe this file exists, because lucene does not correctly handle > > clones. Is this right? > > > > Rarely for me, an ongoing search hits FileNotFoundException because a > > just-now completed merge has deleted the file... > > > > I am yet to pinpoint the problem, as it is quite difficult to reproduce > > it... > > > > I tried a sysout in deleteFile() method and was quite surprised to find > > that refcounts are in negative for some-files.... Is this normally > expected > > behavior? > > > > 14/04/22 13:32:35 INFO refcounter.DirectoryReferenceCounter: Delete file > > [_2y_Lucene41_0.tip] with count [-14] > > > > 14/04/22 13:32:35 INFO refcounter.DirectoryReferenceCounter: Delete file > [_ > > 2s.si] with count [0] > > > > 14/04/22 13:32:35 INFO refcounter.DirectoryReferenceCounter: Delete file > > [segments_4] with count [0] > > > > 14/04/22 13:32:35 INFO refcounter.DirectoryReferenceCounter: Delete file > > [_1z_Lucene41_0.doc] with count [-12] > > > > 14/04/22 13:32:35 INFO refcounter.DirectoryReferenceCounter: Delete file > [_ > > 2y.si] with count [0] > > > > 14/04/22 13:32:35 INFO refcounter.DirectoryReferenceCounter: Delete file > > [_2t.fnm] with count [-1] > > > > 14/04/22 13:32:35 INFO refcounter.DirectoryReferenceCounter: Delete file > > [_24_Lucene41_0.tip] with count [-14] > > > > 14/04/22 13:32:35 INFO refcounter.DirectoryReferenceCounter: Delete file > > [_24_Lucene41_0.tim] with count [-12] > > > > > > -- > > > > Ravi > > >
