On Wed, Apr 23, 2014 at 1:53 AM, Ravikumar Govindarajan < [email protected]> wrote:
> Thanks Aaron, will definitely look at moving to 0.2.2 version... > > There is one problem when performing commit on every mutate call, which I > need your help in understanding > > Our NRT reader re-opens every 10 seconds... Approx... 10 seconds worth of > data is buffered in IndexWriter RAM and then flushed to disk. With 0.2.2, > every mutate is auto flushed to disk because of commit... This will create > a lot of tiny segments and problem gets exacerbated when MergeScheduler > starts accumulating backlog... > > In the case of deletes/updates, the calling thread previously will return > immediately, adding the operation to IW RAM-Buffer. A background NRT thread > will apply these deletes. But now, the calling thread is stalled until the > delete/update is successfully performed... > > One option is to accumulate change-sets and push it to Blur for every 5 > seconds etc..., instead of pushing immediately to blur... > There is a call in 0.2.2 that does this by default (or close). It's called enqueueMutate. Feature wise it gives you NRT updates with a buffer in front of the writer. It delays the visibility of data but allows for a much higher volume of mutates without blocking. Aaron > > -- > Ravi > > > On Tue, Apr 22, 2014 at 10:25 PM, Aaron McCurry <[email protected]> > wrote: > > > I think that you are using 0.2.0. The DirectoryReferenceCounter was a > > flawed piece of code and has been completely removed in 0.2.2. A much > > simpler and more reliable piece of code now exists that handles the same > > function. I can point you to the new way it's all implemented if you > would > > like. However I would suggest that you not use 0.2.0 because of these > > issues as well as many other stability problems. > > > > Aaron > > > > > > On Tue, Apr 22, 2014 at 4:21 AM, Ravikumar Govindarajan < > > [email protected]> wrote: > > > > > DirectoryReferenceCounter wraps an IndexInput with an AtomicInteger for > > > ref-count. > > > > > > I believe this file exists, because lucene does not correctly handle > > > clones. Is this right? > > > > > > Rarely for me, an ongoing search hits FileNotFoundException because a > > > just-now completed merge has deleted the file... > > > > > > I am yet to pinpoint the problem, as it is quite difficult to reproduce > > > it... > > > > > > I tried a sysout in deleteFile() method and was quite surprised to find > > > that refcounts are in negative for some-files.... Is this normally > > expected > > > behavior? > > > > > > 14/04/22 13:32:35 INFO refcounter.DirectoryReferenceCounter: Delete > file > > > [_2y_Lucene41_0.tip] with count [-14] > > > > > > 14/04/22 13:32:35 INFO refcounter.DirectoryReferenceCounter: Delete > file > > [_ > > > 2s.si] with count [0] > > > > > > 14/04/22 13:32:35 INFO refcounter.DirectoryReferenceCounter: Delete > file > > > [segments_4] with count [0] > > > > > > 14/04/22 13:32:35 INFO refcounter.DirectoryReferenceCounter: Delete > file > > > [_1z_Lucene41_0.doc] with count [-12] > > > > > > 14/04/22 13:32:35 INFO refcounter.DirectoryReferenceCounter: Delete > file > > [_ > > > 2y.si] with count [0] > > > > > > 14/04/22 13:32:35 INFO refcounter.DirectoryReferenceCounter: Delete > file > > > [_2t.fnm] with count [-1] > > > > > > 14/04/22 13:32:35 INFO refcounter.DirectoryReferenceCounter: Delete > file > > > [_24_Lucene41_0.tip] with count [-14] > > > > > > 14/04/22 13:32:35 INFO refcounter.DirectoryReferenceCounter: Delete > file > > > [_24_Lucene41_0.tim] with count [-12] > > > > > > > > > -- > > > > > > Ravi > > > > > >
