Thanks Aaron for the clarification... Will try as you suggest. BTW,
I find a setUser() method in Iface class [thrift interface] We need this "User" object passed from BlurClient to ControllerServer and then to ShardServer for every incoming query. Will this happen automatically in current code via thrift marshal/unmarshal? -- Ravi On Wed, Apr 23, 2014 at 4:34 PM, Aaron McCurry <[email protected]> wrote: > On Wed, Apr 23, 2014 at 1:53 AM, Ravikumar Govindarajan < > [email protected]> wrote: > > > Thanks Aaron, will definitely look at moving to 0.2.2 version... > > > > There is one problem when performing commit on every mutate call, which I > > need your help in understanding > > > > Our NRT reader re-opens every 10 seconds... Approx... 10 seconds worth of > > data is buffered in IndexWriter RAM and then flushed to disk. With 0.2.2, > > every mutate is auto flushed to disk because of commit... This will > create > > a lot of tiny segments and problem gets exacerbated when MergeScheduler > > starts accumulating backlog... > > > > In the case of deletes/updates, the calling thread previously will return > > immediately, adding the operation to IW RAM-Buffer. A background NRT > thread > > will apply these deletes. But now, the calling thread is stalled until > the > > delete/update is successfully performed... > > > > One option is to accumulate change-sets and push it to Blur for every 5 > > seconds etc..., instead of pushing immediately to blur... > > > > There is a call in 0.2.2 that does this by default (or close). It's called > enqueueMutate. Feature wise it gives you NRT updates with a buffer in > front of the writer. It delays the visibility of data but allows for a > much higher volume of mutates without blocking. > > Aaron > > > > > > -- > > Ravi > > > > > > On Tue, Apr 22, 2014 at 10:25 PM, Aaron McCurry <[email protected]> > > wrote: > > > > > I think that you are using 0.2.0. The DirectoryReferenceCounter was a > > > flawed piece of code and has been completely removed in 0.2.2. A much > > > simpler and more reliable piece of code now exists that handles the > same > > > function. I can point you to the new way it's all implemented if you > > would > > > like. However I would suggest that you not use 0.2.0 because of these > > > issues as well as many other stability problems. > > > > > > Aaron > > > > > > > > > On Tue, Apr 22, 2014 at 4:21 AM, Ravikumar Govindarajan < > > > [email protected]> wrote: > > > > > > > DirectoryReferenceCounter wraps an IndexInput with an AtomicInteger > for > > > > ref-count. > > > > > > > > I believe this file exists, because lucene does not correctly handle > > > > clones. Is this right? > > > > > > > > Rarely for me, an ongoing search hits FileNotFoundException because a > > > > just-now completed merge has deleted the file... > > > > > > > > I am yet to pinpoint the problem, as it is quite difficult to > reproduce > > > > it... > > > > > > > > I tried a sysout in deleteFile() method and was quite surprised to > find > > > > that refcounts are in negative for some-files.... Is this normally > > > expected > > > > behavior? > > > > > > > > 14/04/22 13:32:35 INFO refcounter.DirectoryReferenceCounter: Delete > > file > > > > [_2y_Lucene41_0.tip] with count [-14] > > > > > > > > 14/04/22 13:32:35 INFO refcounter.DirectoryReferenceCounter: Delete > > file > > > [_ > > > > 2s.si] with count [0] > > > > > > > > 14/04/22 13:32:35 INFO refcounter.DirectoryReferenceCounter: Delete > > file > > > > [segments_4] with count [0] > > > > > > > > 14/04/22 13:32:35 INFO refcounter.DirectoryReferenceCounter: Delete > > file > > > > [_1z_Lucene41_0.doc] with count [-12] > > > > > > > > 14/04/22 13:32:35 INFO refcounter.DirectoryReferenceCounter: Delete > > file > > > [_ > > > > 2y.si] with count [0] > > > > > > > > 14/04/22 13:32:35 INFO refcounter.DirectoryReferenceCounter: Delete > > file > > > > [_2t.fnm] with count [-1] > > > > > > > > 14/04/22 13:32:35 INFO refcounter.DirectoryReferenceCounter: Delete > > file > > > > [_24_Lucene41_0.tip] with count [-14] > > > > > > > > 14/04/22 13:32:35 INFO refcounter.DirectoryReferenceCounter: Delete > > file > > > > [_24_Lucene41_0.tim] with count [-12] > > > > > > > > > > > > -- > > > > > > > > Ravi > > > > > > > > > >
