On Fri, Apr 25, 2014 at 8:29 AM, Ravikumar Govindarajan < [email protected]> wrote:
> > > > If you want to setup a User for a series of requests to and from the Blur > > cluster, use UserContext.setUser(...) and UserContext.reset() and the > > calling thread. Between those 2 calls all requests to the controllers > (and > > shard servers) will be performed as that User. > > > I am not sure I understood your explanation. > > Every call for our mutation/query holds a different User and calls happen > in parallel.. > Understood. > > In other words, > > Every request is allocated a thread in controller & shard-servers. This > thread must unmarshall the User object and set it in it's ThreadLocal. > > Is this current behavior of blur? > The UserContext object handles this for you. The biggest issue with the BlurClientManager (or BlurClient) class is that if you are running multiple blur controllers then each request can go to any of the controllers. So if you call setUser first and the call query, those are 2 different thrift calls and they likely will be sent to different controllers. If you use the UserContext object (which uses ThreadLocals) then it will ensure that the setUser happens on the correct controller for the correct controller session. Does this help? Aaron > > -- > Ravi > > > > On Fri, Apr 25, 2014 at 5:20 PM, Aaron McCurry <[email protected]> wrote: > > > If you want to setup a User for a series of requests to and from the Blur > > cluster, use UserContext.setUser(...) and UserContext.reset() and the > > calling thread. Between those 2 calls all requests to the controllers > (and > > shard servers) will be performed as that User. > > > > Also something else that's new is the trace function. > > > > To see it in action fire up the blur shell and run the trace command > > followed by any other command like a query command (in the shell). Then > go > > a status page on a Blur process (http://<hostname>:40090 for shard > servers > > 40080 for controllers). Click on the trace tab and see what the trace > call > > actually did. It's a nice feature for debugging performance problems. > > > > Aaron > > > > > > On Fri, Apr 25, 2014 at 7:20 AM, Ravikumar Govindarajan < > > [email protected]> wrote: > > > > > Thanks Aaron for the clarification... Will try as you suggest. > > > > > > BTW, > > > > > > I find a setUser() method in Iface class [thrift interface] > > > > > > We need this "User" object passed from BlurClient to ControllerServer > and > > > then to ShardServer for every incoming query. > > > > > > Will this happen automatically in current code via thrift > > > marshal/unmarshal? > > > > > > -- > > > Ravi > > > > > > > > > On Wed, Apr 23, 2014 at 4:34 PM, Aaron McCurry <[email protected]> > > wrote: > > > > > > > On Wed, Apr 23, 2014 at 1:53 AM, Ravikumar Govindarajan < > > > > [email protected]> wrote: > > > > > > > > > Thanks Aaron, will definitely look at moving to 0.2.2 version... > > > > > > > > > > There is one problem when performing commit on every mutate call, > > > which I > > > > > need your help in understanding > > > > > > > > > > Our NRT reader re-opens every 10 seconds... Approx... 10 seconds > > worth > > > of > > > > > data is buffered in IndexWriter RAM and then flushed to disk. With > > > 0.2.2, > > > > > every mutate is auto flushed to disk because of commit... This will > > > > create > > > > > a lot of tiny segments and problem gets exacerbated when > > MergeScheduler > > > > > starts accumulating backlog... > > > > > > > > > > In the case of deletes/updates, the calling thread previously will > > > return > > > > > immediately, adding the operation to IW RAM-Buffer. A background > NRT > > > > thread > > > > > will apply these deletes. But now, the calling thread is stalled > > until > > > > the > > > > > delete/update is successfully performed... > > > > > > > > > > One option is to accumulate change-sets and push it to Blur for > > every 5 > > > > > seconds etc..., instead of pushing immediately to blur... > > > > > > > > > > > > > There is a call in 0.2.2 that does this by default (or close). It's > > > called > > > > enqueueMutate. Feature wise it gives you NRT updates with a buffer > in > > > > front of the writer. It delays the visibility of data but allows > for a > > > > much higher volume of mutates without blocking. > > > > > > > > Aaron > > > > > > > > > > > > > > > > > > -- > > > > > Ravi > > > > > > > > > > > > > > > On Tue, Apr 22, 2014 at 10:25 PM, Aaron McCurry < > [email protected]> > > > > > wrote: > > > > > > > > > > > I think that you are using 0.2.0. The DirectoryReferenceCounter > > was > > > a > > > > > > flawed piece of code and has been completely removed in 0.2.2. A > > > much > > > > > > simpler and more reliable piece of code now exists that handles > the > > > > same > > > > > > function. I can point you to the new way it's all implemented if > > you > > > > > would > > > > > > like. However I would suggest that you not use 0.2.0 because of > > > these > > > > > > issues as well as many other stability problems. > > > > > > > > > > > > Aaron > > > > > > > > > > > > > > > > > > On Tue, Apr 22, 2014 at 4:21 AM, Ravikumar Govindarajan < > > > > > > [email protected]> wrote: > > > > > > > > > > > > > DirectoryReferenceCounter wraps an IndexInput with an > > AtomicInteger > > > > for > > > > > > > ref-count. > > > > > > > > > > > > > > I believe this file exists, because lucene does not correctly > > > handle > > > > > > > clones. Is this right? > > > > > > > > > > > > > > Rarely for me, an ongoing search hits FileNotFoundException > > > because a > > > > > > > just-now completed merge has deleted the file... > > > > > > > > > > > > > > I am yet to pinpoint the problem, as it is quite difficult to > > > > reproduce > > > > > > > it... > > > > > > > > > > > > > > I tried a sysout in deleteFile() method and was quite surprised > > to > > > > find > > > > > > > that refcounts are in negative for some-files.... Is this > > normally > > > > > > expected > > > > > > > behavior? > > > > > > > > > > > > > > 14/04/22 13:32:35 INFO refcounter.DirectoryReferenceCounter: > > Delete > > > > > file > > > > > > > [_2y_Lucene41_0.tip] with count [-14] > > > > > > > > > > > > > > 14/04/22 13:32:35 INFO refcounter.DirectoryReferenceCounter: > > Delete > > > > > file > > > > > > [_ > > > > > > > 2s.si] with count [0] > > > > > > > > > > > > > > 14/04/22 13:32:35 INFO refcounter.DirectoryReferenceCounter: > > Delete > > > > > file > > > > > > > [segments_4] with count [0] > > > > > > > > > > > > > > 14/04/22 13:32:35 INFO refcounter.DirectoryReferenceCounter: > > Delete > > > > > file > > > > > > > [_1z_Lucene41_0.doc] with count [-12] > > > > > > > > > > > > > > 14/04/22 13:32:35 INFO refcounter.DirectoryReferenceCounter: > > Delete > > > > > file > > > > > > [_ > > > > > > > 2y.si] with count [0] > > > > > > > > > > > > > > 14/04/22 13:32:35 INFO refcounter.DirectoryReferenceCounter: > > Delete > > > > > file > > > > > > > [_2t.fnm] with count [-1] > > > > > > > > > > > > > > 14/04/22 13:32:35 INFO refcounter.DirectoryReferenceCounter: > > Delete > > > > > file > > > > > > > [_24_Lucene41_0.tip] with count [-14] > > > > > > > > > > > > > > 14/04/22 13:32:35 INFO refcounter.DirectoryReferenceCounter: > > Delete > > > > > file > > > > > > > [_24_Lucene41_0.tim] with count [-12] > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > Ravi > > > > > > > > > > > > > > > > > > > > > > > > > > > >
