On Tue, Apr 29, 2014 at 8:16 AM, Ravikumar Govindarajan < [email protected]> wrote:
> Thanks Aaron for the explanation... My problem comes with the lack of > knowledge about Thrift internals... > > We create TProtocol object which wraps a Socket connection.. Will > thrift-server always associate/cache a thread for this TProtocol, so that > any number of requests issued on this TProtocol object will re-use the same > thread in Thrift-Server... > First off if you are using Java I would strongly suggest that you use: http://incubator.apache.org/blur/docs/0.2.0/using-blur.html#simple_connection_example_thrift To get the Iface client with everything setup so that you can just use the UserContext class for handling everything. https://git-wip-us.apache.org/repos/asf?p=incubator-blur.git;a=blob;f=blur-util/src/main/java/org/apache/blur/user/UserContext.java;h=0c3dd5f94af1813206fcc4d5d173919d76c4ca9f;hb=apache-blur-0.2 If you are not using Java or want to manage the connections yourself. Then the User setup is configured per connection to each server. So not the TProtocol but rather the TTransport (or TSocket in this case). So by calling setUser on the Iface instance that is backed by a network connection you are telling the server that the connection is for this User. Any calls after that will be assumed to be the User that was set. Does this answer your question? > > By this way, when we call setUser() and then mutate(), the same thread will > be re-used and hence User object is readily available as a thread-local... > So there is some context that needs to be set here. From the server's perspective, the setUser call sets a value in a server side session for the given network connection. Then the User is set inside the server in a ThreadLocal variable for all subsequent calls on the same network connection. On the client side if you are NOT using the BlurClient class for your connection setup, then the setUser call needs to be setup before each call (mutate, query, etc.). If you are using the BlurClient then that UserContext class and the BlurClient handles making the setUser calls for you when it's needed. Aaron > > Am I correct in my understanding? > > -- > Ravi > > > > On Fri, Apr 25, 2014 at 7:39 PM, Aaron McCurry <[email protected]> wrote: > > > On Fri, Apr 25, 2014 at 8:29 AM, Ravikumar Govindarajan < > > [email protected]> wrote: > > > > > > > > > > If you want to setup a User for a series of requests to and from the > > Blur > > > > cluster, use UserContext.setUser(...) and UserContext.reset() and the > > > > calling thread. Between those 2 calls all requests to the > controllers > > > (and > > > > shard servers) will be performed as that User. > > > > > > > > > I am not sure I understood your explanation. > > > > > > Every call for our mutation/query holds a different User and calls > happen > > > in parallel.. > > > > > > > Understood. > > > > > > > > > > In other words, > > > > > > Every request is allocated a thread in controller & shard-servers. This > > > thread must unmarshall the User object and set it in it's ThreadLocal. > > > > > > Is this current behavior of blur? > > > > > > > The UserContext object handles this for you. The biggest issue with the > > BlurClientManager (or BlurClient) class is that if you are running > multiple > > blur controllers then each request can go to any of the controllers. So > if > > you call setUser first and the call query, those are 2 different thrift > > calls and they likely will be sent to different controllers. If you use > > the UserContext object (which uses ThreadLocals) then it will ensure that > > the setUser happens on the correct controller for the correct controller > > session. > > > > Does this help? > > > > Aaron > > > > > > > > > > -- > > > Ravi > > > > > > > > > > > > On Fri, Apr 25, 2014 at 5:20 PM, Aaron McCurry <[email protected]> > > wrote: > > > > > > > If you want to setup a User for a series of requests to and from the > > Blur > > > > cluster, use UserContext.setUser(...) and UserContext.reset() and the > > > > calling thread. Between those 2 calls all requests to the > controllers > > > (and > > > > shard servers) will be performed as that User. > > > > > > > > Also something else that's new is the trace function. > > > > > > > > To see it in action fire up the blur shell and run the trace command > > > > followed by any other command like a query command (in the shell). > > Then > > > go > > > > a status page on a Blur process (http://<hostname>:40090 for shard > > > servers > > > > 40080 for controllers). Click on the trace tab and see what the > trace > > > call > > > > actually did. It's a nice feature for debugging performance > problems. > > > > > > > > Aaron > > > > > > > > > > > > On Fri, Apr 25, 2014 at 7:20 AM, Ravikumar Govindarajan < > > > > [email protected]> wrote: > > > > > > > > > Thanks Aaron for the clarification... Will try as you suggest. > > > > > > > > > > BTW, > > > > > > > > > > I find a setUser() method in Iface class [thrift interface] > > > > > > > > > > We need this "User" object passed from BlurClient to > ControllerServer > > > and > > > > > then to ShardServer for every incoming query. > > > > > > > > > > Will this happen automatically in current code via thrift > > > > > marshal/unmarshal? > > > > > > > > > > -- > > > > > Ravi > > > > > > > > > > > > > > > On Wed, Apr 23, 2014 at 4:34 PM, Aaron McCurry <[email protected] > > > > > > wrote: > > > > > > > > > > > On Wed, Apr 23, 2014 at 1:53 AM, Ravikumar Govindarajan < > > > > > > [email protected]> wrote: > > > > > > > > > > > > > Thanks Aaron, will definitely look at moving to 0.2.2 > version... > > > > > > > > > > > > > > There is one problem when performing commit on every mutate > call, > > > > > which I > > > > > > > need your help in understanding > > > > > > > > > > > > > > Our NRT reader re-opens every 10 seconds... Approx... 10 > seconds > > > > worth > > > > > of > > > > > > > data is buffered in IndexWriter RAM and then flushed to disk. > > With > > > > > 0.2.2, > > > > > > > every mutate is auto flushed to disk because of commit... This > > will > > > > > > create > > > > > > > a lot of tiny segments and problem gets exacerbated when > > > > MergeScheduler > > > > > > > starts accumulating backlog... > > > > > > > > > > > > > > In the case of deletes/updates, the calling thread previously > > will > > > > > return > > > > > > > immediately, adding the operation to IW RAM-Buffer. A > background > > > NRT > > > > > > thread > > > > > > > will apply these deletes. But now, the calling thread is > stalled > > > > until > > > > > > the > > > > > > > delete/update is successfully performed... > > > > > > > > > > > > > > One option is to accumulate change-sets and push it to Blur for > > > > every 5 > > > > > > > seconds etc..., instead of pushing immediately to blur... > > > > > > > > > > > > > > > > > > > There is a call in 0.2.2 that does this by default (or close). > > It's > > > > > called > > > > > > enqueueMutate. Feature wise it gives you NRT updates with a > buffer > > > in > > > > > > front of the writer. It delays the visibility of data but allows > > > for a > > > > > > much higher volume of mutates without blocking. > > > > > > > > > > > > Aaron > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > Ravi > > > > > > > > > > > > > > > > > > > > > On Tue, Apr 22, 2014 at 10:25 PM, Aaron McCurry < > > > [email protected]> > > > > > > > wrote: > > > > > > > > > > > > > > > I think that you are using 0.2.0. The > > DirectoryReferenceCounter > > > > was > > > > > a > > > > > > > > flawed piece of code and has been completely removed in > 0.2.2. > > A > > > > > much > > > > > > > > simpler and more reliable piece of code now exists that > handles > > > the > > > > > > same > > > > > > > > function. I can point you to the new way it's all > implemented > > if > > > > you > > > > > > > would > > > > > > > > like. However I would suggest that you not use 0.2.0 because > > of > > > > > these > > > > > > > > issues as well as many other stability problems. > > > > > > > > > > > > > > > > Aaron > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Apr 22, 2014 at 4:21 AM, Ravikumar Govindarajan < > > > > > > > > [email protected]> wrote: > > > > > > > > > > > > > > > > > DirectoryReferenceCounter wraps an IndexInput with an > > > > AtomicInteger > > > > > > for > > > > > > > > > ref-count. > > > > > > > > > > > > > > > > > > I believe this file exists, because lucene does not > correctly > > > > > handle > > > > > > > > > clones. Is this right? > > > > > > > > > > > > > > > > > > Rarely for me, an ongoing search hits FileNotFoundException > > > > > because a > > > > > > > > > just-now completed merge has deleted the file... > > > > > > > > > > > > > > > > > > I am yet to pinpoint the problem, as it is quite difficult > to > > > > > > reproduce > > > > > > > > > it... > > > > > > > > > > > > > > > > > > I tried a sysout in deleteFile() method and was quite > > surprised > > > > to > > > > > > find > > > > > > > > > that refcounts are in negative for some-files.... Is this > > > > normally > > > > > > > > expected > > > > > > > > > behavior? > > > > > > > > > > > > > > > > > > 14/04/22 13:32:35 INFO > refcounter.DirectoryReferenceCounter: > > > > Delete > > > > > > > file > > > > > > > > > [_2y_Lucene41_0.tip] with count [-14] > > > > > > > > > > > > > > > > > > 14/04/22 13:32:35 INFO > refcounter.DirectoryReferenceCounter: > > > > Delete > > > > > > > file > > > > > > > > [_ > > > > > > > > > 2s.si] with count [0] > > > > > > > > > > > > > > > > > > 14/04/22 13:32:35 INFO > refcounter.DirectoryReferenceCounter: > > > > Delete > > > > > > > file > > > > > > > > > [segments_4] with count [0] > > > > > > > > > > > > > > > > > > 14/04/22 13:32:35 INFO > refcounter.DirectoryReferenceCounter: > > > > Delete > > > > > > > file > > > > > > > > > [_1z_Lucene41_0.doc] with count [-12] > > > > > > > > > > > > > > > > > > 14/04/22 13:32:35 INFO > refcounter.DirectoryReferenceCounter: > > > > Delete > > > > > > > file > > > > > > > > [_ > > > > > > > > > 2y.si] with count [0] > > > > > > > > > > > > > > > > > > 14/04/22 13:32:35 INFO > refcounter.DirectoryReferenceCounter: > > > > Delete > > > > > > > file > > > > > > > > > [_2t.fnm] with count [-1] > > > > > > > > > > > > > > > > > > 14/04/22 13:32:35 INFO > refcounter.DirectoryReferenceCounter: > > > > Delete > > > > > > > file > > > > > > > > > [_24_Lucene41_0.tip] with count [-14] > > > > > > > > > > > > > > > > > > 14/04/22 13:32:35 INFO > refcounter.DirectoryReferenceCounter: > > > > Delete > > > > > > > file > > > > > > > > > [_24_Lucene41_0.tim] with count [-12] > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > Ravi > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
