I think this explanation is quite clear. Sorry for mixing up TProtocol and TSocket.
You have correctly pointed out that UserContext is set per network-connection.. Many thanks for being patient and answering all noob-queries... -- Ravi On Tue, Apr 29, 2014 at 6:33 PM, Aaron McCurry <[email protected]> wrote: > On Tue, Apr 29, 2014 at 8:16 AM, Ravikumar Govindarajan < > [email protected]> wrote: > > > Thanks Aaron for the explanation... My problem comes with the lack of > > knowledge about Thrift internals... > > > > We create TProtocol object which wraps a Socket connection.. Will > > thrift-server always associate/cache a thread for this TProtocol, so that > > any number of requests issued on this TProtocol object will re-use the > same > > thread in Thrift-Server... > > > > First off if you are using Java I would strongly suggest that you use: > > > http://incubator.apache.org/blur/docs/0.2.0/using-blur.html#simple_connection_example_thrift > > To get the Iface client with everything setup so that you can just use the > UserContext class for handling everything. > > > https://git-wip-us.apache.org/repos/asf?p=incubator-blur.git;a=blob;f=blur-util/src/main/java/org/apache/blur/user/UserContext.java;h=0c3dd5f94af1813206fcc4d5d173919d76c4ca9f;hb=apache-blur-0.2 > > If you are not using Java or want to manage the connections yourself. Then > the User setup is configured per connection to each server. So not the > TProtocol but rather the TTransport (or TSocket in this case). So by > calling setUser on the Iface instance that is backed by a network > connection you are telling the server that the connection is for this > User. Any calls after that will be assumed to be the User that was set. > > Does this answer your question? > > > > > > By this way, when we call setUser() and then mutate(), the same thread > will > > be re-used and hence User object is readily available as a > thread-local... > > > > So there is some context that needs to be set here. From the server's > perspective, the setUser call sets a value in a server side session for the > given network connection. Then the User is set inside the server in a > ThreadLocal variable for all subsequent calls on the same network > connection. > > On the client side if you are NOT using the BlurClient class for your > connection setup, then the setUser call needs to be setup before each call > (mutate, query, etc.). If you are using the BlurClient then that > UserContext class and the BlurClient handles making the setUser calls for > you when it's needed. > > Aaron > > > > > > Am I correct in my understanding? > > > > -- > > Ravi > > > > > > > > On Fri, Apr 25, 2014 at 7:39 PM, Aaron McCurry <[email protected]> > wrote: > > > > > On Fri, Apr 25, 2014 at 8:29 AM, Ravikumar Govindarajan < > > > [email protected]> wrote: > > > > > > > > > > > > > If you want to setup a User for a series of requests to and from > the > > > Blur > > > > > cluster, use UserContext.setUser(...) and UserContext.reset() and > the > > > > > calling thread. Between those 2 calls all requests to the > > controllers > > > > (and > > > > > shard servers) will be performed as that User. > > > > > > > > > > > > I am not sure I understood your explanation. > > > > > > > > Every call for our mutation/query holds a different User and calls > > happen > > > > in parallel.. > > > > > > > > > > Understood. > > > > > > > > > > > > > > In other words, > > > > > > > > Every request is allocated a thread in controller & shard-servers. > This > > > > thread must unmarshall the User object and set it in it's > ThreadLocal. > > > > > > > > Is this current behavior of blur? > > > > > > > > > > The UserContext object handles this for you. The biggest issue with > the > > > BlurClientManager (or BlurClient) class is that if you are running > > multiple > > > blur controllers then each request can go to any of the controllers. > So > > if > > > you call setUser first and the call query, those are 2 different thrift > > > calls and they likely will be sent to different controllers. If you > use > > > the UserContext object (which uses ThreadLocals) then it will ensure > that > > > the setUser happens on the correct controller for the correct > controller > > > session. > > > > > > Does this help? > > > > > > Aaron > > > > > > > > > > > > > > -- > > > > Ravi > > > > > > > > > > > > > > > > On Fri, Apr 25, 2014 at 5:20 PM, Aaron McCurry <[email protected]> > > > wrote: > > > > > > > > > If you want to setup a User for a series of requests to and from > the > > > Blur > > > > > cluster, use UserContext.setUser(...) and UserContext.reset() and > the > > > > > calling thread. Between those 2 calls all requests to the > > controllers > > > > (and > > > > > shard servers) will be performed as that User. > > > > > > > > > > Also something else that's new is the trace function. > > > > > > > > > > To see it in action fire up the blur shell and run the trace > command > > > > > followed by any other command like a query command (in the shell). > > > Then > > > > go > > > > > a status page on a Blur process (http://<hostname>:40090 for shard > > > > servers > > > > > 40080 for controllers). Click on the trace tab and see what the > > trace > > > > call > > > > > actually did. It's a nice feature for debugging performance > > problems. > > > > > > > > > > Aaron > > > > > > > > > > > > > > > On Fri, Apr 25, 2014 at 7:20 AM, Ravikumar Govindarajan < > > > > > [email protected]> wrote: > > > > > > > > > > > Thanks Aaron for the clarification... Will try as you suggest. > > > > > > > > > > > > BTW, > > > > > > > > > > > > I find a setUser() method in Iface class [thrift interface] > > > > > > > > > > > > We need this "User" object passed from BlurClient to > > ControllerServer > > > > and > > > > > > then to ShardServer for every incoming query. > > > > > > > > > > > > Will this happen automatically in current code via thrift > > > > > > marshal/unmarshal? > > > > > > > > > > > > -- > > > > > > Ravi > > > > > > > > > > > > > > > > > > On Wed, Apr 23, 2014 at 4:34 PM, Aaron McCurry < > [email protected] > > > > > > > > wrote: > > > > > > > > > > > > > On Wed, Apr 23, 2014 at 1:53 AM, Ravikumar Govindarajan < > > > > > > > [email protected]> wrote: > > > > > > > > > > > > > > > Thanks Aaron, will definitely look at moving to 0.2.2 > > version... > > > > > > > > > > > > > > > > There is one problem when performing commit on every mutate > > call, > > > > > > which I > > > > > > > > need your help in understanding > > > > > > > > > > > > > > > > Our NRT reader re-opens every 10 seconds... Approx... 10 > > seconds > > > > > worth > > > > > > of > > > > > > > > data is buffered in IndexWriter RAM and then flushed to disk. > > > With > > > > > > 0.2.2, > > > > > > > > every mutate is auto flushed to disk because of commit... > This > > > will > > > > > > > create > > > > > > > > a lot of tiny segments and problem gets exacerbated when > > > > > MergeScheduler > > > > > > > > starts accumulating backlog... > > > > > > > > > > > > > > > > In the case of deletes/updates, the calling thread previously > > > will > > > > > > return > > > > > > > > immediately, adding the operation to IW RAM-Buffer. A > > background > > > > NRT > > > > > > > thread > > > > > > > > will apply these deletes. But now, the calling thread is > > stalled > > > > > until > > > > > > > the > > > > > > > > delete/update is successfully performed... > > > > > > > > > > > > > > > > One option is to accumulate change-sets and push it to Blur > for > > > > > every 5 > > > > > > > > seconds etc..., instead of pushing immediately to blur... > > > > > > > > > > > > > > > > > > > > > > There is a call in 0.2.2 that does this by default (or close). > > > It's > > > > > > called > > > > > > > enqueueMutate. Feature wise it gives you NRT updates with a > > buffer > > > > in > > > > > > > front of the writer. It delays the visibility of data but > allows > > > > for a > > > > > > > much higher volume of mutates without blocking. > > > > > > > > > > > > > > Aaron > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > Ravi > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Apr 22, 2014 at 10:25 PM, Aaron McCurry < > > > > [email protected]> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > I think that you are using 0.2.0. The > > > DirectoryReferenceCounter > > > > > was > > > > > > a > > > > > > > > > flawed piece of code and has been completely removed in > > 0.2.2. > > > A > > > > > > much > > > > > > > > > simpler and more reliable piece of code now exists that > > handles > > > > the > > > > > > > same > > > > > > > > > function. I can point you to the new way it's all > > implemented > > > if > > > > > you > > > > > > > > would > > > > > > > > > like. However I would suggest that you not use 0.2.0 > because > > > of > > > > > > these > > > > > > > > > issues as well as many other stability problems. > > > > > > > > > > > > > > > > > > Aaron > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Apr 22, 2014 at 4:21 AM, Ravikumar Govindarajan < > > > > > > > > > [email protected]> wrote: > > > > > > > > > > > > > > > > > > > DirectoryReferenceCounter wraps an IndexInput with an > > > > > AtomicInteger > > > > > > > for > > > > > > > > > > ref-count. > > > > > > > > > > > > > > > > > > > > I believe this file exists, because lucene does not > > correctly > > > > > > handle > > > > > > > > > > clones. Is this right? > > > > > > > > > > > > > > > > > > > > Rarely for me, an ongoing search hits > FileNotFoundException > > > > > > because a > > > > > > > > > > just-now completed merge has deleted the file... > > > > > > > > > > > > > > > > > > > > I am yet to pinpoint the problem, as it is quite > difficult > > to > > > > > > > reproduce > > > > > > > > > > it... > > > > > > > > > > > > > > > > > > > > I tried a sysout in deleteFile() method and was quite > > > surprised > > > > > to > > > > > > > find > > > > > > > > > > that refcounts are in negative for some-files.... Is this > > > > > normally > > > > > > > > > expected > > > > > > > > > > behavior? > > > > > > > > > > > > > > > > > > > > 14/04/22 13:32:35 INFO > > refcounter.DirectoryReferenceCounter: > > > > > Delete > > > > > > > > file > > > > > > > > > > [_2y_Lucene41_0.tip] with count [-14] > > > > > > > > > > > > > > > > > > > > 14/04/22 13:32:35 INFO > > refcounter.DirectoryReferenceCounter: > > > > > Delete > > > > > > > > file > > > > > > > > > [_ > > > > > > > > > > 2s.si] with count [0] > > > > > > > > > > > > > > > > > > > > 14/04/22 13:32:35 INFO > > refcounter.DirectoryReferenceCounter: > > > > > Delete > > > > > > > > file > > > > > > > > > > [segments_4] with count [0] > > > > > > > > > > > > > > > > > > > > 14/04/22 13:32:35 INFO > > refcounter.DirectoryReferenceCounter: > > > > > Delete > > > > > > > > file > > > > > > > > > > [_1z_Lucene41_0.doc] with count [-12] > > > > > > > > > > > > > > > > > > > > 14/04/22 13:32:35 INFO > > refcounter.DirectoryReferenceCounter: > > > > > Delete > > > > > > > > file > > > > > > > > > [_ > > > > > > > > > > 2y.si] with count [0] > > > > > > > > > > > > > > > > > > > > 14/04/22 13:32:35 INFO > > refcounter.DirectoryReferenceCounter: > > > > > Delete > > > > > > > > file > > > > > > > > > > [_2t.fnm] with count [-1] > > > > > > > > > > > > > > > > > > > > 14/04/22 13:32:35 INFO > > refcounter.DirectoryReferenceCounter: > > > > > Delete > > > > > > > > file > > > > > > > > > > [_24_Lucene41_0.tip] with count [-14] > > > > > > > > > > > > > > > > > > > > 14/04/22 13:32:35 INFO > > refcounter.DirectoryReferenceCounter: > > > > > Delete > > > > > > > > file > > > > > > > > > > [_24_Lucene41_0.tim] with count [-12] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > Ravi > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
