I own the pull request that adds (very basic) pooling logic to the Riak client. I left Transports alone (each is a single connection) and decided to have pools be another class you pick just like transport. This allows you instantly use any pooling logic (never remove down servers, delete down servers, round robin, whatever) with *any* transport class: Http, Http keep-alive, PBC, CachedPBC, etc.
I've added pooling and timeouts and use both in production because I'm honestly not sure how you use Riak in a highly available way without them... so we need to make sure they both work well/easily under this new scheme. Do your changes make anything new possible or are they just about cleanup? On Thu, Sep 8, 2011 at 5:17 AM, Greg Stein <[email protected]> wrote: > Yup, agreed. I can certainly see use-cases where a client may have > particular knowledge about data/server locality, and wants to open > connections to *just* those servers. The transports that I am > advocating open connections to servers described by one layer higher. > That layer could say "all Riak servers in the cluster", or it could > say "these three servers holding copies of <this> data". > > An auto-discovery layer (to add and remove) in-between the client and > the transport would be handy. I can envision that, and my proposed > code makes it possible, but (at my current stage) I don't have a use > for it. I'd hope that others can build on my connection pooling work > to see that through. > > Cheers, > -g > > On Thu, Sep 8, 2011 at 07:49, Phil Stanhope <[email protected]> wrote: >> I like the idea of RiakTransport as you describe it. It opens the door to >> other potential underlying transports and isolating a client from knowledge >> of those (websockets, tcp, zeromq messaging come to mind). I'm not >> suggesting that RIAK will ever support these transports by this comment, >> however. >> I also agree with the need to have RiakRingAwareTransport as an additional >> layer that *might* be used by a client. There may be valid reasons why a >> client might want to force particular traffic onto a subset of the ring >> (e.g. M/R config, Search Config, forcing read/write traffic onto different >> nodes, etc). Again, I'm not suggesting that using a subset of the ring for >> particular operations is the best practice. But it may be necessary to do so >> in order to validate and do certain types of testing to prove or disprove >> certain access patterns. >> -phil >> >> On Thu, Sep 8, 2011 at 7:31 AM, Greg Stein <[email protected]> wrote: >>> >>> Hey, all... >>> >>> After a couple comments on my recent work, and some archaeology on the >>> Python/Riak work over the past month... I've realized that I might >>> have a very different view of RiakTransport compared to what I'm >>> seeing in the current work. I figured it best to bring that to the >>> forefront and discuss: >>> >>> In my view, RiakTransport is used by RiakClient (and others) to "talk >>> to the Riak server". >>> >>> Some of the current work, and some proposed pull requests, seem to >>> take the position that a RiakTransport is "one connection to a server, >>> and the client should manage those". >>> >>> Needless to say, I'm in favor of my own position :-) ... I think it is >>> best to transfer *all* responsibility for talking to the server(s) to >>> the transport layer. I really don't think the client/bucket/object >>> layers should know anything about talking to the server(s). I'd like >>> to see the transport layer be told about all server(s) available, and >>> then it Just Works. >>> >>> I'm still a newbie with all this code, and need to keep plugging away >>> at the higher levels of functionality and compensation for problems. >>> I'd like to build up some code that contacts *one* given server, asks >>> for all of the ring servers, and then opens connections to those >>> servers. And then, it should (automatically) maintain client >>> connections based on what is happening with the Riak cluster. The >>> current (proposed) code manages connections to N servers, but has no >>> automatic add/remove based on changes in the cluster status. I think >>> this happens at a layer *just* above the actual transport. ie. >>> something tracks the changes in the ring status and its servers, and >>> transmits those changes into the transport layer, which alters its >>> communication with that cluster (regardless of whether that >>> communication is via HTTP or protobuf). >>> >>> >>> Okee doke. That's the end of my brain dump and future thoughts on the >>> transport and communication layer. I'd really like some feedback, >>> review, and thoughts. >>> >>> Thanks! >>> -g >>> >>> _______________________________________________ >>> riak-users mailing list >>> [email protected] >>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >> >> > > _______________________________________________ > riak-users mailing list > [email protected] > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > _______________________________________________ riak-users mailing list [email protected] http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
