Hi David,
Thanks for your insightful and quick reply... I'm very optimistic about
Thrift's healthy development community!
The fact that last.fm have chosen Thrift for infrastructure puts it
firmly in the Venn set of 'cool kids'.
Bit of a long one. Fairly in depth technical analysis, and offer of
support, follows...
David Reiss wrote:
First, a quick note about the non-blocking server. It uses nonblocking
I/O, but the actual application callback is called synchronously
(either in the main thread or in a worker thread), so it has to be
synchronous (put another way, the return value for the RPC must be
built before the handler function returns).
Yep, saw *some* of that, but not the critical observation that the
return value must be synchronous. Thanks for pointing that one out right
off.
We'd have to fork TNonblockingServer anyway to integrate it into the
XORP EventLoop. So 'some assembly required'.
That's fine -- XRL has been enough of a sticking point anyway that
radical change is still needed.
We don't use libevent, but rather our own stuff. Some background to why
we do that:
I was funded to port XORP to Microsoft Windows a few years ago, and
this was mostly a success, in that we demonstrated Windows Server 2003
in Redmond handling a full BGP routing table. Sadly this is now
bitrotting due to a lack of further interest and attention from them...
integrating with their IP forwarding requires all sorts of shim DLLs and
service magick.
And imagine my surprise and dismay when I learn that their augmented
POSIX stuff, Subsystem for UNIX Applications (SUA), is about to be axed.
The lack of attention to tools is a Development Limiting Move, to my mind.
EventLoop is a critical path in the XORP framework. It was possible
to leverage the existing design (tied to non-blocking BSD socket I/O)
with a number of stealthy workarounds, without threading the entire code
base, as we rely on the OS's socket buffer behaviour for serialization.
Really, ioctlsocket() with FIONREAD -- and some knowledge about how
Winsock operates in terms of helper thread context switch, which only
became visible under careful observation with Mark Russinovich's Process
Explorer. They don't tell you any of this in the docs -- they expect you
to use Win32 threads.
I daresay I need to go off and look at Boost ASIO now to see if someone
else has done similar.
There is currently no explicit support for doing non-blocking RPC
in Thrift (at least from C++. I'm not sure how the Twisted stuff
works). Someone at Facebook is working on support for this, and
we should be able to release it soonish.
Fingers crossed! Rough guesstimates on timescale would be great.
I do most of my development on FreeBSD-STABLE. If there's anything I
can do to help, please do let me know.
I know for a fact that the FreeBSD port is bitrotted -- it is still
shipping the code with 'facebook::' namespace prior to Apache
Incubation, so that is definitely something I can look at in the meantime.
My understanding is that the Thrift code still uses Boost. If that
is still the case, great. There are things we need to do there anyway --
there are a whole bunch of places where XORP's idioms could be
refactored to use Boost.
XORP has been going as a project for so long (2002), that it's worth
remembering we had to invent a lot of our own stuff, which is probably
better supported now by infrastructure such as Boost and Thrift, which
have more flexible and community-supported idioms. Also, thread
libraries weren't as mature then as they are now.
It is also encouraging that Boost, or parts of it, are making their way
into the ISO C++0x specification. That's what we like -- development
must be sustainable.
In the mean time, one thing
you can do is send your message to a TMemoryBuffer, transport that
to the backend using whatever system you want, then receive the
result from another TMemoryBuffer.
OK. I'll transliterate into XORP-speak here for adaptation (mostly for
my own reference, but, it may interest people...)
looking at
http://cvsweb.xorp.org/cgi-bin/cvsweb.cgi/xorp/ospf/xrl_io.cc?rev=1.54
For example, if you have a function
called "foo":
shared_ptr<TMemoryBuffer> buf(new TMemoryBuffer);
Analogous (without transport hooks) to re-using existing XrlRouter
reference when instantiating Xrl client stub.
e.g. _xrl_router.
(An XrlRouter is an object used to connect to the Finder, XORP XRL's
request broker.)
shared_ptr<MyClient> cl(new MyClient(shared_ptr<TProtocol>(new
TBinaryProtocol(buf)));
Analogous to instantiating Xrl generated client stub e.g.
XrlRawPacket4V0p1Client fea_client(&_xrl_router);
cl->send_foo(args);
Analogous to calling fea_client.send_foo(...), although, WITHOUT the
callback binding.
We use member function pointers a lot.
uint8_t* bytes;
uint32_t nbytes;
buf->getBuffer(&bytes, &nbytes);
// stash buf and cl somewhere.
This is what the XrlRawPacket4V0p1Client._sender instance of XrlSender
is going to be doing.
It binds the callback passed in -- and invokes it when the EventLoop
(Reactor pattern) gets activity on the socket(s) used for XRL transport.
// ship nbytes of bytes off to the server
OK. Understand we need to do our own transport stuff here;
understand that TNonblockingServer comes with limitations for transport
used anyway.
// get the server's response
buf->resetBuffer(response, response_len);
int ret = cl->recv_foo();
This is surprisingly similar to what the XRL client glue does for
marshaling the reply and then dispatching the callback.
I still have some unanswered questions about what we'd do about
replacing the Finder. As I see it, Thrift in its default incarnation
leaves the specifics of transport endpoint up to each application (based
on my reading the simple C++ client/service example in the Thrift SVN
tree), and does not present an ORB as such.
This is analogous to XRL's STCP (*not* SCTP) without service
registration/advertisement. AMQP would most likely deal with some of
that for us -- but in the absence of such integration, we'd need a
lightweight broker mechanism which would work with our chosen
transports. I do wonder if someone has already solved that problem
in/around Thrift.
I noticed that there's a patch for dealing with multiple Thrift servers
here (may be bit-rotted, looks as though it's for the pre-Apache tree):-
http://publists.facebook.com/pipermail/thrift/2007-October/000134.html
We'd probably need something like this. At the moment, the XRL framework
takes multiple XIF definitions and spits out bindings coalesced into one
'target' containing every XIF interface implemented for the software
component in a single C++ class. The methods are abstract virtual so the
class needs to be derived from in order to make sure all the bindings
are implemented for each XRL target (analogy: collection of thrift
services).
Obviously Thrift doesn't do this -- C++ bindings are generated for each
server component -- so -- we would need to be able to tie in multiple
Thrift server-side stubs into the same component. But this is something
we can revisit as we figure out the integration.
One of the questions I can see people will have is: will the Thrift
solution perform slower than the XRL solution? At the moment my gut
feeling is that Thrift is going to be faster at binary marshaling, and
there are problems in XORP's scalability which are better solved using
alternate mechanisms i.e. shared memory arenas and/or batch-mode
messaging for virtualized clusters (AMQP beckons...).
Anyway, more points about where I'm coming from:
I'll keep my ears to the ground meanwhile with AMQP. It shouldn't be
too much hassle to adapt what the txAMQP dude has done for Twisted
Python -- although they have the benefit of fantastic language support
for continuations (yield).
Running Thrift RPC as an AMQP application strikes me as a major step
forward for clustering and virtualization.
We have been looking at stuff like Concurrent C++, but it requires
very ambitious tweaks, essentially it ships its own userland scheduler
-- and that is still subject to stuff like timer aliasing between
userland and kernel. It also isn't sufficiently mature in terms of user
base at the moment.
FreeBSD's KSE M:N threading support would be fantastic here, but
it's currently in a questionable state -- development inertia has set
in, and folk have focused on 1:1 threading as that's what has taken hold.
More as it happens...
thanks muchly,
BMS
P.S. Thanks to Esteve for his reply on current developments in Boost.
ASIO isn't covered in my current Boost book which is most likely in need
of an update. ASIO probably captures some of what's already in libxorp,
open to new ways and improvement.
P.P.S. Thank you for kicking CORBA in the crotch. Much love.
